CSC 724 Advanced Distributed Systems Introduction Instructor: Xiaohui

CSC 724 Advanced Distributed Systems Introduction Instructor: Xiaohui (Helen) Gu Spring, 2018

Logistics Instructor: Helen Gu ([email protected]) – Office: 3274 EB II – Office Hours: 2-3pm Tu/Th TA: Onkar Patil ([email protected]) – Office: TBD – Office hours: by appointment More information – http://courses.ncsu.edu/csc724/lec/001/

Course Goal This class is about learning fundamental concepts and the state of the art research results in distributed systems Target audience: PhD students and MS thesis students!

Course Requirements Prerequisites – CSC 501 (operating systems) – Programming skills in C /Java and Linux What to expect – Programming-intensive projects (a real distributed system that you can be proud of) – Research paper reviews for the first half semester

How will you Learn? Lectures – Canonical distributed system problems – Recent advances in distributed system research Read papers – Learn how to read and write by reading the state-ofthe-art papers Projects – The best way to learn distributed systems to design and implement a cool distributed system by yourself Discussion – Ask questions (important) – Give feedback to your peers on their projects

How will you get an A? Write good paper reviews – Two papers each week for the first half semester – 20% Class participation – Make one paper presentation (10%) – Attend the class and participate discussions (10%) Projects – – – – – – Project Proposal (5%) Proposal Presentation (5%) Project MidReview (5%) Project Demo (15%) Final Presentation (10%) Final Writeup (20%) – a research paper!

Get Project Started NOW Form your team – two or three members per group – Learn to conduct team work! Choose project ideas – See course homepage – Recent conference papers: SOSP/OSDI, NSDI, ICDCS, Middleware, DSN – Talk to the instructor Make appointments by email, Demo environments – Amazon AWS, Google Cloud, VCL

What is a Distributed System?

Some Examples Client-Server The Web The Internet A sensor network DNS Kazaa (peer to peer overlays) Data Center Stock Trading System Cluster Grid Lecture 1-9

Google Technology Layers Services and Applications Web search GMail Ads system Google Maps Distributed systems infrastructure Computing platform Cheap PC Hardware Linux Physical networking

Google’s Map/Reduce Execution

Web Hosting Systems Application software distributed among three types of machines – User machine thin client – Middle-tier server Gateway Convert protocols Merge/integrate results from different data sources – Backend server

Online Dictionary Definition A collection of (probably heterogeneous) automata whose distribution is transparent to the user so that the system appears as one local machine. This is in contrast to a network, where the user is aware that there are several machines, and their location, storage replication, load balancing and functionality is not transparent. Distributed systems usually use some kind of client-server organization. Lecture 1-13

Textbook Definitions A distributed system is a collection of independent computers that appear to the users of the system as a single computer. [Andrew Tanenbaum] A distributed system is several computers doing something together. Thus, a distributed system has three primary characteristics: multiple computers, interconnections, and shared state. [Michael Schroeder] Lecture 1-14

Unsatisfactory Why are these definitions short? Why do these definitions look inadequate to us? Because we are interested in the insides of a distributed system – Design/Algorithms/Protocols – Implementation – Maintenance – Management Lecture 1-15

A working definition for us A distributed system is a collection of entities, each of which is autonomous, programmable, asynchronous and failure-prone, and which communicate through an unreliable communication medium. Entity a process on a host Communication Medium Wired or wireless network Lecture 1-16

Distributed System Design Goals Robustness – is the system resilient to host crashes and failures, and to the network dropping messages? Availability – are data, services always there for clients? Transparency – can the system hide its internal workings from the users? i.e., Operating in such a way as to not be perceived by users. Heterogeneity – can the system handle different types of devices? Lecture 1-17

Distributed Systems Design Goals Concurrency – can the server handle multiple clients simultaneously? Efficiency – is it fast enough? Scalability – can it handle 100 million nodes? (nodes clients and/or servers) Security – can the system withstand hacker attacks? Openness – is the system extensible? Lecture 1-18

Distributed System Example -the Internet intranet ISP backbone satellite link desktop: server: network link: Lecture 1-19

The Internet A vast interconnected collection of computer networks of many types. Intranets – subnetworks operated by companies and organizations. ISPs – companies that provide modem links and other types of connections to users. Intranets are linked by backbones – network links of large bandwidth, such as satellite connections, fiber optic cables, and other highbandwidth circuits. Lecture 1-20

Internet Apps: Their Protocols and Transport Protocols Application e-mail remote terminal access Web file transfer streaming multimedia remote file server Internet telephony Application layer protocol Underlying transport protocol smtp [RFC 821] telnet [RFC 854] http [RFC 2068] ftp [RFC 959] proprietary (e.g. RealNetworks) NFS proprietary (e.g., Skype) TCP TCP TCP TCP TCP or UDP TCP Transmission Control Protocol UDP User Datagram Protocol TCP or UDP typically UDP Implemented via network “sockets”. Basic primitive that allows machines to send messages to each other Lecture 1-21

WWW: the HTTP Protocol HTTP: hypertext transfer protocol WWW’s application layer protocol client/server model PC running – client: browser that requests, Explorer receives, and “displays” WWW objects – server: WWW server stores the website, and sends objects in response to requests Mac running Navigator http1.0: RFC 1945 http1.1: RFC 2068 http http requ est r es p onse est u q p re t e t h ons p s p re t t h Server Running cnn.com Web server Lecture 1-22

The HTTP Protocol: More http: TCP transport service: client initiates a TCP connection (creates socket) to server, port 80 server accepts the TCP connection from client http messages (application-layer protocol messages) exchanged between browser (http client) and WWW server (http server) TCP connection closed http is “stateless” server maintains no information about past client requests aside Protocols that maintain “state” are complex! past history (state) must be maintained if server/client crashes, their views of “state” may be inconsistent, and hence must be reconciled. Lecture 1-23

Distributed Software Systems Approaches: – – – – – Client/server model Centralized control Decentralized control Peer-to-peer Transactions

Client/Server Computing Client machines: single-user PCs/workstations – user-friendly interface Each server provides – shared user services Server enables many clients – to share access to same database – to use high-performance computer system (manage database)

Classes of Client/Server Applications 1. Host-based processing – not true client/server computing – traditional mainframe environment 2. Server-based processing – Server: all processing – User: provides graphical interface 3. Client-based processing – Client: all processing data – Server: validation /database logic 4. Cooperative processing – processing optimized: client/server – complex to set up and maintain

Major Class Topics Basic Concepts – Clock Synchronization/Consensus, Replication Large-scale decentralized systems – Peer to peer, service overlay networks Big data – Google MapReduce, data stream processing Autonomic system management Cloud computing Virtualization Research methodology & Presentation skills