Distributed File System: Design Comparisons II Pei Cao Cisco

Distributed File System: Design Comparisons II Pei Cao Cisco Systems, Inc.

Review of Last Lecture Functionalities of Distributed File Systems Implementation mechanism examples – Client side: Vnode interface in kernel – Communications: RPC – Server side: service daemons Design choices – Topic 1: name space construction Mount vs. Global Name Space – Topic 2: AAA in distributed file systems Kerberos, NTLM

Outline of This Lecture DFS design comparisons continued – Topic 3: client-side caching NFS and AFS – Topic 4: file access consistency NFS, AFS, Sprite, and AFS v3 – Topic 5: Locking Implications of these choices on failure handling

Topic 3: Client-Side Caching Why is client-side caching necessary What are cached – Read-only file data and directory data easy – Data written by the client machine when are data written to the server? What happens if the client machine goes down? – Data that are written by other machines how to know that the data have been changed? How to ensure data consistency? – Is there any pre-fetching?

Client Caching in NFS v2 Cache both clean and dirty file data and file attributes File attributes in the client cache are expired after 60 seconds File data are checked against the modified-time in file attributes (which could be a cached copy) – Changes made on one machine can take up to 60 secs to be reflected on another machine Dirty data are buffered on the client machine till file close or up to 30 seconds – If the machine crashes before then, the changes are lost – Similar to UNIX FFS local file system behavior

Implication of NFS v2 Client Caching Data consistency guarantee is very poor – Simply unacceptable for some distributed applications – Productivity apps tend to tolerate such loose consistency Different client implementations implement the “prefetching” part differently Generally clients do not cache data on local disks

Client Caching in AFS Client caches both clean and dirty file data and attributes – The client machine uses local disks to cache data – When a file is opened for read, the whole file is fetched and cached on disk Why? What’s the disadvantage of doing so? However, when a client caches file data, it obtains a “callback” on the file In case another client writes to the file, the server “breaks” the callback – Similar to invalidations in distributed shared memory implementations Implications: file server must keep states!

AFS RPC Procedures Procedures that are not in NFS – Fetch: return status and optionally data of a file or directory, and place a callback on it – RemoveCallBack: specify a file that the client has flushed from the local machine – BreakCallBack: from server to client, revoke the callback on a file or directory What should the client do if a callback is revoked? – Store: store the status and optionally data of a file Rest are similar to NFS calls

Failure Recovery in AFS What if the file server fails – Two candidate approaches to failure recovery What if the client fails What if both the server and the client fail Network partition – How to detect it? How to recover from it? – Is there anyway to ensure absolute consistency in the presence of network partition? Reads Writes What if all three fail: network partition, server, client

Key to Simple Failure Recovery Try not to keep any state on the server If you must keep some states on the server – Understand why and what states the server is keeping – Understand the worst case scenario of no state on the server and see if there are still ways to meet the correctness goals – Revert to this worst case in each combination of failure cases

Topic 4: File Access Consistency In UNIX local file system, concurrent file reads and writes have “sequential” consistency semantics – Each file read/write from user-level app is an atomic operation The kernel locks the file vnode – Each file write is immediately visible to all file readers Neither NFS nor AFS provides such concurrency control – NFS: “sometime within 30 seconds” – AFS: session semantics for consistency

Session Semantics in AFS What it means: – A file write is visible to processes on the same box immediately, but not visible to processes on other machine until the file is closed – When a file is closed, changes are visible to new opens, but are not visible to “old” opens – All other file operations are visible everywhere immediately Implementation – Dirty data are buffered at the client machine until file close, then flushed back to server, which leads the server to send “break callback” to other clients – Problems with this implementation

Access Consistency in the “Sprite” File System Sprite: a research file system developed in UC Berkeley in late 80’s Implements “sequential” consistency – Caches only file data, not file metadata – When server detects a file is open on multiple machines but is written by some client, client caching of the file is disabled; all reads and writes go through the server – “Write-back” policy otherwise Why?

Implementing Sequential Consistency How to identify out-of-date data blocks – Use file version number – No invalidation – No issue with network partition How to get the latest data when read-write sharing occurs – Server keeps track of last writer

Implication of “Sprite” Caching Server must keep states! – Recovery from power failure – Server failure doesn’t impact consistency – Network failure doesn’t impact consistency Price of sequential consistency: no client caching of file metadata; all file opens go through server – Performance impact – Suited for wide-area network?

Access Consistency in AFS v3 Motivation – How does one implement sequential consistency in a file system that spans multiple sites over WAN Why Sprite’s approach won’t work Why AFS v2 approach won’t work Why NFS approach won’t work What should be the design guidelines? – What are the common share patterns?

“Tokens” in AFS v3 Callbacks are evolved into 4 kinds of “Tokens” – Open tokens: allow holder to open a file; submodes: read, write, execute, exclusive-write – Data tokens: apply to a range of bytes “read” token: cached data are valid “write” token: can write to data and keep dirty data at client – Status tokens: provide guarantee of file attributes “read” status token: cached attribute is valid “write” status token: can change the attribute and keep the change at the client – Lock tokens: allow holder to lock byte ranges in the file

Compatibility Rules for Tokens Open tokens: – Open for exclusive writes are incompatible with any other open, and “open for execute” are incompatible with “open for write” – But “open for write” can be compatible with “open for write” --- why? Data tokens: R/W and W/W are incompatible if the byte range overlaps Status tokens: R/W and W/W are incompatible Data token and status token: compatible or incompatible?

Token Manager Resolve conflicts: block the new requester and send notification to other clients’ tokens Handle operations that request multiple tokens – Example: rename – How to avoid deadlocks

Failure Recovery in Token Manager What if the server fails What if a client fails What if network partition happens

Topic 5: File Locking for Concurrency Control Issues – Whole file locking or byte-range locking – Mandatory or advisory UNIX: advisory Windows: if a lock is granted, it’s mandatory on all other accesses NFS: network lock manager (NLM) – – – – NLM is not part of NFS v2, because NLM is stateful Provides both whole file and byte-range locking Advisory Relies on “network status monitor” for server monitoring

Issues in Locking Implementations Synchronous and Asynchronous calls – NLM provides both Failure recovery – What if server fails Lock holders are expected to re-establish the locks during the “grace period”, during which no other locks are granted – What if a client holding the lock fails – What if network partition occurs

Wrap up: Comparing the File Systems Caching: – NFS – AFS – Sprite Consistency – – – – NFS AFS Sprite AFS v3 Locking

Wrap up: Comparison with the Web Differences: – Web offers HTML, etc. DFS offers binary data only – Web has a few but universal clients; DFS is implemented in the kernel Similarities: – Caching with TTL is similar to NFS consistency – Caching with IMS-every-time is similar to Sprite consistency As predicted in AFS studies, there is a scalability problem here Security mechanisms – AAA similar – Encryption?