architecture
diod is an I/O forwarding server that implements a variant of the 9P protocol from the Plan 9 operating system. When paired with a modified version of the v9fs Linux 9P client, diod allows a file system to be exported over a TCP/IP network in a manner similar to NFS.
The file system that is exported can itself be NFS or a parallel file system like Lustre or GPFS. This can be done with minimal loss of distributed semantics because the v9fs client (when used with appropriate mount options) has no page or directory cache - all I/O operations trigger a network request. The page cache effectively moves to the server system, with diod appearing as a multi-threaded user application accessing the file system in the usual way through the VFS.
diod was designed to implement I/O forwarding on CHAOS Linux clusters, as depicted above. A subset of the nodes in the cluster mount Lustre and NFS (the I/O nodes) and the rest (the compute nodes) access these file systems using I/O forwarding. diod is tested with a fan-in of about 32:1 (compute nodes to I/O nodes) but other ratios are possible.
The benefits of this architecture on clusters versus directly mounting a parallel file system on every compute node are:
- System overhead on compute nodes such as asynchronous page cache management and the page cache memory footprint is reduced.
- I/O nodes can be more responsive to file system server-initiated activities such as lock revocation than compute nodes since since I/O nodes do not run heavy computation loads
- Per-client state that must be maintained on file system servers is reduced because the client count is reduced.
- The shared I/O node page cache is potentially hot when multiple compute nodes read the same data, e.g. when loading the same executable.