Approximate comparisons with other interconnects

SATAnet bandwidths will increase as newer, faster Serial ATA standards arise. 450MByte/s and 600MByte/s bandwidths are planned. In addition, multiple ports per node can be utilised, allowing greater per-node bandwidth limited only by the motherboard chipset capabilities and/or PCI/PCI-E SATA expansion cards.

Distributed storage

• Allows abstraction of storage.
• Mount any data (e.g. the world-wide web) as a normal disc and filesystem.
• Manipulate data with “intelligent storage” (e.g. encryption).
• Share memory amongst nodes.

SATAnet makes it possible to provide complete abstraction of any data storage and present it through the SATA interface. For example, the entire world-wide web could be made available as an apparent disc and mounted as a sub-directory, e.g. as /mnt/www with websites accessed as local files. A particularly interesting possibility opened up by this technology is one of bootable websites, whereby a PC can boot up directly from a software vendor’s website over the internet as if from a local disc, revolutionising the distribution, security, and licencing models of operating systems and applications.

Similarly, “intelligent storage” can be implemented, e.g. transparent encryption and compression, RAID, on-site/off-site backups, etc.

Other benefits include: large (and fast) swap files on discless nodes; secure swap files — the underlying storage medium is abstract and not contained within the node; memory can be easily shared amongst nodes by mapping it to a block of sectors or file.

In-flight processing

• Relieves load from the CPU.
• Run-time configurable transformations.

Communication packets can be passed through programmable transformation units within the SATAnet router. This allows the data within a packet to be processed, either to change, filter, or expand the data within the packet or to compute additional data such as checksums.

Algorithmic control

• Relieves load from the CPU.
• Reduces network traffic.
• Removes coding from the algorithm implementation.

Some algorithms require many synchronisations to ensure that data is ready prior to starting execution or it is desireable to run parts in lock-step. Normally this will be implemented with software semaphores and synchronisation primitives with much communication going between nodes. SATAnet can reduce the costs and complexities incurred by such algorithms by itself co-ordinating the synchronisations and execution scheduling, removing the overhead of software control.

Default mode

• Operates as a normal local SATA disc.

The SATAnet router appears as a normal local hard disc to each node it is connected to. If the router has a physical disc installed then this disc is seen by the nodes otherwise the router appears as a blank disc.

Packet-switched mode

• High-bandwidth network communication.
• Co-exists with storage and filesystems.

This is the most flexible networking mode. Data is encapsulated into packets and sent from one node to another using normal SATA protocols. Specifically, reading or writing a packet is simply a case of reading or writing the relevant number and location of sectors.

Both network communication and storage access can co-exist and be freely intermixed over the same SATA interfaces and the networking does not interfere with filesystem operation.

Circuit-switched mode

• Ultra-low latency transfers between applications.

This mode is intended for use for short periods of time between two or more nodes when ultra-low latency communication is desireable. The contents of a variable in a userspace program on one node can be transferred to a variable in a userspace program on another node in under 1µs.

Whilst in this mode, a path through the SATAnet is reserved and so general communication and storage access by other nodes will have temporarily higher latencies.

Distributed storage mode

• Allows abstraction of storage.
• Mount any data (e.g. the world-wide web) as a normal disc and filesystem.
• Manipulate data with “intelligent storage” (e.g. encryption).

This mode co-exists with the other modes and allows SATA commands to be transferred elsewhere for processing, effectively allowing a processing node (normally a SATA host) to behave as a SATA device.

This is a very flexible mode as it allows complete abstraction of any data storage and present it through the SATA interface. For example, the entire world-wide web could be made available and mounted as a sub-directory, e.g. as /mnt/www with websites accessed as local files.

Similarly, “intelligent storage” can be implemented, e.g. transparent encryption and compression, RAID, on-site/off-site backups, etc.

Other benefits include: large (and fast) swap files on discless nodes; secure swap files — the underlying storage medium is abstract and not contained within the node; memory can be easily shared amongst nodes by mapping it to a block of sectors or file.

Networking Characteristics

Bandwidth

• Can determine the dataset sizes for batch-style parallelism — effectively a latency
• Can determine the processing speed for stream-based parallelism
• Low bandwidth is acceptable with highly-independent parallelism
• High bandwidth is beneficial for highly-dependent parallelism

A node can be connected to one or more SATAnet routers and to one or more ports per router. Using multiple ports allows for multiplication of the bandwidth and also lower average latency.

The following table lists bandwidths for packet transfers of varying sizes on a SATA-I interface.

Latency

• Can determine the granularity of the parallelism
• Low-latency is important in synchronous or poorly-parallel applications
• High-latency is tolerable in asynchronous and highly-parallel applications
• Set-up latencies can be offset by merging packets
• In core-rich nodes one core can be allocated to communications to minimise blocking

Packet-switched latencies

The following table lists timings for a packet to transfer through various parts of a Linux-based system.

Transmission on one node is asynchronous to reception on another node, and so the latencies overlap. In practice this means that the overall latency is lower than the simple summation of each part of the transfer. Since the latencies are roughly symmetrical for both transmission and reception, the typical overall latency is T_tad + T_tdh + T_net = 62µs.

Circuit-switched latencies

The following table lists the approximate timings for a 4-byte word to transfer through various parts of a Linux-based system.

Total end-to-end latency is simply T_tah + T_net + T_rha = 0.8µs. The bandwidth achieved for back-to-back transfers of 4-byte words is around 8~12Mbytes/sec.

Storage Characteristics

The following three commands were run under Linux over a SATA-I interface, giving the results shown.

# hdparm -Tt /dev/sdb /dev/sdb: Timing cached reads: 2256 MB in 2.00 seconds = 1127.62 MB/sec Timing buffered disk reads: 178 MB in 3.01 seconds = 59.04 MB/sec

# dd if=/dev/sdb of=/dev/null bs=4096 count=4096 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 0.242752 seconds, 69.1 MB/s

# dd if=/dev/zero of=/dev/sdb bs=4096 count=4096 4096+0 records in 4096+0 records out 16777216 bytes (17 MB) copied, 0.262695 seconds, 63.9 MB/s