Joyent ZFS or Amazon EBS

Courtesy:  Steffen Opel via ServerFault

The ZFS based storage system from Joyent is a local storage system designed to deliver carrier-grade storage and data reliability, see Data Resiliency and Reliability:

We put ZFS on top of a high performance local storage subsystem to ensure that your data is safe, consistent, and always accessible and recoverable. ZFS is a combined file system and logical volume manager designed for pooled local storage. Unlike other file systems deployed for cloud storage, ZFS’ copy-on-write capability guarantees that your image will not be lost. [emphasis mine]

In contrast, EBS is a network block storage system designed to provide highly available, highly reliable storage volumes that can be attached to a running Amazon EC2 instance and exposed as a device within the instance, see section Features of Amazon EBS volumes within Amazon Elastic Block Store (EBS) for details, e.g.:

Amazon EBS volumes are placed in a specific Availability Zone, and can then be attached to instances also in that same Availability Zone.
Each storage volume is automatically replicated within the same Availability Zone. This prevents data loss due to failure of any single hardware component.
Amazon EBS also provides the ability to create point-in-time snapshots of volumes, which are persisted to Amazon S3. These snapshots can be used as the starting point for new Amazon EBS volumes, and protect data for long-term durability. […]

[emphasis mine]

The latter point highlights that EBS does not store its data on S3 in itself, rather provides an easy to use backup mechanism for long-term durability via S3, which implies you will need to assess both scenarios separately in terms of durability and availability though.

Section Amazon EBS Volume Durability further details this architecture:

[…] Amazon EBS volume data is replicated across multiple servers in an Availability Zone to prevent the loss of data from the failure of any single component. The durability of your volume depends both on the size of your volume and the percentage of the data that has changed since your last snapshot. […]

Because Amazon EBS servers are replicated within a single Availability Zone, mirroring data across multiple Amazon EBS volumes in the same Availability Zone will not significantly improve volume durability. However, for those interested in even more durability, Amazon EBS provides the ability to create point-in-time consistent snapshots of your volumes that are then stored in Amazon S3, and automatically replicated across multiple Availability Zones. […]

[emphasis mine]

So while EBS stores data on multiple servers in one availability zone only, S3 provides the extra mile of physical infrastructure separation, see How isolated are Availability Zones from one another?:

Each availability zone runs on its own physically distinct, independent infrastructure […]. Common points of failures like generators and cooling equipment are not shared across Availability Zones. Additionally, they are physically separate, such that even extremely uncommon disasters such as fires, tornados or flooding would only affect a single Availability Zone. [emphasis mine]

This yields a claimed durability of 99.999999999% as outlined in How durable is Amazon S3? and further detailed in How is Amazon S3 designed to achieve 99.999999999% durability?:

Amazon S3 redundantly stores your objects on multiple devices across multiple facilities in an Amazon S3 Region. […] When processing a request to store data, the service will redundantly store your object across multiple facilities before returning SUCCESS. […] [emphasis mine]

Please note, that an availability zone is still constraint to a single region (see Using Regions and Availability Zones for details on this architecture), and their have been respective incidents already, triggering discussions whether region and/or provider redundancy is the way to go for utmost reliability (see Outages below).

Finally, section Amazon S3 Data Consistency Model in Amazon S3 Concepts provides more details how Amazon S3 achieves high availability by replicating data across multiple servers within Amazon’s data centers.


Outages

Both services had at least one major outage in the past – the respective post mortem analysis provides additional insight into the design of each system and allow you to account for this in backup and availability strategies accordingly:

Joyent – Further Strongspace and BingoDisk Update
Amazon – Summary of the Amazon EC2 and Amazon RDS Service Disruption in the US East Region

section Overview of EBS System features an insightful summary of the EBS architecture

The latter outage sparked quite some discussion regarding reliability of cloud computing in general, which interestingly triggered the article Magical Block Store: When Abstractions Fail Us on Joyent’s blog, exploring the differences between both approaches and explaining Joyent’s respective architectural choices (including self-criticism of former failed attempts); while this article obviously might be considered biased, it should still allow you to draw your own conclusions in turn.

Advertisements