AWS EFS: Understanding and Optimizing Elastic File System Performance

AWS Elastic File System (EFS) is a popular fully-managed file storage service that provides simple, scalable, and elastic file storage for use with AWS compute services and on-premises servers. It offers a traditional file system interface and hierarchical structure with the scalability and durability of the AWS cloud.

While EFS is designed to be easy to use, achieving optimal performance requires an understanding of some key concepts, especially around throughput and burst credits. In this article, we‘ll dive deep into how EFS works, share expert tips for getting the most out of it, and help you decide if it‘s the right storage solution for your use case.

Overview of AWS EFS

EFS is one of the three main storage services offered by AWS, alongside Elastic Block Store (EBS) and Simple Storage Service (S3). Some of the key features that set EFS apart include:

Shared file access – EFS file systems can be accessed concurrently by up to 1000s of EC2 instances, from multiple Availability Zones (AZs) in a region. Instances can also access EFS over VPN and Direct Connect, enabling hybrid cloud deployments. This shared access simplifies running workloads and applications across many compute resources.

Elastic and automatic scaling – With EFS, there is no need to provision storage in advance. EFS file systems grow and shrink automatically as files are added and removed, from GBs up to petabytes in size. You only pay for the storage used.

High performance – EFS is designed to provide consistent low latencies and high levels of throughput and IOPS. Throughput scales with the amount of data stored and can reach up to 10+ GB/s and 500,000 IOPS.

Strong consistency and durability – EFS is designed for 11 9‘s of durability and provides strong read-after-write consistency. Data is redundantly stored across multiple AZs.

Integration with AWS services – EFS natively integrates with other AWS services like EC2, ECS, Lambda, and AWS Backup for simplified operations and consolidated billing on the AWS platform.

Understanding EFS Burst Credits

One of the most important concepts to understand about EFS performance is burst credits. EFS uses a credit system to allocate its base and burst throughput performance.

Baseline (permitted) throughput – Every EFS file system (in Standard storage class) receives a baseline rate of 50 KB/s per GB of storage (equivalently 0.05 MB/s per GB).

Burst throughput – To support occasional periods of higher throughput needs, EFS allows file systems to burst up to 100 MB/s of throughput regardless of file system size.

The amount of time a file system can burst is determined by the number of burst credits it has. EFS file systems earn burst credits over time at a rate of 50 KB/s per GB of storage. Burst credits are consumed when file system throughput exceeds the baseline rate, and they regenerate when throughput is below the baseline rate.

For example, a 100 GB EFS file system has a baseline throughput of 5 MB/s (100 GB x 0.05 MB/s per GB) and accumulates burst credits at a rate of 5 MB/s. It can sustain 100 MB/s burst throughput for up to 20 seconds at a time (100 MB/s / 5 MB/s) before burning through its credits.

Importantly, a new empty EFS file system starts with 2.1 GB of initial burst credits, enough to burst at 100 MB/s for 7.2 minutes (2.1 GB / 100 MB/s). Once these initial credits are used, the file system reverts to its baseline performance until it accumulates new credits based on its storage size.

Monitoring burst credits is critical to avoiding unexpected performance drops. You can track burst credit balances using Amazon CloudWatch and set up alarms to alert when credits fall below a threshold. This can help you proactively manage performance and possibly increase storage usage to get more baseline throughput.

Tips for Optimizing EFS Performance

Beyond burst credits, there are several other ways to optimize EFS performance:

1. Choose the right performance mode – EFS offers two performance modes: General Purpose and Max I/O. General Purpose is the default and recommended for most workloads. It provides lower per-operation latencies. Max I/O mode optimizes for higher aggregate throughput vs operation latency. It does this in part by increasing the initial read size from 1 MB to 16 MB. Use Max I/O if you have very high levels of aggregate throughput and your applications can tolerate higher per-operation latencies.

2. Use EFS for the right workloads – EFS is ideal for workloads that require shared file storage that can be concurrently accessed by 100s or 1000s of compute instances. Good examples are content management systems, web serving, and home directories. It is not well-suited for workloads with very high IOPS per GB or per-operation latency requirements, or that have large numbers of small files. Other storage options like Amazon EBS io2 volumes may be better in those cases.

3. Schedule backups and data transfers outside of peak usage – To conserve burst credits for when you most need them, try to schedule file system backups and large data transfers during off-peak hours if possible. You can configure a Network File System (NFS) backup to EFS itself, to Amazon S3, or use the AWS Backup service.

4. Use provisioned throughput mode for greater performance – For applications that require high levels of sustained throughput, you can use EFS provisioned throughput mode. This enables you to provision the throughput your application needs independently of the amount of data stored in the file system. Throughput can be provisioned in increments of 1 MB/s.

5. Enable EFS Intelligent-Tiering – EFS offers an Intelligent-Tiering storage class that automatically moves files to the lower-cost Infrequent Access (IA) storage class based on a default or custom access pattern policy. This can significantly reduce storage costs with minimal performance impact. Access to data remains transparent to applications.

EFS vs EBS vs S3 – Which to Choose?

The choice between AWS‘s three main storage services ultimately depends on your specific use case and requirements. Here are some general guidelines:

Use EFS when:

• You need shared file storage accessible from Linux instances
• Your application requires high levels of aggregate throughput and IOPS
• You have a lot of data or expect your data to grow significantly
• You want to use standard file system semantics and hierarchical structure

Use EBS when:

• You need block storage volumes attached to EC2 instances
• Your workload requires the lowest possible per-operation latency
• You need support for Windows instances in addition to Linux
• You want automated snapshots and the ability to detach/reattach volumes

Use S3 when:

• You want the cheapest storage option for large amounts of data
• Your data consists of large objects or files rather than blocks
• You don‘t need file system semantics or hierarchical structure
• You need a range of storage classes for infrequently accessed data
• You want to access data directly via API without going through compute

Of course, many real-world applications require more than one type of storage. You might use EFS for shared application files, EBS for high-performance database volumes, and S3 for mass data storage, backups, and archives. Choosing the right tool for the job will help you strike the optimal balance of performance, scalability, and cost.

Conclusion

AWS EFS provides a powerful and elastic shared file storage solution for the cloud. To get the most out of it, it‘s important to understand key concepts like burst credits and baseline throughput and how they relate to the amount of data stored.

Monitoring your burst credit balance is critical to avoiding unexpected drops in performance. You can further optimize EFS by choosing the most appropriate performance and throughput modes for your workload, scheduling backups and transfers intelligently, and enabling cost-saving features like Intelligent-Tiering.

When considering EFS, evaluate it alongside other storage options like Amazon EBS and S3. EFS is ideal for use cases that require shared Linux file systems that can scale to petabytes in size and be concurrently accessed by 1000s of instances.

By following the tips and best practices covered in this article, you can maximize the performance, scalability, and cost-efficiency of your EFS deployments. With the right architecture and optimizations, EFS can provide a robust storage foundation for your most demanding cloud workloads.