SoftNAS® on AWS: design & configuration guide.
This design & configuration guide provides step-by-step instructions for the design, configuration, and installation of SoftNAS Cloud NAS on Amazon Web Services (AWS), including selecting the optimum AWS EC2 instance size and storage backend (S3 and/or EBS).
You will need a valid license for SoftNAS Cloud NAS. A license is obtained in one of the following ways:
- Purchase on AWS Marketplace
- Purchase BYOL license keys
- Start a 30 day free trial
Allow 60-90 minutes to complete this guide.
Watch AWS Overview Demo Video
Considerations and Reference Architecture
When designing a file storage system on Amazon EC2, the performance depends on many factors including:
- The Amazon EC2 instance type
- The number and configuration of Amazon Elastic Block Store (Amazon EBS)/S3 volumes
- The type of Amazon EBS volume utilized and the application workload
Review the SoftNAS Cloud NAS Architecture on AWS whitepaper
To help with design, we have created a detailed guide that addresses architectural considerations when deploying SoftNAS Cloud NAS on AWS.
The guide provides:
- Network, Application and Data Security
- Instance to Workload suggestions
- Use Case Recommendations for Disk types
- Data Protection and Redundancy
For best performance results match workload to instance:
M4.2xlarge or equivalent
A good starting point for memory and CPU resources. This instance can handle processing and caching with minimal requirements for network bandwidth.
M4.4xlarge or equivalent
Larger memory-based read cache is beneficial for read-intensive workloads. Additional CPU cores provide better performance when deduplication, compression, and/or RAID is enabled.
M4.10xlarge or equivalent
Suited to heavier workloads that require very high-speed network connection due to the amount of data transferred over the connection. This level of VM also provides much more storage, CPU, and memory capacity
Select Your Backend Storage: Amazon S3 vs EBS
The choice of AWS EBS vs. S3 storage for use case is determined by the performance needed by the backend application. EBS magnetic or SSD backed storage along with S3 object storage have benefits and limitations that work with or against certain workloads. SoftNAS Cloud NAS using Amazon S3 storage can scale cloud storage to practically unlimited capacity. You can provision each cloud disk to hold up to four petabytes (PB) of data. If a larger data store is required, you can use RAID to aggregate multiple cloud disks. When you use a cloud disk, we do suggest using an ephemeral or block device local to the SoftNAS Cloud NAS virtual appliance as a read cache to reduce Amazon S3 I/O charges and improve IOPS and performance for read-intensive workloads. S3 cloud disk is suitable for archive or general access applications where low cost is important but not suitable for applications that need high IOPS performance and very high throughput on random read/write workloads.
- SoftNAS Cloud NAS using Amazon S3 scales to petabytes.
- Adding read/cache will reduce Amazon S3 I/O charges and improve IOPS and performance
- S3 Cloud disk is suitable for archive or General access use cases, but not suitable for use cases that need high IOPS and high throughput on random read/write workloads.
- SoftNAS Cloud NAS using Amazon EBS volumes provides persistent block level storage
- RAIDing or striping the volumes will provide a linear performance increase (Up to the maximum capabilities of the host instance).
- There is an EBS option for every use case from Cold HDD (sc1) and Throughput Optimized HDD (st1) to SSD backed General Purpose (gp2) and Provisioned IOPS (io1) volumes.
Select Your EBS Volume Type
Magnetic-backed volume types support higher block sizes up to 1,024 KB. Throughput Optimized HDD (st1) and Cold HDD (sc1) Amazon EBS volume types are based on magnetic storage technology. The Throughput Optimized HDD (st1) volume type is designed for sequential read/write workloads (e.g., Big Data). It can achieve very high throughput (500 MB/s) for sequential read/write workloads (compared to 160 MB/s and 320 MB/s for SSD-backed gp2 and io1, respectively). Generally, Big Data workloads operate on very large sequential datasets and generate data for storage in a similar way. The st1 volume type has a baseline performance of 40 MB/s per terabyte (TB) of allocated storage and, like gp2, can burst beyond the baseline performance for a short period of time. The Cold HDD (sc1) volume type is designed for high density and infrequent access workloads.
This volume type is suitable for cold storage (infrequent access) applications where low cost is important. Unlike st1, the baseline performance of an sc1 volume is 12 MB/s per TB of allocated storage. It’s important to note that Amazon S3 achieves high availability (HA) by default within a single region, whereas sc1 volumes have to be mirrored across Availability Zones to achieve parity with Amazon S3 in durability and availability of the data (this doubles and triples the cost of sc1 when compared to Amazon S3.) Nevertheless, depending on certain access patterns (e.g., cold versus warm) of the data, the cost of sc1 volumes can be cheaper for certain workloads.
General Purpose (gp2) and provisioned IOPS (io1) SSD volumes can achieve faster IOPS performance and very high throughput on random read/write workloads when compared to magnetic disks, but at a higher price point. However, gp2 and io1 volume types are limited to a throughput of approximately 320 MB/s (160 MB/s for gp2, 320 MB/s for io1). General Purpose (gp2) volumes provide a fixed 1:3 ratio between gigabytes and IOPS provisioned, so a 100 GB General Purpose volume provides a baseline of 300 IOPS. Gp2 volumes less than 1 TB in size can also burst for short periods, up to 3,000 IOPS. You can provision General Purpose volumes up to 16 TB and 10,000 IOPS. Provisioned IOPS (io1) volumes are intended for workloads that demand consistent performance, such as databases. You can create provisioned IOPS volumes up to 16 TB and 20,000 IOPS.
Over a year, Amazon EBS Provisioned IOPS volumes are designed to deliver within 10 percent of the provisioned IOPS performance 99.9 percent of the time. There are differences in total throughput capabilities between Provisioned IOPS (io1) and General Purpose SSD (gp2) volumes. Io1 volumes are designed to provide up to 320 MB/second of throughput while gp2 volumes are designed to provide up to 160 MB/second.
Reference Architecture with HA
Complete installation pre-requirements
Select Your AWS EC2 Instance
SoftNAS Cloud NAS runs on a variety of AWS EC2 instances. Select the optimal EC2 instance by reviewing the SoftNAS Cloud NAS Instance Size Guidance page.
Setup and configure AWS infrastructure
Configure AWS VPCs
If you’ve already set up your AWS environment, please skip to the next section (Configure AWS IAM Policy and Role). Amazon Web Services maintains multiple scenarios for creating VPCs with a step-by-step walkthrough
Use the VPC wizard to create a VPC for running a public-facing web application, while still maintaining non-publicly accessible back-end servers in a second subnet.
Use the VPC wizard to create a VPC for extending your data center into the cloud, and also directly access the Internet from your VPC.
Use the VPC wizard to create a VPC for extending your data center into the cloud, and leverage Amazon’s infrastructure without exposing your network to the Internet.
Use the AWS CLI to create a VPC and and a public and private subnet.
Use the AWS CLI to create a VPC with an associated IPv6 CIDR block, and a public and private subnet each with an associated IPv6 CIDR block.
Assign a public IP address to your EC2 instance so you can access it over the internet (for administration)
Configure AWS IAM Rules
Once you have your VPC created we need to define a SoftNAS Cloud NAS specific IAM policy and attach that policy to a specific role. Login to your AWS console and navigate to ‘Identity and Access Management.’ Now, follow the steps to create your policy and role here: Creating the IAM Role for SoftNAS Cloud NAS®. Please note, when creating the IAM Role, it is case sensitive and MUST be named “SoftNAS_DISK_IAM” (without the quotes).
Create and Configure an Instance in AWS
Now that you have a SoftNAS Cloud NAS IAM disk policy attached to the SoftNAS_DISK_IAM role, we can install the SoftNAS Cloud NAS AMI on an EC2 instance.
A text-based walkthrough
A detailed walk-through of the steps required to create and configure your new SoftNAS Cloud NAS instance can be found here: Configuring Your AWS Instance. Pay special note that you must assign the IAM role you previously create (SoftNAS_DISK_IAM) to this EC2 instance before SoftNAS Cloud NAS will function correctly.