Verifying Snapshots on SoftNAS for Compliance for Halliburton

Verifying Snapshots on SoftNAS for Compliance for Halliburton

As we grow and evolve our Fuusion product, we are constantly finding new use cases, and learning to better respond to customer requirements. Halliburton is one of Buurst’s earliest customers, using our SoftNAS product long before the advent of Fuusion. So, when Halliburton came to us looking for a solution related to our SoftNAS product, we rose to the challenge. And because our solution to Halliburton’s problem involved both of our products, it seemed the perfect opportunity to illustrate at a high level how Fuusion was used to meet their needs.

Halliburton operates over 200 SoftNAS servers across their infrastructure, leveraging both AWS and Azure cloud storage. For compliance purposes, Halliburton needed to ensure and document that for each SoftNAS instance or VM, snapshots were being performed as scheduled. This needed to be performed in an automated fashion, with minimal manual effort across all the 200 virtual machines. This solution was of double interest to us because it allowed us to not only prove the flexibility of our Fuusion product, but to verify that SoftNAS’ built-in snapshot solution operated as intended across a large deployment.

In order to show how our solution works, we created a POC on a smaller scale.

We hope this smaller scale deployment proves an ideal introduction to our Fuusion product in operation.

Identifying Hosts

The first step to verifying each of the 200 SoftNAS deployments operated as intended, is of course to identify the hosts. This proved an easy task, as all it required was compiling a CSV list of all IP addresses or host names. To verify our flow, we started with a small sample size of five IP addresses, two working IP addresses based on a sample environment, and 3 non-existent IP addresses to simulate failures. In Halliburton’s production environment this list would compile the IPs or hostnames of each SoftNAS deployment in their environment.

This CSV is fed into the first processor of the Fuusion flow. The processor, named Get_host_list, obtains files from each live host via the IP addresses or hostnames provided, and a python script running in the background. The processor script grabs snapshot details from each live instance.

Splitting the Records 

Next, the records need to be split on a per instance/VM basis, into successful connections and failures. This is done via a processor we called simply “SplitRecord”. This processor took the data from these instances, existent and non-existent, and created records with the same file name, but each with a different host (UUID) record.

Looking into one of the flowfiles and its attributes, we can see many different attributes that can be called upon in later steps, as necessary. For our purposes, the attribute we are most interested in is the fragment count. The fragment count attribute tells Fuusion that there are 5 different fragments to this single record. Knowing this, and the UUIDs, allows Fuusion to determine the fragments belonging to the record, and allows them to be re-assembled upon request. 

Execute String 

This step verifies the output of each SoftNAS connection, and the data requested from each by the python script in the processor mentioned earlier. As you can see, this record has successfully pulled the hostname, platform, volume name, snapshot count, the age of the last snapshot (LAST_AGE), whether snapshots are enabled, on what schedule, and how long they are retained, for each volume on the instance. The volumes are listed below, with values provided for each of the variables above. 

If the connection fails, on the other hand, this data is not available. So, for the three non-existent IP addresses (purposely created to show how to handle failed connections as you will recall), we instead see execution_status_0 in the Flowfile attributes, indicating a failed connection. This attribute allows us to sort the failures separately, as we will see shortly. 

Update Attribute Step

The Update Attribute processor’s job is simply to rename the files based on success or failure. Successful connections with the data pulled from valid servers are re-named AWS_SNAPSHOT_RESULTS.

The failed nodes (based on the execution_status_0 attribute mentioned earlier), are renamed AWS_ERROR_HOSTS. Remember, even though we have the same filename applied to each file in each category, they are still differentiated by separate UUIDs, and can still be recompiled based on the fragment count attribute.

“Notify” and “Wait”

In typical configurations, Fuusion flows are not configured to perform batch operations. But as you will see, with some creativity and ingenuity, Fuusion is flexible enough to manage just about anything. To manage this requirement, we needed to leverage some pre-existing controller services in a creative fashion, notably a distributed map cache, similar to services such as DynamoDB or Reddis – anything with a key value store. To put it simply, the key is something we need to count, and the value is that count. The Notify processor tells us about that count (the count being the successes or failures to be sorted). The signal identifier is a made up value simply called ‘release’. The signal counter is a key called ‘process’ record, and each process_record will increase the count by an increment of 1. Each increment is then stored in the Distributed Map Cache.

The coolest part of this is that there was no need to set up a separate service such as DynamoDB or Reddis. We were able to leverage the rich variety of controller services already present to create our own solution.

With the “Wait” processor, we are essentially telling the flow when to proceed further, ie, when to run the batch process, by listening for the “release” signal identifier. The Wait processor is going to find that fragment_count attribute mentioned earlier, and look for records from Notify until the counter reaches the value specified by the fragment_count, which we know to be five. Five fragments come into Notify, and five go out, split based on defined variables, all automated in the flow to this point.

Route on Attribute 

So, with Notify and Wait, we tell the flow when to proceed. We’ve split the fragments apart and have attributes labelling them on success and failure, and now we are ready to begin putting them back together. The first thing we need to do is to merge successes together, and merge failures together. We do this by sending the fragments in different directions within the flow, using the Route on Attribute processor. This is done quite simply by sending the fragments in either direction based on a value we’ve seen before, the execution_status. With a simple NIFI Expression Language command, the files are sorted and split to two basically identical processors called MergeRecord 

Fragments (Flowfiles) with an execution_status of 1 are sent to the right (successes). As you can see, 2 files have been sent to the MergeRecord processor, and as we know, 2 of the five IP addresses corresponded to live SoftNAS virtual machines. 

Those with an execution_status of 0 (or failed connections) are sent to the left. As you can see, there are 3 fragments sent to the Merge Record Processor, corresponding to the 3 invalid IP addresses.  

Each of these MergeRecord processors compile the fragments together, successes put together in one, and failures in the other. Finally, the PutFile processor creates a single file out of the merged records, preparing it to be sent out to a shared storage location, in this case an S3 repository.  

So, with this flow, we were able to suck raw data up from a given source (in this case SoftNAS), assign attributes to split and separate the desired data, and tell it where to go, then put it back together in the desired format, all before hitting a central repository.  

Once sent out to S3, it can be retrieved by another flow, to apply additional formatting if necessary, such as formatting it into a CSV or other file format ready for consumption like the one below 

This CSV output, fully automated and tailored to the needs of the client in a standardized format, or a considerably larger one containing all 200 of Halliburton’s servers and their current snapshot status can then be input directly into any business intelligence tool you specify.   

Remember that while this is a very simple example, the same principles can apply to data from any source, and can be split and recompiled in the same manner based on any attribute defined. That’s powerful stuff, if defining a data flow. There are any number of use cases that can benefit from just the basic principles illustrated here 

More Information

Get a Fuusion Demo to find out how we can automate your biggest dataflow challenges, or even just take the hassle out of some of your smaller ones. 

What is Cloud NAS?

What is Cloud NAS?

Cloud NAS (Network Attached Storage) is a popular storage choice for people looking to use cloud storage for applications, user file systems, or data archives. But we still see a lot of confusion when people hear the terms “Cloud network-attached storage”, “Cloud-based NAS”, or cloud NAS service. So What is Cloud NAS?

What is a network-attached storage (NAS)?
What is cloud NAS?
Why do you need Cloud NAS?
Benefits of Using a Cloud NAS service
Use Cases for Cloud NAS

What is a Network Attached Storage (NAS)?

NAS is a common IT term for Network Attached Storage that enables data and file sharing using popular protocols like NFS and CIFS/SMB. iSCSI is typically associated with SAN (Storage Area Networks). NAS storage systems that support NFS, CIFS/SMB, Apple File Protocol (AFP), and iSCSI are termed “unified” storage. SoftNAS provides unified storage designed and optimized for high-performance, higher than normal I/O per second (IOPS), and data reliability and recoverability. It also increases storage efficiency through thin-provisioning, compression, and deduplication.

What is a Cloud NAS?

A cloud NAS works like the legacy, on-premises NAS currently in a lot of data centers. But, unlike traditional NAS or SAN infrastructures, a cloud NAS is not a physical machine. It’s a virtual appliance designed to work with and leverage cloud-based storage to give you all of the functionality you’d expect from a premises-based hardware NAS or SAN.

Cloud NAS is a “ Virtual NAS in the cloud” that uses cloud computing to simplify infrastructure and provide flexible deployment options while reducing costs. Most cloud NAS service solutions work in cloud environments like Amazon Web Services (AWS) and Microsoft Azure. Cloud-based NAS uses easily expandable cloud storage as a central source for storage while still providing common enterprise NAS features.

Traditional Network Attached Storage (NAS) is an intelligent storage device that is connected to a home or office network. A cloud NAS works similarly to any on-premises NAS found in an on-premises data center, only the data is stored in the cloud as opposed to a physical NAS infrastructure.

Demands for Cloud NAS Products

Cloud-based NAS Storage solutions have become an increasingly popular choice for organizations of all sizes as Virtual NAS allows businesses to use cloud storage for their multitudinous applications, systems, and archival functions and do all of this completely virtually.

The cloud-reliant virtualization of the traditional NAS process simplifies infrastructure while providing a variety of flexible deployment options. As the nature of work evolves alongside the rapid increase of data being generated daily, businesses need an efficient, cloud-based NAS to offer them limitless data management, backup, and storage.

A Cloud NAS does (or should) work like the legacy, on-premises NAS currently in many data centers. Typically, a third-party vendor delivers an IP address to the customer while hiding the implementation complexity. In reality, the IP address points to a virtual appliance designed to manage private cloud storage, which multiplies your expenses to pay for the virtual appliance, cloud storage, and bandwidth. In some cases, these third-party vendors act as resellers for another cloud provider, adding additional expenses to use their service. While this is a Cloud NAS, it is not an ideal solution.

Buurst’s SoftNAS Cloud NAS virtual appliance  

For Buurst, the virtual appliance itself is the key to Cloud NAS. It is not a storage capacity and bandwidth service. Storage capacity and bandwidth are already available in multiple formats and offered much cheaper than any private organization can reliably provide by the Cloud, AWS, and Azure monoliths. The goal of Cloud NAS should be to leverage the services of these giants, simplifying infrastructure and providing flexible deployment options while reducing costs. Thus you can leverage the scalability and flexibility of cheap cloud storage and still provide familiar enterprise NAS functionality. By delivering a flexible virtual appliance, we can also allow organizations to host locally and in the cloud, delivering a hybrid solution. 

Why do you need Cloud NAS?

The traditional needs met by locally hosted Network Access Storage are still present today. Users need ready access to multiple file formats without delay and downtime. However, the amount of newly generated data requires an updated solution model to accommodate the explosion of data storage costs.  

According to predictions from various sources (SeedScientific), in 2025, 463 exabytes of data will be created every 24 hours worldwide, with over 175 zettabytes globally. In comparison, estimates for 2020 claim (or predicted) approximately 40 zettabytes. Active data storage for personal and business requirements drives an ever-increasing demand for reliable storage, with most people unaware that most data is stored three times in the cloud. It is still unknown what effect Covid-19 will have on these predictions, but it could be substantial – for example, according to Forbes, remote work has increased 300% over pre-Covid levels. 

What does this mean? It means hosting data on-prem will become exponentially more expensive in hardware costs and associated services such as DR (disaster recovery) because AWS and Azure focus on creating essentially “limitless” cloud storage. It is already far more efficient to leverage these services than to fork-lift hardware-based solutions. In addition, with built-in redundancy, disaster recovery is also much, much cheaper. Finally, with the right set of capabilities, cloud-based NAS offerings significantly shorten the amount of time it takes to migrate from an on-premises NAS to the cloud. 

SoftNAS Cloud NAS Competitive Pricing Advantage

The storage industry has brought its legacy pricing models to the cloud, making you pay twice for your storage. Once to the cloud vendor, and once to the storage vendor. On top of this, if you want to increase performance, you have to increase capacity with it. With Buurst’s SoftNAS Cloud NAS, you’ll never have to pay for your own data.

%

Save up to 60% compared to NetApp ONTAP

%

Save up to 80% over AWS EFS

%

Save up to 60% over Azure NetApp Files

Benefits of Using a Cloud NAS service

Cloud NAS Price/Performance Flexibility

A robust Cloud NAS Filer virtual appliance should provide a great deal of flexibility. It should be available on or compatible with several key cloud platforms (AWS and Azure). It should leverage both lower-performance and lower-priced cloud object storage (AWS S3 or Azure Blob) and higher-performance tier block devices to satisfy HPC (High-Performance Computing) SLAs. You may even be able to utilize both types of storage from the same virtual appliance, depending on your use case. The right Cloud NAS solution will be able to meet your storage requirement for any project.

Eliminate Legacy NAS Systems Refresh

How do you predict precisely how much storage and the performance you will need from that storage over the next 12 months? Have your predictions ever been derailed by an unexpected project requiring additional storage at a different performance level? With legacy hardware NAS solutions, you usually get locked into a long-term contract, and if something changes, you incur the overhead and costs that come with a “fork-lift” upgrade. With a Cloud NAS, you are in control. You can create the storage you need when you need it, for as long as you need it, without signing long-term contracts or renewals. With some solutions, you can leverage your existing hardware to host their virtual appliance to help move your workload to the cloud more easily. 

Built-in Data Resiliency

Most cloud storage has data resiliency built-in by storing multiple copies of data on multiple disks. You can even distribute the data across different availability zones in some cases. This resiliency does not replace the need for High Availability, SnapShots, and backups, but it is nice to have this level of resiliency built right into the storage used by your Cloud NAS. A quality Cloud NAS appliance should build on the available data resilience offered by the cloud provider and expand on it. 

Pay as you Go and Reduce Costs with Cloud NAS

There should not be any additional costs for the storage you need with the right solution beyond the storage provider’s (AWS or Azure) charges. The licensing for the appliance should not change based on use case performance or storage requirements. Cloud storage is becoming cheaper and more flexible over time, as the cloud vendors compete on a massive scale. Suppose your Cloud NAS solution supports the many different storage performance tiers offered by AWS and Azure (as it should). In that case, you can instantly scale your cloud instances to best suit your needs simply by adding additional storage or changing the type of storage you use. You may even create tiered storage and push legacy data to lower-cost storage while maintaining performance for your frequently accessed data in top-tiered storage. 

Use Cases for Cloud NAS

SaaS-Enable Applications

When looking to migrate applications from on-prem to Software-as-a-Service (SaaS) in the cloud, a common hurdle is that traditional applications typically do not support the native cloud storage interfaces. Rewriting your applications to support cloud storage requires application development and is usually complex and costly. For legacy applications that use standard file protocols, the optimal Cloud NAS can offer the expected file services and support for NFS, CIFS/SMB, and iSCSI, along with Active Directory integration support for your existing applications. Choosing a Cloud NAS with the features your application depends on is vital. 

High-Performance Computing (HPC)

HPC requirements for Artificial Intelligence and Machine Learning are becoming standard. The power of the virtual machines in the cloud today matches what you can get on-prem, but you also need your storage to meet HPC SLAs. The level of performance you can get from both compute and storage IOPs and throughput increases every day. With the right Cloud-based NAS, you can achieve 10s and even 100s of thousands of IOPS with massive throughput to cloud storage today. And as the performance of the cloud resources increases, you can use this power as needed and the flexibility to use it only when needed. 

The amount of data created by a business will increase in the aftermath of Covid-19, with more employees working from home and more employers seeking to ensure user productivity at a lower price point. Storage is a critical factor in saving money with a file share repository for users to store shared data or back-end support for a VDI solution. Such a solution needs to be flexible enough to support Windows and Linux users, cost-efficient, and balance performance needs with cost. Suppose user data needs to be kept indefinitely for compliance reasons. In that case, your solution should also help with the migration of legacy data or even the setup of an automated tiered storage solution. 

Backup and Archive Data

The data your company collects is one of its most valuable assets in today’s world. You cannot afford to lose it. When you need to access that data, you need to be able to access it quickly. Using cloud storage to store backup or archive data can be the key. A Cloud NAS with lower-cost cloud storage gives you an “endless” capacity to store backup and archive data. You have the flexibility to decide what level of SLA you need for the retrieval of archived data and match it to the price/performance requirements you have. If your SLA to retrieve archive data is days, you can use archive-level object storage. If you can’t wait days, choose higher performance storage. If you need to plan to keep data accessible for decades for compliance reasons, you will want a Cloud NAS that can help you automatically move the files to less expensive cloud storage for you. The optimum Cloud NAS can help you increase capacity needs and keep costs down by moving those compliance files to more cost-effective storage. 

DevOps and Development

With cloud-based NAS, developers and DevOps can quickly stand up a new storage infrastructure needed for a new project (a new app, a POC, or any other type of project) and then tear it down when done (no long-term storage contract required). The storage needed is always available to be allocated at the performance required for the project

Consider Your Needs Carefully

When considering a Cloud NAS partner, it’s essential to understand your current and potential future requirements. Most cloud NAS offerings include “table stakes” NAS functionality, but the devil can be in the details:

  • Some may charge for hosting as well as the interface
  • Some Cloud NAS’s only offer limited support for various types of cloud storage  
  • Some limit the capacity of supported storage 
  • Some may be fine for primary performance workloads but can be quickly overloaded as performance demands increase 

Top Cloud NAS solutions may offer solution extensions to help you solve problems beyond just storing data in the cloud. In addition, they may offer extensions that help you:
  • Migrate your data from on-prem to the cloud
  • Move cold data from more expensive higher performing storage to more cost-effective, less performant storage 
  • Offer enterprise-class high-availability 
  • Data orchestration that can help you automate the movement and transformation of data. 

So when considering a Cloud NAS partner, take a look under the hood before you buy. Understand what type of mileage you will get with your selection and save time and money by choosing the right tool for the job from the beginning. 

Learn more about Buurst’s SoftNAS 

SoftNAS provides customers a unified, integrated way to aggregate, transform, accelerate, protect and store data and to easily create hybrid cloud solutions that bridge islands of data across SaaS, legacy systems, remote offices, factories, IoT, analytics, AI, and machine learning, web services, SQL, NoSQL and the cloud – any kind of data.

SoftNAS works with the most popular public, private, hybrid, and premises-based virtual cloud operating systems, including Amazon Web Services, Microsoft Azure, and VMware vSphere.

 

AWS NAS Storage Solutions 

Azure NAS Storage Solutions

VMware NAS Storage Solutions

Webinar Recap – The Three Strategies to Increase Performance for Your Applications in AWS

Webinar Recap – The Three Strategies to Increase Performance for Your Applications in AWS

AWS application performance

The above is a recording and follows is a full transcript from the webinar, “3 strategies to increase performance for your applications in AWS.” You can download the full slide deck on Slideshare.

AWS Application Performance

Hi, this is Jeff Johnson. I’m head of product marketing at Buurst. This is a recap of a webinar that we did — it’s called The Three Strategies to Increase Performance for Your Applications in AWS. 

Many customers have tried building these applications in their data centers and they are designed for extremely fast storage. As they are migrating their line of business applications to the cloud, a lot of their applications worked and didn’t have a constraint on access to the storage.  

But a few of those applications would not run or will not run in the cloud with acceptable performance because of the demand, the type of application that it is, and the connections or the throughput or the latency to get the data off of the storage and back into the application or put it back on to the storage. 

They weren’t getting the performance that they needed, so they tried using AWS EFS. They tried using FSx. They tried using maybe some open-source NAS and they weren’t getting the performance until they discovered SoftNAS. 

SoftNAS push to make the AWS application run with great performance

SoftNAS then had the appropriate levers to pull, buttons to push to make the application run with great performance, and we’re going to do a deep dive into why and we have some statistics on some performance benchmarking we did that prove out some of these things. 

Companies trust Buurst for data performance, data migration, data cost control, high availability, control, and security. When we think about this scenario of performance, there are two different camps, let’s say. There are managed storage services, and they are great because someone else is managing your storage layer for you. Then there’s a cloud NAS (network attached storage) like SoftNAS, so this sits in between your cloud clients and your on-premises clients and your storage. 

Managed Storage Service

For managed storage services like AWS EFS or FSx, those solutions…

First of all, the service up many different companies or customers and they throw security in so company A can get to company B. The other end of that is they limit the amount of throughput that each company has to access storage so one of them does not become a noisy neighbor.

Then we have like SSD and HD in cold storage on the backend there that we have access to, so we can always just go to a faster disk or things like that to get better performance, but that comes with a price.

To really think about increasing the performance of a managed storage service, we have two main levers we can pull and push. It’s we can purchase more storage or we can purchase more throughput.

Now, this chart is based on the AWS EFS and FSx websites. What it’s telling me is the more gigs I have purchased; the more throughput I have.

Increase Performance by Increasing Storage

For instance, the very top one on AWS EFS, if I purchase 10 gigs, I get 0.5 per second and it’s going to cost me $3 a month. If my solution requires that I have 100 megs of throughput, I need to purchase two terabytes of storage. That will cost me $614 a month, and that’s fine.

To get more performance, I can increase the amount of storage. I can also increase my throughput by purchasing provisioned throughput.

In this instance here, in this case, I needed that 100 megs of throughput, and I can purchase that. I’m basically buying two terabytes of storage. Even if I only have one terabyte of storage but I need 100 megs of throughput, I’m basically still purchasing two terabytes of storage but I’m provisioned with 100 megs.

The same thing with 350, I’m eight terabytes. With 600 megs, I’m basically purchasing 16 terabytes of storage for the throughput performance that I require. Now that’s great.

Like I said earlier, in a lot of applications, that runs very well and it’s fine. But for those applications that it doesn’t quite work for, they turn to SoftNAS AWS. They turn to a cloud NAS.

SoftNAS for better Application Performance

AWS application performance

With SoftNAS, I can use an efficient protocol. If it’s a SQL Server solution, I can use iSCSI. If it’s Linux, I can use NFS, windows, I can use CIFS, and it could all connect to the same amount of storage I can control and manage off my NAS.

I can use different disks speeds or more disks. I can use more smaller disks than fewer larger disks in my RAID array. I can do things like that. For this conversation, we’re going to talk about the cool ways to increase performance and manage costs.

I can increase the compute instance, and I can increase the read/write cache of my SoftNAS. Let’s dive into that. In AWS, I have compute instances for my NAS. In this case, I have an m3.xlarge. I have 4VCU and 15 megs of RAM, and I get 100-megabyte throughput.

That’s the throughput from my NAS to my clients because all the storage is directly connected to my NAS. I have a c5.9xlarge, 36 VCPUs, 17 megs of RAM. I get 1,200 megs of throughput. Remember that’s from my NAS to my clients.

For my NAS to my storage, that’s directly connected to my NAS. In AWS, I can connect about a petabyte of storage. I can also utilize caching. We know how important caching is. On my NAS, I have an L1 cache which is RAM, and I have an L2 cache which is a dedicated disk to a specific storage pool.

With SoftNAS, by default, I use half of the available RAM. If I have 32 megs of RAM, I can get 16 meg of that by default, available for L1 cache.

For L2 cache, I can use NVMe or SSD. For instance, let’s say I have a SQL Server solution that I’m providing data to and all my storages are on an array. That’s great. I can have NVMe dedicated to that pool servicing that SQL Server and my solution is just humming.

Then I could have another pool, let’s say it’s all my web servers data, that are in an array. I can have all of that with a bigger SSD drive for the cache. I can tune my solution in my NAS controller.

We ran a performance benchmark on AWS EFS against SoftNAS on AWS. Because so many customers were coming to us and saying, “We came to you because you gave us the performance that we needed,” we really wanted to dive down and try to figure out why or how.

To just summarize, throughput is a measurement of how fast your storage can read and write the data. In IOPS, the higher the IOPS; the faster you have access to the data on that disk. Latency is a measurement of the time it takes for a component or a subsystem to process the data request or transaction.

How do we do it?

Well, we have a Linux fio Server with four Linux fio clients through NFS connecting either to SoftNAS with no L2 cache or AWS EFS, with SSD storage on both. We had a basic, medium, and high.

Basic was 100 Megabits per second. Medium was 350 megs per second. High was 600 megs per second. We gave them different amounts of storage to give more throughput to the storage on AWS EFS.

For storage throughput, what we learned was the more RAM, CPU, and network we gave to the NAS the better numbers we got. And we were able to provide continuously sustained throughput and predictable performance because we weren’t throttling any other customers.

Throughput (MiB\s) – Higher is Better

What we found was pretty remarkable. On our basic, we are almost twice the performance. When we got to our higher-end, we are 23% at all times — 23 times the performance in the higher end for read/write sequential and read/write random of a 70/30 combo there.

On IOPS, again what we learned was the more CPU, RAM, and network speed, we got better IOPS. We know we could have used a faster disk such as NVMe, and that’s how we got a million IOPS on AWS. We could have added more disks to an array to aggregate and increase the disk IOPS. We could have done that also, but we didn’t.

IOPs – Higher is Better

What we found is, again, we were almost two times the performance on IOPS on our basic. I think we are around 18 times on the medium and 23 times the performance on the higher end — just huge numbers that we are really proud of.

For the latency, how can I get that data or the time it takes to get that data off and on that disk. What we found is no surprise. If I increase the CPU, the RAM, and network speed, I decrease latency. I could have decreased latency by using NVMe, but that adds a substantial cost.

Latency – Lower is Better

I could have decreased latency by using smaller and more disks than larger disks. Again, these results were kind of the same. We were two times lower. We’re about 18 times lower in our midrange there. In our high end, we’re about 23 times lower.

Now, all of that data, we’ve published. It’s on our blog in burst.com and it’s an I-chart. If you want to dive into this data, we’d be more than happy to talk with you on how we achieve these numbers, try to replicate these numbers, or give you a demo on how we think we can increase the performance of your specific application.

Let’s talk about specific applications where there’s throughput, IOPS, or latency.

For throughput, if you have many client connections like a web server array or even just cloud virtual desktops or actual desktops on-premises, you have lots of clients accessing the data. You need more throughput.

Maybe they are accessing video files, office files, AutoCAD files, or web server content — more throughput. Like one of our great partners, Petronas, had many clients out there that needed access to the content. SoftNAS was able to handle the amount of clients accessing the data.

When we’re talking about IOPS, we’re talking about small block size transactions like a database server or an email server who need to access the data and put data in very small chunks. We have a great partner, Halliburton, for instance. Their application for the oil and gas industry — their landmark application — they take the seismic data, just massive amounts of data.

It has to pull it off of the disk and then render it to show it visually to the plate-tectonic engineers. They were able to get their application, import it, and move to the cloud in record speed with SoftNAS and then have it run at a great performance level with SoftNAS. Absolutely fantastic.

On latency, think about applications like banking, stock exchange, finance who need fast access in and out of that data as fast as they can, or streaming. Netflix is a great partner of ours. To provide the solution for great cloud partners like AWS accessing the cloud storage, providing NetApp to the application.

How do I increase performance with AWS applications?

We’ve got to understand what the bottlenecks are. How does the AWS application perform? Does it need throughput? Does it need IOPS? Does it need low latency?

How to Increase Performance for Your AWS Application

  1. Understand your solution and where the bottlenecks are
    • Throughput
    • IOPs
    • Latency
  2. Then understand if managed storage will work for you
  3. Reach out to SoftNAS to better understand your AWS performance options to develop a solution to meet the specification of your workload

When we understand that, then we can understand if managed storage can work for you. That’s great. If it’s not, come talk to us. Meet a cloud storage performance professional and just talk about some ideas, what you’re seeing out there, why won’t it work. We’ve helped so many customers.

We have helped tons of customers get their applications running on AWS. We were the first NAS up there. We started that whole industry of NAS in the cloud. We have lots of information on performance blogs. We have an e-book.

We have a dedicated performance web page at burst.com/performance. We’d love to show you our product. To get a demo, talk to a performance professional. At Buurst, we are a data performance company. That’s all we want to do, live and breathe and think and provide you the fastest access to your cloud storage for the lowest price.

Try SoftNAS For better AWS Application Performance

SoftNAS can increase cloud application performance with 23x faster throughput, 18x better IOPs, and 24x less latency than other cloud storage solutions. It provides NAS capabilities suitable for the enterprise on AWS, Azure, VMware, and Red Hat Enterprise Linux (RHEL).

SoftNAS provides unified storage designed and optimized for high-performance, higher than normal I/O per second (IOPS), and data reliability and recoverability. It also increases storage efficiency through thin-provisioning, compression, and deduplication.

Check also:

Crank Up the Performance for Your Cloud Applications

AWS NAS design & configuration guide

AWS NAS Storage Solutions

 

Benefits of using Cloud NAS service

Benefits of using Cloud NAS service

The way we work has evolved, but data storage hasn’t changed substantially in over two decades except for the increasing amount of data being generated. With the right set of capabilities, cloud-based NAS offerings significantly shorten the amount of time it takes to migrate from an on-premises NAS to the cloud. With the capacity of cloud storage being “limitless”, NAS in the cloud allows you to expand or reduce the amount of data you manage without the need for forklifting hardware-based solutions. Today we will learn about the benefits of using a Cloud NAS (network attached storage) service.

Benefits of using Cloud NAS service:

Price/Performance Flexibility:

With the right Virtual cloud NAS, you have numerous options on what type of cloud storage to use. Low-performance Cloud Object storage can be used for use cases that don’t require high performance. But for a price, you can also satisfy HPC (high-performance computing) level SLAs. With the right combination of Virtual Machines running your Cloud NAS controller head with the right backend cloud storage, you can meet your storage requirements for just about any project.

Eliminate Legacy NAS Systems Refresh:

How do you predict exactly how much storage and performance from that storage you will need over the next 12 months? Do you ever have an unexpected project come up that requires more storage possibly at a different performance level? With legacy hardware NAS solutions, you usually get locked into a long-term contract, and if something changes, you incur the overhead and costs with a “forklift” upgrade. With a Cloud NAS, you are in control. You can create the storage you need when you need it, for as long as you need it without signing long-term contracts or renewals.

Built-in Data Resiliency:

Most cloud storage has data resiliency built-in by storing multiple copies of data on multiple disks. This resiliency does not replace the need for High Availability, SnapShots, and backups, but it is nice to have this level of resiliency built right into the storage used by your Cloud NAS filer.

Pay as You Go and Reduce Costs:

You only pay your cloud provider for the storage you need. With cloud storage becoming cheaper, you can instantly scale your cloud instances to best suit your needs. Or, you can even use tiered storage and push legacy data to low-cost storage and store frequently-accessed data in top-tiered storage to maintain performance for your “hot” data.

Use cases for a Cloud NAS (network attached storage) include:

SaaS-Enable Applications:

When looking to migrate applications from on-prem to Software-as-a-Service (SaaS) in the cloud, a common hurdle is that traditional applications typically do not support the native cloud storage interfaces. Rewriting your applications to support cloud storage requires application development and is usually complex and costly. For legacy applications that use standard file protocols, the right Cloud NAS can offer the expected file services and support for NFS, CIFS/SMB, iSCSI, and AFP protocols along with Active Directory integration support for your existing applications. Choosing a Cloud NAS with the features your application depends on is key.

New Apps and Proofs of Concept (POC):

A cloud NAS lets developers quickly stand up storage infrastructure for a new application or proof of concept project without any storage hardware. Developers can easily create a storage infrastructure with just a few clicks.

High-Performance Computing (HPC):

HPC requirements for Artificial Intelligence and Machine Learning are becoming standard. The power of the virtual machines you can create in the cloud today matches what you can get on-prem, but you need your storage to meet HPC SLAs also. The level of performance you can get from both compute and storage IOPs and throughput is increasing every day. With the right Cloud-based NAS, you can achieve 10s and even 100s of thousands of IOPS with massive throughput to cloud storage today. And as the performance of the cloud resources increases, you have the option to use this power when ready.

File Services for User File Systems:

The amount of file data being created daily can be overwhelming. When you add the need to keep copies of the data for compliance reasons for extended periods, it gets even worse. The right Cloud NAS can help satisfy the data access needs of your Linux, Windows, and Apple users by supporting the necessary file access protocol support. When you include the need for compliance copies of the files for possibly decades, you want a Cloud NAS that will help you automatically move the files to less expensive cloud storage for you. The right Cloud NAS can help you with both your increasing capacity needs and options to keep those compliance files on more cost-effective storage with low overhead on your part.

Backup and Archive Data:

The data your company collects is one of its most valuable assets in today’s world. You can’t afford to lose it. And when you need to access that data, you need to be able to access it quickly. Using cloud storage to store backup or archive data can be the key. A Cloud NAS with lower-cost cloud storage gives you an “endless” capacity to store backup and archive data. You have the flexibility to decide what level of SLA you need for the retrieval of archived data and match it to the price/performance requirements you have. If your SLA to retrieve archive data is days, you can use archive-level object storage. If you can’t wait days, choose higher-performance storage. And you want a Cloud NAS partner that can help get your data onto the archive storage with ease and low overhead.

DevOps and Development:

With cloud-based NAS, developers and DevOps can quickly stand up a new storage infrastructure needed for a new project and then tear it down when down (no long-term storage contract required). The storage needed is always available to be allocated at the performance required for the project.

When considering a Cloud NAS partner, it’s important to understand your current and potential future requirements. Most cloud NAS offerings include “table stakes” NAS functionality, but the devil can be in the details.

  • Some Cloud NAS’s only offer limited support for the types of cloud storage that can be utilized.
  • Some limit the capacity of storage that can be supported.
  • Some may be fine for basic performance workloads but can be quickly overloaded as performance demands increase.

 

Top Cloud NAS solutions may offer solution extensions to help you solve problems beyond just storing data in the cloud. They may offer extensions that help you:
  • Migrate your data from on-prem to the cloud
  • Move cold data from more expensive higher performing storage to more cost-effective less performant storage
  • Offer enterprise-class high-availability
  • Data orchestration that can help you automate the movement and transformation of data

So when considering a Cloud NAS partner, take a look under the hood before you buy. Understand what type of mileage you will get with your selection and choose the right tool for the job from the beginning and save time and money in the long run.

We hope you found this post helpful as you learn more about cloud NAS. Leave a comment and let us know what we should write about next!

 

Why SoftNAS on AWS?

Why SoftNAS on AWS?

Why Use SoftNAS Cloud NAS on AWS?

SoftNAS Enterprise and SoftNAS Essentials for AWS extend the native storage capabilities of AWS and feature the POSIX-compliant and required storage access protocols needed to create a virtual cloud NAS, without having to re-engineer existing customers’ applications.

SoftNAS products allow customers to migrate existing applications and data to AWS with NFS, CIFS/SMB, iSCSI or AFP support. Customers gain the performance, data protection, and flexibility needed to move to the cloud cost-effectively and ensure successful project outcomes (e.g., snapshots, rapid recovery, mirroring, cloning, high availability, deduplication, and compression).

Each customer’s journey to the cloud is unique and SoftNAS solutions are designed to facilitate adopting cloud projects according to what makes the most immediate business sense, resulting in the highest return on invested resources (budget, people, time). Whether your need is to consolidate file servers globally, utilize the cloud for data archival or backup, migrate SaaS and other business applications, or carry out Big Data or IoT projects, SoftNAS products on AWS deliver effective, tangible results.

aws cloud nas

When to use SoftNAS NAS versus Amazon EFS?

SoftNAS is a best-in-class software-defined, virtual, unified Cloud NAS/SAN storage solution for businesses that need control of their data and frictionless and agile access to AWS cloud storage. SoftNAS supports AWS EBS and S3 Object storage;  SoftNAS is deployed by thousands of organizations worldwide, supporting a wide array of application and workload use cases. Amazon EFS provides basic and scalable file-based NFS access for use with Amazon EC2 instances in the AWS Cloud. As a basic NFS filer, Amazon EFS is easy to use, allowing quick and simple creation and configuration of file systems. The multi-tenant architecture of EFS accommodates elastic growth and scales up and down for AWS customers that require basic cloud file services.

If you need more than a basic cloud filer on AWS, SoftNAS NAS is the right choice:

  1. For mission-critical cloud data requirements that demand low latency, high-availability with the highest-performance cloud storage I/O possible.
  2. SoftNAS ObjectBacker™ increases storage I/O performance up to 400% faster, giving customers performance approaching that of EBS at the price of S3 storage.
  3. For environments with multiple cloud storage projects that require an enterprise-class, virtual NAS storage solution that provides flexible and granular pricing and control with performance and instance-selection capabilities.
  4. For customers with low cloud storage requirements (low TBs) that don’t want to overprovision storage to get desired performance.
  5. For requirements that demand the broadest POSIX-compliant and full NFS 4.1 feature set required storage access support.
  6. For enterprises requiring multi-cloud environment (i.e. AWS & Other Clouds) capabilities, flexibility and data risk mitigation.
  7. For organizations that need the industry’s most complete NAS filer feature set, including data protection provided by the patented cross-zone high availability architecture, at-rest and in-flight encryption (360-degree Encryption™) and full replication using SnapReplicate™. Additional features include: High Availability (SNAP HA™* and DCHA**), Snapshots, Rapid Recovery, Deduplication, Compression, Cloning, and Mirroring.

*   = SNAP HA is available only in SoftNAS Enterprise and SoftNAS Platinum

** = DCHA is available as an optional add-on to SoftNAS Essentials via BYOL only

aws softnas

How and Why is SoftNAS Cloud NAS a Better Way to Store & Control Data?

Moving to SoftNAS on AWS is helping thousands of public organizations and private enterprises around the world move from the old way of controlling and efficiently utilizing data storage into a new and better way. Never-ending backups, storage capacity bottlenecks, the proliferation of file servers, existing applications preventing organizations from moving to the cloud due to complexity, and the spiraling data storage costs become a thing of the past.

softnas

SoftNAS Cloud NAS on AWS

Put SoftNAS to the Test and Try it for 30-Days. Contact a SoftNAS representative for more details and a demo to learn more about both SoftNAS products on AWS and the Beta of SoftNAS Platinum so that you make the right decision.

 

30 days or less and 20TB or less AWS Marketplace Free Trial is available. Just launch and go in minutes.

Moving your On-Premises NAS to the Azure Cloud

Moving your On-Premises NAS to the Azure Cloud

On-Premises NAS to the Azure Cloud

Missed our webinar on “Moving your On-Premise NAS to the Azure Cloud?” Remember to click the button below to test drive SoftNAS Cloud NAS on Microsoft Azure today.

See the slides on SlideShare: Click here

Introduction

Today we’re going to talk about moving from a physical NAS device to the Microsoft Azure Cloud. We’ll cover some of the advantages of using Azure for your cloud storage needs. This is not a new concept. It’s on-premise versus the cloud. Microsoft Azure gives us the option to have VMs running inside of our repository and accessing our virtual NAS, SoftNAS gives you network access control towards all your storage needs within a packaged, usable space.

So how does this benefit us? We have a few use cases we want to highlight below for Azure Storage. The use cases are SaaS enables applications, disaster recovery, and hybrid storage.

Microsoft Azure NAS Storage Use Cases

azure cifs iscsi nfs on-premise nas

For the first use case, the challenge is needing to SaaS enable a customer-facing application on Azure but the app doesn’t support Blob. They also need AD or LDAP integration for that application, so what would the solution be?

The solution would be to rewrite your application to support Blob and AD authentication.  It’s unlikely that that would ever happen, right?  So what else could you do? Instead of rewriting the application to support Blob, you can continue to do business the way you always have. Do you need access via NFS? We’ll just support that via NFS through SoftNAS Cloud NAS. Drop all that data on Azure back in, store it in Blob and let us do the translation. Then we could have access for all our applications on-premise or in the cloud to whatever data resources they need. It could be presented with any protocol that’s listed whether it’s CIFS, NFS, AFP, or iSCSI.

Disaster recovery on Azure Cloud Storage using SoftNAS Cloud NAS

azure on-premise nas disaster recovery

What’s our challenge? We’ve got a company that needs reliable and offsite data protection. They’ve already created a big EMC array at their location that they have several years of support left on and they need to be able to meter its use to it. But they need to be able to have a simple integration solution. So what would be the solution? It would be easy to spin out a SoftNAS instance on premise, access that EMC array, and use the data resources for SoftNAS Cloud NAS. We can then present those air repositories to their application servers and end-users onsite and replicate all that data using SnapReplicate into Microsoft Azure. We would have our secondary Blob storage and replicate all the data that’s on-premise into the cloud.

Now what’s great about this solution is it becomes a gateway to where I’m going to get to the end of support on that EMC array. Well, we’ve got this thing running in Azure already.  Why don’t we just cut the cord? We could just start directing our application resources to Azure. So that’s a great way to get you moving into the cloud and get a migration strategy moving forward.

Hybrid on Premises Storage Gateway to Azure Cloud

azure on-premise nas hybrid storage

The last use case is hybrid on-premise usage. For example, a company has performance-sensitive applications that need a local LAN. They need off-site protection or a capacity, and the solution would be to set up replication to Azure and then have that expand capacity, so whenever they run out of space on-premise we would then be able to burst out into Azure and create more and more virtual machines to access that data, or maybe it’s a web service account that has a web portal UI or something like that needs just a web presence, and then we’re able to many copies of different web servers that are load-balanced, all accessing their same data on top of Microsoft Azure through SoftNAS NAS Storage. So all these use cases are possible. These are all use cases that I’ve had customers experience today.

SoftNAS Microsoft Azure Cloud NAS Storage

azure on-premise nas softnas

At Buurst, we’ve built our architecture to be flexible and adaptable for the cloud. We’ve built a Linux virtual machine on CentOS. It runs ZFS as our file system on that kernel. We run our systems on open controllable systems that we have staff on-site that actually contribute to these open-source amalgams to make these systems better into CentOS and ZFS and we contribute a lot of intellectual property to help advance these technologies into the future.

We run HTML5 as our admin UI and we have PHP. Apache is our web server and so we have all these open systems to allow us to be able to take advantage of a great open source community out there on the internet. And we integrate with many different service users so if you have customers that are currently running in a different public cloud and they’re looking to migrate into Azure, it’s easy for us to come in and help you make that data migration change because in starting a SoftNAS cloud service into both of those service providers and then migrating that data is simple and easy to do the task.

We can do inline deduplication, caching, storage pools, thin provisioning, writeable snapshots, and SnapClones. We can do compression, encryption, all these different offerings that we’re able to give you in a single packaged NAS solution, so once again all the things that you think you come back in like, “Okay, I’m going to have to install all that stuff and I have to buy all these different components and insert it into my hardware.”  Those are all things that are assumed and used, and we’re able to just go ahead and give you our NAS Storage solution.

We’re able to present a storage capacity, so whether it be a CIFS or SMB access medium for Windows users, for some sort of Windows file share, or if it’s an NFS share for some Linux machines, or even just an iSCSI block device or an AFP (Apple File Protocol) for time machine backups, if you have end-users or end devices that need storage repositories of many different protocols, we’re able then to store that data into, say, an Azure Blob storage or even a native Azure storage device.

We’re then able to translate those protocols into an object protocol that is not a native language. We don’t speak in object whenever we’re going through a normal SMB connection, but we do also speak native object into Azure Blob. So we offer the best of both worlds with this solution, just the same as native block devices. We have a native block protocol that we’re able to talk into Azure disks that attach to these machines. We’re able to then create flexible containers that make data accessible to everyone.

Now how does this kind of play out and work in the real world? What we’re going to do is we’re going to present a single IP point of access that all these file systems will land on, so all our CIFS access, all our NFS shares, exports, all the AFP shares will all be enumerated out on a single SoftNAS instance and they will be presented to these application servers and end-users.

The storage pools are nothing more than conglomerations of disks that have been offered up by the Microsoft Azure platform, so whether it’s Microsoft Blob or it’s just native disks, if it’s even another type of object device that you’ve imported into this device, we can support all those device types and create storage pools of different technologies, and we can attach volumes and LUNs that have shares of different protocols to those storage pools so it allows us to have many different connection points to different storage technologies on the back-end, and we do this as a basic translation, and it’s all seamless to the end-user or the end device.

Q&A: Microsoft Azure Cloud NAS Storage

Question 1: What versions of NFS does SoftNAS support?

Answer: We support both version three and four for NFS. Then the follow-up that will be, the question that will be asked is, “What versions of SNB do we support?” We support two and three SNB.

Question 2: What type of RAID does SoftNAS use?

Answer: It’s ‘build your own RAID’.  We don’t tell you what type of RAID you have to use.  It depends on what your situation is.  If you’re inside of Microsoft Azure and you trust their local disk storage is under a low enough AFR that you’re not going to have to worry about RAID in your solution or it’s not that much pressing data, then you can go ahead and use RAID 0 and get the fastest capabilities out of it.  But, if you’re on-premise and you don’t have a hardware RAID solution, we give you the ability to use up to RAID 7. So if you wanted to use RAID 6 to give good performance and redundancy at the same time, you’re welcome to do that.

Question 3: How much would encryption inhibit or prevent deduplication benefits?

Answer: So that one, that’s a tricky question, right? Because deduplication actually happens on the fly, so we’re going to be doing the dedupe inline. Encryptions are not going to come into play there, so the encryption’s going to happen on the actual container itself so we’re going to actually encrypt the channel itself and then whenever we drop the data in there it’s going to dedupe.

Question 4: Does SoftNAS provide performance reports to show or see hot vs. cold data volumes?  

Answer: We do provide a dashboard that gives you access to all that data. You can come in here and see which data disks are getting hit the hardest, where we have data that are just stored as, and asleep, just never touched. We do have availability access for that dashboard to see that data, and it reports in and we can actually export that via SMTP server. So you can integrate it with SMTP or SNMP via things like WhatsUp Gold or like products.

We hope that you found the content useful and that you gained something out of it. Hopefully, you don’t feel we marketed SoftNAS NAS filer too much. Our goal here was just to pass on some information about moving from your on-premise NAS to the Azure cloud. As you’re making that journey to deploying in the cloud or you’re already operational in the cloud, maybe this webinar saved you time from tripping over some of the things that other customers have tripped over.

Expand Azure Storage Efficiency with SoftNAS Cloud NAS

SoftNAS is a software-defined NAS delivered as a virtual appliance running within Azure Computing Service. It provides NAS capabilities suitable for the enterprise, including high availability utilizing Azure availability sets with automatic failover in the Azure cloud storage. SoftNAS runs within your Microsoft Azure account and offers business-critical data protection required for the non-stop operation of applications, websites, and IT infrastructure. It is designed to support a variety of market verticals, use cases, and workload types. Increasingly, SoftNAS deployed on the Azure platform to enable block and file storage services through Common Internet File System (CIFS), NFS, AFP, and iSCSI. Learn why SoftNAS for Microsoft Azure?