Best Practices Learned from 2,000 AWS VPC Configurations

Best Practices Learned from 2,000 Amzon AWS VPC Configuration. Download the full slide deck on Slideshare. 

The SoftNAS engineering and support teams have configured over 2,000 Amazon Virtual Private Cloud (VPC) configurations for SMBs to Fortune 500 companies. In this guide, we share the lessons learned in configuring AWS VPCs, including undocumented guides, tips, and tricks.

Amazon’s Virtual Private Cloud enables you to launch Amazon Web Services (AWS) resources, like EC2 instances, into a virtual network that you’ve defined. They are flexible, secure and a core building block of AWS deployments.

In this Guide, we covered:

• How do packets really flow in an Amazon VPC?
• Common security group misconfigurations.
• Why end-points are good things.
• To NAT or not?
• VPNs and VPCs: a good thing?
• Best practices for AWS VPC management

We’ve configured over 2,000 Amazon AWS VPC

In this post, we’ll be talking about some of the lessons that we’ve learned from configuring over 2,000 VPCs for our customers on Amazon Web Services (AWS_). We’ve configured over 2,000 Amazon VPC for our customers and some of the customers that we’ve configured the VPCs for are listed out here.

We’ve got a wide range of experience in both the SMB and the Fortune500 market. Companies like Nike, Boeing, Autodesk, ProQuest, and Raytheon, have all had their VPCs configured by SoftNAS.

Just to give you a brief overview of what we mean by SoftNAS. SoftNAS is the product that we use for helping manage storage on AWS. You can think of it as a software-defined NAS. Instead of having a physical NAS as you do on a traditional data center, our NAS is software-defined and it’s based fully on the cloud with no hardware required. It’s easy to use.

You can get up and running in under 30 minutes and it works with some of the most popular cloud computing platforms so Amazon, VMware, Azure, and CenturyLink Cloud.

What is an AWS VPC or a Virtual Private Cloud?

It can be broken down in a couple of different ways, and we’re going to break this down from how Amazon Web Service looks at this.

It’s a virtual network that’s dedicated to you. It’s essentially isolated from other environments in the AWS Cloud. Think of it as your own little mini data center inside of the AWS data center.

It’s actually a location where you launch resources into and allow you to virtually logically group them together for control. It gives you configuration flexibility to use your own private IP addresses, create different subnets, routing, whether you want to allow VPN access in, how you want to do internet access out, and configure different security settings from a security group. I’m an access control list point of view.

 The main things to look at that I see are around control.

• What is your IP address range? How is the routing going to work?
• Are you going to allow VPN access?
• Is it going to be a hardware device at the other end?
• Are you going to use Direct Connect? How are you going to architect your subnets?

These are all questions. I’m going to cover some of the tips and tricks that I have learned throughout the years. Hopefully, these will things that will help everyone because there is not really a great AWS VPC book out there or dot guidance. It’s just a smattering of different titbits and tricks from different people.

Security groups and ACL as well as some specific routing rules. There are some specific features that are available only in VPCs. You can configure multiple NIC interfaces.

You can set static private IPs so that you don’t ever lose that private IP when the machine is stopped and started, and in certain instances such as the T2s and the M4s, their primary purpose is to be lost within a VPC.

This is the way that you could perform your hybrid cloud setup or configuration. You could use Direct Connect, for example, to securely extend your premise location into the AWS Cloud, or you could use your VPN connection on the internet to also extend your premise location into the cloud.

You can peer the different VPCs together. You can actually use multiple different VPCs and peer them together for different organizational needs. You can also peer them together with other organizations for access to certain things — think of a backend supplier potentially for inventory control data.

Then there is a bunch of endpoint flow logs that help you with troubleshooting. Think of this, for those of you that have a Linux background or any type of networking background, like a TCP dump or a Wireshark ability to look at the packets and how they flow can be very useful when you’re trying to do some troubleshooting.

Amazon AWS VPC Topology Guidance 

Just some AWS VPC topology guidance so hopefully, you’ll come away will something useful here. Our VPC is used in a single region but will be in multiple availability zones.

It will extend across at least two zones because you’re going to have multiple subnets. Each subnet lives in a single availability zone. If you configure multiple subnets, you can configure this across multiple zones. You can take the default or you can dedicate a specific subnet to a specific zone.

All the local subnets can reach each other and route to each other by default. The subnet sizes can be from a /16 to a /28 and you can choose whatever your IP prefix is.

How can access traffic within the AWS VPC environment?

How can access traffic within the virtual private cloud environment? There is multiple difference between these gateways. What do these gateways mean and what do they do? Do you hear these acronyms IGW, VPG, and VGW? What does all this stuff do?

These gateways generally are provisioned at the time of VPC creation, so keep that also in mind. The internet gateway is an ingress and egress for internet access.

You can essentially in your VPC point to specific machines or different routes to go out over the internet gateway to access resources outside of the VPC or you can restrict that and not allow that to happen. That’s all based on your organizational policy.

A virtual private gateway, which is the AWS side of a VPN connection, if you’re going to have VPN access to your VPC, this is the VPC on the AWS side of that connection and the CG is the customer side of the VPN connection within a specific VPC.

From a VPN option, you have multiple subsets. I mentioned Direct Connect which would essentially give you dedicated bandwidth to your VPC. If you wanted to extend your premise location up into the cloud, you could leverage Direct Connect for your high-bandwidth lower latency type of connections. Or if you wanted to just be able to make a connection faster and you didn’t necessarily need that level of your throughput or performance, you can just tap up a VPN channel.

Most VPN vendors like Cisco and others are supported and you can easily download a template configuration file for those major vendors directly.

Amazon Web Services (AWS) VPC Packets Flow

Let’s talk a little bit about how the packets flow within an AWS VPC. This is one of the things that I really wish I had known earlier on when I was first delving into configuring SoftNAS instances inside of VPCs and setting up VPCs for customers in their environments.

Its, there is not really great documentation out there on how packets get from point A to point B under specific circumstances. We’re going to back to this a couple of different times, but you’ve got to keep this in mind that we’ve got three instances here — instance A, B, and C — installed on three different subnets as you can see across the board.

How do these instances communicate with each other?

Let’s look at how instance A communicates to instance B. The packets hit the routing table. They hit the node table. They go outbound to the outbound firewall.

They hit the source and destination check that occurs and then the outbound security group is checked. Then the inbound security group source and destination check in the firewall.

This gives you an idea if you make different configuration changes in different areas, where they actually impact, and where they actually come into play. Let’s just talk about how instances would talk B to C.

Go back to the first diagram. We’ve already shown how A would communicate with B. How do we get over here to this other network? What does that actually look like from a packet flow perspective?

This is how it looks from an instance B perspective to try to talk to instance C, where it’s actually sitting on two subnets and the third instance (instance C) is on a completely different subnet.

It actually shows how these instances and how the packets would flow out to a completely different network and this would depend on which subnet each instance was configured in.

Amazon AWS VPC Configuration Guide

Some of the lessons that we’ve learned over time. These are personal lessons that I have learned and things that I wish if, on day one, somebody handed me a piece of paper. What would I want to have known going into setting up different VPCs and some of the mistakes that I’ve made throughout my time?

Organize AWS Environment

Number one is to tag all of your resources within AWS. If you’re not doing it today, go do it. It may seem trivial, but when you start to get into multiple machines, multiple subnets, and multiple VPCs, having everything tagged so that you can see it all in one shot really helps not make big mistakes even bigger.

 Plan your CIDR block very carefully. Once you set this VPC up, you can’t make it any bigger or smaller. That’s it, you’re stuck with it. Go a little bit bigger than you think you may need because everybody seems to really wish they hadn’t undersized the VPC, overall. Remember that AWS takes five IPs per subnet. They just take them away for their use. You don’t get them. Avoid overlapping CIDR blocks. It makes things difficult.

Save some room for future expansion, and remember, you can’t ever add any more. There are no more IPs once you set up the overall CIDR.

AWS Subnet Your Way to Success

Control the network properly. What I mean by that is use your ACLs, use the things in the security groups. Don’t be lazy with them. Control all those resources properly. We have a lot of resources and flexibility right there within the ACLs and the security groups to really lock down your environment.

Understand what your AWS subnet strategy is.

Is it going to be smaller networks, or you’re just going to hand out class Cs to everyone? How is that going to work?

If your AWS subnets aren’t associated with a very specific routing table, know that they are associated with the main routing table by default and only one routing table is the main. I can’t tell you how many times I thought I had configured a route properly but hadn’t actually assigned the subnet to the routing table and put the entry into the wrong routing table. Just something to keep in mind — some of these are little things that they don’t tell you.

I’ve seen a lot of people configure things and aligning their subnets to different tiers. They have the DMZ tier, sloe the proxy tier, and the subnet. They are subnets for load balancing, subnets for application, and subnets for databases. If you’re going to use RDS instances, you’re going to have to have at least three subnets, so keep that in mind.

Set your subnet permissions to “private by default” for everything. Use Elastic Load Balancers for filtering and monitoring frontend traffic. Use NAT to gain access to public networks. If you decide that you need to expand, remember the ability to peer your VPCs together.

Endpoint configuration

Also, Amazon has endpoints available for services that exist within AWS such as S3. I highly recommend that you leverage the endpoint’s capability within these VPCs, not only from a security perspective but from a performance perspective.

Understand that if you try to access S3 from inside of the VPC without an endpoint configured, it actually goes out to the internet before it comes back in so the traffic actually leaves the VPC. These endpoints allow you to actually go through the backend and not have to go back out to the internet to leverage the services that Amazon is actually offering.

Control Your Access

Do not set your default route to the internet gateway. This means that everybody is going to be able to get out. And in some of the defaulted wizard settings that Amazon offers, this is the configuration so keep it in mind. Everyone would have access to the internet.

You do use redundant NAT instances if you’re going to go with the instance mode, and there are some cloud formation templates that exist to make this really easy to deploy.

Always use IAM roles. It’s so much better than access keys. It’s so much better for access control. It’s very flexible. Here in the last 10 days, now you can actually attach an IAM role to a running instance, which is fantastic, and even easier to leverage now that you don’t have to deploy the new compute instances to attach and set IAM roles.

How does SoftNAS use Amazon VPC?

How does SoftNAS actually fit into using AWS VPC and why is this important?

We have a high-availability architecture leveraging our SNAP HA feature which provides high availability for failover cross zones, so multiple AZ high-availability. We leverage our own secure block replication using SnapReplicate to keep the nodes in sync, and we can provide a no downtime guarantee within Amazon if you deploy SoftNAS with the multiple AZ configuration in accordance with our best practices.

Cross-Zone HA: AWS Elastic IP

This is how this looks and we actually offer two modes of high availability within AWS. The first is the Elastic IP-based mode where essentially two SoftNAS controllers can be deployed in a single region each of them into a separate zone.

They would be deployed in the public subnet of your VPC and they would be given elastic IP addresses and one of these elastic IPs would act as the VIP or the virtual IP to access both controllers. This would be particularly useful if you have on-premises resources, for example, or resources outside of the VPC that need to access this storage, but this is not the most commonly deployed use case.

Cross-Zone HA: Private Virtual IP Address

Our private virtual IP address configuration is really the most common way that customers deploy the product today, and this is probably at this point about 85 to 90-plus percent of our deployments is in this cross-zone private approach, where you deploy the SoftNAS instance in the private subnet of your VPC.

It’s not sitting in the public subnet, and you pick any IP address that exists outside of the CIDR block of the VPC in order to be able to have high availability, and then you just mount your NFS clients or map your CIFS shares to that private virtual IP that exists outside of the subnet of the CIDR block for the overall VPC.

SoftNAS AWS VPC Common Mistakes

Some common mistakes that we see when people have attempted to deploy SoftNAS in a high availability architecture in VPC mode. You need to deploy two ENIs or Elastic Network Interfaces on each of the SoftNAS instances.

If you don’t catch this right away when you deploy it…Of course, these ENIs can be added to the instance after it’s already deployed, but it’s much easier just to go ahead and deploy the instances with the network interface attached.

Both of these NICs need to be in the same subnet. If you deploy an ENI, you need to make sure that both of them are in the same subnet. We do require ICMP to be open between the two zones as part of our health check.

We also see that the other problem is that people are providing access to S3. We actually as part of our HA provide a third-party witness and that third-party witness is an S3 bucket. So, therefore, we require access to S3, so that would require an endpoint or access out of your data infrastructure.

For Private HA mode, the VIP IP must not be within the CIDR of the VPC in order to overcome some of the networking limitations that exist within Amazon. Taran, I’m going to turn it back over to you. That concludes my portion of the presentation.

we suggest you have a look at our “AWS VPC Best Practices” blog post. In it, we share a detailed look at best practices for the configuration of an AWS VPC and common VPC configuration errors.

SoftNAS Overview

Just to give everyone a brief overview of SoftNAS cloud. Basically, what we are is a Linux virtual appliance that’s available on the AWS marketplace. You are able to go to SoftNAS on AWS and spin up an instance and get up and running in about 30 minutes. As you can see in the image, our architecture is based on ZFS on Linux. We have an HTML5 GUI that’s very accessible and easy to use. We do work on a number of cloud platforms including AWS, as well as Amazon S3 and Amazon EBS.