Provisioning an infrastructure for Kubernetes on AWS with Terraform (Part 1)


Welcome to Part 1 of kubegrid’s 3-part series for setting up Kubernetes and deploying containerized applications! Though kubegrid will automate all this for you, it’s worthwhile to understand what goes into setting up infrastructure, deploying Kubernetes, and shipping your software onto that cluster.

Before you can deploy Kubernetes, you’ll need to create the underlying infrastructure; that is, the cloud machines that Kubernetes (and your application) will run on. This guide will use AWS, but instead of working with AWS directly, you’ll leverage a tool called Terraform to easily build that infrastructure.


Prerequisites

AWS Account

Sign-up for an AWS account (free-tier is fine, though you will need to provision a non-free EC2 instance to run the Kubernetes master node).

Create a new access key and note/download the access key ID and corresponding secret:

https://console.aws.amazon.com/iam/home?#/security_credentials

Install Terraform

Follow the guide here to install Terraform and verify the installation:Installing Terraform | Terraform

SSH Key

You’ll set up SSH access to the EC2 instances as part of the provisioning. If you don’t already have an SSH public/private key pair, follow the guide here to generate one:How to use ssh-keygen to generate a new SSH key

This guide assumes your public key is in ~/.ssh/id_rsa.pub


Section 1: Terraform Variables

We’ll start by creating a file to define configuration variables needed by Terraform.

Create a file named variables.tf.

First up are the AWS access key ID and secret for your AWS account. Since you’ll potentially commit these Terraform configuration files to source control, don’t actually populate those variable here.

variable "aws_access_key" {}
variable "aws_secret_key" {}

Select the AWS region and availability zone that you’ll deploy to. Note that the snippet below is using the region us-west-1. You can use the same or choose a region closer to you.

variable "aws_region" {
default = "us-west-1"
}

variable "availability_zone" {
default = {
"us-west-1" = "us-west-1a"
}
}

The next section defines the Amazon Machine Image (AMI) used to image the EC2 instances. We’ll be using Container Linux.

Be sure to visit https://coreos.com/os/docs/latest/booting-on-ec2.html to select the AMI ID that corresponds to your region.

# CoreOS AMIs (HVM)
# https://coreos.com/os/docs/latest/booting-on-ec2.html
variable "amis" {
default = {
"us-west-1" = "ami-08d3e245ebf4d560f"
}
}

Lastly, add the location of your SSH public key.

variable "ssh_public_key_file" {
default = "~/.ssh/id_rsa.pub"
}

Here’s the complete variables.tf file.

variable "aws_access_key" {}
variable "aws_secret_key" {}

variable "aws_region" {
default = "us-west-1"
}

variable "availability_zone" {
default = {
"us-west-1" = "us-west-1a"
}
}

# CoreOS AMIs (HVM)
# https://coreos.com/os/docs/latest/booting-on-ec2.html
variable "amis" {
default = {
"us-west-1" = "ami-08d3e245ebf4d560f"
}
}

variable "ssh_public_key_file" {
default = "~/.ssh/id_rsa.pub"
}

Section 2: Secrets File

Create a file named secrets.tfvars.tfvars files define variables that can be referenced from .tf files.

Since this file contains your AWS credentials it’s a best practice to not add it to source control.

aws_access_key="<your access key ID>"
aws_secret_key="<your secret access key>"

Section 3: AWS Provider

Create a file named aws.tf.

This tells Terraform that we’ll be using AWS and what credentials to use to access it.

provider "aws" {
access_key = "${var.aws_access_key}"
secret_key = "${var.aws_secret_key}"
region = "${var.aws_region}"
}

Section 4: Network

Create a file named network.tf. This is where you’ll ensure that all the ports that Kubernetes needs to use are open, while not exposing more of your infrastructure to the network than is necessary.

Start by defining the VPC, subnet, internet gateway, and route table / route table association:

# VPC
resource "aws_vpc" "demo" {
cidr_block = "10.0.0.0/16"
tags {
Name = "demo"
}
}

# Subnet
resource "aws_subnet" "demo" {
vpc_id = "${aws_vpc.demo.id}"
cidr_block = "10.0.0.0/24"
availability_zone = "${lookup(var.availability_zone, var.aws_region)}"
tags {
Name = "demo"
}
}

# Internet gateway for subnet
resource "aws_internet_gateway" "demo" {
vpc_id = "${aws_vpc.demo.id}"
tags {
Name = "demo"
}
}

# Route table for subnet
resource "aws_route_table" "demo" {
vpc_id = "${aws_vpc.demo.id}"
route {
cidr_block = "0.0.0.0/0"
gateway_id = "${aws_internet_gateway.demo.id}"
}
tags {
Name = "demo"
}
}

# Route table association for subnet
resource "aws_route_table_association" "demo" {
subnet_id = "${aws_subnet.demo.id}"
route_table_id = "${aws_route_table.demo.id}"
}

Next add a security group that will apply to the master node:

# Security group for Kubernetes master node
# https://kubernetes.io/docs/setup/independent/install-kubeadm/#master-node-s
resource "aws_security_group" "master-node" {
name = "demo-security-group-master"
vpc_id = "${aws_vpc.demo.id}"

# SSH
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

# Kubernetes API server
ingress {
from_port = 6443
to_port = 6443
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

# etcd server client API
ingress {
from_port = 2379
to_port = 2380
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

# Kubelet API
ingress {
from_port = 10250
to_port = 10250
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

# kube-scheduler
ingress {
from_port = 10251
to_port = 10251
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

# kube-controller-manager
ingress {
from_port = 10252
to_port = 10252
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

# allow all outbound traffic
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}

tags {
Name = "demo"
}
}

Follow up with a security group that will apply to the worker node(s):

# Security group for Kubernetes worker node(s)
# https://kubernetes.io/docs/setup/independent/install-kubeadm/#worker-node-s
resource "aws_security_group" "worker-node" {
name = "demo-security-group-worker"
vpc_id = "${aws_vpc.demo.id}"

# SSH
ingress {
from_port = 22
to_port = 22
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

# Kubelet API
ingress {
from_port = 10250
to_port = 10250
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

# NodePort services
ingress {
from_port = 30000
to_port = 32767
protocol = "tcp"
cidr_blocks = ["0.0.0.0/0"]
}

# allow all outbound traffic
egress {
from_port = 0
to_port = 0
protocol = "-1"
cidr_blocks = ["0.0.0.0/0"]
}

tags {
Name = "demo"
}
}

Section 5: Instances

Create a file named instances.tf.

Start by defining a resource for the SSH key pair that you’ll use for login access to your instances. Note that we’re referencing the ssh_public_key_file variable we defined earlier in variables.tf.

# SSH key pair
resource "aws_key_pair" "demo" {
key_name = "my-public-key"
public_key = "${file(var.ssh_public_key_file)}"
}

Next define an instance for the Kubernetes master node. Note that we’re using a t3.small, which is not in the AWS free-tier. A t2-micro is not enough to run the master node.

# Kubernetes master node
resource "aws_instance" "master-node" {
ami = "${lookup(var.amis, var.aws_region)}"
instance_type = "t3.small"
subnet_id = "${aws_subnet.demo.id}"
vpc_security_group_ids = ["${aws_security_group.master-node.id}"]
key_name = "${aws_key_pair.demo.id}"
associate_public_ip_address = true
tags {
Name = "demo-master-node"
}
}

Add 1 or more instances for the Kubernetes worker nodes. For this demo we’ll create 2 workers.

# Kubernetes worker node 1
resource "aws_instance" "worker-node-1" {
ami = "${lookup(var.amis, var.aws_region)}"
instance_type = "t2.micro"
subnet_id = "${aws_subnet.demo.id}"
vpc_security_group_ids = ["${aws_security_group.worker-node.id}"]
key_name = "${aws_key_pair.demo.id}"
associate_public_ip_address = true
tags {
Name = "demo-worker-node-1"
}
}

# Kubernetes worker node 2
resource "aws_instance" "worker-node-2" {
ami = "${lookup(var.amis, var.aws_region)}"
instance_type = "t2.micro"
subnet_id = "${aws_subnet.demo.id}"
vpc_security_group_ids = ["${aws_security_group.worker-node.id}"]
key_name = "${aws_key_pair.demo.id}"
associate_public_ip_address = true
tags {
Name = "demo-worker-node-2"
}
}

Section 6: Output

Now define an output file output.tf that will show you the public IP addresses that will get assigned to your instances during their creation.

You’ll need those for a subsequent tutorial for installing Kubernetes.

output "public_ip_master_node" {
value = "${aws_instance.master-node.public_ip}"
}

output "public_ip_worker_node_1" {
value = "${aws_instance.worker-node-1.public_ip}"
}

output "public_ip_worker_node_2" {
value = "${aws_instance.worker-node-2.public_ip}"
}

Section 7: Launch Infrastructure!

Before we can launch our infrastructure we need to initialize Terraform. You should only need to do this once.

When you run these commands, Terraform will look for all files with extension .tf and use those to define your configuration. In this tutorial, we broke up the configuration into multiple files that contained logically grouped elements, but this is not necessary. You can have as many or as few .tf files as you’d like and organize them in the way that makes sense to you.

terraform init

Now launch the infrastructure!

terraform apply -var-file=secrets.tfvars

You’ll get a full summary from Terraform of what will be deployed. Enter yes to confirm and proceed with the deployment.

After deployment you’ll get the output of the public IP addresses as we defined above in output.tf. If you need to redisplay the output just run the following:

terraform output

Section 8: Deploy Kubernetes

Now your infrastructure is up and running. Follow our blog post Deploying a Kubernetes cluster on Container Linux with kubeadm (Part 2) to deploy Kubernetes onto your freshly created infrastructure!


Section 9: Teardown

To wrap up this exercise and bring down your infrastructure, run the following:

terraform destroy -var-file=secrets.tfvars