TL;DR - This is the first part of two of this blog post series. In this part, the goal is to bundle a simple JRuby Application into a war file using Warbler and then build a base image in AWS (AMI) using Packer. For the second part, we will create the necessary infrastructure to spin up an EC2 instance based on our base image (AMI) using Terraform.

All the code samples are in this GitHub repository.

Disclaimer: This blog does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my opinion.

Context

Nowadays we all hear the buzz word microservices. A lot of companies started to break down their monolithic applications into several microservices. We all hear that microservices are the best thing in the world and only good things come with them. But let me tell you, they’re not.

When talking about microservices, we want them to be resilient, scalable, immutable and fully automated processes for building and deploying and that’s not easy to accomplish!

In this blog post series, I am going to demonstrate how Packer and Terraform can help in the build and deployment processes.

Motivation

Since I joined the platform team at Talkdesk I have been working with several technologies, including Elixir, Kotlin, Packer, Terraform but mainly with JRuby.

I was quite surprised with Packer and Terraform and since DevOps is a topic I’m very interested in I wanted to explore them more in depth on my free time. In these blog posts, I will expose my learnings and experiences.

Let’s get started!

The Application

To better demonstrate how it works it is better to have a real application working. For that reason, I decided to create a simple JRuby Sinatra application that has a root endpoint (GET) which returns a 200 with a “Hello World” in the body.

app/app.rb

require 'sinatra'

get '/' do
  "#{['Hello', 'Hi', 'Hey', 'Yo'][rand(4)]} World!"
end

Building the Base Image (AMI)

Before we start building the base image, we need to understand the requirements to run the application in a production/production-like environment.

Since it’s a JRuby application, we will need to have Java JDK installed and depending on the strategy adopted to deploy it we may also need the JRuby binaries and a Java Web Server.

There are a few strategies for deployment of the application, but for this blog post, I decided to bundle the application using Warbler into a war file. This way we avoid to install JRuby binaries and also a Java Webserver which makes the building process simpler.

Regarding building base images for production you must have in consideration security concerns: you should do your own OS Hardening, ensure that root user doesn’t run your application, logging stuff, permissions, etc.

Disclaimer: The code samples provided here are not intended to be used in production and are not production ready. In case you use them it’s your responsibility.

Creating the Packer configuration source

If you don’t know what Packer is, I advise you to start by reading the starting guide. But in short, Packer is a tool to create images for multiple providers in parallel from a single source configuration. It can also provision those machines using shell scripts, Chef, Puppet, Ansible and other similar tools.

Let’s see how a Packer template looks like.

build/base-image.json

{
  "variables": {
    "aws_access_key": "{{env `AWS_ID`}}",
    "aws_secret_key": "{{env `AWS_SECRET`}}",
    "aws_region": "eu-west-1",
    "aws_base_ami": "ami-402f1a33",
    "aws_instance_type": "t2.micro"
  },
  "builders": [
    {
      "type": "amazon-ebs",
      "access_key": "{{user `aws_access_key`}}",
      "secret_key": "{{user `aws_secret_key`}}",
      "region": "{{user `aws_region`}}",
      "source_ami": "{{user `aws_base_ami`}}",
      "instance_type": "{{user `aws_instance_type`}}",
      "ssh_username": "admin",
      "ami_name": "base-image-openjdk-8-{{timestamp}}"
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "inline": [
        "echo 'Waiting 180 seconds for cloud-init'",
        "timeout 180 /bin/bash -c 'until stat /var/lib/cloud/instance/boot-finished &>/dev/null; do echo waiting...; sleep 10; done'"
      ]
    },
    {
      "type": "shell",
      "inline": [
        "echo 'deb http://ftp.debian.org/debian jessie-backports main' | sudo tee /etc/apt/sources.list.d/backports.list > /dev/null",
        "sudo apt-get update",
        "sudo DEBIAN_FRONTEND=noninteractive apt-get install -y -t jessie-backports -q openjdk-8-jre-headless ca-certificates-java"
      ]
    }
  ]
}

The first “block” is the Variables where the default values are defined. You can also define environment variables instead of values. If you don’t want the defaults, you can always specify the values when running the packer cli by doing this: packer build -var 'aws_region=eu-central-1'.

The second “block” is the Builders that defines in which platform you want to build the image (AWS, Azure, DigitalOcean, etc.). In this configuration we have an Amazon EC2 Builder with a basic configuration. The one thing that’s worth mentioning is the ssh_username, you will have to change it depending on which distribution you use. In this example, we will use Debian Jessie.

The third “block” is the Provisioners. Here we have two Remote Shell provisioners, the first waits for the machine to finish booting. The second installs the Java OpenJDK 8.

So far so good. Now we are in the position to validate our packer configuration file, and if everything is fine we can build our first image.

Validate and build the base image

cd /my_project/build

echo "validating the configuration file"
AWS_ID=my_id AWS_SECRET=my_secret packer validate base-image.json

>Template validated successfully.

echo "build the image"
AWS_ID=my_id AWS_SECRET=my_secret packer build base-image.json

The building process will spin up an EC2 instance based on the AMI created. After the image is ready, it will provision the machine by installing Java OpenJDK 8 and if everything went fine it will stop the EC2 instance and then creates the AMI based on that machine state. After the creation of the AMI, it will terminate the instance and output the AMI id.

At this point the base image is ready. Now it’s time to prepare our application.

Bundle the Application into a war file

Warbler is a gem that enables you to turn your JRuby application into a Java jar or war file, and there are some options while using it that has different results so if you are not aware of it, I suggest you to read the documentation. Below are some references:

• Warbler Documentation
• down the JRuby rabbit hole
• The Frankenjar: The Easiest Ruby App Deployment Ever
• Frankenwar: the Frankenjar revisited

For this example, we will use Warbler to create an executable war file. This means that Warbler will embed Jetty Web Server and that way it will run on its own.

cd /my_project

gem install warbler
bundle install

warble executable war

> rm -f app_name.war
> Creating app_name.war

Let’s run it java -jar app_name.war, it will start the embedded Jetty Web Server, and you should be able to access the application on http://localhost:8080

INFO::main: Logging initialized @264ms
INFO:oejr.Runner:main: Runner
INFO:oejs.Server:main: jetty-9.2.9.v20150224
INFO:/:main: INFO: jruby 9.1.8.0 (2.3.1) 2017-03-06 90fc7ab 
INFO:/:main: INFO: using a shared (threadsafe!) runtime
INFO:oejsh.ContextHandler:main: Started o.e.j.w.WebAppContext@47089e5f{...}
INFO:oejs.ServerConnector:main: Started ServerConnector@29ab3f1d{HTTP/1.1}{0.0.0.0:8080}
INFO:oejs.Server:main: Started @9247ms

Glue them together

We have now our base image and our JRuby application bundled in a war file. We are ready to pack it into an image to be ready to deploy. Let’s create the packer configuration file to build our application image to be deployed.

build/app-image.json

{
  "variables": {
    "aws_access_key": "{{env `AWS_ID`}}",
    "aws_secret_key": "{{env `AWS_SECRET`}}",
    "aws_region": "eu-west-1",
    "aws_base_ami": "{{env `BASE_AMI`}}",
    "aws_instance_type": "t2.micro"
  },
  "builders": [
    {
      "type": "amazon-ebs",
      "access_key": "{{user `aws_access_key`}}",
      "secret_key": "{{user `aws_secret_key`}}",
      "region": "{{user `aws_region`}}",
      "source_ami": "{{user `aws_base_ami`}}",
      "instance_type": "{{user `aws_instance_type`}}",
      "ssh_username": "admin",
      "ami_name": "my-app-name-{{timestamp}}"
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "inline": [
        "echo 'Waiting 180 seconds for cloud-init'",
        "timeout 180 /bin/bash -c 'until stat /var/lib/cloud/instance/boot-finished &>/dev/null; do echo waiting...; sleep 10; done'"
      ]
    },
    {
      "type": "file",
      "source": "toupload/",
      "destination": "/home/admin/"
    }
  ]
}

That’s our configuration file! If you notice it’s identical to the base-image configuration file. But in this case it didn’t install OpenJDK since it’s already available in our base image and the only thing it does it upload our app.war to the machine.

Now probably you should be asking yourself, why didn’t you upload the application in the base-image configuration file? The answer is simple, avoid points of failure. If the application needs OpenJDK 8 to run why download and install it every single time you want to build your application? It will take more time, and it may have problems downloading it.

Spin up an EC2 Instance to run the application

Now that our application image is ready is time to see it working.

Go to AWS Console -> EC2 -> Images AMIs.

Click in Launch -> Next: Configure Instance Details and here we need to do define how does our application start run when a new instance is launched.

Here we are starting our application, of course we could have daemonize it and make it “autostart” on boot and that way it won’t be necessary to add anything here.

Note: The user data you defined when the machine run that commands will be executed by the root user which is not recommended at all.

After adding the User data click in Next: Add Storage -> Next: Add Tags -> Next: Configure Security Group. Here you will want to add a new security group that will allow income traffic to port 8080 which is where our application is running on.

Now hit Review and Launch verify if everything is ok, Launch and now assign your key pair to the instance.

After the instance finishes the initialize process if you go to the given public DNS on port 8080 you should see something like this:

Wrap up

I know there was a lot of stuff being explained here and a lot of concepts if you are quite new to these topics. But we were able to build a base image, bundle the application in a war file, build an application image and then launch it in AWS using custom User Data to start our application.

The part number two will be published in a couple of days will be more lightweight and will focus on how to automate the deploys using terraform.