Container technologies such as Docker provide a standard approach to isolating an application and its dependencies so they can be distributed and executed across environments. Container Orchestration tools such as Kubernetes and Amazon’s Elastic Container Service (ECS) provide platforms for managing the deployment, execution and coordination of Container based applications.

Container Orchestration can bring significant benefits, for example potentially lowering infrastructure costs and complexity. However it can be difficult to determine how to introduce them into existing infrastructure. An ideal approach enables your team to become comfortable using the technology in a low risk way while also providing tangible benefits, even if you eventually decide to not use Containers in Production. This article explores one approach, using Container Orchestration for feature branch deployments.

Feature branch deployment

Feature Branch development is a common workflow when using Distributed version control systems such as Git. Development of Features (or user stories) take place in a dedicated branch and are only merged into the main branch once the feature has been completed and approved.

Creating a separate deployment environment to which the feature branch can be continuously deployed to as it is being developed could be a good starting point for using Containers, especially if you are already using a branch based development workflow. This could for example, allow QA to occur as the feature is being developed without having to wait for it to be integrated and deployed into Staging.

While spinning up a new environment per branch may seem daunting, leveraging “infrastructure as code” tools such as CloudFormation or Terraform provide a straightforward method for creating and destroying these environments on demand. The code can also be reused in Production and Staging if these environments are currently being managed manually.

A Container workflow may also simplify deployments. Container Orchestration platforms such as ECS provide standardised deployment API’s and tooling as opposed to ad hoc approaches often seen in traditional application deployments.

Feature Branch Deployment at Evergiving

Evergiving provide software solutions for the face to face fundraising industry. Fullback Systems collaborated with Evergiving to help them improve their deployment infrastructure.

Evergiving was interested to experiment with Containers and Orchestration in order to evaluate whether it would be a good fit given the small size of their operations team. Some of the aims of using containers were to reduce complexity of the deployment process as well as the infrastructure used to support the applications. Feature branch deployments where chosen as a low risk starting point.

Amazon ECS was chosen as the orchestration platform. Choosing a managed Orchestration solution significantly reduced the maintenance burden for the small operations team. It made sense to use an Amazon Web Services (AWS) product since the existing infrastructure is predominantly hosted within AWS. At the time Elastic Kubernetes Service (EKS) or ECS using the Fargate launch type where not available in the regions used by Evergiving.

Evergiving was already using Feature branch deployments for some applications by spinning up small Amazon Elastic Compute Cloud (EC2) instances for each branch. While this worked well enough, EC2 instances are slow to spin up and provision. AWS also imposes limitations on the number of EC2 that can be spun up per VPC.

The new architecture consists of a single ECS cluster backed by a small number of medium sized EC2 instances which can be scaled up or down depending on utilization. An Application Load Balancer (ALB) is used to route requests to the ECS services.

Since the uptime requirements are less stringent than production Spot EC2 Instances are used resulting in significant cost savings. The ECS scheduler takes care of rescheduling existing services to new instances in the event that spot instances are terminated.

As a first prototype shell scripts where used to push Docker images to Amazon Elastic Container Registry (ECR) repository and deploy the ECS service. These would eventually be replaced with Terraform configuration.

On the initial build of a new branch a Docker image is created and pushed to ECR and a new ECS service is created. A DNS record derived from the branch name is created which points to the ALB and the ALB is configured to perform host based routing to the corresponding ECS service. The development and testing teams are notified via Slack with a URL on which they can access the deployed application once its ready. On subsequent successful builds the service is redeployed using the latest container image.

Branch deletion in the origin Git repository triggers a webhook sending a signed request to an AWS API Gateway endpoint which proxies to a AWS Lambda function. If authorization succeeds the function determines the resources associated with the branch and deletes them.

Initial and subsequent Deployments of a branch are now much faster. Initial deployment of a branch now takes between 30 - 60 seconds as opposed to 5 - 10 minutes using the EC2 instances. Switching to a container based workflow also reduced the amount and complexity of deployment code.

The project gave the operations team hands on experience with Containerising an existing application and deploying it as an ECS service. Branch Deployments are relied upon by Developers and testers for QA which meant that problems around service performance and deployment where uncovered and rectified prior to introducing it to Production.

After approximately 6 months of using ECS for feature branches the team felt comfortable with the technology and started to deploy containers in production. Subsequent work was undertaken to use the ECS Fargate launch type, removing the need to manage EC2 instances in the ECS Cluster.

Conclusions

It can be challenging to introduce new technologies into your infrastructure while minimising risk to Production. Implementing Feature Branch deploys gave Evergiving valuable operational experience with Containers prior to using them in Production as well as the benefits of continuously deployed feature branches.

Moving from deploying to EC2 instances using an ad hoc process to using Container Orchestration with ECS allowed Evergiving to significantly reduce the complexity of deployments and infrastructure maintenance.

If you would like to discuss your own Container Orchestration or Cloud infrastructure needs please feel free to contact us at info@fullbacksystems.