Creating a Canary Deployment Pipeline in Amazon ECS Using AWS App Mesh

This article is a translation of Create a pipeline with canary deployments for Amazon ECS using AWS App Mesh.

This article describes how to implement a canary deployment strategy for an application running on Amazon Elastic Container Service (Amazon ECS) in combination with AWS App Mesh. If you want to do a canary deployment with AWS CodeDeploy using ALB's weighted target groups, see this article.

With container orchestras such as Amazon ECS and Amazon EKS, customers around the world can run complex applications on a large scale. However, to take advantage of the capabilities of these tools, customers need to think about managing connectivity, operations, and security between microservices in a distributed architecture in order to run a highly distributed set of services.

AWS App Mesh is a service mesh that provides application-level networking, facilitating communication between services on different types of computing infrastructure. It also provides end-to-end visibility and helps ensure high availability of your application. App Mesh provides consistent visibility and network traffic control across all services within your application.

1. Canary deploy

Canary deploys / releases allow you to deploy a new version of a particular application and then switch traffic in user-defined percentages according to your business needs. You can also monitor the status of the new version so that traffic can be automatically switched to the older version in the event of a problem. You can reduce the impact of bugs in newer versions of your application on your end customers. Not only does this approach help implement new features, it also addresses the testing needs of complex distributed microservices, allowing you to control some of your traffic and send it to a new version.

2. Architecture overview

The following architectural diagram outlines the AWS services used to create the pipeline.

We will use an application called Yelb to demonstrate this architecture. On Yelb's application, when a user votes for a set of options, such as a restaurant, the pie chart is dynamically updated based on the vote. In addition, Yelb keeps track of the number of pageviews and prints the hostname of the yelb-appserver instance. This hostname will be the IP address (when using awsvpc network mode with Amazon ECS) of the task that receives API requests for voting and page refreshing. Yelb contains the following components:

Yelb's architecture looks like this:

The Yelb configuration uses ephemeral disks for all containers. Please note that running the database this way is for demonstration purposes only.

3. Infrastructure setup

This article uses AWS's Oregon region (us-west-2).

Subsequent steps require an environment with the necessary tools. Use an AWS Cloud9 instance to run this tutorial. If you want to create a Cloud9 instance on your AWS account, see the steps from the beginning of the “Create a Workspace” chapter of the Amazon ECS Workshop to “Validate the IAM role”.

3.1 Requirements

There are some things that need to be installed and configured before you can create a pipeline. Start by cloning your GitHub repository.

 # ヘルパースクリプトと、ブログ投稿のリソースgit clone https://github.com/aws/aws-app-mesh-examples.gitcd aws-app-mesh-examples/blogs/ecs-canary-deployments-pipeline/# サンプルアプリケーションgit clone https://github.com/tiagoReichert/yelb.git && mv yelb microservices

A Makefile has been added to simplify the procedure. This Makefile runs a helper script located under the setup folder of the downloaded resource. These helper scripts have been created and tested on AWS Cloud9 instances and will be used in the following steps to facilitate resource creation. To understand the function of the helper script, please check the contents of the script.

Now let's start by exporting some environment variables (you are free to customize these variables).

 make set-env

You also need an Amazon Virtual Private Cloud (VPC) to provision a logically isolated section of your AWS cloud that launches AWS resources in a defined virtual network. VPCs allow you to launch AWS resources within your own defined virtual network. You have full control over your virtual network environment, including choosing your own IP address range, creating subnets, and configuring route tables and network gateways.

 make create-vpc

3.2 Create a service mesh with an Amazon ECS cluster and AWS App Mesh

Amazon Elastic Container Service (Amazon ECS) is a scalable, fast container management service that makes it easy to run, stop, and manage containers on your cluster. We'll create an empty Amazon ECS cluster with prerequisites for launching the ECS service and a service mesh on AWS App Mesh, which we'll use to perform a canary deployment on the microservice.

 make create-ecs-mesh

3.3 A collection of Prometheus metrics in an Amazon ECS cluster

CloudWatch Container Insights Prometheus Metrics Monitoring automates the detection of Prometheus Metrics from containerized systems and workloads. Prometheus is a toolkit for open source system monitoring and alerting. See “What is Prometheus?” In the Prometheus documentation for more information. Leveraging the concept of a dynamic service discovery mechanism, it discovers his ECS task definition based on the ARN of the ECS task definition based on regular expression-based service discovery.

 make create-monitoring-resources

4. Creating a pipeline

Now that all the requirements are installed and configured, you can proceed with creating a pipeline using AWS CodePipeline for each microservice.

4.1 Shared AWS CloudFormation stack

The shared AWS CloudFormation stack contains multiple AWS Lambda functions and one AWS Step Functions state machine that are responsible for the deployment stage of your pipeline. All configuration parameters are accepted as input, so all deployments can use the same resources.

AWS Step Functions state machines orchestrate multiple Lambda functions that perform canary deployments, using the CloudFormation template helpers available in per-microservices repositories. It switches traffic between multiple versions of the application, taking into account user-defined latency.

State Machine Workflow Overview:

Now let's create a shared CloudFormation stack.

 make create-shared-cloudformation-stack

Proceed to the next step only if the last command returned CREATE_COMPLETE .

4.2 Pipeline AWS CloudFormation Stack

The pipeline's AWS CloudFormation stack is designed to be deployed on a per-microservice basis. Create a pipeline that triggers a single AWS CodeCommit repository and a new commit to that repository's master branch.

Create four stacks with Yelb microservices for the demo. Changing the environment variable USE_SAMPLE_MICROSERVICES to False will create an empty AWS Code Commit repository with all the resources needed for your pipeline and you can use it in your own source code.

Since the Yelb architecture contains four microservices, I wrote a script to create all the AWS CloudFormation stacks I needed.

 make create-sample-microservice-pipeline-stack

Proceed to the next step only if the last command returns " All CloudFormation stacks created successfully! ".

5. Pipeline testing

Now that you have a complete architecture, let's see how it works.

5.1 Configuration file

First, let's take a look at the AWS CodeCommit repository, which has a microservice-specific code repository. There is a folder called specfiles with three files. Let's Dive Deep to these three files.

The following is an example layout of the directory structure of a microservices repository. Everything else in the AWS CodeCommit repository is the source code for each Yelb application.

5.2 Pipeline resources

The easiest way to see the resources created is to open AWS CodePipeline. You can see the four pipelines you just created on the AWS console.

If you open one of them, you will see the " Source ", " Build ", and " Deploy " steps. All of these steps are customizable. For example, you can add security checks, manual approvals, etc. before deploying.

1. The Source step monitors the master branch of your AWS CodeCommit repository for changes. You can access the repository by clicking the AWS CodeCommit link in the Source step.

2. The Build step builds the Docker image within AWS CodeBuild and saves it to Amazon ECR. You can see the build log by clicking the Details link from the Build step.

3. The Deploy step triggers a state machine for AWS Step Functions created by the shared AWS CloudFormation stack. Click the Details link to see its status.

From the Deploy step, click the Details tab to see the run-time inputs and outputs, as well as the AWS Lambda functions and execution logs for each step.

A successful deployment with 100% of the traffic switched to the new version and all health checks passed is shown in the image below.

If you look at the ECS / ContainerInsights / Prometheus namespace in CloudWatch Metrics, you shouldn't see the 5xx response code for the new version ( yelb-appserver- ).

5.3 Deploying a new version

Now let's see the Yelb application working and make some changes to see the pipeline working.

 source ~/.bash_profile && echo http://$APP_URI

Open the URL returned by the above command in your preferred browser. You should see a page similar to the following (it may take a few minutes for the DNS name to propagate):

You can see how the values ​​are updated by clicking on a few voting buttons. Did you notice anything during the vote?

In fact, I deliberately added a problem to the yelb-appserver microservice so that it increments by 2 for each vote. Now let's see how it applies when we fix that problem.

Open the AWS CodeCommit console and go to the yelb-appserver repository. Open the file modules/restaurantsdbupdate.rb and click the Edit button to fix the problem. Change the following line.

con.prepare('statement1', 'UPDATE restaurants SET count = count +2 WHERE name = $1') to con.prepare('statement1', 'UPDATE restaurants SET count = count +1 WHERE name = $1') , Commit

When I open the AWS CodePipeline console, yelb-appserver-pipeline shows In progress after a few seconds. Open it to see the progress of your deployment. Wait for the Deploy stage and refresh the browser tab of your Yelb application several times. You can see that the App Server host name has been switched from the original name as shown in the image below, because the canary has been deployed.

You can see what the weights are at the moment by opening the yelb-appserver microservices virtual router details in the AWS App Mesh console. You can also open the specfiles / deploy.json file in the AWS CodeCommit yelb-appserver repository and see the percent_step and wait_time parameters to see how long it takes to switch all traffic. This example uses the values percent_step: 10 and wait_time: 60 , which takes a total of 10 minutes to switch all traffic.

After the deployment is complete, try voting again in the Yelb application and you'll see that the votes are increasing by one.

5.4 Deploying a version that triggers rollback

Open the yelb-appserver.rb file from yelb-appserver in the AWS CodeCommit repository. Change the port of the production application (line 33) from 4567 to 4568 (or a port number other than 4567) and commit the changes.

This will trigger yelb-appserver-pipeline to deploy a new version of the non-working yelb-appserver microservice. Open yelb-appserver-pipeline in AWS CodePipeline and wait for the Deploy stage to be "In Progress". Then click Details under the Deploy stage. This page shows the visual workflow being deployed.

Try refreshing the browser that has the Yelb application open several times. Even if you reload, the voting data is not returned and you can see that you can not vote. This is because you are redirected to a new version that is not working.

After waiting a few minutes, you can see in the visual workflow that the rollback has occurred because the new version did not pass the health check. The point here is that the health check responsibility in this workflow is an AWS Lambda function that you can customize to suit your needs.

Looking at the CloudWatch metrics ECS / ContainerInsights / Prometheus namespace, the old version still returns response code 2xx , while the new version returns 5xx . After a while, you'll see that the new version of the metric has been removed (auto-rolled back) and no longer exists.

After a few refreshes of the browser that has the Yelb application open, it will work fine with previous versions.

6. Clean up

You can delete the resource created in the previous step with the following command.

 make delete-blog-contents

7. Summary

In this article, I showed you how to leverage AWS App Mesh and work with other AWS services such as AWS Code Pipeline and AWS Step Functions to implement a canary-style deployment strategy.

And if you want to know more about AWS App Mesh, check out the handy links below.

You can see upcoming features on the App Mesh roadmap and try out new features on the App Mesh preview channel. You can also join the App Mesh Slack community to share and discuss your experiences with your team and peers.

The translation was done by Solutions Architect Kaji. The original text is here.