Freddy's AWS 2 cents

Use AWS CodePipeline to build your project on GitHub

Freddy Ho's photo
Freddy Ho
·

8 min read

Use AWS CodePipeline to build your project on GitHub

Table of contents

Introduction

If you are using GitHub repository to store your code and are wanting to create an automated pipeline to build or deploy your code, then you may have turned towards using GitHub Actions. However, another option to consider is AWS CodePipeline. If your build or deployment requires the use of AWS services such as CloudFormation or ElasticBeanstalk, then AWS CodePipeline could be a good choice. AWS Codepipeline has integration with these and many more AWS Services. And even if you're not looking to use AWS services, you can still carry out custom pipeline tasks by using AWS CodeBuild with AWS CodePipelines.

With an AWS CodePipeline GitHub connection set up, any commits pushed to your code repository on GitHub will trigger the CodePipeline to run.

In this post, I'm going to show how easy it is to set up a connection between AWS CodePipeline and GitHub. Then, using AWS CodeBuild with AWS CodePipeline, run some commands to demonstrate its capability to achieve custom pipeline tasks.

Create a CodePipeline GitHub connection

You can create a CodePipeline GitHub connection to your own personal GitHub account or a GitHub organization. If you are connecting to a GitHub organization, then you must be an owner of the organization.

To create the GitHub connection, navigate to AWS CodePipeline in the AWS Console at console.aws.amazon.com/codepipeline

On the left-hand pane named Developer Tools, under Settings, choose Connections.

Choose Create connection.

Choose GitHub and provide a connection name then choose Connect to GitHub.

If you are not already signed in to your GitHub account, this will open a new GitHub sign-in web page to do so.

Choose Install a new app. This will take you to a GitHub page that will prompt you to install AWS Connector for GitHub.

The GitHub page will show a selection of GitHub organizations as well as your personal GitHub account that you can choose to install the AWS Connector for GitHub. Note that any account/organization that already has the AWS Connector installed will be greyed and display "Configure".

The below image shows my personal GitHub account, "FreddyCLH" already installed with the AWS Connector. The Organization "Freddy-CLH-Blog" is a GitHub organization that does not have the AWS Connector installed.

Choose where you want to install the AWS Connector for GitHub.

Select whether you want to provide the AWS Connector with access to All repositories or Only select repositories, then choose Install.

Note that you will be able to change the AWS Connector repository options at a later time (I'll cover this in the next section).

After choosing install, you should be redirected back to the AWS Console Connect to GitHub page. The GitHub Apps field should now be populated. Choose Connect.

Restrict CodePipeline GitHub connection to select repositories

You can configure your installed AWS Connector GitHub app with access to only select repositories. You must first have a GitHub repository created first to select it. To create a GitHub repository see GitHub Docs Creating a new repository.

To configure the AWS Connector with select repository access, from your GitHub account, in the upper-right corner, choose your profile photo, then choose Settings.

In the top-left corner, ensure you have the correct GitHub account (personal or Organization) selected. Navigate to Installed GitHub Apps:

  • For a personal GitHub account: on the left-hand side, under Integrations choose Applications.

  • For a GitHub Organization: on the left-hand side, under Third-party Access choose GitHub Apps.

Locate the app AWS Connector for GitHub and choose Configure.

Under Repository access select Only select repositories. Under the Select repositories drop down select your desired repositories, then choose Save.

Create a CodePipeline for your GitHub project

In this section, we will create a simple CodePipeline with two stages:

  • Source Stage - Clones the project repository from GitHub.

  • Build Stage - Use AWS CodeBuild to run build commands.

Firstly, we need to initialise a new Git project and configure GitHub as the remote repository. Use the following commands to do this, replacing aws-codepipeline-demo and git@github.com:Freddy-CLH-Blog/aws-codepipeline-demo.git with your own repository name and GitHub repository address:

mkdir aws-codepipeline-demo
cd aws-codepipeline-demo
git init
git remote add origin git@github.com:Freddy-CLH-Blog/aws-codepipeline-demo.git
git branch -M main

To provide CodeBuild with the required build commands, within your project directory create a new buildspec.yml file with the following contents:

version: 0.2

phases:
  install:
    runtime-versions:
      python: latest
    commands:
      - echo "Install phase"
  pre_build:
    commands:
      - echo "Pre-build phase"
  build:
    commands:
      - echo "Build phase"
  post_build:
    commands:
      - echo "Post-build phase"

For demonstration purposes, the buildspec.yml file simply installs the latest Python runtime and runs several echo commands. In a real build scenario, you would place your required build commands in the buildspec.yml.

Add and commit the buildspec.yml file then push it to GitHub with the following commands:

git add buildspec.yml
git commit -m "Add CodeBuild buildspec"
git push -u origin main

To create a new CodePipeline, navigate to AWS CodePipeline in the AWS Console at console.aws.amazon.com/codepipeline

On the left-hand pane named Developer Tools, under Pipeline, choose Pipelines.

Choose Create Pipeline.

The AWS CodePipeline Console will direct you with steps to create a new pipeline.

In Step 1 Choose pipeline settings:

  • Provide a Pipeline name.

  • Under Serice role, select New service role. This will create an appropriate IAM role for your CodePipeline to use.

  • Check "Allow AWS CodePipeline to create a service role..."

In Step 2 Add source stage:

  • Under Source provider, select GitHub (Version 2).

  • Under Connection, in the searchable field, select your created connection.

  • Select a Repository name and Branch name of your GitHub repository.

  • Under Output artifact format, choose CodePipeline default.

  • After entering the above, choose Next.

In Step 3 Add build stage:

  • Under Build provider dropdown, select AWS CodeBuild.

  • Under Project name, choose Create project. This will open a new window to create a new AWS CodeBuild project.

Provide a Project name for your CodeBuild project.

Under the Environment fields, choose the following options:

  • Environment image: Managed image

  • Operating system: Ubuntu

  • Runtime: Standard

  • Image: select the latest image

  • Image version: "Always use the latest..."

  • Environment type: Linux

For more information about AWS CodeBuild build environment options see: https://docs.aws.amazon.com/codebuild/latest/userguide/build-env-ref-available.html

Under Service role, select New service role. This will create an appropriate IAM role for CodeBuild to use.

Under Build specifications, choose Use BuildSpec file.

Click Continue to Pipeline, to create the CodeBuild project and return back to creating your CodePipeline.

The Project name field should be now populated, with your created CodeBuild project name.

Choose Next.

In Step 4 Add deploy stage, choose Skip deploy stage.

Review your CodePipeline settings then click Create pipeline.

Your new CodePipeline will run once after creating it. In the next section, we will look at viewing and triggering the CodePipeline.

Viewing and triggering CodePipeline

All your Pipelines are listed in the AWS CodePipeline Console (console.aws.amazon.com/codepipeline).

Choosing your pipeline name will show the most recent release state. The image below shows a release with the Source and Build stages succeeded.

To see the output of the Build stage run commands, choose AWS CodeBuild. This will bring you straight to the CodeBuild project build history.

The last entry under Build history is the most recent build that was run from the AWS CodePipeline.

In the Build history, under the Build run column choose the last entry.

Choose Build logs to see the output of the run commands.

Let's now make changes to the code repository in GitHub to trigger an automated release of our CodePipeline.

In the git project (aws-codepipeline-demo), add the python file app.py with the following contents:

print("Hello world using python!")

Next, modify the buildspec.yml file by adding the line python3 app.py under the build commands. Your updated buildspec.yml should now have the below contents:

 version: 0.2

phases:
  install:
    runtime-versions:
      python: latest
    commands:
      - echo "Install phase"
  pre_build:
    commands:
      - echo "Pre-build phase"
  build:
    commands:
      - echo "Build phase"
      - python3 app.py
  post_build:
    commands:
      - echo "Post-build phase"

Add and commit these changes with a suitable git comment.

git add app.py buildspec.yml
git commit -m "Run python app.py"

Push the changes to GitHub which will then trigger a new CodePipeline release to run.

git push

You should now see a new CodePipeline release in progress. Observe that the git comment is also shown:

After the release is complete, go to CodeBuild build history and check the latest Build logs. You should see the output of the added app.py python file:

Conclusion

CodePipeline can be configured to use a GitHub repository in the pipeline source stage. Any pushed changes to your repository in GitHub will then trigger the pipeline to run. This is powered by a CodePipeline GitHub connection resource with an associated "AWS Connector for GitHub app" installed into your GitHub account or organisation. You can then use CodeBuild with CodePipeline to run your required build commands.

The real power of CodePipeline is its integration with AWS services. In a future blog post, I will cover how to create a CodePipeline to provision a SQL database on Amazon RDS.

AWSaws-codebuildCodePipelineGitHubaws-codepipline