Introduction

In my previous blog post, I showed how to deploy an AWS Lambda function that runs a Postman collection to test the public API, OpenWeather. However, you may have an API that is not (or want to be) publicly accessible. Yet, you may still have a requirement to perform automated API testing.

In this post, I will be demonstrating how to test your API endpoints that are within an AWS VPC private subnet. Once again, I will be using the same Postman collection runner Lambda function but with added Lambda VPC configuration so that it can connect to your private subnet. We will also set up a CloudWatch metric filter on the API test logs to show our API testing results in a graph.

To quickly set up a private API, I will be using Mockoon on an EC2 instance that is launched in a private subnet. I will be mocking the Kubernetes API using Mockoon and creating associated tests for it in Postman.

Pre-requisites

This demonstration requires the following applications and tools:

AWS Account access
Postman - Install Postman
Node.js - Install Node.js 18
AWS SAM CLI - Install the AWS SAM CLI
AWS CLI - Install the AWS CLI

Furthermore, you should have some familiarity with the following:

How to create a VPC with public and private subnets.
How to launch an EC2 with a security group and instance profile.

An API testing scenario

To demonstrate API testing with Postman using AWS Lambda, we have the following mock API testing scenario:

Our mock API will be the Kubernetes API. This will be mocked by using an EC2 instance running Mockoon with a mock sample of Kubernetes API.
Our API endpoint will be located in AWS VPC private subnet with limited private access from outside the subnet. This will be achieved by deploying the EC2 in a private subnet and with a restrictive Security Group.
A NAT Gateway will be required to provide outbound internet access to download Node.js and Mockoon packages onto the EC2.
The Postman collection runner Lambda function will be deployed within the same private subnet as the API. The API network connection will then only traverse within the subnet.

The diagram below shows the described API testing scenario.

Note that AWS EventBridge Schedule can be used to schedule the invocation of the Postman collection runner Lambda function for API testing. This is covered in my previous part 1 blog post.

Develop the API tests using Postman and Mockoon

In this section, we will create API tests in a Postman collection. The API tests will be developed against a running mock Kubernetes API using Mockoon.

Set up Mockoon with the mock sample Kubernetes API

To assist in creating a Postman collection to test the Kubernetes API, we can first run Mockoon with the mock sample Kubernetes API.

Download and install Mockoon to your local development environment. You can download Mockoon for your OS here: https://mockoon.com/download/

Next download the Kubernetes API mock sample project in Mockoon available here: https://mockoon.com/mock-samples/kubernetesio/. The default directory to place the mock API files is "<mockoon install directory>/mockoon/storage".

Launch Mockoon and click the open icon, then locate the downloaded Kubernetes API mock API file and open it.

Mockoon shows all the available API endpoints (paths and methods) as well as their mock responses.

For this demonstration, I will be using the Kubernetes pods status API. You can locate this in Mockoon by entering into the filter field "pods status" and selecting the endpoint GET api/v1/namespaces/:namespace/pods/:name/status.

You will notice that the configured response "Response 1 (200) OK" body contains a JSON payload with many of the field values being an empty string. You could set these field values to your desired mock response, however, for this demonstration we will leave them as empty strings.

Click on the Setting tab and see that the mock API URL is configured to the default value of localhost:3000.

To run the mock API click the green play button.

Leave the Mockoon mock API running so we can build our API tests against it using Postman in the next step.

Create a Postman collection to test your API

In this section, we will create a new Postman collection to test the running Mockoon mock Kubernetes API endpoint:

http://localhost:3000/api/v1/namespaces/mynamespace/pods/mypod/status

If you need a refresher on how to create tests in a Postman collection, see my previous blog post.

In Postman create a new collection called Kube_API_Tests with the following variables:

api_base_url - Initial & current value set to "localhost:3000"
expected-status - Initial & current value left blank

Create a new request named Pod status and configure it to GET the URL {{api_base_url}}/api/v1/namespaces/mynamespace/pods/mypod/status.

Add the following test code:

pm.test("Status code is 200", function () {
    pm.response.to.have.status(200);
});
let expectedStatus = pm.variables.get("expected_status");
pm.test("Check condition status", function () {
    var jsonData = pm.response.json();
    pm.expect(jsonData.status.conditions[0].status).to.eql(expectedStatus);
});

This will test the following:

Expect the response status code to be 200.
Expect the response body JSON status.conditions[0].status to equal the value of the set Postman variable expected_status.

Run the request and you should see both tests pass.

Note that the "Check condition status" test will pass when the Postman variable expected_status is left blank (becoming an empty string). The test will fail if set to any other value. This behaviour will be made use of to produce a mixture of pass and fail results later on.

You can see all the requests sent to your Mockoon mock API under the Logs tab.

Deploy a mock API using an EC2 instance and Mockoon CLI

In this section, we will deploy the Mockoon mock Kubernetes API onto an EC2 instance. The EC2 instance will be launched in a private subnet with a restrictive Security Group. To install Mockoon CLI on the EC2 instance, we will access its session terminal using AWS Systems Manager Session Manager

Launch an EC2 instance into a private subnet

To be able to use AWS Systems Manager Session Manager, create an EC2 IAM role that is attached with the AWS managed policy, AmazonSMMMaangedInastanceCore.

Next, create a Security Group that will be used for the EC2 instance. This Security Group is only assigned a single outbound rule that allows all outbound network access.

Launch an EC2 instance with an Amazon Linux 2 AMI into a private subnet, attached with the created Security Group and EC2 IAM Role. You can use an AMI of your choice however, the commands to install Mockoon may differ from mine provided below.

Make note of the EC2 instance security group ID, private subnet ID and private IP address. These will be used later as inputs when deploying and invoking the Lambda Postman collection runner.

Connect to the EC2 instance using AWS SSM Session Manager. This can be done from the AWS EC2 Console. Select your running EC2 instance then click Connect. Select the Session Manager tab, then click Connect.

This should then open the instance shell terminal within your web browser.

Set up Mockoon CLI with the mock sample Kubernetes API on the EC2 instance

To install Mockoon CLI, we first need to install Node.js.

The following commands can be used to install Node.js 16 using Node Version Manager on Amazon Linux 2:

# Download and install NVM
curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.3/install.sh | bash
. ~/.nvm/nvm.sh

# Insall and use Node.js 16
nvm install 16
nvm use 16

Install Mockoon CLI:

npm install -g @mockoon/cli

Run Mockoon CLI with the mock sample Kubernetes API:

$ mockoon-cli start --data https://raw.githubusercontent.com/mockoon/mock-samples/main/mock-apis/data/kubernetesio.json

Mock started at http://localhost:3000 (pid: 0, name: mockoon-kubernetes)

By default, Mockoon will run the API on port 3000.

Test that you can reach the mock Kubernetes API locally:

curl http://localhost:3000/api/v1/namespaces/mynamespace/pods/mypod/status

You can view the incoming requests to the mock Kubernetes API by tailing the Mockoon logs:

$ tail -f ~/.mockoon-cli/logs/mockoon-kubernetes-out.log

{"level":"info","message":"Server started on port 3000","mockName":"mockoon-kubernetes","timestamp":"2023-04-30T01:09:35.766Z"}
{"level":"info","message":"GET /api/v1/namespaces/mynamespace/pods/mypod/status | 200","mockName":"mockoon-kubernetes","timestamp":"2023-04-30T01:11:01.653Z"}

Deploy AWS Lambda Postman collection runner into the VPC

To perform the API tests, we will use the same AWS Lambda Postman collection runner used in my previous part 1 post with some modifications. The modifications are:

Configuring the Lambda function to be deployed to a VPC private subnet.
Adding to the Lambda log messages the test assertion results in JSON format (for more flexible use with CloudWatch Logs metric filters).

You can find the source code for the Lambda SAM project at: github.com/FreddyCLH/aws-lambda-postman-col..

Create a Security Group for the Lambda function

Lambda functions that are deployed into a VPC require a Security Group. Create a Security Group for the Lambda function with only a single outbound rule that allows all outbound network access.

To allow the Lambda function network access to the mock API hosted on the EC2 instance, we will update the EC2 instance Security Group to allow inbound access from the Lambda function's Security Group. The API test connection will be using TCP port 3000 (Mockoon's default).

Modify the Lambda Postman collection runner for VPC deployment

Clone the AWS Lambda Postman collection runner project:

git clone https://github.com/FreddyCLH/aws-lambda-postman-collection-runner.git

Export the Postman collection that you created in Create a Postman collection to test your API (Kube_API_Tests.postman_collection.json) and place it in the project collections folder.

Modify the SAM deployment template.yaml file to configure the Lambda to be deployed into a VPC subnet.

In the template.yaml Parameters section, add the new parameters subnetIds and securityGroupIds:

Parameters:
  postmanCollection:
    Type: String
    Description: Postman collection file to run.
    Default: default.postman_collection.json
  subnetIds:
    Type: CommaDelimitedList
  securityGroupIds:
    Type: CommaDelimitedList

In the template.yaml, in lambdaFunctionPostmanCollectionRunner resource under Properties, add the following VpcConfig:

 Resources:
  lambdaFunctionPostmanCollectionRunner:
    Type: AWS::Serverless::Function
    Properties:
      Handler: src/handlers/index.handler
      Runtime: nodejs18.x
      # ...ommitted for brevity...
      VpcConfig:
        SubnetIds: !Ref subnetIds
        SecurityGroupIds: !Ref securityGroupIds

Next, modify the source code file src/handlers/index.js to log the API test assertion pass and fail stats in JSON format.

In the try block of the main handler function, under the line:

newmanResult = await newmanRun(outpath, postmanCollection, event);

Add the following code:

    let metricsMessage = {
      "metricsMessage": "true",
      "stat": "assertions passed",
      "count": newmanResult.run.stats.assertions.total - newmanResult.run.stats.assertions.failed,
    }
    console.log(JSON.stringify(metricsMessage));
    metricsMessage = {
      "metricsMessage": "true",
      "stat": "assertions failed",
      "count": newmanResult.run.stats.assertions.failed,
    }
    console.log(JSON.stringify(metricsMessage));

The code should now look like the following:

exports.handler = async (event) => {
  // ...ommitted for brevity...

  console.log(
    "Begining API Tests with Postman collection " + postmanCollection
  );
  console.log("Output path: " + outpath);

  try {
    // Trigger Newman, then encode the resulting XML file to be returned
    newmanResult = await newmanRun(outpath, postmanCollection, event);
    let metricsMessage = {
      "metricsMessage": "true",
      "stat": "assertions passed",
      "count": newmanResult.run.stats.assertions.total - newmanResult.run.stats.assertions.failed,
    }
    console.log(JSON.stringify(metricsMessage));
    metricsMessage = {
      "metricsMessage": "true",
      "stat": "assertions failed",
      "count": newmanResult.run.stats.assertions.failed,
    }
    console.log(JSON.stringify(metricsMessage));

Use AWS SAM to deploy the Lambda

Build and deploy the SAM project:

sam build
sam deploy --guided

Enter the following parameter values when prompted:

postmanCollection - Enter the Postman collection file name Kube_API_Tests.postman_collection.json
subnetIds - Enter the private subnet ID where the mock API EC2 instance is located (see Launch an EC2 instance into a private subnet).
securityGroupIds - Enter the created Lambda function Security Group ID (see Create a Security Group for the Lambda function)

When prompted to "Deploy this Changeset?", enter "y".

Take note of the output lambdaFunctionName. This will be used later when invoking the specified Lambda function name.

Invoke the Lambda function to test the API

Create the file api_test_params.json with the following contents below. This contains the Postman variables values to be used as the Lambda input payload. For api_base_url, provide the mock API EC2 instance private IP address:

{
    "values": [
        {
            "key": "api_base_url",
            "value": "http://10.0.10.165:3000",
            "type": "default",
            "enabled": true
        },
        {
            "key": "expected_status",
            "value": "",
            "type": "default",
            "enabled": true
        }
    ]
}

Invoke the function with the following command, replacing ${FUNCTION_NAME} with your Lambda function name (obtained from the CloudFormation output lambdaFunctionName):

aws lambda invoke --function-name ${FUNCTION_NAME} \
  --payload file://api_test_params.json \
  invoke-response.json \
  --cli-binary-format raw-in-base64-out

Inspect the Lambda functions response in the invoke-response.json file. You should see StatusCode: 200 as well as the metrics about your run Postman collection.

{
  "StatusCode": 200,
  "message": "Run completed successfully!",
  "encoded": "....",
  "stats": {
    ...omitted for brevity...
    "assertions": {
      "total": 2,
      "pending": 0,
      "failed": 0
    },
    "testScripts": {
      "total": 2,
      "pending": 0,
      "failed": 0
    }
  }
}

Use CloudWatch Logs metric filters to graph API test results

The Lambda function writes the API test results into CloudWatch Logs. For better visibility of the API test results, we can create a CloudWatch Logs metric filter to extract the results into CloudWatch metrics and display them in a graph.

Create a metric filter for the API test results

To create a Metric filter, in the AWS Console CloudWatch Logs (console.aws.amazon.com/logs), locate the CloudWatch log group for the Lambda function. The default name for the log group has the name pattern: "/aws/lambda/${FUNCTION_NAME}".

Under Metric filters tab, click Create metric filter.

In Step 1 - Define Pattern, under Filter pattern, enter the following pattern, then click Next.

{ $.metricsMessage = "true" && $.stat = "*" && $.count = "*" }

In Step 2 - Assign Metric, under Filter name, enter "metricsMessage".

Configure the Metric details as follows:

Configure the Dimensions as follows then click Next.

In Step 3 - Review and create, click Create metric filter.

After creating the Metric filter, you can click on the Metric name "AssertionsCount" link to take you to the CloudWatch Metrics page.

View the API test results in a CloudWatch metrics graph

To view the API test results CloudWatch metrics, we first need to generate some API test results. To do this, invoke the AWS Lambda Postman collection runner function a few times. To generate a few failed test results as well, change the value for expected_status in api_test_params.json from an empty string to any other value and invoke the Lambda function.

The CloudWatch metrics are published under the namespace APITestKube and the metric name AssertionCount.

The dimensions "assertions passed" and "assertions failed" are available to view in a graph.

Conclusion

It is possible to test your private APIs (in an AWS VPC private subnet) by using the AWS Lambda Postman collection runner function configured to run in a VPC. In this blog post, we showcased how this can be done by creating a mock private Kubernetes API using an EC2 instance running Mockoon. We then deployed the Lambda function in the same private subnet as the mock API EC2 instance. By invoking the Lambda function, we were able to run our Postman collection API tests against the private Kubernetes API.

To simulate a restrictive inbound network access situation, the mock API EC2 instance was given a Security Group that only allowed inbound connections from the Lambda functions Security Group. Furthermore, because the Lambda function was deployed in the same subnet as the EC2 instance, its network connection for API testing traverses within the same subnet.

The Lambda functions logs that contained the API test results were extracted into CloudWatch metrics by using CloudWatch Logs metric filters. Some of the Lambda function logged messages were modified to output in JSON format. This provided more flexibility to extract values when creating the metric filter. The CloudWatch metrics for the API test result passed and failed assertions were plotted in a graph for better visibility of the API testing.

Freddy's AWS 2 cents