Mastering CloudFormation for API Gateway Deployments

I recently spent a fair amount of time trying to write a CloudFormation template for API Gateway to do precisely what I wanted to do. I struggled to make this happen, and failed to find good community resources to supplement the official documentation. One forum thread seemed to describe exactly what I needed, but the conclusion in the thread seemed to indicate that it wasn’t possible. Nonetheless, it was helpful in understanding some of the problems I was facing. Here is what I wanted CloudFormation to do with API Gateway:

• Create an API instance 
• Configure a stage 
• Deploy API changes from swagger 

Through trial and error and discovering basic concepts in CloudFormation, I was able to get exactly what I wanted. I hope this post helps others write a perfect template for their needs.

Problems I Ran Into

As I attempted different appraoches, I experienced a few common problems:

• Deployment of revised swagger would not occur 
• Updating the stack with revised swagger would error with “stage already exists” 
• No deployment history

CloudFormation Basics

Being new to CloudFormation, I didn’t fully grok some key CloudFormation concepts which was a barrier to getting my template right. The primary concept is that CloudFormation templates dictate desired state, not a set of operations to perform. If a resource is defined in your template, it will be created. If a resource already exists, it will not be created, but can be updated if its properties change. If a resource is removed from the template it will be deleted.

API Gateway Basics

• Swagger is used to define a REST API. 
• That REST API is deployed. 
• A stage references a deployment to make the API available via an endpoint.

The Template

---
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: Setup our API Gateway instances

Parameters:
  StageName:
    Type: String
    Default: 'example_stage'
    Description: 'The name of the stage to be created and managed within our API Gateway instance.'
    
Resources:

  Api:
    Type: AWS::ApiGateway::RestApi
    Properties:
      Name: ExampleApi
      EndpointConfiguration: 
        Types: 
        - REGIONAL

      # The body should contain the actual swagger
      Body: $SWAGGER_DEFINITION$

  # Timestamp is added so that each deployment is unique. Without a new timestamp, the deployment will not actually occur
  ApiDeployment$TIMESTAMP$:
    Type: AWS::ApiGateway::Deployment
    DependsOn: [ Api ]
    # we want to retain our deployment history
    DeletionPolicy: Retain    
    Properties:
      RestApiId:
        Ref: Api
      
  ApiStage:
    Type: AWS::ApiGateway::Stage
    DependsOn: [ApiDeployment$TIMESTAMP$]
    Properties:
      RestApiId:
        Ref: Api
      DeploymentId:
        Ref: ApiDeployment$TIMESTAMP$
      StageName: {Ref: StageName}
      MethodSettings:
        - ResourcePath: "/*"
          HttpMethod: "*"
          LoggingLevel: INFO
          MetricsEnabled: true
          DataTraceEnabled: true
          
Outputs:
  Endpoint:
    Description: Endpoint url
    Value:
      Fn::Sub: 'https://${Api}.execute-api.${AWS::Region}.amazonaws.com'          

Key Points

The deployment resource can specify a StageName and StageDescription. If this is done the deployment will implicitly create a stage that is not managed by CloudFormation. If we attempt to create a stage with the same name we’ll get an error that it already exists. Instead we’re better off creating a deployment that doesn’t prescribe anything about the stage and explicitly defining a stage resource.

Each deployment needs a unique id. Otherwise, CloudFormation will determine the deployment already exists and will not create a deployment. Without a new deployment, changes to our REST API will not be visible. Furthermore, we set a deletion policy on our deployment to retain the deployment. Otherwise, when the timestamp on our deployment changes, CloudFormation will want to delete the old deployment. If this deletion occurs, the deployment is removed from our stage deployment history. With this policy, the deployment is removed from the stack but not deleted from API Gateway.

EDIT:
A script must be written to preprocess the the template and replace the $TIMESTAMP$ token with the timestamp WITHOUT any separators e.g. 05012019355 Thanks to somebody for pointing this out in the comments.


Disclaimer: I wrote this post as an enthusiast. The content is not the official position of Amazon or AWS.

Doing taxes on a virtual machine

A few years back I started using a virtual machine (VM) to run my tax software. I hope by sharing my motivation and method, those new to virtual machines will gain some exposure and find them useful in other endeavors. Since this tackles some advanced configuration, it may also be helpful to those trying to use VirtualBox outside its default configuration. Virtual machines are essentially a completely different computer running as software on a physical computer. They allow you to run a different operating system, test software without affecting your regular machine, and easily snapshot your machine state and restore it at a later time. One limitation is that running a VM will use about twice as much memory, since you're running another instance of an operating system and whatever applications. Another limitation is that performance can be reduced for certain things, for example graphics intensives tasks like games or 3D visualization.

Two different VM's running on my machine

My main motivation for using a virtual machine to do my taxes was that I wanted to be able to use the software from either my laptop or my desktop interchangeably. Normally, one might choose to install on the desktop then simply remote desktop into the desktop. However, I also wanted to be able to access the software while other people were using desktop. Allowing multiple users at the same time on a single machine with Windows normally requires a server edition or modifying Windows in a way that is not consistent with your license agreement (e.g. https://www.serverwatch.com/server-tutorials/remote-desktop-connections-for-multiple-users-on-windows-10-and-windows-server-2012.html).

Another perceived benefit was not having to untidy my desktop with new installations every year. Instead these are all within the virtual machine. Even better, if installing one version uninstalls the previous years software, I would only have to restore a snapshot to retrieve a state that had the correct software installed to amend a return.

Snapshots of a VM, showing multiple states that were "saved", that can be restored or branched from

One key feature I need to enable for my use case of using the virtual machine from any machine inside my house is Remote Display. This is a setting directly on the VM (not the guest OS) that allows me to directly Remote Desktop into the VM, even when somebody is using the host machine.

Another huge reason I chose to use a VM is because it is fun experimenting and playing with new and different things.

Next year, I might play with using a EC2 instance for this so that I don't need to dual purpose my gaming machines for this and worry about memory management.

Prologue

Another drawback of VMs, having to keep additional machines up-to-date: