Deploying Django via AWS Copilot

Recently we needed to deploy a Django application. The brief was simple, it needed to be on AWS, cost-effective, have continuous deployment, and be easy for the client to bring in-house down the line, without needing specialist DevOps expertise. We’re fans of infrastructure as code and containerised solutions, in particular via Kubernetes, but that wasn’t quite fitting the brief.

After gathering further requirements such as expected load etc., we decided to launch on AWS Elastic Container Service (ECS) via AWS Copilot (https://aws.github.io/copilot-cli/). This would allow us to keep the containerisation and scalability, but with a dramatically reduced amount of code and complexity to maintain.

What is AWS Copilot?

Copilot describes itself as: “an open source command line interface that makes it easy for developers to build, release, and operate production ready containerized applications on AWS App Runner, Amazon ECS, and AWS Fargate”

From our perspective, it’s a tool that allows us to create multi-environment infrastructures via low maintenance manifest files and a few commands. Abstracting away the more complicated CloudFormation templates, while still giving us plenty of flexibility and the ability to drop to raw CloudFormation when required.

Django + Nginx example

We’re not going to dive into explaining Copilot, how to install, create a project, setup, exec to a container etc. as the Copilot docs already do a great job of that (https://aws.github.io/copilot-cli/docs/overview/).

Instead, we are going to cover the Django specifics of how our manifest file came together. Below is an example of a manifest for a load-balanced web service (https://aws.github.io/copilot-cli/docs/manifest/lb-web-service/), abbreviated in a couple of places in regards to secrets and variables. It defines a load balancer, 3 containers, environment secrets, and variables per environment for stage and prod.

name: project-name
type: Load Balanced Web Service

http:
  path: /
  target_container: nginx

image:
  build: ./Dockerfile
  port: 8000
  depends_on:
    startup: success

sidecars:
  startup:
    essential: false
    command: ["sh","-c","python manage.py migrate && python manage.py collectstatic --noinput"]
    build: ./Dockerfile
  nginx:
    port: 80
    image:
      build: ./Dockerfile.nginx
    depends_on:
       project-name: start

cpu: 512
memory: 1024
count:
  range: 1-10
  cooldown:
    in: 60s
    out: 30s
  response_time: 2s
exec: true
network:
  connect: true

environments:
  stage:
    http:
      alias: project-name.yourdoamin.com
      hosted_zone: <replace with aws hosted zone>
      healthcheck: "/_health"
    variables:
      DJANGO_CONFIGURATION: Stage
      ...
    secrets:
      DJANGO_DB_HOST:
        secretsmanager: "stage/projectname:django_db_host::"
      DJANGO_DB_PASSWORD:
        secretsmanager: "stage/projectname:django_db_password::"
      ...
    sidecars:
      startup:
        variables:
          DJANGO_CONFIGURATION: StageECS
          ...
        secrets:
          ... as per main container
 prod:
  ...

This manifest file consists of 5 key sections that are worth explaining in more detail, Each category below relates to a key in the above yaml.

http

Our load balancer config, the key thing here is that we point the target of the load balancer to the nginx sidecar (see below) rather than the Django application itself, which is the default in CoPilot. We will also later customise this to be environment specific.

image

Our main container i.e. the Django application. In this case, a Python Dockerfile that executes uwsgi. Note that collectstatic and migrations are not run in this docker file, more on that in sidecars.

sidecars

AWS Copilot has the concept of sidecars (https://aws.github.io/copilot-cli/docs/developing/sidecars/), these are containers that run alongside the main container to perform additional work:

• Startup - A container that runs before the main container (see main container depends on it) to trigger migrations and static collection on each new deployment i.e all the parts of Django we don’t want to be tied to the base application which will be scaled. This container is essential: false meaning once it’s completed it will stop, saving resources. This runs from the same Dockerfile, with an alternate entry point defined as the command. *

• Nginx - Runs nginx via an alternate Dockerfile and proxies to the main container in conjunction with the load balancer, it’s also dependent on the main container running. The Dockerfile and related nginx conf can be quite simple, noting that we allow the health check from any host for the load balancers to correctly assign the targets: *

Dockerfile.nginx

FROM nginx:alpine

RUN rm -f /etc/nginx/conf.d/*
ADD /etc/nginx.conf /etc/nginx/conf.d/nginx.conf

EXPOSE 80

CMD [ "nginx" , "-g" , "daemon off;" ]

---

/etc/nginx.conf

server {
   listen 80;

  location /_health {
    proxy_pass http://localhost:8000;
  }

  location / {
    if ($host !~* ^(stage.example.com|prod.example.com)$ ) {
        return 444;
    }
    
    proxy_pass http://localhost:8000;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection 'upgrade';
    proxy_set_header Host $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_cache_bypass $http_upgrade;

  }
}

cpu, memory etc

Various configurations regarding the size of the containers and the auto scaling options.

environments

Here we set environment-specific config, variables, and secrets. Note that currently variables and secrets need to be repeated across each container.

• http - define the name, hosted zone (the ARN of the AWS hosted zone), and where the health check should go to. As mentioned in the nginx conf we defined an alternate health check, responding to this via a Middleware in Django e.g.

class HealthCheckMiddleware(MiddlewareMixin):
    def process_request(self, request):
        if request.META["PATH_INFO"] == "/_health":
            return HttpResponse("pong")

• variables - environment variables read into the Django settings that don’t need to be secret.

• Secrets - secret environment variables read from AWS secrets manager (remember to assign the CoPilot tags in SSM as noted in the docs - https://aws.github.io/copilot-cli/docs/developing/secrets/). On the server these will be files rather than environment variables. To handle this we wrote a small wrapper that will attempt to read the variable from environ or file to handle this e.g.

settings.py

def get_env_variable(var_name, default=None, prefix="DJANGO"):
    value = None
    if prefix:
        var_name = f"{prefix}_{var_name}"
    if default is not None:
        value = os.environ.get(var_name, default)
    else:
        try:
            value = os.environ[var_name]
        except KeyError:
            error_msg = "Set the {} environment variable".format(var_name)
            print(error_msg)

    if value and os.path.isfile(value):
        with open(value) as f:
            return f.read()
    return value
...

# for the secret or environment variable called DJANGO_DB_HOST:
DATABASE_HOST = get_env_variable("DB_HOST", default="localhost") ##

Anything else?

Following that we have a largely working example of a Copilot deployable Django project. There are lots of other features in Copilot you can also leverage:

• We created an additional service (a separate manifest) for a “Scheduled Job” (https://aws.github.io/copilot-cli/docs/manifest/scheduled-job/) which spins up containers as required (from the same Dockerfile) to run out cron tasks.

• addon manifests (https://aws.github.io/copilot-cli/docs/developing/addons/workload/) are additional CloudFormation templates that can also be deployed, for example, we added a SES-IAM role to allow the containers to send emails via SES. In our project, we created the RDS and S3 for staticfiles (django-storages) separately but you could also add that via addons.

• Pipelines - You can also add pipelines (https://aws.github.io/copilot-cli/docs/manifest/pipeline/) to deploy your project. In our case we already had CI so we added the CD directly into bitbucket using the copilot cli directly.

• Hosted zones and SSL - we briefly covered this above, in our setup we used existing certificates and hosted zones and linked to them in our manifest, you can find out more on this at https://aws.github.io/copilot-cli/docs/developing/domain/

All in all our first experience with CoPilot has been very positive, the autoscaling and deployments work well. It also reduced the time needed to provision the architecture, giving us more time to deliver features. It’s a nice cost-effective approach for our clients who don’t require a more complicated Kubernetes setup and certainly better than provisioning ECS or EC2s manually.