Setting up a development environment for a smooth and seamless experience can be tricky. Docker makes this way easier, but still - different developers have different preferences. In this post we will dig into three underutilized features of Docker Compose that will allow you to give developers the freedom of choice to change the setup based on their needs and preferences.

Utilizing environment substitution

Your docker-compose.yml doesn’t have to be static! Docker Compose supports the substitution of environment variables in your Compose configurations.

I find this especially useful for publishing ports, build arguments, assigning environment variables, and images.

version: "3.7"

services:
  app:
    image: jfahrer/myapp:development
    build:
      context: .
      args:
        - PORT=${MYAPP_PORT:-3000}
    ports:
      - 127.0.0.1:${MYAPP_PORT:-3000}:${MYAPP_PORT:-3000}
    volumes:
      - ./:/usr/src/app:cached
    environment:
      - POSTGRES_HOST=db
      - APP_USER=${USER}
      - APP_ENV=${MYAPP_ENV?You need to set MYAPP_ENV in order to be able to start the service}
    command: ["node", "app.js", "--bind", "0.0.0.0", "--port", "${MYAPP_PORT:-3000}"]

  db:
    image: "postgres:${POSTGRES_VERSION:10.6-alpine}"

As you can tell, there are various places where we use ${} syntax. Docker Compose will perform a variable substitution in those places.

Our Dockerfile takes a build arg PORT that we dynamically set in this Compose configuration. We can pass in the build argument by defining the environment variable MYAPP_PORT. If the environment variable is empty or not defined, we fall back to using 3000. This is achieved using the :- syntax.

We also use MYAPP_PORT in other places: publishing our ports and starting the application.

The containers that we start for our app service definition will have the APP_USER environment variable set to the value of the environment variable USER on our Docker Host.

In addition, we force users to set the MYAPP_ENV environment variable. If it is not set, they will receive an error message:

ERROR: Missing mandatory value for "environment" option in service "app": You need to set MYAPP_ENV in order to be able to start the service

This is achieved using the ? syntax.

And last but not least, we allow users to dynamically change the version of Postgres that will be used as our DBMS. The default is set to 10.6-alpine.

Environment substitution allows us to specify extremely flexible Compose configurations that can be changed by setting environment variables. This is extremely powerful in combination with a .env file - more on that later.

By the way, you can read all about how I use environment substitution for publishing ports here.

Merging Compose files

I often find the need to have different Compose configurations for different scenarios:

  • Use a bind mount for the source code for local development but not when running my tests on CI
  • Switching between using a pre-built image for the frontend for this API application vs creating a bind mount for source code
  • Add additional services for integration testing
  • And many more

These complex changes can not be achieved using environment substitution. But we can merge multiple Compose configurations based on our needs!

We can specify multiple Compose configurations using the the -f flag for docker-compose:

docker-compose -f docker-compose.yml -f docker-compose.ci.yml up -d

We can also use the COMPOSE_FILE environment variable:

export COMPOSE_FILE=docker-compose.yml:docker-compose.ci.yml
docker-compose up -d

Also, if Docker Compose finds a docker-compose.override.yml next to the docker-compose.yml, it will automatically merge the two files (unless you are using the -f flag or the COMPOSE_FILE environment variable is defined`).

Let’s look at a quick example:

docker-compose.yml

version: "3.7"

services:
  app:
    image: jfahrer/myapp:development
    build:
      context: .
    environment:
      - POSTGRES_HOST=db
    command: ["node", "app.js"]

  db:
    image: "postgres:10.6-alpine"

Our base Compose config defines our services. We have our app service that defines the basic directives that we need for all environments.

docker-compose.override.yml

version: "3.7"

services:
  app:
    volumes:
      - ./:/usr/src/app:cached
    ports:
      - 127.0.0.1:3000:3000

The docker-compose.override.yml will automatically be picked up in development. It extends our app service and creates port mapping for us as well a bind mount for our source code.

docker-compose.ci.yml

services:
  app:
    image: jfahrer/myapp:${GIT_SHA}
    command: ["tail", "-f", "/dev/null"]

  db:
    image: "postgres:10.6-alpine"

On CI we would merge this Compose configuration with the main one by setting the COMPOSE_FILE environment variable to docker-compose.yml:docker-compose.ci.yml . In the config we set the tag for our image to the current git sha (assuming it is available via GIT_SHA and change the startup command of our app service. In this case we just want to run a command that doesn’t quit right away (tail -f /dev/null) so that we can use docker-compose exec to send the commands to run the test suite.

The .env file

Docker Compose will automatically read environment variables from a .env file if it is available. Simply specify key/value pairs in it:

APP_ENV=development

This is especially powerful when combined with environment substitution. It allows you to setup your Compose config in a flexible manner and developers can permanently set values in their .env file.

It is also very useful for setting the environment for a service:

version: "3.7"

services:
  app:
    image: jfahrer/myapp:development
    environment:
      - APP_ENV

The above configuration tells Docker to use the value of the environment variable APP_ENV on the Docker Host and make it available in containers for the app service. Since Docker Compose will make all environment variables from the .env available, we can set the value for APP_ENV in it.

And last but not least, we can set the COMPOSE_FILE environment variable that will tell Docker Compose which config files we want to use. This is great when you want to use multiple Compose files for different scenarios and set a default for your system:

COMPOSE_FILE=docker-compose.yml:docker-compose.dev.yml

Exclude the .env file from version control and allow developers to create their own file with their preferences. If you want to set defaults, define them right where you interpolate the environment variable in the Compose file using :- as described above. That being said, I generally appreciate a .env.example file that I can just copy to .env and change according to my needs.