I recently picked up a ‘renewed’ Dell Poweredge R720 because a colleague had gotten one and been telling me about his setup. He was running ESXi and was running multiple services through Traefik and since I wanted to set up a couple of services for myself (Wiki.js, GitLab, and Jenkins) decided to copy his setup. Over a couple of days, we went back and forth with him helping me get the initial setup running and working through issues, and me sharing some additional configurations to make the overall process easier to setup. In the end, we came up with a pretty robust and secure configuration.

In the end of this, you will have robust and easy method to deploy and manage new containers, which will automatically be configured with SSL certificate provided by LetsEncrypt and be accessible via your specified domain.

This specific implementation is running on a private server on a home network, but can be easily applied to a DigitalOcean droplet.

Requirements

1. Ubuntu Server 20.04
2. pfSense 2.7.0 Appliance routing your network (I currently use this from Amazon)
3. Docker
4. Own the domain that you plan on using
5. Domain’s DNS servers are set up to use DigitalOcean
6. A DigitalOcean API Key

What is Traefik

Traefik is a leading modern reverse proxy and load balancer that makes deploying microservices easy. Traefik integrates with your existing infrastructure components and configures itself automatically and dynamically.

Traefik is designed to be as simple as possible to operate, but capable of handling large, highly-complex deployments across a wide range of environments and protocols in public, private, and hybrid clouds. It also comes with a powerful set of middlewares that enhance its capabilities to include load balancing, API gateway, orchestrator ingress, as well as east-west service communication and more. [1]

It also provides a useful dashboard for monitoring and debugging routes and services registered with the traefik service.

What is Portainer

Portainer is an open source tool for managing containerized applications. It works with Kubernetes, Docker, Docker Swarm, Azure ACI in both data centres and at the edge. Portainer removes the complexity associated with orchestrators so anyone can manage containers. It can be used to deploy and manage applications, observe the behavior of containers and provide the security and governance necessary to deploy containers widely. [2]

Getting Started

Install Docker

First, update your existing list of packages:

sudo apt update

Next, we need to install a few prerequisite packages which let apt use packages over HTTPS:

sudo apt install apt-transport-https ca-certificates curl software-properties-common

Then add the GPG key for the official Docker repository:

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Add the Docker repository to APT sources:

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu focal stable"

Update the package database with the Docker packages from the repository we just added:

sudo apt update

Make sure you are about to install from the Docker repository otherwise we will get it from the default Ubuntu repository:

apt-cache policy docker-ce

We should get something similar to the output below:

Output of apt-cache policy docker-ce
docker-ce:
Installed: (none)
Candidate: 5:19.03.9~3-0~ubuntu-focal
Version table:
5:19.03.9~3-0~ubuntu-focal 500
500 https://download.docker.com/linux/ubuntu focal/stable amd64 Packages

Finally, install Docker:

sudo apt install docker-ce

With everything completed, we should have Docker installed and can check the status:

sudo systemctl status docker

Which should output something similar:

Output
● docker.service - Docker Application Container Engine
Loaded: loaded (/lib/systemd/system/docker.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2020-05-19 17:00:41 UTC; 17s ago
TriggeredBy: ● docker.socket
Docs: https://docs.docker.com
Main PID: 24321 (dockerd)
Tasks: 8
Memory: 46.4M
CGroup: /system.slice/docker.service
└─24321 /usr/bin/dockerd -H fd:// --containerd=/run/containerd/containerd.sock

Creating the Docker Compose file

Lets start by setting up our skeleton docker-compose.yml file.

version: '3.3'
services:
  traefik:
    image: traefik:v2.4
    container_name: traefik
    restart: unless-stopped
    networks:

    ports:

    environment:

    volumes:

    command:

    labels:

networks:

This sets up the image and the container name for Traefik as well as its restart policy.

Network

We will need to define a network that allows Traefik to proxy to all of our other containers. Because these containers will not be exposed on their own, every container that Traefik will route to needs to use this network.

    networks:
      - traefik-network

Ports

Here we define what ports Traefik will listen on. Even though we will only be using https, we still listen on port 80 to respond to http requests and forward them to https.

    ports:
      - "80:80"
      - "443:443"

Environments

There are a couple of ways to set up the API token for DigitalOcean. This way is the most secure especially when using Portainer as you would be able to see the API Key in plan text when looking at the container settings. Once obtaining a API Key from DigitalOcean, you would place it in a file named ` digitalocean_api_key` in the same directory as your docker-compose file.

    secrets:
      - digitalocean_api_key
    environment:
      - DO_AUTH_TOKEN_FILE=/run/secrets/digitalocean_api_key
    ...
    secrets:
      digitalocean_api_key:
        file: ./digitalocean_api_key

If you create the digitalocean_api_key file and then $ sudo chmod 600 you will additionally secure the api key from any other users on the computer.

The simpler but less secure way is to just add a DO_AUTH_TOKEN key and value.

    environment:
          - "DO_AUTH_TOKEN=<key>"

If you do use this method, be aware that anyone with access to the Portainer dashboard will be able to see this key in the containers’ configuration window.

Volumes

Not much to mention here, each of these volumes is bound to the Traefik container to facilitate automatic discovery of new containers, store Traefik specific data and LetsEncrypt certificates.

    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./letsencrypt:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik-data/traefik.yml:/traefik.yml:ro
      - ./traefik-data/acme.json:/acme.json\

Command

These lines set up a couple of things, the api.dashboard command will enabled the dashboard so we can see the routes t hat have been configured.

The provider options allow us to define additional Traefik routing for non-docker container services. This give you the ability to have traefik secure and route to an existing service on your network.

Traefik Providers settings

    command:
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.file.directory=/dynamic"
      - "--providers.file.watch=true"
      ...

The entrypoints commands define the different protocols that we want the services to use. In this case, web and websecure will be used when defining other containers to specify the ports to route too.

Setup Entry Points

      command:
      ...
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      ...

LetsEncrypt Options

Finally we want to setup LetsEncrypt.

1. This defines DigitalOcean as the provider we want LetsEncrypt to use to verify domain ownership.

2. delaybeforecheck does exactly what it says, and will provide a delay between the resolver making the update to your DNS records and LetsEncrypt verifying that the records have been added.

3. Email that you wish too received renewal notices from LetsEncrypt

4. Where we will store the LetsEncrypt data

5. What DNS servers LetsEncrypt should use when verifying DNS verification. Without this, you may run into issues where it is using your local DNS cache and not find the records correctly.

      command:
      ...
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=digitalocean"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.delaybeforecheck=0"
      - "--certificatesresolvers.letsencrypt.acme.email=admin@example.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.resolvers=1.1.1.1:53,8.8.8.8:53"

All together we should end up with something like this,

    command:
      - "--api.dashboard=true"
      - "--providers.docker=true"
      - "--providers.file.directory=/dynamic"
      - "--providers.file.watch=true"
      - "--providers.docker.exposedbydefault=false"
      - "--serversTransport.insecureSkipVerify=true"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=digitalocean"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.delaybeforecheck=0"
      - "--certificatesresolvers.letsencrypt.acme.email=admin@example.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.resolvers=1.1.1.1:53,8.8.8.8:53"

Labels

This line sets up the user credentials to log into the Traefik dashboard. There isn’t much you can do in the interface but it is a good place to start if you are having issues resolving a host.

    labels:
      ...
      - "traefik.http.middlewares.api-auth.basicauth.users=user:pass"

To get the password hash use the following. You will need to install apache2-utils to be able to use this command.

    $ sudo apt install apache2-utils
    $ htpasswd -n username

The other options are pretty straight forward and the only other thing to change is the Host to be whatever URL you want to be able to access the Traefik dashboard (ie. traefik.example.com)

All together we should end up with something like this,

    labels:
      - "traefik.enable=true"
      - "traefik.http.middlewares.api-auth.basicauth.users=user:pass"
      - "traefik.http.routers.traefik-router.entrypoints=websecure"
      - "traefik.http.routers.traefik-router.rule=Host(`traefik.domain.tld`)"
      - "traefik.http.routers.traefik-router.tls=true"
      - "traefik.http.routers.traefik-router.tls.certresolver=letsencrypt"
      - "traefik.http.routers.traefik-router.service=api@internal"
      - "traefik.http.routers.traefik-router.middlewares=secured"
      - "traefik.http.middlewares.traefik-redirect-web-secure.redirectscheme.scheme=https"
      - "traefik.http.middlewares.secured.chain.middlewares=traefik-redirect-web-secure,api-auth"

Network Setup

Both my colleague and I run a pfSense appliance for our network and it provides a very convenient way to route all domain traffic to a specific IP without having to set up a separate DNS server.

In your pfSense dashboard you will want to go to Services -> DNS Resolver and near the bottom, you will find an area labeled Custom options. If you don’t see that, look for the Display Custom Options button to toggle them on.

In the Custom options text field, you will want to enter the following

server:
local-zone: "example.com" redirect
local-data: "example.com 3600 IN A <ip addr of docker server>"

Dynamic Traefik Routes (optional)

Another great feature of Traefik is dynamic routes. When you run docker-compose, Traefik will create a dyanmic folder setup by these lines in our config.

      - "--providers.file.directory=/dynamic"
      - "--providers.file.watch=true"

this tells Traefik to look in the dynamic folder for any route files, and load them on the fly. By placing a service-name.yaml file in this directory, you can specify an external service that is running on the network to route through Traefik and still get a valid certificate.

http:
  routers:
    to-demo-service:                      # this is the label for the router
      rule: "Host(`service.domain.tld`)"
      service: demo-service               # this needs to match the name defined under `services`
      entryPoints:
        - https
      tls:
        certResolver: letsencrypt
  services:
    demo-service:                         # this needs to match the name defined under `routers`
      loadBalancer:
        servers:
          - url: "http://[ip address]"

Once you’ve added the file, you will be able to go into your Traefik dashboard and see the new route created under Routes

Portainer Setup (optional)

Here is the configuration for setting up Portainer. This is also a template for any other services that you want to set up through Traefik.

  portainer:
    image: portainer/portainer:latest
    container_name: portainer
    restart: unless-stopped
    networks:
      - local-network
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./portainer-data:/data
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.portainer.rule=Host(`portainer.domain.tld`)"
      - "traefik.http.routers.portainer.service=portainer"
      - "traefik.http.routers.portainer.entrypoints=websecure"
      - "traefik.http.routers.portainer.tls.certresolver=letsencrypt"
      - "traefik.http.services.portainer.loadbalancer.server.port=80"

When configuring the labels for the service be sure to replace the labels and values with the correct entries.

This template should be sufficient enough to copy & paste and replace SERVICE_NAME and HOST.DOMAIN.TLD to your specific implementation and get it running.

    - "traefik.http.routers.<SERVICE_NAME>.rule=Host(`HOST.DOMAIN.TLD`)"
    - "traefik.http.routers.<SERVICE_NAME>.service=<SERVICE_NAME>"
    - "traefik.http.routers.<SERVICE_NAME>.entrypoints=websecure"
    - "traefik.http.routers.<SERVICE_NAME>.tls.certresolver=letsencrypt"
    - "traefik.http.services.<SERVICE_NAME>.loadbalancer.server.port=80"

Upgrading (optional)

When new versions of Traefik or Portainer have been released you may want to upgrade to the latest version. This can be accomplished by using the following steps.

1) Pull the latest images

docker-compose pull

2) Restart containers

docker-compose up -d --remove-orphans

3) Remove obsolete images (optional)

docker image prune

Conclusion

Once that your docker-compose file has been finalized and you have your DigitalOcean API key, you should be able to simply run sudo docker-compose up -d and after a few moments be able to go to both traefik.domain.tld and portainer.domain.tld and see both sites are secured.

TL;DR

Here is the full docker-compose for your copy and paste pleasure, be sure to either create the digitalocean_api_key file or switch it out with the less secure DO_AUTH_TOKEN environment option.

version: '3.3'
services:
  traefik:
    image: traefik:v2.4
    container_name: traefik
    restart: unless-stopped
    networks:
      - local-network
    ports:
      - "80:80"
      - "443:443"
    secrets:
      - digitalocean_api_key
    environment:
      - DO_AUTH_TOKEN_FILE=/run/secrets/digitalocean_api_key
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - ./letsencrypt:/letsencrypt
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./traefik-data/traefik.yml:/traefik.yml:ro
      - ./traefik-data/acme.json:/acme.json
    command:
      - "--providers.docker=true"
      - "--providers.file.directory=/dynamic"
      - "--providers.file.watch=true"
      - "--api.dashboard=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
      - "--entrypoints.web.http.redirections.entryPoint.to=websecure"
      - "--entrypoints.web.http.redirections.entryPoint.scheme=https"
      - "--entrypoints.websecure.address=:443"
      - "--serversTransport.insecureSkipVerify=true"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.provider=digitalocean"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.delaybeforecheck=0"
      - "--certificatesresolvers.letsencrypt.acme.email=admin@example.com"
      - "--certificatesresolvers.letsencrypt.acme.storage=/letsencrypt/acme.json"
      - "--certificatesresolvers.letsencrypt.acme.dnschallenge.resolvers=1.1.1.1:53,8.8.8.8:53"
    labels:
      - "traefik.enable=true"
      - "traefik.http.middlewares.api-auth.basicauth.users=user:pass"
      - "traefik.http.routers.traefik-router.entrypoints=websecure"
      - "traefik.http.routers.traefik-router.rule=Host(`traefik.domain.tld`)"
      - "traefik.http.routers.traefik-router.tls=true"
      - "traefik.http.routers.traefik-router.tls.certresolver=myresolver"
      - "traefik.http.routers.traefik-router.service=api@internal"
      - "traefik.http.routers.traefik-router.middlewares=secured"
      - "traefik.http.middlewares.traefik-redirect-web-secure.redirectscheme.scheme=https"
      - "traefik.http.middlewares.secured.chain.middlewares=traefik-redirect-web-secure,api-auth"

  portainer:
    image: portainer/portainer:latest
    container_name: portainer
    restart: unless-stopped
    networks:
      - local-network
    volumes:
      - /etc/localtime:/etc/localtime:ro
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./portainer-data:/data
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.portainer.rule=Host(`portainer.domain.tld`)"
      - "traefik.http.routers.portainer.service=portainer"
      - "traefik.http.routers.portainer.entrypoints=websecure"
      - "traefik.http.routers.portainer.tls.certresolver=letsencrypt"
      - "traefik.http.services.portainer.loadbalancer.server.port=80"

networks:
  local-network:
    external: true

secrets:
  digitalocean_api_key:
    file: ./digitalocean_api_key

References

[1] “Traefik, The Cloud Native Application Proxy | Traefik Labs.” Traefik Labs: Makes Networking Boring, 2021, traefik.io/traefik.

[2] “Portainer | Open Source Container Management GUI for Kubernetes, Docker, Swarm.” Portaine.Io, 2021, www.portainer.io.

[3] Traefik. (n.d.). [Architecture]. Retrieved August 17, 2021, from https://doc.traefik.io/traefik/