Choosing a secure Docker base image

When you are looking for a secure Docker base image, what are you looking for excactly?

At Flownative, we run a lot of tools and applications in Docker containers – most prominently Flownative Beach. Early on, we decided to not rely on off-the-shelf images for PHP, Redis and all the other components we need, because we wanted to know what we are running and not just hope that image maintainers will keep an eye on security issues.

When you develop a lot of images yourself, you need a good base image to rely on. The base image should be small, capable and secure. And while we had a base image, which worked nicely in the past, it's now time for a new one.

Why a new Base Image?

During a partial security review of our platform, a consultant suggested that we revisit this topic and look into ways to harden our Docker setup. Specifically, Alpine was proposed as a replacement to our slimmed down Ubuntu base image.

This suggestion sounded logical to me – after all, Alpine Linux is known for its minimalist approach and security. So I prepared to redo all our Beach images based on Alpine.

However, while I was at it, I faced some problems with incompatibility to C-extensions we were using and packages which were not available in the Alpine package repository. So I took a step back and thought: What makes Debian, for instance, such a bad choice for a base image? Do I really know?

Let's take a look at some security best practices for base images and, along the lines, compare how Debian and Alpine fit into the puzzle.

Alpine and Debian – Some Background

Alpine Linux has its roots in embedded Linux. It's actually a fork of the LEAF project (Linux Embedded Appliance Framework),  which in turn started as a fork of the so-called "Linux-on-a-floppy" distribution (Linux Router Project), an operating system which fit on a 1,44 MB disk. I guess that it originally became popular in the Docker community mostly due to its small size – after all you don't want to download Gigabytes of data for a simple PHP image.

Debian GNU/Linux is one of the oldest operating systems based on the Linux kernel and is around since the early 1990s. Debian is a feature-rich distribution, due to the tens of thousands of packages it can draw from, and supports a lot of different architectures and needs. Debian – and its derived distribution Ubuntu – are certainly the most used operating systems running Internet servers today [ref].

With this history lesson out of the way, let's continue with best practices for a secure Docker base image:

Minimize Attack Surface

What's not included can't break. That's one of the most important rules for a base image: Make sure that only that software is included which is actually needed. And, as a consequence of this: Make sure that you really know which software is included and how it is working.

Besides including only the minimum amount of packages, you can also restrict access to certain commands to the user actually running the processes.

Least Privilege User

There is no good reason to run processes as root inside a container. That's why in Beach containers we are running processes as a user with id 1000.

It's also important to pay close attention to how you run your containers. For example, in Kubernetes, you should avoid, by all means, running a Pod in privileged mode. And if you have full control over your runtime and images, you can restrict certain actions through Seccomp security profiles.

For example, consider the following file which we'll call profile.json:

{
  "defaultAction": "SCMP_ACT_ALLOW", 
  "syscalls": [
    {
    "name": "mkdir",
    "action": "SCMP_ACT_ERRNO"
    }
  ]
}

… running a container, using the above profile, as follows would deny the container's processes to run mkdir:

docker run --rm -it \
  --security-opt seccomp=profile.json \
  hello-world

Finding and Fixing Vulnerabilities

Security isn't static (or as we say, "security is like trying to nail pudding on the wall"). And realistically you can only keep your images safe if you start using vulnerability scanners and automate the process.

There are various security scanners available for Docker containers, both commercial and free / Open Source variants. The problem with many of these scanners is that they produce a lot of output you need to react on. And even worse: they often produce many false positives.

You also need to know that each team maintaining a Linux distribution may handle security issues differently. Since Debian's policy is to handle all security issues very openly, the list may be longer than in other projects.

Frequent Security Updates

The operating system of your choice should have a well-established process for dealing with vulnerabilities and publish security updates on a regular basis. But even the quickest release of security fixes doesn't help you, if it's not included in your image.

That's a key take away from a couple of projects I had a chance to look into: Security updates for Docker images were never automated and rarely planned through a fixed schedule. In order to get this right, you need to automatically rebuild your base image and then all other images derived from that. Through such a rebuild, you'll usually get all current security updates automatically.

The next challenge then is to roll out these images – preferably automated as well – so that all of your applications are ultimately running on the updated base images.

More Aspects to Consider

There are some soft factors, which I also consider security-relevant.

First, if the software you are using is poorly documented, does not have a big-enough community which could provide you with answers, it's likely that you don't know enough about the system to identify possible security weaknesses.

Second, and somewhat related: You need to know your way around and feel comfortable using the system. If you have years of experience as a Ubuntu user, you may have your problems with Slackware. Seriously though, if you need to choose between two options, and are undecided, your experience with a system can turn the balance.

Third, make sure that the base image provides the functionality for the majority of your use cases. It doesn't help if the base image is super secure, but you need to work around a lot of incompatibilities and by that can cause – possibly security-relevant – errors.

The Pros and Cons

Of course this list isn't exhaustive. It's rather a line-up of points which I considered important during my search.

Alpine

Pro:

• very small images: the community pays a lot attention on minimizing image sizes
• minimum functionality: only absolutely necessary packages contained
• lightweight init system: like Gentoo, Alpine uses OpenRC, a lightweight alternative to systemd
• musl performance: for some cases, musl libc can be more performant than glibc

Con:

• rather poor documentation
• small team: currently there are 3 developers listed as the Alpine Linux team
• possible incompatibilities: musl libc may cause problems with some C-based plugins and adjustments may be necessary if you compile software yourself

Debian

Pro:

• small images: the size of slimmed down Debian images (such as minideb by Bitnami) is almost on par with Alpine (e.g. minideb + Python is just 7 MB larger than Alpine + Python)
• lots of packages: there's hardly any software for Linux which hasn't been packaged for Debian
• well tested: due to its popularity, Debian is used widely and issues are more likely to be found
• comprehensive documentation; also, the community produced a big amount of additional documentations and tutorials
• more security reviews: again, due to its larger community, Debian gets more attention and its more likely that vulnerabilities are discovered, e.g in glibc versus in musl libc (assumption). Debian also has a security audit team, which proactively looks for security issues.
• provenance: validating authenticity of packages is possible, e.g. with debsigs / dpkgsig

Con:

• slightly larger attack surface: minideb consists of about 35 packages (such as bash, grep, hostname, mount …) due to apt depending on it
• more false positives: scanners may report more false positives you need to look at

Conclusion

I think that there's no one-fits-all-purposes base image – you always need to consider the specific use cases as well. As for our scenario, we came to the following conclusion:

Alpine is a valid choice for a base image, even though there may be mixed opinions across experts, if Alpine is a good choice security- and stability-wise. There may be some compatibility problems or bugs related to networking, but in general it's working very well. For some software, no ready-made packages exist, so you'll end up compiling and packaging your own.

What I learned though is, that Debian is (in the meantime) a pretty good choice for a base image, too! The footprint is comparable to that of Alpine. And a stripped down base image contains only a bare minimum of packages which are well maintained and each exist for many years.

The right tooling (scanners, build automation, automatic roll-out) and proper configuration (least privilege, security profiles, dropped container capabilities, network policies) turn out to be as important as a solid base image.

Both – Alpine and a slimmed down Debian – are good choices. However, in the end we'll go for Debian, because we have more trust in the ecosystem, expect a better fit for the software we are going to run on it and can build on the many years of experience we have as Debian / GNU Linux users.