Dos and Do Not Dos for creating containers
#docker #fediverse #bestpractices #homelab
I'd like to say I've stood up my fair share of projects, both personally and professionally over the years. From running open source tiny projects like WriteFreely here to standing up massive microservices in Kubernetes for large companies, I've done my fair share of Docker.
Over the last weekend I've been trying to stand up a Fediverse instance. Those of you who know me know that I'm fairly passionate about the fediverse, allowing us to democratize social media a bit by hosting our own version of Insta/FB/etc and allow our self hosted instances to talk to each other.
However what started as a very fun idea to stand up my own has instead turned into over a week of frustration and questioning, and it inspired this post today.
So today rather than talking about hosting, or Kubernetes, or Docker, today I want to write some essential dos and maybe more importantly do-not-dos for Docker images. None of these things are me saying “You did this wrong”, but more me trying to show the multitude of ways that people use Docker, and how to make sure people find your docker image easy to use, instead of finding it frustrating.
I won't point fingers or use any exact examples, but I will come up with examples of both the negative and positive cases of each.
Don't: Write shell scripts for building/running containers
To be clear I don't mean scripts inside the container, but install scripts to actually run your containers, scripts that actually run docker build or docker run.
I've seen this trend in a lot of open source projects, and it's a bit of a pet peeve because I know maintainers are trying to make images easy to use for their users. After all, we all know plenty of people will demand “Just give me the exe”. (If you don't know, I warn you there's plenty of profanity, but here's the link). However, there's a line that can be crossed, where making it so easy that containers will “just start” hides a lot of the crucial info that is required for actually running a container.
I've spent hours reading through install.sh scripts trying to figure out how maintainers actually build their docker containers, when really all I want to do is figure out what steps I need to build the container, or even how to simply run the container.
The reason this is difficult is simply this – what if I'm not running this container on this machine? What if your user is using Windows? Mac? FreeBSD? What if the machine I'm going to run the container on is in the cloud, across the room, or in my case, I run Kubernetes. I will not be using docker run or docker compose, so while you've made it marginally easier for people using those tools, many other people are left with an even harder to stand up version of your software.
Docker was built to be able to run your code on any machine already, with a standard set of rules and guidelines. Adding custom scripts into the mix means that the process has deviated from the standard Docker guidelines, and new users will need to learn your specific way of standing up your containers.
Do: Write clear documentation on building containers
It may seem backwards, but providing clear and concise documentation in this case of your environment variables, volumes, ports, and other docker-related items is much more helpful than a script.
You can run any container by knowing the environment variables, volumes, and ports – and providing that documentation really is all you should need to do to get your users up and running.
As for the “Give me the exe” folks, well, if you want to self host, learning docker is one of the best ways to get started.
Environment Variables
Below are the environment variables to run , their description, and what their expected values would be.
| Name | Example | Description |
|---|---|---|
| DB_HOST | 127.0.0.1 | The host (or IP address) to locate the database on. | DB_USERNAME | admin | The who will log into the database (located at DB_HOST) |
| DB_PASSWORD | (Your DB password) | The password for DB_USERNAME |
It's tempting to build a nice CLI that allows your users to type that in and it sets it up for you – but by doing that you're only helping a fraction of your users who may run this on their one computer. Help all of your users by instead providing clear documentation.
Don't: Use .env files in Docker
Following our environment variables, another worrying trend I've seen with small projects, the use of .env files in production.
For those who don't know, .env is a growing standard for setting up environment variables for development purposes. I use them myself in python and JS projects. It allows you to set your environment up in a simple file rather than trying to figure out how to set up your system's environment variables while debugging your code.
DB_HOST=127.0.0.1
DB_USER=admin
DB_PASSWORD=foobar!!
The format of a .env file
This is great for development on your machine. This is a maintainability and security nightmare in production. Let's break it down for a few reasons why you should never use .env files in production.
An additional file is an additional dependency
Adding a .env file is another file that needs to be maintained and backed up in the case of a server failing. Rebuilding the environment of a container should not be something that can be lost in case items are not backed up.
Secrets/Credentials
Security here is the most damning for .env files. I gave the database example above to highlight the point, in the case of .env files, your secrets are stored in plain text somewhere on a computer. Docker, kubernetes, openstack, everyone provides some sort of secret store where you can safely place your secrets and inject them into your running container. Personally, I really enjoy using 1password's secret management tools that handle this for me.
Forcing the use of a .env file in production completely negates all of this security work by forcing passwords to be stored in plaintext.
Conflicts with Docker principals
This one I have to throw in, containers aim to be as stateless as possible, by introducing state where there doesn't need to be any (especially when environment variables already exist), goes against what containerization is attempting to do.
All major languages/frameworks allow some way of setting configuration values from either a local file (.env, appsettings.json, etc) or from environment variables and merging them together. I urge developers to learn these settings, it takes a few minutes to learn and will save many headaches later.
Do: Use environment variables
Simple as that. Simply use environment variables to configure your application. Most .env libraries were built to overwrite the environment but will use the environment variables in the background. Let those libraries do their job. If it finds a .env file, they will use it for debugging. If they don't find one, they will use the environment set by the system.
Don't: Share/re-use volumes
I've only seen this on a few projects, but it's a critical one. Never share volumes between containers. It's just more hassle than it's worth, and usually is highlighting an underlying problem with your project.
I see this mostly in the case of something like an app server that serves HTTP, with workers running in other containers. That there is a great paradigm, it allows me to make a separate worker while keeping my HTTP server up. Say though, that you need to access something from the HTTP server. Well, the first approach might be to just share the same volume. HTTP server uploads it to /volume/foo.txt, and then the worker can also mount the same volume and read it, right?
Well, yes – if these containers are always running on the same machine. But – will they? The example I laid out actually kind of contradicts that thought, the point was that we would not want to overwhelm our API server, so wouldn't a natural thought be to move our worker to another machine? If we require a shared volume, that immediately becomes more complex.
Kubernetes offers a few types of volumes, RWO (Read-write Once) allows you to share a volume across running containers (pods), but the limitation is that many can read, only one container can write. Maybe the worker needs to write back to the API container? Well, they offer RWX (Read-write Many) which does work – however it usually requires quite a bit of setup, and how do they accomplish that? Most RWX providers are running an NFS server inside of the cluster, which means that local storage is no longer local, and writes are happening over a network. So to achieve this simple cross-container volume, we must set up quite a bit of overhead to achieve a solution that does not work as well as we wanted it to in the first place.
Do: Use a cache or database
A much cleaner approach, and one that was meant to handle this from the start, is to use a cache. Redis is extremely easy to set up and will solve 99% of the issues you may have here. In the case of text, json, or anything else you may need to pass through, sending it through Redis will probably save you.
For small files, like an avatar or thumbnail, Redis does allow binary files, if you need security most SQL and NoSQL databases allow binary data or attachments. I know many will scoff at the idea, but if the goal is simply to facilitate communication between different containers, this can be a fine solution, say to pass a file back to a worker, who will process it and eventually load it to S3, as an example.
Don't: Write files around the filesystem
A large source of code-smell is if files are written all around the file system. If your code needs some files over under /var and others under /data, maybe it needs to write some files under another location – it becomes very difficult to track all of them. Confusion though is just one problem.
Kubernetes and containerd actually don't let you modify files that aren't in a volume. The entire filesystem of a container running under containerd (k3s for example uses containerd as a default, which is good because it's more lightweight) is completely immutable. Attempting to write to a file that is not explicitly mounted as a writable volume will cause your application to throw “Read-Only filesystem” errors.
There are exceptions to this. /var/log for example is mapped to a temporary cache automatically so logs can be written there, as is standard. /tmp is the same, as anything written to tmp is assumped to be temporary and doesn't matter.
If you need to write to a random file though, say /var/foo/bar, containerd will throw an error, and that directory will need to be explicitly mapped by your users before they can write to it.
Do: Choose one workspace to work out of
There is nothing wrong with needing persistent storage, but try to keep it to the minimum number of volumes. Say you want everything to be under /data, then that allows users to create one and only one volume that will be persisted.
However conversely:
Don't: Store temporary data in persistent storage
The exact opposite of the above, if you have temporary data, store it in temporary storage. Most persistent storage options take on the approach that storage can always grow, but cannot shrink. Meaning if you store logs under /data/logs and /data is a persistent volume but logs can grow to say gigabytes in size, your users will end up paying gigabytes for storing those logs.
Do: Use standard locations for temporary storage
Put logs under /var/logs and temporary storage under /tmp and everything will work perfectly.
Don't: Log to a file
Or don't store logs at all! On the subject of logs, in a containerized world – don't worry about them.
If I have a container acting up, if I want to see the logs I vastly prefer using the docker logs or kubectl logs command over needing to exec into a container to tail a logfile.
Do: Print logs to the console
Kubernetes, Docker, all of the orchestration frameworks then will be able to consume your logs and rotate them accordingly. All logging frameworks will dump logs to the console, there is no need to write your containerized logs to a file at all.
My final one for today, the worst offender I have, this one is important folks.
NEVER: put source code in volumes.
This one is so bad that I'll call out a direct example – Nextcloud. Nextcloud is a PHP application, and their AIO container is one of the most frustrating I've worked with – mostly because their source code is actually stored in the persistent volume. There's many, many reasons for why this is bad practice and goes directly against the very ideas of containerization, but I'll call out a couple.
Containers should never need to be “updated”
Containers should be all encompassing for their environment. There should never be a case when a container starts that some packages may be out of date, or some code may be the wrong version – that's the point of containerization. When a container starts, that is everything it needs to be able to run.
No version control
Since persistent volumes are outside of a git repo, there is no way to update the versions without writing your own update functionality. There are simply too many ways that code in the volume can become inconsistent with what is in the container.
Do: Make use of temporary data and configuration
Code should remain under it's own directory, with persistent data under another completely separate directory. If you're building your own container, I recommend /app for where you put your workspace, and then mounting /data for any persistent storage needs. To go above and beyond, make an environment variable for MOUNT_POINT that maps to /data by default, but let's your users mount anywhere on the file system, that gives them the most flexibility. Then you don't need to worry at all about where in the system your mount points are.
If there are dynamic packages or plugins, there should be a way to define these in environment variables and they should be ran ephemerally. On container start download the plugin to /tmp and use code from there. On container restart, the same thing will happen. This also ensures plugins stay up to date, as plugins will always download the latest version.
An example configuration of a RAID configuration in BIOS
