TripleO

OpenStack Containerization with Podman – Part 5 (Image Build)

For this fifth episode, we’ll explain how we will build containers with Buildah. Don’t miss the first, secondthird and fourth episodes where we learnt how to deploy, operate, upgrade and monitor Podman containers.

In this post, we’ll see the work that we can replace Docker by Buildah to build our container images.

Context

In OpenStack TripleO, we have nearly 150 images (all layers included) for all the services that we can deploy. Of course you don’t need to build them all when deploying your OpenStack cloud, but in our production chain we build them all and push the images to a container registry, consumable by the community.

Historically, we have been using “kolla-build” and the process to leverage the TripleO images build is documented here.

Implementation

kolla-build only supports Docker CLI at this time and we recognized that changing its code to support something else sounded a painful plan, as Docker was hardcoded almost everywhere.

We decided to leverage kolla-build to generate the templates of the images, which is actually a tree of Dockerfile per container.

The dependencies format generated by Kolla is a JSON:

So what we do is that when running:

openstack overcloud container image build --use-buildah

We will call kolla-build with –list-dependencies that generates a directory per image, where we have a Dockerfile + other things needed during the builds.

Anyway, bottom line is: we still use Kolla to generate our templates but don’t want Docker to actually build the images.

In tripleo-common, we are implementing a build and push that will leverage “buildah bud” and “buildah push”.

buildah bud” is a good fit for us because it allows us to use the same logic and format as before with Docker (bud == build-using-dockerfile).

The main challenge for us is that our images aren’t small, and we have a lot of images to build, in our production chain. So we decided to parallelize the last layers of the images (which don’t have childs).

For example, 2 images at the same layer level will be built together, also a child won’t be built in parallel of its parent layer.

Here is a snippet of the code that will take the dependencies dictionary and build our containers:

Without the “fat daemon” that is Docker, using Buildah puts some challenges here where running multiple builds at the same time can be slow because of the locks to avoid race conditions and database corruptions. So we capped the number of workers to 8, to not make Buildah locking too hard on the system.

What about performances? This question is still under investigation. We are still testing our code and measuring how much time it takes to build our images with Buildah. One thing is sure, you don’t want to use vfs storage backend and use overlayfs. To do so, you’ll need to run at least Fedora 28 with 4.18 kernel, install fuse-overlayfs and Buildah should use this backend by default.

Demo

Please select full screen:

In the next episode, we’ll see how we are replacing the docker-registry by a simple web server. Stay tuned!

Software Engineeer at Red Hat, Private Pilot, French guy hiding somewhere in Canada.