An overview of single-purpose Linux distributions

Many people, when they are installing a Linux distribution for a single purpose such as running containers, would prefer an install-and-forget type of deployment. At FOSDEM 2023 in Brussels, several projects of this minimal Linux distribution type were presented. Fedora CoreOS, Ubuntu Core, openSUSE MicroOS, and Bottlerocket OS all tackle this problem in their own way. The talks at FOSDEM gave an interesting overview of how these projects differ in their approaches.

Fedora CoreOS

Akashdeep Dhar and Sumantro Mukherjee, who are both members of the Fedora Council and work at Red Hat as software engineers, explained how they use Fedora CoreOS as the base operating system to run multiplayer game servers in containers. As described in its documentation, Fedora CoreOS is "an automatically updating, minimal, monolithic, container-focused operating system".

Fedora CoreOS (sometimes abbreviated FCOS) provides the host operating system for these containers; it only includes those packages that are needed for a minimal networking-enabled and container-ready setup. At the time of this writing, the latest stable release had 415 packages. It supports the x86_64, aarch64 (including the Raspberry Pi 4), and s390x architectures; it runs on bare metal, virtualized, or on various cloud platforms.

A Fedora CoreOS machine is provisioned using Ignition, which is a tool that partitions disks, formats partitions, enables systemd units, and configures users. Ignition only runs once during the first boot of the system, from the initramfs. An Ignition configuration file is formatted as JSON, but for end users Fedora CoreOS recommends using a Butane configuration, which is a YAML file that Butane translates into an Ignition configuration. The "System Configuration" section in Fedora CoreOS's documentation shows some examples of how to configure storage, network, containers, users and groups, time zones, and more in a Butane configuration. In their talk, Dhar and Mukherjee showed a Butane configuration to set up a Minecraft server in a container, and they also published it in their GitHub repository.

When installing Fedora CoreOS, you choose one of three update streams. "Next" is for experimenting with new features, "testing" represents what is coming in the next stable stream, and "stable" is the stream with changes that have spent a time in the testing stream. Most end users should choose the stable stream. You refer to the Ignition file with your customizations, in a manner that depends on your installation type. For instance, when installing from PXE you append the coreos.inst.ignition_url=URL option to the kernel, referring it to the location of the Ignition file on a web server.

After installation, the system is updated automatically when a new release is rolled out on the chosen stream. The Zincati agent checks for operating-system updates and applies them using rpm-ostree. Zincati can be configured as well; for example, one can configure how "wary" it is to update (that is, how early in the phased rollout cycle it receives updates) and how eager it is to reboot after applying an update (immediately or only within configured maintenance windows). If an update causes problems, the user is always able to manually roll back to the previous system state with:

    $ sudo rpm-ostree rollback -r

There are no dnf or yum commands in Fedora CoreOS. Extending the package set is done with rpm-ostree, which layers the packages on top of the current operating-system image. But, since Fedora CoreOS is a container-focused system, extra services would generally be installed as containers.

Ubuntu Core

Canonical's Valentin David talked about Ubuntu Core. According to the project's home page, it's "a secure, application-centric IoT OS for embedded devices". Ubuntu Core targets high-end embedded devices such as industrial computers for IoT gateways, signage, robotics, and automotive applications; at home it could be useful on a Raspberry Pi to run services such as Nextcloud or home-automation software. The distribution's software is based on Ubuntu's main operating-system builds, but without using deb packages or the dpkg and apt commands. Instead, it only uses snaps to install software. In essence, a snap package is a squashfs image with some metadata about how to install and run the software.

Snaps are isolated from other snaps and the underlying operating system. If a snap is run in strict confinement, it runs in a sandbox, making use of AppArmor, seccomp, and control groups. By default, snaps don't have access to resources outside of the sandbox, but they can get access to specific resources using interfaces.

David explained that there are five types of snaps in Ubuntu Core. The "gadget" snap contains device-specific or architecture-specific components such as the boot loader, device tree, board-specific packages, and configurations. The "kernel" snap comes with the Linux kernel, modules, firmware, and systemd stubs. The "base" snap contains the root file system for the Ubuntu Core operating system. The "snapd" snap has snapd, the daemon that installs and updates all snaps. And last but not least, each application is packaged in an application snap. This includes the root file system of a base snap; it can also make services and commands available to the underlying operating system.

The gadget snap also describes the disk layout. Ubuntu Core typically has four partitions. On UEFI systems, the "seed" partition is the EFI System Partition (ESP), containing the configuration for the first-stage boot loader and at least one recovery system. The "boot" partition contains the second-stage boot loader, a kernel, and initramfs that decrypts the "save" and "data" partitions. The latter two are LUKS2 encrypted. The save partition contains a backup of the device identity and other data to facilitate recovery, while the data partition stores the user and system data.

Most of the Ubuntu Core operating system is read-only. For instance, /etc and /var are read-only by default. However, specific paths are bind-mounted from the data partition, such as /etc/systemd, which allows the system to change any systemd unit files for the services and mount the snap's squashfs image. Transactional updates are handled by snapd: if an update of a snap fails, the system automatically rolls back to the previous version of the snap.

openSUSE MicroOS

Ignaz Forster, research engineer at SUSE, described the design of openSUSE MicroOS. It's a rolling-release distribution of openSUSE Tumbleweed, developed to run as a single-purpose system. A typical target would be for hosting containers, but it can even be used to create a minimal desktop. As with Fedora CoreOS and Ubuntu Core, openSUSE MicroOS automatically updates itself and has a minimal package selection. In openSUSE MicroOS's case, these are just RPM packages from openSUSE's repositories. There's also an enterprise version, SUSE Linux Enterprise Micro, and a community version based on the latter, Leap Micro.

OpenSUSE MicroOS has a read-only root file system, using Btrfs. Transactional updates are handled by a SUSE-specific wrapper script around the package manager zypper, transactional-update. This creates a new Btrfs snapshot of the root file system and then performs an update of the system. If the installation was successful, the script marks the new snapshot as the default snapshot. On errors, the snapshot is discarded and the previous one remains as the default. A reboot activates the new snapshot. Forster announced that, since all of the read-only parts of openSUSE MicroOS have now been moved to /usr, the upcoming 4.2.0 release of transactional-update would also be able to apply new snapshots without rebooting. MicroOS runs a health-checker systemd service that checks whether the system boots as expected after an update. This starts an automatic rollback to the previous default snapshot of the root file system if the system isn't healthy.

The original transactional-update script has been evolving into a generic library for atomic system updates, libtukit. The current implementation only supports Btrfs with openSUSE's snapshot utility Snapper, as used in openSUSE MicroOS. But according to Forster, the API is developed to support other backends.

In contrast to Ubuntu Core, all of /var and /etc are writable, while /usr is read-only. For instance, the default system configuration is put in /usr. Only changes made by the administrative user are in /etc. OpenSUSE's libeconf merges the configuration files placed in several locations. Most of the default MicroOS packages have been changed to work with this. Only /etc/fstab does not follow this convention yet. Forster concluded that openSUSE MicroOS takes a pragmatic approach to use existing infrastructure and packages, and that it's "a functional read-only OS in an imperfect world".

Bottlerocket

While the previous three operating systems originated from a general-purpose mother distribution, Bottlerocket is created by Amazon, tailored to host containers in its Amazon Web Services (AWS) cloud. In his talk, AWS software development engineer Sean McGinnis was quick to emphasize that the operating system is "backed by AWS, but not AWS-only". As an example, the project's GitHub repository has instructions to run it on bare-metal servers.

Bottlerocket was announced in March 2020 and made generally available in August 2020. To keep its footprint as small as possible, Amazon publishes variants for particular use cases. For instance, there's a aws-k8s variant with containerd and kubelet to run as a Kubernetes node on AWS, a vmware-k8s variant to do the same on VMware with Amazon Elastic Kubernetes Service (EKS), and a metal-k8s variant that supports Amazon EKS running on bare metal.

Bottlerocket runs two completely separate container runtimes. One is running host containers for operational tasks. The other one is used for running containers with an orchestrator, such as Kubernetes pods. Both runtimes have different security profiles.

Each container, be it a host container or a for-orchestrator container, is running an API client that talks over a Unix socket to an API server running on Bottlerocket. When Bottlerocket boots, its boot configuration (including user data) is loaded into the API server. User interaction is also typically done through this API to make real-time changes to the system configuration.

One of the host containers is the "control" container, which is launched on boot. This container is used to configure the Bottlerocket host. Another host container is the "admin" container. This isn't launched by default: it should only be launched in exceptional circumstances to troubleshoot the host operating system. It has additional privileges and can use the root process namespace to access the other containers for troubleshooting purposes. The admin container runs an SSH server that is reachable through the host's primary network interface. A final type of host container is the bootstrap container: this bootstraps the host before services like Kubernetes or Docker start. It has additional permissions, for instance to provide access to the underlying host file system.

Security is one of the focal points of Bottlerocket. The root file system is read-only, and /etc is backed by a tmpfs file system that is regenerated on boot. For container images and volumes, a separate user partition is mounted. Moreover, there's no package manager, no shell, and no Python interpreter. "If an attacker is able to escape a container, there are not many tools to work with", McGinnis said.

To check the integrity of the block devices, Bottlerocket uses dm-verity. The kernel boots in lockdown mode, which prevents the root user from modifying the kernel. McGinnis explained that this increases assurance that the running kernel corresponds to the booted kernel. Another security feature he emphasized is that Bottlerocket runs with SELinux in enforcing mode.

For updates, Bottlerocket uses an image-based model. The kernel, system packages, and container runtime packages are all stored inside an operating-system image. The first block device of the host has an active and inactive partition. An upgraded image is downloaded to the inactive partition, and upon reboot the host boots into this partition, which is then made active. The previous Bottlerocket image is still stored in the then inactive partition, and can be rolled back if required.

Conclusion

When looking at the different approaches of these single-purpose Linux distributions, it's clear that there's no one best way. Which one you choose depends on how they align to your goals and what tools you're comfortable with. Are you heavily invested in the API-first AWS or Kubernetes world? Then Bottlerocket seems to be the best fit. Do you prefer snaps to run your services? Then Ubuntu Core is a no-brainer. If you want to run containers on a host system without too much maintenance, then Fedora CoreOS or openSUSE MicroOS are for you. Whether they are using rpm-ostree or Btrfs snapshots under the hood is probably less important when all of the workloads are running in containers anyway.