Building Flashable Images For Your Fedora IoT Remix

It’s nice to be able to pick an RPM-OSTree based system like Fedora IoT and build a remix based on it with tools like RPM-OSTree-Engine. But you know what? Maybe it’s not enough to just manage your Remix’ OSTree, host it somewhere publicly and to automate the whole build process. Maybe you want something more, the next step in OS distribution and management? Well I don’t know about you but hell did I want that feature!

With support and funds from my company I hired someone investigating and implementing this into RPM-OSTree-Engine. After some polishing and refactoring with 0.3.3 we can now start building flashable raw images of Fedora 33 IoT for your Raspberry Pi and Compute Modules.

Let me walk you through how that’s done manually with the RPM-OSTree-Engine container before showing you how it’s automated via Gitlab CI. At the end I’ll talk a bit on the implementation details, in case you wonder and consider implementing something similar looking for an inspiration.

The Art Of Creating OS Images

We’ll use so called loopback devices to mount a file as partitioned virtual SD Card where we then install the OSTree system. In order to do this the loop kernel module has to be available, which most modern systems should provide. Another dependency is SELinux if we intend to build Fedora IoT. Overall it’s recommended to use fairly up to date systems to avoid problems with SELinux policies.

Assuming you have a RPM-OSTree repository laying around from previous play through now all you need to do is running the rpm-ostree-engine-image command:

sudo podman run --privileged --rm -it -v $(pwd):/mirror -w /mirror quay.io/os-forge/rpm-ostree-engine:latest rpm-ostree-engine-image --ref=OSTreeBeard/develop/x86_64 --mirror=/mirror/.deploy-repo

If you want to build an image from a remote RPM-OSTree repository you can additional specify the remote address using the --origin= option:

sudo podman run --privileged --rm -it -v $(pwd):/mirror -w /mirror quay.io/os-forge/rpm-ostree-engine:latest rpm-ostree-engine-image --ref=OSTreeBeard/develop/x86_64 --origin=https://ostree.example.com

Or even combine both options to get an image from your local repository with a public update channel already configured as an alternative to adding a OSTree remotes.d conf.

With the --output= option you can control how the output image is named. It’s default name is os-iot.raw and it will be placed in the working directory. With something like OSTreeBeard-$(date +%Y%m%d) you can get a filename like OSTreeBeard-20210325.raw.

The command requires sudo because the virtualization requires access to the host system. For access to the loopback devices and in order to use dd on os-iot.raw we also need --privileged.

In order to compress the output raw-image you can use the xz command:

xz -0 -T0 ./os-iot.raw

This will give you a file named os-iot.raw.xz which is compatible with fedora arm-image-installer.

I have yet to test if this works cross-architecture. I’m currently building the images on hosts of the respective architecture simply because I already have them set up.

Let Gitlab CI Do The Repetition

With the RPM-OSTree-Engine templates you can build a .gitlab-ci.yml that automatically builds images from provided RPM-OSTree repositories. It’s intended use is as follow up job to building RPM-OSTree commits, but eventually this is up to you. You just need to get the sources at the designated places. The most basic example would be:

include:
  - project: 'os-forge/rpm-ostree-engine'
      ref: 'v0.3.4'
      file: 'gitlab-ci-template.yml'

variables:
  CI_OSTREE_REF_NAME: OSTreeBeard

build-ostree:
  stage: image
  extend: .build-ostree
  tags:
    - arm64
    - selinux

build-image:
  stage: image
  extend: .build-image
  tags:
    - arm64
    - selinux

The template jobs assume that your repository is present under /remote-storage/repo in both jobs. How the repo get’s there is up to you. I use SSHFS in production. In the rpm-ostree-engine-example repository you can find a more complete example .gitlab-ci.yml that makes use of caches to transport the resulting RPM-OSTree repository over to the image build job.

By default the image build job will upload the resulting os-iot.raw.xz as artifact you can then download in gitlab UI. Make sure that the artifacts file size quotas for your gitlab repository are high enough. A typical Fedora IoT 33 image will take up to 750 MB.

Take A Look At The Guts - Eww

The implementation can be found in resources/bin/image.sh.

We begin with writing some zeros into a file which we’ll later mount as virtual device. Using parted we partition this file before setting up the loopback device and mounting the file. Next we can create filesystems, prepare the boot partition, etc. The interesting part is when we use ostree to initialize a new OSTree based system and deploy our latest commit onto the root partition. After that we need to setup some common folders and SELinux permissions. Finally we need to place pre-compiled fstab, grub.cfg and bootloader for RPi3 and RPi4. Thanks to the coreos-assembler and osbuild projects for pointing that out.

Since we are aiming to run this in a container the container has to be executed with CAP_MAC_ADMIN capability in order to have access on mknod and loopback devices. Apparently dd also needs permission to write to os-iot.raw. For simplicity I’ve resorted to --privileged for now. Apart from that rpm-ostree-engine-image or podman for that matter have to be run with sudo so they are allowed to do their magic. That’s something I have to investigate more.

We found that it might be necessary to “probe” the loopX devices with mknod before trying to allocate them with losetup. I have no idea why though. That’s why you will find that strange for-loop in the Gitlab-CI template probing those devices.