Automating The Fedora IoT Remix Build with Gitlab CI

Previously I showed how I build and maintain a Fedora IoT Remix and how to use RPM-OSTree-Engine to ease the process.

This time I want to talk about how I use RPM-OSTree-Engine in Gitlab CI to automate the build process and deployment of the commits to a public OSTree repository. At the end we’ll have a git repository with Gitlab CI configuration to keep our public OSTree repository up-to-date whenever we change something in our git sources.

What we need depends a bit on how to approach this in the end. Let’s start with a minimal setup and I’ll then explain where I take special decisions.

• A git(lab) repository
• Gitlab CI configuration in .gitlab-ci.yml
• Basic setup of RPM-OSTree files
• CI runners

One Gitlab Repository To Rule Them All

So our new git repository, which will be the source of truth for our Fedora IoT Remix, will require a base setup of files required by RPM-OSTree and a .gitlab-ci.yml containing the CI configuration. The neat thing about RPM-OSTree-Engine is, that apart from the container image it also ships with a Gitlab CI Template that you can import to ease out the setup of CI configuration a bit.

In RPM-OSTree-Engines git repository you’ll find a basic repository structure. It’s basically as follows:

/
├── ostree-files/ - Everything to be included into the image
│   ├── ostree.json - treefile
│   └── treecompose-post.sh - post-processing script (shell)
├── hook.d/ - Place custom pre-build scripts here
└── .gitlab-ci.yml - extends rpm-ostree-engine/gitlab-ci-template.yml

The so called treefile (here under ostree-files/ostree.json) contains a description of what files to include where in the system. Note that OSTree takes care of stuff like setting binaries as executable when placed in /usr/bin. The treefile also specifies which packages to install or remove, stuff like that.

The treecompose-post.sh is a script that is executed in the build as one of the last steps. It’s in the scope of the target file system so you may make last-minute adjustments to file ownership or anything in that regard. Note that more complex systems like firewalld, systemd or NetworkManager are not available at this point.

The hook.d directory is a place for scripts that are executed in the scope of your build environment, not the Remix’ filesystem. It’s designed for you to make adjustments to the upstream repository, e.g. with sed or to remove / add files within the build process.

Finally .gitlab-ci.yml is this times focus, as it contains the interesting configuration for building new OSTree commits from our git commits. In this case this file is minimal and just contains extending jobs of the “upstream” template jobs.

One Gitlab CI Configuration To Find Them

A minimal .gitlab-ci.yml looks as follows:

include:
  - remote: https://git.shivering-isles.com/os-forge/rpm-ostree-engine/-/raw/main/gitlab-ci-template.yml

variables:
  CI_OSTREE_REF_NAME: OSTreeBeard

build-ostree-amd64:
  extends: .build-ostree
  # skip sshfs mount
  before_script:
    - true
  tags:
    - amd64

The first include: statement references the template containing the details of the CI configuration. Note that currently the referenced template is just in the main branch. While it should be stable most of the time I plan to use versioned release based references.

The variables: section is used to control the base name of our OSTree ref: myRemote:OSTreeBeard/stable/x86_64. The stable comes from building on the repositories default branch. On other branches the slug is used at this portion of the ref. The architecture is automatically detected from the architecture of the build host.

Last but not least is the actual build job build-ostree-amd64 which extends the template job .build-ostree. Since the job is set to use sshfs mount per default we can skip this for now with the before_script: true statement. Finally the tags are used to control on which build runner to execute this task. This has some significance. On SI-Gitlab and at my work we operate AWS EC2 autoscalers for x86_64 as well as aarch64. Both runner types have different tags so we can control on which of them we want to build our firmware. Adding another supported architecture is as simple as adding another job with a different tag attached. Of course a runner has to be available at Gitlab though. The used AMIs for the runners are also somewhat special, I’ll get more into that in the future. The baseline here is your runners need to have SELinux installed.

Add CD To Bind Them

With this setup the commits in the git repository will cause gitlab CI pipelines to build new OSTree commits automatically. Of course there is one crucial thing missing. Right now all that would happen is that a new commit is build and discarded with the build runner as soon as the job is done.

In order to get this persistent the paths $(pwd)/.deploy-repo or better /remote-storage inside the container need to be synchronized before and after the build job with the remote location that serves the OSTree repository. One way to do this, that I’ve implemented in RPM-OSTree-Engine, is to use sshfs to mount the filesystem containing the repo. That way the container will attempt to mount the remote before the build and sync it after it’s finished.

There are other ways to do this, that could be contributed to RPM-OSTree-Engine or simply implemented in the before_script and after_script sections of the Gitlab CI jobs.

One alternative I know of comes from the OSTree developers, sync-repos. It’s based on rsync and through it’s parallel nature features some interesting properties such as low latency. However it seems that in general a network filesystem is better suited for this kind of job since they provide important properties around synchronization safety.