Manage A Fedora IoT Remix Using RPM-OSTree-Engine

In my post on Building a Fedora IoT OSTree I’ve covered how to manually have some fun with a Fedora IoT Remix. In order to make my life easier and enable usage in CI/CD-pipelines I’ve composed this in an easy-to-use container image.

What we’ll do is:

• Pull the RPM-OSTree-Engine container image
• Prepare a (git) repository with some custom configs
• Run the container and build
• (optional) sign the resulting OSTree commit

RPM-OSTree-Engine

I published a small FOSS tool that is targeted at enabling local developers and CI-pipelines to interact with the Remix’ repository. Let’s have a look on how it works. First pull the image.

podman pull quay.io/os-forge/rpm-ostree-engine:latest

We’ll need a directory which contains our Remix’ OSTree configuration that could be a git repository. Assuming we are in an empty directory you may copy over the example repository config from the RPM-OSTree-Engine project (which is also available in the container image). The example files are the bare minimum required to get the remix up and running. The only “unnecessary” bit is the additional watchdog package specified in ostree-files/ostree.json and it’s corresponding configuration file. The so called treefile ostree.json specifies which of the files in ostree-files/ actually end up in the system as well as which modules to remove, install, etc. Additional scripts, systemd units, configuration files all go into ostree-files/ and are specified in ostree.json.

Since we work on a SELinux enabled host we have to assign the correct labels so we are able to mount this directory when creating the container.

chcon -Rt container_file_t $(pwd)

Now that we’ve got this out of the way we can start the build which could take a while depending on your machines resources.

podman run --privileged --rm -it -v $(pwd):/build -w /build -e "OSTREE_REF_NAME=THEIoT" -e "OSTREE_REF_ARCH=x86_64" quay.io/os-forge/rpm-ostree-engine:latest rpm-ostree-engine-build

Note that the build will create some directories in our working directory. The .deploy-repo is the target destination which contains the repo that you’ll want to publish somewhere later on. If you’d placed a pre-existing repository in .deploy-repo the build would just have added a new commit to the tree instead of initializing an empty repo. The other additional directories are just temporary for the build process.

Next we can sign the newly created commit from the previous step.

chcon -Rt container_file_t <gpg-keyring>
podman run --rm -it -v $(pwd):/build -v <gpg-keyring>:/gnupg-keyring -w /build quay.io/os-forge/rpm-ostree-engine:latest 'export GNUPGHOME=/gnupg-keyring && ostree --repo=".deploy-repo" gpg-sign <commit-id> <gpg-key-id>'

The signing key is placed in a keyring under <gpg-keyring> and mounted in the container. SELinux policy is set to allow the mount. In CI you might want to create a keyring specifically for this purpose. I’ll get into more details about CI/CD in the future. The signature is then applied to the specified commit in your deploy repository.

Note 1: building an OSTree as of today is only possible in mounted host volumes due to limitations of SELinux within the fuseFS filesystem of containers.

Note 2: the --privileged flag is required because of the way rpm-ostree works. I’m still figuring out if individual CAPs will be sufficient.

Done!

Go Ahead And Upload

You may now deploy the contents of the folder .deploy-repo to any webserver so that Fedora IoT devices can start using it. If the directory was present from the start it’d get updated with the latest commit. If you want to know more about what’s going on here and didn’t already have a look I encourage you to go through the previous section on doing all of this manually.

As for deployment and production use I’m using my own specific approach to get my live OSTree repository into the CI/CD pipeline so it get’s updated automatically whenever new commits to the git repo arrive. However that’s stuff for future content, so if that’s something you look forward to stay tuned!

Other Tools Out There

I want to mention some projects and tools having a similar purpose and may provide a more extensive feature set while requiring a different approach.

As for the time of this writing the Fedora IoT upstream project is targeting to switch over from pungi and koji to osbuild/image-builder. I don’t use their current tooling since I want to build on gitlab-ci which is contradicted by relaying on koji, which is in itself Fedoras build, CI and CD system, that pungi is build for. image-builder might be worth a look though, especially to stay close to upstream. This project has it’s own set of remaining problems though.

Another tool-chain used to work with and handle rpm-ostree is coreos/coreos-assembler. This project also handles image creation. Especially the image generation feature of this project strikes me, apart from that I’ve not yet had much of a chance to look closer into it.

As hinted before I’m currently using sshfs to synchronize my remote rpm-ostree repository with my local machine and CI/CD build hosts. An alternative to this is the rsync-repos script that, as the name implies, uses rsync which is much more performant and reliable then network file-systems.