Automated GPG signing in RPM-OSTree-Engine v0.4.0

With the latest update v0.4.0 RPM-OSTree-Engine can now also build Fedora 34 and most importantly handle automated gpg signing of the RPM-OSTree commits and summary file. Either directly after the build or as part of a separate job. Both approaches have their reasoning, we’ll get to that soon enough.

Signing the OSTree commits has huge security benefits. It protects against a range of attacks like update forgery. With this in mind how we approach signing the commits is a crucial decision to make and one to be discussed. In production I use automated signatures for all development branches, test deployments and PoCs while the stable release has it’s own GPG keys.

While this looks pretty straight forward, my generous estimate of three to four hours failed reality by around 8 hours making it a total of roughly 11 hours. So once again the rule of Pi applies and my foolish self was unprepared - again. Anyway, you may read about that in my previous post.

Let Gitlab sign the commits

This new feature is centred around three variables and a new template job. Using the Gitlab CI template adding CI_GPG_KEY_ID, CI_GPG_KEY_PASSPHRASE and CI_GPG_KEY to your CI pipeline environment enables GPG signatures in general.

The CI_GPG_KEY should hold a path to a file containing the key (that’s a feature in Gitlab CI). It’s up to you where those variables come from. One way to provide them is through Gitlab-CI variables. Another way is to use the environment configuration of gitlab-runner. Combining both methods allows you to split up passphrase and key so one compromised system is not enough to loose trust in this GPG identity.

Note: storing parts of secrets like a key or passphrase in the environment of your runner effectively makes the whole runner confidential. Don’t let arbitrary jobs run on this runner, especially by users you don’t trust.

Per default the build job will now attempt to sign the commits. To prevent it from doing so set the variables to the empty string in your .gitlab-ci.yml. You’ll at least want to do this if you aim on using the dedicated signing job and don’t want to sign all your commits.

The signing job will attempt to sign the commit last build in this pipeline. It can be restricted to only run on certain pipelines by adding rules to .gitlab-ci.yml. E.g. to only run on protected branches, if you want to prevent exposure of the secrets to all branches.

    rules
        - if: $CI_COMMIT_REF_PROTECTED == "true"

Or you can prevent the job from running on stable pipelines, because you aim on signing stable commits manually with a different key.

    rules
        - if: $CI_COMMIT_REF_NAME == $CI_DEFAULT_BRANCH

A YubiKey may be used if the key is configured on the runner host. The GNUPG home directory has to be mounted to the rpm-ostree-engine container under /root/.gnupg. The CI_GPG_KEY_PASSPHRASE should contain the YubiKeys user pin, CI_GPG_KEY_ID the ID of the key.

Introduced changes

With v0.4.0 the pre-defined build options have changed. Some of the optional options now have to be explicitly defined in the variable $CI_OSTREE_BUILD_OPTIONS. So instead of specifying CI_OSTREE_FILE=ostree.json it’s now CI_OSTREE_BUILD_OPTIONS="--spec=ostree.json.

The “main-branch-only” rule was removed from the .build-deltas template job. You now have to add that one to your .gitlab-ci.yml to keep that behaviour.

A fun journey again

The past three weeks I’ve been working on this from time to time were thrilling. Not only did I at last finish the automated gpg signing part but I also enhanced the CI pipeline. Previously I’d only run one Gitlab CI test job, with custom code, building an OSTree and corresponding image using the previously build container image for that pipeline. I changed the .gitlab-ci.yml configuration to import the projects own template and let it run an instance of all template jobs. That helped understanding and testing the usage of the template a lot. Now a change in the template would actually be used in the build pipeline of the project itself reducing the duplicated test code.

The time and resources to build this project now are insane though. On the main branch 9 jobs are executed in roughly 50 minutes, transferring 6.3 GiB through the network in the process. Out of desperation and the question how to avoid running this pipeline for simple changes like the docs I found out that one may skip the pipeline with git push -o ci.skip. Neat!