Having Fun With GPG Signatures In Gitlab-CI

Lately I’ve worked on adding automated signing in Gitlab-CI pipelines and jobs of RPM-OSTree-Engine.

It was a sunny, mildly cold Friday morning when I stood there in the kitchen, brewing my first cup of coffee for the day when I wondered which topic it will be for today. This day I fancied solving an issue bugging me for long every time I push new commits to our firmware repository, invoking the CI/CD pipelines: automated firmware signatures. Until now every time I push new commits for the firmware I have to manually keep track of those updates and sign them. While that makes sense and is totally manageable for the stable release branch it becomes quite tedious on develop let alone feature branches. So I set out on this little adventure.

When we last tried…

Last time we tried automated signing of firmware commits we were aiming for something working on all branches in a stable and secure way. We hooked a YubiKey into a build host, installed gitlab-runner and registered a dedicated, confidential runner at our gitlab instance. To this day we did not figure out what exactly went wrong but let me try to sum it up for you.

We configured the YubiKey for a GNUPG home directory on the host which was then mounted into the gitlab-runner container so it could be accessed in containers executed through pipeline jobs. Invoking gpg just like this will now trigger a pinentry-dialogue expecting the YubiKey pin so the key becomes available to the build script. Unfortunately in an unintended CI pipeline there is no one there to enter the pin nor can the pinentry-dialogue be rendered correctly anyway. We tried various things, wrapping the pinentry program into a custom script or even creating a dummy signature with a piped passphrase to unlock the key by using the gpg-agents caching feature. None of this actually worked, so we stopped the effort back than. But of course now, several months later, I have to try this again.

Changing the rules

I wanted to try something different this time, so I changed the rules or idea slightly just so the problem becomes a different one. Let’s keep manual signing for stable releases using the secured key kept on the YubiKey while for other branches we’ll now use a new CI identity more integrated with Gitlab-CI. This way we rule out one factor - access problems with the YubiKey and it’s pinentry program.

I setup a simple test bench with the rpm-ostree-engine project and tried it. Adding the CI_GPG_KEY_ID, CI_GPG_KEY_PASSPHRASE and CI_GPG_KEY variables. Now the Gitlab Instance is holding those secrets. That’s not ideal so I’ll get back to my thoughts about the security implications in a second. The program used for signing is ostree which only allows to specify a gpg-home directory and a gpg key id. Invoking it’s gpg-sign command will then call a pinentry-program. Not what I want to have in an unintended CI pipeline. So I researched and found some plain gpg based solutions.

It seems I’m lucky since gpg >= 2.0 a crucial bug affecting this scenario is solved and the handling for unintended usage has improved in general.

Following a suggestion from jonS90 on stackexchange together with a description on how to work with gpg keys in Gitlab-CI on dev.to I managed to get a working PoC fairly easily. The interesting bit here is to issue a dummy gpg signing command on a temporary file to unlock the key with the gpg supported way of feeding the passphrase via stdin. Plain ostree wouldn’t allow me to do this. Because the gpg-agent will now cache the unlocked key ostree is able to use the key for signing the firmware commit without prompting a new pinentry dialogue. So far so cold the coffee.

Now the Interestingly but frustrating bit: it only worked doing the dummy gpg command right before the ostree gpg-sign and only as part of the same script. In another scenario I would execute gpg before a build script, which internally executes ostree gpg-signafter around 5 minutes of build time. Since the default cache TTL is 10 minutes, or 600 seconds, I expected this to work but it didn’t. I don’t exactly understand why, but my guess is that the unlocked key and cache are not available to subsequent shell scripts which would make sense from a security perspective. Then again OSTree is in itself only a program calling gpg, so under that logic this should fail as well. If you have a clue, please let me know.

Let’s not store the secrets in Gitlab-CI, OK?

Storing our GPG key and corresponding passphrase for this identity is, well, not ideal from a security perspective. Anyone with access to the Gitlab Instance, the Projects variables or even the runner environment may read both the key and the passphrase making it a single point of failure. To mitigate this with Gitlab-CI one can supply either the key or the passphrase on the gitlab-runner host in the runners environment configuration. This way the key and passphrase only come together in the job or pipeline environment making it two systems that have to be compromised.

Keep in mind that the runner storing one half of your credentials should now be considered confidential and probably best only take jobs from this project by fairly trusted identities. One more security measure could be to configure the runner to only execute on signed, trusted commits of the project itself.

I’ll get back to this topic once I’m done with RPM-OSTree-Engine v0.4.0 containing this new signature feature. Have a nice day!