In recent weeks, I’ve been working on adding key rotation support to flatpak, so that repositories which sign their commits and summary files with a key with an expiry date have a way to push updates to that key to all the clients which use them. Currently that’s not possible without each client manually having its config updated to use the new key data (even if the change in key data is just to bump the expiry date).
In the process I’ve learned a few more things about GPG keys, subkeys, signatures, UIDs, etc., which I thought I might dump here in case it’s useful for someone else (or me, in the future). I don’t claim to be an expert, so I may still be misunderstanding some bits. GPG is complex. One thing which has helped ground things is finding the documentation in
RFC 4880
(and related) which defines the GPG packet format.
One thing which keeps flatpak’s use of GPG simple is that it doesn’t use any of the web-of-trust features or trust-on-first-use (TOFU). Its use of GPG is limited to the keyring format (essentially
pubring.gpg
), for storing and publishing public and private keys, subkeys, signatures, revocations, UIDs; and using them to sign and verify OSTree commits and various repository files (
summary
,
summary.idx
, etc.).
Quick primer on the parts of GPG
GPG has keyrings: collections of primary keys. A primary key has public and private/secret parts. The way flatpak uses it, the secret part of a key always stays on the server, and is used to generate signatures. The public part is published by the server and configured in every client using that repository, used to verify the commit signatures. Each client has one keyring per remote it has configured. Typically this contains one primary key, but it could contain several, any of which can be used to verify signatures from that remote.
A key is identified using a fingerprint, a 40-character hex string. It can also be identified using a key ID, which is a substring of the fingerprint. There’s also keygrips, but let’s ignore those.
You can see fingerprints for keys using
gpg --list-keys --fingerprint
.
A primary key has one or more subkeys (at least in flatpak’s usage). All of these keys have public and private/secret parts as above. Each key has a usage which indicates what GPG will let you use it for, such as certifying, signing, authenticating, encrypting. The primary key is typically used to certify its subkeys by creating cross-certification signatures which bind them to the primary key, forming a short chain of trust — anyone who trusts the primary key should trust a subkey which it cross-certifies.
You can see subkeys as the
sub
lines using
gpg --list-keys --with-subkey-fingerprint
. The
[SCE]
,
[S]
,
[E]
(etc.) fields show the usage flags for a key.
The other usage we care about is signing (flatpak doesn’t use authentication or encryption usages). GPG separates keys by usage to prevent attacks where a key is used for a purpose it’s not intended for, and because keys used for different purposes often need to be treated with different levels of care.
In particular, separating by usage means the private/secret part of the primary key can be kept completely offline, and only brought out in a special key signing ceremony when a new subkey needs to be generated and cross-certified. This reduces the risk of the very valuable primary key, which is the root of every client’s trust in the repository, being leaked.
So, we use a signing subkey for day-to-day signing of OSTree commits. For a big flatpak repository, the private/secret part of this subkey might be kept in a hardware security module, so it can’t be exfiltrated from the server if the server were compromised. But there’s still the risk of a compromised server being used to sign things it shouldn’t (such as malicious apps).
That’s a matter for server security, but we can somewhat mitigate against the possibility of the signing subkey being leaked by setting an expiration date on it. Clients might choose not to trust signatures made by it after that date; and
gpg
certainly wouldn’t allow it to be used to create new signatures.
The expiry date of a key is shown as an
expires
field in the
gpg --list-keys --with-subkey-fingerprint
output.
What happens when the subkey expires? By that point, the administrators should have generated another subkey, cross-certified by the primary key in a key signing ceremony (I assume the ceremony involves cake). The private/secret part of the new subkey needs to stay secret, as before; but the public part needs to be distributed to every client’s keyring, along with the new cross-certification signature from the primary key, so the clients know they can trust signatures made by that subkey.
That’s the bit which flatpak is currently lacking.
So in summary: GPG has keyrings. Keyrings have primary keys. Primary keys have one or more subkeys and cross-certification signatures from the primary key on those subkeys. Each subkey has a usage, but flatpak only uses certify (for the primary key) and sign (for the subkeys). Keys can have expiration dates.
And if you want to see the full contents of a keyring, run
gpg --list-keys --with-colons
. It’ll output everything (no filtering) in a machine readable format described
here
(best reference I’ve been able to find), which is sometimes easier to use than remembering which
--with-blah
option to pass to GPG to get it to show the information you want.
What else does GPG have?
Quite a few things. We’ll ignore the big things which are not relevant to flatpak.
Each primary key also has one or more UIDs. These are like subkeys in that they are cross-certified by the primary key. Each UID is a user identity — typically a name and email address. If you were using GPG in a web of trust, the binding between the primary key and a UID is what you sign that you trust when you sign someone’s key in a key signing party.
The UIDs are listed below each primary key in
gpg --list-keys
.
Flatpak doesn’t need UIDs, but they are an unavoidable part of GPG — each primary key must have at least one. A flatpak repository will typically put a server contact email address in the UID and then everyone will ignore it.
UIDs can be revoked; for example if someone loses control of the email address in it and wants their friends to no longer trust emails from it. Flatpak currently doesn’t use this.
What else can be revoked? The cross-certification signatures! You may have heard of a GPG revocation certificate. This is a way of revoking an entire primary key. But there’s also a way of revoking a particular cross-certification signature, meaning that the primary key is still valid/trusted, but the owner of the primary key has lost control of one of the subkeys, and that subkey should no longer be trusted. This is different from key expiration, as it’s a statement that something has explicitly gone wrong.
Because of how GPG is built up as a series of packets of different types, a signature revocation is actually a revocation packet appended to the primary key. This means you can re-cross-certify a subkey after revoking it, by appending another cross-certification packet. And even revoke it again after that. Not sure if there’s a use case for this or if it’s just a consequence of the packet format, but this behaviour does play havoc with working out whether to trust a subkey.
Cross-certification signatures can also have an expiration date built into them, separate from the expiration date of the subkey. I’m not sure of the use case for this either, but there must be one.
Some notes on running GPG on the command line
GPG is historically famously hard to use. I feel this has got better in recent years, particularly for scripting it. In particular it’s added a whole load of
--quick-blah
commands to generate keys, set expiries, etc. from scripts.
One thing which repeatedly tripped me up before I stopped trying to fight it was its concept of a ’homedir’. GPG needs to look for its keyring (and trust database, and various other files) somewhere, and will not run without them, so you always need to pass it a ‘homedir’ to look for them in. By default, this will be
~/.gnupg
, so it’s very easy to accidentally end up operating on your personal GPG keyring when you’re trying to do something in a project.
If using GPG as a tool or in a script, I think you should always create a temporary homedir, pass it as
gpg --homedir=/path/to/temp
and explicitly import whatever keys or context you need into this homedir before doing whatever operation you need.
This is necessary even if ‘all’ you want to do is view a downloaded
.gpg
keyring, because what GPG displays may be affected by the trust database in its homedir. So to view a downloaded keyring you should still do something like
mkdir temp; gpg --homedir=./temp ./path/to/download.gpg
.
If you are trying to sign something, you will typically pass the fingerprint or key ID of a primary key to GPG; for example as
gpg --local-user 0xfingerprint --sign ./path/to/file
. GPG will helpfully use the usage flags of the subkey of that primary key to choose which subkey to sign with. If you want to sign with a
specific
subkey, you need to suffix the fingerprint with an exclamation mark (
!
) otherwise GPG will still choose what it thinks is the most appropriate subkey, which might not align with the subkey you carefully chose. This
!
suffix format is common throughout the GPG command line interface for when you want to specify a specific subkey.
Sorry
That was more of a braindump than I imagined when I set out to write this. I hope some of it is useful; feedback welcome if I’ve got anything wrong. If any GPG experts fancy reviewing key rotation support in flatpak, the draft implementation is
here
.
chevron_right
Tower of Hercules in A Coruña
On the way to lunch
My talk during the second day
I found the official* GNOME-branded ice-cream. (*Not official)
Sponsored by the GNOME Foundation
).
.
