A security token is a very versatile thing. Use it on the web or locally for multi-factor or even passwordless authentication. Or save your PGP key there and have encrypted mail everywhere. Or build your own PKI with it. This article is about the latter: Making it a CA for SSH authentication. It also is a bit of a wake-up call for everyone out there to stop using plain public key authentication. If you're one of them, thank you for not using a password. You are still doing it wrong, though.

The problem with public key authentication

Authentication using keys as the preferred alternative to password login is a well-known and relatively easy thing to set up among administrators: An SSH key pair is generated with ssh-keygen, the public key then written into the authorized_keys file of the remote account you want to login to. Then you use the private key to do so. The classic command for this is ssh-copy-id. This is repeated for each machine from which, and each account to which, you want to log in.

However, this approach has a decisive disadvantage that even long-time administrators tend to make light of. SSH prints the following iconic output, especially during initial contact:

The authenticity of host '<host> (<IP>)' can't be established.
ED25519 key fingerprint is SHA256:<hash>.
Are you sure you want to continue connecting (yes/no/[fingerprint])?

This message is equivalent to a certificate warning in HTTPS: The identity of the host could not be verified. All too often, you simply type yes, and SSH then enters this host in the known_hosts file – the key is now stored permanently. Future connections then use this file to check whether the host has been "approved" and the query is not repeated if this is the case. The yes essentially says I have ascertained that my target host is the intended target. If the fingerprint on the remote host changes (new host keys, or even a man-in-the-middle attack), SSH will warn you.

The problem becomes apparent when such a warning is actually issued. Did something change on the server side, you ask. Or is it even an attack, you ask. Maybe you simply migrated to another host with the same name and now there's a conflict in known_hosts. ssh-keygen -R is then used to remedy this. But then the cycle repeats itself: You type yes, it's in known_hosts permanently and SSH warns you on the next mismatch. You can do so much better.

The solution: Cetificate authentication

Authentication using SSH certificates is a bit more complicated to set up because, similar to TLS, it requires a PKI infrastructure: A trusted certification authority and at least one process for signing keys. However, certificates are particularly valuable because they support host, user, and time restrictions.

With SSH certificates, a distinction is made between host and user certs. Host certs verify hosts and their names to clients (I connect to example.com, is the sent host cert issued to this name?), user certs verify clients to hosts (client tries to log in with user123, is the client allowed to do so according to the client certificate?). Clients and hosts trust the CA that issued these certificates. This way, the client can be sure that example.com is known to the PKI, and the host can be sure that the user is allowed to log in with user123. The proof of trust is no longer individual key pairs, but the cert of a central SSH CA that has certified other keys. Any key pair previously generated with ssh-keygen can become a CA key pair.

With any PKI, securing your private CA key becomes pivotal – it can issue certs willy-nilly, after all. What better location can there be but specialized hardware? Here, a YubiKey 5 is used, but any token supporting Personal Identity Verification (PIV, aka FIPS 201) is capable of doing this (at least Solokey and Nitrokey, so they're mentioned, too). This solves the problem of securing the private key: It is stored on the token and can only be used if the PIV PIN is entered beforehand and, if necessary, the button is touched.

Preparation

Software requirements are pcsclite, p11-kit, and the libykcs11 module (use the ykcs11-p11-kit-module AUR package on Arch Linux). The YubiKey Manager (ykman) is used for interfacing with the YubiKey.

After installing ykman, first check whether the token is recognized and the PIV application is activated. For YubiKeys 4 and older, CCID mode must be activated, which is active by default.

ykman list

WARNING: PC/SC not available. Smart card protocols will not function.
YubiKey <model> (<fw>) [OTP+FIDO+CCID] Serial: <serial>

If you get a warning about PC/SC not being available (see above), make sure the pcscd service is started. Then check whether PIV is enabled.

ykman config usb -l

PIV

Enable it if the output does not contain PIV.

ykman config usb -e PIV

Enable PIV.
Configure USB? [y/N]

Secure the PIV interface, while you're at it. By default PIN, PUK and management key are a kind of thing an idiot would have on his luggage: 123456, 12345678 und 0123456789012345[…]. The following commands are interactive, so simply follow the instructions.

ykman piv change-pin
ykman piv change-puk
ykman piv change-management-key

Steps to better SSH authentication

Creating the CA

First, create a key pair and then a certificate in the PIV store. The cert is not used as is, it's just a requirement of the PKCS#11 standard. The slot used must be 9d, see PIV Slots. With --pin-policy and --touch-policy, you can optionally decide whether and when a PIN query or a button query should appear when this slot is addressed. The default setting here is left unchanged (pin ALWAYS, touch NEVER). The argument sshca.pub is a path, so this is referring to the file in the current working directory. ykman needs this briefly to generate the cert and can then be deleted because the SSH format of the key is needed, and also because this public key can be extracted again at any time with ykman.

ykman piv keys generate 9d sshca.pem
ykman piv certificates generate -s "SSH CA" -d 3650 9d sshca.pem

With ssh-keygen you can now extract the public key from the slot in authorized_keys format. However, -D does not distinguish between slots, so you have to filter by the “Key Management” key. The argument for -D is the path to the libykcs11 module looked up in the standard path for libraries. The option also accepts paths, so you can enter it manually if it cannot be found (usually /usr/lib/libykcs11.so). For demonstration purposes, it is written to /tmp.

ssh-keygen -D libykcs11.so | grep Management > /tmp/sshca.pub

Creating a host certificate

Generate a key pair normally, choose the options as you see fit. Here, ed25519 is used with a comment, then saved to /tmp:

ssh-keygen -t ed25519 -C "Host key of server" -f /tmp/host-myserver

Generating public/private ed25519 key pair.
Enter passphrase (empty for no passphrase): 
Enter same passphrase again: 
Your identification has been saved in /tmp/host-myserver
Your public key has been saved in /tmp/host-myserver.pub
The key fingerprint is:
SHA256:YD40…BwA Host key of server
…

Now you sign it.

• -h is very important. It specifies that a host key is being signed.
• -s normally specifies the private key of the CA to be used for signing. However, together with -D it's used to identify a private key based on the public key specified with -s, so it works like a filter for the output of -D.
• -I is the "name" of the key. An SSH client logs that the key with this name was presented to it by the host, so give it a name you know what to do with. Or name it "Murphy Cooper", for all I care.
• -n specifies the principals for which the cert is to apply, separated by commas; for hosts, these are the names by which the machine can be reached.
• With -V, a validity period can be specified; without it, the certificates are valid indefinitely.

ssh-keygen -h -D libykcs11.so -s /tmp/sshca.pub -I "Host key of server" -n example.com -V +52w host-myserver.pub

Enter PIN for 'YubiKey PIV #<serial>': 
Signed host key host-myserver-cert.pub: id "Host key of server" serial 0 for example.com valid from <start> to <end>

A new file is created: host-myserver-cert.pub. This file, together with the private key host-myserver and the public key of the CA sshca.pub must now be transferred to the host (if generated externally) and ideally saved under /etc/ssh/. Its owner/group should then be changed to root:root.

Now the SSH daemon must be configured to present this host cert instead of some random host keys and accept certs issued by the CA. This is done in sshd_config with the following options:

HostCertificate /etc/ssh/host-myserver-cert.pub
HostKey /etc/ssh/host-myserver
TrustedUserCAKeys /etc/ssh/sshca.pub

Other HostKey options and AuthorizedKeysFile can be commented out. Restart the SSH daemon to apply.

Generating a user certificate

Here, too, you first generate a normal key pair as above (e.g., -f user-user123). The signing step is similar, except that -h is omitted because we are signing a user cert. -n is now a comma-separated list of usernames that are allowed to log in with this cert. To clarify: The following certifies that you are allowed to log in remotely as user123. The user you're doing this as is not considered: User admin could use it to login as user123.

ssh-keygen -D libykcs11.so -s /tmp/sshca.pub -I "User key of user123" -n user123 -V +52w user-user123.pub

Enter PIN for 'YubiKey PIV #<serial>': 
Signed user key user-user123-cert.pub: id "User key of user123" serial 0 for user123 valid from <start> to <end>

A new file is created: user-user123-cert.pub. Now you have to decide whether only certain users of a system or all users should use this cert. Based on your decision, the new file is transferred together with the private key user-user123 either to ~/.ssh of the relevant user or to /etc/ssh/. Let's assume that the cert is tailored to one user, so it would be transferred to ~/.ssh.

Now you need to determine whether there are other users on the system who need to use other certs from the same CA (with different user names, for example) or whether only this single user should use certs from the CA. This determines whether the user-specific known_hosts is used to trust the CA, or the system-wide ssh_known_hosts in /etc/ssh/. The decision will often be the latter, so the CA is entered globally. Therefore, create this file and write the following into it:

@cert-authority example.com ssh-rsa AB3z…

The second section with example.com is a comma-separated list of hostnames for which SSH should attempt to verify certs with the public key that follows. This list accepts globs: * is therefore a valid entry and indicates that all hosts should be verified. As mentioned above, this is followed by the public key of the CA, meaning, the contents of sshca.pub. You may now delete knownhosts from ~/.ssh/ to make SSH reauthenticate all clients.

Testing the connection

Connect to a host primed for cert authentication with a user ready to present a user cert. Use -v to get some clue as to what is happening in the background. Some real output from a setup that is no longer active:

ssh user@host -v

[…]
debug1: identity file /home/<user>/.ssh/User-AC type 3
debug1: certificate file /home/<user>/.ssh/User-AC-cert.pub type 7
[…]
debug1: Authenticating to <host>:<port> as '<user>'
[…]
debug1: Server host certificate: ssh-ed25519-cert-v01@openssh.com SHA256:jsup[…], serial 0 ID "Host-DediServer" CA ssh-rsa SHA256:AGQw[…] valid forever
[…]
debug1: Host '<host>' is known and matches the ED25519-CERT host certificate.
debug1: Found CA key in /etc/ssh/ssh_known_hosts:1
debug1: found matching key w/out port
[…]
debug1: Will attempt key: /home/<user>/.ssh/User-AC-cert.pub ED25519-CERT SHA256:vuX7[…] explicit
debug1: Will attempt key: /home/<user>/.ssh/User-AC ED25519 SHA256:vuX7[…] explicit
[…]
debug1: Next authentication method: publickey
debug1: Offering public key: /home/<user>/.ssh/User-AC-cert.pub ED25519-CERT SHA256:vuX7[…] explicit
debug1: Server accepts key: /home/<user>/.ssh/User-AC-cert.pub ED25519-CERT SHA256:vuX7[…] explicit
Authenticated to <host> ([<IP>]:<port>) using "publickey".
[…]
Last login: <time> from <IP>
[<user>@<hostname> ~]$

The following lines are particularly interesting, as they perfectly summarize what has now been achieved.

• Host ... is known and matches the ED25519-CERT host certificate: The client was able to verify the presented host cert; it is the server it claims to be.
• Offering public key / Server accepts key: The server was able to verify the presented user cert; it authorizes the client to log in with the specified user.

If you receive a message at this point that the host could not be verified, there is a problem somewhere. ssh -v on the client side and the SSH logs on the host side will help with troubleshooting – as will the comment section below. Try it, it's free and doesn't require registration. :)

Source material

These pages served as inspiration for my own experiments. They also describe more advanced topics, such as how to set up a PKI with separate user and host CAs that both trust the SSH CA, or what to do when multiple YubiKeys are in use. However, they do not address the automation of these processes.

• YubiKey as an SSH CA (GitHub Gist)
• Using SSH certificates
• Managing servers with OpenSSH CA

I wrote this originally for GNU/Linux.ch and largely translated it with DeepL (thank you, DeepL) with minor changes in all sections. Check them out if you can speak German. And be sure to use DeepL for your translations instead of some AI – their translations are high quality, and they don't cause RAM shortages.