Sebastian Wiesner

Proton Pass as 1Password alternative?

2026-01-02T00:00:00Z

Proton Pass as 1Password alternative?

In the days before Christmas 1Password had a pretty stupid issue which broke syntax highlighting on all sites using PrismJS. As my 1Password subscription also runs out soon, this nudged me to look at alternatives.

I'd love to use a European provider, or perhaps even self-host my own solution, and I can't help but think that that 1Password issue is a symptom of sloppy engineering. And while I don't have specific doubts about 1Password's security standards, it's still sort of concerning that 1Password apparently just dumps random npm.js dependencies in its extension, which has all my credentials. The NPM ecosystem doesn't have a stellar security track record after all.

So I took a look at Proton Pass. Unfortunately it didn't make the cut, and can't (yet) replace 1Password for me. But that's probably just because I've been using 1Password for a long time (I think it's almost ten years by now), and gotten used to many of 1Password's features. Had I started fresh, I'd quite likely kept using Proton Pass, as it's a pretty good password manager, and the rest of the Proton product suite is pretty cool too. I'll likely take another look at Proton Pass in a year or two, and since it's pretty much impossible to find good in-depth technical comparisons of different password managers, I figured I'd keep note of my testing here.

I think I'll also look at self-hosting Bitwarden with vaultwarden, but since that's a bit more effort to set up, I haven't had the time yet. Maybe I'll have another blog post about this soonish.

What I like

Proton, the company behind Proton Pass, is a strictly European provider with its own servers, i.e. not AWS or Azure like apparently everyone else. Since we can't really trust overseas anymore these days, that's a major point for me. I also appreciate that Proton's major shareholder is a non-profit foundation, and any kind of venture capital.

Proton also has great customer service which responded in less than a day even around Christmas and New Year holidays, and was really helpful and supportive.

Proton Pass itself has native clients for all platforms I care for (Linux, Windows, iOS, Android). The apps look pretty good, and are straight-forward to use. It also features CLI which, despite still being labelled as Beta, appears pretty complete and solid, and already includes SSH support. It can run as SSH agent pre-loaded with all SSH keys from select vaults, and it can also load SSH keys from select vaults into a running SSH agent. It's pretty easy to script, and while I didn't test all of it, it looks like it provides comprehensive access to all features of Proton Pass.

And last but not least its family plan also considerably cheaper than 1Password, and can be combined with subscriptions for other proton products^[1].

What I miss

Importing data from 1password

I started with an import of all my 1Password vaults, and quickly noticed that the import leaves quite a bit to be desired.

Items with lots of custom fields and sections sometimes get mangled during import; the imported items have duplicated fields, and sometimes sections with empty titles. The latter is particularly annoying: Proton Pass doesn't actually allow sections with empty titles, so when you edit the item later on the "Save" button is disabled, with no clear indication or error message. Only a tiny red error icon in empty space points out a missing section title, and it took me a while to figure out that I had to put in section titles to be able to save my edits to the item.

But worse, the import actually lost some data in the process. For a start, none of my passkeys made it over, thought I'm in fact no sure whether 1Password actually exports these. But it definitely does export plain "Document" entries, and these got lost too: Proton Pass imported these as empty items, with correct title, but no attachment. In other words, it lost the actual relevant content of the item, and without any warning in fact. Since we keep quite a lot of such documents in our shared vaults (notably copies of important legal documents such as birth certificates, deeds of possession, etc.), that was a major disappointment.

Per-device registration

Proton Pass secures the vault data with only a single master password. That's no concern if you actually pick a strong master password, but alas, non-technical people (e.g. all my family save me) have quite a different understanding of what's a strong password. As such, I very much appreciate that 1password has two distinct secrets to protect vaults: a strong random secret key which is kept offline (we have printed copies in our strongbox at home) and only needed once every time a new client is registered, and the master password which routinely unlocks the vault in everyday use of the desktop client and apps. The second random secret is an effective companion to somewhat weaker passwords, and makes brute force attacks against the password impossible.

I know that it probably makes not much of a difference effectively, since even a weaker password combined with the usual server-side countermeasures against brute forcing (contemporary PBKDFs, rate-limiting, etc.) should hold up, but I still feel a lot safer and sleep better knowing that with 1Password it's factually impossible to gain access to vault items by brute-forcing passwords.

Other nuisances

Besides these three major points, there are also a few minor nuisances.

Proton Pass won't let you add custom icons to entries^[2]; I've got a lot of custom non-login entries in your vault, so my entire vault just had the generic grey pliers icon all over the place. I might just as well not have any icons at all.

Proton Pass doesn't support plain Documents similar to 1Password; to store document I'd have to create an empty item with a single attachment which is a bit clumsy.

The browser extensions and the CLI aren't linked to the desktop app like with 1Password, where unlocking the desktop app automatically unlocks the CLI and browser extensions as well. So depending on my current tasks I ended up repeatedly unlocking Proton Pass three or more times.

The mobile app has a biometric unlock, and the browser extension can unlock with a short PIN, but neither seems to request the actual password again periodically. Probably not that relevant for security, but having to type the actual master password every once in a while is also a good way to make sure people don't forget it.

The desktop app downloads don't appear to be cryptographically signed, which is probably not too relevant either, but still feels like a strange oversight for a security-minded app.

Summary

I'd really like to use Proton Pass, but I definitely miss a better import from 1Password, tags to organize my vault items, and a two-level security with explicit client registration.

These three issues rule out 1Password for me; all the other things above are mostly minor nuisances which I could easily get used to. I might have used Proton Pass nonetheless in combination with the whole Proton suite, if only Proton Drive had a proper Linux client.

Proton Pass is a good password manager in and by itself, the CLI is great, and it has bonus points for being sort of not-for-profit and Europe-based. Let's see how things evolve; perhaps I'll take another look at Proton Pass in a year or two.

Incidentally, if Proton Drive had a convenient Linux desktop client, I might actually have subscribed to the whole Proton Suite and kept using Proton Pass despite its shortcomings, because the Proton Suite as a whole also has some pretty convincing features on its own. But alas, Proton Drive has no Linux client yet, and all this rclone mount and rclone bisync business is not really great. ↩︎
Curiously Bitwarden doesn't appear to support custom icons either, while still fetching Favicons from websites. I really can't understand why. If you're already storing (fav)icons along with entries, and showing them in place of standard icons, how hard can it be to just have a small image picker to edit these icons? ↩︎

forgejo-runner image with mkosi

2025-11-30T00:00:00Z

forgejo-runner image with mkosi

Codeberg doesn't have much CI capacity so I bring my own action runners to run workflows in my Codeberg repositories. I use mkosi to build a generic system image containing the runner as well as all dependencies I need for CI, which I can then deploy on any system (e.g. spare laptops at home, hosted cloud VMs, etc.) that uses systemd. This post shows the image definition.

Base image

The base image is pretty straight-forward. We build a discoverable disk image based on Archlinux, configure timezone, locale, hostname, and keymap, and install a basic set of packages:

[Output]
ImageId=forgejo-runner
Format=disk
CompressOutput=true

[Distribution]
Distribution=arch

[Content]
Bootable=false
Timezone=UTC
Keymap=us
Locale=C.UTF-8
Hostname=forgejo-runner-????-????
Packages=
    # Base packages
    base
    base-devel
    # pacman-offline to auto-update the runner system
    pacman-offline
    # The forgejo runner daemon itself
    forgejo-runner
    # node 24, as required by actions
    nodejs-lts-krypton
    # Git, obviously
    git
    # Extra dependencies for the actual CI jobs
    rustup
    npm
    # ... whatever you need

We also need a preset file to enable the pacman offline timer for auto-updates as well as the forgejo runner service. We also disable homed while at it, because we don't need user management in a single-purpose disk image. The preset file goes into the default extra tree below mkosi.extra where mkosi automatically picks it up and includes it in the image.

$ cat mkosi.extra/usr/local/lib/systemd/system-preset/10-forgejo-runner.preset
disable systemd-homed.service
disable systemd-homed-activate.service
enable forgejo-runner.service
enable pacman-offline-prepare.timer

Disk layout

By default, mkosi tries to minimize the size of all partitions in the disk image. However, we do need a large /var for toolchains, caches, build outputs, etc., so we adjust the partitions created in the disk image with two repart configuration files:

mkosi.repart/10-root.conf:

[Partition]
Type=root
Format=btrfs
CopyFiles=/
Minimize=guess

mkosi.repart/20-var.conf:

[Partition]
Type=var
CopyFiles=/var:/
Format=btrfs
SizeMinBytes=10G
SizeMaxBytes=10G

Runner configuration

Next we initialize a default configuration for the forgejo-runner:

$ mkdir mkosi.extra/etc/forgejo-runner
$ curl https://code.forgejo.org/forgejo/runner/raw/branch/main/internal/pkg/config/config.example.yaml \
   > mkosi.extra/etc/forgejo-runner/config.yaml

The defaults are good; we just need to adapt them slightly to remove some example values, reduce the job timeout, and pick some more reasonable directories:

diff --git a/mkosi.extra/etc/forgejo-runner/config.yaml b/mkosi.extra/etc/forgejo-runner/config.yaml
index 5addac4..420c43d 100644
--- a/mkosi.extra/etc/forgejo-runner/config.yaml
+++ b/mkosi.extra/etc/forgejo-runner/config.yaml
@@ -25,15 +25,13 @@ runner:
   capacity: 1
   # Extra environment variables to run jobs.
   envs:
-    A_TEST_ENV_NAME_1: a_test_env_value_1
-    A_TEST_ENV_NAME_2: a_test_env_value_2
   # Extra environment variables to run jobs from a file.
   # It will be ignored if it's empty or the file doesn't exist.
   env_file: .env
   # The timeout for a job to be finished.
   # Please note that the Forgejo instance also has a timeout (3h by default) for the job.
   # So the job could be stopped by the Forgejo instance if it's timeout is shorter than this.
-  timeout: 3h
+  timeout: 1h
   # The timeout for the runner to wait for running jobs to finish when
   # shutting down because a TERM or INT signal has been received.  Any
   # running jobs that haven't finished after this timeout will be
@@ -92,7 +90,7 @@ cache:
   #
   # If empty, the cache data will be stored in $HOME/.cache/actcache.
   #
-  dir: ""
+  dir: "/var/lib/forgejo-runner/cache"
   #
   #######################################################################
   #
@@ -191,4 +189,4 @@ container:
 host:
   # The parent directory of a job's working directory.
   # If it's empty, $HOME/.cache/act/ will be used.
-  workdir_parent:
+  workdir_parent: /var/lib/forgejo-runner/act

Register runner from credentials

We also need to register the runner with Codeberg (or any other Forgejo instance). We could do this by hand: open a shell in the image and run forgejo-runner register. However, it's a lot more convenient to do this automatically when booting the image. Systemd has a concept of credentials which we can use to inject registration data into the image when booting. Inside the image we can then automatically register the runner if it's not registered yet.

For this purpose we add a small one-shot service mkosi.extra/usr/local/lib/systemd/system/forgejo-runner-register.service:

[Unit]
Description=Forgejo Runner Registration
After=network-online.target
Wants=network-online.target
ConditionPathExists=!/var/lib/forgejo-runner/.runner

ConditionCredential=forgejo.runner.name
ConditionCredential=forgejo.runner.instance
ConditionCredential=forgejo.runner.labels
ConditionCredential=forgejo.runner.token

[Service]
Type=oneshot
RemainAfterExit=true
ExecStart=/usr/local/bin/forgejo-runner-register-from-credentials
User=forgejo-runner
WorkingDirectory=/var/lib/forgejo-runner

ImportCredential=forgejo.runner.*

This service only runs when the runner is not registered (ConditionPathExists) and all secrets for registration are available (ConditionCredential). It imports all registration credentials from the service manager and calls a small script at mkosi.extra/usr/local/bin/forgejo-runner-register-from-credentials which simply callsforgejo-runner register with the credential values:

#!/usr/bin/bash
set -Euo pipefail
exec /usr/bin/forgejo-runner register \
    --no-interactive \
    --name "$(systemd-creds cat forgejo.runner.name)" \
    --instance "$(systemd-creds cat forgejo.runner.instance)" \
    --labels "$(systemd-creds cat forgejo.runner.labels)" \
    --token "$(systemd-creds cat forgejo.runner.token)"

Then we arrange for this service to be started before the actual forgejo-runner with a small

[Unit]
After=forgejo-runner-register.service
Requires=forgejo-runner-register.service
ConditionFileNotEmpty=/var/lib/forgejo-runner/.runner
ConditionFileNotEmpty=/etc/forgejo-runner/config.yaml

Build and launch

We build the image with mkosi and import the resulting forgejo-runner.raw.zst as a machine image:

$ importctl import-raw --class=machine \
   forgejo-runner.raw.zst codeberg-runner

Now we obtain a registration token and boot the image manually to pass credentials to register the runner:

$ systemd-nspawn --boot --link-journal=try-guest -U \
   --machine codeberg-runner \
   --set-credential=forgejo.runner.instance:https://codeberg.org \
   --set-credential=forgejo.runner.name:foo-runner \
   --set-credential=forgejo.runner.labels:foo:host \
   --set-credential=forgejo.runner.token:THE_TOKEN

From now on we can use machinectl to boot the runner, e.g. machinectl start codeberg-runner.

Launching and registering a new runner is now a matter of importctl, systemd-nspawn, and machinectl, and I can launch a new runner within minutes. We can now spin up an e.g. Alma Linux VM on Hetzner cloud and get a runner up and running within a few minutes, all by hand, even without heavy automation by e.g. terraform, and dispose of it again when the runner is not needed any more which makes for a simple and cost-effective way to do CI on Codeberg.

Build Arch packages with git sources on OBS

2025-11-06T00:00:00Z

Build Arch packages with git sources on OBS

After the infamous xz backdoor incident there's been a general move towards building packages directly from Git sources for the sake of transparency, instead of using upstream provided tarballs.

Now, I like to build my Arch packages on the public openSUSE Build service (OBS) generously sponsored by SUSE, but building packages from Git sources poses a bit of a challenge on OBS: the public instance does not permit network access during builds and requires all sources to be present upfront along with the PKGBUILD. This prevents makepkg from fetching the git sources during build. However, with a small trick we can simply commit a “snapshot” of the source repo along with the PKGBUILD.

We'll build the small wcal utility. We start with a straight-forward PKGBUILD:

$ osc mkpac wcal
$ cd wcal
$ cat >PKGBUILD <<EOF
pkgname=wcal
pkgver=0.1
pkgrel=1
pkgdesc='ISO weekly calendar'
license=('CC0-1.0')
arch=("x86_64")
url='https://github.com/leahneukirchen/wcal'
makedepends=('git')
source=("git+${url}.git#tag=v${pkgver}")

build() {
    make -C "${pkgname}" CFLAGS="${CFLAGS}" LDFLAGS="${LDFLAGS}" PREFIX=/usr all
}

package() {
    make -C "${pkgname}" PREFIX=/usr DESTDIR="${pkgdir}/" install
}

When we run makepkg --geninteg to add the sha256sums:

$ makepkg --geninteg >> PKGBUILD

Then makepkg clones the source URL as a bare Git repository along with the PKGBUILD:

$ ls
PKGBUILD  src  wcal
$ ls wcal
config  description  HEAD  hooks  info  objects  packed-refs  refs

We can just add this bare repository to the package repo:

$ osc add wcal
wcal is a directory, do you want to archive it for submission? (y/n) y
91 blocks
A    wcal.obscpio

This creates an archive containing the bare repository, which we can commit along with the PKGBUILD:

$ osc addremove
$ osc st
A    PKGBUILD
A    wcal.obscpio
$ osc ci -m 'wcal 0.1'
Sending meta data...
Done.
Sending    wcal
Sending    wcal/PKGBUILD
Sending    wcal/wcal.obscpio
Transmitting file data ..
Committed revision 1.

The archive gets uploaded along with the PKGBUILD, and OBS automatically extracts it before running makepkg which then picks up the pre-existing bare repository and uses it as source directory. It still tries to update the repository which fails as the build as no network; this causes a spurious warning in the OBS build logs:

[   26s] ==> Retrieving sources...
[   26s]   -> Updating wcal git repo...
[   26s] fatal: unable to access 'https://github.com/leahneukirchen/wcal.git/': Could not resolve host: github.com

However, we can happily ignore this warning; the package builds successfully.

When updating the package after updating the pkgver in PKGBUILD we run makepkg --geninteg again to update the checksums in the PKGBUILD. This updates the bare repository which we then osc add again; osc warns that the .cpio archive is already under version control but updates it nonetheless.

While not as convenient as using OBS source services to have upstream tarballs fetched automatically it's worth the added build transparency.

Notes on building Arch packages on openSUSE Build Service

2025-07-30T00:00:00Z

Notes on building Arch packages on openSUSE Build Service

I use openSUSE Build Service (OBS) to build most of my packages for Archlinux, sourced from AUR or written by myself, except for a few proprietary packages which OBS does not permit to be built on its infrastructure. There are some caveats, tho, and this post keeps a few notes to work around these.

Set PACKAGER

OBS uses a generated PACKAGER value which effectively identifies the build worker, for instance, i04-ch1b <abuild@i04-ch1b>. There are no direct means to change this, but there's a workaround, albeit a slightly hacky one:

Create a package (say, makepkg-packager) which installs a profile snippet containing export PACKAGER="Jane Doe (OBS) <jdoe@example.com>" at /etc/profile.d/obs-packager.sh (example).
Add the package to Requires: makepkg-packager in the project configuration, via the web interface or osc meta prjconf -e (example).

This effectively sets the $PACKAGER environment variable for the build process of every package in the project, and luckily the Arch build recipe explicitly sources the whole /etc/profile before invoking makepkg. A makepkg drop-in for /etc/makepkg.conf.d won't work, because the Arch OBS build recipe explicitly assembles a custom makepkg file.

The package can be moved to a separate subproject (e.g. "supplemental" or "internal") linked to the main project; this will enable builds from the main project to use the package, but the package won't appear in the repository of the main project.

Disable automated rebuilds

OBS automatically rebuilds packages if dependencies change, but this doesn't work well for Archlinux packages:

Arch packages get updated frequently, which causes excessive rebuilds for packages at the leafs of the dependency tree (e.g. anything depending on gnome-shell will get rebuilt almost daily).
OBS' logic to bump the build number on every automatic rebuild only works for RPM and Debian packages, but does not support Arch's PKGBUILD format.

Automatic rebuilds can be disabled by adding the attribute rebuild="local" to the repository element in the project metadata, via the web interface or osc meta prj -e (example):

   <title>Internal dependencies</title>
   <description></description>
   <person userid="swsnr" role="maintainer"/>
-  <repository name="Arch">
+  <repository name="Arch" rebuild="local">
     <path project="Arch:Extra" repository="standard"/>
     <arch>x86_64</arch>
   </repository>

Resolve multiple providers for dependencies

OBS does not resolve multiple provides automatically. This affects e.g. Rust, which is provided both by rust and rustup; OBS fails to resolve this and needs to be told explicitly which package to prefer via the Prefer setting in the project configuration (example).

Dealing with offline builds

OBS builds packages without network access, so all sources must be available upfront in the package repository. For standard packages with archive sources (i.e. source tarballs) the download_files source service makes OBS automatically download and commit the source archives on every commit. It can be enabled with the following content in the _service file (example):

<services>
  <service name="download_files"/>
</services>

For packages which fetch extra dependencies during build (e.g. NPM, Rust, Go, Maven, etc.) these dependencies need to be vendored upfront and added as an explicit source. The details depend on the package manager being used; for Cargo for instance I usually use cargo vendor, then tar the resulting vendor/ directory and commit it as separate source (example). For go it's similar (example). Generally, it's easy for anything with first-class support for vendoring (e.g. Rust, Go) or which just puts dependencies into a directory in the source tree (e.g. npm). For stuff which doesn't support vendoring and puts dependency artifacts in some other location, it can be quite some work (e.g. Maven, Haskell).

In any case, any PKGBUILD from the AUR which expects to be able to download dependencies during build won't work on OBS; it needs to be adapted, or in some cases entirely rewritten, to account for offline builds.

Simple Secure Boot in Fedora

2024-04-13T00:00:00Z

Simple Secure Boot in Fedora

Fedora doesn't use a proper secure boot setup: It doesn't use unified kernel images and still leaves unsigned initrd files around. Generally, it still seems to consider secure boot support just as means to boot and install on secure-boot locked machines with Microsoft's keys, instead of a proper security tool a user should make use of to really own their own machines.

In this article we'll explore how we can setup secure boot with custom keys. The result works pretty well, and uses no fancy trickery or weird hacks, but is probably still firmly outside of what Fedora supports and will perhaps break with future Fedora releases.

Note: I use Fedora 40 in this article; it may or may not apply to other Fedora 40 versions, both earlier and later. As far as I understand Fedora is actively working on systemd-boot support, UKIs, and secure boot; the whole area is pretty much a moving target and may change a lot in between Fedora releases.

Prerequisites

Before we start let's install necessary tooling:

# dnf install systemd-boot-unsigned systemd-ukify sbsigntools

We install systemd-boot as our bootloader to replace grub, systemd-ukify to build unified kernel images (UKIs) for signing, and sbsigntools to sign EFI binaries for secure boot.

Let's also put sbctl in ~/bin to generate and enroll keys, and to use sbctl verify to check our signatures:

$ curl -LsS https://github.com/Foxboron/sbctl/releases/download/0.13/sbctl-0.13-linux-amd64.tar.gz  | tar xz
$ mkdir -p ~/bin
$ install -m755 sbctl/sbctl ~/bin/sbctl

Finally, we confirm that secure boot is in setup mode:

# bootctl status | grep 'Secure Boot'
systemd-boot not installed in ESP.
   Secure Boot: disabled (setup)

If secure boot is not in setup mode we need to clear the platform key (PK) from the firmware first, via the firmware interface (e.g. systemctl reboot --firmware-setup).

Replace grub with systemd-boot

As a first step we replace the default grub bootloader with systemd-boot because the latter is considerably simpler to set up and can boot from UKIs without further configuration.

As systemd-boot does not need a separate boot partition we first unmount the boot partition and mount the EFI system partition to the standard /efi/ mountpoint:

# umount /boot/efi
# umount /boot
# mkdir -Zm755 /efi/
# sed -i 's_/boot/efi_/efi_' /etc/fstab
# sed -i '/\/boot/d' /etc/fstab
# systemctl daemon-reload
# mount -a

Now we copy systemd-boot to the ESP, uninstall grub and remove its files from the ESP, and enable a utility service which keeps the systemd-boot up to date on the ESP when the corresponding package gets updated:

# bootctl install
Created "/efi/EFI/systemd".
Created "/efi/loader".
Created "/efi/loader/entries".
Created "/efi/EFI/Linux".
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/efi/EFI/systemd/systemd-bootx64.efi".
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi" to "/efi/EFI/BOOT/BOOTX64.EFI".
Random seed file /efi/loader/random-seed successfully written (32 bytes).
Successfully initialized system token in EFI variable with 32 bytes.
Created EFI boot entry "Linux Boot Manager".
# dnf remove 'grub*' --setopt protected_packages=
[…]
# rm -rf /efi/EFI/fedora
# systemctl enable systemd-boot-update.service
Created symlink /etc/systemd/system/sysinit.target.wants/systemd-boot-update.service → /usr/lib/systemd/system/systemd-boot-update.service.

Enable UKIs

Next, we'll reconfigure Fedora to build a unified kernel image instead the traditional pair of a kernel image and initramfs. This creates a single bootable binary to sign for secure boot, instead of leaving an unsigned initramfs around.

Unfortunately, as of 2024-04-12 Fedora 40 patches dracut's kernel install plugin and breaks its UKI support in doing so. We obtain the original plugin and overwrite the one included in the package:

$ curl https://raw.githubusercontent.com/dracut-ng/dracut-ng/main/install.d/50-dracut.install > /tmp/50-dracut.install
$ sudo install -m755 -t /etc/kernel/install.d/ /tmp/50-dracut.install

Now we can enable UKIs, by creating /etc/kernel/install.conf with these contents:

layout=uki
initrd_generator=dracut
uki_generator=ukify

This configuration file changes the kernel installation layout to UKIs, chooses dracut has the initrd generator, and uses ukify to combine the kernel and the initrd to a UKI. While we could use dracut for the latter step as well, in my experiments it failed to take the kernel command line from /etc/kernel/cmdline into account. Besides, ukify has a handy ukify inspect command to inspect the contents of a UKI image, which helps us verify that we build a good image.

We also disable the dracut rescue image because the rescue image setup in Fedora does not yet support UKIs (trying to run the rescue image hook in a UKI setup fails), by creating /etc/dracut.conf.d/50-no-rescue-image.conf with the following contents:

dracut_rescue_image=no

Generate secure boot keys

For secure boot we first need a set of secure boot keys which we generate with sbctl (there are probably also tools for this in Fedora, but sbctl just makes this very easy):

$ sudo ~/bin/sbctl create-keys
Created Owner UUID baa10157-a096-4bba-9b4d-5ae3f2b84895
Creating secure boot keys...✓
Secure boot keys created!
$ sudo ~/bin/sbctl status
Installed:	✓ sbctl is installed
Owner GUID:	baa10157-a096-4bba-9b4d-5ae3f2b84895
Setup Mode:	✗ Enabled
Secure Boot:	✗ Disabled
Vendor Keys:	none

Include the Fedora Secure Boot CA

We also add the Secure Boot Signer CA certificate from Fedora to the secure boot DB which enables us to boot binaries signed by Fedora without having to re-sign them ourselves. Specifically, Fedora's fwupd-efi package comes with a signed EFI binary for fwupd, so firmware updates will just work without further ado if we include the Fedora certificate.

$ sudo dnf install openssl
$ curl https://src.fedoraproject.org/rpms/shim-unsigned-x64/blob/f40/f/fedora-ca-20200709.cer | openssl x509 -inform der -out fedora-ca-20200709.pem
[…]
$ sudo install -m600 -Dt /usr/share/secureboot/keys/custom/db/ fedora-ca-20200709.pem
$ sudo dnf remove openssl

Sign the bootloader

Now we can use our keys to sign the boot loader, and then install the signed binary:

# /usr/bin/sbsign --key /usr/share/secureboot/keys/db/db.key --cert /usr/share/secureboot/keys/db/db.pem /usr/lib/systemd/boot/efi/systemd-bootx64.efi
Signing Unsigned original image
# bootctl install --no-variables
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi.signed" to "/efi/EFI/systemd/systemd-bootx64.efi".
Copied "/usr/lib/systemd/boot/efi/systemd-bootx64.efi.signed" to "/efi/EFI/BOOT/BOOTX64.EFI".
Random seed file /efi/loader/random-seed successfully refreshed (32 bytes).

To keep the signed binary up to date when the package gets updated we amend the systemd-boot-update.service we enabled above to update the signed binary before updating the ESP. To this end we use systemctl edit systemd-boot-update.service --drop-in=10-sbsign to create a new drop-in with the following contents:

[Service]
ExecStartPre=/usr/bin/sbsign --key /usr/share/secureboot/keys/db/db.key --cert /usr/share/secureboot/keys/db/db.pem /usr/lib/systemd/boot/efi/systemd-bootx64.efi

Sign UKIs

Next we configure ukify to sign the generated UKIs, by creating /etc/kernel/uki.conf with the following contents:

[UKI]
SecureBootPrivateKey=/usr/share/secureboot/keys/db/db.key
SecureBootCertificate=/usr/share/secureboot/keys/db/db.pem

Now we re-install all kernels to install them as signed UKIs on the ESP:

root@fedora-40-test:/home/test# kernel-install add-all --verbose
Loading /etc/kernel/install.conf…
layout=uki set via /etc/kernel/install.conf
INITRD_GENERATOR (dracut) set via /etc/kernel/install.conf.
UKI_GENERATOR (ukify) set via /etc/kernel/install.conf.
Loaded /etc/kernel/install.conf.
[…]
Skipping overridden file '/usr/lib/kernel/install.d/50-dracut.install'.
[…]
dracut: *** Creating initramfs image file '/tmp/kernel-install.staging.ooX3PH/initrd' done ***
[…]
KERNEL_INSTALL_LAYOUT and KERNEL_INSTALL_UKI_GENERATOR are good
Using config file: /etc/kernel/uki.conf
+ sbverify --list /usr/lib/modules/6.8.4-300.fc40.x86_64/vmlinuz
+ sbsign --key /usr/share/secureboot/keys/db/db.key --cert /usr/share/secureboot/keys/db/db.pem /tmp/ukij9govtve --output /tmp/kernel-install.staging.ooX3PH/uki.efi
Signing Unsigned original image
Wrote signed /tmp/kernel-install.staging.ooX3PH/uki.efi
[…]
Installing /tmp/kernel-install.staging.ooX3PH/uki.efi as /efi/EFI/Linux/951d91d766c544f2820cb103358d4de2-6.8.4-300.fc40.x86_64.efi
[…]
Installed 1 kernel(s).

Enroll secure boot keys

Now we can verify that everything is signed:

$ sudo ~/bin/sbctl verify
Verifying file database and EFI images in /efi...
✗ /efi/EFI/BOOT/BOOTIA32.EFI is not signed
✓ /efi/EFI/BOOT/BOOTX64.EFI is signed
✗ /efi/EFI/BOOT/fbia32.efi is not signed
✗ /efi/EFI/BOOT/fbx64.efi is not signed
✓ /efi/EFI/Linux/951d91d766c544f2820cb103358d4de2-6.8.4-300.fc40.x86_64.efi is signed
✓ /efi/EFI/systemd/systemd-bootx64.efi is signed

As we can see, all relevant files (the boot loader as well as the kernel image) are signed now (the unsigned files BOOTIA32.EFI, fbia32.efi, and fbx64.efi are left-overs from grub; we can remove these).

Finally, we're ready to enroll our secure boot keys and leave secure boot setup mode:

$ sudo ~/bin/sbctl enroll-keys --custom
Enrolling keys to EFI variables...
With custom keys...✓
Enrolled keys to the EFI variables!

Note that sbctl enroll-keys may abort and very seriously warn you in case it fails to verify the absence of signed option ROMs on your machine. The warning text describes the available options; in case you happen to see it, do read it and take it seriously.

We've now left setup mode, so let's reboot:

# bootctl status | grep 'Secure Boot'
   Secure Boot: disabled
# systemctl reboot

After a reboot secure boot is completely enabled:

$ sudo bootctl | grep 'Secure Boot'
   Secure Boot: enabled (user)
$ sudo ~/bin/sbctl status
Installed:	✓ sbctl is installed
Owner GUID:	baa10157-a096-4bba-9b4d-5ae3f2b84895
Setup Mode:	✓ Disabled
Secure Boot:	✓ Enabled
Vendor Keys:	custom

Configure fwupd

Now that everything works, we should finally configure the firmware updater fwupd for our setup. Normally, fwupd tries to load its EFI capsule (the executable it boots into the install the firmware update on a reboot) through shim to support conventional secure boot setups on systems using Microsoft's keys. However, we don't have shim installed; instead, in our setup we can directly use the signed EFI binary (which is why we included the Fedora CA above). To convince fwupd to not use shim, we update /etc/fwupd/fwupd.conf:

[fwupd]
# use `man 5 fwupd.conf` for documentation

[uefi_capsule]
EnableGrubChainLoad=false
DisableShimForSecureBoot=true

Conclusion

After a few iterations the whole process came out to be a lot simpler than I anticipated, and I now have fully working useful secure boot on my Fedora system. I'm positively surprised on how well Fedora follows systemd upstream without too much of custom tooling and patchery (looking at you, Debian…).

The whole secure boot dance also became a lot simpler in the past years. A few years ago on Arch, sbctl changed the game for secure boot (see Install Arch with Secure boot, TPM2-based LUKS encryption, and systemd-homed , but these days systemd supports this a lot better with ukify, and the remaining gaps are easily filled with good old sbsigntools, to a point that I no longer need sbctl for routine signing of EFI binaries and can solely rely on Fedora packages for this purpose.

That said, I did come across a few oddities in Fedora's packages:

The dracut package patches out UKI support from the kernel-install hook; however this looks like a faulty patch, and probably gets resolved soon. The whole dracut situation is somehwat moving currently with a new upstream emerging after years of stagnation and non-cooperation on the original upstream.
There is a signed fwupd EFI binary, but no signed systemd-boot binary?
The signed fwupd binary is part of fwupd-efi along with the unsigned binary, but for shim the signed and unsigned binaries are split into different packages (and systemd-boot-unsigned suggests this is also planned for systemd-boot)?

But I guess these will resolve soon; the dracut situation will calm down over time, and Fedora actively works to improve UKI support.

Arch Linux rescue image with mkosi

2024-03-24T00:00:00Z

Arch Linux rescue image with mkosi

In this post we'll build a small single-file Arch Linux rescue image for EFI systems.

We will end up with a single EFI executable of about 400 MiB (further optimization can get it down to about 200MiB) which embeds a fully-fledged Arch Linux system. We can then put this image on the EFI partition and sign it for secure boot, which gives us a rescue single-file rescue system to boot into in case the main Arch installation does not boot anymore. From that rescue system we can then chroot into the main installation to repair it.

The image will include metadata which enables systemd-boot to automatically discover the image and add it to its menu when we place in /efi/EFI/Linux (the bootloader specification calls these "type 2 entries", see Type #2 EFI Unified Kernel Images).

You can find my personal version of the image built in this post at https://codeberg.org/swsnr/rescue-image.

I got the idea from a recent post by the mkosi maintainer at https://0pointer.net/blog/a-re-introduction-to-mkosi-a-tool-for-generating-os-images.html which I recommend reading for more information about mkosi. The post calls what we're building here a Unified System Image (USI).

Prerequisites

We'll need a few packages:

# pacman -S mkosi
# pacman -S --asdeps systemd-ukify

mkosi builds the image, and uses systemd-ukify to build a UKI.

Configure mkosi

We configure the image in mkosi.conf:

[Distribution]
Distribution=arch

[Output]
ImageId=archlinux-rescue
Format=uki

[Content]
Hostname=archlinux-rescue
Bootloader=none
Bootable=false
Packages=
    base
    intel-ucode
    linux
    linux-firmware
    wireless-regdb
    iwd
    nano
    less
    mandoc
    man-pages
    arch-install-scripts

select Arch as base distribution for the image,
configure the identifier of the image which also sets the output filenames,
enable the UKI output format,
set a hostname for the image,
and disable bootloader installation (otherwise mkosi would install a somewhat superfluous EFI partition inside the UKI).

We also tell mkosi to install some essential packages into the image. We add

base and a kernel,
firmware binaries, which are essential to boot on any modern hardware,
the regulatory database, which is required for wireless connections,
the iwd daemon to configure wireless interfaces and connect to wireless stations,
the simple nano text editor,
less as pager, and mandoc for the man command (mandoc is somewhat smaller than man-db), and
arch-install-scripts for the arch-chroot command.

Define the systemd preset

We customize the systemd preset to enable the iwd daemon automatically, and disable a few standard systemd services which have no use in a single-user rescue image:

new file mode 100644
index 0000000..4f00b6f
--- /dev/null
+++ b/mkosi.extra/etc/systemd/system-preset/10-rescue-image.preset
@@ -0,0 +1,4 @@
+disable systemd-homed.service
+disable systemd-userdbd.socket
+disable systemd-boot-update.service
+enable iwd.service

mkosi automatically copies the file system tree in the mkosi.extra directory to the image. Hence, our preset file will end up in /etc/systemd/system-preset/10-rescue-image.preset inside the image, where mkosi will pick it up from when it applies the preset as one of its last steps in the build process.

Configure networking

In addition to our custom preset file the default systemd preset applies, which enables systemd-resolved and systemd-networkd and thus provides us with a small yet capable network management stack in the rescue system. However, we still need to configure this stack a bit for proper networking when booting the image on real hardware.

We first put systemd-resolved into the recommended stub mode for the resolv.conf file, to make sure all DNS resolution goes through resolved, by placing the resolv.conf symlink to /run/systemd/resolve/stub-resolv.conf into the mkosi.extra directory (the diff actually describes a symlink in Git):

new file mode 120000
index 0000000..3639662
--- /dev/null
+++ b/mkosi.extra/etc/resolv.conf
@@ -0,0 +1 @@
+/run/systemd/resolve/stub-resolv.conf

From there mkosi copies the symlink itself to the image literally.

By default, systemd-networkd only manages virtual network interfaces of containers and virtual machines, which implies that we automatically have a network connection when testing the image with mkosi qemu. On real hardware however we have real Ethernet and wireless interfaces which we also want systemd-networkd to handle in our rescue image:

new file mode 100644
index 0000000..09fdddf
--- /dev/null
+++ b/mkosi.extra/etc/systemd/network/80-wifi-station.network
@@ -0,0 +1,8 @@
+# SPDX-License-Identifier: MIT-0
+
+[Match]
+Type=wlan
+WLANInterfaceType=station
+
+[Network]
+DHCP=yes
new file mode 100644
index 0000000..0896e7a
--- /dev/null
+++ b/mkosi.extra/etc/systemd/network/89-ethernet.network
@@ -0,0 +1,9 @@
+# SPDX-License-Identifier: MIT-0
+
+# Enable DHCPv4 and DHCPv6 on all physical ethernet links
+[Match]
+Kind=!*
+Type=ether
+
+[Network]
+DHCP=yes

Configure pacman

With networking set up we can now configure pacman. This enables us to install additional software while booted in the rescue image, to handle any kind of recovery task:

new file mode 100644
index 0000000..3bd39de
--- /dev/null
+++ b/mkosi.extra/etc/pacman.d/mirrorlist
@@ -0,0 +1,2 @@
+Server = https://geo.mirror.pkgbuild.com/$repo/os/$arch
+Server = https://mirror.rackspace.com/archlinux/$repo/os/$arch
new file mode 100755
index 0000000..b4db6db
--- /dev/null
+++ b/mkosi.postinst.chroot
@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+echo "Populating pacman keyring"
+pacman-key --init
+pacman-key --populate

We add a mirrorlist using Arch's worldwide geolocating mirrors, to avoid the hassle of setting up a mirror list for the occasional package installation during recovery.

We also initialize the pacman key ring within the image in a post-installation script. mkosi runs this postinst script after package installation, but before configuration and image cleanup. The .chroot extension tells mkosi to run the script while chrooted into the image.

Initialize manpage database

In this post-installation script we also initialize the manpage database now, as mandoc does not do this automatically after package installation:

diff --git a/mkosi.postinst.chroot b/mkosi.postinst.chroot
index b4db6db..03ece97 100755
--- a/mkosi.postinst.chroot
+++ b/mkosi.postinst.chroot
@@ -4,3 +4,6 @@ set -euo pipefail
echo "Populating pacman keyring"
pacman-key --init
pacman-key --populate
+
+echo "Updating manpage database"
+makewhatis /usr/share/man

Now we can use man in the image to read documentation.

Set OS metadata

A UKI image contains release metadata which systemd-boot uses to create a menu entry for the UKI. So far, our image contains the default release metadata of Arch Linux, and thus appears as a regular Arch system in the systemd-boot menu. We add a custom /etc/os-release file to change the identification of the rescue image:

new file mode 100644
index 0000000..4c47e89
--- /dev/null
+++ b/mkosi.extra/etc/os-release
@@ -0,0 +1,6 @@
+NAME="Arch Linux"
+ID=arch
+VARIANT="Rescue Image"
+VARIANT_ID=rescue
+BUILD_ID=rolling
+ANSI_COLOR="38;2;23;147;209"
new file mode 100755
index 0000000..abe3a3e
--- /dev/null
+++ b/mkosi.finalize
@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+echo "Finalizing /etc/os-release"
+source "${BUILDROOT}/etc/os-release"
+echo "PRETTY_NAME=\"${NAME} (${VARIANT} ${IMAGE_VERSION:-n/a})\"" >> "${BUILDROOT}/etc/os-release"

We add a custom /etc/os-release to the image with the mkosi.extra tree, and then use a finalize script, which mkosi runs at the very end of the build process, to generate the $PRETTY_NAME, which systemd-boot uses as the menu label. By generating $PRETTY_NAME dynamically we can include the $IMAGE_VERSION in the name, which mkosi sets from the --image-version argument.

This allows us to build the image with e.g. --image-version=$(git rev-parse --short=10 HEAD)-$(date --utc +%Y%m%d%H%M) to have the git hash and timestamp appear in the menu name, to quickly see how old the rescue image is.

Set a root password

To prevent unauthorized access to the rescue image we set a root password. mkosi reads a plain text or hashed password from the mkosi.rootpw file. We can use openssl passwd to generate a hashed password.

$ touch mkosi.rootpw
$ chmod 600 mkosi.rootpw
$ echo -n hashed: >>mkosi.rootpw
$ openssl passwd -6 >>mkosi.rootpw
Password:
Verifying - Password:

For testing, we can use mkosi -f --autologin qemu to start a VM without having to type the root password.

Build the image

We're now ready to build the image, but before we do so we create two additional directories for mkosi:

$ mkdir mkosi.cache mkosi.output

mkosi writes the generated image to the mkosi.output directory if it exists. We create this directory to move the generated images out of the way, and make it easier to gitignore the build artifacts.

If mkosi.cache exists mkosi will use it for the package manager cache, and re-use the cached package artifacts for subsequent rebuilds which reduces strain on the Arch mirrors a bit and speeds up subsequent builds of the image.

Now, let's build the image:

$ mkosi -f --image-version=$(date --utc +%Y%m%d%H%M%S)
[…]
‣  […]/mkosi.output/archlinux-rescue_20240128082906.efi size is 560.4M, consumes 560.4M.

The image builds successfully. It's fairly large though. To test it with qemu we need to give the VM more memory than the default 2GB mkosi gives to it:

$ mkosi -f --autologin --qemu-mem=6G qemu
[…]
Arch Linux 6.7.1-arch1-1 (ttyS0)

archlinux-rescue login: root (automatic login)

Last login: […] on tty1
[root@archlinux-rescue ~]#

Before we install our image, let's check the metadata:

$ ukify inspect mkosi.output/archlinux-rescue_20240128082906.efi
[…]
.osrel:
  size: 227 bytes
  sha256: 94d2621c5426c1041d463f64c97a41d5d927599d648d4c09438f93f5243b4eb6
  text:
    NAME="Arch Linux"
    ID=arch
    VARIANT="Rescue Image"
    VARIANT_ID=rescue
    BUILD_ID=rolling
    ANSI_COLOR="38;2;23;147;209"
    IMAGE_ID="archlinux-rescue"
    IMAGE_VERSION="20240128082906"
    PRETTY_NAME="Arch Linux (Rescue Image 20240128082906)"
[…]

Our metadata is there; systemd-boot will show this image under the above PRETTY_NAME.

If size is no concern, i.e. if the EFI system partition or the XBOOTLDR partition are sufficiently large to hold this image, we can now install it to the EFI partition and (optionally) sign it for secure boot:

# install -m644 mkosi.output/archlinux-rescue_20240128082906.efi
> /efi/EFI/Linux/archlinux-rescue.efi
# sbctl sign /efi/EFI/Linux/archlinux-rescue.efi

We can however also shrink the image by removing files we do not need.

Shrink the image

Let's first try a few simple cleanups:

index 08d7d01..40ee5a1 100644
--- a/mkosi.conf
+++ b/mkosi.conf
@@ -21,3 +21,9 @@ Packages=
    less
    man-pages
    mandoc
+RemoveFiles=
+    /usr/include/
+    /usr/share/include
+    /usr/share/pkgconfig
+    /usr/lib/**/*.a
+    /usr/share/locale

We remove all headers and pkg-config files, and all static libraries. These files are for compiling programs, and we're quite unlikely to do that in a rescue image. We also remove all locales, because we use C.UTF-8 in the image instead of localized messages.

This gets the image down by a bit:

$ mkosi -f
[…]
‣  […]/mkosi.output/archlinux-rescue.efi size is 528.7M, consumes 528.7M.

To make a real difference, however, we need to shrink the biggest contributors to the size of the image: Kernel modules and firmware. mkosi has builtin support for shrinking the kernel module tree along with firmware:

+KernelModulesExclude=.*
+KernelModulesIncludeHost=true
+KernelModulesInclude=
+    fs/
+    hid/
+    input/
+    usb/
+    dm-.*
+    crypto/
+    tpm/
+    virtio

We first default to excluding all kernel modules: mkosi does not touch kernel modules by default, so excluding all modules is necessary to make the subsequent include rules take effect.

With these include rules we first include all kernel modules currently used on the host system: We'd like to use the image to boot into our hardware, so the set of loaded modules is a good baseline. Then we include a couple of additional driver hierarchies, including all filesystems, all HID, input, and USB devices (to make sure we can plug a USB disk with e.g. an NTFS file system, even if none was plugged into the running system so far). We also include device mapper drivers and the crypto hierarchy to support all kinds of encrypted disks, optionally TPM locked. Finally, we also include all virtio drivers, to still be able to test the image in qemu.

mkosi then removes all modules not matching any of these includes, and also removes all firmware binaries not used by any retained modules.

The resulting image will likely not work on other hardware, but it became a lot smaller:

$ mkosi -f
[…]
‣  […]/mkosi.output/archlinux-rescue.efi size is 244.2M, consumes 244.2M.

Conclusion

I used to keep a rescue image based on GRML around (see GRML on ESP), but stopped doing so once I had a working secure boot setup: GRML ships its initrd and its squash disk image separately, and there were no easy means to combine them into a single image for signing, or sign each part separately.

With mkosi, I finally have a viable alternative which supports secure boot, and allows me to leave my Arch ISO thumb drive at home.

Build Arch Linux packages on OBS

2024-02-22T00:00:00Z

Build Arch Linux packages on OBS

Suse generously sponsors a public build service for packages at https://build.opensuse.org/. Let's try to use it to build Arch packages into a personal repository.

Install osc

The build service works best with a local command line tool called osc. Conveniently, the build service itself offers an Arch repository with binary packages for osc which we can just add to pacman.conf:

[openSUSE_Tools_Arch]
Server = https://download.opensuse.org/repositories/openSUSE:/Tools/Arch/$arch/

We also need to import and trust the signing key for this repo:

# xh 'https://build.opensuse.org/projects/openSUSE:Tools/signing_keys/download?kind=gpg' | pacman-key add -
# pacman-key --lsign-key FCADAFC81273B9E7F184F2B0826659A9013E5B65

Then we can install osc and also python-keyring to let osc store our password in GNOME keyring:

# pacman -S osc python-keyring

Configure repository

Now we can add an Arch repository to our home project on OBS, by navigating to the "Repositories" tab and checking the "Arch Extra" variant under "Arch distributions".

Checkout the home project

For simplicity, we'll just go with the "Home" project which every build service user gets. To start, we check out our home project locally with osc (you'll want to replace swsnr with your OBS username):

$ osc checkout home:swsnr

Your user account / password are not configured yet.
You will be asked for them below, and they will be stored in
/home/basti/.config/osc/oscrc for future use.

Creating osc configuration file /home/basti/.config/osc/oscrc ...
Username [api.opensuse.org]: swsnr
Password [swsnr@api.opensuse.org]:

NUM NAME              DESCRIPTION
1   Secret Service    Store password in Secret Service (GNOME Keyring backend) [secure, persistent]
2   Transient         Do not store the password and always ask for it [secure, in-memory]
3   Obfuscated config Store the password in obfuscated form in the osc config file [insecure, persistent]
4   Config            Store the password in plain text in the osc config file [insecure, persistent]
Select credentials manager [default=1]: 1
done
A    home:swsnr

As it's the first time we're using osc it prompts for authentication.

Create our first package

Now we can add a new package. For simplicity, we'll package my favorite font Vollkorn, because it's dead simple and doesn't involve any complex build system.

First, let's create the package and import the PKGBUILD from AUR:

$ osc mkpac otf-vollkorn
A    otf-vollkorn
$ cd otf-vollkorn/
$ xh 'https://aur.archlinux.org/cgit/aur.git/plain/PKGBUILD?h=otf-vollkorn' > PKGBUILD
$ osc add PKGBUILD
A    PKGBUILD

Let's first edit this PKGBUILD to remove the reference to the changelog which we didn't include, and change the license to a proper SPDX license identifier:

diff --git i/PKGBUILD w/PKGBUILD
index 38de0bc..5cfdd48 100644
--- i/PKGBUILD
+++ w/PKGBUILD
@@ -4,10 +4,9 @@
 pkgname=otf-vollkorn
 pkgdesc="Vollkorn typeface by Friedrich Althausen (OpenType)"
 url='http://vollkorn-typeface.com/'
-license=('OFL')
+license=('OFL-1.1')
 pkgver=4.105
 pkgrel=2
-changelog=ChangeLog.${pkgname}
 arch=('any')

 source=(http://vollkorn-typeface.com/download/vollkorn-${pkgver//./-}.zip)

Next we'll fetch, verify, and commit the font source tarball, because the build service builds packages with network access blocked completely, so all sources need to be committed upfront:

$ makepkg --verifysource
==> Making package: otf-vollkorn 4.105-2 (…)
==> Retrieving sources...
  -> Downloading vollkorn-4-105.zip...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 10.5M  100 10.5M    0     0  6466k      0  0:00:01  0:00:01 --:--:-- 6471k
==> Validating source files with sha256sums...
    vollkorn-4-105.zip ... Passed
$ osc add vollkorn-4-105.zip
A    vollkorn-4-105.zip

Now we can commit the package:

$ osc commit -m 'Upload otf-vollkorn'
Sending meta data...
Done.
Sending    otf-vollkorn
Sending    otf-vollkorn/PKGBUILD
Sending    otf-vollkorn/vollkorn-4-105.zip
Transmitting file data ..
Committed revision 1.

This transfers the package to the build service, which then starts building it. The package page shows the status of the build process, and provides access to the build log.

After the package was built, it moves to the repository and becomes available for download. Note that the build services queues all these operations, so depending on where your package ends up in the queue and how busy the service is, all of these steps may take a while, so even a fast build might take about an hour to finally end up in the repository.

Use the package

To use the package we first need to import the signing keys from the project page, e.g. https://build.opensuse.org/projects/home:swsnr/signing_keys for my home project.

# xh 'https://build.opensuse.org/projects/home:swsnr/signing_keys/download?kind=gpg' | pacman-key add -
# pacman-key --lsign 42D80446DC5C2B66D69DF5B6C1A96AD497928E88

Obviously, you'll want to use the key of your own project here.

We also need to tell pacman about the repository by adding it to pacman.conf:

[home_swsnr_Arch]
Server = https://download.opensuse.org/repositories/home:/swsnr/Arch/$arch/

Then we can finally install our first package from our own OBS Arch repository:

# pacman -Syu
:: Synchronising package databases...
 home_swsnr_Arch                            […]
:: Starting full system upgrade...
 there is nothing to do
# pacman -S otf-vollkorn
resolving dependencies...
looking for conflicting packages...

Package (1)                   New Version  Net Change

home_swsnr_Arch/otf-vollkorn  4.105-2        5,26 MiB

Total Installed Size:  5,26 MiB

:: Proceed with installation? [Y/n]
[…]

Success 🎉

Final words

Using OBS to build Arch packages turned out to be surprisingly simple, but there's a caveat.

The build services builds all packages strictly without network access, something that few PKGBUILDs are prepared for as Arch doesn't have similar restrictions. For traditional C sources that's not much of an issue, but most modern languages like Go, Rust, ECMAScript, etc. have their own dependency manager and routinely try to pull packages from the internet as part of building. Getting PKGBUILDs for packages written in these languages to build on OBS requires some contortions. However, I've managed to get Rust software build on OBS with small modifications to PKGBUILD, but that's for another post.

Generally, you can't expect OBS to just build arbitrary AUR PKGBUILDs; there are other issues, such as OBS failing to resolve alternative dependencies.

But so far I've overcome most of these issues, and certainly plan to use OBS more to build packages for my Arch machines.

Install Arch with Secure boot, TPM2-based LUKS encryption, and systemd-homed

2022-01-06T00:00:00Z

Install Arch with Secure boot, TPM2-based LUKS encryption, and systemd-homed

Update: I no longer use dracut, and the corresponding part of this blog post no longer reflects my setup.

This article describes my Arch Linux setup which combines Secure Boot with custom keys, TPM2-based full disk encryption and systemd-homed into a fully encrypted and authenticated, yet convenient Linux system.

This setup draws inspiration from Authenticated Boot and Disk Encryption on Linux and Unlocking LUKS2 volumes with TPM2, FIDO2, PKCS#11 Security Hardware on systemd 248 by Lennart Poettering.

What this setup does

Authenticate the boot loader, kernel, initramfs, microcode with Secure Boot, using my own custom keys. Nothing can boot which wasn’t signed by my keys.
Everything is either authenticated with my keys (kernel, initramfs, microcode) or encrypted (system partition).
Encrypt the system partition, and unlock it automatically if the boot process was authenticated, by means of a TPM2 key bound to the secure boot state.
Give every user their own dedicated encrypted home directory, which gets unlocked at login and locked again at logout.

What it doesn’t

Show an ugly LUKS password prompt at boot (even with Plymouth it’s not really pretty).
Leave some parts unencrypted and unauthenticated (conventional installations often fail to consider the initramfs).
Ask me twice for my password, once at boot to unlock the disk and then again at login.
Encrypt data of all users with a shared key.

Tools used

Archlinux
systemd >= 250
systemd-boot as bootloader
systemd in initramfs to automatically discover and mount the root filesystem
dracut to generate the initramfs and build signed UEFI binaries
sbctl to create and enroll Secure Boot keys, and sign binaries
systemd-homed to manage user accounts with per-user encrypted home directories
systemd-cryptenroll to add TPM2 and recovery keys tokens to a LUKS partition

The setup

Install the system

We follow the Installation Guide up to and including section “Update the system clock”. Then we partition the disk (/dev/nvme0n1 in our case); we need an EFI system partition of about 500MB and a root partition spanning the rest of the disk. The EFI partition must be unencrypted and have a FAT filesystem; for the root file system we choose btrfs on top of an encrypted partition.

First we partition the disk and reload the partition table; we take care to specify proper partition types (-t option) so that systemd can automatically discover and mount our filesystems without further configuration in /etc/crypttab or /etc/fstab (see Discoverable Partitions Specification (DPS)):

$ target_device=/dev/nvme0n1
$ sgdisk -Z "$target_device"
$ sgdisk -n1:0:+550M -t1:ef00 -c1:EFISYSTEM -N2 -t2:8304 -c2:linux "$target_device"
$ sleep 3
$ partprobe -s "$target_device"
$ sleep 3

Then we setup the encrypted partition for the root file system. We get asked for an encryption password where we pick a very simple encryption password (even “password” is good enough for now, really) to save some typing during installation, as we’ll later replace the password with TPM2 key and a random recovery key:

$ cryptsetup luksFormat --type luks2 /dev/disk/by-partlabel/linux
$ cryptsetup luksOpen /dev/disk/by-partlabel/linux root
$ root_device=/dev/mapper/root

Now we create the filesystems:

$ mkfs.fat -F32 -n EFISYSTEM /dev/disk/by-partlabel/EFISYSTEM
$ mkfs.btrfs -f -L linux "$root_device"

Now we can mount the filesystems and create some basic btrfs subvolumes:

$ mount "$root_device" /mnt
$ mkdir /mnt/efi
$ mount /dev/disk/by-partlabel/EFISYSTEM /mnt/efi
$ for subvol in var var/log var/cache var/tmp srv home; do btrfs subvolume create "/mnt/$subvol"; done

Now we’re ready to bootstrap Arch Linux: We generate a mirrorlist and install essential packages:

$ reflector --save /etc/pacman.d/mirrorlist --protocol https --latest 5 --sort age
$ pacstrap /mnt base linux linux-firmware intel-ucode btrfs-progs dracut neovim

This takes a while to download and installation all packages; afterwards we configure some essential settings. Choose locale settings and the $new_hostname according to your personal preferences.

$ ln -sf /usr/share/zoneinfo/Europe/Berlin /mnt/etc/localtime
$ sed -i -e '/^#en_GB.UTF-8/s/^#//' /mnt/etc/locale.gen
$ echo 'LANG=en_GB.UTF-8' >/mnt/etc/locale.conf
$ echo 'KEYMAP=us' >/mnt/etc/vconsole.conf
$ echo "$new_hostname" >/mnt/etc/hostname

Now we enter the new system and finish configuration by generating locales, enabling a few essential services and setting a root password:

$ arch-chroot /mnt
$ locale-gen
$ systemctl enable systemd-homed
$ systemctl enable systemd-timesyncd
$ passwd root

Still in chroot we now build unified EFI kernel images (including initrd and kernel) for booting and install the systemd-boot boot loader:

$ pacman -S --noconfirm --asdeps binutils elfutils
$ dracut -f --uefi --regenerate-all
$ bootctl install

We do not need to create /etc/fstab or /etc/crypttab; as we assigned the appropriate types to each partition and installed systemd-boot a systemd-based initramfs can automatically determine the disk the system was booted from, and discover all relevant partitions. It can then use superblock information to automatically open encrypted LUKS devices and mount file systems.

At this point we also need to take care to install everything we need for network configuration after reboot. For desktop systems I prefer network manager because it integrates well into Gnome:

$ pacman -S networkmanager

We have finished the basic setup from the live disk now; let’s leave chroot and reboot:

$ exit
$ reboot

After reboot we can complete the system installation, by adding a desktop environment, applications, command line tools, etc.

I like to automate this, and have two bash scripts in my dotfiles, one for boostrapping a new system from a live disk (arch/bootstrap-from-iso.bash) and another one for installing everything after the initial bootstrapping (arch/install.bash).

Create homed user

With the installation finished we create our user account with homectl; let’s name it foo for the purpose of this article. First we should disable copy on write for /home, because this file system feature doesn’t work well with large files frequently updated in place, such as disk images of virtual machines or loopback files as created by systemd-homed:

$ chattr +C /home/

We now create the foo user with an encrypted home directory backed by LUKS and btrfs:

$ homectl create foo --storage luks --fs-type btrfs

By default systemd assigns 85% of the available disk space to the user account, and will balance available space among all user accounts (based on a weight we can configure with —rebalance-weight). On a single user system we may prefer to set an explicit quota for the user account:

$ homectl resize foo 50G

We can also add some additional metadata to the user account:

$ homectl update foo --real-name 'Foo' --email-address foo@example.org --language en_GB.UTF-8 --member-of wheel

man homectl provides a complete list of flags; in particular it also offers support for various kinds of security tokens (e.g. FIDO2) for user authentication, provides plenty of means for resource accounting (e.g. memory consumption) for the user account, and supports different kinds of password policies.

Finally we may run into systemd issues with home areas on btrfs (see below); if login fails with a “Operation on home failed: Not enough disk space for home” message we need to enable LUKS discard:

$ homectl update foo --luks-discard=true

This flag is not safe (heed the warning in man homectl), but until systemd improves its behaviour on btrfs we have no choice unfortunately.

Setup secure boot

First let’s check the secure boot state. We must be in Setup Mode in order to enroll our own keys:

$ sbctl status
Installed:	✓ sbctl is installed
Owner GUID:	REDACTED
Setup Mode:	✗ Enabled
Secure Boot:	✗ Disabled

To enable secure boot we need some keys which we generate with sbctl. For historical reasons sbctl creates these keys in /usr/share/secureboot but plans exists to change this to a more appropriate place (see Github issue 57).

$ sbctl create-keys

Now we tell dracut how to sign the UEFI binaries it builds and rebuild our kernel images to get them signed:

$ cat > /etc/dracut.conf.d/50-secure-boot.conf <

Next we need to sign the bootloader. With -s we ask sbctl to remember this file in its database which later lets us check signatures with sbctl verify and automatically update all signatures with sbctl sign-all. The sbctl package includes a pacman hook which automatically updates signatures when an EFI binary on /efi or in /usr/lib changed. Note that we do not sign the boot loader on /efi but instead place a signed copy in /usr/lib. Starting with systemd 250 bootctl will pick up the signed copy when updating the boot loader. Hence we reinstall the bootloader afterwards to put the signed copy on /efi.

$ sbctl sign -s -o /usr/lib/systemd/boot/efi/systemd-bootx64.efi.signed /usr/lib/systemd/boot/efi/systemd-bootx64.efi
$ bootctl install

We should also do the same for the firmware update to enable seamless firmware updates under secure boot. Again we use -s to remember this file in the sbtctl database:

$ sbctl sign -s -o /usr/lib/fwupd/efi/fwupdx64.efi.signed /usr/lib/fwupd/efi/fwupdx64.efi

Now let’s verify that we have all signatures in place and enroll keys if everything’s properly signed:

$ sbctl verify
Verifying file database and EFI images in /efi...
✓ /usr/lib/fwupd/efi/fwupdx64.efi.signed is signed
✓ /usr/lib/systemd/boot/efi/systemd-bootx64.efi.signed is signed
✓ /efi/EFI/BOOT/BOOTX64.EFI is signed
✓ /efi/EFI/Linux/linux-5.15.12-arch1-1-19ea0ebee1ea4de086128ce1a8e2197b-rolling.efi is signed
✓ /efi/EFI/systemd/systemd-bootx64.efi is signed
$ sbctl enroll-keys

After a reboot we can check the secure boot state again; we’ll see that setup mode is now disabled, secure boot is on, and everything was properly enrolled:

$ reboot
$ sbctl status
Installed:	✓ sbctl is installed
Owner GUID:	REDACTED
Setup Mode:	✓ Disabled
Secure Boot:	✓ Enabled

Enroll TPM2 keys

With the boot process secured we can now configure automatic unlocking of the root filesystem, by binding a LUKS key to the TPM.

We enable the tpm2-tss module in the Dracut configuration, install the dependencies of this dracut module, and regenerate our UEFI kernel images (which will again be signed for secure boot):

$ cat > /etc/dracut.conf.d/50-tpm2.conf <

Now we can enroll a TPM2 token (bound to the secure boot measurement in PCR 7) and a recovery key to our root filesystem. This prompts for an existing passphrase each time. Store the recovery key at a safe place outside of this disk, to have it at hand if TPM2 unlocking ever breaks.

$ systemd-cryptenroll /dev/gpt-auto-root-luks --recovery-key
$ systemd-cryptenroll /dev/gpt-auto-root-luks --tpm2-device=auto

Now reboot and enjoy: The boot process goes straight all the way to the login manager and never shows a LUKS password prompt. The root filesystem is still reasonably secure: The TPM2 key becomes invalid if the secure boot state changes (e.g. new keys are enrolled, or secure boot is disabled), and cannot be recovered if the disk is removed from the system. Consequently only a kernel signed and authenticated with your own secure boot keys can unlock the root disk automatically.

Finally we can wipe the password slots if you like (make sure to have a recovery key at this point):

$ systemd-cryptenroll /dev/gpt-auto-root-luks --wipe-slot=password

If you cannot use secure boot for some reason you can alternatively bind the TPM2 token to a combination of firmware state and configuration and the exact boot chain (up to and including the specific kernel that was started), by specifing the PCR registers 0-5:

$ systemd-cryptenroll /dev/gpt-auto-root-luks --tpm2-device=auto --tpm2-pcrs 0+1+2+3+4+5

This only permits the current kernel and its specific boot chain (e.g bootloader used) to unlock the root filesystem automatically. However this means that we need to reboot and then wipe and re-enroll the TPM2 token after every rebuild of the kernel image… which happens quite often in fact: Dracut updates or configuration changes, kernel updates, systemd updates (due to the EFI shim provided by systemd), bootloader updates, bootloader configuration changes, etc.

Hence I generally recommend to use secure boot if possible in any way.

Issues with this setup

While I am happy with this setup it still has a few drawbacks and issues.

Double encryption

In this setup home directories get encrypted twice, once by homed and then again by the underlying LUKS device. This wastes a bunch of CPU cycles and likely impacts performance a lot, though I haven’t measured the impact and it’s not so bad as to be noticeable in my day-to-day work.

We could optimize this by putting /home/ on a separate partition backed by dm-integrity to authenticate the filesystem (omitting dm-integrity and using a plain file system leaves an attack vector, because linux cannot securely mount untrusted file systems). This setup requires at least systemd 250 or newer, because earlier versions do not support dm-integrity well. With systemd 250 we can setup a HMAC-based integrity device, put the HMAC key on the rootfs (e.g. /etc/keys/home.key) and register the home partition in /etc/integritytab with that key, and then mount it via /etc/fstab.

However, this has a few issues on its own, because dm-integrity has a few design issues and is and nowhere near LUKS/dm-crypt:

There’s no key management like LUKS for dm-crypt, meaning we can’t use passphrases or TPM2 keys for dm-integrity devices; instead we need a key file somewhere on disk.
Unlike LUKS/dm-crypt devices dm-integrity devices aren’t self-describing, because the superblock for dm-integrity doesn’t even contain the algorithm used (see <https://github.com/systemd/systemd/pull/20902). We cannot mount a dm-integrity device without some extra configuration, and worse, getting the configuration wrong can silently corrupt the device.
For these reasons, DPS cannot and does not support dm-integrity partitions, so we need to configure the whole home partition mount, from dm-integrity up to /etc/fstab.

Tooling issues

There are also multiple issues with current tooling that require some more or less safe workarounds:

At the time of writing systemd-home has issues with resizing LUKS home areas on btrfs filesystems, apparently due to fallocate() idiosyncrasies in btrfs. This issue prevents users from logging in, see systemd issues 19398 and 20960, an Arch forums post, and a mail on the systemd-devel list. A workaround is to enable online discard for the user, but this flag is unsafe because it allows overcommitting disk space, which results in IO errors that the kernel and application are not well prepared to handle (which comes down to potential data loss). Keep frequent backups if you need this.

Compose emojis

2021-11-06T00:00:00Z

Compose emojis

On Linux you can define custom sequences for the Compose key. You just need to create a ~/.XCompose file and can start to define new sequences for e.g. emojis:

include "%S/en_US.UTF-8/Compose"

<Multi_key> <period> <p> <r> <a> <y> : "🙏"
<Multi_key> <period> <less> <3> <parenright> : "😍"
<Multi_key> <period> <less> <3> <period> : "❤️"
<Multi_key> <period> <less> <3> <asterisk> : "😘"

man 5 Compose documents the format, though Gtk doesn’t seem to support all of it: It doesn’t handle includes apparently, and always seems to include its own hard-coded list of compose sequences.

There is a nice Gist with some sequences.

The mysterious disapperance of Docker images

2021-08-09T00:00:00Z

The mysterious disapperance of Docker images

A node hosts a Gitlab runner and a small k3s cluster which runs a few services as regular kubernetes deployments. A CI job pinned to that runner builds Docker images for these services services, updates the image of the corresponding deployments, and starts a few system and acceptance tests. The CI job does not push those images to the in-house registry; to avoid polluting the registry with hundreds of images it just builds locally.

Each test then scales each deployment to zero replicas to effectively stop all services, clears the system’s underlying database, and scales the service deployments back to a small number of replicas sufficient for testing.

The whole thing runs fine until one day the replicas randomly fail to start.

The first symptom is that the tests time out while waiting for replicas to start. The services remain unreachable after system reset. Inspecting the deployment reveals that the Docker image doesn’t exist:

$ kubectl describe pod foo-68fbd8c7c5-8grqd
Name:         foo-68fbd8c7c5-8grqd
[…]
Events:
  Type     Reason     Age                   From               Message
  ----     ------     ----                  ----               -------
  Normal   Scheduled  13m                   default-scheduler  Successfully assigned default/foo-68fbd8c7c5-8grqd to node.example.com
  Normal   BackOff    11m (x6 over 13m)     kubelet            Back-off pulling image "registry.example.com/foo/foo:1.2.3-64-affa5225"
  Normal   Pulling    11m (x4 over 13m)     kubelet            Pulling image "registry.example.com/foo/foo:1.2.3-64-affa5225"
  Warning  Failed     11m (x4 over 13m)     kubelet            Error: ErrImagePull
  Warning  Failed     11m (x4 over 13m)     kubelet            Failed to pull image "registry.example.com/foo/foo:1.2.3-64-affa5225": rpc error: code = Unknown desc = Error response from daemon: manifest for registry.example.com/foo/foo:1.2.3-64-affa5225 not found: manifest unknown: manifest unknown
  Warning  Failed     3m11s (x43 over 13m)  kubelet            Error: ImagePullBackOff

Docker doesn’t find the image in the registry because it was never pushed, but it shouldn’t even try to ask the registry: The image was built locally and should already exist on the node. So why is it gone? Perhaps a docker build failure?

However a docker image ls thrown into the pipeline before the test reveals that the image builds fine; in fact it’s there minutes before the test starts and then randomly disappears while the tests are running. Clearly something’s deleting images from the Docker daemon.

Some careful search for the right keywords (“docker images disappear” wasn’t good enough) reveals a Stack Overflow answer pointing to Kubernetes’ own garbage collection which runs every minute, and, under disk space pressure, removes unused Docker images. And indeed, on the affected node kubelet wanted to delete even more:

[…] node.example.com k3s[…]: E0809 […] Image garbage collection failed multiple times in a row: wanted to free 311382016 bytes, but freed 573253440 bytes space with errors in image deletion

Why does kubelet feel pressured to delete images? And why just about 500MiB? That’s not a lot, by Docker’s standards.

Turns out, the system had a very very small /var/lib/docker:

$ dh -h
Filesystem      Size  Used Avail Use% Mounted on
[…]
/dev/sda6       7.0G  2.3G  4.8G  33% /var

Mystery solved: That’s nowhere near enough to satisfy docker’s demand for space, and explains why kubelet feels so much pressure to delete images in almost every build.

As a stop-gap measure all images were pruned; later the day /var/lib/docker was given much more space. The tests haven’t failed since.

Sebastian Wiesner

Proton Pass as 1Password alternative?

Proton Pass as 1Password alternative?

What I like ¶

What I miss ¶

Importing data from 1password ¶

Tags ¶

Per-device registration ¶

Other nuisances ¶

Summary ¶

forgejo-runner image with mkosi

forgejo-runner image with mkosi

Base image ¶

Disk layout ¶

Runner configuration ¶

Register runner from credentials ¶

Build and launch ¶

Build Arch packages with git sources on OBS

Build Arch packages with git sources on OBS

Notes on building Arch packages on openSUSE Build Service

Notes on building Arch packages on openSUSE Build Service

Set PACKAGER ¶

Disable automated rebuilds ¶

Resolve multiple providers for dependencies ¶

Dealing with offline builds ¶

Simple Secure Boot in Fedora

Simple Secure Boot in Fedora

Prerequisites ¶

Replace grub with systemd-boot ¶

Enable UKIs ¶

Generate secure boot keys ¶

Include the Fedora Secure Boot CA ¶

Sign the bootloader ¶

Sign UKIs ¶

Enroll secure boot keys ¶

Configure fwupd ¶

Conclusion ¶

Arch Linux rescue image with mkosi

Arch Linux rescue image with mkosi

Prerequisites ¶

Configure mkosi ¶

Define the systemd preset ¶

Configure networking ¶

Configure pacman ¶

Initialize manpage database ¶

Set OS metadata ¶

Set a root password ¶

Build the image ¶

Shrink the image ¶

Conclusion ¶

Further reading ¶

Build Arch Linux packages on OBS

Build Arch Linux packages on OBS

Install osc ¶

Configure repository ¶

Checkout the home project ¶

Create our first package ¶

Use the package ¶

Final words ¶

Install Arch with Secure boot, TPM2-based LUKS encryption, and systemd-homed

Install Arch with Secure boot, TPM2-based LUKS encryption, and systemd-homed

What this setup does ¶

What it doesn’t ¶

Tools used ¶

The setup ¶

Install the system ¶

Create homed user ¶

Setup secure boot ¶

Enroll TPM2 keys ¶

Issues with this setup ¶

Double encryption ¶

Tooling issues ¶

Compose emojis

Compose emojis

The mysterious disapperance of Docker images

The mysterious disapperance of Docker images

What I like

What I miss

Importing data from 1password

Tags

Per-device registration

Other nuisances

Summary

Base image

Disk layout

Runner configuration

Register runner from credentials

Build and launch

Set PACKAGER

Disable automated rebuilds

Resolve multiple providers for dependencies

Dealing with offline builds

Prerequisites

Replace grub with systemd-boot

Enable UKIs

Generate secure boot keys

Include the Fedora Secure Boot CA

Sign the bootloader

Sign UKIs

Enroll secure boot keys

Configure fwupd

Conclusion

Prerequisites

Configure mkosi

Define the systemd preset

Configure networking

Configure pacman

Initialize manpage database

Set OS metadata

Set a root password

Build the image

Shrink the image

Conclusion

Further reading

Install osc

Configure repository

Checkout the home project

Create our first package

Use the package

Final words

What this setup does

What it doesn’t

Tools used

The setup

Install the system

Create homed user

Setup secure boot

Enroll TPM2 keys

Issues with this setup

Double encryption

Tooling issues