Fixing broken boot from encrypted devices after upgrade to Debian Bookworm

Fixing broken boot from encrypted devices after upgrade to Debian Bookworm

My workstation uses LUKS to encrypt the underlying filesystem on two SSDs (NVMe) devices, atop of which I have lvm2 running to create a big filesystem, which worked nicely the last couple of years (more details will follow below). But after the upgrade from Debian 11 (Bullseye) to Debian 12 (Bookworm) my system refused to boot.

More concretely, I was greeted like always with the prompt to decrypt my root device, which was also working as expected — i.e. I received the “setup device successfully” message from the initramfs. However, the boot failed to mount the resume / swap device and then gave up completely with the information that it couldn’t mount the root device and the lovely message:

ALERT! /dev/mapper/pixie--vg-root does not exist.

Debug attempts

Initramfs debug shell

Searching the web let to a relevant looking thread on forums.debian.net, which gave some initial ideas. In particular, I could make some progress using these two commands:

  vgchange -ay
  vgchange -ay --activationmode partial

Afterwards, my /dev/mapper/ showed the expected devices. However, I couldn’t mount them, hence also couldn’t follow the further advice there.

What is clear to me now but wasn’t at the time is that I should have exited the initramfs emergency shell (via exit) and tried to continue to boot the system. This likely would have worked and allowed me to debug and fix the issue from the machine directly.

Rescue mode from USB stick

Instead, I created a USB stick with the netiso installation image for Debian 12 and booted from the stick into rescue mode (in the advanced options). This nicely detected my hardware, and then prompted me for each of my SSDs to decrypt them.

That’s when I figured out that of course both of my NVMe devices were encrypted separately and not even using the same password. I had forgotten this, but my password store contained of course both passwords, so I could decrypt both devices.

Fixing the problem

Wrong device name

I had read in a reddit thread that Debian uses a modified crypttab(5) format, adding an option initramfs to identify the root device, resume devices and others which needed handling in the initramfs phase of the boot. This option was missing, so I added this to the entries in my crypttab and tried to update the initramfs via update-initramfs -u.

Unfortunetely, this threw an error about a source mismatch for one of my devices and also gave a warning that the cipher method for decryption could not be figured out. After some while, I realized that my /etc/crypttab/ specified nvme0n1p3_crypt as the device but that it should really be nvme1n1p3_crypt (note nvme0 vs. nvme1). I’m not sure if this error had been in there all the time (and if so, why did it work before) or if something changed with Bookworm, triggering a device name change.

After correcting this error, I could finally update the initramfs and rebooted. Unfortunately, I ran into the same problem on boot.

Key file not accessible

This time while debugging in the initramfs emergency shell (and also searching for hints on the web), it became clear to me that my other device nvme0n1p1 was the actual culprit. In particular, I was using a key file (stored in /etc/keys/nvme0n1p1.luks) to decrypt this device automatically, which is also the reason I had forgotten about the two differing passwords for the devices.

So, I edited the crypttab once more to no longer reference the keyfile. So, I changed /etc/crypttab from:

nvme1n1p3_crypt UUID=[elided] none luks,initramfs
nvme0n1p1_crypt UUID=[elided] /etc/keys/nvme0n1p1.luks luks,initramfs

to the new version:

nvme1n1p3_crypt UUID=[elided] none luks,initramfs
nvme0n1p1_crypt UUID=[elided] none luks,initramfs

After another update-initramfs -u -k all, I could finally again boot into my system. Of course, I now have to give both passwords on boot.

I’m not sure if this is a recent change, but apparently now you need to specify a keyfile pattern in /etc/cryptsetup-initramfs/conf-hook. So, I now have the following in this file:

root@pixie:/etc/cryptsetup-initramfs# grep ^KEYFILE_PATTERN conf-hook
KEYFILE_PATTERN=/etc/keys/*

And then reverting the change to the crypttab above, re-running the update-initramfs makes the automatic decryption for the second device work again.

Categories: Linux
Defined tags for this entry:

Who Am I and If So How Many? Using multiple Firefox profiles.

So, I’ve changed my Firefox setup quite a bit over the last months. I now work with multiple profiles, e.g. one for online banking, one for social media use and one for development purposes and one for “default” use (i.e. pretty much else). I’m quite happy with the experience.

Usually I have all three profiles running in parallel, showing on different virtual desktops of my linux box. I.e., on desktop 2, I usually have my development environment open, so I’ll open the “development” browser there, desktop 4 runs all the applications for interacting with other people (mail etc.), so “social” goes there and the default one will show on virtual desktop 5.

Why would you do this?

One direct effect is that the amount of open tabs in any given browser window is a lot smaller and also better grouped than before. I.e., any web page I need for my current development project I will only open in my “development” profile. I will only open my bank’s page in the banking profile, so this browser window will never show anything else.

The other main benefit is that I can have profile specific configurations. E.g., I nail down my “default” profile with NoScript which is not really useful on my “development” profile, whereas I don’t need e.g. the React Dev tools on the “social” or “default” profile.

Dedicated profiles can also help with security, e.g., using a dedicated profile can lower the attack surface for online banking: When you don’t browse to other sites with the same browser/profile, any XSS/CSRF issue on these “other” sites for sure can’t affect your online banking connection.

The profile I use exclusively for online banking is also highly locked down and in addition uses a different theme, so that it is visually obvious that I’m working with this profile.

Get me started, please

To start using multiple profiles, you have to run Firefox with the -P switch, which will start the Profile Manager that allows you to create new profiles, delete profiles etc. Alternatively, if you have Firefox already running, browsing to about:profiles will also allow you to manage your profiles.

For a while, I just started the non-default browsers manually over the command line, by just opening an xterm and running firefox --no-remote -P social &. But I finally created some additional local .desktop files (cf. the Arch wiki page on xdg desktop files), so I can start Firefox from the desktop. I.e., I added a file $HOME/.local/share/applications/socialbrowser-usercreated.desktop with the following content:

[Desktop Entry]
Name=Social Firefox
Comment=Browse the World Wide Web
Comment[de]=Im Internet surfen
Exec=/usr/lib/firefox-esr/firefox-esr --no-remote -P social %u
Terminal=false
X-MultipleArgs=false
Type=Application
Icon=firefox-esr
Categories=Network;WebBrowser;
MimeType=text/html;text/xml;application/xhtml+xml;application/xml;application/vnd.mozilla.xul+xml;application/rss+xml;application/rdf+xml;image/gif;image/jpeg;image/png;x-scheme-handler/http;x-scheme-handler/https;
StartupWMClass=Firefox-social
StartupNotify=true

This will then create a menu entry in the “Internet” submenu in my application starter menu in my desktop environment (because that’s where the given categories will create entries).

Any drawbacks?

One annoying thing is the profile selection that Firefox pops up when you don’t specify a profile on the command line. If you select the option “Use the selected profile without asking at startup”, then it will not be easy anymore to use a different profile — the only way then is really to use -P again.

This default profile selection becomes a problem especially when you want to open a link from a different application (eg. from your mail program), because then you can’t decide which of the running browsers/profiles will open the link, it will always try to use the default one. I have seen varying behavior what Firefox does when you don’t select a default profile: sometimes it just picks one running profile successfully, but I’ve also seen it opening the profile selection dialog again. In that case, if you select a profile that is already in use, Firefox will handle it like an attempt to open up the profile a second time, resulting in an error. My current workaround to this particular issue is to set the default browser to Chromium via xdg-settings set default-web-browser chromium.desktop.

The other hassle working with multiple profiles is bookmark management, as I want some bookmarks only local to one profile but most should be shared. I can use Pocket for the shared ones, of course. However, I often just copy&paste the URL manually to the “default” browser which serves as the main bookmark keeper. I really should move away from this completely and instead use the bookmark extension of my Nextcloud installation.

Overall, for me the benefits clearly outweigh any drawbacks.


Threadripping under Debian

Getting a Threadripper machine to work under Debian

After a long long time of more than eight years my old machine started showing hardware problems: first the power supply failed and had to be replaced. Next the CMOS battery died. It became clear that finally a replacement would be in order. When the first reports about AMDs Ryzen family came out and in particular with the Threadripper tests, I became interested. In the end, I waited until the Threadripper 2950X was available before ordering a custom-built machine from a local vendor. Here is the basic hardware setup:

  • CPU: Threadripper 2950X (32 cores)
  • Mainboard: MSI 399X Carbon
  • Memory: 32 GB
  • GPU: Nvidia 1060
  • SSD: Samsung 970 Evo NVM 1TB

This post will describe what I had to do to on the software / linux side to bring the machine to a usable state.

Basic installation

As a Debian user of old and knowing that the latest Debian stable (Stretch) would not provide a recent enough kernel, I installed Debian Buster. This went more or less flawlessly. However, I can’t help but note that the installation of lvm and disk encryption (ie. lvm2 with lukssetup) provided by the Debian installer is really bad. It hasn’t improved one bit since ages: the automatic partitioning will come up with a crazy sizing scheme and there is absolutely zero useful support when you go manual. It’s easy to end up with an installation which looks like it succeeded but rebooting will end up on the rescue console because mounting the root volume group fails.

The only other noteworthy thing is that I directly installed the proprietary Nvidia drivers because the free nouveau driver doesn’t really support any of the “newer” and more advanced 10X0 cards from Nvidia. As I am a LXDE user, that was of course the desktop environment I installed. However, I installed LXQt and KDE along with it, to also learn about their current state, but went quickly back to LXDE — LXQt is supposed to be the successor of LXDE but it’s currently not really in a comparable state.

I also installed Gnome briefly, only to discover that it’s trying to use Wayland which apparently has problems with the Nvidia drivers. I couldn’t make it work and as I was never a big Gnome fan anyway, I simply threw out Gnome again.

system-udev crashes and kernel compilation

With the system up and running, the first problem I noted was system-udev constantly crashing. This also prevented suspend/hibernate from working. A longer internet search finally revealed this system-udev bug report in Red Hat’s bugzilla. Apparently “the BIOS/firmware is advertising it supports SEV, when in fact it doesn’t” where SEV is AMD’s secure encrypted virtualization technology. If the kernel config option CONFIG_CRYPTO_DEV_SP_PSP is set to yes then the kernel would use it and apparently most distribution provided kernels are setting it — the one from Debian Buster (4.18.0-2 at the time) surely does.

There are apparently three ways to fix this issue:

  • wait until the BIOS/firmware is fixed to no longer provide the wrong information,
  • wait for a newer kernel to provide a workaround,
  • or compile the current kernel with CONFIG_CRYPTO_DEV_SP_PSP set to no.

Apparently, since 4.19-rc5 the kernel has a workaround for the issue, but at the time of writing Buster and Sid only have 4.18. So, off to compile a new kernel without that setting. This turned out to have become a lot more complicated since the last time I had to do this on a Debian system (which was still using make-kpkg at the time, so yes, that’s several years I didn’t had the need to compile a kernel). It also doesn’t help that quite a bit of documentation out there is outdated — the Debian Kernel handbook seems to be the proper documentation.

Unfortunately, it’s still easy to get it wrong. I ran into issues with certificates (why exactly you would need certificates to compile a kernel is beyond me, my wild guess is it’s related to “secure” boot) or my changes to the configuration were overwritten during the process and other issues. In the end, the following process “works for me”:

    # cf. chapter  4.2.3, generate the setup for amd64
    make -f debian/rules.gen setup_amd64_none_amd64

    # fire up the config dialog, now enable RCU-BOOST and disable PSP
    make -C debian/build/build_amd64_none_amd64 xconfig

    # build the kernel
    make -f debian/rules.gen binary-arch_amd64_none_amd64

The only problem with this is that the generated kernel will have the same numbering as the official package. So, a newer minor version of the Debian package can overwrite your manually build package. I haven’t really looked into this yet, but will update this section once I do (and yes, the process described in section 4.3 of said manual didn’t work for me).

Btw., don’t expect kernel compilation to be a fast process. Apparently, the kernel configuration that is used out-of-box for Debian compiles half the world and consequently this takes ages even on such a high-end machine.

The newly compiled kernel fixes the systemd-udev crashes and then also suspend / hibernate worked. However, when I triggered hibernate from the LXDE logout dialog, the machine wouldn’t power-off. Another long round of searching the web, including reading the source to lxsession-logout.c, revealed the solution to disable upower via systemctl disable upower.

Random hangs

During my first attempt to compile a kernel, my system just hang. Completely, not even the magic sysreq worked anymore. That’s apparently another known issue, but there are no good solutions. The attempt to set RCU_NOCB_CPU und RCU_BOOST is one such attempt (cf. the soft hang discussion in the AMD forum, but this didn’t really help me (cf. also the random soft lock discussion on the kernel bugtracker). However, the also linked ZenStates github repo contains a Python script which disables the C6 state.

Again, the forum suggests that the issue might be fixed with newer BIOS versions, but my mainboard has the mentioned AGESA version and the issue still occurred. Disabling C6 state per the script fixes the problem, but results in higher energy consumption and is hence not exactly a perfect solution either. If you want to run zenstates automatically, there is also a systemd template for zenstate. Note that this needs some additional tweaking (which I didn’t get around to yet) to run modprobe msr.

PCI errors

One other thing I noted in the logs were re-occuring PCI errors. There are a number of suggested fixes, cf. this PCI error answer on askubuntu to use pci=nomsi or pci=noaer or this other PCI error suggestion on unixstackexchange to use pci=nommconf. For me, pci=noaer hides the errors successfully and for the moment that’s good enough (read: I haven’t investigated whether the other suggestions would actually fix the issue also).

Virtualization

The latest thing I ran into was that Virtualbox was refusing to start a virtual machine, claiming the AMD feature would be disabled in the BIOS. This turned out to be the case, SVM was disabled. Actually, I couldn’t easily find the option in the first place, I had to use the search in the Bios.

Two things I haven’t tried out yet

The two things I haven’t used yet are the goodies the motherboard provides over the one I originally wanted: Bluetooth and Wlan. I did install the intel firmware to support both features, but can’t really say anything more about it yet.

Conclusion

I don’t have a conclusion just yet. It’s clear this new machine is a lot faster than my old box, but given the Core2Duo cpu is also quite long in the teeth now and that the old one didn’t had a SSD, that’s to be expected. Although this article has gotten longer then I would have hoped for, overall the machine is running quite well (with me running a testing version of Debian which is usually really badly supported). In particular, I can’t say I ran into any issues with the peripherals so far. I guess the machine will have to show it’s power over the next months and hopefully there are some more fixes to BIOS/firmware and the kernel as needed.


Running Linux on Dell systems

Dell is selling Ubuntu equipped systems since about a year now and seems to be quite happy with it. Whatever that effectively means, at least I can tell that I’m quite happy with Linux on Dell systems, too.

Through the last five years, I’ve been using Linux on a number of Dell systems. Under my personal control there have been three laptops (Dell C610, D610 and a Latitude 640) and a desktop (Optiplex 755), on which I have been running Debian Sarge, Ubuntu Dapper, Feisty and now Hardy. We also had several Dell servers at work running more or less smoothly with Debian (sarge, etch). Using Linux wasn’t always without problems: I had trouble with built-in modems, PCMCIA ISDN cards and acpi/hibernation. For example, on my private Latitude 640, I have trouble suspending at all, because of the ipw3945 driver for the wlan. But the important thing to note is that basically all problems were really small and never of a size requiring me to use some other OS in the first place.

The only real issue is not with Dell per se, but more with my favourite OS, Debian: over the years, and especially with the ever-lasting sarge release, getting Debian to run on a recent system got more and more difficult. That’s the main reason why I’ve been using Ubuntu on all recent hardware I had contact with: it’s more or less (more so than less) a Debian system but does run on modern hardware. Main issues here were graphics adapters, sata/scsi hostadapters and network/wifi cards, or to put it otherwise: too old kernels, too old X.org. Both problem sources can simply be solved by using a recent version of Ubuntu. Sorry, Debian, but your release cycle is just too long to be acceptable. Granted, all these problems are mostly an issue when installing a new system, but it’s not always possible to plug in some old disc with a working version of Linux.


Page 1 of 1, totaling 4 entries