You are not logged in.
I've just managed to recover my boot by rolling back to those packages before Tuesday's update.
Those packages are:
glibc-2.31-1.0-pentium4.pkg.tar.zst
imagemagick-7.0.9.22-1.0-pentium4.pkg.tar.zst
libgee-0.20.3-1.0-pentium4.pkg.tar.zst
linux-firmware-20200204.b791e15-1.0-any.pkg.tar.zst
man-pages-5.05-1.0-any.pkg.tar.zst
wpebackend-fdo-1.4.1-1.0-pentium4.pkg.tar.zst
linux-api-headers-5.4.17-1.0-any.pkg.tar.zst
autoconf-2.69-7.1-any.pkg.tar.zst
hwids-20200204-1.0-any.pkg.tar.zst
ca-certificates-mozilla-3.50-1.0-pentium4.pkg.tar.zst
nss-3.50-1.0-pentium4.pkg.tar.zst
python-fastimport-0.9.8-1.0-any.pkg.tar.zst
python-patiencediff-0.1.0-1.0-pentium4.pkg.tar.zst
breezy-3.0.2.3-2.0-pentium4.pkg.tar.zst
libusb-compat-0.1.7-1.0-pentium4.pkg.tar.zst
pango-1:1.44.7+11+g73b46b04-1.0-pentium4.pkg.tar.zst
python-gmpy2-2.1.0b4-1.0-pentium4.pkg.tar.zst
python2-pycryptodome-3.9.5-1.0-pentium4.pkg.tar.zst
python-configobj-5.0.6-6.1-any.pkg.tar.xz
python-certifi-2019.11.28-1.0-any.pkg.tar.xz
python-dulwich-0.19.15-1.0-pentium4.pkg.tar.zstThe errors I was getting began with [FAILED] Failed to start uDev Kernel Device Manager
After that various other things started to go wrong, after my partitions mainly failed to mount.
I tried rolling back linux-firmware and hwids, but linux-firmware rollbacks spit out errors about writing microcode (as did the glibc downgrade, I'll note), but the hwids rollback alone didn't allow my system to boot, and after wasting a day to do that I just decided to splat them all and see if that helps. I also did an update earlier today and that didn't help, which I've since rolled back.
If I had to guess next what might have broken the system I might suspect libusb-compat has broken things, especially because I'm running most of my partitions off a USB card inserted into an internal reader which seems to be connected via USB according to lsusb outputs, but for the time being I'm taking a breather to be able to post this.
Architecture: pentium4, Testing repos: Yes, Hardware: EeePC 901+2GB RAM+OS half on the SD card.
Offline
Try upgrading everything except glibc (should have been removed from the repositories, so "pacman -Syu" should do) - we are currently having issues with that.
regards,
deep42thought
Offline
Okay, thanks for the tip. Upgrading everything from Tuesday except glibc has worked. I've even now managed to update to my Thurday packages. Currently updating to latest is waiting for me to free up enough space on my /usr partition. Seems things I actually use have grown rather, and my /usr partition is limited by the 4GB internal card on this eeepc.
Now that I check there's no version of glibc-2.31 in testing any more, which I guess is where I got this duff version from.
Architecture: pentium4, Testing repos: Yes, Hardware: EeePC 901+2GB RAM+OS half on the SD card.
Offline
Yep, I just wiped one of my real laptops with a glibc 2.31 for testing. Looks like not only systemd-spawn is
completly borked with new glibc 2.31, but also systemd, udev, journal, etc. So far I'm suspecting some
incompatibility of the way glibc 2.31 implements 64-bit time_t with the help of kernel syscalls which might
be missing on a 32-bit kernel. Especially if jailed with libseccomp things go really wrong, so containers see
the first 64-bit syscall to fail with ENOSYS and might interpreted it as EPERM. No mather how many -
-system-call-filter= I added to systemd-nspawn I added, it made no difference.
Furthermore:
- sshd has syscalls to the new time functions and seccomp mitigation, so it needs patching the libseccomp support in sshd
- systemd (and systemd-nspawn) ditto.
- more to follow
For now we stick to glibc 2.30 for all build slaves and for stable.
Offline
Disabling -Dseccomp=false in 64-bit system makes systemd-nspawn work, so the bug
is in systemd.
How can this break pure i686 systems?
Offline
Ah. Seccomp support is also broken for IA-32 and glibc 2.31 for journalling and the rest of systemd. I'm pretty sure
if you compile a full systemd with -Dseccomp=false and boot from it a glibc 2.31 based system, things will just
work. I start to have the same bad feeling about seccomp as about SELinux. It's the wrong hammer for the wrong
nail (or architectural coffin): adding security layers on top of systems (in this case Unix), which were not meant
to run under this type of security requirements, and pushing all the burden to get the seccomp filter calls right into userland,
is just error-prone. Also imagine a C-programmer doesn't want to care about which syscalls a C-library is doing (none,
or one, or one and a fallback?) just to write the right seccomp filters. This is the concept of layered software.
The problem will be, that systemd is just to most severe package affected, we saw already ssh and file/libmagic
suffer from ill-places or missing secomp-filters. And as seccomp is heavily pushed into Linux currently, the situation
is going to only get worse.
We cannot force upstream to disable seccomp for their systemd, just that we can build on our 64-bit build slaves,
that's another point.
It also shows another weakness in current Linux: glibc people, kernel people and systemd people don't work together.
Also everything but 64-bit Intel is left by Redhat/IBM/Intel/Google to be fixed by hobbiist and volunteers, it seems.
Offline
FWIW, since leaving glibc at 2.30 my system seems to be working in the main. I don't use sshd on this system commonly, but I do regularly connect as a client to my server using ssh, so I can say that openssh isn't broken for my uses, even if using it as a daemon might be.
But yeah, this has always to be knowledge been a hobbyist led project; I don't think any of us are getting paid for this. While some software providers do still provide ia32 binaries, that doesn't help for anyone who wants a build from source, plus your ia32 might not be my ia32 with some including sse4 and some topping out at ssse3 or lower for example.
Architecture: pentium4, Testing repos: Yes, Hardware: EeePC 901+2GB RAM+OS half on the SD card.
Offline
openssh 8.2 doesn't allow me to log in anymore, this seems to be an upstream issue, as I also have this now with 64-bit Archlinux!
As I can login to other Linux distros, I guess something is roally borked in sshd itself:
ssh user@somewhere
kex_exchange_identification: read: Connection reset by peer
Furthermore I start to miss packages in /var/cache/pacman/pkg after an update, new packages don't land there anymore!
I guess, ssh 8.2 deprecated some key exchange methods or is not accepting keys generated with certain algorithms.
Though I thought ssh-rsa will be deprecated later due to the SHA1 vulnerability and not in 8.2, but maybe you have to enable a
flag somewhere on the server. This is defenitely worth something for the announcments upstream then, like
"caveat: if you run an Archlinux server and use the bla bla bla keys, you might loose connection to your server." :-)
Offline
Okay, I've not managed to upgrade beyond the packages I downloaded on Thursday locally yet, although I did try to update yesterday I think and it did download the packages. I'm still able to connect to my 64-bit arch server, but for the record I think I disabled rsa1 key exchange in my configs already.
Last edited by levi (2020-02-16 20:50:57)
Architecture: pentium4, Testing repos: Yes, Hardware: EeePC 901+2GB RAM+OS half on the SD card.
Offline
About the systemd/seccomp/glibc 2.31 thingy: I just checked libseccomp on 64-bit, and that one
is missing the clock_nanosleep_time64 (and all time64 related hooks). libseccomp git is fixed
(so is 2.4.2). systemd has clock_nanosleep_time64 in @default for the seccomp filters, so it is
prepared for the new glibc 2.32 time64 system call.
Upstream Arch still has libseccomp 2.4.1, this is worth an out-of-date flagging.
libseccomp 2.4.2 (was released Nov 2019, ahem) also has the new time64 hooks, so that
should be sufficient to make our build slaves work again (and systemd booting again on 32-bit).
So we should keep glibc 2.31 back till libseccomp 2.4.2 hits stable.
Offline
About the ssh thingy: simply upgrade openssh on the 64-bit sever and restart it with systemctl restart sshd
(that is if you have physical access, a ssh shell still open or a KVM switch or a virtual console of sorts, or in
the worst case you can just restart the machine with some remote power-cycling mechanism, a remote management
console or a fancy Raspberry PI with a power relay).
Offline
Cross-posting here: https://bugs.archlinux.org/task/65523
Offline
Okay, has libseccomp been updated on x64, or does this not depend on that?
Architecture: pentium4, Testing repos: Yes, Hardware: EeePC 901+2GB RAM+OS half on the SD card.
Offline
You can follow the discussion on https://bugs.archlinux.org/task/65523.
No, libseccomp 2.4.1 is not up-to-date and discussions go now in the direction of whether 2.4.2 is enough,
or do we even need backports from 2.4.3.
The 64-bit time functions are fine with 2.4.2, so that would be sufficient for "normal" usage of glibc 2.31.
But more esoteric syscalls might still be missing causing hard-to-detect and -debug errors while building
packages (imagine a autoconf test failing silently because of a EPERM in a missing seccomp filter, or
as we had some time ago in file/libmagic).
Offline
Okay, that good to see they didn't just bat it back with a 'we don't support 32-bit x86 builds any more'. Seems there's more than just a weird incompatability with a briefly released glibc backing this up, with multiple issues perhaps all backing up to a wrongly signed source drop.
Architecture: pentium4, Testing repos: Yes, Hardware: EeePC 901+2GB RAM+OS half on the SD card.
Offline
Yes, actually Eli made a good point: there are people using 32-bit containers of say Debian on Archlinux.
I also used to do this: having the newest kernel on Archlinux makes it relatively easy to run all other Linux distros in a
chroot, container for instance for automatized building/testing because the Linux kernel is highly backwards compatible.
Offline