You are not logged in.

#51 2018-09-02 01:25:40

eschwartz
Member
Registered: 2018-08-31
Posts: 11

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

deep42thought wrote:
thx1138 wrote:

In other words, is the availability of hardware to re-build large 32-bit packages quickly an issue here?  Are more build machines needed?

The number of available build slaves is sufficient (but we don't have too much head room). We rather lack the manpower to fix (in a timely manner) all the packages which do not compile on/for 32-bit. To put this in numbers: big rebuilds of python or perl (where a few thousand packages need to be rebuilt) are done in less than 24h - iff most of the packages build without issues; if we need to fix a lot of packages, then it'll take us a lot longer ...

Same rule applies for us. We had an autobuilder which could build between two and four packages at a time, which we used to handle the 1000+ python rebuild. We built in batches, depending on which packages could currently be rebuilt without missing dependencies in a later batch... and this actually finished quite fast, but most batches never completed because of a handful of packages that failed to build. So we'd get a few packages built almost immediately, then progress would stall for a couple days while people, usually me, foutrelis, or felixonmars, figured out how to patch the package to build okay, then pushed a new version to un-jam the pipes.

This is the actual reason, why it took like a month.


Arch Linux (64) Bug Wranger and Trusted User

Offline

#52 2018-09-03 15:42:09

tommi
Member
Registered: 2018-08-25
Posts: 14

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

one simple way to get apps working with python 3.7.
is to use github. for example youtube-dl:
https://github.com/rg3/youtube-dl

Offline

#53 2018-09-03 15:51:32

eschwartz
Member
Registered: 2018-08-31
Posts: 11

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

I'm, uh, just going to ignore you.

Bye.


Arch Linux (64) Bug Wranger and Trusted User

Offline

#54 2018-09-03 16:44:50

tommi
Member
Registered: 2018-08-25
Posts: 14

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

or sudo pip install --upgrade youtube-dl

Offline

#55 2018-09-03 22:45:23

levi
Moderator
From: UK
Registered: 2018-06-16
Posts: 227

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

Stop it!  This is arch lands where we always check the AUR before git cloning anything ourselves.

And that won't help when installed python libs are build against the old version of python, so doesn't really solve anything this thread is actually about.  Thankfully though, I think we're over that episode for now (certainly I've had no problems on the testing repos).

Offline

#56 2018-09-04 05:30:26

andreas_baumann
Administrator
From: Zurich, Switzerland
Registered: 2017-08-10
Posts: 591
Website

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

If you use pip and pacman in parallel for python, make sure you check out in the next upgrade about stale packages.
Yes, always consult the AUR. Also you can build a package locally if it is failing as binary:

asp export python-xxx
cd python-xxx
makepkg -si

and add an IgnorePkg :-)

pip install and direct download are the only option, when things do not exist as package or as AUR package or
the AUR package is bad (but in this case you should also report back to the person maintaing the AUR package).

Offline

#57 2018-09-04 05:33:21

andreas_baumann
Administrator
From: Zurich, Switzerland
Registered: 2017-08-10
Posts: 591
Website

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

A word on github and network downloads: the current way of development where everything is a bunch
of git submodules and a software is recompiled from source is highly flawed!

Packages and shared libraries have been invented to - well - share code.
Who keeps all the local copies of a library up to date? (violates DRY).

The space argument is more or less obsolete, but if you ever built a monster
package like libreoffice, mozilla stuff you know how funny is is to wait all the time
for builds to finish just to get a final error at the end.

And remember: github is owned by M$ :->

Offline

#58 2018-09-04 08:16:36

tommi
Member
Registered: 2018-08-25
Posts: 14

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

Naturally, always prefer the use of the AUR! I never thougt it's necessary to mention that the use of python-pip and pacman in parallel, or any other source is not normal. You have to watch it and should now what you are doing. Pip itself is in the AUR for a workaround in complex situations when system rescue is priority.
I do not have so much time at the moment so i wich you much success and have a nice day.

Last edited by tommi (2018-09-04 08:17:39)

Offline

#59 2018-09-04 08:52:55

andreas_baumann
Administrator
From: Zurich, Switzerland
Registered: 2017-08-10
Posts: 591
Website

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

I pushed youtube-dl to stable.

Offline

#60 2018-09-04 09:06:24

tommi
Member
Registered: 2018-08-25
Posts: 14

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

Thank you very much for you time and effort.

Offline

#61 2018-09-11 08:44:57

shroom
Member
Registered: 2018-08-21
Posts: 9

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

Thanks for fixing all the dependencies! Did a clean install today without any issue smile

Offline

#62 2018-09-11 09:58:05

rogerthat
Member
Registered: 2018-08-14
Posts: 47

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

I have been away from my terminal for several days. What is the status of things looking like for running an upgrade successfully?

Offline

#63 2018-09-12 13:34:10

tommi
Member
Registered: 2018-08-25
Posts: 14

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

gnome-builder runs with the following warnings:
[thomas@home ~]$ gnome-builder
15:21:07.0450                           ide-application[ 6144]:  WARNING: Cannot enable Python 3 plugins: Typelib file for namespace 'Atk', version '1.0' not found
15:21:07.0962                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:07.0963                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:07.0964                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:07.0964                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:07.0965                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0006                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0007                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0008                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0009                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0009                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0011                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0012                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0012                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0013                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0014                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0015                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled
15:21:08.0015                                   libpeas[ 6144]:  WARNING: The 'python3' plugin loader has not been enabled

its no problemfor me. but there may other apps with problem.
anyhow its highly commended to backup the system before update

Last edited by tommi (2018-09-12 13:34:59)

Offline

#64 2018-09-12 14:59:36

levi
Moderator
From: UK
Registered: 2018-06-16
Posts: 227

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

tommi wrote:

anyhow its highly commended to backup the system before update

That's true for things like windows where the update process is opaque, but on most linuxes and certainly on arch you could just back up your package list (pacman -Q) and maybe your /var/cache/pacman/pkg folder (if you expect the arch servers to go down before you can restore).  Back up your user files, but the OS folders can be depended on to be regeneratable from the installed packages provided you don't fill the non home folders with your crap.

Or just roll back specific packages if they ever break, but that requires a greater degree of arch-fu I guess.

Offline

#65 2018-09-12 15:51:17

tommi
Member
Registered: 2018-08-25
Posts: 14

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

ok this sounds pretty much fast to me.

its possible to use fsarchiver http://www.fsarchiver.org/quickstart/ to save your entire system partition.

//for example:

fsarchiver savefs -o -v -/sdb1/Fsarchiver/arch-system.fsa /dev/sda2

//where sda2 is the system partition
//to write back the hole partition:

fsarchiver restfs -o -v  /sdb1/Fsarchiver/arch-system.fsa  id=0,dest=/dev/sda2


/*it needs time and storage with around half the size of the partition to save. but fsarchiver can transfer the system to a different media/partitionttype like a bootable usb device if you want to. so one can create a testsystem for example. you need to change the target device in the  restfs session:*/

fsarchiver restfs -o -v  /sdb1/Fsarchiver/arch-system.fsa id=0,dest=/dev/sdc1

//then you go into that:

sudo mount /dev/sdc1 /mnt

sudo mount --bind /dev /mnt/dev &&
sudo mount --bind /dev/pts /mnt/dev/pts &&
sudo mount --bind /proc /mnt/proc &&
sudo mount --bind /sys /mnt/sys

sudo chroot /mnt

grub-install --recheck /dev/sdc1

sudo grub-mkconfig -o /boot/grub/grub.cfg

exit &&
sudo umount /mnt/sys &&
sudo umount /mnt/proc &&
sudo umount /mnt/dev/pts &&
sudo umount /mnt/dev &&
sudo umount /mnt

//if Bios accepts booting from that device sdc1 (external hard drive for example) you can test things without touching the original system

Last edited by tommi (2018-09-12 16:08:42)

Offline

#66 2018-09-12 17:12:27

tommi
Member
Registered: 2018-08-25
Posts: 14

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

!for the first time you need a live-cd because the system must be unmounted! Later you can use the external disk to run fsarchiver. if uuid is used you need to modify the fstab on that disk before update grub or your favorit bootloader.

if this seems complex dont worry all roads lead to Rome smile

p.s.

also take a look here:
https://wiki.archlinux.org/index.php/LVM#Snapshots
http://www.fsarchiver.org/live-backup/

Last edited by tommi (2018-09-12 19:58:54)

Offline

#67 2018-09-13 02:30:40

rogerthat
Member
Registered: 2018-08-14
Posts: 47

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

Well, I took the plunge and ran a system upgrade today. Everything completed without any unusual warnings or errors!

YAY ArchLinux32!

Offline

#68 2018-09-13 08:15:30

andreas_baumann
Administrator
From: Zurich, Switzerland
Registered: 2017-08-10
Posts: 591
Website

Re: Why are incomplete rebuilds pushed to stable (perl, python3.7)?

Glad to hear that. :-)

Offline

Board footer

Powered by FluxBB