Arch Linux

abaumann

Administrator

From: Zurich

Registered: 2019-11-14

Posts: 985

Website

How to create a build slave?

1. Add the Archlinux32 x86_64/releng repo to pacman.conf:

       [releng]
       Server = https://${ANY_ARCHLINUX32_MIRROR}/$arch/$repo

       Update the repo-db/update packages (pacman -Syu)

       Install archlinux32-keyring-transition

       Install archlinux32-keyring (replaces archlinux32-keyring-transition, you can also use archlinux32-keyring-git from the AUR for the newest keyring)

       Install dependencies (base-devel git wget rsync moreutils devtools32 inetutils moreutils)

       Note: devtools32 might not be up-to-date, there is an AUR package devtools32-git or use the git repo directly from https://git.archlinux32.org/devtools32 directly and install manually with:

   make
   make install

2. Test manually whether you can build packages on the local machine:

   pkgctl repo clone --arch32 --protocol=https sdl2
   cd sdl2
   core-staging-pentium4-build

   You can have a look for the Arch32 marker in the PKGBUILD file, the uipper part is the upstream PKGBUILD, the lower part is the difference from the arch32 package git describing the patches required for Arch32.

3. Create a signing key for signing your packages. Communicate this key to the group.

4. Create a user to run the slave (for instance 'builder')
       visudo and add the build commands to user 'builder':

       Cmnd_Alias BUILD_CMDS = /usr/bin/core-staging-i486-build /usr/bin/core-staging-i686-build /usr/bin/core-staging-pentium4-build /usr/bin/core-stagingpkg /usr/bin/core-staging-with-build-support-i486-build /usr/bin/core-staging-with-build-support-i686-build /usr/bin/core-staging-with-build-support-pentium4-build /usr/bin/core-staging-x86_64-build /usr/bin/core-staging-x86_64_v3-build /usr/bin/core-testing-i486-build /usr/bin/core-testing-i686-build /usr/bin/core-testing-pentium4-build /usr/bin/core-testingpkg /usr/bin/core-testing-x86_64-build /usr/bin/core-testing-x86_64_v3-build /usr/bin/extra-i486-build /usr/bin/extra-i686-build /usr/bin/extra-pentium4-build /usr/bin/extra-staging-i486-build /usr/bin/extra-staging-i686-build /usr/bin/extra-staging-pentium4-build /usr/bin/extra-stagingpkg /usr/bin/extra-staging-with-build-support-i486-build /usr/bin/extra-staging-with-build-support-i686-build /usr/bin/extra-staging-with-build-support-pentium4-build /usr/bin/extra-staging-x86_64-build /usr/bin/extra-staging-x86_64_v3-build /usr/bin/extra-testing-i486-build /usr/bin/extra-testing-i686-build /usr/bin/extra-testing-pentium4-build /usr/bin/extra-testingpkg /usr/bin/extra-testing-x86_64-build /usr/bin/extra-testing-x86_64_v3-build /usr/bin/extra-x86_64-build /usr/bin/extra-x86_64_v3-build
       builder ALL=(ALL) NOPASSWD: BUILD_CMDS

       su to builder
       create .gnupg directory, and add to ~/.gnupg/gpg.conf:

       keyserver-options auto-key-retrieve

5. Check out the code from https://git.archlinux32.org/builder/ into 'builder'. Copy 'conf/slave.conf.example' to 'conf/slave.conf'. Customize the slave by setting at least the 'package_key' to your signing package key. Most importantly set the signing key ID for signing packages:

   package_key='0x15eebadc0de'

6. If you want to run multiple build slaves on the same machine, create Linux users for them like 'slave1', 'slave2' and give them each a separate workspace of the builder. Also, configure separate archbuild chroots for each of them:

   archbuild_chroots='/var/lib/archbuild/slave1'

7. Get the Archlinux32 mirrorlist package 'mirrorlist32' from http://mirror.archlinux32.org/x86_64/releng/ or create one by hand and store it as '/etc/pacman.d/mirrorlist32', pick a mirror close to you (preferably one you have set up for the build slave locally), e.g.:

   Server = https://localmirror/$arch/$repo

   Note: update pacman*.conf files in /usr/share/devtools when building on x86_64 (set Architecture manually to i486, i686, not to auto)

8. You should also make sure, that before building a package to call a mirror syncing script (you configure this also in 'conf/slave.conf':

   mirror_update_command='curl -v https://localmirror/php/sync_mirror.php'

   Such a syncing script could look as follows:

   <?php
   
       header( "Content-Type: text/plain" );
       system( "sudo /data/arch32/scripts/sync_mirror" );

   For sync_mirror have a look how to set up and sync a local mirror.

9. Run the slave for a first time with 'builder/bin/build-package' . to clone upstream's pkgbuild files and verify everything is working. First time you start it you get asked about generating a private/public key pair. Leave the password blank.

   Note: This will generate an error after cloning about not having permission to access buildmaster. Just hit Ctrl+C to quit once you hit this error.

   The git repositories in work/repos/ are packages, community and packages32, they are in bare format, so you have to clone them once, remove the bare once and rename the workspaces back to their original name. Also change the remote origin to point to the repo location at git://git.archlinux.org/svntogit/packages.git,git://git.archlinux.org/svntogit/community.git  resp. https://git.archlinux32.org/packages and not to the bare file system (git://git.archlinux.org/svntogit/packages.git, git://git.archlinux.org/svntogit/community.git and https://git.archlinux32.org/packages.git).

10. It will also generate a slave SSH key pair in 'work/.ssh/' as 'id_rsa'/'id_rsa.pub'. Communicate the 'id_rsa.pub' to the group - along with your name and the name of the build slave, so it can be entered on the buildmaster with:

    manage-slaves add <slave name> <owner of the slave> <public key fingerprint of the build slave>
    manage-slaves enable <slave name>

    From then on you should see the slave appear in https://archlinux32.org/buildmaster/build-slaves.php

11. Import your build key 'work/.ssh/id_rsa' (private key!) to the builder's gpg keyring - strip passphrase if one exists

12. Set PACKAGER in /etc/makepkg.conf on the host to an email address which will end up in all packages you are building.

13. Run builder/bin/build-packages (in a screen/tmux buffer) to start building

14. You can also create a systemd service in '/etc/systemd/system/arch32-slave@.service':

    [Unit]
    Description=Archlinux32 Build Slave (%I)
    After=network.target
    
    [Install]
    WantedBy=multi-user.target
    
    [Service]
    User=%I
    Type=simple
    Restart=on-success
    RestartSec=10
    WorkingDirectory=/home/%I
    ExecStartPre=/usr/bin/git -C /home/%I/builder pull
    ExecStart=/home/%I/builder/bin/build-packages

    Start it with:

    systemctl start arch32-slave@builder.service

    Stop it gracefully with:

    touch /tmp/do-not-run-build-slave

15. You can see what the build slave is doing with:

    journalctl -fl -u arch32-slave@builder

    and build details of the current package building with:

    cd /home/builder/builder/work && tail -f tmp.*/*build-log

16. Usually build slaves should run on 64-bit hosts. Only when packages are recognizing features of the host instead of the chroot, you have to build the i486 packages on a i486 host for instance.