You are not logged in.

#1 2017-06-20 15:55:06

deep42thought
Administrator
From: Jena, Germany
Registered: 2017-06-17
Posts: 315

Discussion of possible means of testing

Hi,

this thread is for discussion and brainstorming possible variants of testing packages and the iso.

The question is what criteria we expect a package to fulfill, before we move it from the testing to the stable repositories.
Furthermore, the question is, how we can automate those tests.

cheers,
deep42thought

Offline

#2 2017-06-20 16:33:45

tyzoid
Administrator
From: Ann Arbor, MI
Registered: 2017-06-17
Posts: 89

Re: Discussion of possible means of testing

I was talking on the #archlinux channel about testing methodology, and the main takeaways are

  • No automated tests are done, outside of the check() functions

  • Isos are not automatically tested

  • Core packages are checked and signed off on, before moving out of testing

  • Testing is "optional" outside of core (maintainers discretion)

Also, that the archlinux-arm community might have some testing infrastructure already available.

Here's the chat log:

12:05:57        tyzoid | I'm curious as to how testing of the packages happens before they're transitioned from testing to the production repos
12:06:16        tyzoid | is it an automated test? Does that promotion happen manually or automatically?
12:07:16         jelle | no automation :)
12:07:49        tyzoid | Some of the pkgbuild files have check functions
12:08:03        tyzoid | is that only to test against broken builds, not for packages that don't work well?
12:08:24     Namarrgon | check() is for upstream tests
12:08:45        tyzoid | like when self-test funtions exist from the developers?
12:08:49     Namarrgon | yup
12:09:10        tyzoid | does that influence the decision to promote out of testing earlier?
12:09:25     Namarrgon | packages for [core] require sign-offs from devs/testers
12:09:56        tyzoid | and for [community]/[extra]?
12:09:59     Namarrgon | packages for other repos optionally go through testing
12:10:05        tyzoid | okay
12:14:26        tyzoid | So Namarrgon: Do you know if the arch install isos get tested at all?
12:15:13     Namarrgon | tyzoid: nothing beyond "does it boot? ship it."
12:16:01     Namarrgon | tyzoid: they don't change much, though, so if the previous booted the next one will probably boot, too.
12:16:05        tyzoid | Namarrgon: And is that a manual test?
12:16:23     Namarrgon | tyzoid: yes
12:18:19        tyzoid | Namarrgon: I'm only asking because over at the arch32 project we had issues with the recent i686 glibc causing a segfault on boot
12:18:38        tyzoid | so I'm trying to see if I can't get some automated testing infrastructure set up there
12:18:47        tyzoid | and was wondering what is done over here
12:19:20        tyzoid | but it sounds like nothing is automated outside of those check() functions in the packagebuilds
12:19:58     Namarrgon | tyzoid: it's all handcrafted :P
12:21:33     Namarrgon | tyzoid: it's usually not a big deal if an issue has some issue, you can just use the previous until it gets fixed
12:21:39     Namarrgon | *an iso has
12:25:21        tyzoid | Namarrgon: do you mind if I include a copy of the chat log about testing on the arch32 forum?
12:27:17     Namarrgon | tyzoid: i don't mind but i'm not responsible for any damage caused by typos, bad jokes or inaccuracies
12:28:04        tyzoid | Alright, thanks
12:36:12     Namarrgon | tyzoid: other "important" packages go through testing but it's at the maintainers discretion. large rebuilds go through the staging repos before the go to testing and finally core/extra/community
12:38:09     Namarrgon | tyzoid: if you want more automation you want to look at what the alarm folks are doing

Offline

#3 2017-06-20 16:44:44

rewbycraft
Server Wrangler
From: Eindhoven, NL
Registered: 2017-06-19
Posts: 10
Website

Re: Discussion of possible means of testing

It would potentially be worth the effort to go and poke the ALarm people.

If nothing else, we could easily reproduce the "Does it boot?" iso test by just starting a VM and seeing if we can ping it within say 5 minutes.

We could potentially put some stuff in place to automatically refuse to move from testing->core if check() fails.

Offline

#4 2017-06-20 17:25:48

tyzoid
Administrator
From: Ann Arbor, MI
Registered: 2017-06-17
Posts: 89

Re: Discussion of possible means of testing

rewbycraft wrote:

If nothing else, we could easily reproduce the "Does it boot?" iso test by just starting a VM and seeing if we can ping it within say 5 minutes.

This could work quite well as a baseline test. 5 minutes is a bit excessive, but that could be determined empirically.

rewbycraft wrote:

We could potentially put some stuff in place to automatically refuse to move from testing->core if check() fails.

If I remember correctly, we don't push anything into staging/testing that fails check in the first place.

Offline

#5 2017-06-20 20:05:05

rewbycraft
Server Wrangler
From: Eindhoven, NL
Registered: 2017-06-19
Posts: 10
Website

Re: Discussion of possible means of testing

tyzoid wrote:
rewbycraft wrote:

If nothing else, we could easily reproduce the "Does it boot?" iso test by just starting a VM and seeing if we can ping it within say 5 minutes.

This could work quite well as a baseline test. 5 minutes is a bit excessive, but that could be determined empirically.

I agree, it was somewhat of a random number for illustration purposes. That said, 5 minutes would account for potential issues in case the testing machines are overloaded.

tyzoid wrote:
rewbycraft wrote:

We could potentially put some stuff in place to automatically refuse to move from testing->core if check() fails.

If I remember correctly, we don't push anything into staging/testing that fails check in the first place.

At the moment, yes.
But one of the points posed in the original post was:

deep42thought wrote:

Furthermore, the question is, how we can automate those tests.

So I figure that we may at some point automate the testing -> stable progression. In which case such a check would be a good idea.

Offline

#6 2017-06-20 22:46:49

bsaner
Member
Registered: 2017-06-20
Posts: 1

Re: Discussion of possible means of testing

hey- i'm the author of AIF-NG (more info here) and i was told my project may be useful to assist in testing builds:

17:44:40 < r00t^2> WarheadsSE: that reminds me, did you see that revival of AIF i wrote? acts more like kickstart than the original AIF
...
17:46:09 < r00t^2> https://aif.square-r00t.net/
17:46:20 < r00t^2> there's a config generator too, it's not documented there but it is at... sec
17:46:42 < r00t^2> https://wiki.archlinux.org/index.php/AIF-NG and https://wiki.archlinux.org/index.php/AI … ation_file

18:02:33 < tyzoid> root^2: Does aif-ng work with the default arch iso?
18:02:38 < tyzoid> Or is it a special iso?
18:03:33 < r00t^2> tyzoid: per the docs, no - it's still in testing stage; i need more feedback. once i've got a bit more input i'll be proposing it for inclusion
18:03:53 < r00t^2> there's a pre-build ISO available, but you'll probably want to change the boot params to point to your own aif.xml
18:04:36 < r00t^2> or you can cook your own ISO; the full manual has instructions
18:04:54 < tyzoid> I'm thinking of it as a possibility for doing automatic build testing for arch32
18:05:04 < tyzoid> so we have our own facilities for iso building
18:05:08 < tyzoid> which isn't too big a problem
18:05:13 < r00t^2> could definitely do that, yeah
18:05:44 < r00t^2> you could even probably get it to build in a qcow2 image so you could test with libvirt/kvm or something
18:06:04 < tyzoid> I was thinking of using virtualbox
18:06:10 < tyzoid> but that might work too
18:06:31 < tyzoid> mostly because I don't have much experience with virtualization outside of vbox
18:06:31 < r00t^2> yeah, vbox, same thing, different process- just manually create the vbox disk image and mount it in one of the pre hooks
18:07:14 < tyzoid> root^2, have you seen the arch32 forum?
18:07:19 < tyzoid> https://bbs.archlinux32.org/
18:07:22 < r00t^2> not yet
18:07:35 < tyzoid> we have a thread over there looking at solutions for automated testing
18:08:09 < r00t^2> oh nice- will register and suggest if you can find a link to the thread
18:08:28 < tyzoid> lol, it's the only thread on the forum right now smile
18:08:30 < tyzoid> https://bbs.archlinux32.org/viewtopic.php?pid=6#p6
18:09:10 < rewbycraft> I want to point out that I've already got a libvirt/kvm based virtualization cluster
18:09:51 < rewbycraft> So if need be, I can just give a hookup to my cluster to let you boot whatever qcow2 image you need

If you have any questions, I may not be monitoring the forums but I keep a persistent irssi session open on Freenode, my nick's r00t^2. feel free to PM; i should reply in 8-12 hours at the latest (but if i'm awake, it's more likely to be 5-10 minutes).

Offline

#7 2017-09-29 15:14:44

andreas_baumann
Administrator
From: Zurich, Switzerland
Registered: 2017-08-10
Posts: 534
Website

Re: Discussion of possible means of testing

I'm currently using a small script to test the ISO and a local mirror I'm rsyncing from the official mirrors,
just to test whether I get a running VM:

https://github.com/andreasbaumann/archauto32

I find the idea of using AIF very good (never used it myself, so I'll have a look into it).

Really interesting would be to have a set of typical installation scenarios (KDE desktop with browser; webserver with mysql, etc.)
lying around as list of packages to install and some configuration.

Offline

#8 2018-11-08 08:40:42

Markus12
Member
Registered: 2018-11-05
Posts: 3

Re: Discussion of possible means of testing

You know it is kind of brainstorm when you try to find out different variants of testing packages and so on. I think it is very useful, sometime we will have totally automatic testing systems cool

Last edited by Markus12 (2018-11-08 08:58:40)

Offline

Board footer

Powered by FluxBB