957dd49ef7
There are two possible interpretations of "expected failure": either
the test *must* fail (exactly the inverse of an ordinary test, with
success becoming failure and failure becoming success), or the test
*may* fail (with success intended, but failure possible in some
environments). Autotools had the second interpretation, which seems
more useful in practice, but Meson has the first.
Instead of using should_fail, we can put the tests in one of two new
suites: "flaky" is intended for tests that succeed or fail unpredictably
according to the test environment or chance, while "failing" is for
tests that ought to succeed but currently never do as a result of a
bug or missing functionality. With a sufficiently new version of Meson,
the flaky and failing tests are not run by default, but can be requested
by running a setup that does not exclude them, with a command like:
meson test --setup=x11_unstable --suite=flaky --suite=failing
As a bonus, now that we're setting up setups and their excluded suites
programmatically, the gsk-compare-broadway tests are also excluded by
default when running the test setup for a non-broadway backend.
When running the tests in CI, --suite=gtk overrides the default
exclude_suites, so we have to specify --no-suite=flaky and
--no-suite=failing explicitly.
This arrangement is inspired by GNOME/glib!2987, which was contributed
by Marco Trevisan.
Signed-off-by: Simon McVittie <smcv@debian.org>
GTK CI infrastructure
GTK uses different CI images depending on platform and jobs.
The CI images are Docker containers, generated either using docker or
podman, and pushed to the GitLab container registry.
Each Docker image has a tag composed of two parts:
${image}: the base image for a given platform, like "fedora" or "debian-stable"${number}: an incremental version number, orlatest
See the container registry for the available images for each branch, as well as their available versions.
Note that using latest as version number will overwrite the most
recently uploaded image in the registry.
Checklist for Updating a CI image
- Update the
${image}.Dockerfilefile with the dependencies - Run
./run-docker.sh build --base ${image} --version ${number} - Run
./run-docker.sh push --base ${image} --version ${number}once the Docker image is built; you may need to log in by usingdocker loginorpodman login - Update the
imagekeys in the.gitlab-ci.ymlfile with the new image tag - Open a merge request with your changes and let it run
Checklist for Adding a new CI image
- Write a new
${image}.Dockerfilewith the instructions to set up a build environment - Add the
pip3 install mesonincantation - Run
./run-docker.sh build --base ${image} --version ${number} - Run
./run-docker.sh push --base ${image} --version ${number} - Add the new job to
.gitlab-ci.ymlreferencing the image - Open a merge request with your changes and let it run
Checklist for Adding a new dependency to a CI image
Our images are layered, and the base (called fedora-base) contains all the rpm payload. Therefore, adding a new dependency is a 2-step process:
- Build and upload fedora-base:$version+1
- Build and upload fedora:$version+1 based on fedora-base:version+1