gitlab CI: switch to using ci-templates

ci-templates encourages building specific containers that can be re-used:
- containers are re-used across pipelines, producing consistent results
- containers are re-used by contributors since they will use the upstream
  containers for their MR, thus guaranteeing the same results.

Containers are automatically rebuild whenever the respective
FDO_DISTRIBUTION_TAG changes. This is particularly interesting now that
Docker Hub will introduce pull limits.

This CI script consists of a config file and a jinja2 template, simply
running 'ci-fairy generate-template' produces the .gitlab-ci.yml.
ci-fairy is part of the freedesktop.org ci-templates and can be pip
installed, see the check-ci-script job.

Functional changes to the previous script:
- new job: check-ci-script, verifies that our gitlab-ci.yml is the one
  generated by the sources
- Added distributions:
  - Fedora 33
- The actual work is now down by a set of scripts in .gitlab-ci/,
  specifically:
  - .gitlab-ci/build.sh is the previous do_build job
  - .gitlab-ci/{fedora|debian}-install.sh are the previous {fedora|debian}_install jobs
  symlinks are in place for centos and ubuntu

Why the scripts instead of steps in the CI? Easer to reading and
reproduce. With the containers being static, it's easy to pull one
locally and re-run the CI job to reproduce an issue. Having everything in a
single script makes that trivial.

Signed-off-by: Peter Hutterer <peter.hutterer@who-t.net>

https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/merge_requests/664

7 files changed, 345 insertions, 0 deletions

diff --git a/.gitlab-ci/build.sh b/.gitlab-ci/build.sh
new file mode 100755
index 0000000000..5f8906ec57
--- /dev/null
+++ b/.gitlab-ci/build.sh
@@ -0,0 +1,19 @@
+#!/bin/bash
+
+date '+%Y%m%d-%H%M%S'; uname -a
+date '+%Y%m%d-%H%M%S'; locale -a
+date '+%Y%m%d-%H%M%S'; env
+date '+%Y%m%d-%H%M%S'; meson --version
+date '+%Y%m%d-%H%M%S'; ! which dpkg || dpkg -l
+date '+%Y%m%d-%H%M%S'; ! which yum  || yum list installed
+date '+%Y%m%d-%H%M%S'; git clean -fdx ; BUILD_TYPE=autotools CC=gcc   WITH_DOCS=1 WITH_VALGRIND=1 contrib/scripts/nm-ci-run.sh
+date '+%Y%m%d-%H%M%S'; rm -rf /tmp/nm-docs-html; mv build/INST/share/gtk-doc/html /tmp/nm-docs-html
+date '+%Y%m%d-%H%M%S'; git clean -fdx ; BUILD_TYPE=meson     CC=gcc   WITH_DOCS=1 WITH_VALGRIND=1 contrib/scripts/nm-ci-run.sh
+date '+%Y%m%d-%H%M%S'; git clean -fdx ; BUILD_TYPE=autotools CC=clang WITH_DOCS=0 contrib/scripts/nm-ci-run.sh
+date '+%Y%m%d-%H%M%S'; git clean -fdx ; BUILD_TYPE=meson     CC=clang WITH_DOCS=0 contrib/scripts/nm-ci-run.sh
+date '+%Y%m%d-%H%M%S'; git clean -fdx ; ! grep -q '^NAME=.*\(Fedora\|CentOS\)' /etc/os-release || ./contrib/fedora/rpm/build_clean.sh -g -w crypto_gnutls -w debug -w iwd -w test -W meson
+date '+%Y%m%d-%H%M%S'; git clean -fdx ; ! grep -q '^NAME=.*\(Fedora\)'         /etc/os-release || ./contrib/fedora/rpm/build_clean.sh -g -w crypto_gnutls -w debug -w iwd -w test -w meson
+date '+%Y%m%d-%H%M%S'; git clean -fdx ; test "$NM_BUILD_TARBALL" != 1 || ( SIGN_SOURCE=0 ./contrib/fedora/rpm/build_clean.sh -r && mv ./NetworkManager-1*.tar.xz /tmp/ && mv ./contrib/fedora/rpm/latest/SRPMS/NetworkManager-1*.src.rpm /tmp/ )
+date '+%Y%m%d-%H%M%S'; git clean -fdx
+date '+%Y%m%d-%H%M%S'; mv /tmp/nm-docs-html ./docs-html
+date '+%Y%m%d-%H%M%S'; test "$NM_BUILD_TARBALL" != 1 || mv /tmp/NetworkManager-1*.tar.xz /tmp/NetworkManager-1*.src.rpm ./
diff --git a/.gitlab-ci/centos-install.sh b/.gitlab-ci/centos-install.sh
new file mode 120000
index 0000000000..78d48447af
--- /dev/null
+++ b/.gitlab-ci/centos-install.sh
@@ -0,0 +1 @@
+fedora-install.sh
\ No newline at end of file
diff --git a/.gitlab-ci/ci.template b/.gitlab-ci/ci.template
new file mode 100644
index 0000000000..9f67dd7891
--- /dev/null
+++ b/.gitlab-ci/ci.template
@@ -0,0 +1,228 @@
+# vim: set expandtab shiftwidth=2 tabstop=8 textwidth=0 filetype=yaml:
+
+{# You're looking at the template here, so you can ignore the below
+   warning. This is the right file to edit #}
+########################################
+#                                      #
+# THIS FILE IS GENERATED, DO NOT EDIT  #
+#                                      #
+########################################
+
+
+.templates_sha: &template_sha d303fafa66bbca44cc5bdb337d5773b5f019b70d # see https://docs.gitlab.com/ee/ci/yaml/#includefile
+
+include:
+{% for distro in distributions|sort(attribute="name") %}
+  # {{ distro.name.capitalize() }} container builder template
+  - project: 'freedesktop/ci-templates'
+    ref: *template_sha
+    file: '/templates/{{distro.name}}.yml'
+{% endfor %}
+
+stages:
+  - prep
+  - test
+  - deploy
+  - triage
+  - container_clean
+
+variables:
+  FDO_UPSTREAM_REPO: NetworkManager/NetworkManager
+  GIT_DEPTH: 1
+  # these tags should be updated each time the list of packages is updated
+  # changing these will force rebuilding the associated image
+  # Note: these tags have no meaning and are not tied to a particular
+  # libinput version
+{% for distro in distributions %}
+  {{"%-13s"| format(distro.name.upper() + '_TAG:')}}'{{distro.tag}}'
+{% endfor %}
+
+{% for distro in distributions %}
+  {{"%-13s"| format(distro.name.upper() + '_EXEC:')}}'bash .gitlab-ci/{{distro.name}}-install.sh'
+{% endfor %}
+
+.nm_artifacts:
+  artifacts:
+    expire_in: 2 days
+    paths:
+      - docs-html
+      - NetworkManager-1*.tar.xz
+      - NetworkManager-1*.src.rpm
+
+#################################################################
+#                                                               #
+#                          prep stage                           #
+#                                                               #
+#################################################################
+
+# Re-generate the .gitlab-ci.yml file and make sure it's the one currently checked in
+# If this job fails, re-generate the gitlab-ci.yml script and git commit it.
+#
+check-ci-script:
+  image: golang:alpine
+  stage: prep
+  before_script:
+    - apk add python3 py-pip git
+    - pip3 install git+http://gitlab.freedesktop.org/freedesktop/ci-templates
+  script:
+    - ci-fairy generate-template
+    - git diff --exit-code && exit 0 || true
+    - echo "Committed gitlab-ci.yml differs from generated gitlab-ci.yml. Please verify"
+    - exit 1
+
+#################################################################
+#                                                               #
+#                     containers stage                          #
+#                                                               #
+#################################################################
+
+# Build a container for each distribution + version. The ci-templates
+# will re-use the containers if the tag doesn't change.
+{% for distro in distributions %}
+{% for version in distro.versions %}
+
+{{distro.name}}:{{version}}@container-prep:
+  extends:
+    - .fdo.container-build@{{distro.name}}
+  stage: prep
+  variables:
+    GIT_STRATEGY: none
+    FDO_DISTRIBUTION_VERSION: '{{version}}'
+    FDO_DISTRIBUTION_TAG: ${{distro.name.upper()}}_TAG
+    FDO_DISTRIBUTION_EXEC: ${{distro.name.upper()}}_EXEC
+{% if not version in distro.always %}
+  when: manual
+{% endif %}
+{% endfor %}
+{% endfor %}
+
+#################################################################
+#                                                               #
+#                   container clean stage                       #
+#                 run during the clean stage                    #
+#                                                               #
+#################################################################
+
+#
+# This stage will look for the container images we currently have in
+# the registry and will remove any that are not tagged with the provided
+# $container_image:$tag
+#
+# This job only runs for a scheduled pipeline.
+#
+# Go to your Profile, Settings, Access Tokens
+# Create a personal token with 'api' scope, copy the value.
+# Go to CI/CD, Schedules, schedule a monthly job.
+# Define a variable of type File named AUTHFILE. Content is that token
+# value.
+.container-clean:
+  stage: container_clean
+  image: golang:alpine
+  before_script:
+    - apk add python3 py-pip git
+    - pip3 install git+http://gitlab.freedesktop.org/freedesktop/ci-templates
+  script:
+    - ci-fairy -v --authfile $AUTHFILE delete-image
+            --repository $FDO_DISTRIBUTION_NAME/$FDO_DISTRIBUTION_VERSION
+            --exclude-tag $FDO_DISTRIBUTION_TAG
+  dependencies: []
+  allow_failure: true
+  only:
+    - schedules
+
+{% for distro in distributions %}
+{% for version in distro.versions %}
+{{distro.name}}:{{version}}@container-clean:
+  extends:
+    - .container-clean
+  variables:
+    GIT_STRATEGY: none
+    CURRENT_CONTAINER_IMAGE: $CI_REGISTRY_IMAGE/{{distro.name}}/$FDO_DISTRIBUTION_VERSION:$FDO_DISTRIBUTION_TAG
+    FDO_DISTRIBUTION_VERSION: '{{version}}'
+    FDO_DISTRIBUTION_TAG: ${{distro.name.upper()}}_TAG
+
+{% endfor %}
+{% endfor %}
+
+#################################################################
+#                                                               #
+#                       build stage                             #
+#                                                               #
+#################################################################
+
+.build@template:
+  stage: test
+  script:
+    - .gitlab-ci/build.sh
+  dependencies: []
+
+#################################################################
+#                                                               #
+#                          test stage                           #
+#                                                               #
+#################################################################
+
+{% for distro in distributions %}
+{% for version in distro.versions %}
+
+t_{{distro.name}}:{{version}}:
+  extends:
+    - .build@template
+    - .fdo.distribution-image@{{distro.name}}
+{% if distro.name == pages_build.name and
+      version == pages_build.version %}
+    - .nm_artifacts
+{% endif %}
+  variables:
+    FDO_DISTRIBUTION_VERSION: '{{version}}'
+    FDO_DISTRIBUTION_TAG: ${{distro.name.upper()}}_TAG
+    {# Where we have extra_variables defined, add them to the list #}
+    {% if distro.build is defined and distro.build.extra_variables is defined %}
+    {% for var in distro.build.extra_variables %}
+    {{var}}
+    {% endfor %}
+    {% endif %}
+  needs:
+    - "{{distro.name}}:{{version}}@container-prep"
+{% endfor %}
+{% endfor %}
+
+#################################################################
+#                                                               #
+#                      specific jobs                            #
+#                                                               #
+#################################################################
+checkpatch:
+  extends:
+  {# run checkpatch on the same image that generates pages because why not #}
+   - t_{{pages_build.name}}:{{pages_build.version}}
+  stage: test
+  script:
+    - date '+%Y%m%d-%H%M%S'; dnf install -y git black gettext /usr/bin/xargs which clang
+    - date '+%Y%m%d-%H%M%S'; NM_CHECKPATCH_FETCH_UPSTREAM=1 contrib/scripts/checkpatch-feature-branch.sh
+    - date '+%Y%m%d-%H%M%S'; black --check . examples/python/gi/nm-wg-set
+    - date '+%Y%m%d-%H%M%S'; git ls-files -z -- 'po/*.po' | xargs -0 -n1 msgfmt -vc
+    - date '+%Y%m%d-%H%M%S'; contrib/scripts/nm-code-format.sh
+  allow_failure: true
+
+pages:
+  stage: deploy
+  script:
+    - mv docs-html public
+  artifacts:
+    expire_in: 20 days
+    paths:
+      - public
+  only:
+    - master
+  dependencies:
+    - t_{{pages_build.name}}:{{pages_build.version}}
+
+triage:issues:
+  stage: triage
+  image: ruby:2.7
+  script:
+    - gem install gitlab-triage
+    - gitlab-triage -d --token $API_TOKEN --source-id $SOURCE_ID
+  only:
+    - schedules
diff --git a/.gitlab-ci/config.yml b/.gitlab-ci/config.yml
new file mode 100644
index 0000000000..1e0d2b0b46
--- /dev/null
+++ b/.gitlab-ci/config.yml
@@ -0,0 +1,55 @@
+# This file contains the configuration for the gitlab ci.
+#
+# To recreate the .gitlab-ci.yml file, run
+#   ci-fairy generate-template
+#
+# The ci-fairy tool is part of
+# https://gitlab.freedesktop.org/freedesktop/ci-templates
+#
+
+# We're happy to rebuild all containers when one changes.
+.default_tag: &default_tag '2020-10-30.0'
+
+
+# The list of all distributions we want to create job for.
+# The template generates manual jobs for all these.
+distributions:
+  - name: fedora
+    tag: *default_tag
+    versions:
+      - '32'
+      - '33'
+      - '31'
+      - '30'
+      - '29'
+      - '28'
+      - 'rawhide'
+    always:
+      - '32'
+  - name: ubuntu
+    tag: *default_tag
+    versions:
+      - 'devel'
+      - 'rolling'
+      - '18.04'
+      - '16.04'
+  - name: debian
+    tag: *default_tag
+    versions:
+      - '9'
+      - '10'
+      - 'testing'
+      - 'sid'
+  - name: centos
+    tag: *default_tag
+    versions:
+      - '7.5.1804'
+      - '7.6.1810'
+      - '7.7.1908'
+      - '7.8.2003'
+      - '8.1.1911'
+
+# specifies which of the above distros is used as source for pages
+pages_build:
+  name: fedora
+  version: '32'
diff --git a/.gitlab-ci/debian-install.sh b/.gitlab-ci/debian-install.sh
new file mode 100755
index 0000000000..ee7f862e03
--- /dev/null
+++ b/.gitlab-ci/debian-install.sh
@@ -0,0 +1,14 @@
+#!/bin/bash
+
+date '+%Y%m%d-%H%M%S'; DEBIAN_FRONTEND=noninteractive apt-get update
+date '+%Y%m%d-%H%M%S'; DEBIAN_FRONTEND=noninteractive NM_INSTALL="apt-get -qq install -y" ./contrib/debian/REQUIRED_PACKAGES
+date '+%Y%m%d-%H%M%S'; dbus-uuidgen --ensure
+date '+%Y%m%d-%H%M%S'; sed -i 's/^# \(pl_PL.UTF-8 .*\)$/\1/p' /etc/locale.gen ; true
+date '+%Y%m%d-%H%M%S'; locale-gen pl_PL.UTF-8
+date '+%Y%m%d-%H%M%S'; grep -q "VERSION=.16.04.[0-9]\+ LTS" /etc/os-release && pip3 install meson==0.53.2
+date '+%Y%m%d-%H%M%S'; grep -q "VERSION=.16.04.[0-9]\+ LTS" /etc/os-release || pip3 install meson
+date '+%Y%m%d-%H%M%S'; contrib/scripts/nm-ci-install-valgrind-in-ubuntu1604.sh
+
+# iproute2 5.2.0 on debian:sid causes our unit tests to fail.
+# Downgrade to a working version. See https://www.spinics.net/lists/netdev/msg584916.html
+date '+%Y%m%d-%H%M%S'; ! ( dpkg -s iproute2 | grep -q '^Version[:] 5.2.0-1\(ubuntu1\)\?$' ) || (curl 'http://ftp.debian.org/debian/pool/main/i/iproute2/iproute2_4.20.0-2_amd64.deb' --output /tmp/iproute2_4.20.0-2_amd64.deb && dpkg -i /tmp/iproute2_4.20.0-2_amd64.deb)
diff --git a/.gitlab-ci/fedora-install.sh b/.gitlab-ci/fedora-install.sh
new file mode 100755
index 0000000000..9c8258e5e5
--- /dev/null
+++ b/.gitlab-ci/fedora-install.sh
@@ -0,0 +1,27 @@
+#!/bin/bash
+
+date '+%Y%m%d-%H%M%S'; ! ( grep -q '^NAME=.*\(CentOS\)' /etc/os-release && grep -q '^VERSION_ID=.*\<7\>' /etc/os-release ) || yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
+date '+%Y%m%d-%H%M%S'; ! ( grep -q '^NAME=.*\(CentOS\)' /etc/os-release && grep -q '^VERSION_ID=.*\<7\>' /etc/os-release ) || yum install -y glibc-common
+date '+%Y%m%d-%H%M%S'; ! ( grep -q '^NAME=.*\(CentOS\)' /etc/os-release && grep -q '^VERSION_ID=.*\<7\>' /etc/os-release ) || localedef -c -i pl_PL -f UTF-8 pl_PL.UTF-8 && locale -a
+date '+%Y%m%d-%H%M%S'; ! ( grep -q '^NAME=.*\(CentOS\)' /etc/os-release && grep -q '^VERSION_ID=.*\<7\>' /etc/os-release ) || yum install -y python36-dbus python36-gobject-base
+
+date '+%Y%m%d-%H%M%S'; ! ( grep -q '^NAME=.*\(CentOS\)' /etc/os-release && grep -q '^VERSION_ID=.*\<8\>' /etc/os-release ) || dnf install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-8.noarch.rpm
+date '+%Y%m%d-%H%M%S'; ! ( grep -q '^NAME=.*\(CentOS\)' /etc/os-release && grep -q '^VERSION_ID=.*\<8\>' /etc/os-release ) || dnf install -y 'dnf-command(config-manager)'
+date '+%Y%m%d-%H%M%S'; ! ( grep -q '^NAME=.*\(CentOS\)' /etc/os-release && grep -q '^VERSION_ID=.*\<8\>' /etc/os-release ) || dnf config-manager --set-enabled PowerTools
+date '+%Y%m%d-%H%M%S'; ! ( grep -q '^NAME=.*\(CentOS\)' /etc/os-release && grep -q '^VERSION_ID=.*\<8\>' /etc/os-release ) || curl https://copr.fedorainfracloud.org/coprs/nmstate/nm-build-deps/repo/epel-8/nmstate-nm-build-deps-epel-8.repo > /etc/yum.repos.d/nmstate-nm-build-deps-epel-8.repo
+
+date '+%Y%m%d-%H%M%S'; NM_NO_EXTRA=1 NM_INSTALL="yum install -y" ./contrib/fedora/REQUIRED_PACKAGES
+date '+%Y%m%d-%H%M%S'; yum install -y glibc-langpack-pl ccache clang which
+
+# containers have "tsflags=nodocs" in /etc/dnf/dnf.conf. We need /usr/shared/gtk-doc/html
+# to generate proper documentation.
+date '+%Y%m%d-%H%M%S'; yum reinstall -y --setopt='tsflags=' glib2-doc
+
+date '+%Y%m%d-%H%M%S'; ! which dnf || dnf install -y python3-dnf-plugins-core
+date '+%Y%m%d-%H%M%S'; ! which dnf || dnf debuginfo-install -y glib2
+date '+%Y%m%d-%H%M%S';   which dnf || debuginfo-install -y glib2
+
+date '+%Y%m%d-%H%M%S'; contrib/scripts/nm-ci-patch-gtkdoc.sh || true
+
+date '+%Y%m%d-%H%M%S'; test -x /usr/bin/ninja || ! test -x /usr/bin/ninja-build || ln -s /usr/bin/ninja-build /usr/bin/ninja
+
diff --git a/.gitlab-ci/ubuntu-install.sh b/.gitlab-ci/ubuntu-install.sh
new file mode 120000
index 0000000000..39fe97d493
--- /dev/null
+++ b/.gitlab-ci/ubuntu-install.sh
@@ -0,0 +1 @@
+debian-install.sh
\ No newline at end of file