GitLab

Authors:

Published On:

Feb 8, 2014

Last Updated:

Oct 15, 2024

GitLab is a web-based “dev ops” software system that provides Git server based hosting, bug tracking, CICD and more for software projects. It is a strong competitor to GitHub, especially for commercial uses.

The GitLab logo¹.

Container Registries

Each repo can have it’s own container registry. You can use this to store Docker images relating to the project.

CI_REGISTRY_IMAGE is provided and is the base path for the container registry.

You can log into the repo’s container registry inside a pipeline with the command:

echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin

Then you be able to push a built image:

docker push "$CI_REGISTRY_IMAGE"

You can download images from the GitLab container registry to your local machine by first logging in:

docker login registry.gitlab.com

You will be prompted to enter your username and password. If you have 2FA enabled, you will have to substitute your password with a personal access token.

You can then download and run an image with:

docker run -it registry.gitlab.com/<org_name>/<project_path> bash

GitLab CI Files

Includes

You can include files in your .gitlab-ci.yml file with the include keyword. This is useful for sharing common configuration between multiple projects.

include:
  - project: 'my-group/my-project'
    file: '/path/to/file.yml'

You can also use the ref to specify a branch, tag or commit SHA when including a file. This is good for “version control” of your include dependencies.

include:
  - project: 'my-group/my-project'
    ref: 'main' # Branch name
    file: '/path/to/file.yml'
  - project: 'my-group/my-project'
    ref: 'v1.0.0' # Tag name
    file: '/path/to/file.yml'
  - project: 'my-group/my-project'
    ref: 'b6901dcb0918db67a81b190d24fa67f0b5f99fdb' # Commit SHA
    file: '/path/to/file.yml'

Conditionally Running Jobs

Sometimes you will only want to run a job under certain conditions. You can do this by using the rules keyword.

Regex can be used with the rules keyword by using =~.

my_job:
  rules:
    - if: $CI_COMMIT_TAG =~ /^docker-image-v.*$/
  script:
    - echo "This job will only run for commits that are tagged with docker-image-v1, docker-image-v2 etc."

Make sure you don’t wrap the regex in quotes. This is still a valid rule, but will not invoke the regex engine. For example, don’t do this:

  rules:
    - if: $CI_COMMIT_TAG =~ "^docker-image-v.*$"

Running Jobs In Docker Containers

You can tell GitLab to run the job inside a Docker container by using the image keyword.

my_job:
  image:
    name: my-docker-image
  services:
    - docker:dind
  script:
    - echo "This job will run in a Docker container with the image my-docker-image"

Remember to always add the docker:dind (Docker in Docker) service to services if you are using Docker images. This starts the Docker daemon inside the container.

  services:
    - docker:dind

The default Docker user will be used by default. If you want to override this, you can change the structure of the image key a bit and add a docker/user key as shown below.

my_job:
  image:
    name: my-docker-image
    docker:
      user: root
  script:
    - echo "This job will run in a Docker container with the image my-docker-image and as user root"

Running Pipelines Locally

GitLab does not have official support for running pipelines locally, but there is a popular third-party tool called gitlab-ci-local (a.k.a. GCL) which allows you to do this.

Installation

To install gitlab-ci-local on Debian-based UNIX distributions (e.g. Ubuntu), you can run the following command²:

sudo wget -O /etc/apt/sources.list.d/gitlab-ci-local.sources https://gitlab-ci-local-ppa.firecow.dk/gitlab-ci-local.sources
sudo apt-get update
sudo apt-get install gitlab-ci-local

Running a Job

Clone the repository that contains the pipeline (i.e. has a .gitlab-ci.yml file) and run the following command:

gitlab-ci-local <job_name>

This will run the job locally and output the results to the console. You may need authentication for the job to run successfully, see the section below for details.

However, this will not run the job’s dependencies, which might be what you were expecting. To run the job and all of it’s dependencies, add the --needs flag:

gitlab-ci-local <job_name> --needs

gitlab-ci-local will cache the “dot env” of dependencies, so you don’t need to keep using --needs during testing if the dependencies haven’t changed. By default, gitlab-ci-local stores the cache of the artifacts at .gitlab-ci-local/artifacts/<job_name>. This is quite a convenient place to check if you need to debug a job.

Manual Jobs

You can trigger a manual job to run during the execution of another job with --manual <job_name>. For example:

gitlab-ci-local build_firmware --manual my_manual_job

GitLab Variables

When a pipeline is running on GitLab’s servers, a number of pre-defined environment variables get provided that the pipeline scripts can use. When running them locally, gitlab-ci-local will do it’s best to provide sensible defaults for these variables. However those dealing with authentication and other URLs will not work, as gitlab-ci-local cannot work out what these should be. In these cases, you can create a file called .gitlab-ci-local-variables.yml in the root directory of the repository and manually provide the variables.

---
MY_VAR_1: <value_1>
MY_VAR_2: <value_2>

CI_REGISTRY defaults to https://registry-1.docker.io/v2/.

CI_PROJECT_DIR will be set to /gcl-builds, which should be where the root directory of the repository resides.

If .gitlab-ci-local-variables.yml ends up containing sensitive information like personal access tokens, do not commit this file to the repository! Make sure to add it to your .gitignore file.

Variables In Variables

It doesn’t look like gitlab-ci-local supports defining variables using other variables. For example, this is works when running on GitLab:

variables:
  MY_PATH: $MY_PATH /another/path

But when this same job is run locally with gitlab-ci-local, the variable does not get updated, as just seems to set to what ever value it was as if this expression was never run.

Authentication

Normal pipelines that are run on GitLab’s servers get provided with a CI_JOB_TOKEN which can be used to authenticate when pulling submodules that are other private GitLab repositories, or when pulling/pushing to the GitLab container registry. If you want access when running locally, first create a personal access token through the GitLab web interface (with the necessary permissions). Then add it to your .gitlab-ci-local-variables.yml file as shown:

---
CI_JOB_TOKEN: <your_personal_access_token>

Image Registry

If you access the repos image registry (or another GitLab repo’s image registry) you will need to set both CI_JOB_TOKEN and CI_REGISTRY_IMAGE in .gitlab-ci-local-variables.yml.

---
CI_JOB_TOKEN: <your_personal_access_token>
CI_REGISTRY_IMAGE: <your_registry_image>

CI_REGISTRY_IMAGE should be in the format registry.gitlab.com/<namespace>/<project>, e.g. registry.gitlab.com/my-group/my-project. You can copy the registry URL from the GitLab web interface by going to the image registry and copying any one image URL to the clipboard, and then removing the :tag from the end of the URL.

Watch Out For Untracked Files

GCL will not sync local files that are not tracked by Git. So if you are playing around and add a completely new file, you might run into a “file not found” error when running your pipeline. To fix this, all you need to do is run git add ... to stage the file in git (it doesn’t need to be committed)².

The file does not need to be continuously re-added to the git staging area when changes are made, as long as the file has been staged (or committed) at least once.

Caching

gitlab-ci-local does not support the GitLab caching feature. gitlab-ci.yml statements like:

cache:
  paths:
    - path/to/cache

will not be cached when running the job locally with gitlab-ci-local, and so expect things to run slower in this situation.

GitLab Runners

GitLab runners are the machines/applications that run your CI pipelines. GitLab runners can be configured and run on GitLab’s servers (GitLab-hosted runners) or self-hosted on your own servers.

You use tags (GitLab tags, not Git tags) to specify what runner should run a job. GitLab-hosted runners will by default (an untagged job) use the small Linux x86-64 runner³.

GitLab-hosted runners support Linux, Windows and MacOS. As of November 2024, the Windows and MacOS runners are in beta³.

To use a Windows runner, you would use the saas-windows-medium-amd64 tag as follows:

my_windows_job:
  tags:
    - saas-windows-medium-amd64

GitLab (2015, Jul 3). Our new logo [blog post]. Retrieved 2024-10-14, from https://about.gitlab.com/blog/2015/07/03/our-new-logo/. ↩
GitHub. firecow/gitlab-ci-local [repo README]. Retrieved 2024-10-02, from https://github.com/firecow/gitlab-ci-local. ↩ ↩²
GitLab. GitLab-hosted runners [docs]. Retrieved 2024-11-11, from https://docs.gitlab.com/ee/ci/runners/index.html. ↩ ↩²