Skip to content
Published On:
Feb 8, 2014
Last Updated:
Jan 15, 2025

GitLab is a web-based “dev ops” software system that provides Git server based hosting, bug tracking, CICD and more for software projects. It is a strong competitor to GitHub, especially for commercial uses.

The GitLab logo.

Container Registries

Each repo can have it’s own container registry. You can use this to store Docker images relating to the project.

CI_REGISTRY_IMAGE is provided and is the base path for the container registry.

You can log into the repo’s container registry inside a pipeline with the command:

Terminal window
echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER $CI_REGISTRY --password-stdin

Then you be able to push a built image:

Terminal window
docker push "$CI_REGISTRY_IMAGE"

You can download images from the GitLab container registry to your local machine by first logging in:

Terminal window
docker login registry.gitlab.com

You will be prompted to enter your username and password. If you have 2FA enabled, you will have to substitute your password with a personal access token.

You can then download and run an image with:

Terminal window
docker run -it registry.gitlab.com/<org_name>/<project_path> bash

GitLab CI Files

Includes

You can include files in your .gitlab-ci.yml file with the include keyword. This is useful for sharing common configuration between multiple projects.

include:
- project: 'my-group/my-project'
file: '/path/to/file.yml'

You can also use the ref to specify a branch, tag or commit SHA when including a file. This is good for “version control” of your include dependencies.

include:
- project: 'my-group/my-project'
ref: 'main' # Branch name
file: '/path/to/file.yml'
- project: 'my-group/my-project'
ref: 'v1.0.0' # Tag name
file: '/path/to/file.yml'
- project: 'my-group/my-project'
ref: 'b6901dcb0918db67a81b190d24fa67f0b5f99fdb' # Commit SHA
file: '/path/to/file.yml'

Conditionally Running Jobs

Sometimes you will only want to run a job under certain conditions. You can do this by using the rules keyword.

Regex can be used with the rules keyword by using =~.

my_job:
rules:
- if: $CI_COMMIT_TAG =~ /^docker-image-v.*$/
script:
- echo "This job will only run for commits that are tagged with docker-image-v1, docker-image-v2 etc."

Running Jobs In Docker Containers

You can tell GitLab to run the job inside a Docker container by using the image keyword.

my_job:
image:
name: my-docker-image
services:
- docker:dind
script:
- echo "This job will run in a Docker container with the image my-docker-image"

The default Docker user will be used by default. If you want to override this, you can change the structure of the image key a bit and add a docker/user key as shown below.

my_job:
image:
name: my-docker-image
docker:
user: root
script:
- echo "This job will run in a Docker container with the image my-docker-image and as user root"

Running Pipelines Locally

GitLab does not have official support for running pipelines locally, but there is a popular third-party tool called gitlab-ci-local (a.k.a. GCL) which allows you to do this.

Installation

To install gitlab-ci-local on Debian-based UNIX distributions (e.g. Ubuntu), you can run the following command2:

Terminal window
sudo wget -O /etc/apt/sources.list.d/gitlab-ci-local.sources https://gitlab-ci-local-ppa.firecow.dk/gitlab-ci-local.sources
sudo apt-get update
sudo apt-get install gitlab-ci-local

Running a Job

Clone the repository that contains the pipeline (i.e. has a .gitlab-ci.yml file) and run the following command:

Terminal window
gitlab-ci-local <job_name>

This will run the job locally and output the results to the console. You may need authentication for the job to run successfully, see the section below for details.

However, this will not run the job’s dependencies, which might be what you were expecting. To run the job and all of it’s dependencies, add the --needs flag:

Terminal window
gitlab-ci-local <job_name> --needs

gitlab-ci-local will cache the “dot env” of dependencies, so you don’t need to keep using --needs during testing if the dependencies haven’t changed. By default, gitlab-ci-local stores the cache of the artifacts at .gitlab-ci-local/artifacts/<job_name>. This is quite a convenient place to check if you need to debug a job.

Manual Jobs

You can trigger a manual job to run during the execution of another job with --manual <job_name>. For example:

Terminal window
gitlab-ci-local build_firmware --manual my_manual_job

GitLab Variables

When a pipeline is running on GitLab’s servers, a number of pre-defined environment variables get provided that the pipeline scripts can use. When running them locally, gitlab-ci-local will do it’s best to provide sensible defaults for these variables. However those dealing with authentication and other URLs will not work, as gitlab-ci-local cannot work out what these should be. In these cases, you can create a file called .gitlab-ci-local-variables.yml in the root directory of the repository and manually provide the variables.

.gitlab-ci-local-variables.yml
---
MY_VAR_1: <value_1>
MY_VAR_2: <value_2>

CI_REGISTRY defaults to https://registry-1.docker.io/v2/.

CI_PROJECT_DIR will be set to /gcl-builds, which should be where the root directory of the repository resides.

If .gitlab-ci-local-variables.yml ends up containing sensitive information like personal access tokens, do not commit this file to the repository! Make sure to add it to your .gitignore file.

Variables In Variables

It doesn’t look like gitlab-ci-local supports defining variables using other variables. For example, this is works when running on GitLab:

variables:
MY_PATH: $MY_PATH /another/path

But when this same job is run locally with gitlab-ci-local, the variable does not get updated, as just seems to set to what ever value it was as if this expression was never run.

Authentication

Normal pipelines that are run on GitLab’s servers get provided with a CI_JOB_TOKEN which can be used to authenticate when pulling submodules that are other private GitLab repositories, or when pulling/pushing to the GitLab container registry. If you want access when running locally, first create a personal access token through the GitLab web interface (with the necessary permissions). Then add it to your .gitlab-ci-local-variables.yml file as shown:

.gitlab-ci-local-variables.yml
---
CI_JOB_TOKEN: <your_personal_access_token>

Image Registry

If you access the repos image registry (or another GitLab repo’s image registry) you will need to set both CI_JOB_TOKEN and CI_REGISTRY_IMAGE in .gitlab-ci-local-variables.yml.

.gitlab-ci-local-variables.yml
---
CI_JOB_TOKEN: <your_personal_access_token>
CI_REGISTRY_IMAGE: <your_registry_image>

CI_REGISTRY_IMAGE should be in the format registry.gitlab.com/<namespace>/<project>, e.g. registry.gitlab.com/my-group/my-project. You can copy the registry URL from the GitLab web interface by going to the image registry and copying any one image URL to the clipboard, and then removing the :tag from the end of the URL.

Watch Out For Untracked Files

GCL will not sync local files that are not tracked by Git. So if you are playing around and add a completely new file, you might run into a “file not found” error when running your pipeline. To fix this, all you need to do is run git add ... to stage the file in git (it doesn’t need to be committed)2.

The file does not need to be continuously re-added to the git staging area when changes are made, as long as the file has been staged (or committed) at least once.

Caching

gitlab-ci-local does not support the GitLab caching feature. gitlab-ci.yml statements like:

cache:
paths:
- path/to/cache

will not be cached when running the job locally with gitlab-ci-local, and so expect things to run slower in this situation.

GitLab Runners

GitLab runners are the machines/applications that run your CI pipelines. GitLab runners can be configured and run on GitLab’s servers (GitLab-hosted runners) or self-hosted on your own servers.

You use tags (GitLab tags, not Git tags) to specify what runner should run a job. GitLab-hosted runners will by default (an untagged job) use the small Linux x86-64 runner3.

GitLab-hosted runners support Linux, Windows and MacOS. As of November 2024, the Windows and MacOS runners are in beta3.

To use a Windows runner, you would use the saas-windows-medium-amd64 tag as follows:

my_windows_job:
tags:
- saas-windows-medium-amd64

Cleaning Up The Repo

You might find that you have committed too many large files to a repository and want to get rid of them. Just deleting them from the repository and committing will not work, as Git will still keep a copy of the file in the repository history.

A screenshot of the project information shown on the landing page of a large GitLab repository.

To permanently remove files from a repositories history, GitLab provides a feature called Repository maintenance which can be used alongside the git-filter-repo tool.4

You may get the error shown below if you don’t run the git-filter-repo commands from a fresh clone of the repository.

$ python .\git-filter-repo --path .\releases\ --invert-paths
Aborting: Refusing to destructively overwrite repo history since
this does not look like a fresh clone.
(expected freshly packed repo)
Please operate on a fresh clone instead. If you want to proceed
anyway, use --force.

You should be able to fix this by performing a fresh clone of the repo. If this will take a long time from the internet and you have a local copy already, you can use:

Terminal window
git clone ./existing-path-to-repo/ ./new-clone-path/ --no-local

The --no-local flag is needed so that the .git directory is in the same state as if cloned from the internet.

Now you need to run git-filter-repo to remove the large files as needed. For example, let’s pretend someone was lazy and rather than releasing build artifacts correctly (e.g. releases section of cloud Git hosting or file storage), decided to commit them to the repository. This folder got rather large and now you want to remove it (and all the Git history associated with it). To do this, you can run:

Terminal window
python .\git-filter-repo --path releases --invert-paths

git-filter-repo creates a commit map at .git/filter-repo/commit-map whenever it is run. This file will look something like this:

old new
00df40253a7d17435864fe5a005f6f0469ef13b2 77966bbe6fc535c8a2b30d0d880a72c583ba0d55
00e98efe0aa8a60c4b69b06918b882718f747bb9 73890befb43ab56b81a27f6e8c874f326c4cc67b
044801170462ad46b077871557a9101964466edc 044801170462ad46b077871557a9101964466edc
0527cd140f2e071093581b61bff8faa768c17ae2 ee4ed5aead68b4c149e73481f8a05b38ab777ee6

Make sure to back this up somewhere as we will be uploading it to GitLab once we have pushed the repo changes.

Next, we need to push the changes that git-filter-repo made locally to the remote repository. You might need to re-add origin as git-filter-repo can remove it for safety reasons. If so, run:

Terminal window
git remote add origin https://gitlab.com/my-group/my-project.git

Then push the changes to the remote repository:

Terminal window
git push origin --force 'refs/heads/*'
git push origin --force 'refs/tags/*'
git push origin --force 'refs/replace/*'

Then, you can run the GitLab cleanup process. The GitLab cleanup process does the following:4

  • Removes any internal Git references to old commits.
  • Runs git gc --prune=30.minutes.ago against the repository to remove unreferenced objects.
  • Unlinks any unused LFS objects attached to your project, freeing up storage space.
  • Recalculates the size of your repository on disk.

Go to Settings > Repository > Repository maintenance in the GitLab web interface. Upload the commit-map file and then click “Start cleanup” as shown in This is a placeholder for the reference: gitlab-ui-repo-maintenance-start-cleanup.

A screenshot of the GitLab UI showing the Repository maintenance page and the “Start cleanup” button.

Footnotes

  1. GitHub. firecow/gitlab-ci-local [repo README]. Retrieved 2024-10-02, from https://github.com/firecow/gitlab-ci-local. 2

  2. GitLab. GitLab-hosted runners [docs]. Retrieved 2024-11-11, from https://docs.gitlab.com/ee/ci/runners/index.html. 2

  3. GitLab. Repository size [documentation]. Retrieved 2025-01-15, from https://docs.gitlab.com/ee/user/project/repository/repository_size.html. 2