GitLab Runner is a popular open source CI tool that can be used for parallel builds of software. It has a built-in job runner that can be used to run jobs in parallel. However, there are some steps you can take to manage GitLab Runner Concurrency for Parallel CI Jobs.

  1. Configure GitLab Runner Configure GitLab Runner so that it runs in parallel mode. This will allow you to run multiple jobs at the same time and share the results between them. To do this, add the following line to your .gitlab-runnerrc file: config.parallel = true

GitLab’s Continuous Integration (CI) pipelines are a popular way to automate builds, tests, and releases each time you push code to your repository. Pipelines run concurrently and consist of sequential stages; each stage can include multiple jobs that run in parallel during the stage. The maximum concurrency of both parallel jobs and cross-instance pipelines depends on your server configuration.

Jobs are run by GitLab Runner instances. Runners operate as isolated processes that receive new jobs from their controlling GitLab server. When a job is issued, the runner will create a sub-process that executes the CI script. There are multiple variables that control when a runner will accept a job and start executing it. In this guide we’ll look at the ways you can configure parallel jobs and pipelines.

Increasing the Runner Count

One way to allow more jobs to run simultaneously is to simply register more runners. Each installation of GitLab Runner can register multiple distinct runner instances. They operate independently of each other and don’t all need to refer to the same coordinating server.

Use the gitlab-runner register command to add a new runner:

You’ll be prompted to supply the registration information from your GitLab server. You can find this on the Settings > CI/CD page of a GitLab project or group, or head to Overview > Runners in the Admin Centre for an instance-level runner. Runners will only execute jobs originating within the scope they’re registered to.

Each registered runner gets its own section in your /etc/gitlab-runner/config.toml file:

If all three runners were registered to the same server, you’d now see up to three jobs running in parallel.

Raising the Concurrency Limit

You can set the permitted concurrency of a specific runner registration using the limit field within its config block:

This change allows the first runner to execute up to four simultaneous jobs in sub-processes. Registering another runner with limit = 2 would raise the concurrency level to a total of six jobs, assuming both runners referenced the same controlling GitLab server.

Handling “Request Concurrency”

The number of live jobs under execution isn’t the only variable that impacts concurrency. GitLab Runner manages the number of job requests it can accept via the separate request_concurrency variable.

This value controls the number of queued requests the runner will take from GitLab. When the server needs to schedule a new CI job, runners have to indicate whether they’ve got sufficient capacity to receive it. The runner won’t accept the job if it’s already got more queued requests than request_concurrency permits.

Consider this example:

This runner will accept up to four concurrent job requests and execute up to two simultaneously. Additional jobs won’t be taken until the initial two have completed.

The Global Concurrency Level

GitLab Runner also maintains a global concurrency factor that places an overall cap on the limit values exposed by individual registrations. You can control this value with the concurrency setting at the top of your config.toml:

Here the configuration of the two runners suggests a total job concurrency of six. However the presence of the global concurrency setting means no more than four jobs will actually run simultaneously. This value limits the total number of sub-processes that can be created by the entire GitLab Runner installation.

Applying Changes

Once you’ve made the changes you need, you can save your config.toml and return to running your pipelines. Modifications to the file are automatically detected by GitLab Runner and should apply almost immediately.

You can try restarting the GitLab Runner process if the new concurrency level doesn’t seem to have applied:

This stops and starts the GitLab Runner service, reloading the config file.

Arranging Your Pipelines for Parallel Jobs

If your jobs in a single pipeline aren’t being parallelized, it’s worth checking the basics of your .gitlab-ci.yml configuration. It’s only jobs that run concurrently by default, not the pipeline stages:

This pipeline defines three stages that are shown horizontally in the GitLab UI. Each stage must complete before the next can begin. The build and deploy stages have two jobs each. These jobs run in parallel if your runners have enough capacity to stay within their configured concurrency limits.

It is possible to break the “stages execute sequentially” rule by using the needs keyword to build a Directed Acyclic Graph:

Here the iOS deployment is allowed to proceed as soon as the build_ios job has finished, even if the remainder of the build stage has not completed. Using needs makes your pipelines more flexible by adding new opportunities for parallelization. However it also brings along complexity which can be harder to maintain over time as you add more jobs to your pipeline.

What About Caching?

Use of concurrency means your jobs may be picked up by different runners on each pass through a particular pipeline. Runners maintain their own cache instances so a job’s not guaranteed to hit a cache even if a previous run through the pipeline populated one. The cache might reside on a different runner to that executing the second job.

You can address this by setting up a shared cache provider using an S3-compatible object storage system. This will cause caches to be uploaded to that provider after the job completes, storing the content independently of any specific runner. Other runner instances will be able to retrieve the cache from the object storage server even if they didn’t create it.

Shared caching can improve performance by increasing the probability of a cache hit, reducing the work your jobs need to complete. Your pipelines shouldn’t require successful cache resolution though: caches are used on a best-effort basis so CI scripts are meant to be resilient to misses.

Summary

GitLab Runner gives you three primary controls for managing concurrency: the limit and request_concurrency fields on individual runners, and the concurrency value of the overall installation. A particular Runner installation won’t execute more than jobs simultaneously, even if the sum of its registrations’ limit values suggests it could take more.

Adding more runners is another way to impact overall concurrency. In general it’s best to raise the concurrency on an existing runner if you simply want to run more jobs with the same configuration. Add a new runner and set its limit value when you need to execute jobs with a new executor or settings that differ from your existing fleet.