Selectors for concurrency synchronization
Argo Workflows has a synchronization mechanism to limit parallel execution of specific workflows or templates within workflows, as required.
The mechanism enforces this with either semaphore or mutex synchronization configurations. For detailed information, see Synchronization.
Codefresh supports an additional level of concurrency synchronization, with selectors, for both workflows and templates.
Selectors enable access to all workflow properties such as annotations
, labels
, or parameters
using Go template formats. Selector-based concurrency configurations support parallel executions of workflows or templates within workflows. And equally important, selectors work with both semaphore and mutex synchronization configurations.
This article has examples for semaphore synchronization configurations.
Semaphore synchronization is configured in the ConfigMap
, which is referenced from workflows or templates within workflows.
Here’s an example of the ConfigMap
with the concurrency definition:
apiVersion: v1
kind: ConfigMap
metadata:
name: semaphore-config
data:
workflow: "1"
Concurrency selector definitions
A concurrency synchronization selector is defined through a name
-template
pair in workflows or templates:
- The
name
is any meaningful/logical name that describes the selector. For example, Git repo or branch. - The
template
is the parameter mapping for the name, that is resolved to the selector value when the pipeline is run. The parameter mapping format must confirm to Sprig templates. For information on Go templates and Sprig functions, see the Go template package documentation, and Sprig Function Documentation. For example:
or
Selectors are added to the synchronization
section, under selectors
, as in the sample YAML below.
When and how does selector concurrency configuration work?
Let’s review a viable use case for selector-based concurrency synchronization, and different scenarios to illustrate when and how it works:
- Default concurrency synchronization without selectors
- Selector concurrency synchronization resolving to the same
template
values - Selector concurrency synchronization resolving to different
template
values
Use case
We have two workflows, synchronization-wf-1
and synchronization-wf-2
.
Both workflows commit to the same Git repo, but to different branches in the repo. The first workflow commits to the main
branch, and the second workflow commits to a feature
branch. When there are commits to the main
and feature
branches, we want the workflows to run in parallel.
Default concurrency synchronization
Both workflows have the same default values for the arguments, and share the same semaphore configuration and key with weight 1.
With the default concurrency configuration, only one workflow at a time can use the key. Meaning that while synchronization-wf-1
is Running
, synchronization-wf-2
is in Pending
status.
Selector concurrency synchronization resolving to same template values
Here are the same workflows, synchronization-wf-1
and synchronization-wf-2
, this time configured with selector concurrency.
Both workflows share the same semaphore key with a weight of 1.
Both workflows also use the same set of selectors. As you can see in the image below, the Git repo and the Git branch arguments have the same values in both.
In this case, the selectors have no impact as the selector templates
in both workflows resolve to the same values, resulting in the same semaphore key.
If you look at the status of both workflows, you can see that they have the same semaphore key.
synchronization:
semaphore:
holding:
- holders:
- synchronization-wf-6lf8b
semaphore: argo/ConfigMap/semaphore-config/workflow?repository=denis-codefresh/argo-workflows&branch=main
Selector concurrency synchronization resolving to different template values
In the third scenario, we have the same workflows, synchronization-wf-1
and synchronization-wf-2
, also configured with the selector concurrency.
They also share the same semaphore configuration and key with weight 1.
The difference is that in this scenario, the selector templates
in both workflows resolve to different values, resulting in different semaphore keys for each workflow.
In synchronization-wf-1
, the Git branch is set to main
, and in synchronization-wf-2
, it is set to feature
.
Now, because the workflows use different semaphore keys, both workflows run in parallel.
If you look at the status of both workflows, you can see that they have different semaphore keys.
synchronization:
semaphore:
holding:
- holders:
- synchronization-wf-6lf8b
semaphore: argo/ConfigMap/semaphore-config/workflow?repository=denis-codefresh/argo-workflows&branch=main
synchronization:
semaphore:
holding:
- holders:
- synchronization-wf-8lf9b
semaphore: argo/ConfigMap/semaphore-config/workflow?repository=denis-codefresh/argo-workflows&branch=feature