Variables in pipelines

Variables in pipelines allow you to parameterize the way your pipeline works.
Codefresh provides a set of system variables, and also provides the capability to define custom variables.

• System variables are predefined variables prefixed with CF.
  System variables are automatically included in every pipeline build. Here are some examples of system variables.
  • CF_BRANCH is the Git branch that was used for this pipeline.
  • CF_REVISION is the Git hash that was used for this pipeline.
  • CF_BUILD_URL is the url of the pipeline build.
• User-defined variables are custom variables which you create.
  You can create user-defined variables for different entities such as projects, pipelines and steps, with or without default values. You can also import variables you may have already defined.

Codefresh supports two syntaxes for variables: the UNIX syntax, and a proprietary syntax for use in YAML files.

Using Codefresh variables in pipelines

There are two ways to use a Codefresh variable in your pipelines:

1. Expose as UNIX environment variables
   This is the default method. All variables in all freestyle steps are exposed as UNIX environment variables. For example, $MY_VARIABLE_EXAMPLE.
2. Directly in YAML properties
   This format, specific to Codefresh, can be used in in YAML properties with the syntax ${{MY_VARIABLE_EXAMPLE}}.

Example: Print out the branch as an environment variable

In this example, we use simple echo commands, but any program or script that reads environment variables can also read them in the same manner.

YAML

MyOwnStep:
  title: Variable example
  image: alpine
  commands:
    - echo $CF_BUILD_ID
    - echo $CF_BRANCH_TAG_NORMALIZED

Example: Use variable directly in YAML properties

YAML

MyAppDockerImage:
  title: Building Docker Image
  type: build
  image_name: my-own-app
  tag: ${{CF_BRANCH_TAG_NORMALIZED}}

Example: Concatenating variables

YAML

MyAppDockerImage:
  title: Building Docker Image
  type: build
  image_name: my-own-app
  tag: ${{CF_BRANCH_TAG_NORMALIZED}}-${{CF_SHORT_REVISION}}

The concatenation above creates Docker images with tags such as:

master-df6a04c
develop-ba1cd68
feature-vb145dh

NOTE
This syntax is specific to Codefresh, and is only available within the Codefresh YAML file itself.
To write scripts or programs that use Codefresh variables, you need to make them aware of the environment variable form.

System variables

System variables are automatically injected to any freestyle step as environment variables.

Git-based system variables

The values of Git-based system variables, such as CF_BRANCH, CF_COMMIT_MESSAGE, CF_REVISION etc. are retrieved directly from the Git provider you use, and therefore have the same limitations of that provider.

For example, GitLab sends less information in Pull Request (PR) events than normal pushes, and Bitbucket sends only the short hash of a commit in PR events. We suggest you read the documentation of your Git provider first to understand what information is available for every Git event.

Gerrit for example, has change-Id and Changes that you can map to CF_PULL_REQUEST variables, such as CF_PULL_REQUEST_ID and more.

The table below describes the system variables.

Context-related variables

Context-related variables are system variables created dynamically during the workflow execution, according to the steps used in the pipeline.

A very common pattern in Codefresh pipelines, is to create a Docker image in one step, and then run a command on its container in the next step (e.g. run unit tests):

YAML

version: '1.0'
steps:
  MyAppDockerImage:
    title: Building Docker Image
    type: build
    image_name: my-own-app
  MyUnitTests:
    title: Running Unit tests
    image: ${{MyAppDockerImage}}
    commands:
      - ./my-unit-tests.sh

In the example above, you can see the MyAppDockerImage variable that denotes a Docker image created dynamically within this single pipeline. In the second step we use it as a Docker context in order to run unit tests. See also the unit testing example app.

Step variables

Every step in a Codefresh pipeline also exposes several built-in variables. You can access them via the global steps parent variable.

Each step creates a variable based on the name of the step. You can then use the members of each variable for status conditions such as: steps.MyUnitTests.result == 'error' for a step called MyUnitTests.

To access variables that have non-standard names with only alphanumeric and underscore (_) characters, use the Variable() function.

Step-member variables

Variables that are created by steps can have members. The members depend on the step type.
For example, if you have a build step named myBuildStep, you can get the ID of the docker image that is created, with echo ${{steps.myBuildStep.imageId}}.

GitHub release variables

GitHub allows you to create releases to mark specific Git tags for general availability.

You can set a trigger for GitHub releases. When there is a GitHub release, the following variables are also available:

GitHub Pull Request variables

The table lists the variables available when a PR (Pull Request) is closed in GitHub.

Gerrit changeId & change message variables

Gerrit has no explicit concept of PRs as in other version control systems to map trigger event payloads to builds. Instead, Gerrit uses Changes which serves a similar purpose and functionality as PRs. You can achieve the same functionality in Codefresh with our CF_PULL_REQUEST group of environment variables.

User-defined variables

User-defined variables are custom variables you add to CI entities such as projects, pipelines, build triggers, and manual build runs. When adding a user-defined variable, you can create a:

• Variable
  A standard variable whose value is visible in plain text unless explicitly encrypted. You can create these manually, import them from a file with predefined values, or define empty variables without assigned values.
• Secret
  A variable whose value is automatically encrypted and masked in logs.

NOTE
Secret variables are currently not supported in on-premises environments.

Methods to define user-defined variables

These are the options available to add user-defined variables.

• Add single variable
  Supported for all entities, this method allows you to add one variable at a time, with or without values. You cannot encrypt empty variables.
• Add multiple variables
  Add a set of variables by entering them manually, or by pasting them into the variable editor, one variable per line.
  This is a quick option to add variables defined locally or in specific environments to a pipeline.
  You simply copy the set of variables and paste them into the text editor.
• Import from file
  Add predefined variables in bulk by importing from a file.

Empty user-defined variables

If you add variables with just the key definitions, the values are automatically populated during pipeline execution or can be defined manually prior to build runs. You can add empty variables to:

• Projects
• Pipelines
• Build triggers

NOTE
Encryption is not supported for empty variables.

Order of precedence for user-defined variables

Because you can add user-defined variables to different entities, you can have the same variables at different levels.
If the variable with the same name is defined for multiple entities, the override rules are based on the priority of the variable. Variables for entities with higher priority override those for entities with lower priority.

For example if a pipeline variable is defined both within a project, and as an execution parameter of a specific build, the final result will be the value of the variable defined as a build parameter. The project-level variable is ignored.

Listed below are the different levels for user-defined variables in order of priority, from the highest to the lowest.

1. Steps
   • export command
     Within the current step using export, or in any subsequent step using cf_export commands.
   • freestyle steps with environment field.

   NOTE
   Step-level variables with export take precedence over freestyle-step variables with environment.

2. Builds
   • Within a specific build execution from the Codefresh UI or from the CLI.
3. Pipeline
4. Shared-configurations
5. Projects

The variables are injected into pipelines from different sources and at different levels. To view the variables actually used by a specific build of the pipeline, see Viewing variables in pipeline builds.

Add user-defined variables

1. Select the entity to which to add variables:
   • Projects
   • Pipelines
   • Steps
   • Builds
2. Select the type of variable to add: Variable or Secret.

Standard variables

• Add single or multiple variables manually as key-value pairs, or import them in bulk from a file.
• To encrypt a variable, click Encrypt, and confirm.

• Secret variables
  • Secrets are automatically encrypted and become immutable once saved.
  • Decrypting a secret resets its value to empty.

Projects

From the list of Projects, select the Project to which to add variables, click the Settings icon, and then click the Variables tab.

Pipelines

• From the list of pipelines, select the Pipeline to which to add variables.
• Click the Settings icon, click the Workflow tab, and then click Variables on the right.

Steps

• Select the pipeline.
• In the Workflow tab, add the variable with the correct syntax to the step as required.

Builds

• From the Builds page, in the Pipelines column, click the pipeline name.
• Click Run.
• Expand Build Variables.

Exporting environment variables from a freestyle step

Steps defined within steps are scoped to the step within which they were created, even when using the export command.
To use variables across steps, we provide a shared file through which you can import and export variables.

There are two ways to add variables to the shared file:

• Using the cf_export command
• Writing directly to the file

Exporting variables with cf_export

Within every freestyle step, the cf_export command allows you to export variables across steps by writing to the shared variables file.

NOTE The default behavior for cf_export is to override the variable value passed at the pipeline level. Depending on when your account was created, existing variables may not inherit the default override behavior.
If you are experiencing situations where you are passing a variable to a pipeline, resetting the value via cf_export, and not seeing the desired value for future steps, please contact Support. Support can verify the behavior, and enable the Feature Flag for the account.

You can either:

• Explicitly state a VAR=VAL pair
• State the name of an existing exported environment variable, for example, EXISTING_VAR

version: '1.0'
steps:
  freestyle-step-1:
    description: Freestyle step..
    title: Free styling
    image: alpine:latest
    commands:
      # Normal export will only work in a single step
      - export EXISTING_VAR=www.example.com

      # CF export will now work in all other subsequent steps
      - cf_export VAR1=alpine:latest VAR2=VALUE2 EXISTING_VAR

  freestyle-step-2:
    description: Freestyle step..
    title: Free styling 2
    image: ${{VAR1}}
    commands:
      - echo $VAR2
      - echo http://$EXISTING_VAR/index.php

By default, cf_export variables work only on subsequent steps, meaning the steps that follow the step in which the variable is defined.
To export a variable both to the current step and to all the remaining steps, do the following:

export MY_VAR='example' # Makes MY_VAR available in this step only
cf_export MY_VAR='example' # Makes MY_VAR available also to all steps after this one

cf_export syntax

cf_export has the same syntax as the bash export command. This means that when you use cf_export, you don’t need to add dollar signs for the variable that is created/assigned.

cf_export $MY_VAR # Don't do this
cf_export MY_VAR # Correct syntax

Masking variables within cf_export

Mask variables within cf_export by defining the --mask flag.
Values of masked variables in cf_export commands are replaced with asterisks in the build logs. This helps to ensure that sensitive information is not exposed also in the variables list, in addition to the logs.

Here is an example with standard and masked versions of the same variable in cf_export commands.

version: '1.0'
steps:
  freestyle-step:
    description: Freestyle step..
    title: Free styling
    image: alpine:latest
    commands:
      - export EXISTING_VAR=some-value

      - cf_export VAR1=alpine:latest VAR2=VALUE2 EXISTING_VAR --mask #masked version

Multi-line variables with cf_export

When exporting a multi-line variable using cf_export, to prevent truncation, encode it in base64.

version: "1.0"
steps:
  assign:
    image: alpine
    commands:
      - apk add --update coreutils
      - export TEST=`echo "line1" && echo "line2" && echo "line3"`
      - echo $TEST
      - cf_export TEST=`echo $TEST | base64 -w 0`
  test:
    image: alpine
    commands:
      - echo $TEST
      - echo `echo $TEST | base64 -d`

Directly writing to the file

For more advanced use cases, you can write directly to the shared variable file that Codefresh reads to understand which variables need to be available to all steps.

This file has a simple format where each line is a variable, and its value in the form of VARIABLE=VALUE.
The cf_export command mentioned in the previous section is just a shorthand for writing on this file.

The variables file is available inside freestyle steps in the following path: ${{CF_VOLUME_PATH}}/env_vars_to_export.

version: '1.0'
steps:
  freestyle-step-1:
    description: Freestyle step..
    title: Free styling
    image: alpine:latest
    commands:
      - echo VAR1=192.168.0.1 >> ${{CF_VOLUME_PATH}}/env_vars_to_export
      - echo hey=alpine:3.9 >> ${{CF_VOLUME_PATH}}/env_vars_to_export

  freestyle-step-2:
    description: Freestyle step..
    title: Free styling 2
    image: ${{hey}}
    commands:
      - echo http://$VAR1/index.php

Use this technique if you have complex expressions that have issues with the cf_export command.

Exporting external link with CF_OUTPUT_URL

Use the CF_OUTPUT_URL variable to export any external in the form of a URL.

This variable can be useful to output the URL of the parent pipeline, and navigate from the child to the parent build. Remember that Codefresh has native support to trigger child builds from parent builds, and to navigate from the parent to the child build through the codefresh-run plugin (link).

Add CF_OUTPUT_URL with cf_export

Simply add a step to the child build with an in-step link to the parent build. The URL link to the parent build is displayed as part of the step details in the Builds page.

Create a link to the parent-build using cf_export and CF_OUTPUT_URL.

Example step in child build to output link to parent

Add the following step at the beginning of the pipeline.

NOTE
The name of the variable exported by cf_export must have the format <name|key_of_step>_CF_OUTPUT_URL. It’s value is the URL of the parent build to link to.

    steps:
      ...
      linkToParentBuild:
        image: codefresh/cli
        commands:
          - export parentBuildId=$(codefresh get annotation build ${{CF_BUILD_ID}} cf_predecessor -o json | jq -r '.value')
          - cf_export linkToParentBuild_CF_OUTPUT_URL="${{CF_URL}}/build/${parentBuildId}"
      ...

On the Builds page, the step details of the child build displays the Output URL which is the link to the parent build, as in the image below.

Masking variables in logs

Codefresh has the built-in capability to automatically mask encyrpted variables in logs. In build logs, the values of encrypted variables are replaced with asterisks.

The variables can be defined in any of the usual ways Codefresh offers, such as shared configuration or within the pipeline:

NOTE This feature is currently available only in Enterprise accounts.

Escape special characters in variables

When passing special characters through environment variables, use \ as an escape character.
For example, to pass a Cassandra connection string, you might do something like Points\=hostname\;Port\=16376\;Username\=user\;Password\=password.

This will safely escape ; and =.

Related articles

Steps in pipelines
Conditional execution of steps