Pipeline definitions YAML

How to define Codefresh pipelines in a declarative manner

Codefresh offers its own built-in format for creating pipelines. The pipeline specification is based on the YAML syntax allowing you to describe your pipelines in a completely declarative manner.

Using Codefresh YAML is the recommended way to create pipelines.

Simple example for codefresh.yml

Here is a very minimal example:

codefresh.yml

version: '1.0'
steps:
  build_image:
    type: build
    description: Building the image...
    image-name: myuser/myservice
    tag: develop # ${{CF_BRANCH}}

  perform_tests:
    image: node:5
    working_directory: ${{main_clone}}
    description: Performing unit tests...
    commands:
      - npm install gulp -g 
      - npm install
      - gulp unit_test

It contains two steps, one named build_image that creates a docker image, and another one called perform_tests that runs unit test with gulp.

If you want to know more about how steps work in Codefresh make sure to read the introduction to pipelines first, before moving on.

Basic pipeline syntax

You can customize your build environment (pipeline) by using the Codefresh YAML file, codefresh.yml. Codefresh uses the build specifications in the codefresh.yml file to execute your build. The codefresh.yml can be basic or it can include intricate build specifications.

A YAML file is comprised of a series of steps that are executed in the order in which they are specified.

codefresh.yml

version: '1.0'

steps:
  step-name:
    [step-contents]
  another-step:
    [step-contents]
  the-very-last-step:
    [step-contents]

You must define a step type for each step, unless you are using a freestyle step. Each step uses Docker images and containers as facilitators for execution. For example, the Freestyle step spins up a container and executes the specified shell commands from the YAML file.

The step names should be unique within the same pipeline. This mainly affects the visualization of the pipeline when it runs.

Each step produces a resource, which you can reference in other steps, and are executed in real-time. For example, a freestyle step can reference an image that was produced by a build step. This allows you to chain steps together and create highly-customized builds.

Variables

You can add variables in the YAML file to chain and reference steps and referencing is possible due to implementation of variables. See Variables in pipelines.

For more information on creating your own step, see the Steps in pipelines.

You can also see the complete YAML specification supported for pipelines. Note however that several fields are only accessible by using the Codefresh API or CLI .

YAML validation

If you are editing Codefresh yaml within the Codefresh UI, the editor will automatically highlight errors as they happen.

This allows you to make quick edits (and possibly run some builds) straight from the GUI. Once you are happy with your pipeline you should commit it to your repository as codefresh.yml (pipeline as code).

You can also validate the pipeline yaml outside of the UI by using the Codefresh CLI. The CLI has a validate parameter that can check one or more files for syntax errors.

$ codefresh validate codefresh.yml
Yaml not valid:
  - "invalid-property" is not allowed

For more information on where the YAML file can be stored, see creating pipelines.

Execution flow

By default, Codefresh will execute all steps in the YAML file and instantly fail the build, if any step presents an error. To change this behavior add the fail_fast:false property in any step that you wish to be ignored in case of errors.

For example, if you have a freestyle step that runs integration tests, and you don’t want the whole pipeline to fail if any of the tests fail, add the fail_fast line to that step:

perform_tests:
    image: node:9
    description: Running integration tests
    fail_fast: false
    commands:
      - gulp integration_test

Now the pipeline will continue to run even if the step perform_tests fails.

Notice also that by default Codefresh pipelines run in sequential mode. All steps will be executed one after the other and in the same order as included in the codefresh.yml file.

If you wish to use parallel steps in your pipelines, see the parallel steps page.

Working directories

In the context of a step, a working directory can be of the following type:

Retrying a step

Sometimes you want to retry a step that has a problem. Network hiccups, transient failures and flaky test environments are common problems that prevent pipelines from working in a predictable manner.

Codefresh allows you to retry any of your steps with the built-in syntax:

yaml

step-name:
    [step-contents]
    retry:
      maxAttempts: 5
      delay: 5
      exponentialFactor: 2

The retry: block has the following parameters, all of which are optional:

• maxAttempts defines the maximum number of retry attempts for the step if there are execution errors, and can range from 1 (the default), to a maximum of 10.
• delay is the interval in seconds between retries, and can range from a minimum of 5 (the default), to a maximum of 60 seconds.
• exponentialFactor is the rate by which to exponentially increase the delay between each retry attempt, and can be from 1 (the default) to a maximum of 5. When the exponentialFactors is greater than 1, each subsequent delay is a multiple of the previous one.

The exponentialFactor works like this:

• exponentialFactor=1 and delay=5 maintains a constant 5-second interval between each retry, regardless of the number of retries.
• exponentialFactor=2 and delay=5 exponentially increases the interval between retries, based on the preceding interval:
  • The first retry has a delay of 5^2 = 25 seconds
  • The second retry has a delay of 25^2 = 625 seconds
  • The same pattern continues for subsequent retries

Here is a complete example:

codefresh.yml

version: '1.0'
steps:
  MyAppDockerImage:
    title: Building Docker Image
    type: build
    image_name: my-own-app
    retry:
      maxAttempts: 2
  MyUnitTests:
    title: Running Unit tests
    image: ${{MyAppDockerImage}}
    commands:
    - ./my_unit_tests.sh
    retry:
      maxAttempts: 3
      delay: 5
  PushingToRegistry:
    type: push
    title: Pushing To Registry
    candidate: ${{MyAppDockerImage}}
    tag: '${{CF_BRANCH}}'
    retry:
      maxAttempts: 3
      delay: 3
      exponentialFactor: 2
      

Notice that Codefresh also provides the following variables that allow you change your script/applications according to the retry attempts:

• CF_CURRENT_ATTEMPT contains the number of current retry attempt.
• CF_MAX_ATTEMPTS contains all the number of total attempts defined.

The retry mechanism is available for all kinds of steps.

Escaping strings

If you want to use strings inside your pipeline that create conflicts with the Codefresh syntax parser (for example they are YAML themselves) you need to escape them using multi-line strings with the >- and |- characters.

The following pipeline is not parsed correctly because the echo command is using the yaml : character

version: "1.0"
steps:
  test:
    title: "Running test"
    type: "freestyle" 
    image: "alpine:3.9" 
    commands:
      - echo hello: world
      

You can fix this issue by using a multi-line YAML string:

version: "1.0"
steps:
  test:
    title: "Running test"
    type: "freestyle" 
    image: "alpine:3.9" 
    commands:
      - |-
        echo hello: world
      

The |- character keeps the line breaks of the text (but removes the last one). Use the >- character if you want to convert line breaks to spaces. For more information see the YAML specification.

Using YAML anchors to avoid repetition

Codefresh also supports yaml anchors, references and extends. These allow you to keep your pipeline DRY.

For example, let’s say that you have two freestyle steps:

1. The first one fills a MySQL server with data.
2. The second one runs integration tests that use the MySQL server.

Here is the respective pipeline:

codefresh.yml

version: '1.0'
steps:
  preLoadDatabase:
    title: Loading Data
    image: alpine
    commands:
      - printenv
      - echo "Loading DB"
    environment: &my_common_envs
      - MYSQL_HOST=mysql
      - MYSQL_USER=user
      - MYSQL_PASS=password
      - MYSQL_PORT=3351  
  runTests:
    title: Integration tests
    image: alpine
    commands:
      - printenv
      - echo "Running tests"
    environment: *my_common_envs  # Same MYSQL_HOST, MYSQL_USER etc.

Instead of repeating the same environment variables in both steps, we can create them once and then just reference them in the second step with the * character.

You also define anchors at the top of the pipeline in the special indicators block:

codefresh.yml

version: '1.0'

indicators:
 - environment: &my_common_envs
      - MYSQL_HOST=mysql
      - MYSQL_USER=user
      - MYSQL_PASS=password
      - MYSQL_PORT=3351
 
steps:
  preLoadDatabase:
    title: Loading Data
    image: alpine
    commands:
      - printenv
      - echo "Loading DB"
    environment: *my_common_envs # Same MYSQL_HOST, MYSQL_USER etc.
  runTests:
    title: Integration tests
    image: alpine
    commands:
      - printenv
      - echo "Running tests"
    environment: *my_common_envs  # Same MYSQL_HOST, MYSQL_USER etc.

Finally. you also extend steps like below:

codefresh.yml

version: '1.0'
steps:
  deploy_to_k8_staging: &my_basic_deployment
    title: deploying to cluster
    type: deploy
    kind: kubernetes 
    cluster:  myStagingCluster
    namespace: sales
    service: my-python-app
  deploy_to_k8_prod: 
    <<: *my_basic_deployment
    cluster:  myProdCluster # only cluster differs, everything else is the same

Here we deploy to two kubernetes clusters. The first step defines the staging deployment. For the second step, we extend the first one and only change the name of the cluster to point to production. Everything else (i.e. namespace and service) are exactly the same.

Related articles

Variables in pipelines
Advanced workflows
Creating pipelines
CI/CD YAML examples