Monitoring & managing GitOps Runtimes

Optimize GitOps Runtimes

The Runtimes page displays the provisioned GitOps Runtimes in your account, both Hybrid, and the Hosted Runtime if you have one.

View Runtime components and information in List or Topology view formats, to manage and monitor them.

GitOps Runtime views

View provisioned GitOps Runtimes in List or Topology view formats.

• List view: The default view, displays the list of provisioned Runtimes, the clusters managed by them, and Git Sources associated with them.
• Topology view: Displays a hierarchical view of Runtimes and the clusters managed by them, with health and sync status of each cluster.

List view

The List view is a grid-view of the provisioned Runtimes.

Here is an example of the List view for runtimes.

Here is a description of the information in the List View.

Topology view

A hierachical visualization of the provisioned GitOps Runtimes. The Topology view makes it easy to identify key information such as versions, health and sync status, for both the provisioned Runtime and the clusters managed by it. Here is an example of the Topology view for GitOps Runtimes.

Here is a description of the information in the Topology view.

Topology View Item	Description
Runtime	the provisioned Runtime. Hybrid Runtimes display the name of the K8s API server endpoint with the cluster. Hosted Runtimes display ‘Codefresh hosted’.
Cluster	The local, and managed clusters if any, for the Runtime. indicates the local cluster, always displayed as `in-cluster`. The in-cluster server URL is always set to `https://kubernetes.default.svc/`. indicates a managed cluster. select to add a new managed cluster. To view cluster components, select the cluster. To add and work with managed clusters, see Managing external clusters in GitOps Runtimes.
Health/Sync status	The health and sync status of the Runtime or cluster. indicates health or sync errors in the Runtime, or a managed cluster if one was added to the runtime. The runtime or cluster node is bordered in red and the name is colored red. indicates that the Runtime is being synced to the cluster on which it is provisioned.
Search and View options	Find a Runtime or its clusters by typing part of the Runtime/cluster name, and then navigate to the entries found. Topology view options: Resize to window, zoom in, zoom out, full screen view.

(Helm Hybrid GitOps) Upgrade GitOps Runtimes

Upgrade provisioned Hybrid GitOps Runtimes to install critical security updates, get new functionality, and the latest versions of all components. The upgrade procedure differs depending on whether the GitOps Runtime has been configured as an Argo CD application or not:

• Argo CD GitOps Runtimes: For Runtimes configured as Argo CD applications, you need to manually update the version in the Helm chart located in the Shared Configuration Repository.
• Non-Argo CD GitOps Runtimes: Run the upgrade command.

If you have managed clusters for Hybrid GitOps Runtimes, upgrading the Runtime automatically updates runtime components within the managed cluster as well.

1. In the Codefresh UI, on the toolbar, click the Settings icon.
2. From Runtimes in the sidebar, select GitOps Runtimes.
3. Switch to either the List View or to the Topology View.
4. List view:
   1. To see all the commits to the Runtime, in the Version column, mouse over Update Available!, and select View Complete Change Log.
   2. Do one of the following:
      • To the right of the row with the Runtime to upgrade, click the context menu and select Upgrade.
      • Click the Runtime name, and then click Upgrade on the top-right.

Topology view: Click the Runtime cluster, and from the panel, click the context menu, and then select Upgrade Runtime.

5. Do one of the following depending on whether you have configured the Runtime as an Argo CD Application or not:
   • Argo CDGitOps Runtimes:Continue from step 6.
   • Non-Argo CD GitOps Runtimes: Continue from step 7.
6. For GitOps Runtimes, do the following:
   1. In your Shared Configuration Repository, go to resources/<runtime_name>/chart where, <runtime_name> is the name of the Hybrid GitOps Runtime to upgrade.
   2. In the chart.yaml, change the version number in both .version and .dependencies.version.
   3. Commit the change, and push to your Git server.

      apiVersion: v2
      appVersion: 1.0.0
      description: Codefresh gitops runtime umbrella chart
      name: codefresh-gitops-runtime
      version: <version>
      dependencies:
           - name: gitops-runtime
       repository: oci://quay.io/codefresh
       version: <version>
      

7. For non-GitOps Runtimes, do the following:
   1. Copy and run the upgrade command: RELEASE_NAME=$(helm ls -n codefresh-gitops-runtime -q) && helm upgrade ${RELEASE_NAME} -n codefresh-gitops-runtime
   2. To exit the upgrade panel, click Close.

Update Git credentials for GitOps Runtimes

Provisioned GitOps Runtimes require valid Git Runtime tokens at all times to authenticate and authorize the Runtime. The Git Runtime token differs from the Git user token which is a personal access token unique to each user. As such, you can use the same Git Runtime token for multiple Runtimes.

When to update
If your Git Runtime token is valid, you can update it at any time by deleting the existing token and replacing it with a new token.

Otherwise, you have to update Git tokens in the following situations:

• Complete Installation status displayed in Sync column for Runtime
  You have installed the Runtime but need to update the Git credentials to complete the installation.
• Invalid, revoked, or expired tokens
  Codefresh automatically flags Runtimes with such tokens. It is mandatory to update the Git tokens to continue working with the platform.

Update methods
The methods for updating any Git token are the same regardless of the reason for the update:

• OAuth2 authorization, if your admin has registered an OAuth Application for Codefresh.
• Git access token authentication, by generating a Git Runtime token in your Git provider account with the correct scopes. You can update your Git Runtime token in the UI or through the CLI.

Update Git Runtime credentials in Codefresh UI

Before you begin

• To authenticate through a Git Runtime token, make sure your token is valid and has the required scopes

How to

1. In the Codefresh UI, on the toolbar, click the Settings icon.
2. From Runtimes in the sidebar, select GitOps Runtimes.
3. Switch to either the List View.
4. Do one of the following:
   • To the right of the row with the Runtime to update, click the context menu and select Update Git Runtime Credentials.
   • Click the Runtime name, click the context-menu on the top-right, and then select Update Git Runtime Credentials.

5. Do one of the following:
   • If your admin has set up OAuth access, click Authorize Access to Git Provider. Go to step 6.
   • Alternatively, authenticate with an access token from your Git provider. Go to step 7.

6. For OAuth2 authorization:

   NOTE
   If the application is not registered, you get an error. Contact your admin for help.

   • Enter your credentials, and select Sign In.
   • If required, as for example if two-factor authentication is configured, complete the verification.

7. For Git token authentication, paste the generated token in the Git runtime token field.
8. Click Update Credentials.

Update Git Runtime token through CLI

If you are using Git Runtime tokens for authentication, you can also update them through the Codefresh CLI.

• Run: cf config update-gitops-settings --shared-config-repo <repo_url> [--git-provider <git-provider>] [--git-api-url <git-api-url>]

  where:

  • <repo_url> is required, and is the URL of the Shared Configuration Repository, and must match the Git provider.
  • --git-provider is optional for cloud providers as Codefresh identifies the provider through the Shared repo URL. Required for on-premises providers, and can be one of the following:
    • GitHub and GitHub Enterprise: github
    • GitLab Cloud and GitLab Server: gitlab
    • Bitbucket Cloud: bitbucket
    • Bitbucket Server: bitbucket-server
  • --git-api-url is optional for all Git providers, including on-premises repo clones.

Configure SSH for GitOps Runtimes

By default, Git repositories use the HTTPS protocol. You can also use SSH to connect Git repositories by entering the SSH private key.

NOTE
When SSH is configured for a GitOps Runtime, when creating/editing Git-Source applications, you can select HTTPS OR SSH as the protocol to connect to the Git repository. See Repository URL in Application Source definitions.

SSH keys For more information on generating SSH private keys, see the official documentation:

• GitHub
• GitLab
• Bitbucket
• Azure
• Gerrit

Before you begin Copy the SSH private key for your Git provider

How to

1. In the Codefresh UI, make sure you are in GitOps Runtimes.
2. From the List View, select the runtime for which to configure SSH.
3. From the context menu with the additional actions on the top-right, select Update Git Runtime Credentials.

4. Expand Connect Repo using SSH, and then paste the raw SSH private key into the field.

5. Click Update Credentials.

View/download logs for GitOps Runtimes

Logs are available for installed GitOps Runtimes, both for the Runtime and for individual Runtime components. Download log files for offline viewing and analysis, or view online logs for a Runtime component and download if needed for offline analysis. Online logs support free-text search, search-result navigation, and line-wrap for enhanced readability.

Log files include events from the date of the application launch, with the newest events listed first.

Download logs for GitOps Runtimes

Download the log file for a GitOps Runtime. The Runtime log is downloaded as a .tar.gz file, which contains the individual log files for each runtime component.

1. In the Codefresh UI, on the toolbar, click the Settings icon.
2. From Runtimes in the sidebar, select GitOps Runtimes.
3. If needed, switch to List View, and then select the runtime for which to download logs.
4. From the context menu, select Download All Logs. The log file is downloaded to the Downloads folder or the folder designated for downloads, with the filename, <runtime-name>.tar.gz. For example, codefreshv2-production2.tar.gz.

View/download logs for Runtime components

View online logs for any Runtime component, and if needed, download the log file for offline viewing and analysis.

NOTE
Downloading logs for Runtime components is supported from Runtime v0.1.39 and higher.

Online logs show up to 1000 of the most recent events (lines), updated in real time. Downloaded logs include all the events, from the date and time of application launch to the date and time of download.

1. In the Codefresh UI, on the toolbar, click the Settings icon.
2. From Runtimes in the sidebar, select GitOps Runtimes.
3. If needed, switch to List View, and then select the Runtime.
4. In the Runtime Components tab, from the context menu of the required component, select View Logs.

4. Do the following:
   • To search for any string, type the free-text to search for, and click the next and previous buttons to navigate between the search results.
   • To switch on line-wrap for readability, click Wrap.

5. To download the log, click Download. The file is downloaded as <component-name>.log.

Reset Shared Configuration Repository for GitOps Runtimes

Codefresh creates and validates the Shared Configuration Repository when you install the first Hybrid or Hosted GitOps Runtime for your account, and uses it for all GitOps Runtimes you add to the same account.

The Shared Configuration Repo is created in your Git provider account. You can reset the repo defined for your account under the following conditions:

• Incorrect/missing URL Mandatory when Codefresh notifies you through the UI that the Shared Configuration Repo URL is either incorrect or missing.

  • Incorrect URL The Shared Config Repo details provided during installation in Account Setup are incorrect. Codefresh could not connect to the Shared Repo with the details provided.
  • Undefined URL
    You installed the GitOps Runtime through a script or an automated mechanism without providing the URL to the Shared Configuration Repository.

• No active Runtimes If Codefresh has already validated the existing Shared Configuration Repository, meaning that at least one GitOps Runtime successfully connected to it, you cannot change the Shared Configuration Repo URL. To do so, you must contact Codefresh Support.

  Otherwise, you can reset the Shared Config Repo URL only after uninstalling all the GitOps Runtimes in your account. This option is useful when moving from a temporary account, for example, a POV account, to your organization’s official account to reset the URL.

Before you begin

• Make sure you have no active GitOps Runtimes in your account

How to

1. Run cf config update-gitops-settings --shared-config-repo <shared_repo_url> where: <shared_repo_url> is the new URL for the Shared Configuration Repository.
2. When prompted, select the Git provider.
3. Confirm to create the Shared Configuration Repo.

(Hybrid GitOps) Configure Deep Links to applications & resources

Deep Links is an Argo CD feature that redirects users to third-party applications/platforms by surfacing links to the same in Argo CD projects, applications, and resources. Read all about it in Argo CD Deep Links.

In Codefresh, you can configure deep links to third-party applications/platforms in the argocd-cm ConfigMap, located in the repo where you installed the Hybrid GitOps Runtime. When configured, deep links are displayed in the application’s Current State tab in Tree view. See Working with resources in Tree View.

NOTE
Deep link configuration in Codefresh requires Runtime v0.1.27 or higher.

1. Go to the <hybrid-gitops-installation-repo>/bootstrap/argo-cd/kustomization.yaml.
2. Configure deep links as in the example below.

...
configMapGenerator:
- name: argocd-cm
  behavior: merge
  literals:
  - |
    resource.links:=- url: https://<mycompany>.splunk.com
        title: Splunk
        description: jf
        icon.class: "fa-book"
  - |
    application.links=- url: https://<mycompany>.splunk.com
        title: Splunk
        description: jf
        icon.class: "fa-book"

where:

• <location>:=- url: defines where the link is displayed and the target URL to link to:
  • location can be application.links (Application) or resource.links (Resource). Codefresh does not show Argo CD projects.
  • url is the target URL in the format https://<url>.com, for example, https://codefresh.io.splunk.com.
• titleis the display name for the link, as will appear in the UI. For example, Splunk.
• descriptionis optional, and presents additional info on the link.
• icon-class is optional, and is the font-awesome icon class displayed to the left of the title.

Argo CD also supports if conditional statements to control when the deep links are displayed. When omitted, configured deep links are always displayed.
For more details, read Configuring Deep Links in Argo CD.

(Helm Hybrid GitOps) Remove GitOps Runtimes

Remove Helm GitOps Runtimes that are offline from the Codefresh UI. The Runtime is not removed from the cluster.

1. In the Codefresh UI, on the toolbar, click the Settings icon.
2. From Runtimes in the sidebar, select GitOps Runtimes.
3. Switch to either the List View or to the Topology View.
4. Do one of the following:
   • To the right of the row with the Runtime to remove, click the context menu and select Remove.
   • Click the Runtime name, click the context-menu on the top-right, and then select Remove.

5. Click Remove to confirm.

Uninstall GitOps Runtimes

Uninstall provisioned GitOps Runtimes that are not in use.

Uninstalling a GitOps Runtime permanently removes:

• The Runtime from the cluster it is provisioned on
• The Git Sources and managed clusters associated with it

How to

1. In the Codefresh UI, on the toolbar, click the Settings icon.
2. From Runtimes in the sidebar, select GitOps Runtimes.
3. Switch to either the List View or to the Topology View.
4. List view:
   • Do one of the following:
     • To the right of the row with the Runtime to upgrade, click the context menu and select Uninstall.
     • Click the Runtime name, click the context-menu on the top-right, and then select Uninstall.

Topology view:

• Click the Runtime cluster, and from the panel, click the context menu, and then select Uninstall.

5. Copy and run the uninstall command:

RELEASE_NAME=$(helm ls -n codefresh-gitops-runtime -q) && helm uninstall ${RELEASE_NAME} -n codefresh-gitops-runtime

6. Click Close to exit the upgrade panel.

(Legacy CLI Hybrid GitOps) Restoring provisioned GitOps Runtimes

In case of cluster failure, restore the provisioned Hybrid GitOps Runtime from the existing runtime installation repository. For partial or complete cluster failures, you can restore the Runtime to either the failed cluster or to a different cluster. Restoring the provisioned Runtime reinstalls it, leveraging the resources in the existing Runtime repo.

Restoring the runtime:

• Applies argo-cd from the installation manifests in your repo to your cluster
• Associates argo-cd with the existing installation repo
• Applies the Runtime and argo-cd secrets to the cluster
• Updates the Runtime config map (<runtime-name>.yaml in the bootstrap directory) with the new cluster configuration for these fields: cluster ingressClassName ingressController ingressHost

Restore a Hybrid GitOps Runtime

Reinstall the Hybrid Runtime from the existing installation repository to restore it to the same or a different cluster.

Before you begin

• Have the following information handy:

  All values must be the identical to the Runtime to be restored.

  • Runtime name
  • Repository URL
  • Codefresh context
  • Kube context: Required if you are restoring to the same cluster

How to

1. Run: cf runtime install --from-repo
2. Provide the relevant values when prompted.

3. If you are performing the runtime recovery in a different cluster, verify the ingress resource configuration for app-proxy, workflows, and default-git-source. If the health status remains as Progressing, do the following:

   • In the Runtime installation repo, check if the ingress.yaml files for the app-proxy and workflows are configured with the correct host and ingressClassName:

     apps/app-proxy/overlays/<runtime-name>/ingress.yaml apps/workflows/overlays/<runtime-name>/ingress.yaml

   • In the Git Source repository, check the host and ingressClassName in cdp-default-git-source.ingress.yaml:

     resources_<runtime-name>/cdp-default-git-source.ingress.yaml

   See the example below.

4. If you have managed clusters registered to the hybrid runtime you are restoring, reconnect them. Run the command and follow the instructions in the wizard: cf cluster add

5. Verify that you have a registered Git integration: cf integration git list --runtime <runtime-name>

6. If needed, create a new Git integration: cf integration git add default --runtime <runtime-name> --provider github --api-url https://api.github.com

Ingress example

This is an example of the ingress.yaml for workflows.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    ingress.kubernetes.io/protocol: https
    ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/backend-protocol: https
    nginx.ingress.kubernetes.io/rewrite-target: /$2
  creationTimestamp: null
  name: runtime-name-workflows-ingress
  namespace: runtime-name
spec:
  ingressClassName: nginx
  rules:
  - host: your-ingress-host.com
    http:
      paths:
      - backend:
          service:
            name: argo-server
            port:
              number: 2746
        path: /workflows(/|$)(.*)
        pathType: ImplementationSpecific
status:
  loadBalancer: {}

(Hybrid GitOps) Troubleshoot communication problems

You have a communication problem if you see either of these notifications: Unable to communicate with a runtime_name OR Unable to communicate with two or more runtimes

Refer to our troubleshooting section for a list of possible causes and the corresponding corrective actions.

(Hybrid GitOps) View notifications in Activity Log

The Activity Log is a quick way to monitor notifications for Runtime events such as upgrades. A pull-down panel in the Codefresh toolbar, the Activity Log shows ongoing, success, and error notifications, sorted by date, starting with today’s date.

1. In the Codefresh UI, on the top-right of the toolbar, select Activity Log.
2. To see notifications for provisioned Runtimes, filter by Runtime.

3. To see more information on an error, select the + sign.

Related articles

Managing Git Sources in GitOps Runtimes
Managing external clusters in GitOps Runtimes
Shared Configuration Repository