Skip to main content

Clone multiple code repos in one pipeline

In addition to the pipeline's default codebase, you can use Git Clone step to clone additional code repos into the pipeline's workspace.

For example, you can use this to:

  • Build multiple artifacts in the same pipeline. Suppose you use Packer and Ansible to build artifacts automatically, and you have separate repositories for Packer, Ansible, and code. You can clone all three repositories into the pipeline's workspace.
  • Pull code from separate code and build repositories. For instance, if your code files are in a repository managed by the Engineering team and your Dockerfiles are in a different repository managed by the Security team, you can clone both repositories into the pipeline's workspace.
  • Clone codebases without using the built-in clone codebase function.

Add a Git Clone step​

Assume the default codebase is a repo that contains app code files, and the Dockerfile necessary to build the app image is in a different repo. You can use Git Clone step to clone the second repo into the workspace, or fetch specific files/directories. Then, you can use a Build and Push step to build and push an image using files from both repos.

Add a Git Clone step to clone a second repo into the pipeline's workspace.

    - step:
        type: GitClone
        name: clone second repo
        identifier: clone_second_repo
        spec:
          connectorRef: account.git
          repoName: myOrg/mySecondRepo
          build:
            type: branch
            spec:
              branch: main

For details, visit the git clone step page.

Add a Run step​

You can use git commands in Run steps to clone multiple repos into a stage. You can also provide arguments to clone subdirectories, clone recursively, and so on.

For example, this step clones a GitHub repository.

              - step:
                  type: Run
                  identifier: clone
                  name: clone
                  spec:
                    shell: Sh
                    command: |-
                      git clone https://GH_PERSONAL_ACCESS_TOKEN@github.com/ACCOUNT_NAME/REPO_NAME.git

To use this command, you would replace:

  • ACCOUNT_NAME with your GitHub account name.
  • REPO_NAME with the name of the GitHub repo to clone.
  • PERSONAL_ACCESS_TOKEN with a GitHub personal access token that has pull permissions to the target repository. Additional permissions may be necessary depending on the Action's purpose. Store the token as a Harness secret and use a variable expression, such as <+secrets.getValue("YOUR_TOKEN_SECRET")>, to call it.

For details about the Run step settings, check out the Run scripts tab under the Run step settings page.

tip

When cloning additional codebases that use the same credentials as your default codebase, you can use your default codebase connector's credentials in your Run step.

Build an artifact from both code repos​

Now that the files from both repos will be cloned into a common workspace, you can add a step to build an image using code from both repos, such as a Build and Push to Docker step.

Pay attention to settings like the Dockerfile setting that assume files are located at the codebase's root directory if not otherwise specified. This is because the pipeline's default codebase files are cloned in the root folder (/harness), while other codebase files are cloned into subfolders.

Depending on the default codebase, you might need to specify a non-root path for build files. You can also use commands, such as cp, in Run steps to move cloned files around the workspace before building the image.

YAML examples​

The following YAML example describes a pipeline that clones two code repos, one as the default codebase (cloneCodebase: true) and the second in the GitClone step.

pipeline:
  name: Clone two repos
  identifier: Clone_two_repos
  projectIdentifier: my_project
  orgIdentifier: default
  tags: {}
  properties:
    ci:
      codebase:
        connectorRef: account.git1
        repoName: my-code-repo
        build: <+input>
  stages:
    - stage:
        name: build from two repos
        identifier: build_from_two_repos
        description: ""
        type: CI
        spec:
          cloneCodebase: true
          platform:
            os: Linux
            arch: Amd64
          runtime:
            type: Cloud
            spec: {}
          execution:
            steps:
              - step:
                  type: GitClone
                  name: clone second repo
                  identifier: clone_second_repo
                  spec:
                    connectorRef: account.git2
                    build:
                      type: branch
                      spec:
                        branch: main
              - step:
                  type: BuildAndPushDockerRegistry
                  name: BuildAndPushDockerRegistry_1
                  identifier: BuildAndPushDockerRegistry_1
                  spec:
                    connectorRef: mydockerhubconnector
                    repo: username/imagerepo
                    tags:
                      - <+pipeline.executionId>
                    dockerfile: /path/to/dockerfile

Use a Plugin step​

As an alternative to the Git Clone and Run steps, you can clone a codebase by running the Git Drone plugin in a Plugin step.