Part 3 of 6

40 min

Building Your First Pipeline

Create a complete CI/CD workflow that tests, builds, and deploys a Node.js application to staging and production

With the core concepts understood, it's time to build a real-world pipeline. In this guide, we'll create a complete CI/CD workflow that tests a Node.js application, builds a Docker image, and deploys it to your server.

Project Structure

We'll work with a typical Node.js web application. Here's the structure we'll build CI/CD for:

Project layout

my-node-app/
├── src/
│   └── index.js
├── tests/
│   └── app.test.js
├── Dockerfile
├── package.json
└── ci/
    ├── pipeline.yml
    ├── tasks/
    │   ├── test.yml
    │   ├── build-image.yml
    │   └── deploy.yml
    └── scripts/
        ├── run-tests.sh
        ├── build-docker.sh
        └── deploy.sh

The ci/ directory contains all Concourse-specific configurations, keeping your pipeline definitions version-controlled alongside your application code.

Creating Task Definitions

First, let's create the individual task definitions that our pipeline will use.

Test Task

ci/tasks/test.yml

platform: linux

image_resource:
  type: registry-image
  source:
    repository: node
    tag: "20-alpine"

inputs:
- name: source

caches:
- path: source/node_modules

run:
  path: sh
  args:
  - -exc
  - |
    cd source
    npm ci
    npm run lint
    npm test

Build Image Task

ci/tasks/build-image.yml

platform: linux

image_resource:
  type: registry-image
  source:
    repository: concourse/oci-build-task

inputs:
- name: source

outputs:
- name: image

params:
  CONTEXT: source
  DOCKERFILE: source/Dockerfile

run:
  path: build

Deploy Task

ci/tasks/deploy.yml

platform: linux

image_resource:
  type: registry-image
  source:
    repository: alpine
    tag: latest

inputs:
- name: source

params:
  DEPLOY_HOST: ""
  DEPLOY_USER: ""
  SSH_PRIVATE_KEY: ""
  IMAGE_TAG: ""

run:
  path: sh
  args:
  - -exc
  - |
    apk add --no-cache openssh-client
    
    # Setup SSH
    mkdir -p ~/.ssh
    echo "$SSH_PRIVATE_KEY" > ~/.ssh/id_rsa
    chmod 600 ~/.ssh/id_rsa
    ssh-keyscan -H "$DEPLOY_HOST" >> ~/.ssh/known_hosts
    
    # Deploy
    ssh "$DEPLOY_USER@$DEPLOY_HOST" << EOF
      docker pull myorg/my-node-app:${IMAGE_TAG}
      docker stop my-node-app || true
      docker rm my-node-app || true
      docker run -d --name my-node-app -p 3000:3000 myorg/my-node-app:${IMAGE_TAG}
    EOF

The Complete Pipeline

Now let's build the pipeline that orchestrates these tasks:

ci/pipeline.yml

resource_types:
- name: slack-notification
  type: registry-image
  source:
    repository: cfcommunity/slack-notification-resource
    tag: latest

resources:
# Source code repository
- name: source-code
  type: git
  icon: github
  source:
    uri: ((git-uri))
    branch: main
    private_key: ((git-private-key))

# Docker registry for built images
- name: app-image
  type: registry-image
  icon: docker
  source:
    repository: ((docker-repo))
    username: ((docker-username))
    password: ((docker-password))
    tag: latest

# Slack notifications
- name: slack
  type: slack-notification
  icon: slack
  source:
    url: ((slack-webhook-url))

# Semantic versioning
- name: version
  type: semver
  icon: tag
  source:
    driver: git
    uri: ((git-uri))
    branch: version
    file: version
    private_key: ((git-private-key))

jobs:
#
# TEST JOB
#
- name: test
  public: true
  plan:
  - get: source-code
    trigger: true
  
  - task: run-tests
    file: source-code/ci/tasks/test.yml
    input_mapping:
      source: source-code
  
  on_failure:
    put: slack
    params:
      text: |
        :x: *Tests failed* for `((git-uri))`
        Build: $ATC_EXTERNAL_URL/builds/$BUILD_ID

#
# BUILD JOB
#
- name: build
  public: true
  serial: true
  plan:
  - in_parallel:
    - get: source-code
      trigger: true
      passed: [test]
    - get: version
      params: { bump: patch }
  
  - task: build-docker-image
    privileged: true
    file: source-code/ci/tasks/build-image.yml
    input_mapping:
      source: source-code
  
  - put: app-image
    params:
      image: image/image.tar
      additional_tags: version/version
    get_params:
      skip_download: true
  
  - put: version
    params: { file: version/version }
  
  on_success:
    put: slack
    params:
      text: |
        :white_check_mark: *Build successful*
        Image: `((docker-repo)):$(cat version/version)`
  
  on_failure:
    put: slack
    params:
      text: |
        :x: *Build failed* for `((git-uri))`
        Build: $ATC_EXTERNAL_URL/builds/$BUILD_ID

#
# DEPLOY STAGING JOB
#
- name: deploy-staging
  public: false
  serial: true
  plan:
  - in_parallel:
    - get: source-code
      passed: [build]
    - get: app-image
      trigger: true
      passed: [build]
    - get: version
      passed: [build]
  
  - load_var: image-tag
    file: version/version
  
  - task: deploy-to-staging
    file: source-code/ci/tasks/deploy.yml
    input_mapping:
      source: source-code
    params:
      DEPLOY_HOST: ((staging-host))
      DEPLOY_USER: ((staging-user))
      SSH_PRIVATE_KEY: ((staging-ssh-key))
      IMAGE_TAG: ((.:image-tag))
  
  on_success:
    put: slack
    params:
      text: |
        :rocket: *Deployed to staging*
        Version: `((.:image-tag))`
        URL: https://staging.example.com

#
# DEPLOY PRODUCTION JOB
#
- name: deploy-production
  public: false
  serial: true
  plan:
  - in_parallel:
    - get: source-code
      passed: [deploy-staging]
    - get: app-image
      passed: [deploy-staging]
    - get: version
      passed: [deploy-staging]
  
  - load_var: image-tag
    file: version/version
  
  - task: deploy-to-production
    file: source-code/ci/tasks/deploy.yml
    input_mapping:
      source: source-code
    params:
      DEPLOY_HOST: ((production-host))
      DEPLOY_USER: ((production-user))
      SSH_PRIVATE_KEY: ((production-ssh-key))
      IMAGE_TAG: ((.:image-tag))
  
  on_success:
    put: slack
    params:
      text: |
        :tada: *Deployed to production*
        Version: `((.:image-tag))`
        URL: https://app.example.com
  
  on_failure:
    put: slack
    params:
      text: |
        :rotating_light: *PRODUCTION DEPLOY FAILED*
        Build: $ATC_EXTERNAL_URL/builds/$BUILD_ID

Setting Up Credentials

Before deploying the pipeline, configure your secrets. If you're not using a credential manager, create a variables file:

⚠️ Never commit credentials.yml to version control!

credentials.yml

git-uri: git@github.com:myorg/my-node-app.git
git-private-key: |
  -----BEGIN OPENSSH PRIVATE KEY-----
  ...your private key...
  -----END OPENSSH PRIVATE KEY-----

docker-repo: myorg/my-node-app
docker-username: myuser
docker-password: mypassword

slack-webhook-url: https://hooks.slack.com/services/XXX/YYY/ZZZ

staging-host: staging.example.com
staging-user: deploy
staging-ssh-key: |
  -----BEGIN OPENSSH PRIVATE KEY-----
  ...staging key...
  -----END OPENSSH PRIVATE KEY-----

production-host: app.example.com
production-user: deploy
production-ssh-key: |
  -----BEGIN OPENSSH PRIVATE KEY-----
  ...production key...
  -----END OPENSSH PRIVATE KEY-----

Deploying the Pipeline

Now deploy your pipeline to Concourse:

# Login to Concourse
fly -t main login -c http://your-concourse:8080

# Set the pipeline with credentials
fly -t main set-pipeline \
  -p my-node-app \
  -c ci/pipeline.yml \
  -l credentials.yml

# Review the diff and confirm (y)

# Unpause the pipeline
fly -t main unpause-pipeline -p my-node-app

Once deployed, visit your Concourse web UI to see the pipeline visualization. Each job will be shown as a box, with lines connecting them to show dependencies.

Understanding the Pipeline Flow

Here's how the pipeline executes:

                    ┌─────────────────────────────────────────────────────────────┐
                    │                        Pipeline Flow                         │
                    └─────────────────────────────────────────────────────────────┘

[source-code] ───trigger───> [test] ───passed───> [build] ───passed───> [deploy-staging]
                                │                     │                       │
                                │                [app-image]                  │
                                │                 [version]                   │
                                │                     │                       │
                                └─────────────────────┴───────passed───> [deploy-production]
                                                                          (manual trigger)

Job Dependencies

1. test

Triggers automatically on new commits. Runs linting and unit tests.

2. build

Only runs after test passes. Builds Docker image with semantic version tag.

3. deploy-staging

Automatically triggers when build produces a new image. Deploys to staging server.

4. deploy-production

Requires manual trigger. Only accepts versions that passed staging.

Pipeline Operations

Triggering Builds

# Manually trigger a job
fly -t main trigger-job -j my-node-app/test

# Trigger and watch output
fly -t main trigger-job -j my-node-app/test --watch

# Trigger production deployment (after staging verification)
fly -t main trigger-job -j my-node-app/deploy-production

Viewing Build Output

# Watch a running build
fly -t main watch -j my-node-app/build

# View specific build
fly -t main watch -j my-node-app/build -b 42

# Get build logs
fly -t main builds -j my-node-app/test

Pausing and Unpausing

# Pause a job (prevents triggering)
fly -t main pause-job -j my-node-app/deploy-production

# Unpause
fly -t main unpause-job -j my-node-app/deploy-production

# Pause entire pipeline
fly -t main pause-pipeline -p my-node-app

Intercepting Containers

Debug failed builds by accessing the container:

# List containers from recent builds
fly -t main containers

# Intercept a specific build's container
fly -t main intercept -j my-node-app/test -s run-tests

# You're now in the container
ls -la source/
cat source/package.json
npm test  # Re-run tests interactively

Adding Pipeline Groups

For complex pipelines, organize jobs into visual groups:

Add at the top of pipeline.yml

groups:
- name: development
  jobs:
  - test
  - build

- name: deployment
  jobs:
  - deploy-staging
  - deploy-production

- name: all
  jobs:
  - test
  - build
  - deploy-staging
  - deploy-production

# ... rest of pipeline

Groups create tabs in the web UI, making large pipelines navigable.

Common Patterns

Running Jobs in Parallel

Speed up pipelines by parallelizing independent steps

- name: test
  plan:
  - get: source-code
    trigger: true
  
  - in_parallel:
    - task: lint
      file: source-code/ci/tasks/lint.yml
    - task: unit-tests
      file: source-code/ci/tasks/unit-test.yml
    - task: integration-tests
      file: source-code/ci/tasks/integration-test.yml

Conditional Steps with try

Allow non-critical steps to fail

- name: build
  plan:
  - get: source-code
  - task: build
    file: source-code/ci/tasks/build.yml
  - try:
      task: upload-coverage
      file: source-code/ci/tasks/coverage.yml

Aggregating Multiple Resources

Build matrix-style pipelines

resources:
- name: node-18
  type: registry-image
  source:
    repository: node
    tag: "18"

- name: node-20
  type: registry-image
  source:
    repository: node
    tag: "20"

jobs:
- name: test-matrix
  plan:
  - get: source-code
    trigger: true
  - in_parallel:
    - task: test-node-18
      image: node-18
      file: source-code/ci/tasks/test.yml
    - task: test-node-20
      image: node-20
      file: source-code/ci/tasks/test.yml

Time-Based Triggers

Run jobs on a schedule

resources:
- name: nightly
  type: time
  source:
    start: 2:00 AM
    stop: 3:00 AM
    location: America/New_York

jobs:
- name: nightly-build
  plan:
  - get: nightly
    trigger: true
  - get: source-code
  - task: full-test-suite
    file: source-code/ci/tasks/full-tests.yml

Debugging Tips

Check Resource Versions

# See what versions Concourse knows about
fly -t main resource-versions -r my-node-app/source-code

# Check why a resource isn't triggering
fly -t main check-resource -r my-node-app/source-code

Validate Pipeline Syntax

# Validate before setting
fly -t main validate-pipeline -c ci/pipeline.yml

# Format pipeline YAML
fly -t main format-pipeline -c ci/pipeline.yml

View Job Configuration

# See effective job config
fly -t main get-pipeline -p my-node-app

Next Steps

You now have a fully functional CI/CD pipeline! In Part 4, we'll explore advanced patterns including fan-in/fan-out workflows, dynamic pipelines, and multi-environment deployments.

Part 2: Core Concepts Part 4: Advanced Patterns