Mastering Google Cloud Build Config Syntax

Say the magic words, and GCB opens up with hidden potential

UPDATE: There’s now a documentation page on cloud.google.com with a more thorough and otherwise better explanation of how to do bash scripting in Cloud Build. Read that, instead of this!

Google’s Cloud Build automation platform is modeled on a series of containerized “builder” steps, specified in a YAML config file. For each step, GCB spins up a Docker container, and runs a specific command within the context of that container, like gradle, or gcloud, or wget. You can pass one or more arguments to that command using the args field in your cloudbuild.yaml file.

This syntax makes it easy to get a basic build pipeline up and running. But as you add complexity to your pipeline, you may find that your YAML becomes hard to parse and maintain, or even that there are things you need to do that simply can’t be expressed. In this post, we’ll explore alternative ways to write your Cloud Build config. By applying these best practices, you’ll have more readable, maintainable config files. Even better, you can harness the hidden power of Cloud Build to create advanced CI/CD pipelines.

These config files all do the exact same thing. Which one is best? That depends on your use case.

As an example, we’ll make a build step that fetches text from a remote API, and prints it to the console. Let’s start with the simplest version of the config:

Simple Syntax

steps:
- name: 'gcr.io/cloud-builders/curl'
  args: ['https://pets.doingdevops.com/pet','-s','--max-time','10']

This is the most common syntax found in documentation, and it’s usually the right place to start.

Best Practice: Use this for simple jobs: it’s concise and easy to read.

Expanded syntax (collection style)

It also works if we add line breaks to the simple syntax:

steps:
- name: 'gcr.io/cloud-builders/curl'
  args:
    [
      'https://pets.doingdevops.com/pet',
      '-s',
      '--max-time', '10', # related args on one line
    ]

By adding line breaks, we make room for long arguments, and make it easier to see where one argument ends and the next one begins. This is especially helpful when using variables or data structures within arguments, which can easily become hard to read.

However, the one-argument-per-line rule isn’t actually enforced. So, if multiple arguments are related, they can be grouped one line. In this example, the arguments --max-timeand 10are on the same line, to make it evident that one refers to the other.

Best Practice: Use this syntax whenever basic syntax becomes cumbersome: many arguments, long arguments, complexity within arguments, etc. And don’t forget to add a dangling comma!

Expanded syntax (list style)

Alternatively, we can write the step using a YAML list, with one argument per line, prefixed with dashes. The arguments may be unquoted, but that’s a bad idea; it’s very hard to read an argument like — -s: is that dash-space-dash? or space-dash-dash? or…?

steps:
- name: 'gcr.io/cloud-builders/curl'
  args:
  - 'https://pets.doingdevops.com/pet'
  - '-s'
  - '--max-time'
  - '10'

While this is similar to “Expanded syntax (collection style),” it has a few disadvantages: 1) having a dash per argument reduces readability, especially since many arguments will have their own dashes; 2) you can’t group multiple arguments on a single line; 3) it’s harder to convert from “Simple syntax.”

Best Practice: Don’t use. (Prefer “collection style.”)

Breakout syntax

Here’s where things get interesting! Each builder has a default entrypoint, which typically correlates to that builder’s purpose. For example, the Gradle builder’s entrypoint is gradle. When a gradle build step is invoked, the gradle command is run in the container, with everything in the args array passed as arguments to Gradle.

But we’re not limited only to the default command: we can “break out,” and run any command available in the container, simply by specifying an alternate entrypoint. Most builders have bash installed, which gives us ultimate flexibility to do anything we wish within our step. Here’s the same operation, but invoking bash directly:

steps:
- name: 'gcr.io/cloud-builders/curl'
  entrypoint: 'bash'
  args:
  - '-c' # pass what follows as a command to bash
  - |
    curl -s 'https://pets.doingdevops.com/pet' --max-time 10

Here, a block YAML syntax is used (the argument is prefaced with “|”), so we can pass a multiline unquoted string as the command. And what exactly have we achieved? Nothing! This still does the same thing as the “simple” example. But now we have the full power to run arbitrary bash commands. So we can do more. Much more…

Let’s suppose that the remote API is flaky; sometimes it fails. In that case, we want to retry until it succeeds. Here’s an example that uses multiple commands and conditional logic to ensure that we get a valid response:

steps:
- name: 'gcr.io/cloud-builders/curl'
  entrypoint: 'bash'
  args:
  - '-c'
  - |
    PET="$$(curl -s https://pets.doingdevops.com/pet_flaky --max-time 10)"
    while [ "$$PET" == "ERROR" ] ; do
      echo "Error: API failed to respond with a pet! Try again..."
      PET="$$(curl -s https://pets.doingdevops.com/pet_flaky --max-time 10)"
  done
  echo "Success! $${PET}"

Note the use of double dollar signs ($$) as escape characters. This ensures that Cloud Build will pass them to the container, rather than interpreting them as substitutions.

This is more than just a one-off curl command! By overriding the entry point, and stringing together bash commands, we can do anything that’s possible within the confines of this builder container. And if you need to do something that’s not available in an existing builder, you can make your own custom builder image — be sure to include a shell (e.g. bash) if you plan to use “breakout” syntax.

What’s something interesting you’ve done with Cloud Build? Leave a note in the comments!

Example config files, and source for the remote API services, can be found at github.com/davidstanke/gcb-syntax-blog