A Step-by-Step Guide to Create a CI/CD Pipeline with Google Cloud Services

In today‘s fast-paced software development world, the ability to rapidly and reliably deliver new features and fixes is critical. Continuous Integration and Continuous Delivery (CI/CD) has emerged as a key practice to streamline the build, test and release process. Google Cloud provides a powerful set of tools to implement CI/CD pipelines for your applications.

In this guide, we‘ll walk through all the steps needed to create a complete CI/CD pipeline using various Google Cloud services including Cloud Source Repositories for source control, Container Registry for Docker image storage, Cloud Build for building and testing, and Google Kubernetes Engine for deploying the application. By the end, you‘ll have a fully automated pipeline that builds, tests and deploys your app every time you push new code. Let‘s get started!

Prerequisites

Before diving in, make sure you have the following:

• A Google Cloud account (you can sign up for $300 in free credits if you‘re new)
• Basic knowledge of Google Cloud concepts and navigation
• A code editor and terminal
• Git installed, and familiarity with basic Git commands
• Docker installed (recommended but not required)

We‘ll be using a sample web application written in Python and Flask for this tutorial. You can find the code in this GitHub repo or use your own application.

Activating Google Cloud APIs

First, we need to enable some Google Cloud APIs for the services we‘ll be using. From the Google Cloud console, go to the API Library and search for and enable the following APIs:

• Cloud Build API
• Kubernetes Engine API
• Container Registry API
• Cloud Source Repositories API

Setting up a Source Repository

Google Cloud Source Repositories provides Git version control to store and manage your application code. You can use it to host private Git repos right on Google Cloud.

To create a repository:

1. Go to the Source Repositories page and click "Add repository"
2. Select "Create a new repository" and give it a name like "hello-cloudbuild"
3. Choose your Google Cloud project and click "Create"

Now clone the empty repo to your local machine:

gcloud source repos clone hello-cloudbuild
cd hello-cloudbuild

Copy your application files into this repo directory. The sample Python app directory structure looks like this:

hello-cloudbuild
│   Dockerfile  
│   app.py
│   requirements.txt
│
└───kubernetes
    │   deployments.yaml
    │   services.yaml

The app.py and requirements.txt contain the Python application code and dependencies. We‘ll look at the Dockerfile, deployments.yaml and services.yaml in a bit.

Now commit and push your code to the Source Repository:

git add .
git commit -m "Initial commit"
git push origin main

Containerizing the Application

To deploy and run the application on Kubernetes, we first need to package it as a Docker container image. A Dockerfile specifies all the commands needed to assemble the image.

Here‘s the Dockerfile for the sample Python application:

# Use the official lightweight Python image.
FROM python:3.9-slim

# Allow statements and log messages to immediately appear in the Knative logs
ENV PYTHONUNBUFFERED True

# Copy application dependency manifests to the container image.
COPY requirements.txt ./

# Install production dependencies.
RUN pip install -r requirements.txt

# Copy local code to the container image.
COPY . ./

# Run the web service on container startup. 
# Use gunicorn webserver with one worker process and 8 threads.
# For environments with multiple CPU cores, increase the number of workers
# to be equal to the cores available.
CMD exec gunicorn --bind :$PORT --workers 1 --threads 8 --timeout 0 main:app

This Dockerfile uses the official Python base image, copies the application files, installs dependencies, and specifies the command to run the application.

To build the Docker image, you can run:

docker build -t gcr.io/PROJECT-ID/hello-cloudbuild:v1 .

Replace PROJECT-ID with your Google Cloud project ID. This tags the image and prepares it for pushing to the Google Container Registry, which we‘ll do from the CI/CD pipeline.

Defining the Kubernetes Manifests

Google Kubernetes Engine allows you to deploy and manage your containerized applications on a cluster of Compute Engine nodes. To deploy the app to GKE, we need to define Kubernetes manifest files.

The deployments.yaml file describes the desired state of the application deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-cloudbuild
spec:
  replicas: 1
  selector:
    matchLabels:
      app: hello-cloudbuild
  template:
    metadata:
      labels:
        app: hello-cloudbuild
    spec:
      containers:
      - name: hello-cloudbuild
        image: gcr.io/PROJECT-ID/hello-cloudbuild:v1

It specifies the container image to deploy and the number of replicas.

The services.yaml manifest defines a load balancer service to expose the application externally:

apiVersion: v1
kind: Service
metadata:
  name: hello-cloudbuild
spec:
  type: LoadBalancer
  selector:
    app: hello-cloudbuild
  ports:
    - port: 80
      targetPort: 8080

We‘ll deploy these manifests to GKE from the CI/CD pipeline.

Creating a GKE Cluster

Next, let‘s create the GKE cluster where the application will run. From the Cloud Console, navigate to the Kubernetes Engine page and click "Create cluster".

Select the Standard cluster template, give the cluster a name, select a region/zone, and customize the number of nodes if desired. Then click "Create" to provision the cluster.

Once created, you can connect to the cluster from your local terminal:

gcloud container clusters get-credentials CLUSTER-NAME --zone COMPUTE-ZONE --project PROJECT-ID  

Configuring the CI/CD Pipeline

With the source code, Docker image, and Kubernetes manifests ready, it‘s time to configure Cloud Build to define the pipeline.

Cloud Build uses a cloudbuild.yaml configuration file to specify the build steps. Here‘s an example that builds the Docker image, pushes it to the registry, and deploys to GKE:

steps:
# Build the Docker image
- name: ‘gcr.io/cloud-builders/docker‘
  args: [‘build‘, ‘-t‘, ‘gcr.io/$PROJECT_ID/hello-cloudbuild:v1‘, ‘.‘]

# Push the image to Container Registry  
- name: ‘gcr.io/cloud-builders/docker‘
  args: [‘push‘, ‘gcr.io/$PROJECT_ID/hello-cloudbuild:v1‘]

# Deploy to GKE  
- name: ‘gcr.io/cloud-builders/kubectl‘
  args: [‘apply‘, ‘-f‘, ‘kubernetes/‘]
  env:
  - ‘CLOUDSDK_COMPUTE_ZONE=COMPUTE-ZONE‘
  - ‘CLOUDSDK_CONTAINER_CLUSTER=CLUSTER-NAME‘

Each step uses a specific builder (Docker, kubectl) to execute commands like building the image, pushing to the registry, and deploying to the cluster.

To create a build trigger that automatically runs this pipeline when new commits are pushed:

1. Go to the Cloud Build triggers page
2. Click "Create Trigger"
3. Give the trigger a name and description
4. Choose your source repository and branch to watch (e.g. the hello-cloudbuild repo on the main branch)
5. Choose "Cloud Build configuration file" and specify the cloudbuild.yaml location
6. Click "Create" to finish

Now whenever you push a new commit to the watched branch, Cloud Build will automatically run the pipeline to build, push and deploy the app!

You can also manually run the pipeline by clicking "Run trigger" from the triggers list.

Testing the Application

After the Cloud Build pipeline finishes successfully, the sample Python app will be deployed and accessible through the Kubernetes load balancer service.

To get the external service endpoint:

kubectl get services

Look for the EXTERNAL-IP of the hello-cloudbuild service. Open that IP in a web browser and you should see the running application!

To test the CI/CD pipeline, try making a small change to the application code (e.g. change the homepage text). Commit and push the code change. This will trigger an automatic Cloud Build which you can monitor from the Cloud Console.

Once the pipeline completes, refresh the application URL to see the updated version deployed!

Cleaning Up

To avoid incurring ongoing costs for the resources used in this tutorial, be sure to clean up when you‘re done.

You can go to each Google Cloud service used and delete the associated resources:

• Cloud Build – delete the trigger and any history/artifacts
• Container Registry – delete the hello-cloudbuild repository and images
• Kubernetes Engine – delete the clusters
• Source Repositories – delete the hello-cloudbuild repository

Next Steps

Congratulations, you‘ve successfully set up a CI/CD pipeline on Google Cloud that automatically builds, tests and deploys your application! Some next steps to explore:

• Adding unit/integration tests to the pipeline
• Setting up PR/staging/release branch flows
• Configuring deployment strategies like blue/green or canary
• Monitoring pipeline metrics
• Implementing GitOps and config sync

I hope this guide gave you a solid foundation for building CI/CD pipelines on Google Cloud. You can adapt the concepts shown here for your own applications and extend them in many ways. For further reading, check out the Google Cloud CI/CD documentation. Thanks for following along!