Database migrations in Kubernetes

In-app migrations, initContainers, Kubernetes Job and Helm Hooks.

Introduction

In the era of microservices and Kubernetes, managing database migrations has become more complex than ever. Traditional methods of running migrations during application startup are no longer sufficient.

This article explores various approaches to handling database migrations in a Kubernetes environment, with a focus on Golang-based solutions.

The challenge of migrations in Kubernetes

Kubernetes introduces new challenges for database migrations:

• Multiple replicas starting simultaneously.

• Need for coordination to avoid concurrent migrations.

• Separation of concerns between application and migration logic.

Popular migration tools for Golang

As mentioned in another post, there are a different tools you can use to manage your migrations like:

golang-migrate

• Widely used and supports numerous databases.

• Simple CLI and API.

• Supports various migration sources (local files, S3, Google Storage).

goose

• Supports main SQL databases.

• Allows migrations written in Go for complex scenarios.

• Flexible versioning schemas.

atlas

• Powerful database schema management tool

• Supports declarative and versioned migrations.

• Offers integrity checks and migration linting.

• Provides GitHub Actions and Terraform provider.

Running migrations in Kubernetes

Inside your code

A naive implementation would be to run the code of the migration directly inside your main function before you start your server.

Example using golang-migrate:

package main

import (
    "database/sql"
    "fmt"
    "log"
    "net/http"

    "github.com/golang-migrate/migrate/v4"
    "github.com/golang-migrate/migrate/v4/database/postgres"
    _ "github.com/golang-migrate/migrate/v4/source/file"
    _ "github.com/lib/pq"
)

func main() {
    // Database connection parameters
    url := "postgres://user:pass@localhost:5432/dbname"

    // Connect to the database
    db, err := sql.Open("postgres", url)
    if err != nil {
        log.Fatalf("could not connect to database: %v", err)
    }
    defer db.Close()

    // Run migrations
    if err := runMigrations(db); err != nil {
        log.Fatalf("could not run migrations: %v", err)
    }

    // Run the application, for example start the server
    if err := http.ListenAndServe(":8080", nil); err != nil {
        log.Fatalf("server failed to start: %v", err)
    }
}

func runMigrations(db *sql.DB) error {
    driver, err := postgres.WithInstance(db, &postgres.Config{})
    if err != nil {
        return fmt.Errorf("could not create database driver: %w", err)
    }

    m, err := migrate.NewWithDatabaseInstance(
        "file://migrations", // Path to your migration files
        "postgres",          // Database type
        driver,
    )
    if err != nil {
        return fmt.Errorf("could not create migrate instance: %w", err)
    }

    if err := m.Up(); err != nil && err != migrate.ErrNoChange {
        return fmt.Errorf("could not run migrations: %w", err)
    }

    log.Println("migrations completed successfully")
    return nil
}

However, these could cause different issues like your migrations being slow and Kubernetes considering the pod didn’t start successfully and therefore killing it. You could run those migrations in a Go routine, but how do you handle failures then?

In case when multiple pods are created at the same time, you would have a potential concurrency problem.

It also means your migrations need to be inside your Docker image.

InitContainers

By using an initContainers in your Kubernetes Deployment, it will run the migration before the main application container starts. That’s a good first solution for when scaling is not a problem yet.

If the initContainer fails, the blue/green deployment from Kubernetes won’t go further and your previous pods stays where they are. It prevents having a newer version of the code without the planned migration.

Example:

initContainers:
  - name: migrations
    image: migrate/migrate:latest
    command: ['/migrate']
    args: ['-source', 'file:///migrations', '-database','postgres://user:pass@db:5432/dbname', 'up']

Separate job

You could create a Kubernetes Job that runs your migrations, and trigger that job during the deployment process before rolling out the application.

Example:

apiVersion: batch/v1
kind: Job
metadata:
  name: db-migrate
spec:
  template:
    spec:
      containers:
      - name: migrate
        image: your-migration-image:latest
        command: ['/app/migrate']

You can also combine it with initContainers making sure that the pod starts only when the job is successful.

initContainers:
  - name: migrations-wait
    image: ghcr.io/groundnuty/k8s-wait-for:v2.0
    args:
      - "job"
      - "my-migration-job"

Helm hook

If you use Helm, it has hooks that you could use for running migrations during chart installation/upgrade. Define a pre-install or pre-upgrade hook in your Helm chart.

pre-install-hook.yaml:

apiVersion: batch/v1
kind: Job
metadata:
  name: {{ include "mychart.fullname" . }}-migrations
  annotations:
    "helm.sh/hook": pre-install,pre-upgrade
    "helm.sh/hook-weight": "-5"
    "helm.sh/hook-delete-policy": hook-succeeded
spec:
  template:
    spec:
      containers:
        - name: migrations
          image: your-migrations-image:tag
          command: ["./run-migrations.sh"]

There are pre-install and post-install hooks.

Best practices for Kubernetes migrations

1. Decoupling Migrations from Application Code

   1. Create separate Docker image for migrations.

   2. Use tools like Atlas to manage migrations independently.

2. Version Control for Migrations

   1. Store migration files in your Git repository.

   2. Use sequential or timestamp-based versioning.

3. Idempotent Migrations

   1. Ensure migrations can be run multiple times without side effects.

4. Rollback Strategy

   1. Implement and test rollback procedures for each migration.

5. Monitoring and Logging

   1. Use tools like Atlas Cloud for visibility into migration history.

Conclusion

Managing database migrations in a Kubernetes environment requires careful planning and execution.

By leveraging tools like golang-migrate, goose, or atlas, and following best practices, you can create robust, scalable, and maintainable migration strategies.

Remember to decouple migrations from application code, use version control, and implement proper monitoring to ensure smooth database evolution in your Kubernetes-based architecture.