Resource Management with the CockroachDB Operator

On this page

This page explains how to configure Kubernetes cluster resources such as memory, CPU, and storage.

On a production cluster, the resources you allocate to CockroachDB should be proportionate to your machine types and workload. Cockroach Labs recommends that you determine and set these values before deploying the cluster, but you can also update the values on a running cluster.

Memory and CPU

You can set the CPU and memory resources allocated to the CockroachDB container on each pod.

Specify CPU and memory values in cockroachdb.crdbCluster.resources.limits and cockroachdb.crdbCluster.resources.requests in the values file used to deploy the cluster:

cockroachdb:
  crdbCluster:
    resources:
      limits:
        cpu: 4000m
        memory: 16Gi
      requests:
        cpu: 4000m
        memory: 16Gi

Apply the new settings to the cluster:

helm upgrade --reuse-values $CRDBCLUSTER ./cockroachdb-parent/charts/cockroachdb --values ./cockroachdb-parent/charts/cockroachdb/values.yaml -n $NAMESPACE

Cockroach Labs recommends using identical values for resources.requests and resources.limits. When setting the new values, note that not all of a pod's resources will be available to the CockroachDB container. This is because a fraction of the CPU and memory is reserved for Kubernetes.

For more information on how Kubernetes handles resources, see the Kubernetes documentation.

Cache and SQL memory size

Each CockroachDB node reserves a portion of its available memory for its cache and for storing temporary data for SQL queries. For more information on these settings, see the Production Checklist.

The CockroachDB operator dynamically sets cache size and SQL memory size each to 25% (the recommended percentage) of the available memory, which depends on the memory request and limit you specified for your configuration. These values can be modified by adding the cache or max-sql-memory fields to cockroachdb.crdbCluster.flags, which is equivalent to appending --cache or --max-sql-memory as cockroach start flags.

Persistent storage

When you start your cluster, Kubernetes dynamically provisions and mounts a persistent volume into each pod. For more information on persistent volumes, see the Kubernetes documentation.

The storage capacity of each volume is set in cockroachdb.crdbCluster.dataStore.volumeClaimTemplate.spec.resources in the values file used to deploy the cluster:

cockroachdb:
  crdbCluster:
    dataStore:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: "10Gi"

You should provision an appropriate amount of disk storage for your workload. For recommendations on this, see the Production Checklist.

Expand disk size

If you discover that you need more capacity, you can expand the persistent volumes on a running cluster. Increasing disk size is often beneficial for CockroachDB performance.

Specify a new volume size in the values file used to deploy the cluster:

cockroachdb:
  crdbCluster:
    dataStore:
      volumeClaimTemplate:
        spec:
          resources:
            requests:
              storage: "100Gi"

Apply the new settings to the cluster:

helm upgrade --reuse-values $CRDBCLUSTER ./cockroachdb-parent/charts/cockroachdb --values ./cockroachdb-parent/charts/cockroachdb/values.yaml -n $NAMESPACE

The CockroachDB operator updates all nodes and triggers a rolling restart of the pods with the new storage capacity.

To verify that the storage capacity has been updated, run kubectl get pvc to view the persistent volume claims (PVCs). It will take a few minutes before the PVCs are completely updated.

Network ports

The CockroachDB operator separates network traffic into three ports:

Specify alternate port numbers in cockroachdb.crdbCluster.service.ports of the CockroachDB operator's custom resource (for example, to match the default port 5432 on PostgreSQL):

cockroachdb:
  crdbCluster:
    service:
      ports:
        sql: 5432

Apply the new settings to the cluster:

helm upgrade --reuse-values $CRDBCLUSTER ./cockroachdb-parent/charts/cockroachdb --values ./cockroachdb-parent/charts/cockroachdb/values.yaml -n $NAMESPACE

The CockroachDB operator updates all nodes and triggers a rolling restart of the pods with the new port settings.

Ingress

You can configure an Ingress object to expose an internal HTTP or SQL ClusterIP service through a hostname.

In order to use the Ingress resource, your cluster must be running an Ingress controller for load balancing. This is not handled by the CockroachDB operator and must be deployed separately.

Specify Ingress objects in cockroachdb.crdbCluster.service.ingress. Set ingress.enabled to true and specify ingress.ui (HTTP) or ingress.sql (SQL) in the values file used to deploy the cluster:

cockroachdb:
  crdbCluster:
    service:
      ingress:
        enabled: true
        ui:
          ingressClassName: nginx
          annotations:
            key: value
          host: ui.example.com
        sql:
          ingressClassName: nginx
          annotations:
            key: value
          host: sql.example.com

• ingressClassName specifies the IngressClass of the Ingress controller. This example uses the nginx controller.

• The host must be made publicly accessible. For example, create a route in Amazon Route 53, or add an entry to /etc/hosts that maps the IP address of the Ingress controller to the hostname.

  Note:

  Multiple hosts can be mapped to the same Ingress controller IP.

• TCP connections for SQL clients must be enabled for the Ingress controller. For an example, see the nginx documentation.

  Note:

  Changing the SQL Ingress host on a running deployment will cause a rolling restart of the cluster, due to new node certificates being generated for the SQL host.