Edge Regions

Bootstrapping edge regions for CrateDB Cloud

Warning

CrateDB Edge (CrateDB Cloud Edge) was renamed to CrateDB Cloud on Kubernetes in February 2024.

Warning

This document is intended for internal use only!

Warning

CrateDB Cloud Edge is beta-level software at the time of writing and changes a lot. Things may not work as expected and generally only the happy path is tested. See the Known Issues section below for known problems and possible solutions.

Overview

CrateDB Edge is a package of components that can be installed on any compatible Kubernetes cluster and connected to CrateDB Cloud as a “region”.

Some of these components are well-known, such as the Grafana suite (Grafana, Prometheus, Loki).

CrateDB Edge Architecture Overview

Components

CrateDB Edge installs several components on the customer’s Kubernetes cluster:

  • Cloud Agent - The HTTPS API endpoint that CrateDB Cloud talks to when interacting with an Edge region.

  • Crate Operator - The Kubernetes operator responsible for provisioning and managing CrateDBs. The Crate Operator is Open Source.

  • Cloud Operator - The Kubernetes operator responsible for provisioning and managing CrateDB Projects.

  • Cloudflare Argo Agent - A reverse proxy / tunnel that exposes the Cloud Agent to be accessible by CrateDB Cloud. The Argo Agent is Open Source.

  • Ingress Nginx - An instance of ingress nginx that is used as an ingress controller for the Cloud Agent.

  • Kubernetes Replicator - An instance of a Kubernetes replicator that is used to replicate certain secrets between namespaces.

  • Prometheus - An instance of Prometheus DB that stores metrics from the Kubernetes cluster and the CrateDBs running on it.

  • Loki - An instance of Loki DB that stores logs from running CrateDBs and other components.

  • Grafana - An instance of Grafana, that is pre-configured to access Prometheus and Loki. Grafana can be used to view logs and metrics from running CrateDBs, and comes pre-configured with a few helpful dashboards.

  • Cert Manager - An instance of cert-manager.io that is used to issue and renew certificates for CrateDBs. Note that self-signed certificates are used by default.

Feature flag

CrateDB Edge is currently behind a feature flag that is only enabled for specific users. The feature flag controls the following actions:

  • Creating new regions

  • Deleting exiting regions

  • Generating the cloud edge Kubernetes manifest

In order to change the feature flag the following commands can be executed:

FLASK_APP=cloud.api.main:create_app flask pancake flags enable-group FEATURE_EDGE_REGIONS user [USER ID]
FLASK_APP=cloud.api.main:create_app flask pancake flags disable-group FEATURE_EDGE_REGIONS user [USER ID]

Please note in order to run these commands in production you must execute them from within a cloud-api app pod.

CloudFlare Argo

CrateDB Edge uses CloudFlare Argo tunnels to access the Agent and Prometheus from behind firewalls. This means that you can install CrateDB Edge on any k8s cluster, even those not exposed to the public internet.

Steps to bootstrap an edge region

What the customer needs to bring

  1. A Kubernetes cluster: The customer needs to set up a Kubernetes cluster that fulfills the following requirements:

    • at least 3 nodes (to run a high-availability setup)

    • depending on the CrateDB cluster you want to run, reserve 1 CPU for OS/node & 2-3 CPUs for other things like cloud-agent, external-dns, Prometheus

    • K8s version >1.15

    • a storage class named crate-premium and crate-local - the installation script will guide with the setup for these.

  2. A CrateDB Cloud account with an organization: The customer needs to create a CrateDB Cloud account and create an organization. This organization will, later on, be the owner of the edge region. This means all the organization users will have access to the edge region; the organization administrators will be allowed to deploy resources to the edge region.

Creating an Edge Region

This is done by going to the Cloud Console (Regions tab) and creating one there. If you don’t see an option to create one, the feature flag might not be enabled.

Installing everything into your Kubernetes cluster

Once the Edge Region is created in the Cloud Console, you will be offered a script you can run to install everything. Copy that and execute it:

bash <(wget -qO- https://console.cratedb-dev.cloud/edge/cratedb-cloud-edge.sh) [YOUR TOKEN]

The script is generally self-explanatory and will inform you if something goes wrong, or if something needs to be done manually.

Accessing Grafana

If Promtail/Loki/Grafana are installed as part of the installation script (and that is recommended), they can only be accessed from within the cluster by default (for security reasons).

You need to run the following commands to access Grafana:

# First obtain the generated password
$ kubectl get secret --namespace crate-loki crate-grafana \
    -o jsonpath="{.data.admin-password}" | base64 --decode ; echo

# Then get the pod name
$ export POD_NAME=$(kubectl get pods --namespace crate-loki \
    -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=crate-grafana" \
    -o jsonpath="{.items[0].metadata.name}")

# And finally create the tunnel
$ kubectl --namespace crate-loki port-forward $POD_NAME 3000

Grafana can now be accessed on http://localhost:3000.

When a region is created, an Argo tunnel configuration is also created for accessing Grafana, although it is not enabled by default (also for security reasons). To enable it, you need to do the following:

# Edit the Argo configmap
$ kubectl -n cloud-app edit configmap cloudflare-tunnel-agent-config

Now find the configuration for tunnel.yaml and add a new entry to the ingress list, for Grafana, pointing to the internal service URL of http://crate-grafana.crate-loki.svc.cluster.local. Your updated tunnel.yaml will look something like this:

tunnel.yaml: |-
  tunnel: [your-tunnel-id]
  credentials-file: /etc/cloudflared/tunnel.json

  ingress:
  - hostname: prometheus-[your-region-id]-edge.cratedb-edge-proxy-dev.net
    service: http://ingress-nginx.cloud-ingress-nginx.svc.cluster.local
  - hostname: grafana-[your-region-id]-edge.cratedb-edge-proxy-dev.net
    service: http://crate-grafana.crate-loki.svc.cluster.local
  - service: http://ingress-nginx.cloud-ingress-nginx.svc.cluster.local

Grafana can now be accessed by going to https://grafana-[your-region-id]-edge.cratedb-edge-proxy-dev.net, courtesy of Argo.

Warning

Please be aware that this makes Grafana publicly accessible, and only protected by the password that was configured when installing it.

Known Issues

Connectivity

Creating Projects and CrateDBs is a fire-and-forget operation - if the cloud-agent on the edge cluster is not reachable, you will not see any errors in the UI and it will seem like it worked.

Deleting and re-installing

If you attempt to uninstall a region and then re-install it, deletion might hang when removing the Project CRD. If you Ctrl+C and attempt to re-install the region, you might find that the CRD is missing and cloud-k8s-operator fails to start. If this happens, simply re-run the installation script and it should fix itself.

Google Kubernetes Engine

The default local storage class does not work out of the box. GKE seems to have issues with the hostpath provisioner. To circumvent, create a copy of the premium storage class.

“hostpath” Storage Class

It’s possible to use the “hostpath” provisioner for the crate-premium storage class (same as crate-local). However, the kubelet_volume_stats_used_bytes metric in Prometheus will not exist as it is only reported for “real” volumes. Consequently, the metric will not be shown in the UI.