============ Edge Regions ============ **Bootstrapping edge regions for CrateDB Cloud** .. warning:: CrateDB Edge (CrateDB Cloud Edge) was renamed to CrateDB Cloud on Kubernetes in February 2024. .. warning:: This document is intended for internal use only! .. warning:: CrateDB Cloud Edge is beta-level software at the time of writing and changes a lot. Things may not work as expected and generally only the happy path is tested. See the Known Issues section below for known problems and possible solutions. Overview ======== CrateDB Edge is a package of components that can be installed on any compatible Kubernetes cluster and connected to CrateDB Cloud as a "region". Some of these components are well-known, such as the Grafana suite (Grafana, Prometheus, Loki). .. image:: diagrams/cratedb-edge-architecture.svg :width: 100% :alt: CrateDB Edge Architecture Overview Components ========== CrateDB Edge installs several components on the customer's Kubernetes cluster: - **Cloud Agent** - The HTTPS API endpoint that CrateDB Cloud talks to when interacting with an Edge region. - **Crate Operator** - The Kubernetes operator responsible for provisioning and managing CrateDBs. The `Crate Operator`_ is Open Source. - **Cloud Operator** - The Kubernetes operator responsible for provisioning and managing CrateDB Projects. - **Cloudflare Argo Agent** - A reverse proxy / tunnel that exposes the Cloud Agent to be accessible by CrateDB Cloud. The `Argo Agent`_ is Open Source. - **Ingress Nginx** - An instance of `ingress nginx`_ that is used as an ingress controller for the Cloud Agent. - **Kubernetes Replicator** - An instance of a Kubernetes `replicator`_ that is used to replicate certain secrets between namespaces. - **Prometheus** - An instance of Prometheus DB that stores metrics from the Kubernetes cluster and the CrateDBs running on it. - **Loki** - An instance of Loki DB that stores logs from running CrateDBs and other components. - **Grafana** - An instance of Grafana, that is pre-configured to access Prometheus and Loki. Grafana can be used to view logs and metrics from running CrateDBs, and comes pre-configured with a few helpful dashboards. - **Cert Manager** - An instance of `cert-manager.io`_ that is used to issue and renew certificates for CrateDBs. Note that self-signed certificates are used by default. Feature flag ============ CrateDB Edge is currently behind a feature flag that is only enabled for specific users. The feature flag controls the following actions: - Creating new regions - Deleting exiting regions - Generating the cloud edge Kubernetes manifest In order to change the feature flag the following commands can be executed: .. code-block:: bash FLASK_APP=cloud.api.main:create_app flask pancake flags enable-group FEATURE_EDGE_REGIONS user [USER ID] FLASK_APP=cloud.api.main:create_app flask pancake flags disable-group FEATURE_EDGE_REGIONS user [USER ID] Please note in order to run these commands in production you must execute them from within a cloud-api app pod. CloudFlare Argo =============== CrateDB Edge uses CloudFlare Argo tunnels to access the Agent and Prometheus from behind firewalls. This means that you can install CrateDB Edge on any k8s cluster, even those not exposed to the public internet. Steps to bootstrap an edge region ================================= What the customer needs to bring -------------------------------- 1. **A Kubernetes cluster**: The customer needs to set up a Kubernetes cluster that fulfills the following requirements: - at least 3 nodes (to run a high-availability setup) - depending on the CrateDB cluster you want to run, reserve 1 CPU for OS/node & 2-3 CPUs for other things like cloud-agent, external-dns, Prometheus - K8s version ``>1.15`` - a storage class named ``crate-premium`` and ``crate-local`` - the installation script will guide with the setup for these. 2. **A CrateDB Cloud account with an organization**: The customer needs to create a CrateDB Cloud account and create an organization. This organization will, later on, be the owner of the edge region. This means all the organization users will have access to the edge region; the organization administrators will be allowed to deploy resources to the edge region. Creating an Edge Region ----------------------- This is done by going to the Cloud Console (Regions tab) and creating one there. If you don't see an option to create one, the feature flag might not be enabled. Installing everything into your Kubernetes cluster -------------------------------------------------- Once the Edge Region is created in the Cloud Console, you will be offered a script you can run to install everything. Copy that and execute it: .. code-block:: bash bash <(wget -qO- https://console.cratedb-dev.cloud/edge/cratedb-cloud-edge.sh) [YOUR TOKEN] The script is generally self-explanatory and will inform you if something goes wrong, or if something needs to be done manually. Accessing Grafana ----------------- If Promtail/Loki/Grafana are installed as part of the installation script (and that is recommended), they can only be accessed from within the cluster by default (for security reasons). You need to run the following commands to access Grafana: .. code-block:: bash # First obtain the generated password $ kubectl get secret --namespace crate-loki crate-grafana \ -o jsonpath="{.data.admin-password}" | base64 --decode ; echo # Then get the pod name $ export POD_NAME=$(kubectl get pods --namespace crate-loki \ -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=crate-grafana" \ -o jsonpath="{.items[0].metadata.name}") # And finally create the tunnel $ kubectl --namespace crate-loki port-forward $POD_NAME 3000 Grafana can now be accessed on http://localhost:3000. When a region is created, an Argo tunnel configuration is also created for accessing Grafana, although it is not enabled by default (also for security reasons). To enable it, you need to do the following: .. code-block:: bash # Edit the Argo configmap $ kubectl -n cloud-app edit configmap cloudflare-tunnel-agent-config Now find the configuration for ``tunnel.yaml`` and add a new entry to the ingress list, for Grafana, pointing to the internal service URL of ``http://crate-grafana.crate-loki.svc.cluster.local``. Your updated tunnel.yaml will look something like this: .. code-block:: yaml tunnel.yaml: |- tunnel: [your-tunnel-id] credentials-file: /etc/cloudflared/tunnel.json ingress: - hostname: prometheus-[your-region-id]-edge.cratedb-edge-proxy-dev.net service: http://ingress-nginx.cloud-ingress-nginx.svc.cluster.local - hostname: grafana-[your-region-id]-edge.cratedb-edge-proxy-dev.net service: http://crate-grafana.crate-loki.svc.cluster.local - service: http://ingress-nginx.cloud-ingress-nginx.svc.cluster.local Grafana can now be accessed by going to ``https://grafana-[your-region-id]-edge.cratedb-edge-proxy-dev.net``, courtesy of Argo. .. warning:: Please be aware that this makes Grafana publicly accessible, and only protected by the password that was configured when installing it. Known Issues ============ Connectivity ------------ Creating Projects and CrateDBs is a fire-and-forget operation - if the ``cloud-agent`` on the edge cluster is not reachable, you will not see any errors in the UI and it will seem like it worked. Deleting and re-installing -------------------------- If you attempt to uninstall a region and then re-install it, deletion might hang when removing the Project CRD. If you Ctrl+C and attempt to re-install the region, you might find that the CRD is missing and ``cloud-k8s-operator`` fails to start. If this happens, simply re-run the installation script and it should fix itself. Google Kubernetes Engine ------------------------ The default local storage class does not work out of the box. GKE seems to have issues with the hostpath provisioner. To circumvent, create a copy of the premium storage class. "hostpath" Storage Class ------------------------ It's possible to use the "hostpath" provisioner for the ``crate-premium`` storage class (same as ``crate-local``). However, the ``kubelet_volume_stats_used_bytes`` metric in Prometheus will not exist as it is only reported for "real" volumes. Consequently, the metric will not be shown in the UI. .. _cert-manager.io: https://cert-manager.io/ .. _ingress nginx: https://kubernetes.github.io/ingress-nginx/ .. _replicator: https://github.com/mittwald/kubernetes-replicator .. _Crate Operator: https://github.com/crate/crate-operator .. _Argo Agent: https://github.com/cloudflare/cloudflared