Merge Cloud App and Cloud Product¶

CCEP	2
Title	Merge Cloud App and Cloud Product
Version	1
Author	Christian Haudum
Date	2019-03-08
Status	Implemented

Introduction¶

On Dec 3rd, 2018 we started implementing a new CrateDB Cloud service for products, named Product API, which resides as a standalone service with its own Github repository. The aim was to create a separation of concerns between core functionality of the Cloud and product functionality, such as deploying, listing, editing and deleting Cloud resources, such as CrateDB clusters and various types of ingest consumers.

However, over the time, it turned out that the decoupling causes various problems:

The relation between products and projects, which is essential, is “broken” by the service boundary.
Also, the service boundary makes any changes to the Product API that depend on the Cloud App incredibly hard to implement and to deploy, since we cannot wait for changes of the Cloud App to be on production before implementing and testing new features of the Product API.
The Apollo client used in the Console (web interface) of the Cloud does not support multiple endpoints, e.g. /graphql for the core service and /product/graphql for the product service.

These and other difficulties observed in the development process led to this proposal.

Proposal¶

This CCEP proposes to merge the Product API project located at crate/cloud-product into the Cloud App project located at crate/cloud-app to form a single project that contains the complete publicly available CrateDB Cloud API.

Project Structure¶

In order to keep the code of the previously decoupled services more modular, they keep their own Python module with the application, thus having core, product and telemetry modules with the app module.

Additionally there may be other modules, such as utils. Root level modules must not depend on entity modules. The project structure should make spotting possible dependencies really easy in code reviews.

Example layout:

.
├── app
│   ├── account
│   │   ├── organizations
│   │   ├── projects
│   │   ├── roles
│   │   └── users
│   ├── core
│   │   ├── db
│   │   ├── graphql
│   │   └── monitoring
│   ├── product
│   │   ├── clusters
│   │   ├── consumers
│   │   └── products
│   └── telemetry
│       ├── logs
│       ├── metrics
│       ├── notifications
│       └── snapshots
├── deploy
│   └── manifests
├── docs
├── images
│   ├── backup
│   ├── metrics_exporter
│   └── restore
└── tests
    ├── doctests
    └── unittests

Each module contains at least a models.py (which keeps the SQLAlchemy model definitions), a types.py (for GraphQL types), a queries.py (which keeps the GraphQL query object types and their respective resolvers), a mutations.py (which keeps the GraphQL mutations), and a controllers.py (which contains the controllers for this module). Additionally other modules, such as utils.py (for utilities) may be present.

Truncated example layout:

.
├── app
│   ├── __init__.py
│   ├── account
│   │   ├── __init__.py
│   │   ├── organizations
│   │   │   ├── __init__.py
│   │   │   ├── controllers.py
│   │   │   ├── models.py
│   │   │   ├── mutations.py
│   │   │   ├── queries.py
│   │   │   └── types.py
...

Database Layout¶

Merging the two projects together also allows to simplify database access. All relations are moved into a single database schema named core. This then requires only a single database user core to have DQL and DMQ access on schema core.

The database schema and access of the telemetry cluster may stay as is.

Impact¶

This CCEP does not have any impact on other proposals.