=================== A RESTful Structure =================== +-------------+------------------------------------------------+ | CCEP | 5 | +-------------+------------------------------------------------+ | Title | A RESTful Structure | +-------------+------------------------------------------------+ | Version | 3 | +-------------+------------------------------------------------+ | Author | Markus Holtermann | +-------------+------------------------------------------------+ | Date | 2019-04-05 | +-------------+------------------------------------------------+ | Updated | 2019-11-13 | +-------------+------------------------------------------------+ | Status | Accepted | +-------------+------------------------------------------------+ Introduction ============ The :doc:`CCEP-0004 ` makes a recommendation for using `flask-marshmallow`_ with the underlying `marshmallow`_ (de)serialization and validation library. With the acceptance of that CCEP, this CCEP will outline the project and code structure for ``cloud-app``. Proposal ======== File Structure -------------- The file system structure proposed herein is based on the structure outlined in :doc:`CCEP-0002 ` for the merge of the ``cloud-app`` and ``cloud-product`` services. Entity Modules ^^^^^^^^^^^^^^ Each "entity module" (eg. ``app.core.organizations``, ``app.product.clusters``, ``app.telemetry.logs``) will gain two to three additional Python modules: - ``schemas.py`` - ``validators.py`` - ``views.py`` The ``schemas.py`` module will be the REST equivalent to GraphQL's ``types.py``. It contains the `marshmallow schema definitions`_. Once the GraphQL API can be turned off, removing the ``types.py`` MUST NOT have any implications on the functionality and integrity of the REST API. The ``validators.py`` module is an optional module that may or may not exist. It's meant to hold small, specific validation methods used by the schemas for the same entity module. The ``views.py`` module will contain the native `Flask`_ views. In the long term, these views will replace GraphQL's ``mutations.py`` and ``queries.py`` modules. The same constraint that must hold for ``schemas.py`` must hold for ``views.py`` as well: the module MUST NOT have any dependency on any existing GraphQL code. The requirement to not depend on any GraphQL specific code supports the future removal of the GraphQL. Core Module ^^^^^^^^^^^ Similar to the ``graphql`` Python package ``app.core.graphql``, this CCEP recommends to maintain a ``app.core.marshmallow`` package. This package will be used to maintain project-wide marshmallow specific code, such as nested schemas (e.g. for Dublin Core) in ``schemas.py``, custom fields in ``fields.py``, validators in ``validators.py``, etc:: . └── app └── core └── marshmallow ├── __init__.py ├── decorators.py ├── fields.py ├── schemas.py └── validators.py Code Structure and Naming ------------------------- Schemas ^^^^^^^ Marshmallow 2.x allows non-strict schema validation. Marshmallow 3.x enforces strict schema validation. In order to get the same behavior for Marshmallow 2.x (version 3.x is still in Beta state) and simplify the upgrade process once 3.x is stable, this CCEP recommends to enforce schema validation. This can be achieved by setting the ``strict`` property on a schema's the ``Meta`` class. This is particularly useful for schemas used for deserialization, as it raises errors over silent error flags to check for. As such, the bare layout for a ``schemas.py`` module looks like this: .. code-block:: python from marshmallow import fields from app.core.marshmallow import ma class ProjectSchema(ma.Schema): id = fields.String(dump_only=True) name = fields.String(required=True) class Meta: strict = True project_schema = ProjectSchema() projects_schema = ProjectSchema(many=True) Marshmallow 3.x, unlike 2.x raises errors when fields are provided on the API that don't exist on a schema. To accomplish the same behavior as it was present on Marshmallow 2.x, one MUST include the ``unknown = "exclude"`` schema option on a schema's ``Meta`` class: .. code-block:: python from marshmallow import EXCLUDE, fields from app.core.marshmallow import ma class ProjectSchema(ma.Schema): id = fields.String(dump_only=True) name = fields.String(required=True) class Meta: unknown = EXCLUDE project_schema = ProjectSchema() projects_schema = ProjectSchema(many=True) .. important:: Schemas MUST include a suffix "Schema" in their class name to prevent name collisions on imports when SQLAlchemy models are imported into the same scope. Views ^^^^^ Views should be simple and not encapsulate any logic apart from input validation, permission checks, fetching objects, and rendering. Furthermore, views should only handle the HTTP methods they're meant to do. Furthermore, in order to prevent importing hundreds of views when registering them in the app, each "entity module" should define one or more `Flask Blueprints`_. The Blueprints will then be imported in ``app.main`` and registered in ``register_blueprints()``. Following this concept allows entity modules to decide which endpoints they want and allows the overall application to decide on where it's mounting a set of endpoints. With that in mind, a boilerplate ``views.py`` could look like this: .. code-block:: python from flask import Blueprint, g from app.account.auth.decorators import require_superuser from app.core.marshmallow.decorators import load_schema from .controllers import ProjectController from .models import Project from .schemas import project_schema, projects_schema projects_blueprint = Blueprint("projects", __name__) @projects_blueprint.route("/", methods=["GET"]) @require_superuser def projects_list(): projects = Project.query().all() return projects_schema.jsonify(projects) @projects_blueprint.route("/", methods=["POST"]) @require_superuser @load_schema(project_schema) def project_create(): project = ProjectController().create(g.data) return project_schema.jsonify(project), 201 @projects_blueprint.route("//", methods=["GET"]) @require_superuser def project_detail(id): project = Project.query().filter(Project.id == id).one() return project_schema.jsonify(project) @projects_blueprint.route("//", methods=["PUT"]) @require_superuser @load_schema(project_schema) def project_update(id): project = Project.query().filter(Project.id == id).one() ProjectController().edit(project, g.data) return project_schema.jsonify(project) @projects_blueprint.route("//", methods=["DELETE"]) @require_superuser def project_delete(id): project = Project.query().filter(Project.id == id).one() ProjectController().delete(project) return "", 204 .. note:: The HTTP response status code for the ``POST`` and ``DELETE`` methods are not the usual `200`_ but `201`_ and `204`_ respectively. One should also note that the input validation by using the ``load_schema`` decorator happens *after* the user authentication and basic permission checks. By having the input validation at a later stage, validators attached to a schema can make assumptions about an authenticated user (or not) or the authenticated user being a superuser (or not), etc. A schema that takes the primary key of a related object can thus evaluate if the requesting user has access to the related object. URL Routing ^^^^^^^^^^^ Unlike GraphQL, API versioning is an integral part of a RESTful API. All API endpoints MUST be part of a versioned URL. As outlined in the previous section, entity modules will provide one or more `Flask Blueprints`_. The ``app.main.register_blueprints`` method will gain additional steps adding the blueprints: .. code-block:: python def register_blueprints(app) -> None: # internal-auth routes are used for internal services but since they are # accessible from the outside they are secured via basic-auth. app.register_blueprint(snapshot, url_prefix="/internal-auth/snapshot") app.register_blueprint(webhook, url_prefix="/internal-auth/webhook") app.register_blueprint(projects_blueprint, url_prefix="/api/v2/projects") References ========== There is a `Proof of Concept pull-request`_ showcasing how a more complex marshmallow REST API could look like. Furthermore, there is a `pull-request with a basic implementation`_ for of the ``app.core`` changes .. _flask-marshmallow: https://github.com/marshmallow-code/flask-marshmallow .. _marshmallow: https://marshmallow.readthedocs.io/en/stable/ .. _marshmallow schema definitions: https://marshmallow.readthedocs.io/en/2.x-line/quickstart.html#declaring-schemas .. _Flask: http://flask.pocoo.org/ .. _Flask Blueprints: http://flask.pocoo.org/docs/1.0/blueprints/ .. _200: https://httpstatuses.com/200 .. _201: https://httpstatuses.com/200 .. _204: https://httpstatuses.com/200 .. _Proof of Concept pull-request: https://github.com/crate/cloud-app/pull/715 .. _pull-request with a basic implementation: https://github.com/crate/cloud-app/pull/727