A RESTful Structure¶

CCEP	5
Title	A RESTful Structure
Version	3
Author	Markus Holtermann
Date	2019-04-05
Updated	2019-11-13
Status	Accepted

Introduction¶

The CCEP-0004 makes a recommendation for using flask-marshmallow with the underlying marshmallow (de)serialization and validation library. With the acceptance of that CCEP, this CCEP will outline the project and code structure for cloud-app.

Proposal¶

File Structure¶

The file system structure proposed herein is based on the structure outlined in CCEP-0002 for the merge of the cloud-app and cloud-product services.

Entity Modules¶

Each “entity module” (eg. app.core.organizations, app.product.clusters, app.telemetry.logs) will gain two to three additional Python modules:

schemas.py
validators.py
views.py

The schemas.py module will be the REST equivalent to GraphQL’s types.py. It contains the marshmallow schema definitions. Once the GraphQL API can be turned off, removing the types.py MUST NOT have any implications on the functionality and integrity of the REST API.

The validators.py module is an optional module that may or may not exist. It’s meant to hold small, specific validation methods used by the schemas for the same entity module.

The views.py module will contain the native Flask views. In the long term, these views will replace GraphQL’s mutations.py and queries.py modules. The same constraint that must hold for schemas.py must hold for views.py as well: the module MUST NOT have any dependency on any existing GraphQL code.

The requirement to not depend on any GraphQL specific code supports the future removal of the GraphQL.

Core Module¶

Similar to the graphql Python package app.core.graphql, this CCEP recommends to maintain a app.core.marshmallow package. This package will be used to maintain project-wide marshmallow specific code, such as nested schemas (e.g. for Dublin Core) in schemas.py, custom fields in fields.py, validators in validators.py, etc:

.
└── app
    └── core
        └── marshmallow
            ├── __init__.py
            ├── decorators.py
            ├── fields.py
            ├── schemas.py
            └── validators.py

Code Structure and Naming¶

Schemas¶

Marshmallow 2.x allows non-strict schema validation. Marshmallow 3.x enforces strict schema validation. In order to get the same behavior for Marshmallow 2.x (version 3.x is still in Beta state) and simplify the upgrade process once 3.x is stable, this CCEP recommends to enforce schema validation. This can be achieved by setting the strict property on a schema’s the Meta class. This is particularly useful for schemas used for deserialization, as it raises errors over silent error flags to check for.

As such, the bare layout for a schemas.py module looks like this:

from marshmallow import fields

from app.core.marshmallow import ma


class ProjectSchema(ma.Schema):
    id = fields.String(dump_only=True)
    name = fields.String(required=True)

    class Meta:
        strict = True


project_schema = ProjectSchema()
projects_schema = ProjectSchema(many=True)

Marshmallow 3.x, unlike 2.x raises errors when fields are provided on the API that don’t exist on a schema. To accomplish the same behavior as it was present on Marshmallow 2.x, one MUST include the unknown = "exclude" schema option on a schema’s Meta class:

from marshmallow import EXCLUDE, fields

from app.core.marshmallow import ma


class ProjectSchema(ma.Schema):
    id = fields.String(dump_only=True)
    name = fields.String(required=True)

    class Meta:
        unknown = EXCLUDE


project_schema = ProjectSchema()
projects_schema = ProjectSchema(many=True)

Important

Schemas MUST include a suffix “Schema” in their class name to prevent name collisions on imports when SQLAlchemy models are imported into the same scope.

Views¶

Views should be simple and not encapsulate any logic apart from input validation, permission checks, fetching objects, and rendering. Furthermore, views should only handle the HTTP methods they’re meant to do.

Furthermore, in order to prevent importing hundreds of views when registering them in the app, each “entity module” should define one or more Flask Blueprints. The Blueprints will then be imported in app.main and registered in register_blueprints(). Following this concept allows entity modules to decide which endpoints they want and allows the overall application to decide on where it’s mounting a set of endpoints.

With that in mind, a boilerplate views.py could look like this:

from flask import Blueprint, g

from app.account.auth.decorators import require_superuser
from app.core.marshmallow.decorators import load_schema

from .controllers import ProjectController
from .models import Project
from .schemas import project_schema, projects_schema

projects_blueprint = Blueprint("projects", __name__)


@projects_blueprint.route("/", methods=["GET"])
@require_superuser
def projects_list():
    projects = Project.query().all()
    return projects_schema.jsonify(projects)


@projects_blueprint.route("/", methods=["POST"])
@require_superuser
@load_schema(project_schema)
def project_create():
    project = ProjectController().create(g.data)
    return project_schema.jsonify(project), 201


@projects_blueprint.route("/<id>/", methods=["GET"])
@require_superuser
def project_detail(id):
    project = Project.query().filter(Project.id == id).one()
    return project_schema.jsonify(project)


@projects_blueprint.route("/<id>/", methods=["PUT"])
@require_superuser
@load_schema(project_schema)
def project_update(id):
    project = Project.query().filter(Project.id == id).one()
    ProjectController().edit(project, g.data)
    return project_schema.jsonify(project)


@projects_blueprint.route("/<id>/", methods=["DELETE"])
@require_superuser
def project_delete(id):
    project = Project.query().filter(Project.id == id).one()
    ProjectController().delete(project)
    return "", 204

Note

The HTTP response status code for the POST and DELETE methods are not the usual 200 but 201 and 204 respectively.

One should also note that the input validation by using the load_schema decorator happens after the user authentication and basic permission checks. By having the input validation at a later stage, validators attached to a schema can make assumptions about an authenticated user (or not) or the authenticated user being a superuser (or not), etc. A schema that takes the primary key of a related object can thus evaluate if the requesting user has access to the related object.

URL Routing¶

Unlike GraphQL, API versioning is an integral part of a RESTful API. All API endpoints MUST be part of a versioned URL. As outlined in the previous section, entity modules will provide one or more Flask Blueprints.

The app.main.register_blueprints method will gain additional steps adding the blueprints:

def register_blueprints(app) -> None:
    # internal-auth routes are used for internal services but since they are
    # accessible from the outside they are secured via basic-auth.
    app.register_blueprint(snapshot, url_prefix="/internal-auth/snapshot")
    app.register_blueprint(webhook, url_prefix="/internal-auth/webhook")
    app.register_blueprint(projects_blueprint, url_prefix="/api/v2/projects")

References¶

There is a Proof of Concept pull-request showcasing how a more complex marshmallow REST API could look like.

Furthermore, there is a pull-request with a basic implementation for of the app.core changes