Building an application on keepup

Getting started

You need Python 3.11 or newer. SQLite is enough for development; PostgreSQL is for production.

python3 -m venv .venv && . .venv/bin/activate
# the full panel: the HTTP layer and sign-in, the sections, the database and one driver
pip install "keepup-admin[panel,postgres]" uvicorn

Capabilities arrive as extras: one extra, one distribution. The base is the constructor only — the kernel, the plugin catalogue, the service registry, the lifecycle and the transport seam. It declares no library of anybody else's, not even FastAPI; every capability declares its own.

The panel extra installs everything the panel needs but the driver; the driver is chosen separately — postgres in production, sqlite for development. Two drivers of one required service stop the start on purpose.

An application is a file that hands over settings and gets an ASGI application back. The tables plugins declare (get_declared_tables()) are created by the kernel at start; the first accounts are created by keepup-auth:

import os

from keepup import KeepupSettings, create_app

PLUGINS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "plugins")

app = create_app(KeepupSettings(
    title="Your product",
    project_name="yours",
    plugins_dir=PLUGINS_DIR,
    static_mounts=(),
))

{"plugins": [], "roles": []}

Two values are set by whoever deploys, not by the code:

export SECRET_KEY="$(python3 -c 'import secrets; print(secrets.token_urlsafe(48))')"
export ADMIN_INITIAL_PASSWORD="choose-your-own"
uvicorn main:app --port 8000

What answers after that:

• /selfcare — the admin panel: sign in as admin, eight sections; the shell and the sections are installed by keepup-http.
• /api/docs and /api/redoc — the API description.
• /api/health — the health of the process.

The default database is SQLite at data/keepup_db, relative to the working directory.

Checks

The tree for development is installed from the repository: pip install -e ".[test]" -e packages/keepup-*. One command runs all of it: .github/scripts/check.sh — the lint, the suite with coverage, the release and the graph, the security audit and the build of every wheel; --fast skips the wheels and the audit. Tests in the tree are named *_tests.py and run by path: python -m pytest tests/your_tests.py -v.

Project layout

The framework reads data relative to the process's working directory, and plugin code from the directory the application names. So start the process from your application's root:

yours/
├── main.py                   # KeepupSettings -> app
├── plugins/
│   └── hello.py              # class HelloPlugin(BasePlugin)
├── config/
│   ├── modules.json          # plugins, panel sections, roles
│   ├── auth.yaml             # provider: local (optional)
│   └── postgres.properties   # where the database is (optional)
└── static/                   # your own front end, if any

Derive the plugins directory from the file's location rather than from the working directory — then plugins are found wherever the process was started:

import os

PLUGINS_DIR = os.path.join(os.path.dirname(os.path.abspath(__file__)), "plugins")
# The settings name the directory further on: plugins_dir=PLUGINS_DIR.

Two keepup applications in one directory are not untidy but wrong: at start the panel's section catalogue is topped up from config/modules.json, and the second application would receive the first one's sections.

KeepupSettings

The one place to configure. Anything the framework cannot know arrives here; the defaults close rather than open.

profile is a patch over the plugin catalogue (plugins/builtin.json declares "panel" and "metrics-only"), not a second configuration format: one codebase serves a full panel, a worker and a process that answers only /metrics. The metrics-only profile replaced the retired disable_http_server setting. The app_tables and extra_setup settings and the keepup.schema module with its init_db() were removed in 0.5.0.

Plugins and routes

A feature of the application is a plugin: a subclass of BasePlugin in a file of its own. The framework loads it as declared in config/modules.json.

The class name follows from the id

The file is plugins/<id>.py, the class is f"{id.capitalize()}Plugin": hello → HelloPlugin, integration_logs → Integration_logsPlugin. A wrong name does not stop the start: the plugin simply does not load, and the log keeps a warning.

Hooks

A route is a dictionary

There are no decorators: the framework builds the FastAPI wrapper for every route itself, so sign-in, parameter handling and the audit are the same for all of them.

A handler takes current_user: dict and plain keyword arguments: path and query parameters arrive by name, the body of a write arrives in request. An unknown parameter is answered with 400, not 500: when the route declares a mask, the mask decides and converts types; otherwise the handler's signature does (a handler with **kwargs takes everything).

def get_api_routes(self):
    return [{
        "path": "/api/shops/{shop_id}/offers",
        "methods": ["GET"],
        "handler": self.list_offers,
        "params": {
            "shop_id": {"type": "int"},
            "status": {"type": "str", "choices": ["active", "archived"]},
        },
    }]

async def list_offers(self, current_user: dict, shop_id: int, status: str = "active"):
    ...

Declaring and enabling

{
  "plugins": [
    {"id": "hello", "name": "Hello", "enabled": true, "priority": 100,
     "config": {"greeting": "hi"}}
  ],
  "roles": [
    {"name": "ADMIN", "modules": [], "plugins": ["hello"]}
  ]
}

Whether a declared plugin runs is decided in order, strongest first: PLUGINS_DISABLE → PLUGINS_ENABLE → the switch in the panel → the enabled flag in the file. A lower priority initialises earlier. A role's plugins list is visibility in /api/plugins, not enablement.

The kernel

A plugin is loaded by the constructor — the kernel. A plugin declares a descriptor: id, its kind (required, optional, transport), what it needs (requires, wants) and what it gives (provides, contributions). A catalogue is composed of those descriptors: the framework's own catalogue (plugins/builtin.json), the application's file (config/modules.json) and a profile (settings.profile) — each later layer patches the one before. Plugins publish services into the registry (provides) and take other plugins' services through requires; the lifecycle runs them in two phases — first those whose requirements are already met, then the rest. A required plugin and a transport are enabled by being installed; an optional one by the file, by PLUGINS_ENABLE, by a profile or by an administrator.

from keepup.kernel import create_runtime

runtime = create_runtime(settings)   # a Runtime, not yet started

The service names a plugin may require and publish are listed in doc/service-catalogue.md; the constructor's contract is in doc/plugin_constructor.md. If the application built a plugin manager itself, the plugin_manager setting hands it to the kernel.

The panel

The panel lives at /selfcare. Its shell — theme pages, script and styles — ships with keepup-http and is served from shell_mount; the base carries no front end, and there is nothing to copy into your tree.

Eight framework sections

keepup-http installs them and the shell. They are entered into the catalogue at start and granted to ADMIN by default:

• Users — accounts, roles, sessions.
• Sections — the panel's section catalogue and its grants to roles.
• Cluster — replicas, their plugins and state; stop, start, restart.
• Metrics — CPU and memory per replica.
• Themes — the panel's look.
• Events — the application's event log.
• Background tasks — scheduler jobs and distributed locks.
• Integration log — outgoing calls to external systems.

A section of your own

A section is a script and, optionally, a stylesheet declared in modules[] of the same modules.json, plus a line in a role's modules. The shell loads the script and calls window[initFunction]().

{
  "modules": [{
    "id": "orders",
    "name": "Orders",
    "js": "/static/modules/js/orders.js",
    "css": "/static/modules/css/orders.css",
    "initFunction": "initOrdersModule",
    "routePath": "/selfcare/modules/orders",
    "sectionId": "orders-section",
    "config": {}
  }],
  "roles": [{"name": "CLIENT", "modules": ["orders"], "plugins": []}]
}

The catalogue lives in the database; the file is its seed. At start only sections and grants that have no row yet are added — what an administrator switched off in the panel stays switched off.

Themes

The framework imposes no theme of its own: the themes are the keepup-ui distribution (keepup_ui.themes), and the application lists its own in built_in_themes as (name, page_file, brand, logo) tuples. On an empty database the first becomes active; on an existing one the administrator's choice stays.

Database

Where the database is

When DB_TYPE is set, everything comes from the environment: DB_HOST, DB_PORT, DB_NAME, DB_USER, DB_PASSWORD, DB_PATH, DB_POOL_*. When it is not — from config/postgres.properties (db.* keys), with the password still allowed to come from DB_PASSWORD. With neither — SQLite.

export DB_TYPE=postgres DB_HOST=db.internal DB_NAME=yours DB_USER=yours
export DB_PASSWORD=…

Queries

DatabaseManagerV2 is a SQLAlchemy engine with a pool. Parameters are named only.

from keepup_db import DatabaseManagerV2

rows = DatabaseManagerV2.execute(
    "SELECT id, name FROM orders WHERE owner_id = :owner", {"owner": user_id})

with DatabaseManagerV2.get_session() as session:   # commits or rolls back itself
    ...

A table is declared once

There are no migrations. A table is a SQLAlchemy Core declaration for both dialects; ensure_tables() creates whatever is missing at every start: CREATE TABLE IF NOT EXISTS, missing columns, indexes. A new column is just a new Column.

from sqlalchemy import Column, Integer, Text
from keepup_db import tables

orders = tables.table(
    "orders",
    Column("id", Integer, primary_key=True, autoincrement=True),
    Column("owner_id", Integer, nullable=False),
    Column("title", Text, nullable=False),
    tables.foreign_key("owner_id", "users", ("id",), ondelete="CASCADE"),
)

tables.ensure_tables(orders)

Dialect differences live in the declaration (with_variant, ddl_if, tables.per_dialect(postgres=…, sqlite=…)). A column on a table somebody else owns — users, say — is added with tables.ensure_columns("users", Column(…)), not with a second declaration.

Several replicas against one database

An application on keepup is meant to run as several processes behind a load balancer. All coordination goes through the shared database — replicas do not know each other's addresses.

A distributed lock

Background work that must run in one replica is taken under a lock in the distributed_locks table. The holder renews it while working, so the work may take as long as it needs; only the lock of a holder that went quiet is taken over.

from keepup.locks import distributed_lock, with_distributed_lock

@with_distributed_lock("nightly_report", timeout=300)
async def nightly_report():
    ...   # runs in one replica; the others skip this run

async def purge():
    try:
        async with distributed_lock("purge_journal", timeout=1):
            ...
    except Exception:
        pass  # another replica is purging -- that is as good

With the lock taken, the decorator skips the run and returns None; the context manager raises.

The message bus

Set notification_channel and the framework starts a bus on PostgreSQL LISTEN/NOTIFY before the plugins initialise. deliver() sends an envelope to the other replicas and hands it to this one's subscribers; publish() — to the others only.

from keepup import notification_bus

bus = notification_bus.get_notification_bus()
if bus is not None:
    bus.subscribe(self.on_envelope)          # in the plugin's initialize()

await notification_bus.deliver({"kind": "order_changed", "order_id": 42})

The cluster registry

Every replica publishes itself in cluster_members and collects the commands addressed to it from cluster_commands. The panel's Cluster section shows replicas and their plugins and knows three commands: stop — the process stays alive but answers 503 to the API and pauses its scheduler; start — back; restart — a normal shutdown and a re-run of the same command line.

Background jobs

The scheduler keepup-tasks installs (keepup_tasks.scheduler) is one per process; its jobs are visible in the Background tasks section and can be run from there by hand. What keeps a job from running in every replica is not the scheduler but the lock around the job itself.

Users and sign-in

Accounts live in the application's database (provider: local in config/auth.yaml; without the file — the same). There are two roles, ADMIN and CLIENT (keepup.roles), but a user holds a set of roles: it lives in the user_roles table, and the sections and plugins of every role in the set are glued together. Ask with keepup_auth.user_roles.has_role(user, ROLE_ADMIN) (in the panel, window.AppRoles.has(user, role)) rather than comparing user["role"]. The users.role column is gone: in the answers of a sign-in (/api/auth/me and the user list) the role field remains, computed from the set — for older panels. An empty set is refused on write, and access is closed by blocking an account rather than by taking roles away. Signing in issues a token signed with SECRET_KEY, and the panel keeps its session in a cookie.

• Registration is closed until SELF_REGISTRATION_ENABLED is true.
• The password rule is password_rule(password, username) in the settings: return a problem text or None.
• Your own routes outside plugins get the user through the get_current_user and get_current_admin dependencies from keepup_auth.dependencies.

Sign-in through an external provider

OpenID Connect is switched on with one field. Without it the /api/auth/oidc/login and /api/auth/oidc/callback routes are not even registered.

import os

from keepup import KeepupSettings
from keepup.settings import OidcSettings

settings = KeepupSettings(
    title="Your product",
    oidc=OidcSettings(
        issuer="https://id.example.com/realms/yours",
        client_id="yours",
        client_secret=os.environ["OIDC_CLIENT_SECRET"],
        redirect_uri="https://yours.example.com/api/auth/oidc/callback",
        roles_claim="groups",
    ),
)

Without oidc_account_policy, a person the application has never seen is refused. A policy is a function of the provider's claims; ready ones are in keepup_auth.oidc_policy: refuse_unknown(), create_account(), create_if_email_domain([...]) — the last compares the address only when the provider says it verified it.

Observability

Four independent channels, all wired into the application's lifespan.

• An audit of incoming requests — every plugin route is written to incoming_requests through a buffer. Of a request body, only field names are kept by default; otherwise audit_redaction decides (ready ones: keepup_audit.audit.names_only and keep_as_is).
• An event log — await emit_event(event_type, event_text, event_data) from keepup_audit.events; the Events section filters by declared_event_types.
• Metrics — /metrics for Prometheus (without credentials only with metrics_public=True) and CPU and memory snapshots per replica for the panel.
• Logs — rotated locally, and with remote_log_url set, also shipped in batches to your collector.

from keepup_audit.events import emit_event

await emit_event("order_accepted", "Order #42 accepted", {"order_id": 42})

Secure by default

Where a default would have to guess, the framework closes. The application says what it needs open.

The callables plugins give each other through get_handlers() are not published over HTTP. Verification of a provider's tokens refuses symmetric algorithms and none, and answers every failure the same way, so an answer does not tell which check failed.

Public interface and versions

A module an application may import from declares __all__. Everything else is internal and may change without notice, including any name starting with an underscore. A test in the framework's repository holds this against real applications: a name taken from the framework and not declared public fails the run.

A version is three numbers. Major changes what people already rely on: an API that answers differently, a path walked differently. Minor adds capability or removes what is outgrown. The third is an urgent fix of a defect and nothing else.

Where the modules moved

Almost every keepup.<something> name moved to a distribution of its own. There is no compatibility layer: an old import raises ModuleNotFoundError.

There is no keepup/ package directory either: the repository root is the package. What stays in the base is keepup.roles, keepup.locks, keepup.notification_bus, keepup.kernel, keepup.plugins.base, keepup.settings, keepup.security, keepup.cache, keepup.instance, keepup.api_versions and keepup.logging_setup.

What changed from version to version is in the CHANGELOG.