Architecture

Marketing data moves through Agoranomos in one direction, from ad platforms to dashboards and AI clients, and every step is code that runs on infrastructure you control.

1. Load. dlt pulls data from ad platform APIs, web analytics, CRM systems and files into PostgreSQL.
2. Transform. dbt turns the raw tables into one unified model of advertising statistics.
3. Orchestrate. Apache Airflow schedules the loads and the dbt build, and refreshes Metabase afterwards.
4. Explore. Metabase serves ready-made dashboards, metrics and a glossary from the final layer of the warehouse.
5. Ask. AI clients reach the same data through MCP servers with least-privilege defaults.

Overview

flowchart TB
    sources["<b>Sources</b><br/>ad platform APIs, GA4, CRM, files,<br/>demo generator of fictional platforms"]
    dlt["<b>dlt</b><br/>incremental load"]
    airflow["<b>Apache Airflow</b><br/>schedules loads,<br/>dbt builds and<br/>Metabase refreshes"]

    subgraph warehouse["PostgreSQL warehouse"]
        direction TB
        raw["raw_#lt;source#gt;"] -->|dbt| staging --> intermediate --> marts
    end

    metabase["<b>Metabase</b><br/>dashboards, metrics, glossary"]
    mcp["<b>MCP servers</b><br/>Metabase, Airflow, warehouse, dbt<br/>least privilege by default"]
    clients["<b>AI clients</b><br/>Claude, ChatGPT,<br/>Claude Code, Cursor,<br/>VS Code, Codex"]

    sources --> dlt --> warehouse
    warehouse -->|marts only| metabase
    metabase --> mcp
    warehouse --> mcp
    airflow --> mcp
    mcp --> clients
    airflow -.-> dlt
    airflow -.->|dbt build| warehouse
    airflow -.->|refresh| metabase

Solid arrows show how data moves. Dotted arrows show what Airflow runs.

Components

Load: dlt

dlt takes care of incremental loading, schema evolution, merging and name normalization. Every source is a dlt source that follows one shared contract:

• Date window. Each load covers a date window in the time zone of the ad account: final data up to yesterday, plus a sliding reload window because attribution keeps maturing after the fact.
• Safe repeats. Each load replaces the facts of each account in the days of its window, deleted and inserted in one transaction, the days that come back empty included, so a repeat adds no duplicate.
• Explicit retries. How failed requests are retried is set explicitly, not left to library defaults.
• Loud failures. If an account fails, the run fails or raises an alert that lists the affected accounts and the original API error.

The connectors of real platforms are to add four rules to this contract (planned with the connectors):

• An app of the deployment. Each deployment registers its own app or project with each platform and keeps its secrets in .env: the access level, the quotas and the OAuth client of a platform belong to the app, so one app cannot serve every deployment.
• Tokens that expire. A source refreshes its access token, and says loudly before a token expires and needs a new sign-in: a long-lived user access token of Meta lasts about 60 days (as of October 2026).
• Asynchronous reports. Where a platform builds a report on request, as Microsoft Advertising does, a source submits it, polls until it is ready and downloads it, with the same explicit retries and loud failures as any other request.
• Daily quotas. The daily load of every account of a deployment fits the daily quota of the lowest access level of each platform that reads real accounts, such as the 2,880 operations a day of the Explorer access of the Google Ads API (as of October 2026).

The planned sources are ad platform APIs (for example Google, Meta, TikTok, Microsoft Advertising and LinkedIn), GA4, CRM systems and files. The demo needs no keys: its generator makes 18 months of a fictional business on four fictional platforms, in the shapes of the APIs of Google Ads, Meta, TikTok and GA4, and a fictional source of exchange rates, up to yesterday in the time zone of each account. The DAG agoranomos_demo_history loads them as the loader role, one source after another, into raw_demo_<platform>, and demo-init runs it on the first start (ADR-0025). The first task of each run of the transform also loads, with dlt, the settings of the model, the clients of the deployment and its conversion actions, from extensions/model/ into raw_settings (ADR-0024). The DAG agoranomos_demo_daily brings the new days of the demo every day at 09:17 UTC: for each account the days since its last load, and at least the recent days whose numbers can still change on its platform; demo-init unpauses it after the history (See the demo).

Decision record: ADR-0004: Ingest data with dlt.

Transform and store: dbt and PostgreSQL

Agoranomos runs two PostgreSQL 18 instances:

• warehouse holds the analytical data.
• platform holds the metadata database of Airflow and the application database of Metabase, each with its own owner, and celery_results, which only the CeleryExecutor variant uses, with the results of its tasks.

Keeping them apart means heavy analytical queries cannot slow the scheduler, and each instance has its own tuning, backups and upgrade path. These two databases are small, so one instance with separate databases and owners is enough for both ("database per service" without extra instances).

Inside the warehouse, data moves through layers, each in its own schema:

Access is split by role. dlt loads the raw_<source> schemas and raw_settings as the loader role, and dbt builds staging, marts and ops, and intermediate once that schema exists, as the transformer role. Metabase connects as the metabase_reader role, which sees marts and, in ops, the views of the Ops collection only, and the warehouse MCP server connects as the ai_reader role, which reads marts and staging in read-only sessions with a statement timeout.

Today the dbt project in dbt/ builds the model from the demo data (The data model). Each platform has a staging view for each of its tables, each a call of one macro for the shape of its API, which the connector of that platform is to call too. The steps that the platforms share union them and resolve the clients, the conversion actions and the exchange rates, as views in staging. Nine marts are tables that dbt rebuilds in full on each run: the facts ad_stats_campaign_daily, ad_conversions_campaign_daily and web_sessions_daily, and the dimensions ad_platforms, ad_accounts, clients, ad_campaigns, ad_campaign_names and fx_rates_daily. The critical tests of the staging views and of the shared steps guard the marts: when one of them fails, the models that read it are not rebuilt, and the marts keep their last good data.

Every model, seed, column and source table of the project has a description. Every mart also has a display name for Metabase, and each of its columns a display name and a semantic type (Describe the marts). dbt writes the descriptions into the comments of the tables, views and columns that it builds, so SQL clients read them in the warehouse, and so will the planned AI access. Each mart has an enforced contract: its columns, their types and its primary key are fixed in its YAML, and a build that would change them fails.

dbt Core with dbt-postgres does the transformations, with tests from the first day. The package elementary records the result of every model, seed and test of each run in the schema ops, which the Ops collection shows; its report is planned (ADR-0029). The marts layer is one model of advertising statistics that spans all platforms (ADR-0025):

• Additive metrics are stored by campaign and day, with the conversions by action in a table of their own; ad groups and ads come with the first real connector.
• Non-additive metrics, such as reach and frequency, are to be separate period facts with calendar keys.
• Money is kept in the currency of the ad account and in one reporting currency, at the rate of the day, and each day is the day of the time zone of the ad account.
• The facts carry the current names of campaigns, and a dimension keeps the names that each campaign had, as dlt records them with its scd2 strategy.
• One deployment holds the data of one business, and clients group its ad accounts (ADR-0024).

Decision records: ADR-0003: Run two PostgreSQL instances and ADR-0005: Transform data with dbt Core.

Orchestrate: Apache Airflow

Airflow 3.3 runs as separate services (API server, scheduler, DAG processor and triggerer) on an image of its own, built from the official apache/airflow image: airflow/Dockerfile adds dlt and Cosmos to Airflow's environment and dbt in a virtual environment of its own. By default it uses the LocalExecutor, which runs each task as a process of the scheduler, so there is no broker and no worker container, and a single machine keeps the memory for the pipelines (ADR-0015). The tasks share the memory of the scheduler container, and Airflow's documentation warns that in a container a task that runs out of memory can restart the scheduler: a plugin in the image therefore makes each task process the first that the kernel stops at the memory limit, on a host whose kernel restricts ptrace with Yama, such as Ubuntu; on Docker Desktop for macOS, which has no Yama, a task can still stop the scheduler (System requirements). A deployment that needs separate workers, possibly on other machines, selects the CeleryExecutor variant with COMPOSE_FILE=compose.yaml:compose.celery.yaml in .env. That variant adds Valkey as the message broker (BSD-licensed, speaking the Redis protocol) and an airflow-worker service; Celery results go to a database of their own on the platform instance, celery_results, so that the worker holds neither the password of the metadata database of Airflow nor the key that signs its tokens, as the security model of Airflow asks; workers can be scaled out horizontally. AIRFLOW_PARALLELISM caps the tasks of the whole stack in both variants, so it rises together with the workers. Authentication uses Airflow's FAB auth manager with the Admin, Op and Viewer roles.

Airflow Assets chain the pipeline, so each step starts when the previous one has produced its data: ingest_<source>, then dbt build, then a Metabase cache refresh. Today the demo chains the first two steps this way: each of the five loads of agoranomos_demo_history, and of agoranomos_demo_daily, updates the asset of the main table of its platform, and agoranomos_demo_transform runs when all five have updated since its last run. Its first task loads the settings of the model, and the tasks of dbt follow. The Metabase refresh is planned. dbt runs through Cosmos, in its own virtual environment so that dbt's dependencies stay separate from Airflow's. Cosmos renders each model and each seed of dbt as one Airflow task, which builds it and then runs its tests (dbt build, ADR-0028), from the manifest of the dbt project, which metadata-export writes on every start, and Airflow runs at most two of them at a time (ADR-0027).

Agoranomos uses Astronomer's open-source projects (Cosmos and astro-airflow-mcp) but not the Astro CLI or Astro Runtime. The stack stays vendor-neutral and builds on the official Apache Airflow image.

Decision records: ADR-0002: Orchestrate with Apache Airflow 3 and, for the executor, ADR-0015: Run LocalExecutor by default.

Explore: Metabase

Metabase OSS reads the marts schema, and the views of the Ops collection in ops, through the metabase_reader role. It comes with ready-made dashboards, metrics and a glossary. Two one-shot jobs set Metabase up through its API: metabase-init creates the admin, connects the warehouse and keeps the Ops collection, a dashboard about the pipeline itself (Watch the pipeline), and, with the demo profile, demo-init puts the demo collection, its questions and its two dashboards into the collection Agoranomos from code after it runs the demo pipeline once (See the demo). On every start, metabase-init also applies the metadata of the marts: the display names, descriptions and semantic types that the one-shot metadata-export job reads from the dbt project with dbt ls, and the overrides of a deployment in extensions/metadata/overrides.toml (Rename a field or add your own definitions). The code always wins: an edit of these in the Metabase UI is reverted at the next start, and the log of the job says so (ADR-0016). demo-init applies them as well, after its sync, and gives the demo questions the titles of their columns. The built-in MCP server (available since Metabase 0.60) follows the level of AI access in .env, which a new installation starts at build, and metabase-init sets the permissions of the level on every start, as ADR-0017 decides. The content of the package is in the collection Agoranomos, view-only for everyone but admins, and the setup jobs put it back as the code says on every start, finding it by the entity IDs that Metabase gave it (ADR-0018).

Decision record: ADR-0006: Use Metabase as the BI layer.

Ask: MCP servers

AI access goes through four MCP servers. Each starts with least-privilege defaults. v0.3.0 is to ship three of them, with the Airflow and warehouse servers in local mode only, and dbt MCP in a later release (ADR-0030). The servers of Metabase, of Airflow and of the warehouse run today; dbt MCP is planned:

These are the defaults of ADR-0007; the Metabase row is as ADR-0017 sets it. In Metabase a client may ask for any OAuth scope, and the person, not Metabase, chooses which to grant, so the limit is the permissions of the account. A deployment sets a level in .env: a ceiling, and a default that every account gets, in the Metabase UI too, on every database and every shared collection. An admin raises people above the default through groups that the setup creates. The content of the package and the warehouse stay out of reach at every level, and Agoranomos sets up no email server or Slack for Metabase, so no level sends alerts or subscriptions. The permissions of the Airflow and warehouse servers are those of ADR-0007, and each of the two also has a switch of its own in .env (Configuration). In server mode the proxy closes the registration of OAuth clients and lets a token reach only the MCP server; ./agoranomos ai connect registers the clients, and ./agoranomos ai revoke and the level off delete them (ADR-0031).

Local mode serves coding agents such as Claude Code on the same machine; the desktop and mobile apps of Claude and ChatGPT reach Metabase's MCP server over HTTPS in server mode.

• Why stdio through docker exec instead of HTTP ports. For the stdio servers, no port is opened, there is no risk of DNS rebinding against localhost, and credentials stay inside the container. In server mode, remote clients reach Metabase's built-in server over HTTPS with OAuth. The stdio servers do not run there in v0.3.0, as the memory budget has no room for their container (ADR-0030), and they stay on stdio until a gateway is added.
• Ready-made context. The repository provides AGENTS.md (the cross-agent standard) and a CLAUDE.md that imports it. Its .mcp.json gives Claude Code the Airflow and warehouse servers; the server of Metabase needs a client ID of each deployment, which ./agoranomos ai connect claude-code registers. Its product skills in .claude/skills/ add a source, diagnose a failed pipeline, build a dashboard and explain a metric, and the guide Talk to your data connects each client. A Claude Code plugin and configuration for Cursor, VS Code and Codex are planned for later releases.
• Untrusted text. Campaign names and ad texts are untrusted data, because anyone can write them. AGENTS.md states that data is never an instruction, and least-privilege defaults limit the damage if an injection attempt succeeds.

Decision records: ADR-0007: Give AI clients least-privilege access through MCP and, for the transport in server mode, ADR-0011: Support local and server modes.

Run: containers

The stack runs as one Docker Compose project, split into profiles so that you start only what you need:

• core: both PostgreSQL instances, Airflow and Metabase, and the one-shot jobs that set them up on every start: db-init (the roles, databases and schemas), airflow-init (the migration of the metadata database and the admin), metadata-export (the export of the metadata of the marts from dbt) and metabase-init (the first setup of Metabase, its connection to the warehouse, and the metadata of the marts). The CeleryExecutor variant of Airflow, which compose.celery.yaml adds, puts the Valkey broker and a worker in it.
• demo: the one-shot demo-init job, which makes the demo ready on the first start; .env turns it on by default (See the demo).
• server: the proxy service of server mode, Caddy in front of Metabase on ports 80 and 443; ./agoranomos init --server turns it on (ADR-0019).
• ops: the backup service, which takes a backup of the databases every day; ./agoranomos init --server turns it on (ADR-0020).
• ai: one container with the Airflow and Toolbox MCP servers, in local mode, which an AI client reaches over stdio with ./agoranomos mcp, through docker compose exec; a new .env turns it on in local mode, and ./agoranomos init --server takes it out. The dbt MCP server is to come later, in a container of its own (ADR-0030).

Each container gets the Linux privileges that its job needs and no more. Every service drops all Linux capabilities, starts as a user that is not root and may not gain privileges, and every one-shot job but airflow-init has a read-only file system. The secrets of a container can be wider than its job: the containers that run the tasks, the scheduler with the LocalExecutor and the worker with the CeleryExecutor, hold keys of Airflow and the passwords of the warehouse roles, and the scheduler also the password of the metadata database of Airflow, and task code can read them (the worker holds neither that password nor the key that signs the tokens of Airflow, but task code there can still run code in the scheduler through the results of Celery, which the scheduler unpickles: ADR-0015) (ADR-0013 and ADR-0015 record the passwords; the keys are in the same environment, Configuration lists them). The databases and the broker are on an internal network with no route to the internet, and no container mounts the Docker socket. Every service has a memory limit, so that, on the recommended memory (8 GB, or 12 GB for the CeleryExecutor variant), one that runs short of memory cannot take it from the others (System requirements says what holds on less). Metabase asks no service on the internet where the addresses of its users are (Configuration).

The stack has two modes, local and server. Both are first-class, and CI tests both. Local mode runs today, with its ports published on 127.0.0.1 only (Run Agoranomos locally). Server mode runs today too: Caddy, the server profile, puts Metabase on HTTPS on ports 80 and 443 only, and Airflow stays reachable through an SSH tunnel (ADR-0019, Deploy on a server). Its daily backups, the ops profile, run too (ADR-0020, Back up and restore).

The stack needs 8 GB of RAM available to Docker (6 GB as a minimum, and 12 GB recommended for the CeleryExecutor variant), 4 CPUs and 20 GB of disk, on x86-64-v2 or ARM64 hardware such as Apple Silicon. The memory limits of the services come from measurements on the demo (System requirements).

Decision records: ADR-0010: Support amd64 and arm64 within a memory budget covers the architectures, ADR-0022: Budget the memory by variant and mode the memory budget, and ADR-0011: Support local and server modes the two modes.

Decisions

The reasons behind the choices about each component, the container layout and the two modes, and the alternatives that were considered, are recorded as architecture decision records. The sections above link to them. Decisions lists all records, including those on the license and the documentation, and ADR-0001 explains which decisions are recorded and how.