Skip to content

Orchestrate with Apache Airflow 3

The default executor is set in ADR-0015; the description of the CeleryExecutor below applies to the variant with separate workers.

Context and Problem Statement

Agoranomos has to load data from ad platforms on a schedule, build the warehouse models after each load and refresh the dashboards once the models are ready. The steps depend on one another, they have to be retried when they fail, and a failure has to be visible. The stack must also run anywhere from a laptop to a small server, on infrastructure that the user controls.

Which orchestrator runs these pipelines, how does it execute tasks, and which distribution of it do we build on?

Decision Drivers

  • Tasks should run apart from the scheduler, and it should be possible to add workers as the load grows.
  • The stack should stay vendor-neutral: it has to run on the official Apache image, without a vendor's CLI or runtime.
  • The orchestrator should have a maintained dbt integration and a maintained MCP server, so that neither has to be written from scratch.
  • The message broker should have a permissive open-source license.
  • The whole stack has to fit the memory budget set in ADR-0010.

Considered Options

  • Airflow with the CeleryExecutor and a Valkey broker, on the official apache/airflow image
  • Airflow with the CeleryExecutor and a Redis broker
  • Airflow with the LocalExecutor, without a broker
  • Astro CLI and Astro Runtime as the base
  • Dagster
  • Prefect

Decision Outcome

Chosen option: "Airflow with the CeleryExecutor and a Valkey broker, on the official apache/airflow image", because Airflow is governed by the Apache Software Foundation, runs on an official image without vendor-specific tooling, has a broad ecosystem of provider packages and works with Cosmos for dbt and with astro-airflow-mcp for AI access, and because Celery workers keep task execution apart from the scheduler and scale out horizontally.

In practice this means:

  • Airflow 3.3 runs as separate services: API server, scheduler, DAG processor, triggerer and workers. Agoranomos extends the official apache/airflow image with its own layer and pins the Python tag explicitly, for example 3.3.2-python3.13.
  • CeleryExecutor runs every task on a worker. Celery keeps task results in the Airflow metadata database (ADR-0003), so no separate result backend is needed.
  • Valkey 9 is the Celery broker. It is licensed under the BSD 3-Clause license and speaks the Redis protocol.
  • Authentication uses Airflow's FAB auth manager with the Admin, Op and Viewer roles.
  • Our own Compose files. Agoranomos builds on the image and not on Airflow's own docker-compose.yaml, which is documented as not meant for production and ships default credentials.
  • Astronomer's open-source projects are used where they fit: Cosmos, which runs dbt models as Airflow tasks (ADR-0005), and astro-airflow-mcp, the MCP server for Airflow (ADR-0007). The Astro CLI and Astro Runtime are not used.

The other options were not chosen for these reasons:

  • Redis as the broker. Redis 8.x is licensed under the RSAL, SSPL or AGPL, none of which is as permissive as the BSD license of Valkey, which is a drop-in replacement.
  • LocalExecutor. It is simpler and lighter, with no broker and no worker containers, but it runs tasks as subprocesses of the scheduler on the same machine, so it cannot scale out to separate workers.
  • Astro CLI and Astro Runtime. They tie the stack to one vendor's tooling and bring a Compose setup of their own, while the stack has to run on the official Apache image.
  • Dagster and Prefect were considered but not evaluated in depth. Airflow was the starting premise of the project: its maintainers have operated it in production, and it is widely known among the data engineers the project is aimed at. Both Dagster and Prefect have dbt integrations and MCP servers, so this choice is not a verdict on them.

Consequences

  • Good, because tasks run on workers that are isolated from the scheduler, and more workers can be added as the load grows.
  • Good, because the stack depends on no vendor's CLI or runtime and stays portable.
  • Good, because the dbt integration and the AI access to Airflow come from maintained projects.
  • Good, because the broker has a permissive license, unlike recent Redis releases.
  • Bad, because the CeleryExecutor needs more containers and more memory than the LocalExecutor: a broker and separate workers come on top of Airflow's own services. This puts pressure on the memory budget, and a lighter lite profile based on the LocalExecutor is likely to be added if measurements call for it.
  • Bad, because Airflow 3 is itself made of several services, so there are many containers to pin, upgrade and monitor.
  • Bad, because Dagster and Prefect were not evaluated in depth, so the choice of Airflow rests on the starting premise of the project and not on a side-by-side comparison.

More Information

  • Implementation status, as of October 2026: in place for v0.1.0: Airflow 3.3.2 runs as separate services, with the FAB auth manager, the LocalExecutor by default and Valkey as the broker of the CeleryExecutor variant (ADR-0015). The layer of its own is in place too: airflow/Dockerfile adds dlt and Cosmos to the official image and dbt in a virtual environment of its own, and ./agoranomos up builds it. astro-airflow-mcp comes with the AI layer in v0.3.0 (ADR-0007). From v0.1.5 the results of Celery are not in the metadata database, as the decision above expected, but in a database of their own on the platform instance of ADR-0003, so that the worker holds no password of the metadata database (ADR-0015); they still need no service of their own.
  • Versions as of October 2026: Airflow 3.3 (3.3.2) and Valkey 9 (the latest release is 9.1.2, and the 9.0 line still gets patch releases, the latest being 9.0.6). Airflow 3.3 supports Python 3.10 to 3.14 and PostgreSQL 14 to 18, which covers the choices in ADR-0003 and ADR-0012.
  • The memory use of the stack was measured in October 2026, with the LocalExecutor by default (ADR-0015): it fits the budget, so no lite profile is planned (System requirements).
  • The Celery executor documentation describes the executor and its broker settings. Valkey is the broker project.