The Clock Is Not a Dependency: Rethinking Orchestration for the Agentic Data Era!

Table of contents

The market just admitted the problem

On July 13, Prefect announced it is acquiring Dagster Labs, bringing together the two most widely adopted successors to Apache Airflow into a combined suite serving data pipelines, ML operations, and AI agent infrastructure. The framing of the deal is telling: Prefect positions the combination as runtime execution plus governed agent access (via FastMCP) plus Dagster’s declarative outcomes — letting teams automate agentic workflows the way they’ve always automated pipelines, with Dagster’s materialization tracking and freshness policies catching data problems at the point of computation while Prefect’s event-driven execution reacts as state changes.

Read that as a confession. The two leading Airflow challengers concluded that neither half — imperative event-driven execution nor declarative asset semantics — is sufficient alone for the AI era. They’re right. But a merger of two control planes is not the same as the architecture the moment demands. To see why, look closely at what the incumbents’ own documentation says they can and cannot do.

Airflow: real progress, and a revealing warning label

Airflow 3 deserves credit for genuine movement. The scheduling docs now organize around assets, not just cron: asset definitions, asset-aware scheduling with conditional expressions, and even asset partitions now appear alongside the traditional timetables. Event-driven scheduling landed in 3.0: DAGs can be triggered by external events rather than predefined time-based schedules, with an AssetWatcher monitoring an external event source such as a message queue and triggering an asset update when a relevant event occurs.

But the same documentation contains the most revealing paragraph in the entire project. Only triggers inheriting from BaseEventTrigger are allowed for event-driven scheduling, because triggers that wait for an external resource to reach a given state — a file existing, a job succeeding, a row being present — can cause infinite rescheduling: once the condition becomes true it stays true, so the DAG fires repeatedly on every check. The docs explicitly caution against conditions that remain permanently true once met.

Think about what that warning means architecturally. The one thing a data-dependency-driven orchestrator must express — a predicate over data state — is precisely the thing Airflow’s trigger model documents as dangerous. The system can react to edges (a message arrived) but cannot natively reason about levels (the partition is complete, the watermark passed T, quality checks hold) because it has no state ledger recording which condition-satisfactions have already been consumed by which downstream runs. Airflow’s asset remains fundamentally a URI that gets “touched”; the vocabulary is presence, not content. And its placement vocabulary — pools, queues, priority weights — still knows nothing about GPUs, inference quota, or data locality.

Dagster: the right nouns, a centralized poll underneath

Dagster’s concept model is the most data-native among mainstream orchestrators, and the docs are explicit about the inversion: an asset represents a logical unit of data such as a table, dataset, or ML model, with dependencies on other assets forming the lineage — it’s the core abstraction. Around it sit the pieces a data-oriented orchestrator needs: asset checks ensuring expectations around quality, freshness, or completeness; partitions representing logical slices that enable incremental processing on relevant subsets; and automation conditions describing when an asset should execute based on the status of the asset and its upstream dependencies, composable to express complex scheduling logic without custom sensor code.

This is genuinely dependency-driven orchestration in the declarative sense. But the docs also name the mechanism: automation conditions are evaluated by an automation condition sensor — a control-plane loop ticking against a metadata store. That’s a smarter, centralized poll: a real improvement over per-task sensors, but still a scan whose cost grows with asset count, which matters when agents inflate asset counts by orders of magnitude. The asset graph is also fundamentally curated — definitions deployed in code locations — which fits long-lived governed pipelines and fits awkwardly with ephemeral, agent-spawned micro-workflows. And placement is fully delegated to executors and run launchers; readiness and compute feasibility are never one decision.

The Prefect deal doesn’t change this analysis — it packages the declarative layer with a more reactive runtime, which is directionally correct, and the combined roadmap explicitly targets AI-native workflows and earlier detection of data issues via materialization and freshness policies. What it does not add is a data-content dependency plane or resource-aware admission.

Durable execution: the missing kernel, minus the data

The durable-execution camp solves a different slice almost perfectly. Restate’s architecture is the cleanest illustration of what an agent-scale control plane should look like: a partitioned, log-centric runtime where the replicated log is the primary durability layer — an operation “happens” when the partition leader appends its record and receives quorum acks, and that commit point defines the durable order of invocations, steps, state updates, messages, timers, and completions. Waiting costs nothing: durable promises and timers follow the same log-first pattern — each action is appended to the log, and upon reading the committed record the processor applies it, registering timers or resolving promises. Dynamic cross-workflow coordination is native: when a handler targets a different key, the event is recorded in the origin partition’s log and delivered exactly once to the destination partition. That is the “dynamic dependency handshake” primitive — a consumer awaits a promise, a producer resolves it, delivery is pushed and journaled, no polling anywhere.

Temporal argues the same model from the workflow side: replacing DAG-first orchestration with code-first durable workflows, integrating application and data teams through signals and Nexus for cross-boundary calls, and coordinating long-running, human-in-the-loop, and agentic AI workflows — while letting data remain where it lives and managing scale without moving large datasets. And the ecosystem has noticed the agentic fit: durable execution is becoming a core reliability layer for production AI agents, converging on persisting completed execution boundaries and recovering without repeating tool calls or external mutations.

What none of these engines have is a data vocabulary. No assets, partitions, watermarks, completeness semantics, quality state, lineage, or cost model. They are superb workflow kernels. Handing an agent a durable-execution engine and calling it data orchestration is handing someone a filesystem and calling it a database.

The five gaps, restated against the evidence

Time is a proxy that no longer proxies. Asset-update triggers (Airflow 3, Dagster) replace the clock with a touch-event. But the real dependency is a predicate over data content — completeness, watermark, distribution stability, freshness relative to consumption — and Airflow’s own docs show its trigger model structurally cannot host state predicates safely, while Dagster’s conditions evaluate presence and status, not content signals from the data itself.

Agent-generated volume breaks control-plane cardinality assumptions. DAG-file parsing and per-tick metadata scans are O(assets × tick-rate). At millions of ephemeral agent-spawned micro-workflows, the control plane must look like Restate’s — sharded orchestration and state by key, hot paths partition-local, cross-partition work explicit — where admission is a log append and waiting is an index entry.

Readiness and placement must be one decision. A task can be data-ready but GPU-infeasible, or resource-cheap right now in a way that should pull optional work forward. Every current orchestrator decides “whether” and delegates “where.” Nobody jointly optimizes over data readiness, GPU/inference headroom, token budgets, warehouse credits, and locality.

Dynamic graphs need dynamic handshakes. Durable promises exist and work — in engines with no data semantics. Asset dependencies exist and work — in engines with no runtime rendezvous primitive. Nobody has attached the promise to the data state.

Delegated poll sensors are a decentralized tax. This critique has been on the record since 2021: hundreds of flows poking the metastore for the same popular table’s partitions overwhelms the underlying systems, and the fix is a centralized dependency service where the vast majority of metadata inspections hit a state store instead of being amplified downward. Deferrable operators centralized the waiting; they never centralized the knowing.

The strongest counterargument: won’t Airflow just absorb this?

Before proposing new architecture, the incumbency case deserves a fair hearing, because it’s the argument every pragmatic platform lead will make: Airflow is already moving in the right direction, and installed base beats architecture almost every time.

The evidence for the steelman is real. Airflow 3 was not a cosmetic release — the project demonstrated it can ship paradigm-scale change. The scheduling documentation now treats asset definitions, asset-aware scheduling with conditional expressions, and asset partitions as first-class citizens alongside cron and timetables, and event-driven scheduling landed in 3.0, letting DAGs react to external events via AssetWatchers attached to message queues rather than predefined time-based schedules. The trajectory from 2.4’s Datasets to 3.2’s asset partitions shows a community steadily rebuilding its scheduling vocabulary around data. Add the surrounding advantages: the largest provider ecosystem in the category, a massive trained workforce, commercial backing, and the boring-but-decisive fact that Airflow is already approved, deployed, and paid for inside thousands of enterprises. History offers plenty of precedents for incumbents absorbing the challenger’s idea: Postgres absorbed JSON and later vector search and stayed the default database; Kafka absorbed stream processing adjacencies; Kubernetes absorbed operators, batch, and GPUs. On this reading, “data-state-driven orchestration” becomes Airflow 3.5 or 4.0 — another trigger source, another scheduling expression — and the challengers get commoditized before they reach escape velocity.

If that’s right, the correct strategy is to wait. So it matters a great deal whether the assumption holds. It has three flaws, one of which is close to fatal.

Flaw one: the paradigm can’t be absorbed at the edge — it replaces the kernel. The absorption precedents share a property worth naming: the incumbent could adopt the new idea as a peripheral feature without touching its core. JSON in Postgres was a new column type; the storage engine, planner, and transaction model stayed intact. Data-state-driven orchestration is not like that for Airflow. Airflow’s kernel is a relational metadata database plus a scheduler loop whose unit of account is the DAG run; assets, watchers, and event triggers are adapters that convert external signals into DAG runs. A dependency plane built on content predicates requires a different kernel: a ledger of data-state transitions with consumption tracking, push-based promise resolution, and admission that scales as log appends rather than periodic scans of a database. You can see the kernel’s limits surface in Airflow’s own documentation: only triggers inheriting from BaseEventTrigger may drive scheduling, because conditions that wait for a resource to reach a state — a file existing, a job succeeding — remain true once met and cause the DAG to be triggered indefinitely on every check. That warning is not a missing feature; it is the model telling you it has no memory of which condition-satisfactions have been consumed. Fixing it properly means a stateful dependency ledger at the heart of the scheduler — at which point you have rewritten Airflow’s core inside Airflow, with all of the migration burden and none of the clean-slate advantages.

Flaw two: the installed base is an anchor precisely where it looks like an asset. Airflow’s compatibility surface — DAG-file semantics, execution dates, operators, XComs, thousands of provider packages, and untold millions of lines of user DAG code — is the moat, and the moat dictates that every increment must preserve DAG-run semantics. This is the innovator’s dilemma in open-source form: the community cannot make data state the primitive, because that would demote the DAG run to a derived concept and break the contract with every existing user. So each release rationally does what Airflow 3 did — graft the new idea onto the old unit of account. Meanwhile the governance that makes Airflow trustworthy (Apache process, consensus among a huge contributor base, long deprecation windows) sets the pace of change. The category’s own history shows how this ends when the required change is kernel-deep: Hive added ACID and LLAP but the future went to engines built query-first; Jenkins added pipelines-as-code but CI moved to systems born that way; and an incumbent with strong adoption can slide into maintenance relevance in just a few years once the architectural center of gravity moves. Notably, the ecosystem itself has just voted on this question: the two most widely adopted successors to Airflow chose to merge with each other rather than bet that the future would be won inside the Airflow codebase.

Flaw three: the marginal “user” of the next decade has no switching costs. Installed-base advantage is a human-organizational phenomenon — retraining, migration risk, procurement. The workload growth this post is about comes from agents, and agents don’t have those frictions. An agent generating a thousand micro-workflows a day selects an execution substrate by API capability, latency, and cost, per task, with zero loyalty. Airflow’s ten-year accumulation of human familiarity buys it very little with a caller that reads an OpenAPI spec in milliseconds. The battle for the agent-generated tier starts much closer to even than the installed-base argument assumes — and that tier is where the volume explosion is.

The honest synthesis. None of this predicts Airflow’s decline — quite the opposite. The most probable outcome is stratification: Airflow (and the merged Prefect–Dagster estate) keeps the human-curated, governed tier, where its asset events and messaging integrations make it an excellent citizen of a data-state-driven architecture — a first-class producer and consumer of state transitions. What the incumbency argument gets wrong is the conclusion that Airflow will therefore become the dependency plane. The kernel mismatch, the compatibility anchor, and the agent-tier dynamics all point the same way: the data-state ledger, the promise broker, and the joint scheduler will emerge as a layer that Airflow plugs into, not a feature Airflow ships. Betting on absorption means waiting for an Apache project to replace its own heart while preserving full backward compatibility — a bet with very few historical winners.

What to build: a data-state-driven orchestration plane

Since no challenger closes all five gaps, here is the system that does — five layers, three of them assemblable from existing parts, which is the cost-effectiveness argument.

The data state plane is a canonical catalog of assets and partitions enriched with content signals, not just existence: watermarks, completeness percentages against source, row counts, sketch-backed statistics (HyperLogLog cardinality, KLL quantiles), check results, and schema fingerprints, so that any data-state predicate is evaluable in milliseconds from metadata alone. Table-format commit logs already carry much of this; a sketch sidecar fills the rest. Crucially, this plane owns state transition semantics — completeness rollups across time hierarchies (5-minute windows composing to hours composing to days, so daily flows trigger without a daily job existing), and quality transitions (valid → invalid → backfilled) that propagate to dependents with policy, not just events.

The event backbone publishes every commit, check result, and signal update once, on a partitioned log, regardless of how many consumers care — converting O(waiters × poll-rate) into O(true changes).

The dependency broker is the genuinely novel component: it accepts declarative predicates (“runnable when orders/2026-07-16 completeness ≥ 99.995% AND row count within 2σ of trend AND feature freshness ≤ 4h”), compiles them into subscriptions, and resolves durable promises when predicates flip true — including runtime-registered await points addressable by artifact identity, so agent-spawned workflows rendezvous without pre-declared edges. Because the broker owns consumption records (which satisfactions triggered which runs), level-based predicates are safe — dissolving exactly the infinite-rescheduling hazard Airflow documents.

The durable execution kernel — Restate or Temporal, essentially off the shelf — owns journaled steps, exactly-once side effects, suspension around awaits, and recovery.

The joint scheduler consumes readiness from the broker and feasibility from the resource layer (GPU pools, inference-endpoint headroom, warehouse budgets, locality maps) and performs admission and placement as one decision, with budget governors and priority preemption for agent-originated work. Agents submit intents with contracts — desired artifact, freshness requirement, budget ceiling, deadline — not DAGs.

What to do on Monday

Keep the curated estate on Dagster or Airflow 3 — the asset model is right for stable, governed pipelines, and the Prefect–Dagster combination will likely strengthen that tier. Put the dynamic, agent-generated tier on a durable execution engine now, where promises and signals replace sensors immediately. Stand up the signal sidecar on your table formats so content predicates become cheap. Then build the one missing piece — the dependency broker binding catalog predicates to durable promises — and grow the joint scheduler behind it.

Airflow defined the batch era by making the clock programmable. The consolidation happening this very week shows the incumbents know that’s over. The next baseline belongs to whoever makes data state programmable — and wires it, push-first and cost-aware, into where and when compute actually runs.

References

Want to look beyond Airflow with smooth infrastructure migration? Learn about Stateful Data or join our pilot program.