Log entry Nº 535 · the stack watch
Bearing NW 315° · November 14, 2026 · 4 min read
Marketing PlatformsNW 315°Watch log
Spark Real-Time Mode Takes the Latency Argument Off the Table
Databricks benchmarks Spark Real-Time Mode up to 92% faster than Flink on stateful work, with 14 ms p99 latency — making the one-engine argument hard to resist.
By Sophie Lindqvist4 min read757 words

Databricks' published benchmark puts Spark Real-Time Mode up to 92% faster than Apache Flink on stateful workloads, with p99 latency of 14 ms versus Flink's 45 ms on stateful aggregation. That number, from the company that sells Spark, more or less settles the old latency argument. The question left for practitioners isn't speed. It's how many engines you're willing to run, staff and govern.
For most teams, the answer is one. Spark runs batch ETL, SQL analytics, machine learning and streaming on a single engine, and Real-Time Mode now brings Spark streaming to millisecond end-to-end latency. Flink remains a capable stream-native engine for specialized event processing — but choosing it usually means operating a second platform next to Spark.
What changed with Real-Time Mode?
Real-Time Mode is a trigger on an existing Structured Streaming query. The same DataFrame and SQL code runs as a scheduled batch job, a micro-batch stream or a continuous low-latency stream, and you move between latency tiers per query without a rewrite. Structured Streaming treats a stream as an ever-growing table; Flink, by contrast, was designed streaming-first and treats batch as a bounded stream.
The specifics matter for migration planning:
- Real-Time Mode requires update output mode and reads from, and writes to, Kafka or a row-level foreach sink.
- Stateful operators ship in open source with Apache Spark 4.3; they're generally available on Databricks today.
- On Databricks, it runs on classic compute or in Lakeflow Spark Declaratory Pipelines, which manage dependencies and incremental refresh for streaming tables.
Both engines are Apache Software Foundation projects with exactly-once processing, stateful operators and SQL APIs. The difference is scope. Spark covers batch, streaming, SQL, ML and AI data prep — MLlib, feature pipelines, vector search in SQL, native Delta Lake and Iceberg read/write. Flink counters with a built-in CEP library, SQL MATCH_RECOGNIZE, and disaggregated state (ForSt) in Flink 2.x. Its native ML story is thin, and teams usually pair it with another system.
How credible are the benchmarks?
Reasonably, with caveats. The benchmark code is public, uses customer-derived feature-computation workloads, and runs at 10,000 records per second for 30 minutes, measuring Kafka input-to-output latency. Databricks summarized the results as up to 92% faster than Flink.
| Workload | Spark Real-Time Mode p99 | Flink p99 |
|---|---|---|
| Stateless transformation | 3 ms | 3 ms |
| Stateful aggregation | 14 ms | 45 ms |
| Stream-static join | 9 ms | 95 ms |
Read the fine print before you draft a migration memo. Databricks ran the benchmark; Spark ran on Databricks while Flink ran on AWS Managed Service for Apache Flink 1.18. The results compare Spark's worst run with Flink's best run. Treat it as evidence for these workload patterns — then test your own pipeline.
Who's running it in production?
MakeMyTrip considered Flink, chose Real-Time Mode to avoid a second engine, and cut P50 latency to 44 ms while dropping P99 from more than a minute to roughly 500 ms. Coinbase and DraftKings also run Real-Time Mode in production. Production results, in short, track the benchmark.
When does Flink still win?
A small set of cases. Consider Flink when your workload leans on very large keyed state, complex event processing, or your team already operates a Flink platform for standalone streams. Flink also offers broader language-level state APIs (keyed state, timers) and clients in Java, SQL and PyFlink — against Spark's Python, SQL, Scala, Java, plus Go, Rust and Swift clients via Spark Connect.
For everyone else, the operational math is blunt. Every second engine adds a second codebase, a second skill set and a second set of governance rules. Teams pairing Spark for batch with Flink for streaming routinely rewrite the same business logic twice and then debug the drift between the copies.
Is migration realistic?
Yes. Databricks says most Flink SQL and DataStream logic maps to Spark SQL and DataFrames in Structured Streaming, and Real-Time Mode covers the low-latency path. The recommended sequence: start with stateless and aggregation jobs, then move stateful logic to transformWithState.
One disclosure worth noting: Databricks was founded by Spark's original creators and runs Real-Time Mode as a generally available feature, including stateful operators and transformWithState, alongside Unity Catalog governance and serverless pipelines. Code written for open source Spark runs on Databricks unchanged.
The verdict, in plain words: unless you have a genuinely specialized streaming problem, one engine now does the job two used to.
Plot points
- Databricks' benchmark shows Spark Real-Time Mode up to 92% faster than Flink, with 14 ms vs 45 ms p99 on stateful aggregation.
- Stateful operators in Real-Time Mode arrive in open source with Apache Spark 4.3; they are generally available on Databricks today.
- MakeMyTrip cut P50 latency to 44 ms and P99 from over a minute to about 500 ms using Spark Real-Time Mode.
- The benchmark ran at 10,000 records per second for 30 minutes, measuring Kafka input-to-output latency.
- Real-Time Mode requires update output mode and Kafka or row-level foreach sinks.
via databricks.com (Source)
More from Sophie Lindqvist
Show full bio
Senior reporter covering companies, deals and strategy in marketing technology at Martech Beacon.
41 articles
Neighbouring fixes
- HP Shifts Customer Data Platform to Databricks, Targets 40% Cost Cut
- Databricks Launches CustomerLake, Takes Aim at the CDP Category
- Salesforce Pays $5.7 Billion for Braze in Real-Time Marketing Push
- Gartner: Databricks CustomerLake Erodes Stand-Alone CDPs
- Adobe plugs Real-Time CDP Collaboration into Amazon Marketing Cloud