Log entry Nº 535 · the stack watch

Bearing NW 315° · November 14, 2026 · 4 min read

Marketing PlatformsNW 315°Watch log

Spark Real-Time Mode Takes the Latency Argument Off the Table

Databricks benchmarks Spark Real-Time Mode up to 92% faster than Flink on stateful work, with 14 ms p99 latency — making the one-engine argument hard to resist.

By Sophie Lindqvist4 min read757 words

Apache Spark vs Apache Flink: which should you use?
Fix Nº 535Apache Spark vs Apache Flink: which should you use? — Lucent_Designs_dinoson20 / pixabay

Databricks' published benchmark puts Spark Real-Time Mode up to 92% faster than Apache Flink on stateful workloads, with p99 latency of 14 ms versus Flink's 45 ms on stateful aggregation. That number, from the company that sells Spark, more or less settles the old latency argument. The question left for practitioners isn't speed. It's how many engines you're willing to run, staff and govern.

For most teams, the answer is one. Spark runs batch ETL, SQL analytics, machine learning and streaming on a single engine, and Real-Time Mode now brings Spark streaming to millisecond end-to-end latency. Flink remains a capable stream-native engine for specialized event processing — but choosing it usually means operating a second platform next to Spark.

What changed with Real-Time Mode?

Real-Time Mode is a trigger on an existing Structured Streaming query. The same DataFrame and SQL code runs as a scheduled batch job, a micro-batch stream or a continuous low-latency stream, and you move between latency tiers per query without a rewrite. Structured Streaming treats a stream as an ever-growing table; Flink, by contrast, was designed streaming-first and treats batch as a bounded stream.

The specifics matter for migration planning:

  • Real-Time Mode requires update output mode and reads from, and writes to, Kafka or a row-level foreach sink.
  • Stateful operators ship in open source with Apache Spark 4.3; they're generally available on Databricks today.
  • On Databricks, it runs on classic compute or in Lakeflow Spark Declaratory Pipelines, which manage dependencies and incremental refresh for streaming tables.

Both engines are Apache Software Foundation projects with exactly-once processing, stateful operators and SQL APIs. The difference is scope. Spark covers batch, streaming, SQL, ML and AI data prep — MLlib, feature pipelines, vector search in SQL, native Delta Lake and Iceberg read/write. Flink counters with a built-in CEP library, SQL MATCH_RECOGNIZE, and disaggregated state (ForSt) in Flink 2.x. Its native ML story is thin, and teams usually pair it with another system.

How credible are the benchmarks?

Reasonably, with caveats. The benchmark code is public, uses customer-derived feature-computation workloads, and runs at 10,000 records per second for 30 minutes, measuring Kafka input-to-output latency. Databricks summarized the results as up to 92% faster than Flink.

Workload Spark Real-Time Mode p99 Flink p99
Stateless transformation 3 ms 3 ms
Stateful aggregation 14 ms 45 ms
Stream-static join 9 ms 95 ms

Read the fine print before you draft a migration memo. Databricks ran the benchmark; Spark ran on Databricks while Flink ran on AWS Managed Service for Apache Flink 1.18. The results compare Spark's worst run with Flink's best run. Treat it as evidence for these workload patterns — then test your own pipeline.

Who's running it in production?

MakeMyTrip considered Flink, chose Real-Time Mode to avoid a second engine, and cut P50 latency to 44 ms while dropping P99 from more than a minute to roughly 500 ms. Coinbase and DraftKings also run Real-Time Mode in production. Production results, in short, track the benchmark.

When does Flink still win?

A small set of cases. Consider Flink when your workload leans on very large keyed state, complex event processing, or your team already operates a Flink platform for standalone streams. Flink also offers broader language-level state APIs (keyed state, timers) and clients in Java, SQL and PyFlink — against Spark's Python, SQL, Scala, Java, plus Go, Rust and Swift clients via Spark Connect.

For everyone else, the operational math is blunt. Every second engine adds a second codebase, a second skill set and a second set of governance rules. Teams pairing Spark for batch with Flink for streaming routinely rewrite the same business logic twice and then debug the drift between the copies.

Is migration realistic?

Yes. Databricks says most Flink SQL and DataStream logic maps to Spark SQL and DataFrames in Structured Streaming, and Real-Time Mode covers the low-latency path. The recommended sequence: start with stateless and aggregation jobs, then move stateful logic to transformWithState.

One disclosure worth noting: Databricks was founded by Spark's original creators and runs Real-Time Mode as a generally available feature, including stateful operators and transformWithState, alongside Unity Catalog governance and serverless pipelines. Code written for open source Spark runs on Databricks unchanged.

The verdict, in plain words: unless you have a genuinely specialized streaming problem, one engine now does the job two used to.

Plot points

  1. Databricks' benchmark shows Spark Real-Time Mode up to 92% faster than Flink, with 14 ms vs 45 ms p99 on stateful aggregation.
  2. Stateful operators in Real-Time Mode arrive in open source with Apache Spark 4.3; they are generally available on Databricks today.
  3. MakeMyTrip cut P50 latency to 44 ms and P99 from over a minute to about 500 ms using Spark Real-Time Mode.
  4. The benchmark ran at 10,000 records per second for 30 minutes, measuring Kafka input-to-output latency.
  5. Real-Time Mode requires update output mode and Kafka or row-level foreach sinks.

via databricks.com (Source)

Filed under

  • apache-spark
  • apache-flink
  • databricks
  • real-time-mode
  • stream-processing
Share this article:

More from Sophie Lindqvist

Sophie Lindqvist

Show full bio

Senior reporter covering companies, deals and strategy in marketing technology at Martech Beacon.

41 articles

Neighbouring fixes

‹ Previous articleNext article ›