Batch processing won the last decade. It was predictable, correct, and easy to reason about. But AI in 2026 does not wait for the next ETL run. Fraud detection, real-time personalization, predictive maintenance, and agentic systems all require...
AI Editorial Team
Collective Intelligence
Editorial packet
data engineering / Stream Processing / real-time / streaming / architecture
Batch processing won the last decade. It was predictable, correct, and easy to reason about. But AI in 2026 does not wait for the next ETL run. Fraud detection, real-time personalization, predictive maintenance, and agentic systems all require decisions made while the data is still warm—sometimes within milliseconds. Streaming is no longer a niche concern for hedge funds and ad-tech. It is strategic infrastructure.
The good news: the tooling has matured. What used to require bespoke distributed systems is now accessible through well-documented platforms with managed offerings, clear APIs, and strong community ecosystems. The bad news: streaming is still harder than batch. It demands new mental models for state, time, failure, and cost.
This article covers the architecture patterns, technologies, and operational practices that make real-time data pipelines actually work for machine learning in 2026.
At the heart of any streaming architecture is the event log: an immutable, ordered, replayable stream of events that acts as the single source of truth for real-time data.
Apache Kafka remains the de facto standard. With over 150,000 organizations running it in production, Kafka has evolved from a scalable messaging layer into a central nervous system for digital business. Kafka 3.8+ runs in KRaft mode (ZooKeeper-free), supports tiered storage for cost-efficient long-term retention, and provides exactly-once semantics through idempotent producers and transactional APIs.
Apache Pulsar offers multi-tenant messaging with geo-replication and tiered storage natively. It separates compute and storage more cleanly than Kafka, which can simplify operations at massive scale. However, ecosystem maturity and tooling density still lag behind Kafka, and some analysts project Pulsar may fade from the landscape as Kafka-compatible platforms consolidate market share.
Redpanda is a C++ reimplementation of the Kafka protocol, designed for simplicity and performance. It runs as a single binary without a ZooKeeper or KRaft dependency, starts in seconds, and delivers lower p99 latencies in many benchmarks. For teams that need Kafka compatibility without JVM operational complexity, Redpanda is increasingly the pragmatic choice.
Key design decisions for your event backbone:
min.insync.replicas=2 provides durability without excessive write latency.{domain}.{entity}.{version} (trading.orders.v2) prevent topic sprawl.Events sitting in a log are inert. Stream processors bring them to life.
Apache Flink is the current gold standard for stateful stream processing. Flink 2.0 unified batch and streaming APIs, meaning the same code handles historical backfills and live processing. It provides sub-second latency, event-time semantics, and exactly-once state checkpoints. Flink’s RocksDB state backend can manage terabytes of state, and incremental checkpointing keeps recovery times manageable.
For ML use cases, Flink excels at:
Spark Structured Streaming remains the top choice for teams already invested in the Spark ecosystem. It offers SQL, Python, Scala, and Java APIs; deep integration with Delta Lake; and mature monitoring dashboards. Spark is micro-batch under the hood, which adds latency (typically seconds, not milliseconds) but simplifies operations and integrates cleanly with batch workflows.
ksqlDB provides a SQL interface over Kafka Streams for simpler transformations, filtering, and materialized views. It lowers the barrier to entry for teams without dedicated stream-processing engineers. However, ksqlDB is best for lightweight use cases. Complex stateful logic, large-scale joins, and custom ML feature computation still belong in Flink or Spark.
Lambda Architecture maintains two parallel pipelines: a batch layer for correctness and historical completeness, and a speed layer for low-latency insights. The serving layer merges both. Lambda is robust but operationally expensive—you maintain two codebases, two teams, and two failure modes.
Kappa Architecture eliminates the batch layer entirely. All data is treated as a stream. Historical reprocessing is achieved by replaying archived events through the same stream processor. This simplifies operations, unifies business logic in one place, and reduces the surface area for bugs.
In 2026, Kappa is the preferred default for new systems. Batch is simply a special case of streaming: a bounded stream with a start and an end. Flink 2.0 and Spark Structured Streaming both support this unified model. Choose Lambda only when you have genuinely divergent latency and correctness requirements that cannot be served by a single engine, or when legacy batch infrastructure cannot be retired.
The gap between raw events and model-ready features is where most streaming ML projects stall. A feature store bridges this gap by providing:
Feast (open-source) ingests precomputed features from batch pipelines or streaming sources (Kafka via Spark Streaming) and serves them through a unified API. It decouples ML teams from underlying data infrastructure. Redis or DynamoDB typically back the online store, with gRPC serving for lowest latency.
Tecton (managed) goes further by internalizing feature pipelines. Data scientists define transformations in Python or SQL; Tecton materializes batch, streaming, and real-time features automatically. It also includes built-in monitoring for feature freshness, null rates, and data drift.
For real-time AI, feature freshness is the critical metric. Measure it end-to-end: change a source value, then time how long until the served feature reflects it. Sub-second is achievable with Flink feeding Redis; minutes is typical for batch-sync architectures.
Online inference means scoring a model while the user is still on the page, the transaction is still in flight, or the sensor is still emitting. Latency budgets are tight:
Architecture pattern:
Model serving and feature serving must be co-optimized. If p99 feature retrieval latency is 30ms and model inference is 20ms, you have 50ms before the user notices. Every millisecond counts.
Exactly-once processing means every event is processed precisely one time, even through failures, restarts, and rebalances. This is essential for financial transactions, inventory management, and ML feature accuracy.
Kafka provides exactly-once through idempotent producers and transactional writes. Flink provides exactly-once through distributed snapshots (checkpoints) that capture operator state consistently. In practice, exactly-once adds operational overhead: longer checkpoint intervals increase replay volume; shorter intervals increase I/O load. A 1–5 minute checkpoint interval is a common production default.
Backpressure occurs when downstream consumers cannot keep pace with upstream producers. Without handling, this leads to memory exhaustion, dropped messages, or cascading failures. Flink handles backpressure naturally by propagating pressure upstream—slow sinks slow the entire topology. Kafka consumers can be tuned with max.poll.records and dynamic fetch sizes. Monitor consumer lag relentlessly; it is the single most important streaming health metric.
You cannot debug a distributed stream by tailing a log file. Observability must be designed in from the start.
Critical metrics:
Tools: Prometheus + Grafana for infrastructure metrics; Confluent Control Center or Redpanda Console for Kafka-specific visibility; Flink’s Web UI for job-level diagnostics; and OpenTelemetry for distributed tracing across the inference pipeline.
Streaming can be expensive if left unmanaged. Key strategies in 2026:
Fraud Detection (< 100ms): A global payments platform ingests transaction events into Kafka, enriches them with Flink CEP for pattern matching, computes rolling risk features, and serves them to an XGBoost model via Redis-backed Feast. Suspicious transactions are declined before authorization completes.
Real-Time Personalization: An e-commerce platform captures clickstreams into Redpanda, computes session-level features in Flink (dwell time, category affinity, cart abandonment signals), and serves them to a two-tower neural network. Product rankings update within 200ms of a user action.
IoT Predictive Maintenance: Thousands of industrial sensors emit telemetry via MQTT into Kafka. Flink rolling statistics detect anomaly patterns, trigger alerts, and update predictive models. Maintenance crews receive notifications before equipment fails.
Real-time data for AI is no longer science fiction. The tools exist, the patterns are proven, and the failures are well-documented. The teams that succeed are the ones that treat streaming as a first-class discipline—designing for failure, measuring obsessively, and optimizing for both latency and correctness from day one.
data-engineeringFeature stores bridge data engineering and production ML by centralizing feature definitions, versioning, and low-latency serving. This guide compares 2026 architectures and platforms while addressing training-serving skew, governance, and adoption tradeoffs.
data-engineeringRetrieval-augmented generation (RAG) has become the default architecture for production AI in 2026. You cannot fit an enterprise knowledge base into even the largest context windows, so you embed your documents, store the vectors, and retrieve the...
foundation-modelsOn May 28, 2026, OpenAI unveiled GPT-5.5, a mid-cycle release that nonetheless signals a major inflection point in how large language models handle complex reasoning tasks. While the version number suggests an incremental update, the underlying...