Structured Streaming Archives

June 12, 2026

Featured image: Handling late-arriving events, four numbered steps: measure lateness, size the watermark, pick the output mode, and correct what is dropped with a MERGE

Handling Late-Arriving Events in Real-Time Data Pipelines

Why events arrive late, how Spark watermarks and Stream Analytics tolerances decide what counts, and six strategies to measure lateness, size watermarks and correct late data instead of losing it.

June 10, 2026

Featured image: Spark Structured Streaming in practice, a PySpark code card that applies a 10-minute watermark and a five-minute window, then writes to Delta with append output mode, a checkpoint and an availableNow trigger

Apache Spark Structured Streaming: A Practical Guide

A hands-on guide to Spark Structured Streaming: the micro-batch model, sources and sinks, output modes, triggers, watermarks, foreachBatch upserts and monitoring, with output from a local run.

June 8, 2026

Featured image: Streaming from Event Hubs into Databricks, a left-to-right pipeline of Azure Event Hubs, the Kafka connector, Azure Databricks and bronze and silver Delta tables

Building Streaming Data Pipelines with Azure Event Hubs and Databricks

Step-by-step: stream Event Hubs events into bronze and silver Delta tables on Azure Databricks with the Kafka connector, a Unity Catalog service credential, watermarks and deduplication.

June 2, 2026

Featured image: Real-time data architectures on Azure, with Azure Event Hubs at the centre of an orbit of Databricks, Cosmos DB, Eventhouse, Stream Analytics, IoT Hub and Event Grid icons

Designing Real-Time Data Processing Architectures on Azure

How to design a real-time data platform on Azure: latency tiers, Event Hubs ingestion, choosing Stream Analytics, Databricks, Fabric or Functions, and the decisions on event time, partitions and replay.