Ozon Priority-Aware Data Pipeline
A priority-aware big-data pipeline designed to process and organise high-volume e-commerce data through a structured data engineering workflow.
- Type
- Big Data Engineering · E-commerce
- Role
- Data Engineer
- Contribution
- Pipeline architecture · Cloud infrastructure · Data processing
- Year
- 2026
The problem
An e-commerce platform produces a constant stream of events, and they do not all matter equally. When everything flows through one queue, a surge of routine events can delay the ones that need immediate action. The pipeline had to handle high volume without letting urgent data wait behind routine data, while keeping the flow organised enough to transform, process and analyse downstream.
What I built
A priority-aware streaming pipeline for e-commerce event data, built on Azure Event Hubs. Incoming events are classified by urgency and routed into one of four priority tiers: critical, high, standard and low.
Each tier runs as its own Event Hubs namespace, so capacity, partitions and consumers are set per tier and a flood of standard traffic cannot starve critical events. The critical stream is split across four partitions for parallel consumption, and the namespaces run with zone redundancy for resilience. Downstream, each stream is transformed and processed on its own path before being written out for storage and analysis, with namespace-level metrics used to confirm that requests, messages and throughput reach the intended tier.
Approach
Priority as architecture
Key decisions
Isolate tiers, don’t just label them
Each priority runs in its own Event Hubs namespace rather than sharing one queue with a priority field, so contention in one tier cannot delay another.
Size each tier to its load
Partitions and consumers are configured per tier. The critical stream gets four partitions for parallel processing without over-provisioning the rest.
Resilience at the platform level
Zone redundancy is enabled on the namespaces, so the pipeline tolerates the loss of a single availability zone.
Verify routing with live metrics
Namespace-level request, message and throughput metrics confirm that test traffic lands in the intended tier.
Priority tiers
- Critical
- ozon-eh-critical · 4 partitions
- High
- ozon-eh-high
- Standard
- ozon-eh-standard
- Low
- ozon-eh-low
Outcome
- Isolated priority tiers
- 4
- Standard-tier events in the recorded test run
- 2.4k
- Server errors across tiers
- 0
In the recorded test run, events reached their intended tiers, around 2,400 through the standard tier and 12 through the high tier, with no server errors on any namespace. The architecture enforces priority structurally, by isolating streams, rather than relying on ordering inside one shared queue.
- Tools
- Capabilities
Working on something similar?
Start a conversation