Replenish batch data pipeline for retail procurement

Nightly batch pipeline turning orders from 15 stores and stock from 5 warehouses into supplier orders: HDFS data lake with 3-way replication, Trino federated SQL over HDFS + PostgreSQL, Airflow orchestration.

Replenish is a batch-only procurement pipeline: every night between 22:00 and midnight it reads the day's orders from 15 points of sale and stock levels from 5 warehouses, computes what each supplier must deliver, and writes the orders out. The constraint was deliberate: no streaming, a clear split between the OLTP database (PostgreSQL) and the analytical layer (HDFS).

Raw files land in an HDFS cluster with 4 datanodes and 3-way replication. Trino runs federated SQL across HDFS and PostgreSQL so the business logic is plain SQL over both, and Airflow schedules and monitors the whole run. Everything is packaged with Docker Compose and documented for a second engineer to run.

It is the classic data-engineering exercise (lake, warehouse, orchestration, separation of concerns) done end to end and reproducibly. Source code is public on GitHub.