Bronze → Silver → Gold

Enterprise medallion pipeline turning raw event streams into production-grade business metrics. Click stages, explore data transformations, and trace lineage from Gold back to Bronze.

0

Services

0

S3 Buckets

0

Quality Gates

0

Bronze Records

Live Pipeline Simulator

Click any stage to see transformation details. Particles flow in real-time showing data movement. Record counters increment live.

Data Sample Explorer

Row	event_id	raw_payload	ts
1	evt_A7B2	{"usr":"3421",...	14:23:11.042
2	evt_B9C3	{"usr":"8172",...	14:23:11.089
3	evt_C3D8	{"usr":"null",...	14:23:11.124

Hover rows to highlight. Toggle layers to see data transformation from raw events to aggregated business metrics.

Streaming Console

pipeline-stream.log

Live

Quality Gate Inspector

Click each gate to expand validation checks and pass-rate gauges.

DAG Execution Timeline

Airflow-style task timeline. Hover tasks to highlight. Watch running tasks animate.

ingest_kafka0s – 3s

write_bronze3s – 5s

validate_bronze5s – 6s

clean_silver6s – 10s

validate_silver10s – 11s

aggregate_gold11s – 14s

validate_gold14s – 15s

notify_complete15s – 15.5s

Success

Running

Waiting

Service Topology

Click any service to see connections. Highlighted services are in the data path.

Data Lineage Tracer

Click Trace Lineage to see how Gold metrics flow back through Silver to Bronze sources.

Gold→daily_revenue

Column: total_revenue

Silver→transactions_cleaned

Column: amount

Bronze→raw_events

Column: payload.amt

Source→kafka_topic_payments

Column: raw_json

Stack

PySpark 3.5Delta LakeRedpandaMinIOAirflow 2.9dbtGreat ExpectationsPostgreSQL 16GrafanaOpenMetadataDatabricks