[Two Tracks]
Multi-Source Data Architecture
Every source behind the dashboards inventoried — what it is, where it lands, who owns it — and two tracks that never mix.
The company sells through several payment gateways, affiliates and call centers; its costs live in a finance ERP, spreadsheets and ad platforms. Dashboards read a data lake, a legacy MySQL and Postgres mirrors.
Mapped and measured every source the margin dashboard needed and proposed a track for each.
The same metric disagreed between screens because each dashboard reached its sources by its own path — and some sources were spreadsheets nobody owned.
01 · Two tracks
Sales and payment data go through the lake's canonical gold layer. Operational data (spreadsheets, task tool) go through a scheduled edge function with a read-only service account into staging and modeled Postgres tables.
02 · Three kinds of source
PIPELINE (ingest it), RULER (a reference table — version it) and CONFERENCE (a gateway statement — only reconcile against it, never ingest).
03 · One card per source
Grain, cadence, owner, landing place and failure mode — each fact with the date it was measured.
04 · Look before asking
Before requesting a new integration, search the warehouse: ad spend at ad × day grain and call-center sales were already landing daily.
05 · Timezones written down
The gateway closes its day in UTC−4, the lake uses UTC and the legacy DB UTC−3. Every comparison says which day it uses.
what you're seeing
Sources on the left, the two hubs in the middle (lake gold and Postgres), dashboards on the right. Each source has a kind, grain, cadence, owner and failure mode; the colored dot is its freshness.try this
- 1.Click a source to trace its path to the dashboards.
- 2.Filter by RULER and CONFERENCE — they're never ingested like a pipeline.
- 3.Click “manual source goes stale” and see which dashboards inherit the warning.
real vs simplified
The classification and the two tracks are the real design. Source names are generic and the wiring is simplified.! brokeNet sales in the new model came out far above legacy.
→ changedTraced in three steps to commission computed on the collected amount (tax included) instead of the price — fixed in the source definition, not in each consumer.
! brokeA 5–12% gap against the gateway looked like bad data.
→ changedIt was timezone, tax and fees defined differently. A reconciliation note now comes before any comparison.