databricks-cost-leak-hunter · data planes

Where the numbers come from

Two planes feed this skill, and only one of them is allowed to produce a dollar. No customer data, workspace, or dollar figure appears on this page — this is the shape of the pipeline, not a result.

Billing plane and control plane feeding the four leak detectors A left-to-right data flow. On the left, the billing plane: system.billing.usage carries billed DBUs and system.billing.list_prices carries USD per unit; both flow into the focal priced CTE, which joins them on sku_name and usage_unit inside the price-effective window. The priced CTE fans out into four leak detectors — idle clusters and jobs on All-Purpose compute, both marked confirmed; overprovisioned clusters, marked estimated; and the Photon premium, marked at-risk. Beneath the detectors sits the control plane: live cluster config and events from databricks-workspace-mcp plus the system.compute tables. Its connector is dashed and enters the detector band, not the priced CTE, because the control plane corroborates the cause of each flagged row and never produces a dollar. All four detectors converge on the deterministic ranker, which ranks by monthly dollar impact and keeps confirmed, estimated and at-risk sums separate. BILLING PLANE · AUTHORITATIVE FOR DOLLARS FOUR LEAK DETECTORS CONTROL PLANE · AUTHORITATIVE FOR CAUSE, NEVER DOLLARS BILLED DBUs USD PER UNIT PRICED ROWS LEAK + KIND CORROBORATE ×4 system.billing.usage — the billed-DBU ledger. Read through the Databricks CLI Statement Execution API; supplies usage_quantity, usage_unit, sku_name, billing_origin_product and the usage_metadata struct. SQL Billed usage system.billing.usage system.billing.list_prices — the USD price book. Supplies pricing.default per sku_name and usage_unit, each row valid only inside its price_start_time to price_end_time window. SQL Price book system.billing.list_prices priced CTE (focal) — usage joined to list_prices on sku_name AND usage_unit inside the price-effective window, restricted to currency_code USD. The single dollar primitive every detector query reuses. CTE Priced rows priced CTE usage × list_prices sku_name + usage_unit Leak 1, idle clusters — confirmed. All-Purpose spend on clusters whose latest config row has auto_termination_minutes = 0. CONFIRMED Idle clusters auto_termination_minutes = 0 Leak 2, scheduled jobs on All-Purpose compute — confirmed. A usage row carrying a job_id while billing_origin_product is ALL_PURPOSE, re-priced at the Jobs Compute rate. CONFIRMED Jobs on All-Purpose job_id + ALL_PURPOSE Leak 3, overprovisioned clusters — estimated, not billed. Mean CPU from system.compute.node_timeline below 25 percent, modelled as spend times one minus CPU percent. ESTIMATED Overprovisioned avg_cpu_pct < 25 Leak 4, Photon premium — at-risk, pending review against actual runtime gain. Photon is billing-visible through the SKU, not as a column on system.compute.clusters. AT-RISK Photon premium sku_name ILIKE '%PHOTON%' Deterministic ranker — rank-and-report.py. Sums each category by its kind, ranks descending by monthly dollar impact, and never blends confirmed dollars with estimated or at-risk ones. The model does not do the arithmetic. PY Deterministic ranker rank-and-report.py ranks by $/month sums kinds separately Control plane — live cluster config and events. The databricks-workspace-mcp server exposes five read tools (clusters_get, clusters_events, clusters_list, instance_pools_list, pipelines_get); system.compute.clusters and node_timeline supply config history and utilisation. Neither produces a dollar. REST Live config & events databricks-workspace-mcp system.compute.clusters LEGEND PRICED CTE (FOCAL) BILLING TABLE DETECTOR CONTROL EVIDENCE CORROBORATION · NO $

You are looking at where every figure in the cost-leak report is allowed to come from. The billing plane on the left — system.billing.usage joined to system.billing.list_prices on sku_name and usage_unit inside the price-effective window, in USD — resolves into the priced CTE, the one dollar primitive every detector query reuses.

The control plane at the bottom — live cluster config and events from the databricks-workspace-mcp server, plus the system.compute tables — never produces a dollar. Its connector is dashed and it enters the detector band, not the CTE, because its job is to explain why a flagged row leaks: auto-termination switched off, an autoscale floor set too high, Photon enabled on a job it does not accelerate.

That separation is the reason the tool can attribute spend to a cause at all. The number comes from what was billed; the cause comes from what is configured; the two only meet at the detector, and each detector carries a kind — confirmed, estimated, or at-risk — so the ranker sums them separately and a modelled figure is never added to a billed one.

← databricks-cost-leak-hunter

Sample output: the four leak categories, ranked, with the confidence split and every published rate cited.

/databricks-cost-leak-hunter/

Grounded in databricks-pack v2.27.0 · skills/databricks-cost-leak-hunter/SKILL.md