databricks-cost-leak-hunter — pipeline architecture
A Claude Code skill that audits a Databricks workspace for real-dollar cost leaks. It reads two planes — the billing plane (the CLI Statement Execution API over system.*, which produces the dollars) and the control plane (the databricks-workspace-mcp server, which corroborates live config into a one-change fix) — and runs a detect → compute → rank → report pipeline.
Pipeline topology
grounded in skills/databricks-cost-leak-hunter/{SKILL.md · scripts/ · references/}system.billing.usage
billed-DBU ledger
system.billing.list_prices
price book · windowed
grant-chain probe
Step 1 · fail fast
priced CTE
Step 2 · usage × list_prices
system.compute.clusters
auto_termination_minutes
system.compute.node_timeline
avg_cpu_pct
Leak 1 · idle clusters
auto_termination = 0
confirmed
Leak 2 · jobs on All-Purpose
re-priced at Jobs rate
confirmed
Leak 3 · overprovisioned
spend × (1 − CPU%)
estimated
Leak 4 · Photon premium
~2× premium portion
at-risk
rank-and-report.py
Step 7 · deterministic ranker
CFO report
split headline · ranked · confidence
databricks-workspace-mcp
control plane · 5 read tools
- clusters_get · clusters_events
- clusters_list · instance_pools_list
- pipelines_get
Component details
Billing plane — the number
- Dollars come from
system.billing.usagejoined tolist_prices— thepricedCTE. - Matched on
sku_nameANDusage_unitwithin the price-effective window, in USD — a re-priced SKU never double-counts. - Confirmed leak dollars are
SUM(usd)over real billed rows — never an estimate. - UC enforces a metastore-admin grant chain on every
system.*read.
The four leak detectors
- Idle clusters (confirmed) —
auto_termination_minutes = 0on All-Purpose spend. - Jobs on All-Purpose (confirmed) — re-price the same DBUs at the Jobs rate; the delta is the savings.
- Overprovisioned (estimated) —
spend × (1 − CPU%)fromnode_timeline. - Photon premium (at-risk) — the
~2×premium portion, pending a runtime-gain check.
Confidence is load-bearing
- Every leak carries a
kind: confirmed / estimated / at-risk. - The ranker sums confirmed and unconfirmed dollars separately — never one number under one verb.
- The CFO report's headline splits "burning ~$X (confirmed)" from "up to ~$Y pending review."
Control plane — the fix
databricks-workspace-mcpexposes 5 read-only control-plane tools.- Each flagged leak is corroborated against live config (
clusters_get,clusters_events, …). - The SQL produces the number; the MCP turns it into a verified single-config-change fix.
- Absent the MCP, the skill still produces dollars and accepts pasted config — it degrades, never fails silently.
Deterministic ranker
rank-and-report.pyingests one JSON object per leak category.- Converts 30-day → monthly, ranks descending by dollar impact, annualizes the headline + #1 line.
- Tolerates LLM-formatted currency (
"$1,200.50") and case-variant kinds so no row is silently dropped. - The LLM never eyeballs a number — the arithmetic is in Python.
Grant chain · fail fast
- Step 1 probes
SELECT 1 FROM system.billing.usagebefore any scan. - On
PERMISSION_DENIEDit reports the exactUSE CATALOG → USE SCHEMA → SELECTchain and stops. - The most common real-world failure, surfaced upfront instead of mid-flow.