Databricks pack · 5 skills

The Databricks bill went up and nobody can name which cluster took the money.

Five Claude Code skills that read your own workspace — system.billing.usage, cluster events, streaming state — and name the specific cluster, job, or table behind each finding. Every dollar figure is labelled Confirmed, Estimated, or At-risk, so nothing modeled ever gets read as cash.

You know the shape of it. The bill moved and the answer is “more usage.” A cluster took 25 minutes to start and the platform reports one aggregate number. A stream died overnight after a routine VACUUM, and the error names a missing Parquet file instead of the maintenance job that deleted it. In all three cases the platform told you the outcome and kept the attribution to itself.

Get the Databricks pack 5 skills · v2.27.0 · MIT · backed by the databricks-workspace-mcp server See what the cost report looks like Sample output from cost-leak-hunter, with every published rate cited

The three labels, and what each one means

Confirmed
Money actually billed, computed from your own system.billing.usage.
Estimated
A model of spend against idle CPU. Never presented as billed waste.
At-risk
Pending review. Never counted as recoverable cash.

The five skills

5

Flagship

databricks-cost-leak-hunter

Cost Leak Hunter

Audits your own system.billing.usage table and ranks each leak in dollars per month, keeping what was actually billed separate from what is only modeled.

What it detects

  • Interactive clusters with auto_termination_minutes = 0 — billing the full All-Purpose rate around the clock for compute nobody is attached to. Ranked by 30-day billed spend, corroborated against the live config and the idle gap between RUNNING and TERMINATING in cluster events. Labelled Confirmed: money actually billed.
  • Scheduled jobs still running on All-Purpose compute (~$0.55/DBU) instead of Jobs Compute (~$0.15/DBU) — the signature leak where a notebook prototype got shipped to prod on the same interactive cluster. Savings are a deterministic re-pricing of the same DBUs at the current Jobs rate, not a guess.
  • Photon billing its ~2× premium without delivering the speedup. UDF, Pandas-UDF and RDD workloads silently fall back to JVM Spark while the cluster keeps billing the Photon rate. Found via sku_name ILIKE '%PHOTON%' because system.compute.clusters has no runtime_engine column at all. Labelled At-risk: pending review, never counted as recoverable cash.
  • Clusters averaging under 25% CPU across system.compute.node_timeline, plus the instance-pool variant where min_idle_instances keeps warm VMs that the Databricks dashboard reports as $0 while the cloud provider bills you directly. Labelled Estimated — a model of spend × idle-CPU%, never presented as billed waste.
  • The metastore-admin grant gap on system.billing.usage, where a full workspace admin still gets PERMISSION_DENIED because system-schema access is granted separately at the metastore level. Probed before any analysis, so you get the exact GRANT USE CATALOG / USE SCHEMA / SELECT statements instead of a crash halfway through.

Who feels thisThe data-platform lead or FinOps owner who has to sit in the month-end close and explain, line by line, why the Databricks number moved — without per-cluster, per-job dollar attribution to point at.

See the sample cost report

databricks-streaming-guardian

Streaming Guardian

Blocks the destructive op before it runs — and when one already ran, tells you which recovery path costs you duplicates versus data.

What it detects

  • CREATE OR REPLACE minting a brand-new table UUID under a byte-identical name, so every active consumer dies on its next batch with DIFFERENT_DELTA_TABLE_READ_BY_STREAMING_SOURCE while the producer team is certain they changed nothing. A PreToolUse hook blocks the DDL when it confirms live consumers via system.streaming.query_progress — and fails open, so it never false-blocks.
  • The nightly OPTIMIZE racing an auto-compaction nobody knew was on. Databricks silently enables delta.autoOptimize.autoCompact on any table touched by MERGE/UPDATE/DELETE, so two compactors tombstone the same files and the job dies intermittently with ConcurrentDeleteDeleteException. A shipped probe returns SAFE or COLLISION RISK before you run it.
  • VACUUM reaping the exact Parquet files a lagging checkpoint still pins by path — DELTA_FILE_NOT_FOUND_DETAILED, which ignoreMissingFiles=true does not reliably cover. Only bites streams that fell behind the OPTIMIZE-then-VACUUM cycle, which is why it lands after a restart or a traffic spike.
  • The fan-out MERGE that was fine on a partitioned table and started throwing ConcurrentAppendException the week you moved to Liquid Clustering. LC drops the partition folders, so the ON predicate no longer proves writers touch disjoint file sets — a subagent reads clusteringColumns off DESCRIBE DETAIL and narrows it.
  • A driver or executor OOM while every heap metric reads flat and green, because the memory is off-heap RocksDB state store — so raising spark.executor.memory and tuning GC does nothing. Diagnosed off stateOperators.numRowsTotal / memoryUsedBytes climbing, not heap dashboards.

Who feels thisThe streaming on-call engineer paged at 2am when an overnight Delta stream dies after somebody else’s routine maintenance job ran — the person who has to decide before the morning SLA whether the recovery reprocesses, duplicates, or permanently loses rows.

databricks-uc-migration-pilot

UC Migration Pilot

Before the Hive Metastore goes read-only, it tells you table by table which ones will actually move to Unity Catalog, and names the specific reason each of the rest will not.

What it detects

  • Managed tables sitting on the DBFS root (dbfs:/user/hive/warehouse/…). SYNC rejects them because they are managed, not external, so each one needs a physical DEEP CLONE or CTAS rather than a metadata registration. The most common blocker in a mature workspace: every CREATE TABLE written without a LOCATION landed there.
  • Azure legacy-scheme tables on wasbs:// or adl://. A UC external location cannot be created over the Blob or ADLS Gen1 drivers at all, only abfss://. adl:// is doubly stuck: Microsoft retired Gen1 on 2024-02-29, so the data has to be copied to a Gen2 account before anything can be registered.
  • DENY-based Table ACLs with no Unity Catalog equivalent. UC is allow-only, so a grant set that used DENY to carve an exception out of a broad GRANT cannot be auto-mapped and has to be re-authored as group restructuring. Flagged alongside clusters still on LEGACY_TABLE_ACL / LEGACY_SINGLE_USER / LEGACY_PASSTHROUGH access modes, which cannot talk to UC at all.
  • The shallow-clone trap: CREATE TABLE … SHALLOW CLONE looks like a migration but copies no data. Time travel to older versions fails, and dropping or VACUUMing the HMS source destroys the clone’s data exactly when you think cutover is done. DEEP CLONE is the required verb when the source will be retired.
  • Post-migration PERMISSION_DENIED where the grant, the group and the user all exist. The Entra SCIM connector provisions direct members only and never flattens nested groups, so a user who is ‘in analysts’ in Entra is not in analysts in the Databricks account group graph — and hand-patching it in Databricks fails again because the SCIM-created group is IdP-managed.

Who feels thisThe data-platform lead who owns the HMS-to-Unity-Catalog cutover date and has to say how many weeks it will take — and the on-call engineer who takes the ‘I can’t read this table anymore’ ticket the morning after a migration batch, where the grant looks correct and access still fails.

databricks-cluster-forensics

Cluster Forensics

Databricks gives you one generic umbrella code and an aggregate timer — this names which startup stage actually spiked, and which of the five possible causes actually fired.

What it detects

  • A cold start reported as one number. The skill splits the PENDING window into provisioning / init-scripts / spark-startup, so a ‘random 25-minute start’ resolves to ‘provisioning 21m (84%)’ and points at cloud VM allocation or subnet/DNS rather than your init scripts. Boundary events that are absent are reported as unmeasured and never folded into another stage.
  • CLOUD_PROVIDER_LAUNCH_FAILURE and NPIP_TUNNEL_SETUP_FAILURE treated as one problem when they are five: subnet IP exhaustion, custom DNS that cannot resolve the control-plane or SCC-relay hostnames, an NSG or security-group rule blocking egress 443 to the relay, a subnet or VNet an IaC apply deleted out from under the workspace, and cloud capacity versus quota. Each has a different fix, which is why blanket-retry is the wrong reflex.
  • DBR upgrade landmines that do not fail the cluster, they fail the job or the numbers: the DBR 14.0 move of the working directory to the workspace filesystem, where a >500 MB intermediate write now errors; 15.1 removing DBFS-root library storage and JDK 11; and 15.4 flipping spark.sql.legacy.jdbc.useNullCalendar to true, which shifts JDBC TIMESTAMP values on pre-1582 and timezone-edge rows with no exception. An AST scanner finds the at-risk relative-path writes and a bytecode scanner inventories JAR JDK targets before the bump, not after cutover.
  • Spot reclaims that surface as ‘stage failure’ and never mention spot. The reclaimed worker’s shuffle blocks are gone, the reduce side throws FetchFailedException, Spark recomputes — and a second reclaim inside that window burns consecutive attempts against spark.stage.maxConsecutiveAttempts until the job aborts. The finding underneath is usually a driver sitting on spot, where a single reclaim is total loss.
  • Event-stream truncation misread as a hang. Databricks prunes old cluster events, so a missing INIT_SCRIPTS_FINISHED may mean ‘pruned’, not ‘init script hung’ — the skill flags the truncation instead of inferring a cause from an absence.

Who feels thisThe data-platform or SRE engineer on call at 2am when compute is broken and the cluster page shows a code with no English behind it — and the same platform owner who has to state, before a DBR upgrade window, exactly what will break.

databricks-bundle-medic

Bundle Medic

Your Asset Bundle deploys cleanly once and then bricks on the second run — and the deploy-time infrastructure around it fails or leaks money in ways nothing in the console tells you about.

What it detects

  • The second databricks bundle deploy dying on Error: reading terraform.tfstate: opening: unexpected EOF, so the promote-to-prod pipeline can never run a second deploy (databricks/cli#4986, reproduced on macOS and Linux across CLI v0.288.0 through v0.296.0). The state file is not actually corrupt — the CLI’s streaming read is. A PreToolUse hook caches a validated known-good backup before every deploy.
  • First deploy to a fresh workspace failing with User does not have CREATE TABLE on Schema even though the grants are declared — the transient GRANT-ordering bug (databricks/cli#4573) where the failed pass already applied the grants, so exactly one retry succeeds. If the same error survives a retry, the skill tells you to stop retrying: it is a real missing privilege.
  • The bundle-bind gap for UC catalogs and external locations (databricks/cli#4842). Brownfield catalogs and external locations built by the UI or an earlier Terraform run cannot come under GitOps without hand-editing terraform.tfstate (undocumented schema, one bad field breaks every future deploy) or destroy-and-recreate (impossible once dependent tables exist). The skill emits a review-first Terraform import plan instead.
  • PrivateLink enabled and the NAT bill up anyway. PrivateLink privatizes the control plane only — the data plane’s S3, STS and Kinesis calls still exit through the NAT gateway, turning normally-free same-region S3↔EC2 transfer into paid NAT data-processing bytes plus cross-AZ transfer, with no errors and no failed jobs to tip you off. Gateway-endpoint coverage is per-route-table, so one missed private subnet keeps quietly paying.
  • A CMK rotation scoped as a quick change window, when the cluster-volume key binds at boot and is never re-read. The account-level workspace PATCH is rejected until every cluster, instance pool and SQL warehouse is terminated, making it a workspace-wide outage. The managed-services key rotates live; conflating the two is what turns a zero-downtime change into an unplanned window.

Who feels thisThe platform engineer who owns the Databricks deploy pipeline and workspace infrastructure — woken when the promote-to-prod bundle deploy fails on its second run and cannot be unstuck, and the same person who has to explain the NAT gateway line item to FinOps at month-end.

How the numbers are produced

Every dollar figure comes from a query against your own system.billing.usage joined to list_prices, and the arithmetic runs in a bundled Python ranker rather than in the model’s head. Findings carry a confidence label that is never collapsed: Confirmed means money actually billed, Estimated means a model of spend against idle CPU, At-risk means pending review and is never counted as recoverable cash. The live config that explains why a leak exists comes from the databricks-workspace-mcp server reading your workspace REST API — and when that server is not registered, the skills say so and drop to advisory mode instead of guessing.

Objections

We already have cost dashboards.

Dashboards report totals per SKU and per workspace. They do not tell you that a specific job is billing the All-Purpose rate (~$0.55/DBU) for batch work that belongs on Jobs Compute (~$0.15/DBU), or that your Photon line item has no runtime gain behind it. This re-prices the same DBUs at the current Jobs rate to produce a deterministic delta, then corroborates against the live cluster config so the output is a named cluster and a single config change, not a chart. It also catches the case dashboards structurally cannot: instance pools whose min_idle_instances keeps warm VMs that Databricks reports as $0 while your cloud provider bills you directly.

Can it touch our data?

It reads system tables — system.billing.usage, system.compute.*, system.streaming.query_progress — and the workspace REST API. It does not read your business tables. Auth comes entirely from your environment (the CLI’s own token, or the MCP server’s PAT/U2M/M2M); nothing is hardcoded. System-table access requires a metastore-admin grant chain (USE CATALOG / USE SCHEMA / SELECT), so your metastore admin controls the scope and can grant read-only. Only one hook acts rather than reports: the streaming guard blocks a destructive SQL op when it confirms live consumers, and it fails open. The one operation that terminates compute — the workspace drain for CMK rotation — is dry-run by default and needs an explicit --execute.

Is this just an LLM guessing?

The detection is SQL and scripts; the model does not do the arithmetic. Every HMS table’s READY/BLOCKED/ORPHAN verdict comes from a Python classifier — the model never eyeballs a storage URI. The dollar ranking is a script. The cold-start stage split is a script, and when a boundary event is missing it reports ‘unmeasured’ rather than folding the time into another stage. The upstream bugs cite issue IDs you can look up yourself (databricks/cli#4986, #4573, #4842). And the grant probe runs before any analysis, so a missing permission stops the run with the exact GRANT statements instead of failing halfway through with partial numbers.