Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 36 ยท Governance and Cost

Cost Optimization and FinOps for Data Platforms

Cut data platform spend: understand storage, compute and egress costs, prune partitions, right-size clusters, use lifecycle tiers, and build cost visibility and budgets.

Cloud data platforms scale effortlessly, including their bills. One careless query that scans a 50 TB table every five minutes can cost more than the engineer who wrote it. FinOps (cloud financial operations) is the habit of making cost visible, assigning it to owners, and engineering it down without hurting the business.

The good news: most savings come from a short list of ideas you have already met in this course.

Where the money goes

Cost area Driver Typical lever
Storage GB stored per month, tier, number of copies compression, lifecycle tiers, delete old data
Compute cluster hours or bytes scanned right-sizing, auto-stop, pruning, caching
Data transfer bytes out of a region or cloud (egress) co-locate compute and data, compress, cache
Requests API calls to object storage (small files!) compaction, larger files
Managed services per-GB ingested, per-seat, per-connector pick the right service, avoid duplicate tools

Two pricing models matter: provisioned (you pay for clusters or slots while they exist, so idle time is waste) and on-demand (you pay per byte scanned or per second, so bad queries are waste).

Lever 1: scan less data

In a warehouse billed by bytes scanned, every technique from this chapter pays directly:

  • Columnar storage and column selection: SELECT * reads every column; naming two columns can cut the bytes by 90%.
  • Partition pruning and clustering: filter on the partition or clustering column.
  • Compression and efficient types: zstd; int or date instead of long strings.
  • Pre-aggregation: a daily summary table serves dashboards cheaply instead of scanning raw events.
Python Estimate scan cost with and without pruning

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

If the baseline query costs $20 a run, the last scenario costs a fraction of that, and the saving multiplies by how often the query runs.

Lever 2: right-size and auto-stop compute

  • Auto-suspend warehouses and clusters after a few idle minutes. An 8-node cluster idle for the weekend is pure loss.
  • Right-size: start small and scale up on evidence. Doubling a cluster often halves runtime for the same total cost on parallel work, but not for small queries.
  • Use spot/preemptible instances for fault-tolerant batch jobs (Spark retries lost tasks); never for latency-critical services.
  • Separate workloads: a heavy nightly job should not share compute with interactive dashboards; give each a size and budget of its own.
  • Autoscaling with limits: set a maximum so a runaway query cannot spin up unlimited resources.
Python Compare always-on vs auto-stopped cluster

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Lever 3: storage tiers and lifecycle

Not all data is equally "hot". Object stores offer tiers (standard, infrequent access, archive) at falling per-GB prices but rising retrieval costs and delays. A lifecycle policy moves or deletes data by age automatically.

flowchart LR
  H["Hot<br/>0-30 days<br/>standard"] --> W["Warm<br/>30-180 days<br/>infrequent access"] --> C["Cold<br/>180+ days<br/>archive"] --> X["Delete after retention"]
Python Monthly storage cost with a lifecycle policy

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Also: delete temporary and orphaned data (failed job output, old snapshots after retention), keep fewer copies of derived data you can rebuild, and expire table-format snapshots.

Lever 4: do less work

  • Incremental processing instead of full refreshes (see the SQL patterns lesson).
  • Do not recompute what can be cached or materialized once.
  • Reduce frequency where freshness is not valuable: an hourly dashboard refresh that nobody checks overnight can be paused.
  • Deduplicate pipelines: three teams building the same metric three times pay three times.
  • Drop unused tables and dashboards; lineage and query logs show what nobody reads.

Visibility, ownership and budgets

You cannot reduce what you cannot see:

  1. Tag everything (team, project, environment) so cost can be attributed.
  2. Chargeback / showback: show each team its spend, or bill it. People optimize what they pay for.
  3. Budgets and alerts: notify at 50/80/100% of a monthly budget, and anomaly alerts for sudden jumps.
  4. Query-level reporting: top queries by bytes scanned or cost; fix the worst offenders first.
  5. Unit economics: track cost per pipeline run, per million events, or per active user, so growth that is merely proportional does not alarm anyone.
Python Find the most expensive queries from a log

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The report shows that one ad hoc query costs more than a dashboard run many times. Targeted fixes, such as adding a partition filter, usually beat broad restrictions.

Trade-offs

Cheaper is not always better. Spot instances can be interrupted, archive tiers add hours to retrieval, and over-aggressive pruning of data can violate retention requirements. Compare savings with engineering time and risk; a $50/month saving that costs a week of work is a bad deal, while a $5,000/month one is not.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A warehouse bills per byte scanned. Which change cuts the cost of a query most reliably?

    Column pruning and partition pruning reduce the bytes read.
  2. 2. What does auto-suspend do for a provisioned cluster?

    You pay for provisioned compute while it exists, so stopping idle compute saves money.
  3. 3. What is a storage lifecycle policy?

    Lifecycle rules shift cold data to cheaper storage and expire data past retention.
  4. 4. Why tag resources by team and environment?

    Attribution creates ownership; people reduce costs they can see and are accountable for.

Summary

  • Costs come from storage, compute, transfer and requests; know whether your engine bills by time or by bytes.
  • Scan less (columns, partitions, pre-aggregation), stop idle compute, tier and expire storage, and process incrementally.
  • Tag, report, budget and track unit costs; weigh savings against effort and risk.

Next: Project: A Batch ETL Pipeline