Skip to content
Tutorials Center
My learning

Topics

Lesson 29 of 45 ยท Data and Analytics

Dataflow, Dataproc and Batch Pipelines

Process big data on Google Cloud: the Apache Beam model on Dataflow, Spark on Dataproc, windows, streaming vs batch and Cloud Composer, with a runnable pipeline.

Sooner or later data outgrows a single script. A log file becomes a thousand files, a daily report becomes a stream of events that never ends, and the one machine that ran your loop is not enough. Google Cloud offers managed services that spread the work across many machines for you: Dataflow for pipelines written with Apache Beam, Dataproc for Spark and Hadoop, and Cloud Composer for scheduling and coordinating jobs.

The first question is whether you need them at all. For many tasks, a SQL query in BigQuery is simpler: load the data, write a query, done. Reach for pipelines when you must transform data before it lands, process streams from Pub/Sub continuously, run existing Spark code, or do work that is awkward in SQL.

Pick a tool for the job

Dataflow Beam pipelines, batch or streaming, fully managed.
Dataproc Spark and Hadoop clusters you already know.
Composer Airflow to schedule and chain jobs.
BigQuery Often enough: just write SQL.

Start from the simplest tool that solves the problem.

1 / 2

The Beam model

Apache Beam is an open-source programming model for data processing, with SDKs for Java, Python and Go. You describe what should happen to the data, and a runner decides how to execute it. Dataflow is Google's managed runner: you hand it the pipeline and it provisions workers, splits the work, scales the number of workers up and down, and retries failed pieces.

A Beam pipeline has a few core ideas:

  • A pipeline is the whole job, from reading input to writing output.
  • A PCollection is a dataset flowing through the pipeline. It can be bounded (a fixed set, like files) or unbounded (a stream that never ends).
  • A transform takes one or more PCollections and produces another: Map, Filter, FlatMap, GroupByKey, Combine.

The classic example counts words. In Beam's Python SDK it reads like this (read-only here, because it needs the apache-beam package and a runner):

Python
import apache_beam as beam

with beam.Pipeline() as p:
    (
        p
        | beam.io.ReadFromText("gs://my-bucket/books/*.txt")
        | beam.FlatMap(lambda line: line.lower().split())
        | beam.Map(lambda word: (word, 1))
        | beam.CombinePerKey(sum)
        | beam.io.WriteToText("gs://my-bucket/out/wordcounts")
    )

The | operator chains steps. To run it on Dataflow you pass options such as the runner name, the project and a temporary storage location; the same code runs on your laptop with the default local runner for testing.

Because the transforms are independent per element until a GroupByKey, the runner can process pieces on many machines at once. The grouping step is where data with the same key is brought together, a step often called the shuffle.

Batch and streaming

The same model handles both kinds of data:

  • Batch: the input is finite, such as yesterday's files. The job starts, finishes and stops billing.
  • Streaming: the input is unbounded, such as a Pub/Sub subscription. The job keeps running and results arrive continuously.

Streaming raises a question batch does not: when is a result "complete"? An event that happened at 10:00:05 may reach the pipeline at 10:00:40 because a phone was offline. Beam separates event time (when it happened) from processing time (when you saw it) and uses three concepts to cope:

  • Windows cut an endless stream into finite pieces for aggregation: fixed (every minute), sliding (the last five minutes, updated every minute) or session (a burst of activity ended by a gap).
  • A watermark is the system's estimate of "event time has probably reached this point", so a window can be closed.
  • Triggers and allowed lateness control when to emit results early and how long to accept stragglers.

You will run a simplified version of these ideas in the stand-in below.

Dataflow in practice

Dataflow is serverless from your point of view. You pay for the workers (CPU, memory, disk) while the job runs, and autoscaling adds workers when the backlog grows. Some practical points:

  • A Flex Template packages a pipeline as a container so others can launch it with parameters, without installing Beam.
  • Google also provides templates for common jobs, such as Pub/Sub to BigQuery, so no code is needed.
  • Pipelines read from and write to Cloud Storage, BigQuery, Pub/Sub and many other systems.
  • Run workers with their own service account having only the roles the job needs.
  • Design for idempotent writes: a retried piece of work may run again.
Terminal
# Read-only: launch a Google-provided template.
gcloud dataflow jobs run pubsub-to-bq \
  --gcs-location=gs://dataflow-templates/latest/PubSub_Subscription_to_BigQuery \
  --region=europe-west1 \
  --parameters=inputSubscription=projects/my-project/subscriptions/billing-sub,outputTableSpec=my-project:shop.events

The template path and parameter names above are an example of the shape: check the current template documentation for the exact names.

Dataproc and Spark

Dataproc runs Apache Spark and Hadoop on managed clusters. If your company already has Spark jobs, or your team knows Spark and its libraries, Dataproc lets you move them with few changes. A cluster starts in about a minute or two, so a common pattern is ephemeral clusters: create one for a job, run it, delete it, and pay only for the minutes used. Input and output live in Cloud Storage rather than on the cluster's disks, so deleting the cluster loses nothing. There is also a serverless option where you submit a Spark job (a batch) without managing a cluster at all.

Terminal
# Read-only sketch of an ephemeral-cluster workflow.
gcloud dataproc clusters create nightly --region=europe-west1 --num-workers=2
gcloud dataproc jobs submit pyspark gs://my-bucket/jobs/report.py \
  --cluster=nightly --region=europe-west1
gcloud dataproc clusters delete nightly --region=europe-west1

Choose between Dataflow and Dataproc this way: for new pipelines, especially streaming, Dataflow with Beam is the usual choice; for existing Spark or Hadoop code and ecosystem tools, Dataproc is the shorter path.

Cloud Composer

Real data work is rarely one job. Load the files, then transform them, then refresh a table, then email a report, and run all of that every night, retrying the steps that fail. Cloud Composer is managed Apache Airflow, where you write a DAG (directed acyclic graph) in Python: tasks plus dependencies between them. Composer runs the scheduler and workers; tasks can start Dataflow jobs, run BigQuery queries or create Dataproc clusters. It is powerful but is a service that runs continuously and costs money even when idle, so for one simple job a scheduled query or a Cloud Scheduler trigger may be enough.

flowchart LR
  F["Files land in<br/>Cloud Storage"] --> D["Dataflow job<br/>clean and transform"]
  D --> B["BigQuery table"]
  B --> R["Report query"]
  C["Composer DAG"] -.schedules.-> D
  C -.schedules.-> R

Composer does not move data itself; it starts the services that do and tracks the order.

Cloud Data Fusion: pipelines without code

Cloud Data Fusion is a managed service for building data pipelines by dragging and connecting boxes in a web interface. It is based on the open-source CDAP project. A pipeline is a graph of plugins: sources (Cloud Storage, BigQuery, databases, Pub/Sub), transforms and sinks. The studio shows each stage's schema and lets you preview sample data before you deploy.

Two parts to know:

  • Transformations are the middle boxes: filter rows, rename or drop columns, join two inputs, group and aggregate, convert types, or run a small script.
  • Wrangler is the visual data-preparation tool. You load a sample, click on a column and apply steps ("parse as CSV", "split on comma", "fix dates", "drop empty rows"). Each step is saved as a directive, so the clean-up becomes a repeatable recipe that runs on the full data.

When you run a deployed pipeline, Data Fusion creates an ephemeral Dataproc cluster, runs the work there, and deletes the cluster. Choose it for teams that prefer a visual tool, or for standard integrations with many connectors. Choose Dataflow or Spark code when you need fine control, unusual logic, or software-engineering practices such as unit tests and code review for every change.

Databricks on Google Cloud

Databricks is a data and AI platform built around Apache Spark, available on Google Cloud as a partner service you buy through the Marketplace. Its architecture has two halves:

  • The control plane is run by Databricks: the web application, notebooks, job scheduler and cluster manager.
  • The compute plane is in your Google Cloud project: the clusters that process the data and the buckets that hold it. Your data stays in your account.

Key ideas:

  • Lakehouse. Files stay in cheap object storage (Cloud Storage), but a table format on top gives them warehouse features. Delta Lake adds ACID transactions, schema enforcement and time travel to Parquet files, so a "lake" can be queried and updated like a database.
  • Medallion architecture. Data moves through quality layers: bronze (raw, as it arrived), silver (cleaned and joined) and gold (aggregated, ready for dashboards and models). If a rule changes, you rebuild silver and gold from bronze.
  • Data governance. Unity Catalog is the central place for permissions, lineage and auditing across tables, files and models.
  • Notebooks. You work in shared notebooks (Python, SQL, Scala, R) attached to a cluster. The first steps are to sign in, create a workspace, create a cluster, then create a notebook and run a cell.

Databricks overlaps with Dataproc (both run Spark) and with BigQuery (both can be the home of analytics). Teams often choose it for a strong notebook and machine-learning experience, or for a multi-cloud strategy, and BigQuery or Dataproc for a Google-native stack. The next stand-in imitates a bronze, silver and gold flow with plain Python.

Python Bronze, silver, gold

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Stand-in: a tiny pipeline in Python

Beam itself needs packages and a runner that this page cannot provide, so the first stand-in rebuilds the shape of a pipeline with plain Python. Each step is a small object, and | chains them like Beam does. There is no parallelism and no cluster; the point is to see map, group and reduce as separate stages. Running it in your browser uses a real Python interpreter (Pyodide), but the pipeline is our own code, not Beam.

Python Stand-in: word count as map, group, reduce

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that Map and Filter work on each element alone (a runner can spread them over many machines), while GroupByKey needs to see all values for a key together. That grouping is the expensive step in a real cluster.

Stand-in: fixed windows and late data

The second stand-in groups timestamped events into one-minute fixed windows and drops events that arrive too late. Real Beam watermarks are estimates based on the sources; here the "watermark" is simply the largest event time seen so far, which is a deliberate simplification. Each event has an event time (when it happened) and an arrival time (when the pipeline saw it).

Python Stand-in: windows, watermark and lateness

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The event at time 50 arrived after the pipeline had seen time 65, yet it still counted in the first window because it was within the allowed lateness. The last event was too late and was dropped. In Beam you decide this policy explicitly, trading completeness against waiting time.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What is Dataflow?

    Dataflow runs Beam pipelines, handling workers, scaling and retries for batch and streaming jobs.
  2. 2. Which situation points to Dataproc rather than Dataflow?

    Dataproc runs Spark and Hadoop, so existing code and tools carry over.
  3. 3. What does a window do in stream processing?

    Fixed, sliding and session windows group events by event time so they can be aggregated.
  4. 4. What is Cloud Composer used for?

    Composer is managed Apache Airflow: it coordinates tasks that run in other services.
  5. 5. Why use ephemeral Dataproc clusters with data kept in Cloud Storage?

    Storing input and output outside the cluster lets you create the cluster for a job and delete it afterwards.

Summary

  • Dataflow runs Apache Beam pipelines (batch and streaming) without cluster management.
  • A Beam pipeline chains transforms over PCollections; grouping brings equal keys together.
  • Streaming needs windows, watermarks and a lateness policy because events arrive late and out of order.
  • Dataproc runs Spark and Hadoop, ideally as short-lived clusters with data in Cloud Storage.
  • Cloud Composer (Airflow) schedules and orders jobs; BigQuery SQL is often the simplest answer.

Next: Cloud Logging, Monitoring and Alerting