Skip to content
Tutorials Center
My learning

Topics

Lesson 40 of 45 ยท Projects

Project: Event-Driven Image or Data Pipeline

Build an event-driven pipeline: an upload to Cloud Storage publishes to Pub/Sub, a Cloud Run function processes it and writes the result to Firestore.

An event-driven pipeline reacts to things happening instead of running on a timer. In this project a user uploads a file to Cloud Storage, the upload publishes a message to Pub/Sub, a Cloud Run function wakes up to process the file, and it writes a small result document to Firestore. Nothing runs, and nothing is billed beyond storage, while no files arrive.

This pattern fits thumbnails, CSV imports, document scanning and audit trails. You need a Google Cloud project to build the real thing, so those commands are read-only here. The runnable part is a Node stand-in that models the same four stages in one process, including the at-least-once delivery that makes idempotent handlers essential.

Four stages

Upload A file lands in a bucket.
Pub/Sub The event becomes a message.
Function Code processes the file.
Firestore A result document is saved.
1 / 2
sequenceDiagram
  participant U as User
  participant B as Cloud Storage
  participant P as Pub/Sub topic
  participant F as Cloud Run function
  participant D as Firestore
  U->>B: upload photo.jpg
  B->>P: OBJECT_FINALIZE notification
  P->>F: deliver message
  F->>B: read the object
  F->>D: write result document
  F-->>P: success, acknowledge

If the function fails, Pub/Sub redelivers the message instead of losing it.

Step 0: Set up

Review Cloud Storage, Pub/Sub, Cloud Run functions and Firestore if needed.

Terminal
export PROJECT_ID=my-project-id
export REGION=us-central1
export BUCKET=my-uploads-12345

gcloud config set project $PROJECT_ID
gcloud services enable storage.googleapis.com pubsub.googleapis.com \
  firestore.googleapis.com cloudfunctions.googleapis.com run.googleapis.com \
  cloudbuild.googleapis.com artifactregistry.googleapis.com eventarc.googleapis.com

Step 1: Bucket, topic and notification

Create a private bucket for uploads, a Pub/Sub topic, and a notification that publishes a message to the topic whenever an object is finished uploading.

Terminal
gcloud storage buckets create gs://$BUCKET --location=$REGION --uniform-bucket-level-access
gcloud pubsub topics create uploads

gcloud storage buckets notifications create gs://$BUCKET \
  --topic=uploads \
  --event-types=OBJECT_FINALIZE

Cloud Storage publishes as a Google-managed service agent, which needs permission to publish to your topic. If the notification command complains, grant the Pub/Sub Publisher role to the storage service agent (you can see its address with gcloud storage service-agent):

Terminal
gcloud pubsub topics add-iam-policy-binding uploads \
  --member="serviceAccount:$(gcloud storage service-agent)" \
  --role=roles/pubsub.publisher

OBJECT_FINALIZE fires when an object is created or overwritten, so uploading the same name twice sends two messages. Filtering by prefix (--object-prefix=incoming/) keeps your function from reacting to its own output files, a classic cause of infinite loops.

Step 2: Firestore

Create a Firestore database in Native mode. A collection named uploads will hold one document per processed file.

Terminal
gcloud firestore databases create --location=$REGION

Step 3: The function

The function receives a message whose data is the base64-encoded JSON description of the object (name, bucket, size, content type, generation). It reads the metadata, builds a result, and saves it. Using a stable document ID makes the handler idempotent: if the same message is delivered twice, the second write overwrites the first with identical content rather than creating a duplicate.

JavaScript
// index.js
import functions from "@google-cloud/functions-framework";
import { Firestore } from "@google-cloud/firestore";

const db = new Firestore();

functions.cloudEvent("onUpload", async cloudEvent => {
  const message = cloudEvent.data.message;
  const object = JSON.parse(Buffer.from(message.data, "base64").toString());

  // Stable id: bucket + name + generation (changes only when the file changes)
  const id = encodeURIComponent(`${object.bucket}/${object.name}/${object.generation}`);

  await db.collection("uploads").doc(id).set({
    name: object.name,
    bytes: Number(object.size),
    contentType: object.contentType,
    kind: object.contentType?.startsWith("image/") ? "image" : "data",
    processedAt: new Date().toISOString()
  });
  console.log(JSON.stringify({ severity: "INFO", message: "processed", name: object.name }));
});

A real image job would also read the file with the Cloud Storage client, resize it, and write the thumbnail to a second bucket or the thumbs/ prefix. The structure stays the same.

Give the function its own service account with just the roles it needs: read objects from the bucket and write to Firestore.

Terminal
gcloud iam service-accounts create uploads-fn

gcloud storage buckets add-iam-policy-binding gs://$BUCKET \
  --member="serviceAccount:uploads-fn@$PROJECT_ID.iam.gserviceaccount.com" \
  --role=roles/storage.objectViewer
gcloud projects add-iam-policy-binding $PROJECT_ID \
  --member="serviceAccount:uploads-fn@$PROJECT_ID.iam.gserviceaccount.com" \
  --role=roles/datastore.user

Step 4: Deploy and trigger

Terminal
gcloud functions deploy on-upload \
  --gen2 \
  --region=$REGION \
  --runtime=nodejs22 \
  --source=. \
  --entry-point=onUpload \
  --trigger-topic=uploads \
  --service-account=uploads-fn@$PROJECT_ID.iam.gserviceaccount.com \
  --max-instances=10

Upload a file and watch it flow:

Terminal
echo "sample" > hello.txt
gcloud storage cp hello.txt gs://$BUCKET/incoming/hello.txt
gcloud functions logs read on-upload --region=$REGION --gen2 --limit=5

Then open Firestore in the console and find the document in uploads. If nothing appears, work down the chain: is the notification listed (gcloud storage buckets notifications list gs://$BUCKET), does the topic's subscription show undelivered messages, and does the function log an error? See common mistakes for the permission checklist.

Step 5: Make failure safe

Pub/Sub guarantees at-least-once delivery, so duplicates are normal and failures are retried. Prepare for both:

  • Keep handlers idempotent, as above.
  • Enable retries on the trigger so a thrown error causes redelivery, and attach a dead-letter topic to the subscription so a message that keeps failing is parked after a number of attempts instead of looping forever.
  • Set --max-instances so a flood of uploads cannot overwhelm downstream systems.

Cleanup

Terminal
gcloud functions delete on-upload --gen2 --region=$REGION
gcloud pubsub topics delete uploads
gcloud storage rm --recursive gs://$BUCKET
gcloud iam service-accounts delete uploads-fn@$PROJECT_ID.iam.gserviceaccount.com

The Firestore database stays until you delete it or the project; remove test documents or the project when done.

Stand-in: the whole pipeline in one process

This Node program models the stages with plain objects: a bucket that emits events, a topic with a subscription that retries failures, a handler, and a Map standing in for Firestore. The first delivery of one message fails on purpose, and one message is delivered twice, to show retries and idempotency. It is a teaching model of the behavior, not the Pub/Sub API.

Node.js Stand-in: upload, message, process, store

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with node:readline or process.stdin), it gets the next line from this box.

There are three messages but only two documents: the duplicate of cat.jpg overwrote its own document, and report.csv succeeded on the second try after a negative acknowledgement. If you change MAX_ATTEMPTS to 1, the failing message lands in the dead-letter list instead.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which Cloud Storage event type fires when an object is created or overwritten?

    OBJECT_FINALIZE signals a successful upload, including overwrites of an existing name.
  2. 2. Why must the handler be idempotent?

    Duplicate deliveries are normal. A stable document ID makes a repeat write harmless.
  3. 3. What does a dead-letter topic do?

    After the configured number of attempts, a failing message is moved aside so you can inspect it.
  4. 4. How can a function that writes thumbnails into the same bucket avoid triggering itself forever?

    Each new thumbnail is a new finalize event. Filtering by prefix or writing elsewhere breaks the loop.

Summary

  • A bucket notification turns uploads into Pub/Sub messages that trigger a Cloud Run function.
  • Delivery is at least once: write idempotent handlers with stable IDs.
  • Use retries and a dead-letter topic so failures are neither lost nor endless.
  • Give the function a dedicated service account with only storage read and Firestore write access, and cap its instances.

Next: Project: Log and Sales Analytics in BigQuery