Skip to content
Tutorials Center
My learning

Topics

Lesson 30 of 37 ยท Connectors and Deployment

Deployment Modes and Clusters

Learn the Flink cluster architecture (JobManager, TaskManagers, slots), operator chaining, application and session modes, Kubernetes and YARN, and high availability.

A Flink job that works on your laptop must eventually run on a cluster that survives machine failures, scales with traffic and can be upgraded without data loss. This lesson explains the pieces of a Flink cluster, how a program is turned into parallel tasks and slots, the main deployment modes, and what high availability means.

Cluster commands and configuration are shown read-only; the Java stand-ins model slot and task scheduling so you can calculate resource needs yourself. It follows Exactly-Once Sinks and Two-Phase Commit.

Cluster roles

JobManager Schedules tasks, coordinates checkpoints.
TaskManager Runs tasks in slots.
Dispatcher Accepts job submissions, serves the web UI.
Resource manager Asks the platform for TaskManagers.
1 / 2

Cluster architecture

A Flink cluster has two kinds of processes:

  • The JobManager (which contains the Dispatcher, the ResourceManager and one JobMaster per running job) turns your job into an execution graph, schedules tasks, triggers checkpoints and coordinates recovery.
  • One or more TaskManagers are worker JVMs. Each has a fixed number of task slots, the unit of resources a task can occupy, and executes tasks in threads.

A slot gets a fixed share of the TaskManager's managed memory (CPU is not isolated between slots). A TaskManager with 4 slots can run four parallel pipelines at once. Slots from different operators of the same job may share: by default, a whole pipeline slice (source -> map -> keyBy -> window -> sink of the same parallel index) fits in one slot, a feature called slot sharing. Therefore the number of slots a job needs equals its maximum operator parallelism, not the total number of tasks.

Operator chaining

Running every operator in its own thread would mean copying records between threads. Flink chains adjacent operators that need no network shuffle (for example map then filter) into a single task, so records pass by plain method call. A shuffle (keyBy, rebalance) breaks the chain. You rarely need to touch it, but startNewChain(), disableChaining() and slotSharingGroup(...) exist for rare cases, such as isolating a heavy operator.

The stand-in calculates tasks and slots for a pipeline described by operators and parallelisms. Chains and slot sharing explain why 10 tasks can fit in 4 slots.

Java Stand-in: chaining and slot calculation

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Deployment modes

How a job reaches the cluster is the deployment mode:

  • Application mode starts a dedicated cluster for one application. The main() method runs on the JobManager, so the client does not need to ship the job graph. The cluster ends when the application ends. It gives the best isolation (one job cannot hurt another) and is the recommended mode for production streaming jobs.
  • Session mode runs a long-lived cluster to which many jobs are submitted. Startup is instant and resources are shared, which suits many short, small jobs or interactive SQL, at the cost of weaker isolation: a misbehaving job can take down the JobManager and affect others.

(Older per-job mode, which started a cluster per submitted job graph, has been superseded by application mode.)

Platforms

Flink runs on:

  • Kubernetes: the most common choice today. Two ways: native Kubernetes (Flink talks to the Kubernetes API and requests pods as it needs them) and the Flink Kubernetes Operator, which adds a FlinkDeployment custom resource for declarative deployment, upgrades with savepoints, autoscaling and health checks.
  • YARN: for existing Hadoop clusters.
  • Standalone: processes you start yourself (./bin/start-cluster.sh), good for learning and bare metal.
  • Managed services: such as Amazon Managed Service for Apache Flink, Confluent Cloud for Apache Flink and Ververica.

A FlinkDeployment for the Kubernetes operator looks like this:

YAML
apiVersion: flink.apache.org/v1beta1
kind: FlinkDeployment
metadata:
  name: click-counter
spec:
  image: registry.example.com/click-counter:1.4.0
  flinkVersion: v2_0
  flinkConfiguration:
    taskmanager.numberOfTaskSlots: "2"
    state.backend.type: rocksdb
    execution.checkpointing.interval: 60 s
    state.checkpoints.dir: s3://bucket/checkpoints
    state.savepoints.dir: s3://bucket/savepoints
    high-availability.type: kubernetes
    high-availability.storageDir: s3://bucket/ha
  serviceAccount: flink
  jobManager:
    resource: { memory: "2048m", cpu: 1 }
  taskManager:
    resource: { memory: "4096m", cpu: 2 }
  job:
    jarURI: local:///opt/flink/usrlib/click-counter.jar
    parallelism: 4
    upgradeMode: savepoint
    state: running

Changing image or parallelism and applying the file makes the operator take a savepoint, stop the job and start the new version from it. See State Backends and Savepoints.

High availability

The JobManager is a single point of failure unless you enable high availability (HA). With HA, Flink stores job graphs, checkpoint pointers and leader information in a durable place (Kubernetes ConfigMaps or ZooKeeper, plus a storage directory), and runs standby JobManagers. If the leader dies, a standby takes over, finds the latest completed checkpoint and resumes the job. TaskManager failures are handled with or without HA: the JobManager restarts the affected tasks from the last checkpoint.

Restart behavior is governed by a restart strategy, for example fixed delay or exponential delay (restart-strategy.type: exponential-delay). Exponential delay is a good default: it retries quickly at first and slows down so a persistent failure does not hammer external systems.

Java Stand-in: exponential-delay restart schedule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Sizing a first deployment

A useful starting recipe: choose the parallelism from your throughput target and the per-core capacity you measured in a test, set numberOfTaskSlots to the number of cores per TaskManager (commonly 1 to 4), give each TaskManager enough memory for the RocksDB managed memory plus the heap, and size the JobManager at 1-2 CPU and 2-4 GB unless you run thousands of tasks. Test with a realistic data volume before trusting any number.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. How many slots does a job need with slot sharing enabled?

    A pipeline slice of different operators shares a slot, so the maximum parallelism decides the slot need.
  2. 2. Which deployment mode gives the best isolation for a production streaming job?

    Each application gets its own JobManager and TaskManagers, so jobs cannot interfere.
  3. 3. What does the Flink Kubernetes Operator do on an image update with upgradeMode savepoint?

    It automates the stateful upgrade procedure.
  4. 4. Why enable high availability?

    HA persists metadata needed to recover the job after a JobManager failure.

Summary

  • A cluster has a JobManager (scheduling, checkpoints, recovery) and TaskManagers with slots that run tasks.
  • Operator chaining merges operators without shuffles; slot sharing means slots needed equals max parallelism.
  • Application mode (a cluster per application) is the production default; session mode shares a cluster.
  • Kubernetes (native or the Flink Kubernetes Operator) is the usual platform; YARN and standalone also work.
  • Enable HA and set a restart strategy such as exponential delay.

Next: Monitoring and Backpressure