Skip to content
Tutorials Center
My learning

Topics

Lesson 2 of 37 ยท Getting Started

Flink Architecture

Understand how a Flink cluster runs a job: client, JobManager, Dispatcher, ResourceManager, TaskManagers, task slots, parallelism and how work is spread over machines.

A Flink program you write on your laptop is just a description of a dataflow. To run it at scale, Flink needs a cluster: some processes that decide what runs where, and others that do the heavy lifting. Knowing the roles of these processes makes error messages, dashboards and tuning advice much easier to understand.

This lesson tours the components, explains task slots (the most confusing term for beginners) and shows with a small simulation how a job with a given parallelism is spread over machines. It follows What Is Apache Flink?.

The main components

flowchart TB
    C["Client<br/>builds the job graph and submits it"] --> D["Dispatcher<br/>REST API and web UI"]
    D --> JM["JobManager (one per job)<br/>scheduling, checkpoints, recovery"]
    JM <--> RM["ResourceManager<br/>owns the slots"]
    RM --> TM1["TaskManager 1<br/>slot slot slot"]
    RM --> TM2["TaskManager 2<br/>slot slot slot"]
    JM --> TM1
    JM --> TM2

The control plane (client, Dispatcher, JobManager, ResourceManager) and the workers (TaskManagers).

  • Client. Your main method runs here. It builds the job graph from your code and submits it. It is not part of the running cluster; once the job is submitted the client may disconnect (in detached or application modes).
  • Dispatcher. Accepts job submissions through a REST endpoint, serves the web UI and starts a JobManager for every submitted job.
  • JobManager. The coordinator of one job. It turns the job graph into an execution graph, asks for resources, deploys tasks, triggers checkpoints and handles failures by restarting tasks from the last checkpoint.
  • ResourceManager. Knows the cluster's resources and hands out slots. It talks to YARN, Kubernetes or the standalone cluster to start or stop TaskManagers.
  • TaskManagers (workers). JVM processes that run the actual operators. They exchange data with each other over the network and report to the JobManager.

In older text you will see the word "master" for the JobManager side and "worker" for a TaskManager. Both are JVM processes; in a small test setup, one machine can host all of them.

Task slots

A TaskManager does not run "any number of tasks". It offers a fixed number of task slots, a slot being a slice of the TaskManager's managed memory (CPU is not isolated). A slot is the unit of scheduling: one slot can run one parallel pipeline of your job, for example "source 3 โ†’ map 3 โ†’ window 3".

Settings you will meet:

  • taskmanager.numberOfTaskSlots is how many slots each TaskManager offers. A common rule is one slot per CPU core.
  • parallelism.default is the parallelism used when you do not set one in code.

By default Flink uses slot sharing: several different operators of the same job can sit in the same slot, as long as they belong to the same parallel pipeline. So a job needs only as many slots as its highest parallelism, not as the sum of all operators' parallelism.

Job Operator parallelism Slots needed (with sharing)
source โ†’ map โ†’ sink, all 4 4, 4, 4 4
source (2) โ†’ window (6) โ†’ sink (1) 2, 6, 1 6

Simulating slot assignment

The stand-in below plays the part of the ResourceManager: it has two TaskManagers with three slots each and places the parallel subtasks of a job so that each slot holds one parallel pipeline (slot sharing).

Java Stand-in: assigning subtasks to slots

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Change the window parallelism to 8 and run it again: the job asks for more slots than the cluster has, which in real Flink leaves the job in a scheduling state until more TaskManagers join (or fails after a timeout).

From your code to running tasks

Your program goes through several graph forms:

  1. StreamGraph. One node per operator call, built by the client.
  2. JobGraph. Operators that can be chained are merged into one node so they run in the same thread and avoid network and serialization costs (for example map followed by filter).
  3. ExecutionGraph. The JobManager expands each node into parallelism many parallel subtasks and decides where each runs.

The next lesson, The Dataflow Model, shows chaining in detail.

Deployment modes

How the processes above are started depends on the mode:

  • Session mode. A long-running cluster accepts many jobs. Cheap to submit to, but jobs share resources and can disturb each other.
  • Application mode. The cluster is started for one application and the main method runs on the JobManager. This is the recommended mode for production, with full isolation.
  • (The old per-job mode is deprecated in favour of application mode.)

Resource providers include standalone (you start the processes), Kubernetes (the Flink Kubernetes Operator is the usual choice) and YARN. Deployment is covered in detail in the later chapters of the Apache Flink track.

What a TaskManager's memory looks like

A TaskManager is one JVM, and Flink splits its memory into pools: framework memory, task heap for your objects, managed memory (used by RocksDB state backends and batch sorting), network buffers and JVM overhead. Slots divide managed memory evenly, so more slots per TaskManager means a smaller share for each. This is why the usual advice is: a few mid-sized TaskManagers with several slots each, instead of thousands of tiny ones.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which component coordinates one job, triggers checkpoints and restarts failed tasks?

    The JobManager schedules tasks, triggers checkpoints and handles recovery for its job.
  2. 2. What is a task slot?

    Slots are the scheduling units of a TaskManager. With slot sharing, one slot runs one parallel slice of the whole pipeline.
  3. 3. A job has operators with parallelism 2, 6 and 1. With slot sharing, how many slots does it need?

    With slot sharing, the job needs as many slots as its highest operator parallelism, here 6.
  4. 4. Why does Flink chain operators such as map and filter?

    Chained operators exchange records by plain method calls inside one task instead of sending them over the network.

Summary

  • The Client builds and submits the job; the Dispatcher accepts it; a JobManager coordinates it; the ResourceManager manages slots; TaskManagers run the work.
  • A slot is the unit of scheduling, and slot sharing means a job needs only as many slots as its highest parallelism.
  • Code becomes StreamGraph, then JobGraph (with chaining), then an ExecutionGraph of parallel subtasks.
  • Application mode is the recommended production deployment mode.

Next: Setting Up Flink and Running Your First Job.