Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 37 ยท Getting Started

The Dataflow Model and the Job Graph

Learn how a Flink program becomes a dataflow graph: sources, transformations, sinks, operator chaining, forward and hash exchanges, and how to read the plan.

Every Flink program is, underneath, a directed graph of operators connected by streams. You write code that looks like a chain of method calls; Flink turns it into a graph and then decides how to cut it into tasks. Understanding that graph helps you predict where data crosses the network, why one slow operator slows the whole job, and what the boxes in the web UI mean.

This lesson builds on Setting Up Flink and Running Your First Job.

Sources, transformations and sinks

Three kinds of node make up the graph:

  • Source. Reads data into the job (Kafka topic, file, socket, generator).
  • Transformation / operator. Takes one or more streams and produces a stream: map, filter, flatMap, keyBy, window, process, union, connect.
  • Sink. Writes the final data out (database, Kafka topic, files, print).
Java
DataStream<String> events = env.fromSource(source, watermarks, "events");   // source
DataStream<Event>  parsed = events.map(Event::parse);                        // operator
DataStream<Event>  valid  = parsed.filter(e -> e.amount() > 0);              // operator
valid.keyBy(Event::userId).sum("amount").sinkTo(sink);                       // operators + sink

A transformation does not process data when you call it. It adds a node to the graph and returns a new DataStream handle that points to that node. Only env.execute() ships the finished graph to the cluster.

Streams between operators

Two operators connected by a stream can exchange records in two basic ways:

  • One-to-one (forwarding). Subtask i of the upstream operator sends to subtask i of the downstream operator. Order is preserved and no data needs to be re-partitioned. map after filter is a typical case.
  • Redistributing. The stream is repartitioned and each upstream subtask may send to several downstream subtasks. Examples are keyBy (by hash of the key), rebalance() (round-robin), rescale(), broadcast() and shuffle() (random). These cross the network and do not preserve order between different upstream subtasks.
flowchart LR
    subgraph A["Source (p=2)"]
      A1["s1"]
      A2["s2"]
    end
    subgraph B["map+filter (p=2)"]
      B1["m1"]
      B2["m2"]
    end
    subgraph C["window sum (p=2)"]
      C1["w1"]
      C2["w2"]
    end
    A1 --> B1
    A2 --> B2
    B1 --> C1
    B1 --> C2
    B2 --> C1
    B2 --> C2

Straight lines: forwarding. Crossing lines: a keyBy redistribution.

Operator chaining

Sending a record over the network means serializing it, buffering it and deserializing it. When two operators are connected one-to-one and have the same parallelism, Flink chains them into a single task: they run in one thread and pass records by ordinary method call. In the picture above source, map and filter may be chained into one task, and the window stays a second task because keyBy forces a redistribution.

You normally let Flink decide, but you can influence it:

Java
env.disableOperatorChaining();                 // for the whole job (debugging)
stream.map(...).disableChaining();             // this operator starts a new chain and is not chained at all
stream.map(...).startNewChain();               // start a new chain here
stream.map(...).slotSharingGroup("heavy");     // put this operator (and followers) in another slot sharing group

Disable chaining only to isolate a slow operator in the metrics, or to give a heavy operator its own slot. Chains are good for performance.

From StreamGraph to tasks

flowchart LR
    P["Your code"] --> SG["StreamGraph<br/>one node per call"]
    SG --> JG["JobGraph<br/>chains merged"]
    JG --> EG["ExecutionGraph<br/>parallel subtasks"]
    EG --> T["Tasks in slots"]

The same program in four shapes. The JobGraph is what the web UI shows.

A stand-in for chaining

The stand-in below groups a list of operators into chains using the rule "an edge can be chained if it is forward and the parallelism is equal", the same rule Flink applies.

Java Stand-in: which operators chain into one task?

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Three tasks come out: the first three operators are chained, the keyed window starts a new task (hash exchange) and takes the formatter along, and the sink starts a third task because its parallelism is different. A parallelism change always breaks a chain.

Parallel dataflows and data skew

Each operator has one or more subtasks, and one stream has one or more stream partitions. With parallelism 4, the same map function runs as four independent subtasks on different data. This is why the lambda you write must not depend on shared mutable static state.

When the redistribution is by key, a few hot keys can send most records to one subtask while the rest idle. That is data skew, the most common reason a job is slower than the hardware suggests. The metrics in the web UI show it as one subtask with high busy time while others are low.

Java Stand-in: skewed keys overload one subtask

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

One subtask receives 800 of the 1000 events while another receives none. Adding more parallelism will not help, because the popular key still hashes to one place. Remedies are covered later: pre-aggregating before the keyBy, adding a random "salt" to the key and re-merging, or choosing a better key.

Reading the plan

env.getExecutionPlan() returns JSON that can be pasted into Flink's plan visualizer. In the web UI, each job-graph box shows the chained operator names, its parallelism, records and bytes in/out, and backpressure and busy indicators. Use these three to find the bottleneck: the slowest busy operator is usually just downstream of the first one flagged as backpressured. See Backpressure and Flow Control.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. When does Flink actually start processing data after you write source.map(...).keyBy(...).sum(...).print()?

    Transformations only build the graph; execute() submits it.
  2. 2. Which operation forces records to be redistributed across the network?

    keyBy partitions records by the hash of the key, so records move between subtasks.
  3. 3. What is the benefit of operator chaining?

    Chaining removes buffers and serialization between connected operators.
  4. 4. A few very popular keys make one subtask much busier than the others. What is this called?

    Skew means unevenly distributed keys. More parallelism alone does not fix it.

Summary

  • A Flink program is a graph of sources, operators and sinks; transformations are lazy until execute().
  • Streams between operators are either forwarding (one-to-one) or redistributing (keyBy, rebalance, broadcast).
  • Forward-connected operators with equal parallelism are chained into one task for speed.
  • StreamGraph becomes JobGraph (chained) becomes ExecutionGraph (parallel subtasks).
  • Skewed keys overload single subtasks; the web UI helps you spot them.

Next: Bounded and Unbounded Streams.