Skip to content
Tutorials Center
My learning

Topics

Lesson 3 of 37 ยท Getting Started

Setting Up Flink and Running Your First Job

Install Apache Flink, start a local cluster, create a Maven project and run a streaming WordCount job. Includes the web UI tour and a runnable stand-in for the logic.

Time to build something. In this lesson you install Flink locally, start a small cluster, create a Java project with the right dependencies and run the classic first streaming program: counting words. The commands and the Flink code are read-only here (they need a real machine), but the word-count logic is also written as a runnable stand-in so you can follow every step.

You should have read Flink Architecture first, so the words JobManager and TaskManager already make sense.

Prerequisites

  • Java 17 or 21 (Flink 2.x no longer supports Java 8; 17 or 21 is the safe choice). Check with java -version.
  • Maven or Gradle for building jobs.
  • Linux, macOS or Windows with WSL. The Flink scripts are Bash scripts, so on Windows use WSL or Docker.

Download and start a local cluster

Terminal
# download a release from flink.apache.org/downloads and unpack it
tar -xzf flink-2.1.0-bin-scala_2.12.tgz
cd flink-2.1.0

./bin/start-cluster.sh        # starts a JobManager and one TaskManager

Open http://localhost:8081. This is the Flink web UI, served by the Dispatcher. You will see the available task slots (by default 1 per TaskManager), running and completed jobs, and links to logs.

Useful commands:

Terminal
./bin/flink run examples/streaming/WordCount.jar   # submit a bundled example job
./bin/flink list                                    # list running jobs
./bin/flink cancel <jobId>                          # stop a job
./bin/stop-cluster.sh                               # shut the cluster down

Output of print() sinks goes to the TaskManager's stdout file (log/flink-*-taskexecutor-*.out), also reachable from the web UI under Task Managers > Stdout.

Create a project

A Flink job is a normal Java application packaged as a JAR. A minimal pom.xml fragment (versions shown are examples; use the release you installed):

XML
<properties>
  <flink.version>2.1.0</flink.version>
</properties>

<dependencies>
  <dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-streaming-java</artifactId>
    <version>${flink.version}</version>
    <scope>provided</scope>
  </dependency>
  <dependency>
    <groupId>org.apache.flink</groupId>
    <artifactId>flink-clients</artifactId>
    <version>${flink.version}</version>
    <scope>provided</scope>
  </dependency>
</dependencies>

Scope provided keeps the Flink classes out of your JAR because the cluster already has them. If you want to run the job directly from your IDE, enable "include dependencies with provided scope" in the run configuration, or add a Maven profile that switches the scope to compile.

The streaming WordCount

Java
import org.apache.flink.api.common.typeinfo.Types;
import org.apache.flink.api.java.tuple.Tuple2;
import org.apache.flink.streaming.api.datastream.DataStream;
import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;

public class WordCount {
    public static void main(String[] args) throws Exception {
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        DataStream<String> lines = env.fromData(
            "to be or not to be",
            "that is the question");

        lines
            .flatMap((String line, org.apache.flink.util.Collector<Tuple2<String, Integer>> out) -> {
                for (String word : line.split(" ")) {
                    out.collect(Tuple2.of(word, 1));
                }
            })
            .returns(Types.TUPLE(Types.STRING, Types.INT))
            .keyBy(pair -> pair.f0)
            .sum(1)
            .print();

        env.execute("Streaming WordCount");
    }
}

Line by line:

  1. getExecutionEnvironment() returns a local environment inside an IDE and the cluster environment when submitted with flink run.
  2. fromData creates a small bounded source for tests. Real jobs use Kafka, files or other connectors (see the connectors chapter).
  3. flatMap splits each line into words, emitting (word, 1) pairs.
  4. .returns(...) is needed with lambdas because Java erases generic types. See Map, Filter and FlatMap.
  5. keyBy groups all pairs with the same word onto the same parallel instance.
  6. sum(1) keeps a running sum of tuple field 1 per key and emits the new total for every incoming pair.
  7. print() writes each result to stdout.
  8. Nothing runs until env.execute(...) is called. Everything before it only builds the dataflow graph.

Package with mvn clean package and submit: ./bin/flink run target/my-job.jar. A possible output ordering looks like (to,1), (be,1), ... (to,2), (be,2): each word's count grows, and lines from different keys may interleave differently on every run when parallelism is above 1.

The same logic as a stand-in

Java Stand-in: streaming WordCount logic

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Notice the output: like Flink, the stand-in emits an updated total for every incoming word, so to and be appear twice with growing numbers. A batch word count would print each word once.

Seeing the job graph

You can ask the environment for the plan before running:

Java
System.out.println(env.getExecutionPlan());   // JSON, paste into the plan visualizer

Or open the job in the web UI. Each box in the graph is a chained task; the number is its parallelism. Setting env.setParallelism(2) or passing -p 2 to flink run creates two parallel subtasks of each operator, which needs two free slots.

Java Stand-in: two parallel subtasks, keyed by hash

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Each word always lands on the same subtask, so each subtask owns a disjoint part of the state. Real Flink routes keys through key groups, explained in Parallelism, keyBy and Key Groups.

Common beginner mistakes

  • Forgetting env.execute(). The program exits immediately and nothing happens.
  • Missing type information with lambdas (an InvalidTypesException): add .returns(...) or use an anonymous or named function class.
  • Looking for print() output in the terminal when submitted to a cluster. It goes to the TaskManager's .out file.
  • Not enough slots. The job stays in Scheduling if parallelism exceeds free slots.
  • Packaging Flink into the JAR (scope not provided) causing class conflicts on the cluster.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Where does the output of print() appear for a job submitted to a cluster?

    print() runs inside a TaskManager, so its output goes to that TaskManager's .out file, visible in the web UI.
  2. 2. What happens if you build a DataStream pipeline but never call env.execute()?

    Transformations are lazy. execute() submits the built graph.
  3. 3. Why is the Flink dependency usually declared with scope provided?

    Bundling Flink into your JAR duplicates classes and can cause version conflicts on the cluster.
  4. 4. What does sum(1) on a keyed stream of Tuple2 do?

    Rolling aggregations emit an updated result for each incoming element of the key.

Summary

  • Install Java 17+, unpack Flink, run ./bin/start-cluster.sh, and open the UI at port 8081.
  • A job is a JAR built with the Flink dependencies in provided scope and submitted with flink run.
  • Build the graph with transformations, then call env.execute(...) to run it.
  • Rolling aggregations emit an updated value for every event; print() writes to the TaskManager's stdout.

Next: The Dataflow Model and the Job Graph.