Skip to content
Tutorials Center
My learning

Topics

Lesson 10 of 37 ยท DataStream API

DataStream Basics: Environment, Sources and Sinks

Learn the structure of every Flink DataStream program: the execution environment, creating sources, data types and TypeInformation, sinks, and running with execute.

The DataStream API is Flink's core Java API for streams. You describe what should happen to each event with a chain of transformations, and Flink distributes the execution. Every DataStream program, whether it has five lines or five thousand, has the same four-part skeleton, and this lesson walks through it: environment, source, transformations, sink.

Code in this lesson is real Flink code shown read-only, followed by runnable stand-ins of the same structure. It builds on Backpressure and Flow Control.

The skeleton of a job

Java
public class Pipeline {
    public static void main(String[] args) throws Exception {
        // 1. environment
        StreamExecutionEnvironment env = StreamExecutionEnvironment.getExecutionEnvironment();

        // 2. source
        DataStream<String> lines = env.fromData("alpha", "beta", "gamma");

        // 3. transformations
        DataStream<String> upper = lines.map(String::toUpperCase);

        // 4. sink
        upper.print();

        // 5. run
        env.execute("my first pipeline");
    }
}

The execution environment

StreamExecutionEnvironment is the entry point. Where you run decides what you get:

  • getExecutionEnvironment() picks automatically: a local mini cluster inside your IDE or a normal main run, and the cluster when submitted with flink run.
  • createLocalEnvironmentWithWebUI(new Configuration()) runs locally with the web UI (needs flink-runtime-web), handy for exploring.
  • getExecutionEnvironment(configuration) lets you pass configuration, such as pipeline.name or execution.runtime-mode.

Typical settings:

Java
env.setParallelism(4);
env.setMaxParallelism(128);
env.setRuntimeMode(RuntimeExecutionMode.STREAMING);
env.enableCheckpointing(30_000);

Creating sources

Sources create the first DataStream. The choices, from simple to production:

Java
// a few fixed elements, bounded: tests and demos
DataStream<Integer> numbers = env.fromData(1, 2, 3, 4, 5);

// from a collection
DataStream<String> fromList = env.fromData(List.of("a", "b", "c"));

// a generator for load tests (flink-connector-datagen)
DataGeneratorSource<String> gen = new DataGeneratorSource<>(
    index -> "event-" + index, 1_000, RateLimiterStrategy.perSecond(10), Types.STRING);
DataStream<String> generated = env.fromSource(gen, WatermarkStrategy.noWatermarks(), "generator");

// a real connector, unbounded
KafkaSource<String> kafka = KafkaSource.<String>builder()
    .setBootstrapServers("broker:9092")
    .setTopics("clicks")
    .setGroupId("clicks-job")
    .setStartingOffsets(OffsetsInitializer.earliest())
    .setValueOnlyDeserializer(new SimpleStringSchema())
    .build();
DataStream<String> clicks = env.fromSource(kafka, WatermarkStrategy.noWatermarks(), "clicks");

fromSource takes a source, a watermark strategy (see Watermarks; noWatermarks() means no event time yet) and a name that shows up in the web UI. The older addSource(SourceFunction) API is removed in Flink 2.x; every new source implements the unified Source interface.

Data types and TypeInformation

Flink must know the type of every record to serialize it efficiently and to give the right state serializers. It supports:

  • Java primitives and boxed types, String, BigDecimal, dates.
  • Tuples: Tuple2<String, Integer> up to Tuple25, with fields f0, f1, ... (compact, positional, awkward to read).
  • POJOs and Java records: public class with a no-argument constructor and public fields or getters/setters (records are supported and are the nicest choice).
  • Arrays, List, Map, Row (used by the Table API), Avro and Protobuf classes.

Types that Flink cannot analyse fall back to Kryo (slow and bad for state evolution). You can see the fallback in logs as "class X cannot be used as a POJO type". With lambdas, Java's erasure hides generics, so you tell Flink the output type:

Java
DataStream<Tuple2<String, Integer>> pairs = words
    .map(w -> Tuple2.of(w, 1))
    .returns(Types.TUPLE(Types.STRING, Types.INT));

A record makes this unnecessary in most cases:

Java
public record Click(String userId, String page, long timestamp) {}
DataStream<Click> clicks = lines.map(Click::parse);   // type known from the method reference

Sinks

A sink is the last node. A few common ones:

Java
stream.print();                              // stdout of the TaskManager (debugging)
stream.printToErr();

stream.sinkTo(KafkaSink.<String>builder()    // write to Kafka
    .setBootstrapServers("broker:9092")
    .setRecordSerializer(KafkaRecordSerializationSchema.builder()
        .setTopic("results")
        .setValueSerializationSchema(new SimpleStringSchema())
        .build())
    .build());

stream.sinkTo(FileSink                       // rolling files
    .forRowFormat(new Path("/out"), new SimpleStringEncoder<String>("UTF-8"))
    .build());

sinkTo takes the unified Sink interface; the old addSink(SinkFunction) is deprecated in 1.x and removed in 2.0.

Running the job

  • env.execute("name") submits the job and blocks until it finishes (never for unbounded jobs).
  • env.executeAsync("name") submits and returns a JobClient immediately, for programs that manage several jobs or want to poll the status.
  • Everything between the creation of the environment and execute only builds the graph. If your main has an exception before execute, nothing was submitted. If code after execute matters, remember that for a streaming job it will not run until the job ends.

Runnable stand-in: a tiny environment

To see the "build first, run later" principle, the stand-in below implements a miniature pipeline builder. Transformations are only registered; execute() runs them.

Java Stand-in: build a pipeline lazily, run on execute()

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

The "plan built" line prints before any data is touched: exactly how Flink treats your chain of transformations.

Runnable stand-in: types matter

Flink cares about types because serialization and state depend on them. The stand-in shows why a tuple and a record carry the same data but differ in readability.

Java Stand-in: tuple fields versus a named record

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Common mistakes

  • Printing results in a cluster and looking in the wrong place (see the first-job lesson).
  • Reading from fromData and expecting it to run forever: it is bounded and the job finishes.
  • Forgetting .returns(...) for lambdas that produce generic types.
  • Using tuples everywhere; records or POJOs make code easier to read and keep state compatible when you add a field.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does getExecutionEnvironment() return when the program is submitted with flink run?

    It adapts: local in an IDE, cluster environment when submitted.
  2. 2. What is the third argument of env.fromSource(source, strategy, name)?

    It is just a name for the operator.
  3. 3. Why do lambdas that return generic types often need .returns(...)?

    Type erasure hides generics from Flink's type extraction.
  4. 4. What does env.executeAsync() do differently from env.execute()?

    executeAsync returns immediately with a handle to the running job.

Summary

  • A DataStream program has an environment, sources, transformations, sinks and an execute call.
  • getExecutionEnvironment() adapts to local runs and cluster submission.
  • Sources use fromData for tests, DataGeneratorSource for load and real connectors like Kafka in production, via fromSource and sinkTo.
  • Prefer records or POJOs over tuples, and give Flink type information for lambdas.

Next: Map, Filter and FlatMap.