Skip to content
Tutorials Center
My learning

Topics

Lesson 28 of 37 ยท Connectors and Deployment

Connectors: Sources and Sinks

Understand Flink connectors: the unified Source API with splits and enumerators, KafkaSource and KafkaSink, FileSink, JDBC and CDC, and offsets that make replay possible.

A stream processor is only useful if data flows in and results flow out. Flink calls the entry points sources and the exits sinks, and ships (or hosts as separate projects) connectors for Kafka, files and object stores, JDBC databases, Elasticsearch/OpenSearch, Pulsar, Kinesis, CDC for MySQL and Postgres, and many more. The connector decides how data is read in parallel, how positions are remembered for checkpoints, and which delivery guarantees are possible.

This lesson explains the common model and shows the real Kafka and file connectors read-only; the runnable Java stand-ins reproduce the split and offset mechanics. It begins chapter 7 after SQL Windows, Joins and Top-N.

Real versus stand-in. No Kafka or Flink runs on this site. Connector configuration below is real and read-only; the Java programs model splits and offsets with in-memory lists.

A source has three parts

Split A slice of the input, like a partition.
Enumerator Hands out splits to readers.
Reader Reads one split in a parallel task.

This is the unified Source API.

1 / 2

The unified Source API

Modern Flink sources are built on one API with three roles. A split is a unit of work, for example a Kafka partition or a file (or part of one). The split enumerator runs once in the JobManager, discovers splits, and assigns them to source readers, which run in parallel tasks and read the records of the splits they hold. Reader progress (such as the offset inside each split) is part of the checkpoint, so after a failure each reader starts exactly where the last checkpoint says.

This design works for both bounded and unbounded input: a file source finishes when all splits are consumed, while a Kafka source keeps polling for new records and can discover new partitions while running.

The stand-in assigns "partitions" to readers round-robin, reads a few records at a time and remembers an offset per split, as the real components do.

Java Stand-in: enumerator, splits and readers with offsets

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

KafkaSource

The Kafka connector is the most common source. It reads topics (one split per partition), stores partition offsets in checkpoints, and can start from earliest, latest, a timestamp or committed offsets.

Java
KafkaSource<String> source = KafkaSource.<String>builder()
    .setBootstrapServers("broker:9092")
    .setTopics("clicks")
    .setGroupId("click-job")
    .setStartingOffsets(OffsetsInitializer.committedOffsets(OffsetResetStrategy.EARLIEST))
    .setValueOnlyDeserializer(new SimpleStringSchema())
    .build();

DataStream<String> clicks = env.fromSource(
    source,
    WatermarkStrategy.forBoundedOutOfOrderness(Duration.ofSeconds(10)),
    "Kafka clicks");

Notes worth remembering:

  • Flink does not rely on Kafka's consumer-group offsets for recovery. It relies on the offsets in its own checkpoints. The offsets committed to Kafka on checkpoint completion exist only for monitoring tools.
  • Parallelism above the partition count leaves some readers idle, which also stalls watermarks unless you configure an idleness timeout (withIdleness(Duration.ofMinutes(1)) on the watermark strategy).
  • Pass the WatermarkStrategy into fromSource so watermarks are generated per partition, inside the source.

Sinks: KafkaSink and FileSink

A sink writes results. Its delivery guarantee is a property of the connector configuration.

Java
KafkaSink<String> sink = KafkaSink.<String>builder()
    .setBootstrapServers("broker:9092")
    .setRecordSerializer(KafkaRecordSerializationSchema.builder()
        .setTopic("alerts")
        .setValueSerializationSchema(new SimpleStringSchema())
        .build())
    .setDeliveryGuarantee(DeliveryGuarantee.EXACTLY_ONCE)
    .setTransactionalIdPrefix("alerts-job")
    .build();

alerts.sinkTo(sink);

AT_LEAST_ONCE may write duplicates after a recovery; EXACTLY_ONCE uses Kafka transactions, committed when a checkpoint completes (the next lesson explains how). FileSink writes files into a folder structure such as dt=2026-11-12/hour=09/, with row formats (JSON lines, CSV) or bulk formats (Parquet, ORC, Avro):

Java
FileSink<String> files = FileSink
    .forRowFormat(new Path("s3://lake/clicks/"), new SimpleStringEncoder<String>("UTF-8"))
    .withBucketAssigner(new DateTimeBucketAssigner<>("yyyy-MM-dd--HH"))
    .withRollingPolicy(DefaultRollingPolicy.builder()
        .withRolloverInterval(Duration.ofMinutes(15))
        .withInactivityInterval(Duration.ofMinutes(5))
        .withMaxPartSize(MemorySize.ofMebiBytes(256))
        .build())
    .build();

Files being written are "in-progress", become "pending" when rolled, and turn "finished" only when a checkpoint completes. This is why FileSink needs checkpointing enabled to ever publish files.

Tables, CDC and other connectors

In SQL and the Table API connectors appear as 'connector' = '...' options (kafka, upsert-kafka, filesystem, jdbc, elasticsearch-7, datagen, print, blackhole). Three more families deserve a mention:

  • JDBC sources and sinks for relational databases; the sink can upsert by primary key. A JDBC lookup table supports enrichment joins with a cache.
  • CDC connectors (Flink CDC) read a database's change log, producing full inserts, updates and deletes, with an initial snapshot phase followed by streaming changes.
  • Custom connectors implement Source/Sink (DataStream) or DynamicTableSource/DynamicTableSink (SQL) and register through a factory.

When choosing a connector check four properties: whether it is bounded or unbounded, whether it can replay (resume at a saved position), whether the sink supports transactions or idempotent writes, and how it behaves when parallelism and partitions differ.

Java Stand-in: file sink states pending and finished

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Readers downstream only see finished files, which is how a file sink can offer exactly-once output even though a crash may leave half-written files behind.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a split in the Source API?

    The enumerator discovers splits and assigns them to parallel readers, which track progress inside each split.
  2. 2. How does Flink resume a Kafka source after a failure?

    Offsets live in the checkpoint, together with operator state, so the job rewinds consistently. Offsets committed to Kafka are for monitoring.
  3. 3. Why does FileSink need checkpointing enabled?

    Pending files are committed on checkpoint completion; without checkpoints they stay pending forever.
  4. 4. What goes wrong if source parallelism exceeds the number of Kafka partitions?

    An idle reader emits no watermark, and the minimum across readers holds event time back.

Summary

  • The unified Source API has splits, an enumerator and parallel readers; progress is saved in checkpoints.
  • KafkaSource reads partitions as splits and recovers from checkpointed offsets; pass the watermark strategy to fromSource.
  • KafkaSink and FileSink offer delivery guarantees; file output becomes visible only after a checkpoint.
  • SQL connectors include kafka, upsert-kafka, filesystem, jdbc, CDC and the learning helpers datagen, print and blackhole.
  • Choose connectors by boundedness, replay ability and sink transaction support.

Next: Exactly-Once Sinks and Two-Phase Commit