Skip to content
Tutorials Center
My learning

Topics

Lesson 18 of 39 ยท Data Formats and the Ecosystem

Kafka Connect

Learn Kafka Connect: source and sink connectors, workers, tasks, converters, single message transforms, offsets and the REST API, with a runnable pipeline stand-in.

Most of the code people write around Kafka is plumbing: copy rows from a database into a topic, copy a topic into a search index or a data warehouse, and do it reliably, with retries, restarts and no lost or repeated rows. Kafka Connect is the framework that does this plumbing so you configure it instead of programming it. You give it a JSON description of a connector, and a cluster of Connect workers runs it.

Connect is part of the Apache Kafka distribution. The connectors themselves (for databases, object stores, search engines, message queues) are plugins from the community and vendors, so this lesson uses a fictional connector class name to stay product-neutral. It follows Schemas, Avro, Protobuf and JSON.

Real versus stand-in. The REST calls and JSON configuration below are real in shape and shown read-only: the site cannot run a Connect cluster. The TryIt blocks are plain JavaScript stand-ins that copy the moving parts (a source that remembers an offset, a transform, a converter and a sink) in one process.

Two directions

Source connector External system into Kafka.
Sink connector Kafka out to an external system.
1 / 3

Workers, connectors and tasks

A connector is the named job: "copy the customers table into topic customers". It is only a coordinator. When you submit it, the connector decides how to split the work into tasks, up to tasks.max. For example, a database source could give each task a subset of tables, and a sink gives each task a share of the topic's partitions. Tasks are the units that actually read and write.

Tasks run inside workers, plain Java processes you start with the Kafka Connect scripts. There are two modes:

  • Standalone: one worker, configuration in files, offsets stored in a local file. Handy for development.
  • Distributed: many workers form a cluster through a group.id. Connectors and tasks are balanced across them, and if a worker dies its tasks move to the others. Configuration, offsets and status live in three internal Kafka topics (config.storage.topic, offset.storage.topic, status.storage.topic), so no worker holds unique state.

Configuring a connector over REST

In distributed mode you manage connectors through a REST API on each worker (port 8083 by default). Here is a made-up source connector:

JSON
{
  "name": "customers-source",
  "config": {
    "connector.class": "com.example.connect.CustomersDbSourceConnector",
    "tasks.max": "2",
    "connection.url": "jdbc:example://db.internal/shop",
    "tables": "customers",
    "topic.prefix": "shop-",
    "key.converter": "org.apache.kafka.connect.storage.StringConverter",
    "value.converter": "org.apache.kafka.connect.json.JsonConverter",
    "value.converter.schemas.enable": "false",
    "transforms": "mask",
    "transforms.mask.type": "org.apache.kafka.connect.transforms.MaskField$Value",
    "transforms.mask.fields": "email"
  }
}

You create it with POST /connectors (or PUT /connectors/<name>/config), and inspect and control it with a few more calls:

TEXT
GET    /connectors                      list connector names
GET    /connectors/customers-source/status
PUT    /connectors/customers-source/pause
PUT    /connectors/customers-source/resume
POST   /connectors/customers-source/restart
DELETE /connectors/customers-source

Status shows the connector and each task as RUNNING, PAUSED, FAILED and so on. A failed task is not restarted automatically; you look at its error and restart it.

Converters

Connect keeps its own internal data model so connectors do not care about wire formats. A converter translates between that model and the bytes in Kafka, and it is configured separately for keys and values. Common choices are StringConverter, JsonConverter (with schemas.enable choosing whether each message embeds its schema) and Avro, Protobuf or JSON Schema converters backed by a schema registry. Because converters are configuration, the same connector can write Avro on one cluster and JSON on another without a code change.

Single message transforms

A single message transform (SMT) edits each record as it passes through, one record at a time, without touching other records. Kafka ships a set: InsertField (add a field such as a timestamp or static value), ReplaceField (rename, include or exclude fields), MaskField (blank out a value), TimestampConverter, RegexRouter (change the topic name) and more, and you can chain several in order. They fit small fixes like masking an email. Anything that needs joins, aggregations or several records at once belongs in a stream processor (see Kafka Streams and Stream Processing).

Offsets in Connect

Connect must resume after a restart without skipping or repeating data, so it tracks positions, but differently per direction:

  • A source task attaches a source partition and source offset to each record it produces (for a database table: perhaps the table name and the highest row id or log position copied). Connect writes these offsets to the offsets topic after the records are accepted by Kafka. On restart the task asks Connect for the last offset and continues from there. This is at-least-once by default; newer versions can support exactly-once source connectors if the connector and the worker are configured for it (exactly.once.source.support).
  • A sink task is simply a consumer. Connect manages a consumer group named connect-<connector name>, and commits offsets after the task's put and flush complete. You inspect lag with the normal consumer-group tools (see Offsets and Commits).

For problem records, sink connectors can use errors.tolerance=all with errors.deadletterqueue.topic.name to route bad records to a dead letter topic instead of failing the task (see Retries, Dead Letter Topics and Poison Messages).

A source-to-sink pipeline you can run

The stand-in below puts the pieces into one program: a source that reads rows with id greater than its stored offset, two transforms (mask the email, tag the origin), a JSON "converter" and a sink that upserts into a "warehouse". Watch the offsets let the pipeline pick up exactly where it left off.

JS Source, SMT, converter, sink

Press Run (or Ctrl + Enter) to see the console output here.

No output.

Nothing was copied twice, even though the source ran in three separate bursts. In real Connect, the same effect comes from the persisted offsets, with the at-least-once caveat that a crash between "records accepted" and "offset saved" can replay a batch.

Tasks and failover

How do tasks end up on workers? Connect balances them over the live workers, and rebalances whenever membership changes. This stand-in assigns six tasks round-robin and then removes a worker.

JS Task assignment and failover

Press Run (or Ctrl + Enter) to see the console output here.

No output.

The real assignor tries to keep as many tasks as possible where they are, and recent Connect versions use incremental cooperative rebalancing so that unaffected tasks keep running. The stand-in only shows the end result.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the job of a connector, as opposed to a task?

    The connector holds configuration and plans the work. Tasks run in workers and do the actual reading or writing.
  2. 2. Where does a sink connector keep its position?

    A sink task is a consumer, so Connect commits its offsets in a normal consumer group, which makes lag easy to inspect.
  3. 3. Which is the right job for a single message transform?

    SMTs edit one record at a time. Joins and aggregations need a stream processor.
  4. 4. In distributed mode, what happens if a worker crashes?

    Configuration, offsets and status are in Kafka topics, so another worker can pick up the tasks.

Summary

  • Connect moves data between Kafka and other systems through configuration: source connectors write into Kafka, sink connectors read out.
  • Connectors plan the work, tasks do it, and workers run the tasks; distributed mode shares state in internal topics and survives worker failures.
  • Converters decide the wire format; SMTs make small per-record edits.
  • Source offsets are stored by Connect; sink offsets are an ordinary consumer group.
  • Manage connectors through the REST API and route bad sink records to a dead letter topic.

Next, process the data itself in Kafka Streams and Stream Processing.