Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 47 ยท Data Ingestion and Processing Models

Getting Data In: Sqoop, Flume, NiFi and Kafka

Learn batch and streaming data ingestion for Hadoop: Sqoop imports, Flume agents, NiFi flows, Kafka topics, change data capture and delivery guarantees.

A cluster is only useful once data is inside it. Data lives in relational databases, application logs, message queues, sensors and files on other servers, and ingestion is the work of moving it into HDFS or another store reliably, repeatedly and without losing or duplicating records.

This lesson separates batch from streaming ingestion, introduces the classic tools (Sqoop, Flume, NiFi, Kafka), explains change data capture and delivery guarantees, and ends with a runnable pipeline stand-in. It follows Choosing a NoSQL Store.

None of these tools run on the site. The commands and configuration files are read-only and written to show the real shape. The runnable examples are Python stand-ins for the ideas: incremental import, a log pipeline and at-least-once delivery.

Batch or streaming?

Batch ingestion moves a bounded chunk of data at a scheduled time: "copy yesterday's orders every night at 02:00". It is simple, efficient for large volumes, and easy to retry, but the data is only as fresh as the last run.

Streaming ingestion moves each record shortly after it is created. Freshness is seconds, but you must handle ordering, bursts, back-pressure and restarts all the time.

Pick the ingestion style

Batch Scheduled copies. Simple and cheap.
Streaming Continuous flow. Fresh but harder.
CDC Stream the changes of a database.

Many platforms use all three side by side.

1 / 2

Sqoop: relational databases to HDFS

Apache Sqoop ("SQL to Hadoop") copies tables between a relational database and HDFS or Hive. It turns an import into a MapReduce job in which every map task reads one slice of the table through JDBC, so the copy runs in parallel. The slices come from a split column, normally the primary key: Sqoop reads its minimum and maximum and divides the range among the mappers.

Terminal
sqoop import \
  --connect jdbc:mysql://db.example.com/shop \
  --username etl --password-file /user/etl/.pw \
  --table orders \
  --target-dir /data/raw/orders \
  --split-by order_id \
  --num-mappers 4 \
  --as-parquetfile

For tables that grow, you do not want to reload everything. An incremental import only fetches rows beyond a stored high-water mark:

Terminal
sqoop import --connect jdbc:mysql://db.example.com/shop --table orders \
  --incremental append --check-column order_id --last-value 120000 \
  --target-dir /data/raw/orders

Sqoop was retired to the Apache Attic in 2021. The idea lives on in Spark JDBC reads, NiFi, and CDC tools, so it is still the clearest way to learn it. Watch out for load on the production database: four mappers means four concurrent queries.

The first stand-in shows the logic: split a key range for parallel readers, then import only new rows by remembering a high-water mark.

Python Parallel split and incremental import

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that an incremental import based on order_id > last_value finds new rows only. It misses updates and deletes of old rows, which is the main reason for change data capture.

Change data capture (CDC)

Change data capture reads the database's own transaction log (the MySQL binlog, the PostgreSQL write-ahead log) and emits every insert, update and delete as an event with the old and new values. Debezium is a well-known open-source tool for this, and it publishes the events to Kafka. Compared with repeated SELECT queries, CDC:

  • puts almost no query load on the source,
  • captures updates and deletes,
  • keeps the exact order of changes,
  • gives you a history you can replay.

A change event is typically a small JSON document with the operation (c create, u update, d delete), the row before and the row after, and a timestamp or log position. Downstream jobs apply these events to a table, a process often called a merge or upsert.

Flume: log events to HDFS

Apache Flume collects log-style events. An agent is a small JVM process wired as source โ†’ channel โ†’ sink. The source receives events (tailing a file, listening on a port), the channel buffers them (memory, or file for durability), and the sink writes them somewhere, for instance HDFS.

PROPERTIES
a1.sources  = r1
a1.channels = c1
a1.sinks    = k1

a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /var/log/web/access.log
a1.sources.r1.channels = c1

a1.channels.c1.type = file
a1.channels.c1.capacity = 100000

a1.sinks.k1.type = hdfs
a1.sinks.k1.channel = c1
a1.sinks.k1.hdfs.path = /data/raw/weblogs/%Y/%m/%d
a1.sinks.k1.hdfs.rollInterval = 300
a1.sinks.k1.hdfs.fileType = DataStream

Flume transfers events from channel to sink in transactions: an event leaves the channel only after the sink confirms it. That gives at-least-once delivery. The rollInterval matters because HDFS files are written once; rolling files every few minutes avoids both endless open files and the small-files problem if rolling is too frequent. Today, new designs mostly use Kafka or Filebeat plus Kafka instead of Flume.

NiFi: visual data flows

Apache NiFi is a web-based tool where you drag processors onto a canvas and connect them. Each connection is a queue with back-pressure, and every piece of data (a FlowFile) carries attributes and provenance, a full audit trail of where it came from and what changed it. NiFi is strong for routing, light transformations and bringing in many different protocols (SFTP, HTTP, databases, Kafka) without writing code. It is not meant for heavy computation: it moves and shapes data, and the cluster does the processing.

Kafka: the durable log

Apache Kafka is a distributed, replicated, append-only commit log. Producers append messages to a topic, which is split into partitions. Within one partition, every message has a sequential offset, and the order is guaranteed. Consumers read at their own pace and remember their offset, so many independent consumers can read the same data, and a consumer can rewind to replay history.

  • Messages with the same key go to the same partition (hash of the key), so per-key order is kept. This is the same hash idea as consistent partitioning.
  • Each partition is replicated to several brokers, and one is the leader.
  • A consumer group shares the partitions: each partition is read by exactly one consumer of the group, so adding consumers (up to the partition count) scales reading.
  • Kafka Connect runs ready-made connectors: a source connector (database, files) or a sink connector (HDFS, S3, a search index).
Terminal
kafka-topics.sh --create --topic clicks --partitions 6 --replication-factor 3 \
  --bootstrap-server broker1:9092
kafka-console-producer.sh --topic clicks --bootstrap-server broker1:9092
kafka-console-consumer.sh --topic clicks --from-beginning \
  --bootstrap-server broker1:9092

Kafka is the common buffer between producers and consumers: if the HDFS sink is down for an hour, the log keeps the messages and the sink catches up from its stored offset.

flowchart LR
    A["App logs"] --> K["Kafka topic<br/>(partitions)"]
    D["Database<br/>CDC"] --> K
    K --> H["HDFS / object store<br/>(Connect sink)"]
    K --> S["Stream processor"]
    K --> X["Search index"]

Kafka decouples many producers from many consumers.

Delivery guarantees

Any pipeline may crash between "write the record" and "remember that it was written". That is why you must know the guarantee you get:

  • At-most-once: never duplicate, may lose data.
  • At-least-once: never lose, may duplicate (the usual default: retry until acknowledged).
  • Exactly-once: the effect appears once. In practice this is built from at-least-once delivery plus idempotent writes (writing the same record twice has the same result as once, for example an upsert by key) or transactions.

The second stand-in is a small pipeline: a source reads log lines, a sink writes them, and a crash happens in the middle. At-least-once delivery re-sends from the last committed offset, and an idempotent sink removes the duplicates.

Python A source-to-sink pipeline with a crash and recovery

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Without idempotence, the lines written before the crash are written again, giving 10 rows for 8 source lines. With an idempotent sink (keyed by offset), the duplicates are dropped and exactly 8 rows remain. This is the practical recipe for "effectively exactly-once".

Which tool when?

Need Reasonable choice
Nightly copy of tables Spark JDBC read, NiFi, or a managed ETL tool (Sqoop in older clusters)
Every change of a database, with deletes CDC (Debezium) into Kafka
Application events and logs at scale Kafka, with Connect sinks to HDFS or object storage
Many file and protocol sources, light routing, audit trail NiFi
Existing Flume agents on servers Keep them, plan to migrate to Kafka

Whatever you pick, land data first in a raw zone unchanged, as described in Data Lakes, Warehouses and Lakehouses, so you can always reprocess.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does an incremental Sqoop import with --check-column and --last-value fetch?

    It uses a high-water mark. It finds new rows but cannot see updates of old rows or deletes, which is a reason to use CDC.
  2. 2. Why is CDC better than repeated SELECT queries for keeping a copy in sync?

    Log-based CDC captures every committed change in order and does not run heavy queries against the tables.
  3. 3. In Kafka, what guarantees message order?

    Each partition is an ordered log with sequential offsets. Using the same key for related messages sends them to the same partition.
  4. 4. How do you get an effectively exactly-once result from at-least-once delivery?

    Retries may duplicate a record. If the sink upserts by key or offset, the duplicate changes nothing.

Summary

  • Batch ingestion is scheduled and simple; streaming is continuous and fresher; CDC streams the changes recorded in a database log.
  • Sqoop copied tables in parallel using a split column and incremental high-water marks; Flume moved logs with source, channel and sink; NiFi builds visual flows with provenance; Kafka is the durable, partitioned log that decouples producers from consumers.
  • Retries give at-least-once delivery; combine them with idempotent writes for effectively exactly-once results.
  • Land raw data unchanged first so you can always reprocess it.

Next: Batch vs Stream Processing.