Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 76 ยท What Data Science Is

Data Processing: Batch, Real-time and Online

Learn the stages of data processing and the main types: batch, real-time (stream), online (interactive) and offline processing, with trade-offs and runnable pandas examples.

Raw data does not arrive ready for decisions. Something has to collect it, clean it, combine it, summarise it and deliver the result to a person or a program. That chain of steps is data processing, and the way you run it decides how fresh the answers are, what the system costs and how it fails.

This lesson describes the stages every pipeline shares and the main processing styles: batch, real-time (stream), online (interactive) and offline. You will see each one in a small simulation and learn how to choose. It fits between the high-level view in The Data Science Workflow and the practical habits of Reproducible Projects.

The stages of data processing

Whatever the style, the stages are the same.

flowchart LR
    A["Collect<br/>apps, sensors, files, APIs"] --> B["Prepare<br/>clean, validate, deduplicate"]
    B --> C["Transform<br/>join, aggregate, encode"]
    C --> D["Store<br/>warehouse, lake, database"]
    D --> E["Analyse<br/>SQL, statistics, models"]
    E --> F["Deliver<br/>report, dashboard, alert, API"]
    F -.->|"feedback"| A

The processing cycle. In production the last step often feeds new data back into the first.

  1. Collect: gather data from databases, files, logs, sensors, forms or APIs; see Data Sources.
  2. Prepare: validate, remove duplicates, fix types and handle missing values (Data Quality Audit).
  3. Transform: join tables, aggregate, create features, encode categories.
  4. Store: write to a database, data warehouse or file store so others can query it.
  5. Analyse: run queries, statistics or models.
  6. Deliver: present a report, send an alert, or serve a prediction through an API.

Batch processing

In batch processing, data is collected over a period, then processed all at once on a schedule: every night, every hour, every month. Think of payroll, monthly invoices, a nightly sales report or retraining a model each week.

  • Strengths: simple, efficient for large volumes, easy to rerun and test, cheap because jobs can run when resources are idle.
  • Weaknesses: results are as old as the last run; an error is found only when the batch finishes.
  • Typical tools: cron jobs, SQL scripts, Apache Spark, Airflow, pandas scripts.
Python A batch job: yesterday's orders into a daily report

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The whole day arrives as one table and the result is one tidy summary. If you found a bug tomorrow, you could rerun the same job over the same input and get the corrected answer. That repeatability is the main advantage of batch.

Real-time (stream) processing

In stream or real-time processing, each record is handled as it arrives, within milliseconds to seconds. Examples: flagging a stolen card at the moment of payment, a ride app updating a driver's position, a smoke alarm, live traffic maps.

  • Strengths: low latency, can trigger actions immediately.
  • Weaknesses: harder to build and debug; must cope with late, duplicated and out-of-order events; always running, so more costly.
  • Typical tools: Apache Kafka, Apache Flink, Spark Structured Streaming, cloud streaming services.

Stream processors rarely look at one event alone. They keep a small state (a running total, the last 5 minutes of events) and update it with every new record.

Python A stream: rolling window and an alert on each event

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each event is processed in constant time, the state is small, and the alert fires at the moment the suspicious transaction arrives, not the next morning. Notice that at t=70 the early events of card A have left the window. Real systems must also decide what to do with an event that arrives late (an event-time versus processing-time problem).

Online (interactive) processing

Online processing, often called OLTP (online transaction processing) or interactive processing, answers a user's request immediately and keeps the data consistent: booking a seat, transferring money, saving a form. Each request is small, there are many of them at once, and correctness matters more than complex analysis. This is the world of operational databases.

It is easy to confuse with "online learning". In machine learning, online learning means updating a model one example or a small batch at a time instead of retraining on all data, which suits streams and huge datasets.

Python Online learning: update a running estimate one event at a time

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The running mean needs only the count and the current mean, and after the last event it equals the batch answer. Many online algorithms (online gradient descent, running variance, count-min sketches) follow the same pattern: a small state, updated per event.

Offline processing

Offline is the opposite of online in a different sense: work that does not need an immediate response and may not even need a connection to the live system. Examples are training a model on last month's data, a monthly financial audit, or analysis on a laptop with an exported file. In practice, batch and offline overlap. Offline analysis is where data scientists explore and train; online systems then serve what was learned.

Choosing a style

Question Batch Real-time (stream) Online (interactive)
How fresh must results be? Hours to days Milliseconds to seconds Instant, per request
Data volume per run Very large Continuous flow Small per request
Complexity to build Low High Medium
Typical use Reports, payroll, model training Fraud alerts, live dashboards Bookings, payments, apps
Reprocessing after a bug Easy Hard Not applicable

Guidance:

  • Start with batch. If a daily report answers the question, a stream is over-engineering.
  • Use streaming only when a delay costs something: money, safety or user experience.
  • Hybrid designs are common. A "lambda" or "kappa" architecture keeps a fast stream for fresh approximations and a batch job to recompute the exact answers later.
  • Decide how to handle bad data in every style: invalid, late or duplicate records need a rule, and the rule should be tested; see Testing Data Code.

Ask "what is the cost of being one hour late?" If the answer is "nothing", batch is enough. If it is "we lose the sale or the customer", consider real-time.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A bank must flag a suspicious card payment before it is approved. Which processing style fits best?

    The decision must happen within milliseconds of the event, so each event has to be processed as it arrives.
  2. 2. What is the main advantage of batch processing?

    Batch jobs run on a schedule over a fixed input, which makes them simple, cheap and repeatable. The cost is freshness.
  3. 3. In the pipeline stages, which step comes right after validating and removing duplicates?

    Prepare is followed by transform: joining, aggregating and encoding, before the data is stored and analysed.
  4. 4. What does online learning mean in machine learning?

    Online learning updates the model incrementally as new data arrives, instead of retraining on the full history each time.

Summary

  • Data processing is a chain: collect, prepare, transform, store, analyse, deliver.
  • Batch processing works on accumulated data on a schedule: simple, cheap and repeatable, but not fresh.
  • Stream (real-time) processing handles each event as it arrives with small state, at the price of complexity.
  • Online processing answers interactive requests; online learning updates a model incrementally.
  • Offline work (training, audits, exploration) needs no immediate response.
  • Start with the simplest style that meets the freshness requirement, and decide how bad data is handled in each.

Next: Framing the Question.