Skip to content
Tutorials Center
My learning

Topics

Lesson 1 of 47 ยท Big Data Basics

What Is Big Data?

Understand what makes data big: volume, velocity and variety, why one machine hits a wall, scale up versus scale out, and the workloads that need a cluster.

"Big data" sounds like a marketing phrase, and sometimes it is. But behind it sits a very practical engineering problem: at some point your data no longer fits on one computer, or it arrives faster than one computer can process it, or it is so messy that your usual database cannot hold it. When that happens, the tools you know stop working and you need a different way to store and compute.

This tutorial teaches that different way, using Apache Hadoop and the ideas around it. In this first lesson you will learn a useful working definition of big data, see why a single machine is not enough, and compare the two ways of growing a system: bigger machines or more machines. You do not need any Hadoop knowledge, only basic Python for the examples.

Hadoop itself cannot run on this site: there is no cluster in your browser. Every real command and configuration in this tutorial is shown in read-only code blocks. The runnable TryIt examples are stand-ins: small Python programs that reproduce the idea (splitting data, moving blocks, shuffling keys) so you can see how it works inside. Each lesson tells you which parts are real and which are stand-ins.

A working definition: the three Vs

There is no official size at which data becomes "big". A more useful test is whether the data pushes you past what a single, ordinary machine and a single ordinary database can handle. Three properties usually cause that, and they are known as the three Vs.

Three reasons data gets big

Volume Too much data to store on one disk.
Velocity Data arrives faster than one server can digest it.
Variety Logs, JSON, images and tables mixed together.

Any one of the three can be enough to need a cluster.

1 / 2

Volume is the obvious one: terabytes and petabytes. A web company that logs every click can produce hundreds of gigabytes a day. After a year, nobody opens that file in a text editor.

Velocity is about speed. A payment system or a sensor network may emit tens of thousands of events per second. Even if each event is tiny, the stream never stops, and decisions (is this payment fraud?) are only useful if they come quickly.

Variety means the data does not look like neat rows. You may have web server logs, JSON from mobile apps, CSV exports from partners, images and free text. A traditional relational database wants a schema up front. Big data systems tend to store the raw files first and decide the structure later, which you will meet again as "schema-on-read" in Data Lakes.

Some authors add more Vs (veracity, value). They are fair questions to ask, but the first three are what drive the architecture.

Why one machine is not enough

A single server has four limits, and they do not grow together:

  • Disk capacity. Even at tens of terabytes per server, a petabyte needs many servers.
  • Disk speed. A good hard disk reads roughly 100 to 200 MB per second. Reading one terabyte from one disk takes more than an hour just to scan it, before doing any work.
  • Memory. RAM is fast but small compared with the data. Anything that does not fit has to be streamed from disk.
  • Failure. One machine means one point of failure. When it dies, everything stops, and the data may be lost.

The second point is the one people underestimate. The problem is often not "can I store it" but "can I read it in a reasonable time". The way out is to read from many disks at the same time.

Let us see that idea with a small stand-in. The program below creates a list of numbers, then "scans" it in two ways: all in one piece, and split into chunks as a cluster would split it, with each chunk summed separately and the partial results combined. The data is tiny and runs on one CPU, so this does not make anything faster in Python. What it demonstrates is that the answer is identical, which is the property that makes splitting work possible.

Python Split work, same answer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The important design rule is visible: each chunk is processed without looking at the others, and only the small partial results are combined at the end. Operations like sums, counts, maximums and filters work this way. Later, MapReduce turns this rule into a full programming model.

Scale up or scale out

When one machine is too small you can grow in two directions.

Scaling up (vertical scaling) means buying a bigger machine: more CPU cores, more RAM, bigger disks. It is simple, because your program does not change. But it has a ceiling, the price rises faster than the capacity (a server with ten times the memory costs far more than ten times as much), and it is still one point of failure.

Scaling out (horizontal scaling) means adding more ordinary machines and spreading the data and the work across them. There is almost no ceiling, because you can keep adding nodes, and a failed node is a nuisance rather than a disaster. The price is complexity: your software must split data, move code to the data, and survive failures. Hadoop exists to hide that complexity.

flowchart LR
  subgraph Up["Scale up"]
    A["One big server<br/>more RAM, more disks"]
  end
  subgraph Out["Scale out"]
    B1["Node 1"]
    B2["Node 2"]
    B3["Node 3"]
    B4["Node N"]
  end

Scaling up makes one box bigger; scaling out adds more boxes that share the job.

Let us put rough numbers on it. The next stand-in estimates how long a full scan takes if each disk reads 150 MB per second, as the number of disks working in parallel grows. This is plain arithmetic, not a measurement of a real cluster, and real clusters lose some time to coordination, but it shows why scale-out is attractive.

Python Scan time for 1 TB as disks are added

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

One disk needs a little over 111 minutes. A hundred disks reading at once need about one minute. That is the whole promise of big data systems: many cheap disks and CPUs working at the same time on different slices of the data.

Structured, semi-structured and unstructured data

The "variety" of big data means three broad kinds of data, and each is stored and processed differently:

Kind What it looks like Examples Typical storage and tools
Structured fixed columns and types, every row the same shape database tables, CSV exports, billing records relational databases, Hive tables, Parquet
Semi-structured self-describing, with tags or keys, but fields can vary or nest JSON, XML, logs with key=value pairs, Avro HDFS files, document stores, Hive with JSON or Avro readers
Unstructured no built-in data model for fields free text, images, audio, video, PDFs HDFS or object storage, then specialized processing (text mining, vision)

Most real organizations hold all three, and a large share of the data (often quoted as most of it) is semi-structured or unstructured. That is a main reason Hadoop uses schema-on-read (see Hadoop vs a Relational Database): you can land any of these kinds as files and decide how to interpret them later. Semi-structured formats such as JSON are flexible but cost more to parse and take more space than columnar formats, so pipelines usually convert them to Parquet or ORC after landing (see Row vs Column Formats).

Python Three kinds of data, three ways to read them

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Types of analytics

What you do with big data falls into several levels of ambition, each answering a different question:

  1. Descriptive analytics: what happened? Totals, averages, daily counts, dashboards. This is the most common and is mostly SQL aggregation.
  2. Diagnostic analytics: why did it happen? Drill down and compare segments to find causes (why did sales drop in one region?).
  3. Predictive analytics: what is likely to happen? Statistical models and machine learning estimate churn, demand or fraud risk from history.
  4. Prescriptive analytics: what should we do? Recommendations and optimization choose actions, such as the best price or the best delivery route.
  5. Real-time (streaming) analytics: what is happening right now? Events are analyzed within seconds to detect fraud or system failures (see Batch vs Stream Processing).
  6. Text analytics: what do the words say? Extracting topics, sentiment and entities from reviews, tickets and documents (the unstructured data above).

Descriptive and diagnostic questions look backward and are cheap; predictive and prescriptive ones need more data, more compute and more skill, but bring more business value. Hadoop and Spark support all of them, from Hive queries to machine learning.

More than three Vs

The three Vs explain why a cluster is needed. Practitioners often extend the list to six: veracity (how trustworthy and clean the data is: missing values, duplicates, sensor errors), value (whether the data is worth the cost of storing and analyzing) and variability (the meaning and shape of the data change over time, such as a word whose sentiment shifts with context or a source that changes format). Veracity matters as much as volume: ten billion wrong records are worse than ten million correct ones, which is why Security, Governance and Data Quality is part of this course.

Typical big data workloads

You will meet the same kinds of jobs again and again:

  • Log and clickstream analysis: counting page views, finding errors, building sessions (see the log analysis project).
  • ETL and data preparation: extracting data from many sources, cleaning it, and loading it into a warehouse.
  • Search and indexing: building an index over a huge document collection (search index project).
  • Recommendations and analytics: "customers who bought X also bought Y" (recommendations project).
  • Machine learning at scale: training on data that does not fit in memory.
  • Real-time monitoring: alerts from streams of events, covered in Batch vs Stream Processing.

Most of these are batch jobs that read a big dataset once, in order, and write a smaller result. That access pattern (large sequential reads, few random updates) shapes almost every design decision in the rest of this tutorial.

Do you really have big data?

Be honest before you build a cluster. A modern laptop can analyse a few gigabytes comfortably, and a single database server can handle hundreds of gigabytes. A cluster adds cost and operational work. A good rule of thumb: if the data fits on one machine's disk, and a single-machine tool finishes in an acceptable time, use the simple tool. Reach for distributed systems when data volume, growth, or required speed truly outgrows one machine.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which of these is a good reason to move from one server to a cluster?

    A cluster is justified by real limits of capacity or speed. It adds operational work, so popularity or fashion is not a reason.
  2. 2. What does 'velocity' describe?

    Velocity is the speed of incoming data. Volume is the amount, and variety is the mix of formats.
  3. 3. What is the main downside of scaling up (a bigger single server)?

    Bigger machines are simple to use but costly, limited in size, and when one dies everything stops.
  4. 4. Why does splitting a sum into chunks give the same result as one scan?

    Sum is an operation that can be computed per chunk and then merged. That independence is what lets many machines share the work.

Summary

  • Data is "big" when volume, velocity or variety outgrow one machine and one database.
  • A single machine is limited by disk size, disk speed, memory and failure.
  • Reading from many disks at once is the key to scanning huge data quickly.
  • Scale up means a bigger machine; scale out means more machines. Hadoop is built for scale out.
  • Typical workloads are mostly batch jobs with large sequential reads: logs, ETL, indexing, analytics.
  • Do not build a cluster before you really need one.

Next: Distributed Systems in Plain Words.