Skip to content
Tutorials Center
My learning

Topics

Hadoop and Big Data Tutorial

Learn Hadoop and Big Data step by step: 47 lessons in 9 chapters with examples, quizzes, diagrams and projects.

Start learning →

of lessons completed

Big Data Basics

  1. 1 What Is Big Data? Understand what makes data big: volume, velocity and variety, why one machine hits a wall, scale up versus scale out, and the workloads that need a cluster.
  2. 2 Distributed Systems in Plain Words Learn the ideas under every big data tool: nodes and networks, partial failure, replication, consistency, the CAP idea and why commodity hardware works.
  3. 3 Hadoop vs a Relational Database Compare Hadoop with a relational database: schema-on-read vs schema-on-write, scaling, integrity and ACID, workloads and how to choose between them.
  4. 4 The Hadoop Ecosystem at a Glance A map of the Hadoop ecosystem: HDFS, YARN, MapReduce, Hive, Pig, HBase, Spark, Kafka and ZooKeeper, what each tool does and how they fit together.
  5. 5 Hadoop: History, Role Today and Alternatives Where Hadoop came from, from Google's papers to Yahoo and the vendors, why cloud object stores replaced much of HDFS, and when Hadoop is still the right choice.
  6. 6 Hadoop 2 vs Hadoop 3 Compare Hadoop 2 and Hadoop 3: erasure coding, multiple standby NameNodes, Java 8 baseline, intra-DataNode balancer, new ports, YARN features and cloud connectors.
  7. 7 Installing Hadoop: Modes, Config Files and Daemons Understand Hadoop's three modes, the core-site, hdfs-site, yarn-site and mapred-site files, formatting the NameNode, starting daemons and web UIs, on Linux and Windows.

HDFS

  1. 8 HDFS Architecture Learn how HDFS stores huge files: the NameNode and its metadata, DataNodes, large blocks, the write-once model and why a 128 MB block makes sense.
  2. 9 Replication and Block Placement How HDFS keeps data safe: the replication factor, rack awareness, the default placement policy, re-replication after failures and an introduction to erasure coding.
  3. 10 The HDFS Read and Write Path Follow a file through HDFS: how a client asks the NameNode, streams blocks down a DataNode pipeline, verifies checksums and how heartbeats keep metadata fresh.
  4. 11 Working with HDFS from the Shell Learn the everyday HDFS shell commands: ls, put, get, cat, rm, permissions, quotas, fsck and distcp, with a runnable Python mini file system to practice on.
  5. 12 Filesystems Hadoop Can Use Learn Hadoop's FileSystem abstraction and its implementations: local, HDFS, WebHDFS, S3A, ViewFs, Azure and Google Cloud connectors, and how URI schemes pick one.
  6. 13 NameNode HA, Federation and Operations Learn how HDFS removes the NameNode single point of failure with standby nodes, journal nodes and ZooKeeper, plus federation, small files and the balancer.

YARN and Resource Management

  1. 14 YARN: Resources and Scheduling Understand YARN, Hadoop's resource manager: ResourceManager, NodeManagers, containers, the ApplicationMaster and the life of an application, with a simulation.
  2. 15 Capacity and Fair Schedulers Compare the FIFO, Capacity and Fair schedulers in YARN, learn queues, preemption and how to size containers and tune memory settings, with a fair-share allocator.

MapReduce

  1. 16 The MapReduce Programming Model Learn how MapReduce works: key/value pairs, the map, shuffle and reduce phases, data locality and why the model scales, with word count in plain Python.
  2. 17 Inside the Shuffle and Sort Open the black box between map and reduce: partitioning, spills, merging, sorting, combiners, choosing the reducer count and fighting skewed keys.
  3. 18 MapReduce Design Patterns Learn the reusable MapReduce patterns: filtering, summarization, inverted index, map-side and reduce-side joins, top-N and secondary sort, each in a runnable simulator.
  4. 19 Writing a MapReduce Job in Java Write a complete Hadoop MapReduce job in Java: Mapper, Reducer, Writable types, the Job driver, input and output formats and running it with hadoop jar.
  5. 20 MapReduce Worked Examples: Character Count, Weather and Titanic Three complete MapReduce programs: character count, hot and cold day detection from weather records, and average age by gender, with jars, commands and output inspection.
  6. 21 Hadoop Streaming with Python Write MapReduce jobs in Python with Hadoop Streaming: stdin/stdout mappers and reducers, why reducer input is sorted, and how to test a job with a local pipe.
  7. 22 Fault Tolerance, Tuning and Limits of MapReduce How MapReduce survives failures with task retries and speculative execution, how to tune a job, and why disk-based iteration led to Spark replacing it.

Data Formats and Storage

  1. 23 Row vs Column Formats Compare text, CSV, JSON, SequenceFile and columnar layouts for big data: splittability, compression codecs and why column formats scan so much less data.
  2. 24 Avro and Schema Evolution Learn how Avro stores data with a schema, how its compact binary encoding works, and which schema changes are safe, with a runnable encoder and resolver.
  3. 25 Parquet and ORC Internals Open a Parquet file: row groups, column chunks, pages, dictionary and run-length encoding, min/max statistics and predicate pushdown, with runnable stand-ins.
  4. 26 Partitioning, Bucketing and File Layout Organize big data files so queries read less: directory partitioning, partition pruning, bucketing, the small-files problem and compaction, with runnable simulations.

SQL and NoSQL on Hadoop

  1. 27 Hive: SQL over Files Learn how Hive puts SQL on top of files in HDFS: the metastore, managed and external tables, HiveQL, SerDes and how a query is compiled into distributed stages.
  2. 28 Hive Performance: Execution Engines and Tuning Make Hive fast: why Tez replaced MapReduce, vectorized execution, the cost-based optimizer, map-join versus shuffle-join, and ACID tables, with cost-model examples.
  3. 29 Interactive SQL: Presto/Trino and Impala Learn how Presto/Trino and Impala run fast interactive SQL on Hadoop data: MPP architecture, connectors, federated queries, and how they differ from Hive.
  4. 30 Pig Latin Dataflows Learn Apache Pig's dataflow language: relations, LOAD, FILTER, GROUP, JOIN and FOREACH, user-defined functions and how Pig differs from Hive, with a Python stand-in.
  5. 31 HBase: Wide-Column Storage Learn how HBase stores huge sparse tables: row keys, column families, regions, the LSM tree with memstore, HFiles and compaction, and good row-key design.
  6. 32 Choosing a NoSQL Store Compare key-value, document, wide-column and graph databases, and learn consistent hashing and the Dynamo and Cassandra ideas behind scalable stores.

Data Ingestion and Processing Models

  1. 33 Getting Data In: Sqoop, Flume, NiFi and Kafka Learn batch and streaming data ingestion for Hadoop: Sqoop imports, Flume agents, NiFi flows, Kafka topics, change data capture and delivery guarantees.
  2. 34 Batch vs Stream Processing Understand bounded and unbounded data, latency versus throughput, event time, tumbling and sliding windows, watermarks, and Lambda and Kappa architectures.
  3. 35 Spark, Flink and What Came After MapReduce See how Spark's RDDs, DataFrames and lazy DAGs replaced MapReduce, how Flink does true streaming, and what Apache Beam adds, with a lazy-evaluation demo.
  4. 36 Machine Learning on Hadoop: Mahout and Beyond Learn what Apache Mahout did for machine learning on Hadoop: recommenders, clustering and classification on MapReduce, why iteration was slow, and what replaced it.

Architecture and Operations

  1. 37 Data Lakes, Warehouses and Lakehouses Compare data lakes, warehouses and lakehouses: zones, schema-on-read, Delta, Iceberg and Hudi table formats, and a runnable versioned table with snapshots.
  2. 38 Security, Governance and Data Quality Learn how Hadoop clusters are secured with Kerberos, Ranger and encryption, how lineage and catalogs help, and how to check data quality with rules.
  3. 39 Sizing, Monitoring and Running a Cluster Plan Hadoop cluster capacity, watch the right metrics, recognise common failures and compare on-premises with cloud costs, with a capacity calculator.
  4. 40 ZooKeeper for Coordination Learn what ZooKeeper does in Hadoop: znodes, ensembles and quorum, sessions, watches, leader election, and its role in HDFS and YARN high availability and HBase.

Projects, Practice and Review

  1. 41 Project: Web Log Analysis with MapReduce Build a complete MapReduce pipeline that parses web server access logs, finds top pages and errors per hour, and uses a combiner to cut shuffle data.
  2. 42 Project: Search Index over Documents Build an inverted index and TF-IDF ranking with chained MapReduce jobs, then split the index build into partitions the way a cluster would.
  3. 43 Project: Co-purchase Recommendations Build a 'customers also bought' recommender with chained MapReduce jobs: pair counting, a top-N per item and a lift score to avoid popularity bias.
  4. 44 Project: Clickstream Sessionization Group raw click events into user sessions with a secondary sort, a 30-minute gap rule, stream-style session windows and a columnar Parquet-style output.
  5. 45 Practice Exercises with Solutions Twelve hands-on Hadoop and big data exercises on HDFS blocks, replication math, MapReduce jobs, partitioning, skew and file formats, each with a runnable solution.
  6. 46 Hadoop and Big Data Interview Questions Prepare for Hadoop and big data interviews: clear answers on HDFS, YARN, MapReduce, Hive, HBase, file formats and batch vs stream, with small runnable checks.
  7. 47 Hadoop and Big Data Cheat Sheet A one-page Hadoop and big data cheat sheet: HDFS, YARN and Hive commands, key defaults, file format comparison and a table for choosing the right tool.