Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 36 ยท Getting Started

Data Sources, Types and Formats

Understand where data comes from, the difference between structured, semi-structured and unstructured data, and how CSV, JSON and logs are parsed in practice.

Before you can build a pipeline you need to know what you are pulling from. Data comes from applications, devices, third-party services and people typing into spreadsheets, and each source has its own quirks. This lesson classifies sources and formats and shows how to parse the three you will see most: CSV, JSON and log lines.

Where data comes from

Common data sources

App databases Orders, users, payments.
APIs SaaS tools and partners.
Logs and events Clicks, requests, errors.
Files CSV, Excel, exports.

Each source fails in its own way.

1 / 2

Those four questions are the classic "four Vs". For every new source, write down its size, arrival speed, shape and trustworthiness before designing anything.

Structured, semi-structured, unstructured

  • Structured data has a fixed schema: rows and columns with known types. Relational tables and well-formed CSV files are structured.
  • Semi-structured data carries its own structure but allows variation: JSON, XML, Avro records. Fields may be missing, nested or added over time.
  • Unstructured data has no predefined model: free text, images, audio, PDFs. You extract structure from it (OCR, text analysis) or store it as objects.

Most engineering effort goes into turning semi-structured data into structured tables, because tables are what SQL and dashboards understand.

CSV: simple and treacherous

CSV looks trivial and is full of traps: delimiters inside values, missing headers, inconsistent quoting and everything-is-a-string. Always read it with a proper parser, never with split(",").

Python Parse CSV with quotes and gaps

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the amount column: because one value is missing, pandas stores it as float and shows NaN. Explicit types (dtype=) are safer than guessing.

JSON: nested and flexible

JSON records often nest objects and arrays. Flattening them into columns is a daily task, and pandas.json_normalize does most of it.

Python Flatten nested JSON

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Ben has no country, so the flattened column holds NaN. That is the semi-structured reality: absent fields are normal, and your pipeline must decide whether it is an error or a default.

Logs: text with a hidden schema

Server logs, application events and access logs are lines of text whose structure lives in a convention. Regular expressions turn them into rows.

Python Parse web-server log lines

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Again the rejects are counted, not discarded. In production you would write them to a dead-letter file.

Schema, types and nulls

A schema says which columns exist, their types, and whether they may be null. Pipelines should enforce one at the boundary, rather than letting surprises travel downstream.

Python Validate a schema at the boundary

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real tools express the same idea with typed schemas (Avro, Protobuf, pydantic, table DDL). The lesson is the same: decide what "valid" means and check it where data enters.

Time zones and encodings

Two small things cause large bugs. First, time: store timestamps in UTC, with a time zone, and convert for display only. Second, text encoding: use UTF-8 end to end, and decode bytes explicitly. A file exported from an old spreadsheet may be in a different encoding, and reading it as UTF-8 produces garbage or errors.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which of these is semi-structured data?

    JSON carries its own structure but allows missing, extra or nested fields.
  2. 2. Why should you not parse CSV with line.split(',')?

    Quoted fields such as "Smith, Ann" contain delimiters. A real CSV parser handles quoting.
  3. 3. What are the four Vs used to describe data sources?

    Volume, velocity, variety and veracity describe size, speed, shape and trustworthiness.
  4. 4. What is a good policy for log lines that fail to match the parser?

    Rejects should be visible. A dead-letter store keeps them for analysis without blocking good records.

Summary

  • Sources include databases, APIs, logs and files; ask about volume, velocity, variety and veracity.
  • Structured, semi-structured and unstructured data need different handling; most work turns semi-structured data into tables.
  • Parse CSV, JSON and logs with proper tools and keep the rejects.
  • Validate schemas at the boundary, store time in UTC and use UTF-8.

Next: The Data Engineering Lifecycle