Skip to content
Tutorials Center
My learning

Topics

Lesson 15 of 32 ยท Retrieval and Vector Stores

Document Loaders and Text Splitters

Load text, PDFs and web pages into LangChain Document objects with metadata, then split them into well-sized overlapping chunks with RecursiveCharacterTextSplitter, plus a runnable splitter.

RAG starts with raw material: files, web pages, database rows. LangChain represents each piece of text as a Document, and prepares it for search in two steps: a loader produces Documents from a source, and a text splitter cuts long Documents into chunks small enough to embed and retrieve precisely.

The Document object

A Document has just two fields: page_content (the text) and metadata (a dictionary describing it).

Python
from langchain_core.documents import Document

doc = Document(
    page_content="Refunds are issued within 14 days.",
    metadata={"source": "policies/refunds.md", "section": "Refunds", "updated": "2026-01-10"},
)

Metadata is not decoration. It lets you cite sources, filter ("only documents from 2026"), and debug retrieval. Decide what metadata you need before indexing, because adding it later means re-indexing.

Document loaders

Loaders live mostly in langchain-community and in provider-specific packages. They all expose .load() (a list) and .lazy_load() (an iterator that keeps memory low):

Python
from langchain_community.document_loaders import TextLoader, DirectoryLoader, PyPDFLoader, WebBaseLoader

docs = TextLoader("notes.txt", encoding="utf-8").load()
docs = DirectoryLoader("handbook/", glob="**/*.md", loader_cls=TextLoader).load()
pages = PyPDFLoader("report.pdf").load()          # one Document per page, with page numbers
web = WebBaseLoader("https://example.com/docs").load()

Other common sources are CSV files, JSON, Word documents, Notion, Google Drive, GitHub, SQL databases and S3 buckets. PDF loaders differ in quality: scanned PDFs need OCR, and tables often come out as messy text, so inspect the result before trusting it.

Always print a few loaded Documents. Navigation menus, cookie banners and repeated headers in web pages pollute the index, and cleaning them at load time is the cheapest quality win in RAG.

Why split?

Embedding a whole 50-page document gives one vector that blurs everything it says, and sending it all to the model wastes the prompt. Splitting produces chunks, each about one idea, so retrieval can return exactly the relevant paragraph.

Two numbers control splitting:

  • chunk_size: the maximum length of a chunk, measured in characters (or tokens, if you use a token-based length function).
  • chunk_overlap: how much of the end of one chunk is repeated at the start of the next, so a sentence cut at a boundary still appears whole in at least one chunk.
Python
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(chunk_size=800, chunk_overlap=100)
chunks = splitter.split_documents(docs)
print(len(chunks), chunks[0].metadata)

split_documents keeps each source's metadata on its chunks. Typical starting points are 500 to 1,000 characters with 10 to 20 percent overlap; tune on your own questions.

How the recursive splitter works

RecursiveCharacterTextSplitter tries a list of separators in order of preference: paragraph breaks ("\n\n"), then line breaks, then spaces, then single characters. It splits on the biggest separator that makes pieces small enough, so it keeps paragraphs together when it can and only cuts mid-sentence as a last resort. Here is the algorithm, simplified, in plain Python:

Python Stand-in: a recursive text splitter

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice how the chunks follow paragraph boundaries first, and the long paragraphs get split on spaces.

Overlap (runnable)

Overlap repeats the tail of one chunk at the start of the next. This stand-in applies a sliding window over words, which makes the effect easy to see.

Python Stand-in: chunk overlap with a sliding window

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at where each chunk begins: the first words repeat the last three words of the previous chunk.

Splitting by structure

Plain character splitting ignores document structure. Better options exist:

  • MarkdownHeaderTextSplitter splits by headings and stores the heading path in metadata ({"h1": "Refunds"}), so each chunk knows what section it came from.
  • Language-aware splitters (RecursiveCharacterTextSplitter.from_language(Language.PYTHON)) split code on classes and functions.
  • Token-based splitters measure length in tokens, which matches the embedding model's limits better than characters.
  • Semantic chunkers split where the topic changes, using embeddings; powerful but slower and not always better.

Adding context to chunks

A chunk that says "It takes 3 to 5 days" is meaningless alone. Fixes: keep the heading in the chunk text or metadata, use overlap, and keep chunks big enough for one complete idea. For tables and lists, splitting by line often destroys meaning, so keep them whole.

Python Stand-in: carry the section heading into every chunk

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Practical advice

  • Start simple: recursive splitter, 800 characters, 100 overlap.
  • Inspect the chunks by eye. If a chunk does not make sense alone, the index will not either.
  • Keep chunk ids stable (for example source plus position) so you can update one document without rebuilding everything.
  • Store the original source and position in metadata for citations.
  • Re-split when you change chunk size, and re-embed; they are linked.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What are the two fields of a LangChain Document?

    page_content holds the text; metadata is a dictionary describing it, used for citations and filtering.
  2. 2. What is chunk_overlap for?

    Overlap keeps context that would otherwise be split between two chunks.
  3. 3. How does RecursiveCharacterTextSplitter prefer to split?

    It tries the coarsest separator that keeps pieces under the size limit.
  4. 4. Why store metadata like section and source on every chunk?

    Metadata supports citations and filtered search, and tells you where a chunk came from.

Summary

  • A Document is page_content plus metadata; loaders create them from files, PDFs, web pages and more.
  • Split long documents into chunks of about one idea; typical start is 500 to 1,000 characters with about 10 to 20 percent overlap.
  • RecursiveCharacterTextSplitter respects paragraphs first; structure-aware splitters keep headings as metadata.
  • Inspect chunks yourself; clean load output; keep stable ids.

Next: What Is an Agent?