Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 38 ยท Distribution and Docker

Private Indexes, Mirrors and Vendoring

Host internal Python packages and install offline. Learn the PEP 503 simple index, index-url vs extra-index-url, dependency confusion, wheelhouses and vendoring, with runnable examples.

PyPI is public. Many organizations also have code that must not be public: an internal library shared between services, a patched fork, a package with a commercial license. Others work in places with no internet access at all, or want builds that do not break when PyPI is slow. For all of these, you need to control where packages come from.

This lesson explains how Python package indexes actually work, the options for private and offline distribution, and a security problem you must understand: dependency confusion. Parts of it run for real in this page, using only the standard library.

How an index works

An installer needs two things from an index: the list of files available for a project, and the files themselves. The standard way to provide that is the simple repository API (PEP 503, with later additions): a folder of HTML pages.

  • /simple/ lists projects.
  • /simple/myproject/ lists the project's files as links, with a hash in each URL fragment.

That is all pip, uv or Poetry need. Because the format is just static HTML, an index can be a plain web server or a folder on a network drive.

Python Generate and read a simple index page

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice the name normalization: Internal_Utils, internal-utils and internal.utils are the same project, and the index URL uses the normalized form. And notice that the hash is part of the link, so a client can verify every file it downloads.

Options for private distribution

Option What it is Good for
Hosted registry A cloud service (GitHub Packages, GitLab package registry, AWS CodeArtifact, Azure Artifacts, Google Artifact Registry) Teams that already use that platform
Self-hosted index devpi, Artifactory, Nexus, pypiserver Companies that need full control, caching and policies
A folder of files Any directory or web path used with --find-links Tiny setups, offline installs
Direct references pkg @ git+https://example.com/[email protected] Quick forks and unreleased code

A mirror or proxy cache (devpi, Artifactory, Nexus) sits in front of PyPI: the first install fetches from PyPI and stores the file, later installs use the copy. This speeds up CI, survives PyPI outages and lets security teams block packages. Often the same server also hosts your private packages, giving developers one URL for everything.

Telling installers where to look

Terminal
# Replace PyPI completely
pip install --index-url https://pypi.internal.example/simple/ myproject

# Add a second index IN ADDITION to PyPI (dangerous: see below)
pip install --extra-index-url https://pypi.internal.example/simple/ myproject

Make it permanent in configuration instead of typing it every time. For pip, pip.conf (or pip.ini on Windows) or environment variables:

INI
[global]
index-url = https://pypi.internal.example/simple/
TOML
# uv, in pyproject.toml
[[tool.uv.index]]
name = "internal"
url = "https://pypi.internal.example/simple/"
explicit = true      # only used for packages assigned to it

[tool.uv.sources]
internal-utils = { index = "internal" }

Credentials belong in environment variables, .netrc, a keyring or your CI's secret store. Never commit passwords or tokens inside URLs.

Dependency confusion

Suppose your company has a private package acme-billing version 1.2.0, and your build uses --extra-index-url to find it. The installer queries both indexes and takes the highest version. An attacker who notices the name uploads acme-billing version 99.0.0 to public PyPI. Now every fresh build installs the attacker's code. This is a dependency confusion attack, and it has hit real companies.

Python Dependency confusion in miniature

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the first run acme-billing comes from the public index at version 99.0.0: the attack succeeds. In the second, the package is bound to the private index and the attacker's upload is ignored.

How to defend:

  1. Use a single index URL that points at your proxy (index-url, not extra-index-url). The proxy decides what is internal and what is proxied from PyPI, and can block public packages with your internal names.
  2. Bind packages to indexes (uv explicit = true with tool.uv.sources).
  3. Use hash-pinned lock files (lock files) so unexpected files are refused.
  4. Reserve your internal names on public PyPI with placeholder packages, if you can.
  5. Prefer an internal prefix or namespace, such as acme-billing, and never publish internal names by accident.

Offline installs: the wheelhouse

When a server has no internet, build a wheelhouse: a folder of wheels for everything you need, then install from it only.

Terminal
# On a machine with internet
pip download -r requirements.txt -d wheelhouse/
# or: pip wheel -r requirements.txt -w wheelhouse/

# Copy the folder to the offline machine, then:
pip install --no-index --find-links wheelhouse/ -r requirements.txt

--no-index guarantees pip never reaches for the network, so a missing file is a clear error rather than a silent download. Download for the target platform and Python version (--platform, --python-version, --only-binary=:all:) when the build machine differs from the server.

Vendoring

Vendoring means copying a dependency's source directly into your project, for example under myapp/_vendor/. Pip does this itself for its internal libraries. It removes an install-time dependency, but you must keep the copy updated, handle its license, and rewrite its imports (tools such as vendoring automate this). Treat it as a last resort, appropriate for tiny pure-Python helpers or tools that must have zero dependencies, not for large or fast-moving packages.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is a Python package index at its simplest?

    Installers need only the project listing pages and the files, so even a static web server or a folder can act as an index.
  2. 2. What makes dependency confusion possible?

    If an attacker publishes your internal package name with a higher version on the public index, the highest-wins rule selects it.
  3. 3. Which command installs only from a local folder of wheels, never from the network?

    --no-index disables the package index and --find-links points at the folder.
  4. 4. When is vendoring a reasonable choice?

    Vendoring has a maintenance cost, so keep it for small, stable code where having no dependency is valuable.

Summary

  • An index is static pages in the simple API format: project listings, file links and hashes.
  • Private options include hosted registries, self-hosted servers (devpi, Artifactory, Nexus), plain folders and Git URLs; mirrors cache PyPI.
  • Prefer one index-url; mixing a private and the public index with --extra-index-url invites dependency confusion.
  • Bind internal packages to their index, hash-pin lock files and reserve internal names.
  • Wheelhouses with --no-index --find-links support offline installs; vendoring copies code in as a last resort.

Next: Project: Publish a Command-Line Package