Skip to content
Tutorials Center
My learning

Topics

Lesson 60 of 149 ยท Modules and the Standard Library

Files and Folders with os and shutil

Work with folders in Python. List and walk directories with os, read environment variables, copy, move and delete trees with shutil, use tempfile.

pathlib gives you path objects for working with single files and folders. Two older modules fill in the rest of the picture. os is Python's interface to the operating system: the current directory, folder listings, environment variables, process information. shutil ("shell utilities") does the high-level jobs you'd otherwise do in a file manager: copying, moving and deleting whole folder trees, and making zip archives.

You'll see both in almost every automation script, so in this lesson you'll learn the parts you actually need: listing and walking folders, the os.path functions you'll meet in existing code, environment variables, shutil copy, move and delete operations, and the tempfile module for scratch folders that clean up after themselves.

Note: The editors on this page have their own in-memory file system, so every example here works on real (temporary) files and can't touch your computer. Each example creates its own temporary folder, so you can run it as often as you like.

Where am I? The working directory

Relative paths like "notes.txt" are resolved against the current working directory: the folder your program was started from, not necessarily the folder your script is in. os.getcwd() tells you what it is, and os.chdir() changes it:

Python The working directory

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

os.getcwd() returns a str. Its sibling os.getcwdb() returns the same folder as bytes, exactly as the operating system stores it, which matters only when a folder name isn't valid text in the default encoding. Most programs never need it:

Python
import os
print(type(os.getcwd()).__name__, type(os.getcwdb()).__name__)  # str bytes
print(os.getcwdb().decode() == os.getcwd())                     # True

os.chdir() affects the whole program, which makes code hard to follow. contextlib.chdir changes the directory only inside a with block and restores it afterwards. Better still, avoid changing directory at all and build full paths instead, as the pathlib lesson shows with Path(__file__).parent.

Listing folders

os offers three ways to look inside folders, from simplest to most powerful:

  • os.listdir(path) returns a list of names (strings) in one folder.
  • os.scandir(path) yields DirEntry objects that already know whether each item is a file or folder, plus its size and dates, with fewer system calls.
  • os.walk(path) goes through a folder and every folder inside it.

Let's build a small project tree and look at it each way:

Python listdir and scandir

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

os.listdir() gives only names, in no guaranteed order, so sort them if order matters, and join them with the folder path before opening them. scandir() is the faster choice when you need to know what each entry is. Use it in a with block so the operating system handle is closed promptly.

Walking a whole tree

os.walk() visits every folder below a starting point. For each folder it yields a tuple: the folder's path, the names of its subfolders, and the names of its files:

Python os.walk

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The trick in the middle is worth remembering: os.walk() looks at the subfolders list after yielding it, so removing names from it (in place) stops the walk from entering those folders. That's how you skip .git, node_modules or virtual environment folders. For simple pattern searches, Path(root).rglob("*.py") is shorter; os.walk() wins when you want to prune folders or handle each folder as a unit.

os.path in older code

Before pathlib, paths were plain strings handled by os.path. You'll read this style in countless existing scripts, so know the common functions:

Python os.path functions

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The pathlib lesson has a table that maps each of these to its Path equivalent. For new code, prefer pathlib; mixing the two is fine, because every os and shutil function accepts Path objects as well as strings.

Creating, renaming and deleting with os

Task os function Notes
make one folder os.mkdir(p) fails if the parent is missing
make nested folders os.makedirs(p, exist_ok=True) creates every missing level
rename or move os.rename(src, dst) on Windows, fails if dst exists
rename, replacing os.replace(src, dst) overwrites dst on every system
delete a file os.remove(p)
delete an empty folder os.rmdir(p) fails if not empty

On Linux and macOS, os.chown(path, uid, gid) changes a file's owner and group (usually only the superuser may do that), and os.chmod(path, mode) changes its permission bits. os.chown doesn't exist on Windows, so check hasattr(os, "chown") in code that must be portable.

Python
import os, stat

os.chmod("report.txt", stat.S_IRUSR | stat.S_IWUSR)   # owner read/write only (0o600)
if hasattr(os, "chown"):                               # not available on Windows
    os.chown("report.txt", os.getuid(), os.getgid())   # keep the current owner and group

os.replace() is a useful building block for safe saves: write the new content to a temporary file next to the target, then replace the target in one step. If the program crashes halfway, the old file is still intact.

Environment variables

Every process has a set of environment variables: name/value strings set by the operating system, the shell, or a hosting platform. Programs use them for settings that shouldn't be hard-coded, such as API keys, database addresses or a "debug" switch. os.environ works like a dictionary:

Python Reading environment variables

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Values are always strings, so convert numbers and booleans yourself. Use os.environ["NAME"] when a setting is required (a clear KeyError at startup beats a mysterious failure later), and os.environ.get("NAME", default) when it's optional. Setting a variable only affects your own process and the programs it starts, never the shell that launched you.

Tip: Keep secrets like API keys in environment variables (or a secrets manager), not in your code, so they never end up in version control. Many projects load local values from a .env file with the third-party python-dotenv package.

shutil: copy, move and delete trees

os works on one file or one empty folder at a time. shutil handles whole trees:

Function What it does
shutil.copy(src, dst) copy a file's content and permissions; dst may be a folder
shutil.copy2(src, dst) like copy, but also keeps timestamps
shutil.copytree(src, dst) copy a whole folder tree
shutil.move(src, dst) move a file or folder, even across drives
shutil.rmtree(path) delete a folder and everything in it
shutil.make_archive(base, "zip", root) create a .zip (or .tar.gz) of a folder
shutil.which("git") find a program on the PATH, or None
Python Backing up a project with shutil

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

copytree() fails if the destination already exists; pass dirs_exist_ok=True to merge into an existing folder. The ignore=shutil.ignore_patterns(...) argument skips names that match glob patterns, which is perfect for leaving out caches and build output.

shutil.rmtree() deletes permanently. There's no recycle bin and no confirmation. Double-check the path before calling it, never build it from unchecked user input, and consider printing what would be deleted on the first run of a new script.

Temporary files and folders

Programs often need scratch space: a file to download into, a folder to unpack an archive, test data. The tempfile module creates them in the system's temp location with unique names, so two programs never collide, and it can delete them for you:

Python tempfile

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

TemporaryDirectory is the one you'll use most. Tests rely on it heavily (pytest's tmp_path fixture is the same idea), and it's a safe home for any intermediate file. Every example in this lesson used it, which is why you could rerun them without leftovers.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Which function gives you every folder below a starting point, together with its subfolders and files?

    os.walk() yields (folder, subfolders, files) for the starting folder and every folder inside it.
  2. 2. How do you stop os.walk() from entering folders named .git?

    os.walk() reads the subfolders list after yielding it, so subfolders.remove(".git") prunes that branch.
  3. 3. Your program needs DATABASE_URL to run. What's the best way to read it?

    For required settings, failing early with a clear error is better than running with an empty value. Use .get() with a default only for optional settings.
  4. 4. Which call deletes a folder that still contains files?

    os.rmdir() only removes empty folders and os.remove() removes files. shutil.rmtree() deletes a whole tree, permanently.

Summary

  • os.getcwd() shows the working directory; prefer full paths, or contextlib.chdir for a temporary change.
  • os.listdir() returns names, os.scandir() returns entries that know their type and size, and os.walk() visits a whole tree (prune it by editing subfolders in place).
  • os.path functions (join, basename, splitext, exists...) appear in older code; pathlib does the same with objects.
  • os.makedirs(exist_ok=True), os.replace(), os.remove() and os.rmdir() handle single items.
  • os.environ holds environment variables as strings; use [...] for required settings and .get() with a default for optional ones.
  • shutil copies (copy2, copytree), moves, zips (make_archive) and deletes trees (rmtree, permanently).
  • tempfile.TemporaryDirectory() gives you a scratch folder that cleans itself up.

Next: Reading and Writing Files, where you'll open files with open(), read them line by line and choose the right mode.