Skip to content
Tutorials Center
My learning

Topics

Lesson 56 of 149 ยท Modules and the Standard Library

Saving Objects with pickle and shelve

Save almost any Python object to bytes or a file with pickle, store objects by key with shelve, avoid the security trap, and copy with deepcopy.

JSON is great for exchanging data, but it only understands a handful of types. Save a set, a date or a tuple and you get back a list, a string or another list, if it works at all. Sometimes you just want to save a Python object exactly as it is and get the same thing back later. That's what pickle does.

The pickle module turns nearly any Python object into a stream of bytes (this is called serializing or "pickling") and rebuilds the object from those bytes later ("unpickling"). In this lesson you'll pickle data to bytes and files, see what can and can't be pickled, learn why you must never unpickle untrusted data, store objects by key with shelve, and finish with the copy module, which uses the same machinery to duplicate objects.

dumps and loads

The function names match the json module: dumps() returns bytes, loads() rebuilds the object.

Python A round trip through pickle

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Every value came back with its original type: the date is still a date, the set is still a set, the Decimal keeps its exact value. The result is equal to the original but a separate object.

Pickle also remembers shared references. If two places in your data point to the same list, they still point to one shared list after loading, and even an object that contains itself survives:

Python Shared and circular references

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

JSON can't do either of these: it would duplicate the list, and it raises an error on the cycle.

Pickle files

dump() and load() (no "s") write to and read from a file. Pickle data is binary, so open the file in binary mode, "wb" and "rb", and don't pass an encoding:

Python Saving game state to a file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Pickle files aren't meant to be read by people or by other languages. If you open one in a text editor you'll see a jumble of bytes. That's fine for caches and save files that only your Python program reads.

What can be pickled?

Pickle handles all the built-in data types, most standard library classes, and instances of your own classes. But it stores functions and classes by name, not their code: it writes down "the function sqrt from the module math" and imports it again when loading. Anything without an importable name, or tied to the running program, fails:

Python Things pickle refuses

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A lambda has no name to look up, a generator is a paused piece of a running program, and an open file belongs to the operating system. Database connections, locks and network sockets fail for the same reason.

Your own classes

Instances of your classes pickle automatically: pickle saves the class's module and name plus the instance's attribute dictionary. When loading, it imports the class and fills in the attributes without calling __init__. That means the class must be importable when you load, so define classes you pickle in a module, not inside a function.

When an object holds something that can't be pickled, such as an open file or a cache, control what gets saved with __getstate__ and __setstate__:

Python Choosing what to pickle

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Pickled data can also break when a class changes: rename a class or move it to another module and old pickles can no longer find it. That's one reason pickle suits short-lived data (caches, checkpoints, data passed between processes) better than long-term storage.

Protocols

Pickle has several format versions called protocols. Newer protocols are more compact and faster, and dumps() picks a modern default for you. You only need to choose one when an older Python version must read the data:

Python
import pickle

print(pickle.DEFAULT_PROTOCOL, pickle.HIGHEST_PROTOCOL)
data = pickle.dumps([1, 2, 3], protocol=pickle.HIGHEST_PROTOCOL)

loads() detects the protocol automatically.

Never unpickle untrusted data

Here's the most important fact about pickle: loading a pickle can run any code. A pickle doesn't just store data; it stores instructions for rebuilding objects, and an object can tell pickle to rebuild it by calling any function with any arguments. This harmless demo makes loads() call print(); an attacker would call something that deletes files or installs malware:

Python Why pickle is unsafe

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The victim never imported or defined Surprise. The bytes alone told loads() to call print. Swap print for os.system and the bytes could run any shell command.

Only unpickle data that your own program created and that nobody else could have changed. Never load pickles from users, uploads, emails, or the internet. For data that crosses a trust boundary, use JSON or another data-only format.

shelve: a dictionary on disk

The shelve module combines pickle with a simple database file. A shelf behaves like a dictionary whose keys are strings and whose values can be any picklable object, but it lives in a file, so the data survives between runs of your program:

Python
import shelve

# flag="n" always starts with a new, empty shelf (handy when rerunning)
with shelve.open("library", flag="n") as db:
    db["isbn-0001"] = {"title": "Tidal Maps", "copies": 2, "tags": {"travel"}}
    db["isbn-0002"] = {"title": "Small Engines", "copies": 1, "tags": set()}

# Later, maybe in another run of the program:
with shelve.open("library") as db:
    print(sorted(db.keys()))
    book = db["isbn-0001"]
    print(book["title"], book["copies"])

    # Changing a value you read does NOT update the file...
    db["isbn-0002"]["copies"] = 5
    print(db["isbn-0002"]["copies"])     # still 1

    # ...you must assign it back:
    entry = db["isbn-0002"]
    entry["copies"] = 5
    db["isbn-0002"] = entry
    print(db["isbn-0002"]["copies"])     # now 5

    del db["isbn-0001"]
    print(len(db), "book left")

Note: Run that example on your computer. shelve.open() stores data through Python's dbm module, which the in-browser Python on this site leaves out. The editor below uses shelve.Shelf, which wraps any dictionary, so you can still try the same operations live.

Python A shelf you can watch

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Each time you read a key, shelve unpickles a fresh copy, so changing that copy doesn't touch the stored data. Assign the value back, as shown, or open the shelf with writeback=True, which caches every value you access and writes them all back when the shelf closes (at the cost of memory).

The flag argument works like a file mode: "c" (the default) opens or creates the shelf, "r" is read-only, "w" opens an existing one for writing, and "n" always creates a new, empty shelf. Shelve may create one file or several, depending on the database backend Python finds, so treat the name you pass as a base name rather than a full file name.

Shelve is handy for small tools, caches and prototypes. Because it uses pickle, the same security rule applies, and it isn't designed for several programs writing at once. When you outgrow it, move to a real database like SQLite, which you'll meet in Databases with sqlite3.

Copying objects: copy and deepcopy

Pickle's ability to rebuild objects has a close cousin in the copy module. You've seen list.copy() and copy.deepcopy() in List Methods; here's the full picture:

  • Assignment (b = a) copies nothing. Both names point to the same object.
  • copy.copy(a) makes a shallow copy: a new outer object whose items are the very same inner objects.
  • copy.deepcopy(a) copies recursively, so nested lists, dictionaries and your own objects are all duplicated. Like pickle, it keeps shared references and cycles intact.
Python Shallow vs deep copies

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The shallow copy kept its old name, because rebinding original.name doesn't affect it, but it shares the members list, so it sees "Cai". The deep copy shares nothing. Classes can customize this by defining __copy__ and __deepcopy__, though the defaults are right for most classes.

You might see pickle.loads(pickle.dumps(obj)) used as a "deep copy" in older code. copy.deepcopy() is clearer, works with more objects, and is the right tool.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. How should you open a file for pickle.dump()?

    Pickle produces bytes, so the file must be opened in binary mode, without an encoding.
  2. 2. A web form lets users upload a .pkl file that your server loads with pickle.load(). What's the problem?

    A pickle can instruct load() to call any function. Never unpickle data from an untrusted source; use JSON instead.
  3. 3. Why does pickle.dumps(lambda x: x + 1) fail?

    Functions and classes are saved as references like math.sqrt, not as code. A lambda has no name pickle could import later.
  4. 4. You read book = db["k"] from a shelf (default options), change book["copies"], and close the shelf. Is the change saved?

    Each read returns a fresh unpickled copy, so changes stay in memory unless you store the value again.
  5. 5. b = copy.copy(a) where a is a list of lists. What is shared between a and b?

    A shallow copy creates a new outer list holding the same inner objects. Use copy.deepcopy() to copy the inner lists too.

Summary

  • pickle.dumps()/loads() convert objects to bytes and back; dump()/load() use files opened with "wb"/"rb".
  • Pickle keeps exact types, shared references and cycles, which JSON can't.
  • Functions and classes are pickled by name, so lambdas, generators, open files and connections fail; your classes must be importable when loading.
  • __getstate__ and __setstate__ control what an object saves and how it's restored.
  • Never unpickle untrusted data: loading a pickle can run arbitrary code.
  • shelve is a persistent, dictionary-like store built on pickle; assign changed values back or use writeback=True.
  • copy.copy() makes shallow copies and copy.deepcopy() makes fully independent ones.

Next: XML with ElementTree, where you'll parse, search and build XML documents.