Skip to content
Tutorials Center
My learning

Topics

Lesson 33 of 49 ยท Working with Real Data

Saving and Loading Arrays

Save and load NumPy arrays with np.save, savez and savez_compressed, convert to raw bytes with tobytes and frombuffer, compare file sizes and avoid pickle.

Computing an array can take minutes; saving it takes milliseconds. NumPy ships its own binary file formats that store the dtype, shape and values exactly, load back instantly and work on every platform. Text formats such as CSV lose the dtype, take far more space, and parse slowly.

This lesson covers the .npy format for one array, .npz archives for several, raw byte conversion, how big the files really are, and why you should be cautious with pickle. All examples write to io.BytesIO, an in-memory "file", so they run anywhere. With a real file you simply pass a path such as "data.npy" instead.

One array: np.save and np.load

np.save writes a .npy file: a small text header describing the dtype and shape, followed by the raw bytes of the array. np.load reads it back as an identical array.

Python A .npy round trip in memory

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The file is 128 bytes larger than the array's data: that is the header. The first bytes are the magic string \x93NUMPY. Because dtype and shape are in the header, nothing is lost: an int16 array comes back as int16, and a (3, 4) array comes back as (3, 4).

On a real disk the calls are np.save("a.npy", a) and np.load("a.npy"). If you leave off the extension when saving to a path, NumPy adds .npy.

Several arrays: savez and savez_compressed

np.savez bundles named arrays into a single .npz file (a zip archive of .npy files). np.savez_compressed does the same with compression, which pays off when the data is repetitive or sparse.

Python Comparing savez with savez_compressed

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Loading an .npz returns a lazy object: arrays are read only when you index them by name. Use it in a with block so the file is closed. Compression costs CPU time on both save and load, so use it for archives you keep and send, and skip it for scratch files you reload constantly. Random floating-point data barely compresses at all.

Raw bytes: tobytes and frombuffer

Sometimes you do not want a file format at all, only the bytes. A network protocol, a database blob or a binary image file all need a plain byte sequence. arr.tobytes() returns the array's data as bytes and np.frombuffer(data, dtype=...) builds an array over bytes.

Python Arrays to bytes and back

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Three lessons in this output:

  • Bytes carry no dtype or shape. You must remember both, or the numbers make no sense. .npy exists precisely to store that information.
  • frombuffer does not copy. The array is a view of the bytes object, and since bytes is immutable the array is read-only. Use bytearray or call .copy() to get a writable array.
  • Byte order matters. Reading little-endian integers as big-endian (">i4") gives wildly different values. Use explicit "<" or ">" in the dtype for files shared across machines.

Raw files: tofile and fromfile

arr.tofile(path) writes the array's bytes straight to a file, and np.fromfile(path, dtype=...) reads them back as a flat 1-D array. It is tobytes and frombuffer with a file in between, and it is the right tool for headerless binary formats such as raw sensor dumps or raw image data. As with raw bytes, the file stores no dtype, shape or byte order, so you must supply them when you read.

fromfile also accepts count (how many items to read) and offset (how many bytes to skip first), which lets you jump past a fixed-size header or read a single record. With sep="," the pair switches to a simple text mode, but np.savetxt and np.loadtxt are the better choice for text.

Python tofile and fromfile

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Notice that the wrong dtype never raises an error: reading the same file as uint8 or as big-endian simply produces different numbers. If a raw file looks like noise, suspect the dtype, the byte order or the shape first. np.memmap (see Memory mapping huge files below) is the better choice when the file is too large to load at once.

How big are the files?

Different formats trade size, speed and portability. Here is the same one-million-element int64 array in several forms.

Python File size comparison

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Binary formats store exactly itemsize * size bytes plus a header. Text is far larger and also slower, because every number must be formatted and parsed. The cheapest way to shrink a file is often a smaller dtype (Data Types): float32 halves the size of float64, and uint8 is eight times smaller than int64. Only do it when the precision is not needed.

flowchart TD
    A["What do you need to store?"] --> B{"How many arrays?"}
    B -->|"One"| C["np.save / np.load (.npy)"]
    B -->|"Several"| D["np.savez or<br/>np.savez_compressed (.npz)"]
    B -->|"Just bytes<br/>for a protocol or blob"| E["tobytes / frombuffer<br/>(remember dtype and shape)"]
    A --> F{"Human-readable<br/>or other tools?"}
    F -->|"Yes"| G["np.savetxt (CSV)"]

Use binary formats for exact, fast storage and text only when people or other programs must read it.

Memory mapping huge files

If an array is bigger than your RAM, np.load("big.npy", mmap_mode="r") returns a numpy.memmap: an array backed by the file, where only the pieces you touch are read from disk. It needs a real file on disk, so this block is read-only:

Python
big = np.load("big.npy", mmap_mode="r")   # nothing loaded yet
window = big[1_000_000:1_000_010]          # reads just this slice from disk

Modes are "r" (read-only), "r+" (read and write back to the file), "w+" (create) and "c" (copy-on-write). A memmap is excellent for reading slices of a large dataset and poor for random access across a slow disk.

Why not pickle?

Python's pickle can serialize NumPy arrays, and np.save will pickle arrays of Python objects. But pickle is risky: loading a pickle can run arbitrary code, so a file from an untrusted source can take over your machine. It is also tied to Python and library versions. That is why np.load refuses pickled data unless you pass allow_pickle=True.

Python np.load refuses pickled object arrays

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Keep arrays numeric when you can: plain numeric .npy and .npz files never need pickle. Only pass allow_pickle=True for files you created yourself.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does a .npy file store besides the array's values?

    The header (128 bytes in the example) records dtype, shape and memory order, so np.load reproduces the array exactly.
  2. 2. You call b = np.frombuffer(data, dtype=np.int32) where data is a bytes object. What is true about b?

    frombuffer shares memory with the buffer. Because bytes is immutable, the array is read-only, and shape must be reapplied with reshape.
  3. 3. Which call saves two named arrays into one compressed archive?

    savez_compressed stores keyword-named arrays in a zip-compressed .npz. savez does the same without compression.
  4. 4. Why does np.load default to allow_pickle=False?

    Unpickling untrusted data is a security risk because pickles can run code. NumPy requires you to opt in explicitly.

Summary

  • np.save/np.load store one array in .npy with dtype and shape; np.savez and np.savez_compressed store several named arrays in .npz.
  • Pass a path for real files; a file-like object such as io.BytesIO works the same way (rewind with seek(0) before loading).
  • tobytes and np.frombuffer convert to and from raw bytes, but you must supply dtype, shape and byte order yourself, and frombuffer gives a read-only view for bytes.
  • Binary files are itemsize * size plus a small header; choose a smaller dtype or compression to shrink them.
  • mmap_mode="r" reads slices of huge files without loading everything.
  • Avoid pickle for untrusted data; keep arrays numeric and leave allow_pickle off.

Next, treat arrays as pictures in Arrays as Images and Plots.