Skip to content
Tutorials Center
My learning

Topics

Lesson 4 of 38 ยท Getting Started

Reading and Writing Data Files

Use scipy.io to read and write MATLAB .mat files, WAV audio and Matrix Market data, and choose between npy, npz, text and sparse formats for your arrays.

Numerical code is only useful if numbers can get in and out. Measurements come from instruments and colleagues, often as MATLAB files or WAV recordings; results have to be saved for the next script, or for a collaborator who uses another language. The module scipy.io reads and writes several scientific formats, and NumPy and scipy.sparse add formats of their own. This lesson shows the ones you will use most, and helps you pick one.

Every example here uses in-memory buffers (io.BytesIO) instead of files on disk, because the code runs in your browser. On your own machine replace the buffer with a file name, for example sio.savemat("results.mat", data); nothing else changes. Both functions accept either a path or a file-like object.

MATLAB files: savemat, loadmat and whosmat

MATLAB and Octave store variables in .mat files, and scipy.io reads and writes the classic format (versions 4 to 7.2). savemat(file, dict) saves a dictionary whose keys become the variable names; loadmat(file) returns a dictionary in the other direction; whosmat(file) lists the variables of a file (name, shape, MATLAB class) without loading the data, which is a cheap way to inspect a large file.

Three details surprise newcomers:

  • MATLAB has no scalars and no true 1-D arrays: everything is at least 2-D. A Python 7 comes back as [[7]], and np.arange(4) comes back as a (1, 4) row vector. Pass squeeze_me=True to loadmat to remove the extra dimensions again.
  • loadmat adds three bookkeeping keys that start with two underscores (__header__, __version__, __globals__). Skip them when you loop over the result.
  • Strings and structs map to character arrays and structured NumPy arrays. For MATLAB structs, struct_as_record=False gives attribute-style access.

Newer MATLAB files saved as version 7.3 are really HDF5 files, and loadmat raises NotImplementedError for them: open those with h5py (or the mat73 package). Options of loadmat worth knowing are variable_names=[...] (load only some variables) and mat_dtype=True (keep MATLAB's floating-point type); savemat accepts do_compression=True and oned_as="row" or "column".

Python Save, inspect and load a MATLAB file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The array of floats comes back bit for bit, the integer arrives as a 1 by 1 matrix, and whosmat found all four variables without reading their contents. In real projects, call squeeze_me=True as the first thing, unless you need the exact MATLAB shapes.

WAV audio files

scipy.io.wavfile reads and writes uncompressed WAV files. wavfile.write(file, rate, data) takes the sample rate in samples per second and a NumPy array; wavfile.read(file) returns (rate, data). Mono audio is a 1-D array, stereo a 2-D array of shape (samples, channels). The array's dtype decides the file's sample format, and read gives it back unchanged:

dtype Meaning
int16 CD-quality integers, range -32768 to 32767
int32 high-resolution integers
uint8 old 8-bit audio, silence at 128
float32 floating point, nominal range -1.0 to 1.0

The most common bug is writing a float array that is not scaled: floats must stay within -1 to 1, and integers must use their full range. Convert a float signal with (x * 32767).astype(np.int16), and the other way round with x / 32768.0.

Python Writing and reading a WAV file

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

For the analysis of such a recording (spectrograms, filters), continue with Spectral Analysis and Peaks and Filtering.

Other formats in scipy.io

scipy.io has readers for several more specialised formats. They follow the same pattern (a function that takes a path or file object), so the documentation of each is all you need once you know the idea:

Module or function Format Typical use
mmread, mmwrite Matrix Market .mtx, a text format for sparse (and dense) matrices exchanging sparse matrices with other tools and research collections
hb_read, hb_write Harwell-Boeing older sparse-matrix collections
netcdf_file NetCDF 3 climate and ocean data (for NetCDF 4 use the netCDF4 package)
arff.loadarff ARFF machine-learning data sets from the Weka tool
FortranFile unformatted Fortran binary output of simulation programs written in Fortran
readsav IDL save files astronomy data

Matrix Market is the one you meet most often. A short read-only example (it uses a thread pool internally, which is why it is not run on this page):

Python
from scipy.io import mmread, mmwrite
from scipy import sparse

A = sparse.random_array((5, 5), density=0.3, format="coo")
mmwrite("matrix.mtx", A)        # a text file: header line, size line, one line per nonzero
B = mmread("matrix.mtx")        # a sparse array again (use spmatrix=False for arrays)

NumPy and sparse formats

You do not need scipy.io for data that only Python will read. The native formats are faster, smaller and keep dtypes and shapes exactly:

Function Format When to use
np.save / np.load .npy, one array quick storage of one array
np.savez_compressed / np.load .npz, several named arrays, compressed a few arrays that belong together
np.savetxt / np.loadtxt plain text small tables that people or spreadsheets must read
sparse.save_npz / sparse.load_npz .npz for a sparse matrix any sparse matrix, keeps the format (CSR, CSC...)

sparse.save_npz stores only the nonzero entries, so a matrix that would need 80 KB as a dense array can fit in a few kilobytes.

Python npy, npz, text and sparse files

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The 100 by 100 sparse matrix with 200 nonzero entries occupies about 3 KB on disk against 80 KB for the dense array. See Sparse Matrices for how the formats work.

Never load .npy/.npz files with allow_pickle=True, or unpickle .mat object arrays, from sources you do not trust. Pickle can execute arbitrary code when it is loaded. The defaults refuse it; keep them.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. You load a MATLAB file and the scalar you saved as 7 arrives as [[7]]. What is the simplest fix?

    MATLAB arrays always have at least two dimensions. squeeze_me=True removes the dimensions of length 1 so scalars and vectors look like ordinary Python data.
  2. 2. What does whosmat do?

    whosmat only reads the headers, so it is cheap even for files of several gigabytes.
  3. 3. A float64 audio array with values between -1 and 1 is written with wavfile.write. What do you get back from wavfile.read?

    The dtype of the array decides the sample format of the file, and read returns the same dtype. Check that floats stay in the range -1 to 1.
  4. 4. Which format is the best choice to store one large sparse matrix that only Python programs will read?

    save_npz stores only the nonzero entries and the format information, so the file stays small and the matrix returns as the same sparse type.

Summary

  • scipy.io.savemat, loadmat and whosmat exchange variables with MATLAB and Octave (classic formats; version 7.3 files need h5py).
  • MATLAB data is always at least 2-D; use squeeze_me=True to get ordinary scalars and vectors back, and skip the keys starting with __.
  • wavfile.read and wavfile.write handle WAV audio: the array dtype is the sample format, and shape is (samples, channels) for stereo.
  • mmread, mmwrite, netcdf_file, arff.loadarff and others cover specialised formats.
  • Use np.save, np.savez_compressed, np.savetxt and sparse.save_npz when only Python needs the data.
  • Never unpickle data from sources you do not trust.

Next: Matrices, Solving Systems and Inverses.