Skip to content
Tutorials Center
My learning

Topics

Lesson 32 of 49 ยท Working with Real Data

Structured Arrays and Text Data

Store mixed-type records in structured arrays, read CSV text with loadtxt and genfromtxt, process strings with np.char, and compute with datetime64 dates.

A plain NumPy array holds one dtype. Real tables do not: a column of names, a column of ages, a column of weights. NumPy covers this with structured arrays, which pack several named fields into each element. It also reads delimited text into arrays, offers vectorized string functions, and has proper date and time types.

This lesson is the toolbox for turning small text files and mixed records into arrays. For large tables with labels and rich missing-value handling you will eventually reach for pandas, but these NumPy tools are lightweight, fast and often enough. They build on Data Types and Sorting and Searching.

Structured arrays

A structured dtype is a list of (name, type) pairs. Each element of the array is a record; each field behaves like a column you can read or assign as a normal array.

Python A structured array of people

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Key points:

  • The type codes follow the usual rules: "U10" is a Unicode string of up to 10 characters, "i4" a 32-bit integer, "f8" a 64-bit float. Fixed-width strings silently truncate longer text.
  • people["age"] returns a view of one column, so writing to it modifies the records (Views vs Copies).
  • Boolean masks, sorting with order=, and slicing all work on whole records.
  • A recarray adds rec.name attribute access. It is convenient but slower than plain structured arrays, so most code sticks to people["name"].

Reading text with loadtxt and genfromtxt

Both functions parse delimited text into arrays. Use np.loadtxt for clean, regular data. Use np.genfromtxt when the file has missing values, column names or mixed types. They accept a file name or any file-like object, so the examples below use io.StringIO, which pretends a string is a file.

flowchart LR
    A["Text or file<br/>(io.StringIO)"] --> B{"Clean and numeric?"}
    B -->|"Yes"| C["np.loadtxt"]
    B -->|"Missing values,<br/>headers, mixed types"| D["np.genfromtxt"]
    C --> E["2D float array"]
    D --> F["Structured array<br/>or filled array"]

Pick loadtxt for tidy numbers and genfromtxt when the data is messy.

Python Parsing CSV text and saving it back

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Look at the genfromtxt output: the empty qty for "nut" became -1 and the empty price for "washer" became nan. Missing integers have no NaN, so genfromtxt substitutes a default; pass filling_values to choose your own. With names=True the first row becomes the field names, and dtype=None makes NumPy guess each column's type.

np.savetxt writes a 1D or 2D array as text. Give it a fmt for each column (or one for all), a delimiter and an optional header. Remember the file is plain text: it loses dtypes and is slow for big arrays. For binary storage see Saving and Loading Arrays.

Vectorized string functions

Arrays of strings support elementwise operations through np.char (the long-standing module) and np.strings (the newer, faster ufunc-based module in NumPy 2). They mirror Python's str methods: upper, strip, replace, startswith, find, add, str_len and more.

Python String operations on whole arrays

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The default "U" dtype pads every element to the longest string in the array, which wastes memory when lengths vary. StringDType, new in NumPy 2, stores variable-length strings and never truncates. The np.strings functions work on both kinds.

String arrays do not make text processing fast the way numeric arrays do. For heavy text work, pandas string methods or plain Python are usually more convenient. Use NumPy strings when the text is one column of a larger array computation.

Dates and times

datetime64 stores a date or timestamp as an integer count of units since 1970-01-01. The unit is part of the dtype: datetime64[D] for days, [M] for months, [h] for hours, and so on. Subtracting two datetimes gives a timedelta64.

Python Dates, ranges and differences

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Things worth remembering:

  • Casting to a coarser unit (astype("datetime64[M]")) truncates; to a finer unit it is exact.
  • Dividing a timedelta64 by np.timedelta64(1, "D") gives a plain float number of days.
  • NaT ("not a time") is the missing value for dates; test it with np.isnat, as with NaN in Missing Data and NaN.
  • np.busday_count and np.is_busday handle weekdays (Monday to Friday) and accept custom holidays.
  • datetime64 has no time zones. Convert to UTC before storing.

Text arrays in depth: the np.char and np.strings families

The earlier examples used a handful of string functions. There are many more, and they group naturally. Everything below exists in both np.char and, for most functions, the faster np.strings module of NumPy 2.

Job Functions
Case upper, lower, capitalize, title, swapcase
Trim and pad strip, lstrip, rstrip, center, ljust, rjust, zfill
Search find, rfind, count, startswith, endswith, index, isin-style tests with ==
Edit replace, add, multiply, partition, mod
Split and join np.char.split, np.char.join
Classify isalpha, isdigit, isnumeric, isspace, isupper, islower
Compare np.char.equal, less, compare_chararrays
Python Cleaning a column of messy codes

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

partition returns an extra axis of size 3 (before, separator, after), which is why the examples index [:, 0] and [:, 2]. np.char.split returns an object array of Python lists, so it is not faster than a plain loop; prefer partition when you only need one split.

datetime64 in depth

The basics of datetime64 are above. This section covers the parts that matter in real data work: the available units, how mixed units behave, timedelta64 arithmetic, generating date ranges and the business-day tools.

  • Units: Y, M, W, D, h, m, s, ms, us, ns. Mixing units promotes to the finer one.
  • Ranges: np.arange(start, stop, step, dtype="datetime64[D]"), where the step may be a timedelta64.
  • Calendar arithmetic: adding months to a datetime64[M] is clean; adding np.timedelta64(1, "M") to a day-level date is an error because a month has no fixed length.
  • Business days: np.busday_offset, np.busday_count, np.is_busday, with a weekmask and holidays for local calendars.
Python Units, ranges and business days

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The weekday trick works because day 0 of the epoch (1970-01-01) was a Thursday, so adding 3 maps Monday to 0. The more readable route is np.is_busday with a weekmask, or pandas when you need real calendar features.

When you read dates from text, give the dtype explicitly (dtype="datetime64[D]"). Strings such as "2024-03-15" convert directly, but formats such as "15/03/2024" do not: reformat them first or let pandas parse them.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Given a structured array people with a field 'age', what does people['age'] return?

    Field access returns a view into the same memory. Modifying people['age'] modifies the records.
  2. 2. Which function should you use to read CSV text that has empty cells and a header row of column names?

    genfromtxt handles missing values and header names (names=True); loadtxt expects complete, regular data.
  3. 3. What is an advantage of np.dtypes.StringDType over the fixed-width 'U' dtype?

    StringDType (NumPy 2) is variable-width, so one long string does not force every element to be that wide, and assignments never truncate.
  4. 4. What does np.datetime64('2024-03-15') + 30 produce?

    For a day-unit datetime64, integers are interpreted as days. Adding 30 gives 2024-04-14.

Summary

  • A structured dtype packs named, typed fields into each element; arr["field"] returns a column view, and sorting with order= works on records.
  • np.loadtxt reads clean numeric text; np.genfromtxt handles headers, missing values and mixed types. Both accept io.StringIO.
  • np.savetxt writes text with fmt, delimiter and header.
  • np.char and np.strings apply string methods elementwise; StringDType stores variable-length strings in NumPy 2.
  • datetime64 and timedelta64 do date arithmetic, ranges with np.arange, unit casting and business-day counts; NaT marks a missing date.

Next, learn how to store arrays efficiently in Saving and Loading Arrays.