Skip to content
Tutorials Center
My learning

Topics

Lesson 24 of 64 ยท Transforming Data

Creating and Changing Columns

Add and change columns with assignment, assign(), insert(), rename() and drop(), and learn how pandas 3 Copy-on-Write ends chained-assignment surprises.

Analysis is mostly about making new columns from old ones: a total from price and quantity, a flag for large orders, a cleaner version of a messy label. This chapter, "Transforming Data", starts with the mechanics of adding, changing, renaming and removing columns, and with one rule from pandas 3 that changes how you think about copies: Copy-on-Write.

If you need a refresher on picking columns out of a table first, see Selecting Columns and Rows and loc and iloc.

Adding a column by assignment

Assign to a new column name and pandas creates it. The right-hand side can be a single value (repeated on every row), another column, an expression built from columns, or a list or Series of the right length.

Python Assigning new columns

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Assignment either creates the column or replaces it if the name exists. Lists must match the number of rows exactly; otherwise pandas raises a ValueError about the length.

Assigning a Series aligns on the index

When the right-hand side is a Series, pandas matches values to rows by index label, not by position. Rows with no matching label get a missing value.

Python Index alignment when assigning a Series

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

This is usually helpful, and sometimes a trap after filtering or sorting. The Index explains alignment in depth.

Conditional values

To set values for only some rows, use loc with a mask. For two-way or many-way choices, np.where and np.select are usually clearer (covered in Vectorised Operations).

Python Setting values for some rows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

In the first case the new bulk column is missing for rows where the mask was false. In the second, every row gets a value.

assign(): add columns without changing the original

df.assign(name=expr, ...) returns a new DataFrame with the extra columns and leaves the original untouched. Because it returns a DataFrame, you can chain several steps in one readable pipeline. Pass a function (usually a lambda) when a new column needs a column created earlier in the same chain.

Python Chaining with assign()

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The lambda d: receives the DataFrame as it is at that step, including columns added earlier in the chain. That is why tax can use total. Chains like this avoid half-finished intermediate variables, and Building Data Pipelines extends the idea.

To create a column whose name has spaces, unpack a dict: df.assign(**{"total price": ...}).

insert(): choose the position

Assignment appends new columns at the end. df.insert(loc, name, values) places a column at a numeric position and changes the DataFrame in place. It refuses to overwrite an existing name unless you pass allow_duplicates=True.

Python Inserting at a position

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Renaming and dropping

rename changes labels without touching the data. Give it a dict of old-to-new names, or a function that is applied to every label. drop(columns=[...]) removes columns. Both return a new DataFrame.

Python rename and drop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Normalising names to lower_snake_case early in the pipeline makes the rest of the code easier to write, since you can use df.unit_price and avoid quoting. pop removes a column from the DataFrame in place and returns it.

Copy-on-Write in pandas 3

Which operations make a copy, and which share data? In old pandas, the answer depended on obscure rules, and a famous warning, SettingWithCopyWarning, tried to tell you when an assignment might not reach the original. pandas 3 replaces all that with one rule:

Every DataFrame or Series that results from an operation behaves as its own copy. Changing it never changes any other object.

Under the hood pandas still avoids copying data until you modify something (that is the "copy on write"), so this is cheap. For you it means you can never change one object by accident through another. The code below demonstrates it.

flowchart LR
    A["df"] --> B["column = df['x']"]
    A --> C["subset = df[df.x > 1]"]
    B -- "change B" --> D["only B changes"]
    C -- "change C" --> E["only C changes"]
    A --> F["df stays as it was"]

In pandas 3, columns and subsets taken from a DataFrame are independent objects: changing them never changes the source.

Python Copy-on-Write in action

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The last case matters: alias = df does not copy anything, it gives the same object a second name. Use df.copy() when you want an independent table you can edit freely.

Why chained assignment no longer works

Chained assignment means two index operations in a row, with an assignment at the end:

Python
df[df["x"] > 20]["x"] = 0        # first selects rows, then sets on the result
df["x"][df["x"] > 20] = 0        # first gets the column, then sets on the result

In older versions these sometimes worked and sometimes silently did nothing, depending on whether the first step returned a view or a copy; that uncertainty was the SettingWithCopyWarning. In pandas 3 the first step always returns an independent object, so the assignment always changes only that temporary object and never reaches df. pandas emits a ChainedAssignmentError warning when it spots the pattern, as a pointer to the fix.

Python The chained pattern versus loc

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The fix is always the same: do row and column selection in one loc call (df.loc[mask, "col"] = value), or build the new column with assign or np.where. Code you find online that relies on chained assignment (or that adds .copy() just to silence SettingWithCopyWarning) can be simplified in pandas 3. The inplace=True argument is similarly unnecessary; returning new objects is cheap, and chains read better. Common pandas Pitfalls revisits these traps.

Note: Copy-on-Write is always on in pandas 3; there is no option to turn it off. If you are reading a tutorial written for pandas 1 or 2, any advice about "views versus copies" no longer applies.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. What does df.assign(total=lambda d: d['price'] * d['qty']) do to df?

    assign never modifies the original; it returns a new DataFrame. Assign the result to a name to keep it.
  2. 2. You assign a Series with index [0, 2] to a column of a 3-row DataFrame (index 0, 1, 2). What does row 1 get?

    A Series is aligned on its index labels. Label 1 is absent, so that row is missing. A plain list would be placed by position.
  3. 3. In pandas 3, what does df[df['x'] > 20]['x'] = 0 do to df?

    Copy-on-Write makes the first selection an independent object, so the assignment cannot reach df. Use df.loc[df['x'] > 20, 'x'] = 0.
  4. 4. Which method places a new column at a chosen position?

    insert(loc, name, values) adds a column at position loc, in place. Plain assignment and assign append to the end.
  5. 5. After alias = df, you modify alias['x']. What happens to df?

    Plain name assignment does not copy. Copy-on-Write only applies to objects derived by operations such as df['x'] or df[mask]. Use df.copy() for an independent table.

Summary

  • Create or replace a column with df["new"] = ... using a value, list, column expression or aligned Series.
  • Use df.loc[mask, "col"] = value to set values on selected rows in one step.
  • assign(...) returns a new DataFrame, so you can chain steps; use lambda d: to reference earlier steps.
  • insert places a column at a position; rename and drop(columns=...) return new DataFrames; pop removes and returns a column.
  • pandas 3 Copy-on-Write: results of operations are independent objects, and alias = df is the only way two names share data.
  • Chained assignment (df[mask]["col"] = v) never updates df; use a single loc.

Next: Vectorised Operations.