Skip to content
Tutorials Center
My learning

Topics

Lesson 54 of 111 ยท Feature Engineering and Pipelines

Feature Creation and Transformation

Create better model inputs: ratios, interactions, log transforms, binning with KBinsDiscretizer, date and cyclic features, FunctionTransformer and splines.

A model can only use the columns you give it. If the signal sits in a relationship between two columns, in the shape of a skewed distribution, or in the hour of the day, a plain linear model will miss it, and even a flexible model will need more data to find it. Feature engineering means building new columns that make the pattern easy to see. It is often the cheapest way to improve a model.

This lesson covers the most common recipes: ratios and interactions, log transforms, binning, date and time features, cyclic encoding, and smooth spline features. You will also see FunctionTransformer, which turns any function into a scikit-learn step so that the recipe can live inside a pipeline. Scaling, encoding and missing values have their own lessons: Scaling and Normalization, Encoding Categorical Features and Handling Missing Data.

Five ways to make a feature

Ratios Price per square meter beats price and area alone.
Interactions Products let a linear model see "A matters when B is high".
Log transforms Squash long right tails into a friendlier shape.
Bins Turn a number into ranges when only the range matters.
1 / 2

Ratios and differences

Raw columns often hide the quantity that really matters. A house with a price of 300,000 and 100 square meters is a different story from one with the same price and 200 square meters. The ratio price / area states it directly. Differences work the same way: the gap between a loan amount and an income, or between a sensor reading and its baseline.

A few habits keep ratios safe. Guard against division by zero, keep the unit obvious in the column name, and remember that a ratio built from the target is leakage (see Nested Cross-Validation and Data Leakage). The target must never feed a feature.

Interactions

A linear model adds up effects. If advertising helps only for products that are in stock, no single coefficient can say that, but the product ad_spend * in_stock can. Tree models can find such rules alone, yet they need splits and data to do it, while a handcrafted interaction gives even a linear model the idea for free. PolynomialFeatures(interaction_only=True) builds all pairwise products automatically (Polynomial Regression and Nonlinearity).

Log transforms for skewed data

Money, counts, durations and populations usually have a long right tail: many small values and a few huge ones. Those outliers dominate squared-error losses and distance calculations. Taking the logarithm turns multiplicative differences into additive ones and pulls the tail in. Use np.log1p(x) (that is log(1 + x)) when zeros are possible, and np.expm1 to go back.

Python Ratios, interactions and a log transform

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The price was generated as a product of area and quality, which turns into a sum once you take logs. The log of area is therefore the right feature for a linear model of log price, and the engineered set scores better.

If you transform the target, as we did here, remember to transform predictions back before reporting errors in real units. scikit-learn's TransformedTargetRegressor does that for you.

Binning with KBinsDiscretizer

Sometimes only the range of a value matters: age bands in insurance, tax brackets, "new / regular / veteran" customers. KBinsDiscretizer learns bin edges from the training data and can return one-hot columns, so a linear model gets a separate coefficient for each range and can draw a step-shaped curve. The strategy argument chooses the edges: "uniform" (equal width), "quantile" (equal counts) or "kmeans".

Binning throws information away inside each bin, so it is usually a trade: more flexibility for a linear model, less precision. For trees it is mostly pointless, because trees choose their own split points.

FunctionTransformer: any function as a step

FunctionTransformer(func) wraps a plain function so it behaves like a scikit-learn transformer. Because it has nothing to learn, fit does nothing and transform calls your function. Wrapping a recipe in a transformer keeps training and prediction identical, and lets the cross-validation machinery apply it fold by fold.

Python KBinsDiscretizer and FunctionTransformer

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Date and time features

A timestamp is a single number that a model cannot read well. Break it into parts that carry meaning: year, month, day of week, hour, whether it is a weekend, days since some event. pandas gives you these through the .dt accessor (see Dates and Times). Which parts help depends on the problem: electricity demand follows the hour and weekday, retail sales follow the month and holidays.

Cyclic features

Hour 23 and hour 0 are one hour apart, but the numbers 23 and 0 look far apart to a model. The same goes for months, weekdays and compass angles. Mapping a cyclic value onto a circle fixes this: compute sin(2ฯ€ ยท value / period) and cos(2ฯ€ ยท value / period). Two columns together identify the position on the circle, and neighbors across the wrap-around end up close together.

flowchart LR
    H["hour (0 to 23)"] --> S["sin(2ฯ€ ยท hour / 24)"]
    H --> C["cos(2ฯ€ ยท hour / 24)"]
    S --> M["model sees a circle"]
    C --> M

One cyclic number becomes two columns that describe a point on a circle.

Python Date parts and cyclic encoding

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Do not shuffle time-ordered rows when you validate features built from dates. Use a time-aware split (Cross-Validation for Groups and Time) so that no future information reaches the past.

SplineTransformer: smooth flexible curves

Polynomials can swing wildly at the edges. Splines build a curve from many small, smooth, local pieces (B-spline basis functions). SplineTransformer expands one column into several overlapping bumps, and a linear model on top fits a smooth, flexible curve with a stable shape. Raise n_knots for more flexibility and degree for smoother pieces.

Python Splines give a linear model a smooth curve

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

A feature-engineering checklist

  • Start from the domain: ask what quantity a human expert would compute.
  • Build features inside the pipeline so that statistics (bin edges, scaling) are learned on training folds only.
  • Compare against the plain feature set with cross-validation. A new feature that does not help is noise, and noise has a cost.
  • Name features clearly and keep the recipe in code, not in a spreadsheet.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why use np.log1p instead of np.log on a column that can contain zeros?

    np.log(0) gives -inf, which breaks most models. log1p computes log(1 + x), so zero maps to zero, and expm1 undoes it.
  2. 2. Hour 23 and hour 0 should be neighbors. Which encoding achieves this?

    Sine and cosine place the hour on a circle, so the wrap-around from 23 to 0 is a short step. The raw number and its rescaled version keep 23 and 0 at opposite ends.
  3. 3. What does FunctionTransformer learn during fit?

    FunctionTransformer is stateless: fit checks the input and transform calls your function. That is exactly what you want for log, ratio or date recipes.
  4. 4. Which KBinsDiscretizer strategy gives bins with about the same number of samples each?

    quantile places edges at quantiles of the training data. uniform gives equal widths, kmeans uses one-dimensional k-means centers, and equal is not a valid strategy.
  5. 5. A teammate creates a feature from the target column to help the model. What is the problem?

    At prediction time the target is unknown, so any feature built from it gives a score that looks great in validation and collapses in production.

Summary

  • New features make patterns visible: ratios, differences, interactions and log transforms are the first things to try.
  • KBinsDiscretizer turns numbers into ranges (uniform, quantile, kmeans); FunctionTransformer wraps any function as a pipeline step.
  • Break timestamps into parts and encode cyclic values with sine and cosine.
  • SplineTransformer gives a linear model a smooth, local, flexible curve.
  • Judge every feature with cross-validation, and never build a feature from the target.

Next: Text Features.