Skip to content
Tutorials Center
My learning

Topics

Lesson 27 of 45 ยท Data and Files

Cleaning and Preparing Data

Clean data in MATLAB and Octave: find missing values, detect outliers, fill gaps with interpolation, smooth with movmean, normalize, sort and group, with Python stand-ins.

Real measurements are messy. Sensors drop readings, someone types 120 instead of 12.0, and two columns are measured in different units. Before you fit a model or draw a conclusion, you clean and prepare the data. This is usually the longest part of an analysis, and it pays off: a single bad value can change a mean or a fit completely.

This lesson covers the standard steps: find missing values, detect outliers, fill gaps, smooth noise, rescale, and sort or group. MATLAB and Octave are not available in your browser here, so the MATLAB code is read-only; each step has a runnable Python/NumPy stand-in that produces the same numbers. Where a function exists only in newer releases, the lesson says so and shows a portable alternative.

A cleaning checklist

Missing Find NaN values first.
Outliers Spot values that do not fit.
Fill Interpolate or remove.
Rescale Normalize before comparing.
1 / 2

Missing values

MATLAB marks a missing number with NaN ("not a number"). Reading an empty cell from a CSV file produces one (see Reading and Writing Files). NaN is contagious: any arithmetic with it gives NaN, and NaN == NaN is false, so you must test with isnan.

MATLAB
x = [4 5 NaN 7 120 6 NaN 5 4 6];
sum(isnan(x))                 % 2 missing values
mean(x)                       % NaN, the missing values spoil the mean
mean(x, 'omitnan')            % ignores them (MATLAB R2015a and later)
xclean = x(~isnan(x));        % logical indexing: keep only real numbers
xclean2 = rmmissing(x);       % same result (MATLAB R2016b and later)

The 'omitnan' flag works for sum, max, std and others in MATLAB. In Octave use nanmean-style helpers from the statistics package or filter first with x(~isnan(x)), which works everywhere. Logical indexing is covered in Logical Indexing and find; NaN-aware statistics also appear in Built-in Math and Statistics.

Python Find and drop missing values

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Outliers

An outlier is a value far from the rest. It may be an error (120 where 12.0 was meant) or a rare but real event, so look before you delete. Two common rules:

  • Z-score rule: flag values more than 3 standard deviations from the mean. Simple, but the outlier itself inflates the mean and standard deviation, which can hide it.
  • Median rule: use the median and the median absolute deviation (MAD), which are barely affected by extreme values. MATLAB's isoutlier uses this by default: a value is an outlier if it is more than three scaled MADs from the median, where scaled MAD is 1.4826 * median(abs(x - median(x))).
MATLAB
xc = x(~isnan(x));
isoutlier(xc)                              % logical mask (MATLAB R2017a and later)
filloutliers(xc, 'linear')                 % replace them (R2017a and later)

% portable version:
m   = median(xc);
mad = 1.4826 * median(abs(xc - m));
mask = abs(xc - m) > 3 * mad;
xc(mask)                                   % shows the offending value(s)
Python Detect the outlier with the median rule

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

You will see that the 120 is caught by the median rule even though its z-score stays under 3: the outlier pulled the mean and standard deviation toward itself. This is why the median rule is the safer default for small samples.

Filling gaps by interpolation

After removing outliers or finding NaN, you can leave gaps, delete rows, or fill them. For ordered data such as a time series, interpolation estimates a missing point from its neighbours. Use interp1 (works in MATLAB and Octave) to evaluate a line through the known points at every index:

MATLAB
y = x;
y(abs(x - m) > 3*mad) = NaN;               % blank out the outlier
idx = 1:numel(y);
ok = ~isnan(y);
filled = interp1(idx(ok), y(ok), idx);     % linear interpolation at every index
% or, in MATLAB R2016b and later:
filled = fillmissing(y, 'linear');

Linear interpolation draws straight lines between the surrounding known points; 'previous' repeats the last value, 'nearest' copies the closest, and 'spline' fits a smooth curve (more in Polynomials and Curve Fitting). Gaps at the very start or end have no neighbour on one side, so interp1 returns NaN there unless you pass 'extrap'. Do not interpolate across large gaps; the filled values are invented data.

Python Blank the outlier, then interpolate

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Smoothing with a moving average

Noise hides trends. A moving average replaces each value by the mean of a window centered on it. movmean(x, 3) uses the point and its two neighbours; at the ends, the window shrinks to what exists.

MATLAB
smooth = movmean(filled, 3);               % MATLAB R2016a and later
smooth = filter(ones(1,3)/3, 1, filled);   % portable, but delays the signal by one sample

A larger window smooths more but also blunts real peaks. filter is causal (it looks only backward), so its result is shifted; movmean is centered, so peaks stay in place. The portable centered version is conv(filled, ones(1,3)/3, 'same'), with the end points distorted by the implicit zeros.

Python Moving average with shrinking edge windows

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Normalizing and rescaling

Columns measured in different units (meters and kilograms, say) are not comparable, and many methods work badly when one column is a thousand times larger than another. Two standard fixes:

MATLAB
z = (x - mean(x)) / std(x);                % z-score: mean 0, standard deviation 1
r = (x - min(x)) / (max(x) - min(x));      % rescale to the range [0, 1]
z = normalize(x);                          % MATLAB R2018a and later (z-score by default)
r = normalize(x, 'range');                 % to [0, 1]

For a matrix, mean and std work per column by default, and implicit expansion subtracts each column's mean from that column (see Arithmetic: Matrix vs Element-wise). std divides by n - 1 unless told otherwise. Compute the mean and standard deviation on the training data only and reuse them for new data, otherwise you leak information.

Sorting and grouping

sort returns the sorted values and, optionally, where they came from; sortrows orders a matrix by one or more columns (see Reshaping, Joining and Sorting). To summarize by group, unique gives the group labels and accumarray adds up or averages values per label:

MATLAB
group = [1 2 1 3 2 1];
value = [10 20 30 40 50 60];
[labels, ~, g] = unique(group);            % g maps each row to its group number
totals = accumarray(g(:), value(:))        % [100; 70; 40]
means  = accumarray(g(:), value(:), [], @mean)   % [33.33; 35; 40]
Python Normalize, sort and group

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Keep the original data

Work on a copy and keep the raw data untouched, so you can redo any step. Put the cleaning code in a script that goes from raw file to cleaned variable, so the result is repeatable and others can check what you did. And plot before and after (see 2-D Plots): a picture shows at once whether the fill or the smoothing did something unreasonable.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. Why does x == NaN never find the missing values?

    Every comparison with NaN is false, so use isnan(x) to find missing values.
  2. 2. Why is the median rule often better than the z-score rule for a small sample?

    The median and MAD barely move when one extreme value is present, whereas the mean and standard deviation are pulled toward it.
  3. 3. What does interp1(idx(ok), y(ok), idx) do?

    It evaluates the interpolant built from the known points at every index, which fills the gaps.
  4. 4. How does movmean(x, 3) treat the first element?

    At the ends the window shrinks, so the first value is the mean of the first two elements.
  5. 5. Which is a sound habit when cleaning data?

    A script from raw data to cleaned data is repeatable and auditable. Overwriting or blindly deleting loses information.

Summary

  • Missing values are NaN; test with isnan, then remove (x(~isnan(x)), rmmissing) or fill them.
  • Detect outliers with the median/MAD rule (isoutlier, or three scaled MADs by hand); look before deleting.
  • Fill gaps with interp1 or fillmissing, and only across small gaps.
  • movmean smooths with a centered window; filter is portable but delays the signal.
  • Normalize with a z-score or range rescale before comparing columns; reuse training statistics on new data.
  • sort, sortrows, unique and accumarray order and group data.
  • Keep the raw data and script every step.

Next, put cleaned data to work in Polynomials and Curve Fitting.