Skip to content
Tutorials Center
My learning

Topics

Lesson 36 of 48 ยท MLlib

MLlib Regression: Linear Regression Walkthrough

Predict numbers with Spark MLlib LinearRegression: feature vectors, train/test split, coefficients, RMSE, R2, residuals and ridge, lasso and elastic net.

Regression predicts a number: the price of a house, tomorrow's demand, the minutes a delivery will take. Linear regression is the simplest and most useful starting point. It models the target as a weighted sum of the inputs plus a constant, and learns the weights that make the predictions as close as possible to the real values.

This lesson walks through a complete Spark MLlib regression job: building the feature vector, fitting LinearRegression, reading the coefficients and the training summary, scoring with RMSE and R2 on held-out data, and using regularization to keep the model honest. It builds on MLlib Overview and Pipelines. Spark cannot run on this site, so the PySpark code is read-only; the TryIts are plain-Python stand-ins that implement the same math on a small data set.

The idea in one formula

For two inputs the model is price = w1 * rooms + w2 * area + b. Training chooses w1, w2 and the intercept b to minimize the mean squared error between predictions and true values. Squaring makes big misses count more and gives a smooth function that has a closed-form answer (the normal equations) as well as iterative solvers. For a few thousand features Spark uses the normal equations; for more it uses the L-BFGS optimizer, which only needs to sum gradients over partitions and therefore scales with the number of rows.

Building and fitting the model

Python
from pyspark.ml import Pipeline
from pyspark.ml.feature import VectorAssembler
from pyspark.ml.regression import LinearRegression

houses = spark.createDataFrame([
    (2, 50.0, 110000.0), (3, 70.0, 150000.0), (3, 80.0, 165000.0),
    (4, 95.0, 190000.0), (4, 110.0, 215000.0), (5, 120.0, 240000.0),
    # ... more rows
], ["rooms", "area", "price"])

assembler = VectorAssembler(inputCols=["rooms", "area"], outputCol="features")
lr = LinearRegression(featuresCol="features", labelCol="price", predictionCol="predicted_price")

train, test = houses.randomSplit([0.8, 0.2], seed=7)
model = Pipeline(stages=[assembler, lr]).fit(train)

lr_model = model.stages[-1]
print("coefficients:", lr_model.coefficients)     # [w_rooms, w_area]
print("intercept:", lr_model.intercept)

The label column here is price and must be numeric. The coefficient list lines up with inputCols, so reading them is straightforward: w_area = 1,700 means one extra square meter adds about 1,700 to the predicted price while the other inputs stay fixed. Compare coefficients only when features are on comparable scales; set standardization=True (the default) so the solver scales features internally and reports coefficients in the original units.

Evaluating on unseen data

The training summary reports how well the model fit the data it saw:

Python
s = lr_model.summary
print(s.rootMeanSquaredError, s.r2)

That number is optimistic. The honest score comes from the test split:

Python
from pyspark.ml.evaluation import RegressionEvaluator

predictions = model.transform(test)
for metric in ["rmse", "mae", "r2"]:
    value = RegressionEvaluator(labelCol="price", predictionCol="predicted_price",
                                metricName=metric).evaluate(predictions)
    print(metric, round(value, 3))

What the three metrics mean:

  • RMSE (root mean squared error) is in the same unit as the label. An RMSE of 9,000 means "typical errors are around 9,000 euros". It punishes large errors.
  • MAE (mean absolute error) is the average size of the error, less sensitive to a few big misses.
  • R2 (coefficient of determination) compares your model to the trivial model that always predicts the mean. 1.0 is perfect, 0.0 is no better than the mean, and negative means worse. A high R2 does not mean the model is right, only that it explains variance; always look at residuals too.

Look at the residuals (price - predicted_price). If they scatter randomly around zero the linear model is reasonable. If they bend (large for expensive houses, small for cheap ones) a feature transformation (for example the log of the price) or a tree model may fit better.

Python Linear regression by the normal equations, with RMSE and R2

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Regularization: ridge, lasso and elastic net

When there are many features, or features are strongly correlated, plain least squares can fit the noise: weights become huge and unstable and test error rises. Regularization adds a penalty on the size of the weights:

Setting Penalty Effect
regParam=0 none ordinary least squares
regParam>0, elasticNetParam=0.0 L2 (ridge) shrinks all weights smoothly
regParam>0, elasticNetParam=1.0 L1 (lasso) drives some weights to exactly zero, a form of feature selection
regParam>0, 0<elasticNetParam<1 a mix (elastic net) both effects
Python
lasso = LinearRegression(featuresCol="features", labelCol="price",
                         regParam=0.5, elasticNetParam=1.0)
m = lasso.fit(assembler.transform(train))
print(m.coefficients)          # irrelevant features show up as 0.0

Pick regParam and elasticNetParam with CrossValidator and a ParamGridBuilder exactly as shown in the overview lesson, scoring with rmse. A scaling step (StandardScaler) is not needed for LinearRegression because of standardization, but it matters for algorithms that do not do it themselves.

Python How ridge shrinks the weights

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. Real LinearRegression runs on distributed DataFrames with the normal-equation or L-BFGS/OWLQN solvers, handles sparse vectors, weights and standardization, and reports a rich summary (pValues, tValues for the normal solver). The stand-ins solve tiny systems in one process with Gaussian elimination. The metrics (RMSE, R2) and the ridge idea are the same.

Beyond straight lines

If residuals show curves, there are two simple remedies. Create new feature columns (area * area, log(area), an interaction such as rooms * area) before the assembler; the model stays linear in its weights. Or swap the estimator for RandomForestRegressor, GBTRegressor or DecisionTreeRegressor, which capture non-linear patterns and interactions without feature engineering, at the cost of readable coefficients. Because the pipeline shape is identical, you can compare them with one evaluator on the same test split.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. A regression model has RMSE = 9,000 on a house-price test set. What does this tell you?

    RMSE is in the same unit as the label, so it reads as a typical error size, with big misses weighted more.
  2. 2. What does R2 = 0 mean?

    R2 compares the model with the mean-only baseline. 1 is perfect, 0 equals the baseline, negative is worse than the baseline.
  3. 3. Which setting gives lasso (L1) regularization in LinearRegression?

    elasticNetParam=1.0 selects a pure L1 penalty, which can push weights to exactly zero.
  4. 4. Why evaluate on a test split instead of using the training summary?

    Only rows the model has never seen show how it will perform in practice.

Summary

  • Linear regression predicts a number as a weighted sum of features; LinearRegression learns the weights by least squares.
  • Build one features vector with VectorAssembler, split with randomSplit, and read coefficients and intercept from the fitted model.
  • Judge with RegressionEvaluator on held-out data: RMSE (error in label units), MAE and R2 (compared with the mean baseline).
  • Inspect residuals, not just R2, to see whether a straight line is enough.
  • Regularization (regParam, elasticNetParam) gives ridge, lasso or elastic net and stabilizes models with many or correlated features.

Next: find structure without labels in MLlib Clustering.