Skip to content
Tutorials Center
My learning

Topics

Lesson 34 of 48 ยท MLlib

MLlib Overview and Pipelines

Learn Spark MLlib: Transformer, Estimator and Pipeline, VectorAssembler, train/test split, cross-validation and saving models, with runnable stand-ins.

Training a model on a laptop is easy until the data stops fitting in memory. MLlib is Spark's machine-learning library: it implements common algorithms (regression, classification, clustering, recommendation) and the plumbing around them (feature preparation, evaluation, tuning) so that the work is spread across a cluster. It reads and writes ordinary DataFrames, which means the same tools you use for ETL also feed your models.

This lesson explains MLlib's three core ideas, Transformer, Estimator and Pipeline, then shows a complete flow: assemble features, train, evaluate on held-out data, tune with cross-validation and save the result. Spark cannot run on this site, so the real PySpark code is read-only and the TryIts are small stand-ins built with Python and NumPy that follow the same API shape. Make sure you are comfortable with DataFrames (DataFrames: Creating and Inspecting).

Two APIs, one to use

MLlib has two packages. pyspark.mllib is the old RDD-based API; it is in maintenance mode. pyspark.ml is the DataFrame-based API and the one to learn and use. When people say "Spark ML" or "MLlib" today they mean pyspark.ml.

Spark's built-in algorithms are best for large tabular data with standard models (linear and logistic regression, decision trees, random forests, gradient-boosted trees, k-means, ALS recommendations). For deep learning or very specialized models you normally train elsewhere and only use Spark to prepare data or run predictions at scale.

Transformer, Estimator, Pipeline

Three building blocks cover almost everything:

  • A Transformer turns one DataFrame into another by adding columns. It has a transform(df) method. VectorAssembler is a transformer, and so is a trained model, which adds a prediction column.
  • An Estimator learns something from data. It has a fit(df) method that returns a Transformer (the fitted model). LinearRegression is an estimator; LinearRegressionModel is what it returns. StandardScaler is an estimator too, because it must learn means and spreads before it can scale.
  • A Pipeline chains stages (transformers and estimators) in order. pipeline.fit(df) fits each estimator in turn and returns a PipelineModel, itself a transformer that applies every fitted stage.
flowchart LR
    D["Raw DataFrame"] --> A["VectorAssembler<br/>(Transformer)"]
    A --> S["StandardScaler<br/>(Estimator)"]
    S --> L["LinearRegression<br/>(Estimator)"]
    L --> M["PipelineModel<br/>(Transformer)"]
    M --> P["Predictions<br/>DataFrame"]

Fitting a pipeline fits each estimator in order; the resulting PipelineModel applies all fitted stages to new data.

Why bother with a pipeline? Because the exact same steps must run on training data and on the data you predict later. If you scale with numbers learned from the training set, new rows must be scaled with those same numbers. A pipeline stores that and removes a whole class of bugs.

Features live in one vector column

MLlib algorithms expect two columns: a numeric label and a features column that holds all inputs as a single vector. You build the vector with VectorAssembler:

Python
from pyspark.ml.feature import VectorAssembler

assembler = VectorAssembler(inputCols=["rooms", "area", "age"], outputCol="features")
prepared = assembler.transform(houses)
prepared.select("features", "price").show(3, truncate=False)
# +---------------+-------+
# |features       |price  |
# +---------------+-------+
# |[3.0,85.0,12.0]|210000 |
# ...

Text categories must become numbers first: StringIndexer maps each category to an index and OneHotEncoder turns indexes into sparse vectors. Other common stages are StandardScaler, Imputer (fill missing values), Tokenizer/HashingTF/IDF for text, and Bucketizer.

The stand-in builds the same three roles (assembler, scaler, regression) as small classes with fit and transform. You can check that fitting a pipeline calls fit on estimators only, in order.

Python A miniature Transformer / Estimator / Pipeline

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Train, test and evaluate

A model that is judged on the data it learned from looks better than it is. The remedy is a train/test split: fit on one part and measure on the part the model has never seen.

Python
train, test = houses.randomSplit([0.8, 0.2], seed=42)

from pyspark.ml import Pipeline
from pyspark.ml.regression import LinearRegression
from pyspark.ml.evaluation import RegressionEvaluator

pipeline = Pipeline(stages=[assembler, LinearRegression(featuresCol="features", labelCol="price")])
model = pipeline.fit(train)

predictions = model.transform(test)
rmse = RegressionEvaluator(labelCol="price", predictionCol="prediction",
                           metricName="rmse").evaluate(predictions)
print(f"RMSE on test data: {rmse:,.0f}")

randomSplit takes weights and a seed; without the seed, a rerun may split differently (and note that the split also depends on partitioning, so cache the DataFrame if you need exactly repeatable splits). Evaluators exist for regression (rmse, mae, r2), binary classification (areaUnderROC) and multiclass classification (accuracy, f1).

Hyperparameter tuning with cross-validation

Algorithms have hyperparameters: settings you choose before training, such as regularization strength (regParam) or tree depth. To pick them without peeking at the test set, MLlib offers k-fold cross-validation. The training data is cut into k folds; each fold takes a turn as the validation set while the model trains on the other k-1. Averaging the scores gives a fair estimate for each setting. The best setting is then refit on the full training set.

Python
from pyspark.ml.tuning import ParamGridBuilder, CrossValidator

lr = LinearRegression(featuresCol="features", labelCol="price")
pipeline = Pipeline(stages=[assembler, lr])

grid = (ParamGridBuilder()
        .addGrid(lr.regParam, [0.0, 0.1, 1.0])
        .addGrid(lr.elasticNetParam, [0.0, 0.5])
        .build())                                         # 3 x 2 = 6 combinations

cv = CrossValidator(estimator=pipeline,
                    estimatorParamMaps=grid,
                    evaluator=RegressionEvaluator(labelCol="price", metricName="rmse"),
                    numFolds=3,
                    parallelism=2,
                    seed=42)

cv_model = cv.fit(train)           # trains 6 x 3 = 18 models, then refits the best on all of train
print(cv_model.avgMetrics)         # one average score per grid point
best = cv_model.bestModel          # a PipelineModel

Cost grows quickly: combinations times folds times one training run. TrainValidationSplit is the cheaper variant with one split instead of k folds. Cache the training DataFrame before tuning (see Caching and Persistence), because every fit rereads it.

The stand-in tunes the strength of ridge regression on noisy data using 4-fold cross-validation. It picks the setting with the lowest average validation error.

Python k-fold cross-validation to choose a regularization strength

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Saving and loading models

Training is expensive, so save the fitted pipeline and reuse it in another job, such as a streaming query that scores events:

Python
model.write().overwrite().save("s3://models/house-price/v1")

from pyspark.ml import PipelineModel
loaded = PipelineModel.load("s3://models/house-price/v1")
loaded.transform(new_data).select("prediction").show()

Models are stored as Parquet and JSON metadata. The format is tied to the Spark version family, so record the Spark version next to the model. Saving the whole pipeline (not just the last model) guarantees preprocessing is applied the same way at prediction time.

Real versus stand-in. Real MLlib stages operate on distributed DataFrames, with solvers that run iterative algorithms (for example L-BFGS or normal equations) by aggregating over partitions, and CrossValidator can train several models in parallel. The stand-ins hold everything in NumPy arrays on one machine and use direct linear algebra. They reproduce the API shape (fit returns a model, a pipeline fits stages in order, cross-validation averages fold scores), not the distributed execution.

Text features: Tokenizer, TF-IDF and Word2Vec

Models need numbers, so text must be turned into vectors. MLlib offers a chain of small stages that you can place in a pipeline in front of any classifier (see MLlib Classification).

Python
from pyspark.ml.feature import Tokenizer, StopWordsRemover, HashingTF, IDF, Word2Vec

tokenizer = Tokenizer(inputCol="text", outputCol="words")                  # lowercase, split on whitespace
remover = StopWordsRemover(inputCol="words", outputCol="clean")             # drop "the", "and", ...
tf = HashingTF(inputCol="clean", outputCol="tf", numFeatures=1 << 18)       # hashed term counts
idf = IDF(inputCol="tf", outputCol="features", minDocFreq=2)                # estimator: learns idf weights

text_pipeline = Pipeline(stages=[tokenizer, remover, tf, idf])
features = text_pipeline.fit(docs).transform(docs)
  • Tokenizer splits on whitespace; RegexTokenizer splits with a pattern and is the better choice for punctuation.
  • StopWordsRemover removes very common words using a language list.
  • HashingTF maps each word to a position with a hash function and counts occurrences, so it needs no vocabulary pass and keeps memory fixed. Different words can collide in the same slot; a large numFeatures (a power of two) makes that rare. CountVectorizer instead learns an exact vocabulary and lets you map indexes back to words.
  • IDF (inverse document frequency) down-weights words that appear in many documents. The score for a word is its count in the document multiplied by log((N + 1) / (df + 1)), where N is the number of documents and df the number containing the word. Rare, specific words get big weights; words that appear everywhere get close to zero.
  • Word2Vec learns a dense vector (for example 100 numbers) per word from the contexts in which it appears, so similar words get nearby vectors. The model averages the word vectors of a document into one document vector, and findSynonyms("spark", 5) lists the closest words. It needs a lot of text to be useful.
Python Term frequency and IDF by hand

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. MLlib's HashingTF uses a murmur3 hash into sparse vectors and IDF runs as a distributed aggregation; Word2Vec trains a skip-gram neural model across partitions. The stand-in computes exact counts and the same smoothed IDF formula on four sentences, which is enough to see why rare words get larger weights. It does not train word vectors.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does an Estimator's fit() method return?

    fit() learns from data and returns a Model, which is a Transformer. Calling transform() on it adds columns such as prediction.
  2. 2. Why use a Pipeline instead of applying the stages by hand?

    A PipelineModel stores what each stage learned, such as scaling statistics, so new data is processed exactly like the training data.
  3. 3. What does VectorAssembler produce?

    MLlib algorithms read one vector column, usually called features. VectorAssembler combines the input columns into it.
  4. 4. In 3-fold cross-validation with a grid of 4 parameter combinations, how many models are trained before the final refit?

    Each of the 4 combinations is trained once per fold: 4 x 3 = 12. The best combination is then refit on all training data.

Summary

  • Use pyspark.ml, the DataFrame-based API; pyspark.mllib is in maintenance mode.
  • A Transformer has transform, an Estimator has fit and returns a Transformer, and a Pipeline chains stages into a PipelineModel.
  • Algorithms read a label column and one vector column of features, built with VectorAssembler.
  • Judge models on held-out test data (randomSplit with a seed) using an evaluator such as RegressionEvaluator.
  • CrossValidator with a ParamGridBuilder tries every combination over k folds; cost is combinations times folds.
  • Save and load the whole pipeline with write().save() and PipelineModel.load().

Next: make your jobs fast with Performance Tuning.