Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 48 ยท MLlib

MLlib Classification: Logistic Regression and ROC-AUC

Train a binary classifier with Spark MLlib: StringIndexer, VectorAssembler, LogisticRegression, thresholds, BinaryClassificationEvaluator and ROC-AUC.

Classification predicts a category: will this customer churn, is this email spam, is this transaction fraud? When there are two outcomes it is called binary classification, and the most common first model is logistic regression. Despite the name it is a classifier: it turns a weighted sum of the inputs into a probability between 0 and 1, and you pick a threshold to turn that probability into a yes or no.

This lesson walks through a full Spark MLlib classification job: preparing categorical and numeric columns, training LogisticRegression, reading probabilities, and judging the model with ROC-AUC and a confusion matrix. It assumes the vocabulary from MLlib Overview and Pipelines (Transformer, Estimator, Pipeline). Spark cannot run on this site, so the PySpark code is read-only. The runnable examples are plain-Python stand-ins that implement the same math on a tiny data set.

The data and the goal

Imagine a table of telecom customers with a usage number, a contract type (a string) and a churned flag:

Python
customers = spark.createDataFrame([
    (12.0, "monthly", 1), (45.0, "yearly", 0), (7.5, "monthly", 1),
    (60.0, "yearly", 0), (20.0, "monthly", 0), (5.0, "monthly", 1),
    # ... thousands more rows
], ["hours", "contract", "churned"])

The label column must be numeric: 0.0 and 1.0 for binary problems. The inputs must end up in one vector column. Strings cannot go into a vector, so they need encoding first.

Preparing features

Three stages do the preparation:

Python
from pyspark.ml import Pipeline
from pyspark.ml.feature import StringIndexer, OneHotEncoder, VectorAssembler

indexer = StringIndexer(inputCol="contract", outputCol="contract_idx", handleInvalid="keep")
encoder = OneHotEncoder(inputCols=["contract_idx"], outputCols=["contract_vec"])
assembler = VectorAssembler(inputCols=["hours", "contract_vec"], outputCol="features")
  • StringIndexer is an estimator: it learns the distinct strings and maps each to an index (the most frequent string gets 0). handleInvalid="keep" puts unseen strings into an extra bucket instead of failing at prediction time.
  • OneHotEncoder turns the index into a sparse vector, so the model does not think "yearly = 1" is twice "monthly = 0.5".
  • VectorAssembler combines the numeric column and the encoded vector into features.

If you already have a string label, index it too: StringIndexer(inputCol="status", outputCol="label").

Training logistic regression

Python
from pyspark.ml.classification import LogisticRegression

lr = LogisticRegression(featuresCol="features", labelCol="churned",
                        maxIter=50, regParam=0.01, elasticNetParam=0.0)

pipeline = Pipeline(stages=[indexer, encoder, assembler, lr])
train, test = customers.randomSplit([0.8, 0.2], seed=42)
model = pipeline.fit(train)

predictions = model.transform(test)
predictions.select("churned", "probability", "prediction").show(3, truncate=False)
# +-------+----------------------------------------+----------+
# |churned|probability                             |prediction|
# +-------+----------------------------------------+----------+
# |1      |[0.18325406,0.81674594]                 |1.0       |
# |0      |[0.93418851,0.06581149]                 |0.0       |
# +-------+----------------------------------------+----------+

The model adds three columns: rawPrediction (the raw score), probability (a vector holding the chance of class 0 and class 1) and prediction (the class after applying the threshold, 0.5 by default). regParam adds a penalty on large weights to reduce overfitting; elasticNetParam mixes L2 (0.0) and L1 (1.0) penalties, with L1 pushing some weights to exactly zero. After fitting, the last pipeline stage holds the learned numbers:

Python
lr_model = model.stages[-1]
print(lr_model.coefficients, lr_model.intercept)

A positive coefficient means a larger value of that feature raises the churn probability.

Python Logistic regression from scratch with gradient descent

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Evaluating: ROC-AUC and accuracy

Accuracy (the share of right answers) is misleading when classes are unbalanced: a model that always says "no fraud" is 99.9% accurate and useless. A better single number is the area under the ROC curve (AUC). The ROC curve plots the true-positive rate against the false-positive rate as the threshold moves from 1 down to 0. The area has a friendly meaning: it is the probability that a randomly chosen positive example gets a higher score than a randomly chosen negative one. 0.5 is a coin flip, 1.0 is a perfect ranking.

Python
from pyspark.ml.evaluation import BinaryClassificationEvaluator, MulticlassClassificationEvaluator

auc = BinaryClassificationEvaluator(labelCol="churned", rawPredictionCol="rawPrediction",
                                    metricName="areaUnderROC").evaluate(predictions)
acc = MulticlassClassificationEvaluator(labelCol="churned", predictionCol="prediction",
                                        metricName="accuracy").evaluate(predictions)
print(f"AUC = {auc:.3f}, accuracy = {acc:.3f}")

predictions.groupBy("churned", "prediction").count().show()   # the confusion matrix

The confusion matrix counts four outcomes: true positives, true negatives, false positives (alarm, but fine) and false negatives (missed churner). Which error is worse depends on the business. If a retention call is cheap and a lost customer is expensive, lower the threshold with lr.setThreshold(0.3) so more customers are flagged. Remember that AUC does not depend on any threshold, while precision, recall and accuracy do. metricName="areaUnderPR" is another option that suits very rare positives.

Spark's BinaryClassificationEvaluator also needs the same discipline as every model: evaluate on the test split, and use cross-validation when you tune regParam.

Python AUC, accuracy and the confusion matrix

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Real versus stand-in. MLlib's LogisticRegression fits by the L-BFGS optimizer, aggregating gradients over partitions, and supports regularization, weights and multiclass (softmax) models. The stand-in uses plain gradient descent on a dozen rows and no regularization. The formulas (sigmoid, log loss, threshold, AUC as pair ranking) are the real ones; the scale and the optimizer are not.

Other classifiers

Because every MLlib classifier has the same fit / transform shape, switching models is a one-line change in the pipeline: DecisionTreeClassifier, RandomForestClassifier, GBTClassifier (binary only), NaiveBayes and LinearSVC. Tree models need no feature scaling and handle non-linear patterns; logistic regression gives interpretable coefficients and is a strong baseline. Compare them with the same evaluator on the same test split.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What does the probability column of a fitted LogisticRegression model hold?

    For binary problems it is a two-element vector: P(class 0) and P(class 1). prediction is derived from it using the threshold.
  2. 2. Why is AUC often better than accuracy for imbalanced data?

    A model that predicts the majority class always can have high accuracy; its AUC would be 0.5.
  3. 3. What does StringIndexer do?

    It is an estimator that learns the vocabulary, with the most frequent value getting index 0.
  4. 4. You want to catch more churners and accept more false alarms. What should you do?

    A lower threshold labels more customers positive, which raises recall and lowers precision.

Summary

  • Binary classification needs a numeric 0/1 label and one features vector column.
  • StringIndexer, OneHotEncoder and VectorAssembler prepare categories and numbers; put them in a Pipeline with LogisticRegression.
  • The model adds rawPrediction, probability and prediction; the threshold (default 0.5) turns probability into a class.
  • Use BinaryClassificationEvaluator (areaUnderROC) on a held-out test set, and inspect the confusion matrix for the errors that matter.
  • Swap in tree models, naive Bayes or SVC by changing one pipeline stage.

Next: predict numbers instead of classes in MLlib Regression.