Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 36 ยท Projects and Practice

Project: A Nightly Data Cleanup Pipeline

Build a nightly Apex pipeline: a Schedulable launches a stateful batch that archives stale records, collects failures and chains a queueable summary, with tests and a Java simulation.

Data quality decays. Leads go stale, duplicate-looking contacts pile up, old tasks clutter reports. In this project you build a nightly cleanup pipeline from the pieces of the last chapters: a scheduled job starts a batch that archives stale leads, counts successes and failures, and at the end hands the results to a queueable that records a summary. Each part is small and testable, and together they show how Salesforce asynchronous tools fit.

This needs an org, so the Apex listings are read-only. The Java stand-in at the end simulates the pipeline (scheduler, chunks, partial failures, summary) so you can experiment with scope sizes and failure rates.

Requirements

  1. Every night at 02:00, find leads that are not converted and have had no activity for a year.
  2. Set their status to Archived. A failure on one lead must not stop the others.
  3. After the last chunk, write a summary record with the totals and the first few error messages.
  4. Everything must be bulk-safe and testable.

The data model

Create a custom object Cleanup_Log__c with fields Run_Date__c (Date), Processed__c (Number), Failed__c (Number) and Errors__c (Long Text Area). It is where the summary goes.

Part 1: the batch

APEX
public with sharing class StaleLeadCleanup implements Database.Batchable<SObject>, Database.Stateful {

    public Integer processed = 0;
    public Integer failed = 0;
    public List<String> errors = new List<String>();
    private static final Integer MAX_ERRORS_KEPT = 20;

    public Database.QueryLocator start(Database.BatchableContext bc) {
        return Database.getQueryLocator(
            'SELECT Id, Status FROM Lead WHERE IsConverted = false AND Status != \'Archived\' ' +
            'AND LastActivityDate < LAST_N_DAYS:365');
    }

    public void execute(Database.BatchableContext bc, List<Lead> scope) {
        for (Lead l : scope) {
            l.Status = 'Archived';
        }
        List<Database.SaveResult> results = Database.update(scope, false);
        for (Integer i = 0; i < results.size(); i++) {
            if (results[i].isSuccess()) {
                processed++;
            } else {
                failed++;
                if (errors.size() < MAX_ERRORS_KEPT) {
                    errors.add(scope[i].Id + ': ' + results[i].getErrors()[0].getMessage());
                }
            }
        }
    }

    public void finish(Database.BatchableContext bc) {
        System.enqueueJob(new CleanupSummaryJob(processed, failed, errors));
    }
}

Points to notice: the query excludes already-archived records, so a rerun does no extra work (the job is idempotent); Database.update(scope, false) allows partial success; errors are capped so the state stays small (it is serialised between chunks); and the summary is created in a queueable rather than in finish so it has its own transaction and limits.

Part 2: the summary job

APEX
public with sharing class CleanupSummaryJob implements Queueable {
    private Integer processed;
    private Integer failed;
    private List<String> errors;

    public CleanupSummaryJob(Integer processed, Integer failed, List<String> errors) {
        this.processed = processed;
        this.failed = failed;
        this.errors = errors;
    }

    public void execute(QueueableContext ctx) {
        insert new Cleanup_Log__c(
            Run_Date__c = Date.today(),
            Processed__c = processed,
            Failed__c = failed,
            Errors__c = String.join(errors, '\n')
        );
    }
}

Part 3: the schedule

APEX
public with sharing class NightlyCleanup implements Schedulable {
    public void execute(SchedulableContext sc) {
        Database.executeBatch(new StaleLeadCleanup(), 200);
    }
}

Schedule it once from anonymous Apex:

APEX
System.schedule('Nightly lead cleanup', '0 0 2 * * ?', new NightlyCleanup());

Part 4: tests

APEX
@isTest
private class StaleLeadCleanupTest {

    @isTest
    static void archivesLeadsAndWritesSummary() {
        List<Lead> leads = new List<Lead>();
        for (Integer i = 0; i < 100; i++) {
            leads.add(new Lead(LastName = 'Old ' + i, Company = 'Co', Status = 'Open'));
        }
        insert leads;

        Test.startTest();
        Database.executeBatch(new StaleLeadCleanup(), 200);
        Test.stopTest();

        Cleanup_Log__c log = [SELECT Processed__c, Failed__c FROM Cleanup_Log__c LIMIT 1];
        Assert.areEqual(100, log.Processed__c);
        Assert.areEqual(0, log.Failed__c);
    }

    @isTest
    static void scheduleIsRegistered() {
        Test.startTest();
        Id jobId = System.schedule('Test cleanup', '0 0 2 * * ?', new NightlyCleanup());
        Test.stopTest();
        Assert.areEqual('0 0 2 * * ?', [SELECT CronExpression FROM CronTrigger WHERE Id = :jobId].CronExpression);
    }
}

Note that brand-new test leads have a recent LastActivityDate, so the production query would not select them. A common technique is to move the WHERE clause into a @TestVisible property that tests override, for example @TestVisible private String queryCondition = 'LastActivityDate < LAST_N_DAYS:365';. Keep this in mind when you write the real thing.

Try the idea: simulate the pipeline (Java stand-in)

Java Stand-in: scheduler, chunked batch, partial failures, summary (Java)

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Try different scope sizes and failure rates. The totals do not change, only the number of chunks (and therefore transactions).

Extend it

  • Make the stale-days threshold a Custom Metadata record so an admin can change it.
  • Email the summary using Messaging.SingleEmailMessage from the queueable.
  • Add an Archived_Date__c stamp and a report on it.
  • Chain a second batch for contacts, started from finish.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. Why does the query exclude Status = 'Archived'?

    Excluding already-processed records makes retries safe and cheap.
  2. 2. Why use Database.Stateful here?

    Without Stateful the counters would reset for every execute call.
  3. 3. Why write the summary from a queueable started in finish?

    finish is one transaction; handing work to a queueable isolates it with fresh limits.
  4. 4. Why cap the number of stored error messages?

    Keeping state small protects heap and serialisation cost.

Summary

  • The schedulable launches the batch, the batch does the chunked work, the queueable records the result.
  • Make jobs idempotent, allow partial success, and keep stateful data small.
  • Test the batch with fewer than one chunk of records and test that the schedule registers.
  • Parameterise thresholds so admins can tune them without a deploy.

Next: Practice: Governor Limits, Patterns and Interview Questions