Skip to content
Tutorials Center
My learning

Topics

Lesson 88 of 119 ยท Spring Boot

Batch Processing with Spring Batch

Learn Spring Batch - jobs, steps, chunk-oriented processing, readers, processors and writers, job parameters, restart, skip and retry, with runnable models of each idea.

Not all work happens while a user waits. Every night a bank posts interest to millions of accounts, a retailer imports a supplier's price list of two million rows, a library archives last year's loans, and a data team recalculates statistics. This is batch processing: large volumes of data, processed without interaction, usually on a schedule, and required to be reliable. If the job dies at record 1,400,000 you want to resume there, not start over, and you want to know exactly what was processed, skipped or rejected.

A plain for loop with a database insert can't give you that. Spring Batch is the framework for it. It provides the vocabulary (jobs, steps, chunks), the building blocks (readers for files and databases, writers, processors), the bookkeeping (a repository that records every run, so jobs can be restarted) and the policies (skip bad records, retry flaky calls, process in parallel). In this lesson you will learn how the pieces fit, with runnable plain-Java models of the chunk loop and of restart behavior, followed by the real Spring Batch configuration. You should know Spring Boot basics, Spring Data JPA and JDBC transactions first.

The vocabulary

classDiagram
  class Job {
    name
  }
  class Step {
    name
  }
  class JobInstance {
    job name + identifying parameters
  }
  class JobExecution {
    status, start and end time
  }
  class StepExecution {
    readCount, writeCount, skipCount, commitCount
  }
  Job "1" --> "1..*" Step : consists of
  Job "1" --> "*" JobInstance : logical runs
  JobInstance "1" --> "*" JobExecution : attempts
  JobExecution "1" --> "*" StepExecution : one per step

A job describes the work; a job instance is one logical run (for example "the import for 2026-03-01"); each attempt to run that instance, including restarts, is a job execution.

  • A Job is the whole process, a sequence (or flow) of Steps.
  • A Step is an independent phase. It is either chunk-oriented (read, process, write many items) or a tasklet (one piece of work, such as deleting a temporary directory).
  • JobParameters identify a run. importMembersJob with inputFile=members-2026-03-01.csv is one JobInstance. Running the same job with the same identifying parameters again continues or is refused; a different parameter value starts a new instance.
  • A JobExecution is one attempt to run an instance, with a status (STARTING, STARTED, COMPLETED, FAILED, STOPPED) and an exit code. Each step in it has a StepExecution with counters: items read, filtered, written, skipped, and the number of commits and rollbacks.
  • The JobRepository stores all of this in database tables (BATCH_JOB_INSTANCE, BATCH_JOB_EXECUTION, BATCH_STEP_EXECUTION, and context tables). That history is what makes restart possible.

Chunk-oriented processing

The central pattern is the chunk. A step reads items one at a time, processes each one, collects the results, and when chunkSize items have been read it writes them all at once inside one transaction, then commits and starts the next chunk.

flowchart LR
  subgraph Chunk["One chunk = one transaction"]
    direction LR
    R["ItemReader<br/>read() x N"] --> P["ItemProcessor<br/>process() x N<br/>(null = filter out)"] --> W["ItemWriter<br/>write(all N results)"]
  end
  Chunk --> C["Commit + save<br/>progress"]
  C --> Chunk

Reading and processing are item by item; writing is bulk. A failure in a chunk rolls back that chunk only.

The three interfaces are small:

  • ItemReader<I> returns the next item, or null when the input is exhausted.
  • ItemProcessor<I, O> transforms or validates one item. Returning null filters the item out (it is not written, and it is counted as filtered, not as an error).
  • ItemWriter<O> receives the whole chunk and writes it, usually with a single batched SQL statement.

Why chunks instead of one big transaction, or one transaction per row? A single transaction over two million rows holds locks and undo logs for hours and loses everything on failure. A transaction per row is slow because of the commit overhead. Chunks (commonly 100 to 5,000 items) balance speed and recoverability. This model of the loop makes the mechanics visible:

Java A model of the chunk loop

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Count the commits: eight rows with a chunk size of three make three chunks (3, 3 and 2 items read). Notice that the chunk size counts items read, not items written; the first chunk wrote only two members because one was filtered.

A real Spring Batch job

Add spring-boot-starter-batch (and a database for the job repository; H2 is fine locally). Spring Boot configures the JobRepository and creates the batch tables for embedded databases. Don't add @EnableBatchProcessing in Spring Boot 3 or later: it switches Boot's auto-configuration off.

This job imports members from a CSV file, keeps adults, and stores them with one batched INSERT per chunk. The code follows Spring Batch 5, the version in Spring Boot 3. Spring Batch 6, used by Boot 4, simplified some builder signatures (for example the transaction manager argument became optional), so check the reference for the version you use.

Java
record MemberRow(int id, String name, String email, int age) {}
record Member(int id, String name, String email) {}

@Configuration
class ImportMembersJobConfig {

    @Bean
    @StepScope                       // created per step run, so it can see the job parameters
    FlatFileItemReader<MemberRow> memberReader(@Value("#{jobParameters['inputFile']}") String inputFile) {
        return new FlatFileItemReaderBuilder<MemberRow>()
            .name("memberReader")
            .resource(new FileSystemResource(inputFile))
            .linesToSkip(1)                                   // header row
            .delimited().names("id", "name", "email", "age")
            .targetType(MemberRow.class)
            .build();
    }

    @Bean
    ItemProcessor<MemberRow, Member> memberProcessor() {
        return row -> row.age() < 18
            ? null                                            // filtered, not an error
            : new Member(row.id(), row.name().strip(), row.email().toLowerCase());
    }

    @Bean
    JdbcBatchItemWriter<Member> memberWriter(DataSource dataSource) {
        return new JdbcBatchItemWriterBuilder<Member>()
            .dataSource(dataSource)
            .sql("INSERT INTO member (id, name, email) VALUES (:id, :name, :email)")
            .beanMapped()
            .build();
    }

    @Bean
    Step importStep(JobRepository jobRepository, PlatformTransactionManager transactionManager,
                    FlatFileItemReader<MemberRow> reader, ItemProcessor<MemberRow, Member> processor,
                    JdbcBatchItemWriter<Member> writer) {
        return new StepBuilder("importStep", jobRepository)
            .<MemberRow, Member>chunk(500, transactionManager)
            .reader(reader)
            .processor(processor)
            .writer(writer)
            .build();
    }

    @Bean
    Step cleanupStep(JobRepository jobRepository, PlatformTransactionManager transactionManager) {
        return new StepBuilder("cleanupStep", jobRepository)
            .tasklet((contribution, chunkContext) -> {
                // one-off work: move the processed file to an archive folder
                return RepeatStatus.FINISHED;
            }, transactionManager)
            .build();
    }

    @Bean
    Job importMembersJob(JobRepository jobRepository, Step importStep, Step cleanupStep) {
        return new JobBuilder("importMembersJob", jobRepository)
            .start(importStep)
            .next(cleanupStep)                                // runs only if importStep completed
            .build();
    }
}

The ready-made readers and writers cover most needs: FlatFileItemReader and FlatFileItemWriter (CSV, fixed width), JdbcCursorItemReader and JdbcPagingItemReader (database), JpaPagingItemReader, JsonItemReader, StaxEventItemReader (XML), JdbcBatchItemWriter, JpaItemWriter, and adapters for Kafka and AMQP. Write a custom reader or writer when the source is something else; the interface is one method.

Parameters, instances and restarting

Spring Boot runs all Job beans at start-up by default. For a scheduled or on-demand job, set spring.batch.job.enabled=false and launch it yourself, with parameters:

Java
@Service
class ImportService {
    private final JobLauncher jobLauncher;
    private final Job importMembersJob;

    ImportService(JobLauncher jobLauncher, Job importMembersJob) {
        this.jobLauncher = jobLauncher;
        this.importMembersJob = importMembersJob;
    }

    JobExecution importFile(String path) throws Exception {
        JobParameters parameters = new JobParametersBuilder()
            .addString("inputFile", path)                     // identifying: part of the JobInstance key
            .addLong("requestedAt", System.currentTimeMillis(), false)   // not identifying
            .toJobParameters();
        return jobLauncher.run(importMembersJob, parameters);
    }
}

The rules for restarts follow from the repository model:

  • Launching a job with parameters that match a completed instance throws JobInstanceAlreadyCompleteException. That protects you from importing the same file twice by accident.
  • Launching the same parameters after a failed execution creates a new execution of the same instance. Each step that already completed is skipped, and a failed chunk-oriented step resumes from its last saved position, because readers keep their position in the step's ExecutionContext (for example "line 1,400,000") which is committed together with each chunk.
  • If you want a new run every time (for example a nightly job without natural parameters), use a JobParametersIncrementer such as RunIdIncrementer.

This is the mechanism worth seeing in action. The model below uses a real H2 database: each chunk inserts rows and saves the checkpoint inside the same transaction, so a failing chunk rolls back both. A malformed amount in item 7 stops the first run; after the data is fixed, the second run resumes right after the last committed item.

stateDiagram-v2
  [*] --> STARTING
  STARTING --> STARTED
  STARTED --> COMPLETED : all steps finished
  STARTED --> FAILED : exception not skipped
  STARTED --> STOPPED : stop requested
  FAILED --> STARTING : restart (same instance)
  STOPPED --> STARTING : restart (same instance)
  COMPLETED --> [*]

A completed instance is final. Failed and stopped executions can be restarted and continue where they left off.

Java Chunk transactions, rollback and restart with H2

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with IO.readln() or a Scanner), it gets the next line from this box.

Study the first run. Chunks of items 1-3 and 4-6 were committed; the third chunk (items 7-9) was rolled back entirely, so no half-written chunk exists, and the checkpoint still says 6. The restart picked up at item 7. In real Spring Batch the checkpoint lives in BATCH_STEP_EXECUTION_CONTEXT, and the repository tracks the status of each step, but the principle is identical.

Skip, retry and fault tolerance

Large inputs always contain a few bad records, and external calls sometimes time out. Spring Batch lets you declare what to tolerate with faultTolerant():

Java
return new StepBuilder("importStep", jobRepository)
    .<MemberRow, Member>chunk(500, transactionManager)
    .reader(reader).processor(processor).writer(writer)
    .faultTolerant()
    .skip(FlatFileParseException.class)                       // a malformed line: log it, carry on
    .skip(ValidationException.class)
    .skipLimit(50)                                            // but fail the step after 50 skips
    .retry(TransientDataAccessException.class)                // deadlocks, timeouts, brief outages
    .retryLimit(3)
    .noRollback(ValidationException.class)                    // optional: don't roll back for this one
    .listener(new SkipLogger())                               // SkipListener: write skipped items to an audit table
    .build();
  • Skip: the item is left out, the counters record it and a SkipListener can store it for later inspection. A skip limit is essential; if half the file is bad, the step should fail instead of silently importing the rest.
  • Retry: the failed operation is attempted again, optionally with a back-off. Use it for transient problems only.
  • Skip while writing: when the writer throws for a chunk, Spring Batch cannot tell which item caused it. It rolls the chunk back and re-processes it item by item, each in its own transaction, until it finds and skips the bad one. This "scan" is correct but slower, and your processor runs again for those items, so make processors free of side effects.

Scaling a step

Start simple and measure. When a step is too slow, the options in increasing order of complexity are:

  1. Tune the chunk size and use bulk writes. A well-batched INSERT is often ten times faster than row-by-row. Check reWriteBatchedInserts or the equivalent for your driver.
  2. Multi-threaded step: give the step a TaskExecutor (.taskExecutor(executor), a virtual-thread executor works) so chunks run in parallel. The reader must be thread-safe: JdbcPagingItemReader is, JdbcCursorItemReader and the file readers are not (wrap them in a SynchronizedItemStreamReader). Disable restartability for such steps with saveState(false), because there is no single position to save.
  3. Parallel steps: run independent steps at the same time using a split flow (.split(taskExecutor).add(flowA, flowB)).
  4. Partitioning: divide the data (for example ids 1-1,000,000 and so on) into partitions, each processed by its own step execution, locally in threads or remotely on other machines. This scales across servers and keeps restart per partition.
  5. Remote chunking sends read items to workers over messaging (Messaging covers brokers) for expensive processing.

Operating batch jobs

  • Scheduling: trigger jobs with @Scheduled (Scheduling, Caching and Async), Kubernetes CronJobs, or an external scheduler. With several app instances, make sure only one starts the job; the job repository refuses to run two executions of the same instance at once, which is a useful safety net.
  • Observability: Spring Batch publishes Micrometer metrics (job and step durations, item counts), and JobExecutionListener/StepExecutionListener hooks let you log summaries or send alerts on failure (Actuator and Observability).
  • Idempotence: even with restart support, design writers so that repeating a chunk is harmless, for example MERGE/upsert statements, because a crash between the database commit and the repository update is possible in some setups.
  • Testing: @SpringBatchTest provides JobLauncherTestUtils to run a single step or the whole job against an embedded database, then assert on the StepExecution counters and the table contents.
  • When not to use it: for a few thousand rows a simple @Scheduled method may be enough. For continuous, low-latency event streams use a streaming framework or consumers on a broker instead; Spring Batch is for bounded data sets with a beginning and an end.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A step has a chunk size of 100 and reads 250 items. How many transactions (commits) does the step perform?

    Chunks of 100, 100 and 50 items are each written and committed in their own transaction.
  2. 2. What happens when an ItemProcessor returns null?

    Returning null is the supported way to drop an item, for example one that fails a business rule. It is not an error.
  3. 3. A job instance failed at item 1,400,000. You launch it again with the same identifying parameters. What happens?

    The job repository stores each step's progress in its execution context, which is committed with every chunk, so a restart continues from the last commit.
  4. 4. Why should a step that uses skip also set a skip limit?

    Skipping a handful of bad records is normal; skipping thousands usually means the input or the code is wrong and a person needs to look.
  5. 5. Which reader is safe to share between threads in a multi-threaded step?

    The paging reader fetches independent pages and is thread-safe. Cursor and file readers keep a single position and must be wrapped (or the step run single-threaded).

Summary

  • Spring Batch gives batch jobs structure and reliability: a Job of Steps, each either chunk-oriented or a tasklet, with run history stored in a JobRepository.
  • A chunk reads and processes items one by one, then writes them together and commits. Chunk size trades speed against rollback cost; returning null from a processor filters an item.
  • A JobInstance is identified by job name and identifying parameters; JobExecutions are its attempts. Completed instances can't be rerun; failed ones restart where they stopped, because progress is saved in the same transaction as each chunk.
  • Use faultTolerant() with a skip limit for bad records and retry for transient errors; writer failures trigger an item-by-item scan of the chunk.
  • Scale in steps: batch writes, multi-threaded steps (thread-safe readers only), parallel flows, partitioning, remote chunking.
  • Launch jobs yourself with parameters (spring.batch.job.enabled=false), schedule them, monitor them and make writers idempotent.

Next: Layered Architecture steps back from individual frameworks to how you organize an application's packages and responsibilities.