Skip to content
Tutorials Center
My learning

Topics

Lesson 35 of 36 ยท Projects and Practice

Practice: Benchmark GA, DE and PSO

Compare a real-coded genetic algorithm, differential evolution and particle swarm optimization fairly on standard benchmark functions, with equal budgets and many seeds.

You now know several optimizers: genetic algorithms, differential evolution, particle swarm optimization and local search methods like simulated annealing. Which one should you use? The honest answer is "it depends on the problem". The skill to learn is how to find out with a fair experiment.

In this lesson you build a small benchmarking harness. It runs three algorithms on four classic test functions, gives each the same budget of fitness evaluations, repeats each run over many seeds and reports a table. The same harness applies to your own problems.

The benchmark functions

Standard test functions have known optima and different difficulties. All of these have the global minimum 0 and are used in d = 5 dimensions here.

Function Shape What it tests
Sphere A smooth bowl Basic convergence speed
Rosenbrock A narrow, curved valley Following a curved path; variables interact
Rastrigin A bowl covered with regular bumps Escaping many local minima
Ackley Flat outer plateau, deep central hole Finding a small attractive region

Results on benchmarks are only an indication. A method that wins on all four may still lose on your real problem, but the harness tells you how an algorithm behaves on smooth, valley-shaped and bumpy landscapes.

The fairness rules

  1. Equal budget: every algorithm gets exactly the same number of function evaluations (here 6,000).
  2. Many seeds: each combination runs 8 times. Report the median (robust to bad runs) and the best and worst.
  3. Same bounds and starting distribution.
  4. Reasonable defaults, not hand-tuned for one algorithm.
  5. Report failures: the worst run matters as much as the best.
flowchart TD
    F["Functions: sphere, rosenbrock, rastrigin, ackley"] --> H["Harness"]
    A["Algorithms: GA, DE, PSO"] --> H
    H --> R["For each pair: 8 seeds, budget = 6000 evaluations"]
    R --> T["Table: median / worst final value"]

Three algorithms times four functions times eight seeds: 96 runs.

Python A fair benchmark of three optimizers

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Reading the table

Do not expect one algorithm to win everywhere. Typical patterns (your exact numbers depend on the settings and seeds):

  • On the sphere, all three converge, but DE and PSO (which exploit smoothness) reach values many orders of magnitude smaller than the blend-crossover GA with the same budget.
  • On Rosenbrock, the valley is tricky. Compare the median with the worst column: a method can have a great median and still fail badly in some runs.
  • On Rastrigin, no method solves it exactly within this budget; the bumps punish premature convergence, so look at the worst column.
  • On Ackley, DE and PSO find the central hole almost exactly, while this GA stalls near 0.01.

This is the "no free lunch" idea in practice: averaged over all possible problems, no optimizer beats any other, and performance comes from matching the algorithm's bias to the problem's structure.

Performance measures beyond the final value

The final best value after a fixed budget is the simplest measure. Others are often useful:

  • Convergence curves: best value versus evaluations, to see speed.
  • Success rate: fraction of runs reaching a target (for example, f < 1e-3).
  • Evaluations to target: how many evaluations until the target is reached.
  • Statistical tests: use a rank-based test (Wilcoxon or Mann-Whitney) to see whether a difference between two algorithms is more than noise.
Python Success rate and a rank test on Rastrigin

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

The test answers a question the table alone cannot: is the gap between two settings larger than run-to-run noise?

Turning the harness on your own problem

  1. Replace the benchmark function with your fitness function (vectorized if possible).
  2. Keep the three optimizers and add your own GA design.
  3. Fix a budget you can afford in production.
  4. Run at least 10 seeds and look at the median and the worst case.
  5. Pick the algorithm whose worst case you can live with, not whose best case is prettiest.

Common mistakes

  • Giving one algorithm more evaluations. The budget must be identical, including initial populations.
  • Reporting only the best run. That measures luck.
  • Tuning one algorithm and not the others. Use defaults for all, or tune all equally.
  • Concluding from one function. Use a spread of landscapes.

Check your understanding

Quick check ยท 4 questions

of 4 answered You scored / 4 ยท perfect!

  1. 1. What is the fairest way to compare optimizers with different population sizes?

    Evaluations are the cost currency, since fitness evaluation usually dominates run time.
  2. 2. Why report the median and the worst run, not only the best?

    You must live with the typical and the bad runs in practice.
  3. 3. What does the 'no free lunch' idea say?

    Performance comes from the match between the algorithm's biases and the problem structure.
  4. 4. Which benchmark especially punishes premature convergence because of many local minima?

    Rastrigin has a regular grid of local minima.

Summary

  • Compare algorithms with equal evaluation budgets, identical bounds and many seeds.
  • Use functions with different landscapes: smooth, valley-shaped, bumpy and plateau-like.
  • Report median, worst case, success rate and, where possible, a rank-based test.
  • Expect no universal winner; learn how each algorithm's bias matches your problem.
  • Reuse the harness for your own problems.

Next: Practice Exercises and Where to Go Next