3.7 Gradient-Based and Gradient-Free Optimization
Two algorithm families¶
A gradient-based algorithm uses derivatives of objectives and constraints to choose a search direction. Sequential quadratic programming and interior-point methods are common choices for smooth constrained CCD problems. Their main advantage is scalability: one accurate gradient can guide movement in a space containing hundreds or thousands of variables.
Gradient-free algorithms use objective and constraint evaluations without explicit derivatives. Pattern search, covariance-matrix adaptation, particle methods, and evolutionary algorithms can be helpful for discrete choices, noisy models, failed simulations, or small nonsmooth problems.
Algorithm choice follows model structure¶
| Problem feature | Preliminary preference |
|---|---|
| Smooth, many continuous variables, sparse derivatives | Gradient based |
| A few expensive variables and no reliable derivatives | Derivative free |
| Binary architecture decisions | Mixed-integer or enumerative strategy |
| Many local optima | Multi-start, global exploration, or hybrid strategy |
| Noisy or discontinuous simulation logic | Repair model or use a noise-tolerant method |
Gradient-free does not mean assumption-free, and it does not guarantee a global optimum. Its evaluation cost can grow rapidly with dimension. Conversely, gradient-based methods are only as trustworthy as their derivative information and scaling.
Derivative sources¶
Finite differences are easy but sensitive to step size and simulation noise. Complex-step differentiation can be highly accurate for compatible analytic code. Algorithmic differentiation applies the chain rule through the computational graph. Direct and adjoint sensitivities exploit model structure; adjoints are particularly attractive when there are many design variables but few scalar outputs.
A practical hybrid workflow¶
Use physics and coarse screening to select plausible regions, explore distinct basins if necessary, then refine continuous designs with accurate gradients. Re-evaluate the best candidates at tighter tolerances and higher fidelity.