Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Constrained Optimization, Scaling, and Conditioning

Why ∇f=0\nabla f=0 is insufficient

For an unconstrained differentiable optimum, ∇f(x∗)=0\nabla f(\mathbf{x}^*)=0 is necessary. At a constrained optimum, the gradient need not vanish because active constraints block motion in improving directions.

Objective contours meeting an active constraint boundary at a constrained optimum, with objective and constraint gradients balancing one another.

At an active boundary, the objective gradient is balanced by the active constraint gradient.

Lagrangian and KKT conditions

Define the Lagrangian

L(x,λ,ν)=f(x)+∑i=1mλigi(x)+∑j=1pνjhj(x).\mathcal{L}(\mathbf{x},\boldsymbol{\lambda},\boldsymbol{\nu}) =f(\mathbf{x})+\sum_{i=1}^{m}\lambda_i g_i(\mathbf{x})+\sum_{j=1}^{p}\nu_jh_j(\mathbf{x}).
(1)

Under suitable regularity conditions, a local optimum satisfies the Karush–Kuhn–Tucker conditions:

∇xL=0,gi(x)≤0,hj(x)=0,λi≥0,λigi(x)=0.\begin{aligned} \nabla_{\mathbf{x}}\mathcal{L}&=0,\\ g_i(\mathbf{x})&\leq0,\\ h_j(\mathbf{x})&=0,\\ \lambda_i&\geq0,\\ \lambda_i g_i(\mathbf{x})&=0. \end{aligned}
(2)

The last condition is complementary slackness: an inequality is either inactive with zero multiplier, or active and potentially influential.

A multiplier measures the local value of relaxing its constraint. A large multiplier indicates that a small relaxation could significantly improve the objective and can help prioritize requirements.

Major constrained algorithms

Modern nonlinear-programming solvers often use SQP or interior-point methods. CCD users should understand the required inputs and returned diagnostics even if they do not implement these solvers.

Gradient-based and gradient-free solvers in CCD practice

SQP and interior-point methods are gradient-based: they exploit derivative information and typically converge in far fewer function evaluations than derivative-free alternatives, but they converge to the local optimum nearest the initial guess. Gradient-free algorithms—such as covariance matrix adaptation evolution strategy (CMA-ES), genetic algorithms (GA), and particle swarm optimization (PSO)—require only function evaluations, tolerate noisy or discontinuous objectives, and search more globally, but the number of evaluations they need tends to grow rapidly with the number of design variables, so they scale poorly to high-dimensional problems even after they have located a promising region.

A recent review of wind-turbine control co-design practice illustrates how these families are actually combined rather than chosen exclusively. Because CCD problems often combine a low-dimensional, potentially multimodal plant-design search with a much higher-dimensional control-trajectory search, a common hybrid strategy assigns a gradient-free optimizer (CMA-ES, GA, or PSO) to the plant-design variables and a gradient-based optimizer (SQP or an interior-point method) to the control-design variables, matching each discipline to the solver family best suited to its dimensionality and smoothness. Widely used implementations include CMA-ES and COBYLA on the gradient-free side, and SNOPT (a sparse SQP implementation) and IPOPT (an interior-point implementation) on the gradient-based side.

Why scaling matters

Mathematically equivalent formulations can behave very differently numerically. A solver may handle variables near 10-6 and 106 unevenly, especially when using common step and convergence tolerances.

Elongated objective contours and inefficient search steps before scaling, compared with balanced contours and direct steps after scaling.

Scaling produces a numerically more balanced landscape.

Variable, objective, and constraint scaling

A physical variable may be represented as

xi=xi,ref+six~i,x_i=x_{i,\mathrm{ref}}+s_i\tilde{x}_i,
(3)

with typical x~i\tilde{x}_i values near one. Bound-based scaling maps a variable to [0,1][0,1]:

x~i=xi−xL,ixU,i−xL,i.\tilde{x}_i=\frac{x_i-x_{L,i}}{x_{U,i}-x_{L,i}}.
(4)

Functions can similarly be scaled using meaningful nonzero references:

f~=ffref,g~i=gigi,ref.\tilde{f}=\frac{f}{f_{\mathrm{ref}}}, \qquad \tilde{g}_i=\frac{g_i}{g_{i,\mathrm{ref}}}.
(5)

Conditioning and tolerances

Conditioning describes solution sensitivity to small changes in data. Poor conditioning can cause slow convergence, sensitivity to tolerances, inaccurate finite differences, unstable linear algebra, and large design changes from small numerical errors.

Scaling improves algorithm behavior but cannot remove genuine physical ill-conditioning. If two variables have nearly identical effects, the design may be fundamentally poorly identifiable.

Solver tolerances operate in the provided coordinates. A residual tolerance of 10-6 has a different meaning for a normalized constraint than for a dimensional force balance. Scale first, then select tolerances.

Example 3.3: poorly scaled quadratic

For

f(x1,x2)=106x12+x22,f(x_1,x_2)=10^6x_1^2+x_2^2,
(6)

the curvature in x1x_1 is a million times larger than in x2x_2. Gradient descent needs a small step for stability in x1x_1, making progress in x2x_2 slow. Define

x~1=1000x1,x~2=x2.\tilde{x}_1=1000x_1,\qquad\tilde{x}_2=x_2.
(7)

Then f=x~12+x~22f=\tilde{x}_1^2+\tilde{x}_2^2, which has balanced curvature.

Activity 3.2: KKT Analysis of a Minimum-Mass Cantilever Beam

Activity 3.2: KKT Analysis of a Minimum-Mass Cantilever Beam

A rectangular cantilever beam has width bb, height hh, length LL, elastic modulus EE, material density ρ\rho, and an end load PP. Its mass is

m(b,h)=ρLbh.m(b,h)=\rho Lbh.
(8)

The maximum bending stress and tip displacement are

σ(b,h)=6PLbh2,δ(b,h)=4PL3Ebh3.\sigma(b,h)=\frac{6PL}{bh^2}, \qquad \delta(b,h)=\frac{4PL^3}{Ebh^3}.
(9)

The optimization problem is

min⁡b,hρLbh,subject to6PLbh2−σmax⁡≤0,4PL3Ebh3−δmax⁡≤0,bmin⁡≤b≤bmax⁡,hmin⁡≤h≤hmax⁡.\begin{aligned} \min_{b,h}\quad &\rho Lbh,\\ \text{subject to}\quad &\frac{6PL}{bh^2}-\sigma_{\max}\leq0,\\ &\frac{4PL^3}{Ebh^3}-\delta_{\max}\leq0,\\ &b_{\min}\leq b\leq b_{\max},\\ &h_{\min}\leq h\leq h_{\max}. \end{aligned}
(10)

Use

L=1.2 m,P=8 kN,E=70 GPa,L=1.2\ \mathrm{m}, \qquad P=8\ \mathrm{kN}, \qquad E=70\ \mathrm{GPa},
(11)
ρ=2700 kg/m3,σmax⁡=120 MPa,δmax⁡=4 mm,\rho=2700\ \mathrm{kg/m^3}, \qquad \sigma_{\max}=120\ \mathrm{MPa}, \qquad \delta_{\max}=4\ \mathrm{mm},
(12)

and

0.01≤b≤0.12 m,0.02≤h≤0.20 m.0.01\leq b\leq0.12\ \mathrm{m}, \qquad 0.02\leq h\leq0.20\ \mathrm{m}.
(13)

  1. Write the Lagrangian, including multipliers for the stress, displacement, and bound constraints.

  2. Write all KKT conditions: stationarity, primal feasibility, dual feasibility, and complementary slackness.

  3. Assume initially that the stress and displacement constraints are active and all bounds are inactive. Solve the two active constraints analytically for bb and hh.

  4. Check whether the resulting design satisfies the variable bounds.

  5. Compute the associated Lagrange multipliers and determine whether the assumed active set satisfies dual feasibility.

  6. Enumerate all physically plausible active sets involving:

    1. stress only;

    2. displacement only;

    3. stress and displacement; and

    4. one performance constraint and one variable bound.

  7. Determine the globally optimal feasible design by comparing all valid KKT candidates.

  8. Interpret each nonzero multiplier as the local value of relaxing its corresponding engineering requirement.

Activity 3.3: Scaling, Conditioning, and Gradient-Method Convergence

Activity 3.3: Scaling, Conditioning, and Gradient-Method Convergence

Consider the quadratic optimization problem

min⁡x∈R3f(x)=12xTHx−bTx,\min_{\mathbf{x}\in\mathbb{R}^3}\quad f(\mathbf{x}) =\frac{1}{2}\mathbf{x}^TH\mathbf{x}-\mathbf{b}^T\mathbf{x},
(14)

where

H=[108000100010−4],b=[104110−2].H= \begin{bmatrix} 10^8&0&0\\ 0&1&0\\ 0&0&10^{-4} \end{bmatrix}, \qquad \mathbf{b}= \begin{bmatrix} 10^4\\ 1\\ 10^{-2} \end{bmatrix}.
(15)

  1. Compute the exact minimizer

    x∗=H−1b.\mathbf{x}^*=H^{-1}\mathbf{b}.
    (16)

  2. Compute the 2-norm condition number κ2(H)\kappa_2(H).

  3. For fixed-step gradient descent,

    x(k+1)=x(k)−α∇f(x(k)),\mathbf{x}^{(k+1)} =\mathbf{x}^{(k)} -\alpha\nabla f\left(\mathbf{x}^{(k)}\right),
    (17)

    derive the largest stable step size.

  4. Derive the optimal fixed step size

    α∗=2λmax⁡+λmin⁡,\alpha^*=\frac{2}{\lambda_{\max}+\lambda_{\min}},
    (18)

    and the corresponding worst-case error contraction factor

    q∗=κ2(H)−1κ2(H)+1.q^*=\frac{\kappa_2(H)-1}{\kappa_2(H)+1}.
    (19)

  5. Estimate the number of iterations required to reduce the error norm by a factor of 10-6.

  6. Introduce the scaled variables

    x=Sz,S=[10−40001000102].\mathbf{x}=S\mathbf{z}, \qquad S= \begin{bmatrix} 10^{-4}&0&0\\ 0&1&0\\ 0&0&10^2 \end{bmatrix}.
    (20)

  7. Derive the scaled Hessian

    H~=STHS\widetilde{H}=S^THS
    (21)

    and compute its condition number.

  8. Compare gradient descent, Newton’s method, and BFGS on the original and scaled formulations from

    x(0)=[111].\mathbf{x}^{(0)}= \begin{bmatrix} 1\\ 1\\ 1 \end{bmatrix}.
    (22)

  9. Explain why variable scaling changes numerical behavior without changing the underlying physical optimum.

  10. Give one example of physical ill-conditioning that cannot be eliminated by simple coordinate scaling.

Chapter 3: Engineering Design Optimization
Gradient-Based Optimization
Chapter 3: Engineering Design Optimization
Multidisciplinary Design Optimization