Exercises - Optimal Control

Equality path constraints and the augmented Hamiltonian
Consider the Bolza problem
$\min_{\boldsymbol{u}(\cdot)} \; J = \Phi\!\left(\boldsymbol{x}(t_f)\right) + \int_{t_0}^{t_f} L(\boldsymbol{x},\boldsymbol{u},t)\,\mathrm{d}t,$
(1)
subject to
$\dot{\boldsymbol{x}} = \boldsymbol{f}(\boldsymbol{x},\boldsymbol{u},t)$
(2)
and the mixed equality path constraint
$\boldsymbol{c}(\boldsymbol{x},\boldsymbol{u},t) = \boldsymbol{0}.$
(3)
(a) Introduce a time-varying multiplier $\boldsymbol{\mu}(t)$ for the path constraint and construct the augmented Hamiltonian
$\overline{H} = L + \boldsymbol{\lambda}^{\mathsf T}\boldsymbol{f} + \boldsymbol{\mu}^{\mathsf T}\boldsymbol{c}.$
(4)
(b) Derive the state equation, costate equation, and control-stationarity condition in terms of $\overline{H}$ .
(c) Show that variation with respect to $\boldsymbol{\mu}(t)$ recovers the path constraint.
(d) Specialize the result to a control-only constraint
$\boldsymbol{c}(\boldsymbol{u},t) = \boldsymbol{0}.$
(5)
(e) Specialize the result to a state-only constraint
$\boldsymbol{c}(\boldsymbol{x},t) = \boldsymbol{0}$
(6)
and identify the additional term in the costate equation.
Inequality path constraints and complementarity
Suppose a minimization problem contains the scalar inequality path constraint
$c(\boldsymbol{x},\boldsymbol{u},t) \leq 0.$
(7)
Using
$\overline{H} = H+\mu c,$
(8)
derive the pointwise conditions
$c\leq0, \qquad \mu\geq0, \qquad \mu c=0.$
(9)
Then:
(a) derive the necessary conditions on an inactive arc;
(b) derive the necessary conditions on an active arc;
(c) explain why the active-set structure is generally unknown in advance;
(d) for the control bound
$u(t)-u_{\max}\leq0,$
(10)
determine the multiplier behavior when $u(t)<u_{\max}$ and when $u(t)=u_{\max}$ ;
(e) state how the inequality would be imposed in a direct-collocation nonlinear program.
Order of a state inequality constraint
Consider the double integrator
$\dot{x}_1=x_2, \qquad \dot{x}_2=u,$
(11)
with the state path constraint
$c(\boldsymbol{x}) = x_1-x_{\max} \leq 0.$
(12)
(a) Differentiate $c$ repeatedly along the dynamics until the control appears explicitly.
(b) Show that the constraint has order two.
(c) On a constrained arc, derive the tangency conditions
$x_1=x_{\max}, \qquad x_2=0.$
(13)
(d) Determine the control required to remain on the constrained arc.
(e) State the consistency conditions that must hold at entry into the constrained arc.
(f) Explain why enforcing $x_1\leq x_{\max}$ only at collocation nodes does not guarantee continuous-time feasibility.
(g) Formulate a dense-grid verification test for the reconstructed state trajectory.
Principle of optimality and the HJB equation
Define the optimal cost-to-go
$J^*(\boldsymbol{x},t) = \min_{\boldsymbol{u}(\cdot)} \left[ \Phi\!\left(\boldsymbol{x}(t_f)\right) + \int_t^{t_f} L(\boldsymbol{x}(\tau),\boldsymbol{u}(\tau),\tau) \,\mathrm{d}\tau \right].$
(14)
Starting from Bellman’s principle of optimality over the short interval $[t,t+\Delta t]$ :
(a) write the dynamic-programming recursion;
(b) expand $J^*(\boldsymbol{x}(t+\Delta t),t+\Delta t)$ to first order;
(c) derive the Hamilton–Jacobi–Bellman equation
$0 = J_t^* + \min_{\boldsymbol{u}} \left[ L + \nabla_{\boldsymbol{x}}J^{*\mathsf T} \boldsymbol{f} \right];$
(15)
(d) derive the terminal condition for $J^*$ ;
(e) show that, along the optimal trajectory,
$\frac{\mathrm{d}J^*}{\mathrm{d}t} = - L(\boldsymbol{x}^*,\boldsymbol{u}^*,t);$
(16)
(f) identify the relationship between the value-function gradient and the costate.
Scalar HJB equation and Riccati feedback
Consider
$\dot{x} = -2x+u$
(17)
with
$J = \frac{1}{2}x^2(t_f) + \frac{1}{2} \int_t^{t_f} \left( x^2+u^2 \right)\,\mathrm{d}\tau.$
(18)
Assume
$J^*(x,t) = \frac{1}{2}P(t)x^2.$
(19)
(a) Derive $J_t^*$ and $J_x^*$ .
(b) Substitute the assumed value function into the HJB equation.
(c) Minimize the HJB expression with respect to $u$ .
(d) Derive the optimal feedback law.
(e) Show that $P(t)$ satisfies
$\dot{P} = P^2+4P-1.$
(20)
(f) Show that the terminal condition is
$P(t_f)=1.$
(21)
(g) Derive the optimal closed-loop state equation.
(h) Show that
$\lambda(t) = P(t)x^*(t).$
(22)
Matrix HJB formulation of finite-horizon LQR
Consider
$\dot{\boldsymbol{x}} = A(t)\boldsymbol{x} + B(t)\boldsymbol{u},$
(23)
with performance index
$J = \frac{1}{2} \boldsymbol{x}^{\mathsf T}(t_f) S_f \boldsymbol{x}(t_f) + \frac{1}{2} \int_t^{t_f} \left[ \boldsymbol{x}^{\mathsf T}Q\boldsymbol{x} + \boldsymbol{u}^{\mathsf T}R\boldsymbol{u} \right] \,\mathrm{d}\tau,$
(24)
where $Q(t)=Q^{\mathsf T}(t)\succeq0$ and $R(t)=R^{\mathsf T}(t)\succ0$ .
Assume
$J^*(\boldsymbol{x},t) = \frac{1}{2} \boldsymbol{x}^{\mathsf T} S(t) \boldsymbol{x}.$
(25)
(a) Derive
$J_t^* = \frac{1}{2} \boldsymbol{x}^{\mathsf T} \dot{S} \boldsymbol{x}.$
(26)
(b) Derive the minimizing control.
(c) Derive the Riccati differential equation
$-\dot{S} = A^{\mathsf T}S + SA - SBR^{-1}B^{\mathsf T}S + Q.$
(27)
(d) Derive the terminal condition
$S(t_f)=S_f.$
(28)
(e) Derive the feedback gain and the closed-loop state equation.
(f) Show that
$\boldsymbol{\lambda} = S\boldsymbol{x}^*.$
(29)
(g) Explain why the Riccati equation is integrated backward while the state equation is integrated forward.
Bang–bang and singular control
Consider
$\min_{u(\cdot)} \; J = \int_0^1 x_2(t)u(t)\,\mathrm{d}t$
(30)
subject to
$\dot{x}_1=x_2, \qquad \dot{x}_2=-x_2+u,$
(31)
$0\leq u(t)\leq2,$
(32)
and
$x_1(0)=0, \qquad x_2(0)=1, \qquad x_1(1)=1, \qquad x_2(1)=1.$
(33)
(a) Construct the Hamiltonian.
(b) Derive the costate equations.
(c) Define the switching function
$\phi(t) = x_2(t)+\lambda_2(t).$
(34)
(d) Derive the bang–bang minimization rule for $\phi>0$ and $\phi<0$ .
(e) Impose
$\phi=\dot{\phi}=0$
(35)
on a candidate singular arc.
(f) Derive the singular control.
(g) Verify that the singular state and control satisfy all endpoint conditions.
(h) Determine the order of the singular arc and state the corresponding generalized Legendre–Clebsch check.
Analytical direct shooting for the minimum-energy double integrator
Consider
$\min_{u(\cdot)} \; J = \frac{1}{2} \int_0^T u^2(t)\,\mathrm{d}t$
(36)
subject to
$\dot{x}_1=x_2, \qquad \dot{x}_2=u,$
(37)
with fixed endpoint data
$x_1(0)=a_0, \qquad x_2(0)=v_0, \qquad x_1(T)=a_f, \qquad x_2(T)=v_f.$
(38)
(a) Derive the Hamiltonian and costate equations.
(b) Show that $\lambda_1$ is constant and $\lambda_2$ is affine in time.
(c) Derive the optimal control as an affine function of time.
(d) Integrate the state equations analytically.
(e) Reduce the terminal conditions to a $2\times2$ linear system for the two unknown initial costates.
(f) Solve this system explicitly.
(g) Explain why the original infinite-dimensional problem has been reduced to a finite-dimensional shooting problem.
General neighboring-optimal-control derivation
Let $(\boldsymbol{x}^*,\boldsymbol{u}^*,\boldsymbol{\lambda}^*)$ be a nominal nonlinear optimal trajectory. Define
$\delta\boldsymbol{x} = \boldsymbol{x}-\boldsymbol{x}^*, \qquad \delta\boldsymbol{u} = \boldsymbol{u}-\boldsymbol{u}^*, \qquad \delta\boldsymbol{\lambda} = \boldsymbol{\lambda}-\boldsymbol{\lambda}^*.$
(39)
Assume the neighboring dynamics and second-order cost are
$\delta\dot{\boldsymbol{x}} = F(t)\delta\boldsymbol{x} + G(t)\delta\boldsymbol{u},$
(40)
$\delta^2J = \frac{1}{2} \delta\boldsymbol{x}^{\mathsf T}(t_f) P_f \delta\boldsymbol{x}(t_f) + \frac{1}{2} \int_{t_0}^{t_f} \left[ \delta\boldsymbol{x}^{\mathsf T}Q\delta\boldsymbol{x} + 2\delta\boldsymbol{x}^{\mathsf T}M\delta\boldsymbol{u} + \delta\boldsymbol{u}^{\mathsf T}R\delta\boldsymbol{u} \right] \,\mathrm{d}t.$
(41)
(a) Construct the neighboring Hamiltonian.
(b) Derive the neighboring costate equation.
(c) Derive the stationarity condition.
(d) Show that
$\delta\boldsymbol{u} = - R^{-1} \left( M^{\mathsf T}\delta\boldsymbol{x} + G^{\mathsf T}\delta\boldsymbol{\lambda} \right).$
(42)
(e) Assume
$\delta\boldsymbol{\lambda} = P(t)\delta\boldsymbol{x}$
(43)
and derive the generalized Riccati equation.
(f) Show that
$P(t_f)=P_f.$
(44)
(g) Derive the neighboring feedback gain
$K(t) = R^{-1} \left( M^{\mathsf T} + G^{\mathsf T}P \right).$
(45)
(h) Show that the standard time-varying LQR Riccati equation is recovered when $M=\boldsymbol{0}$ .
Neighboring feedback for a nonlinear scalar system
Consider the nonlinear dynamics
$\dot{x} = x+x^2+u$
(46)
and the nominal trajectory
$x^*(t)=0, \qquad u^*(t)=0.$
(47)
The neighboring cost is
$\delta^2J = \frac{1}{2}p_f\delta x^2(t_f) + \frac{1}{2} \int_0^{t_f} \left( q\delta x^2 + 2m\delta x\,\delta u + r\delta u^2 \right) \,\mathrm{d}t,$
(48)
with $r>0$ .
(a) Linearize the dynamics about the nominal trajectory and identify $F$ and $G$ .
(b) Derive the scalar neighboring Riccati equation.
(c) Derive the feedback gain
$K(t) = \frac{m+G P(t)}{r}.$
(49)
(d) Write the neighboring correction
$\delta u(t) = -K(t)\delta x(t).$
(50)
(e) Write the complete implemented control $u=u^*+\delta u$ .
(f) Show that the correction is zero on the nominal trajectory.
(g) Describe a numerical test using several initial perturbation magnitudes.
(h) State the residuals and qualitative behaviors that should be checked to determine when the neighboring approximation is no longer reliable.