HJB Formulation of the Finite-Horizon Linear Quadratic Regulator
This session applies the Hamilton–Jacobi–Bellman equation to the finite-horizon linear quadratic regulator (LQR). The objective is to recover, using dynamic programming, the same matrix Riccati differential equation and state-feedback law obtained previously from the calculus of variations and Pontryagin’s minimum principle.
The importance of this derivation is not merely algebraic. It shows that the Riccati equation is a direct consequence of the HJB partial differential equation when the system is linear and the cost is quadratic.
Finite-Horizon LQR Problem ¶ Consider the time-varying linear system
x ˙ ( t ) = A ( t ) x ( t ) + B ( t ) u ( t ) , x ( t 0 ) = x 0 . \boxed{
\dot{\boldsymbol{x}}(t)=\boldsymbol{A}(t)\boldsymbol{x}(t)+\boldsymbol{B}(t)\boldsymbol{u}(t),
\qquad
\boldsymbol{x}(t_0)=\boldsymbol{x}_0.
} x ˙ ( t ) = A ( t ) x ( t ) + B ( t ) u ( t ) , x ( t 0 ) = x 0 . The finite-horizon quadratic cost is
J = 1 2 x T ( t f ) S f x ( t f ) + 1 2 ∫ t 0 t f [ x T ( t ) Q ( t ) x ( t ) + u T ( t ) R ( t ) u ( t ) ] d t . \boxed{
J=
\frac{1}{2}\boldsymbol{x}^\mathsf{T}(t_f)\boldsymbol{S}_f\boldsymbol{x}(t_f)
+
\frac{1}{2}
\int_{t_0}^{t_f}
\left[
\boldsymbol{x}^\mathsf{T}(t)\boldsymbol{Q}(t)\boldsymbol{x}(t)
+
\boldsymbol{u}^\mathsf{T}(t)\boldsymbol{R}(t)\boldsymbol{u}(t)
\right]\,\mathrm{d} t.
} J = 2 1 x T ( t f ) S f x ( t f ) + 2 1 ∫ t 0 t f [ x T ( t ) Q ( t ) x ( t ) + u T ( t ) R ( t ) u ( t ) ] d t . Assume
Q ( t ) = Q T ( t ) ⪰ 0 , R ( t ) = R T ( t ) ≻ 0 , S f = S f T ⪰ 0. \begin{aligned}
\boldsymbol{Q}(t)&=\boldsymbol{Q}^\mathsf{T}(t)\succeq 0,\\
\boldsymbol{R}(t)&=\boldsymbol{R}^\mathsf{T}(t)\succ 0,\\
\boldsymbol{S}_f&=\boldsymbol{S}_f^\mathsf{T}\succeq 0.
\end{aligned} Q ( t ) R ( t ) S f = Q T ( t ) ⪰ 0 , = R T ( t ) ≻ 0 , = S f T ⪰ 0. The positive definiteness of R \boldsymbol{R} R
Structural Assumptions ¶ For the classical finite-horizon LQR problem, one often assumes that
( A ( t ) , B ( t ) ) (\boldsymbol{A}(t),\boldsymbol{B}(t)) ( A ( t ) , B ( t ))
( A ( t ) , Q 1 / 2 ( t ) ) (\boldsymbol{A}(t),\boldsymbol{Q}^{1/2}(t)) ( A ( t ) , Q 1/2 ( t ))
all matrices are sufficiently smooth functions of time.
These assumptions ensure that the problem is well posed and that the resulting feedback law has the expected properties.
Cost-to-Go Function ¶ At an arbitrary state x \boldsymbol{x} x t t t
J ∗ ( x , t ) = min u ( ⋅ ) [ 1 2 x T ( t f ) S f x ( t f ) + 1 2 ∫ t t f ( x T Q x + u T R u ) d τ ] . J^*(\boldsymbol{x},t)
=
\min_{\boldsymbol{u}(\cdot)}
\left[
\frac{1}{2}\boldsymbol{x}^\mathsf{T}(t_f)\boldsymbol{S}_f\boldsymbol{x}(t_f)
+
\frac{1}{2}
\int_t^{t_f}
\left(
\boldsymbol{x}^\mathsf{T}\boldsymbol{Q}\boldsymbol{x}
+
\boldsymbol{u}^\mathsf{T}\boldsymbol{R}\boldsymbol{u}
\right)\,\mathrm{d}\tau
\right]. J ∗ ( x , t ) = u ( ⋅ ) min [ 2 1 x T ( t f ) S f x ( t f ) + 2 1 ∫ t t f ( x T Q x + u T R u ) d τ ] . The terminal condition is
J ∗ ( x , t f ) = 1 2 x T S f x . \boxed{
J^*(\boldsymbol{x},t_f)
=
\frac{1}{2}\boldsymbol{x}^\mathsf{T}\boldsymbol{S}_f\boldsymbol{x}.
} J ∗ ( x , t f ) = 2 1 x T S f x . The running cost is
L ( x , u , t ) = 1 2 x T Q x + 1 2 u T R u . L(\boldsymbol{x},\boldsymbol{u},t)
=
\frac{1}{2}
\boldsymbol{x}^\mathsf{T}\boldsymbol{Q}\boldsymbol{x}
+
\frac{1}{2}
\boldsymbol{u}^\mathsf{T}\boldsymbol{R}\boldsymbol{u}. L ( x , u , t ) = 2 1 x T Q x + 2 1 u T R u . The dynamics are
f ( x , u , t ) = A x + B u . \boldsymbol{f}(\boldsymbol{x},\boldsymbol{u},t)
=
\boldsymbol{A}\boldsymbol{x}+\boldsymbol{B}\boldsymbol{u}. f ( x , u , t ) = A x + B u . The HJB Hamiltonian is therefore
H = 1 2 x T Q x + 1 2 u T R u + J x ∗ T ( A x + B u ) . \boxed{
\mathcal{H}
=
\frac{1}{2}\boldsymbol{x}^\mathsf{T}\boldsymbol{Q}\boldsymbol{x}
+
\frac{1}{2}\boldsymbol{u}^\mathsf{T}\boldsymbol{R}\boldsymbol{u}
+
J_{\boldsymbol{x}}^{*\mathsf{T}}
(\boldsymbol{A}\boldsymbol{x}+\boldsymbol{B}\boldsymbol{u}).
} H = 2 1 x T Q x + 2 1 u T R u + J x ∗ T ( A x + B u ) . Here,
J x ∗ = ∇ x J ∗ ( x , t ) . J_{\boldsymbol{x}}^*
=
\nabla_{\boldsymbol{x}}J^*(\boldsymbol{x},t). J x ∗ = ∇ x J ∗ ( x , t ) . Optimal Control from Hamiltonian Minimization ¶ The optimal control minimizes the Hamiltonian:
u ∗ = arg min u H . \boldsymbol{u}^*
=
\arg\min_{\boldsymbol{u}}\mathcal{H}. u ∗ = arg u min H . Differentiate with respect to u \boldsymbol{u} u
∂ H ∂ u = R u + B T J x ∗ . \frac{\partial\mathcal{H}}{\partial\boldsymbol{u}}
=
\boldsymbol{R}\boldsymbol{u}
+
\boldsymbol{B}^\mathsf{T}J_{\boldsymbol{x}}^*. ∂ u ∂ H = R u + B T J x ∗ . The first-order condition is
R u ∗ + B T J x ∗ = 0 . \boldsymbol{R}\boldsymbol{u}^*
+
\boldsymbol{B}^\mathsf{T}J_{\boldsymbol{x}}^*
=
\boldsymbol{0}. R u ∗ + B T J x ∗ = 0 . Because R ≻ 0 \boldsymbol{R}\succ0 R ≻ 0
u ∗ = − R − 1 B T J x ∗ . \boxed{
\boldsymbol{u}^*
=
-
\boldsymbol{R}^{-1}\boldsymbol{B}^\mathsf{T}J_{\boldsymbol{x}}^*.
} u ∗ = − R − 1 B T J x ∗ . This is already a feedback relationship, although the value-function gradient is not yet known.
Reduced Optimal Hamiltonian ¶ Substitute
u ∗ = − R − 1 B T J x ∗ \boldsymbol{u}^*
=
-
\boldsymbol{R}^{-1}\boldsymbol{B}^\mathsf{T}J_{\boldsymbol{x}}^* u ∗ = − R − 1 B T J x ∗ into the Hamiltonian.
First,
1 2 u ∗ T R u ∗ = 1 2 J x ∗ T B R − 1 B T J x ∗ . \begin{aligned}
\frac{1}{2}\boldsymbol{u}^{*\mathsf{T}}\boldsymbol{R}\boldsymbol{u}^*
&=
\frac{1}{2}
J_{\boldsymbol{x}}^{*\mathsf{T}}
\boldsymbol{B}\boldsymbol{R}^{-1}\boldsymbol{B}^\mathsf{T}
J_{\boldsymbol{x}}^*.
\end{aligned} 2 1 u ∗ T R u ∗ = 2 1 J x ∗ T B R − 1 B T J x ∗ . Second,
J x ∗ T B u ∗ = − J x ∗ T B R − 1 B T J x ∗ . \begin{aligned}
J_{\boldsymbol{x}}^{*\mathsf{T}}\boldsymbol{B}\boldsymbol{u}^*
&=
-
J_{\boldsymbol{x}}^{*\mathsf{T}}
\boldsymbol{B}\boldsymbol{R}^{-1}\boldsymbol{B}^\mathsf{T}
J_{\boldsymbol{x}}^*.
\end{aligned} J x ∗ T B u ∗ = − J x ∗ T B R − 1 B T J x ∗ . Therefore,
H ∗ = 1 2 x T Q x − 1 2 J x ∗ T B R − 1 B T J x ∗ + J x ∗ T A x . \boxed{
\mathcal{H}^*
=
\frac{1}{2}\boldsymbol{x}^\mathsf{T}\boldsymbol{Q}\boldsymbol{x}
-
\frac{1}{2}
J_{\boldsymbol{x}}^{*\mathsf{T}}
\boldsymbol{B}\boldsymbol{R}^{-1}\boldsymbol{B}^\mathsf{T}
J_{\boldsymbol{x}}^*
+
J_{\boldsymbol{x}}^{*\mathsf{T}}\boldsymbol{A}\boldsymbol{x}.
} H ∗ = 2 1 x T Q x − 2 1 J x ∗ T B R − 1 B T J x ∗ + J x ∗ T A x . Hamilton–Jacobi–Bellman PDE ¶ The HJB equation is
J t ∗ + H ∗ = 0. J_t^*
+
\mathcal{H}^*
=
0. J t ∗ + H ∗ = 0. Thus,
J t ∗ + 1 2 x T Q x − 1 2 J x ∗ T B R − 1 B T J x ∗ + J x ∗ T A x = 0. \boxed{
J_t^*
+
\frac{1}{2}\boldsymbol{x}^\mathsf{T}\boldsymbol{Q}\boldsymbol{x}
-
\frac{1}{2}
J_{\boldsymbol{x}}^{*\mathsf{T}}
\boldsymbol{B}\boldsymbol{R}^{-1}\boldsymbol{B}^\mathsf{T}
J_{\boldsymbol{x}}^*
+
J_{\boldsymbol{x}}^{*\mathsf{T}}\boldsymbol{A}\boldsymbol{x}
=
0.
} J t ∗ + 2 1 x T Q x − 2 1 J x ∗ T B R − 1 B T J x ∗ + J x ∗ T A x = 0. The terminal boundary condition is
J ∗ ( x , t f ) = 1 2 x T S f x . \boxed{
J^*(\boldsymbol{x},t_f)
=
\frac{1}{2}\boldsymbol{x}^\mathsf{T}\boldsymbol{S}_f\boldsymbol{x}.
} J ∗ ( x , t f ) = 2 1 x T S f x . Classification of the PDE ¶ The HJB equation is a first-order nonlinear partial differential equation:
first order in time because it contains J t ∗ J_t^* J t ∗
first order in the state because it contains J x ∗ J_{\boldsymbol{x}}^* J x ∗
nonlinear because J x ∗ J_{\boldsymbol{x}}^* J x ∗
The terminal condition is specified at t f t_f t f
Why an Additional Structural Assumption Is Needed ¶ The PDE contains derivatives with respect to both time and state. For a general nonlinear problem, one must solve the PDE over the full state-time domain.
The LQR problem is special because the terminal cost is quadratic and the system and running cost preserve quadratic structure. This motivates a quadratic value function.
Quadratic Value-Function Ansatz ¶ Assume
J ∗ ( x , t ) = 1 2 x T S ( t ) x , \boxed{
J^*(\boldsymbol{x},t)
=
\frac{1}{2}\boldsymbol{x}^\mathsf{T}\boldsymbol{S}(t)\boldsymbol{x},
} J ∗ ( x , t ) = 2 1 x T S ( t ) x , where
S ( t ) = S T ( t ) . \boldsymbol{S}(t)=\boldsymbol{S}^\mathsf{T}(t). S ( t ) = S T ( t ) . This form is consistent with the terminal boundary condition.
At t = t f t=t_f t = t f
1 2 x T S ( t f ) x = 1 2 x T S f x . \frac{1}{2}\boldsymbol{x}^\mathsf{T}\boldsymbol{S}(t_f)\boldsymbol{x}
=
\frac{1}{2}\boldsymbol{x}^\mathsf{T}\boldsymbol{S}_f\boldsymbol{x}. 2 1 x T S ( t f ) x = 2 1 x T S f x . Hence,
x T [ S ( t f ) − S f ] x = 0 for all x . \boldsymbol{x}^\mathsf{T}
\left[
\boldsymbol{S}(t_f)-\boldsymbol{S}_f
\right]
\boldsymbol{x}
=
0
\quad
\text{for all }\boldsymbol{x}. x T [ S ( t f ) − S f ] x = 0 for all x . Therefore,
S ( t f ) = S f . \boxed{
\boldsymbol{S}(t_f)=\boldsymbol{S}_f.
} S ( t f ) = S f . State Gradient of the Value Function ¶ Since S \boldsymbol{S} S
J x ∗ = S ( t ) x . \boxed{
J_{\boldsymbol{x}}^*
=
\boldsymbol{S}(t)\boldsymbol{x}.
} J x ∗ = S ( t ) x . To see this,
∂ ∂ x ( 1 2 x T S x ) = 1 2 ( S + S T ) x = S x . \frac{\partial}{\partial\boldsymbol{x}}
\left(
\frac{1}{2}\boldsymbol{x}^\mathsf{T}\boldsymbol{S}\boldsymbol{x}
\right)
=
\frac{1}{2}
(\boldsymbol{S}+\boldsymbol{S}^\mathsf{T})\boldsymbol{x}
=
\boldsymbol{S}\boldsymbol{x}. ∂ x ∂ ( 2 1 x T S x ) = 2 1 ( S + S T ) x = S x . Time Derivative of the Value Function ¶ The partial derivative with respect to time holds x \boldsymbol{x} x
J t ∗ = 1 2 x T S ˙ ( t ) x . \boxed{
J_t^*
=
\frac{1}{2}
\boldsymbol{x}^\mathsf{T}\dot{\boldsymbol{S}}(t)\boldsymbol{x}.
} J t ∗ = 2 1 x T S ˙ ( t ) x . Only the explicit time dependence of S ( t ) \boldsymbol{S}(t) S ( t )
Substitution into the HJB Equation ¶ Substituting
J x ∗ = S x , J t ∗ = 1 2 x T S ˙ x \begin{aligned}
J_{\boldsymbol{x}}^*&=\boldsymbol{S}\boldsymbol{x},\\
J_t^*&=\frac{1}{2}\boldsymbol{x}^\mathsf{T}\dot{\boldsymbol{S}}\boldsymbol{x}
\end{aligned} J x ∗ J t ∗ = S x , = 2 1 x T S ˙ x into the HJB equation gives
0 = 1 2 x T S ˙ x + 1 2 x T Q x − 1 2 x T S B R − 1 B T S x + x T S A x . \begin{aligned}
0
={}&
\frac{1}{2}\boldsymbol{x}^\mathsf{T}\dot{\boldsymbol{S}}\boldsymbol{x}
+
\frac{1}{2}\boldsymbol{x}^\mathsf{T}\boldsymbol{Q}\boldsymbol{x}
-
\frac{1}{2}
\boldsymbol{x}^\mathsf{T}
\boldsymbol{S}\boldsymbol{B}\boldsymbol{R}^{-1}\boldsymbol{B}^\mathsf{T}\boldsymbol{S}
\boldsymbol{x}
\nonumber\\
&+
\boldsymbol{x}^\mathsf{T}\boldsymbol{S}\boldsymbol{A}\boldsymbol{x}.
\end{aligned} 0 = 2 1 x T S ˙ x + 2 1 x T Q x − 2 1 x T S B R − 1 B T S x + x T S A x . The final term is scalar. Therefore,
x T S A x = 1 2 x T ( S A + A T S ) x . \boldsymbol{x}^\mathsf{T}\boldsymbol{S}\boldsymbol{A}\boldsymbol{x}
=
\frac{1}{2}
\boldsymbol{x}^\mathsf{T}
\left(
\boldsymbol{S}\boldsymbol{A}+\boldsymbol{A}^\mathsf{T}\boldsymbol{S}
\right)
\boldsymbol{x}. x T S A x = 2 1 x T ( S A + A T S ) x . Symmetrization of the Cross Term ¶ For any square matrix M \boldsymbol{M} M
x T M x = x T [ 1 2 ( M + M T ) ] x . \boldsymbol{x}^\mathsf{T}\boldsymbol{M}\boldsymbol{x}
=
\boldsymbol{x}^\mathsf{T}
\left[
\frac{1}{2}
(\boldsymbol{M}+\boldsymbol{M}^\mathsf{T})
\right]
\boldsymbol{x}. x T M x = x T [ 2 1 ( M + M T ) ] x . The skew-symmetric part contributes nothing because
x T [ 1 2 ( M − M T ) ] x = 0. \boldsymbol{x}^\mathsf{T}
\left[
\frac{1}{2}
(\boldsymbol{M}-\boldsymbol{M}^\mathsf{T})
\right]
\boldsymbol{x}
=
0. x T [ 2 1 ( M − M T ) ] x = 0. Applying this to M = S A \boldsymbol{M}=\boldsymbol{S}\boldsymbol{A} M = S A
x T S A x = 1 2 x T ( S A + A T S ) x . \boldsymbol{x}^\mathsf{T}\boldsymbol{S}\boldsymbol{A}\boldsymbol{x}
=
\frac{1}{2}
\boldsymbol{x}^\mathsf{T}
(\boldsymbol{S}\boldsymbol{A}+\boldsymbol{A}^\mathsf{T}\boldsymbol{S})
\boldsymbol{x}. x T S A x = 2 1 x T ( S A + A T S ) x . The HJB equation becomes
1 2 x T [ S ˙ + Q − S B R − 1 B T S + S A + A T S ] x = 0. \frac{1}{2}
\boldsymbol{x}^\mathsf{T}
\left[
\dot{\boldsymbol{S}}
+
\boldsymbol{Q}
-
\boldsymbol{S}\boldsymbol{B}\boldsymbol{R}^{-1}\boldsymbol{B}^\mathsf{T}\boldsymbol{S}
+
\boldsymbol{S}\boldsymbol{A}
+
\boldsymbol{A}^\mathsf{T}\boldsymbol{S}
\right]
\boldsymbol{x}
=
0. 2 1 x T [ S ˙ + Q − S B R − 1 B T S + S A + A T S ] x = 0. Because this must hold for every x \boldsymbol{x} x
S ˙ + Q − S B R − 1 B T S + S A + A T S = 0 . \boxed{
\dot{\boldsymbol{S}}
+
\boldsymbol{Q}
-
\boldsymbol{S}\boldsymbol{B}\boldsymbol{R}^{-1}\boldsymbol{B}^\mathsf{T}\boldsymbol{S}
+
\boldsymbol{S}\boldsymbol{A}
+
\boldsymbol{A}^\mathsf{T}\boldsymbol{S}
=
\boldsymbol{0}.
} S ˙ + Q − S B R − 1 B T S + S A + A T S = 0 . Part II develops the Riccati equation, feedback law, implementation, interpretation, and verification.
Summary ¶ The finite-horizon LQR problem has linear dynamics and quadratic cost.
The HJB Hamiltonian is minimized with respect to the control.
The minimizing control is
u ∗ = − R − 1 B T J x ∗ . \boldsymbol{u}^*=-\boldsymbol{R}^{-1}\boldsymbol{B}^\mathsf{T}J_{\boldsymbol{x}}^*. u ∗ = − R − 1 B T J x ∗ . The HJB PDE is nonlinear in the value-function gradient.
A quadratic value-function ansatz is consistent with the problem structure.
The terminal condition gives
S ( t f ) = S f . \boldsymbol{S}(t_f)=\boldsymbol{S}_f. S ( t f ) = S f . Substitution reduces the HJB PDE to a matrix differential equation.
Connection. The quadratic ansatz now yields the Riccati equation, the optimal feedback law, and a practical backward-in-time numerical procedure.