Introduction

Suppose you held up an object and began rotating it at a fixed speed. After some amount of time, you stop rotating it. If I know the starting pose of the object and the amount of time that has passed, under what conditions can I also tell you the rotational velocity (both direction and speed) of the spinning object?

There's really just one condition: that the rotational distance (i.e. angle) between the initial and final positions does not exceed some maximum value. For example, if you rotate 180° in any direction, it's impossible for me to know whether you rotated the object clockwise or counterclockwise. Worse, if you rotate a full 360°, I couldn't know whether you didn't move the object at all or performed a whole rotation or a million rotations. This maximum allowed distance that allows one to still infer the initial and final orientation is called the injectivity radius^[1].

The question motivating this post is, what is the injectivity radius for the rotations in not just 2 dimensions and 3 dimensions but any dimension? And more generally, what is it for the unitary group and its most common subgroups? Since these groups are featured in many (all?) of the introductory texts on Lie groups and manifolds, and since the injectivity radius is a basic property introduced in differential geometry textbooks, I was surprised that I could not find a single reference giving these radii for these groups.

In this post I'll work out these radii. Marvelously, we don't need any differential geometry or group theory to do this, just linear algebra! Nevertheless, this post assumes familiarity with these topics and for the sake of space will try not define all common terms or notation.^[2]

The injectivity radius

Consider a point \(p\) on some manifold \(\mathcal{M}\) with tangent vectors \(X,Y \in T_p \mathcal{M}\). Assume a Riemannian metric \(g_p\) defining an inner product \(g_p\colon (X, Y) \mapsto \left\langle X, Y \right\rangle_g\), which induces a norm \(\left\lVert X \right\rVert_g\).

Let's denote the exponential map \(\exp_p\colon X \mapsto q\) for \(q \in \mathcal{M}\) and the logarithmic map \(\log_p\colon q \mapsto Y\). The injectivity radius at \(p\) is defined as the norm of the smallest \(X\) for which \(X \ne Y = \log_p (\exp_p X)\), or in other words, the smallest \(X\) for which the logarithmic map no longer is the inverse of the exponential map. We further define the global injectivity radius of \(\mathcal{M}\) as the infimum of the injectivity radii at all points on the manifold.

Notationally, we define this global injectivity radius as

We'll also consider the related supremum

These two quantities form the lower and upper bound radii, respectively, of two geodesic balls within which the exponential map is invertible.

The unitary group(s)

The Unitary group \(\mathrm{U}(n, \mathbb{F})\) over some number system \(\mathbb{F}\) is the group of all \(n \times n\) matrices \(p \in \mathbb{F}^{n \times n}\) for which \(p^\mathrm{H} p = I_n\), where \({\cdot}^\mathrm{H}\) denotes the matrix adjoint. \(\mathbb{F}\) could be the real numbers \(\mathbb{R}\), complex numbers \(\mathbb{C}\), or quaternions \(\mathbb{H}\).

We will also deal with the following subgroups:

• \(\mathrm{SU}(n)\): the special unitary group, which consists of complex unitary matrices whose determinant is +1

• \(\mathrm{O}(n) \equiv \mathrm{U}(n, \mathbb{R})\): the orthogonal group, i.e. the group of rotations and reflections

• \(\mathrm{SO}(n)\): the special orthogonal group, i.e. the group of real rotations

We will focus on the real and complex fields, but the unitary quaternionic case immediately follows from the complex one.

Relevant geometric properties

The unitary group is a compact group and when equipped with the Frobenius inner product \(g\colon (X, Y) \mapsto \left\langle X, Y \right\rangle_\mathrm{F}\) becomes a Riemannian manifold.^[3] The Riemannian exponential \(\exp_p\) and logarithm \(\log_p\) are related to the Lie group exponential \(\operatorname{Exp}\) and logarithm \(\operatorname{Log}\), which for these matrix groups are just the matrix exponential and logarithm.

In the following then, \(p\) is always the identity matrix and will not be mentioned, while \(X\) is always an element of the Lie algebra, that is, the tangent space at the identity matrix.

The orthogonal group \(\mathrm{O}(n)\) is comprised of two submanifolds, \(\mathrm{SO}(n)\), whose elements have determinant +1, and another subgroup whose elements have determinant -1. These submanifolds are disconnected, so that the geodesic cannot join two points from the different submanifolds.

Relevant linear algebraic properties

All unitary matrices have a unit determinant \(|\det(q)| = 1\). The inverse of any unitary matrix \(q\) is just its adjoint \(q^{-1} = q^\mathrm{H}\)

The logarithm of any unitary matrix is a skew-hermitian matrix \(X = -X^\mathrm{H}\).^[4] Unitary and skew-hermitian matrices are normal matrices, which means they are always diagonalizable with unitary eigenvectors. Let \(q=VSV^\mathrm{H}\) be the eigendecomposition of \(q\) and \(X = U \Lambda U^\mathrm{H}\) be the eigendecomposition of \(X\).

The unitary/skew-Hermitian condition then implies

The norm of \(X\) under the Frobenius metric is

The special unitary group has the additional constraint that the determinant is +1. This constraint implies that \(\sum_{i=1}^n \theta_i = 0\). Given \(n-1\) values of \(\theta_i\), the \(n\)th value is thus fixed.

Special orthogonal matrices are real, and real matrices have the extra property that if they have a complex eigenvalue \(\lambda_i\) then they also have a complex eigenvalue \(\lambda_i^*\), i.e. the complex eigenvalues come in conjugate pairs. The sum of the elements in each pair is thus 0, and they don't contribute to the the sum \(\sum_{i=1}^n \theta_i\). When \(n\) is odd, at least one eigenvalue is 0.

Using the eigendecomposition, the matrix exponential and logarithm can be computed by applying the corresponding function to the eigenvalues. Then

The injectivity radii

We are finally ready to work out the injectivity radii. For \(\mathrm{U}(n, \mathbb{F})\), the eigenvalues of \(X\) are \(\lambda_i = \mathrm{i} \theta_i\) for \(\theta_i \in \mathbb{R}\).

The matrix exponential is invertible when \(\exp(\mathrm{i} \theta_i)\) is invertible. But this is just rotation in the complex plane by the angle \(\theta_i\), which is invertible for \(\theta_i \in (-\pi, \pi]\). The injectivity radii are then computed by the constraint \(|\theta_i| = \pi\) for at least one \(i \in 1\ldots n\).

For \(\mathrm{U}(n, \mathbb{F})\) and \(\mathbb{F} \not\equiv \mathbb{R}\), the largest value of \(\left\lVert \theta \right\rVert\) achievable subject to the constraints occurs when \(|\theta_i| = \pi\) for all \(i\). Then \(\left\lVert \theta \right\rVert = \pi \sqrt{n}\). The smallest value of \(\left\lVert \theta \right\rVert\) occurs when \(|\theta_i| = \begin{cases}\pi, & i = k\\ 0, & i \ne k\end{cases}\) for any \(k \in 1\ldots n\). So

For \(\mathrm{SU}(n)\), we have the additional constraint that \(\sum_{i=1}^n \theta_i = 0\). For even \(n\), \(\left\lVert \theta \right\rVert\) is maximized subject to these constraints when \(n/2\) entries in \(\theta\) are \(+\pi\) and when \(n/2\) entries are \(-\pi\). On the other hand, for odd \(n\), it is maximized when \((n-1)/2\) values each are \(+\pi\) and \(-\pi\) and the \(n\)th value is \(0\). \(\left\lVert \theta \right\rVert\) is minimized subject to these constraints when there is a single nonzero pair \(\theta_i = -\theta_j = \pi\) for \(j \ne i\). As a result,

The constraints on \(\theta\) required for \(\mathrm{O}(n)\) and \(\mathrm{SO}(n)\) are also satisfied by the lower and upper bounds of the norms considered for \(\mathrm{SU}(n)\), so

It's always a good idea to numerically check that the results make sense. For the 2D and 3D rotations \(\mathrm{SO}(2)\) and \(\mathrm{SO}(3)\), respectively, we then have

These injectivity radii correspond to a rotation of 180°. ^[3] Coming back to our motivating example, if we start from any pose and rotate an object more than 180° in any direction, we can no longer uniquely determine the initial rotational velocity.

\(\mathrm{U}(1, \mathbb{C})\), the complex unit circle, is isomorphic to \({\mathrm{SO}(2)}\), so it may seem like a contradiction that its injectivity radius is \(\pi\). But this difference is again caused by the choice of metric, which causes the inner product on \({\mathrm{SO}(2)}\) to be scaled by \(\frac{1}{2}\) compared to \(\mathrm{U}(1, \mathbb{C})\).^[3]

Analogously, \(\mathrm{U}(1, \mathbb{H})\) represents the unit quaternions (also the compact symplectic group), which are an alternative way to represent 3D rotations, and is equivalent to \(\mathrm{SU}(2)\). The injectivity radii again only differ by the factor of \(\sqrt{2}\) due to the scaling convention of the metric.

Conclusion

This has been one of the rare moments where we got to dabble with group theory and manifolds without needing too much geometry. I hope it was enjoyable!

[1]	The injectivity radius is so called because it is the radius of a geodesic ball within which the exponential map is injective (one-to-one).

[2]	It's really hard to write anything about manifolds or groups without writing a whole introductory text to differential geometry or group theory.

[3]

Because of the skew-hermitian nature of the elements of the Lie algebra, some texts use the scaled Frobenius metric \(g(X, Y) = \frac{1}{2}\left\langle X, Y \right\rangle_\mathrm{F}\). In some cases, this allows the norm of a tangent vector to be interpreted as the angle of the rotation. To get the injectivity radii for this metric, one would just divide ours by \(\sqrt{2}\).

[4]

This comes from differentiating the constraint \(q^\mathrm{H}q=I_n\). Then we have \(\mathrm{d}{(q^\mathrm{H}q)} = (\mathrm{d}{q})^\mathrm{H}q + q^\mathrm{H} \mathrm{d}{q} = \mathrm{d}(I_n) = 0\). Letting \(X = \mathrm{d}{q}\), then \(X^\mathrm{H}q = -q^\mathrm{H} X = -(X^\mathrm{H}q)^\mathrm{H}\). When \(q\) is the identity matrix, \(X = -X^\mathrm{H}\).

[5]	For the complex unitary group, \(\mathrm{i}\) is the usual imaginary number, while for quaternions, it would be a pure unit quaternion.

Introduction

For a square matrix \(A \in \mathbb{C}^{m \times n}\), the LU decomposition is useful for solving systems of equations, inverting \(A\), and computing the determinant.

This post assumes familiarity with terminology of forward- and reverse-mode differentiation rules. For a succinct review, I recommend reading the ChainRules guide on deriving array rules.

Definition

The LU decomposition of \(A\) is

Where \(q = \min(m, n)\), \(L \in \mathbb{C}^{m \times q}\) is a unit lower triangular matrix, that is, a lower triangular matrix whose diagonal entries are all ones. \(U \in \mathbb{C}^{q \times n}\) is an upper triangular matrix. \(P \in \mathbb{R}^{m \times m}\) is a permutation matrix whose action on a matrix \(X\) (i.e. \(P X\)) reorders the rows of \(X\). As a permutation matrix, \(P P^\mathrm{T} = P^\mathrm{T} P = I\).

Uses

If we have a system of the form of \(A X = B\) and would like to solve for \(X\), we can use the LU decomposition to write \(L U X = P B\). Then \(X = L^{-1} (U^{-1} (P B))\). Note that the row-swapping action \(P B\) can be computed in-place. We can also easily compute the left-division by the triangular matrices in-place using forward and back substitution.

By setting \(B\) in the above equation to be the identity matrix, then we can compute the inverse of \(A\), that is \(X = A^{-1}\).

The determinant of \(A\) is \(\det(A) = \det(P) \det(L) \det(U)\). The determinant of a triangular matrix is the product of its diagonal entries, so \(\det(L) = 1\), and \(\det(U) = \prod_{i=1}^n U_{ii}\). \(\det(P) = (-1)^s\), where \(s\) is the number of row swaps encoded by the permutation matrix. So \(\det(A) = (-1)^s \prod_{i=1}^n U_{ii}\), which is very cheap to compute from the decomposition.

The LU decomposition can still be computed when \(A\) is wide (\(m < n\)) or tall (\(m > n\)). However, none of the above applications make sense in this case, and I don't know what it's useful for.

Motivation

Often the LU decomposition is computed using LAPACK subroutines, which cannot be automatically differentiated through. Hence, it is necessary to implement custom automatic differentiation rules. ^[1] derived a pushforward (AKA forward-mode differentiation rule or Jacobian-vector-product) for the LU decomposition for the square case, but I couldn't find a rule for the wide or tall cases. This is a problem, because I wanted to implement a generic rule for ChainRules.jl that would work for all dense matrices in Julia, where square and non-square dense matrices are all implemented using the same type Matrix (more specifically, as the union of types StridedMatrix). It is not ideal to write a custom rule that will work for only a subset of matrices of a given type.

We could always pad \(A\) with zeros to make it square. However, if \(m \ll n\) or \(m \gg n\), then this is wasteful. JAX seems to use this approach, though it's possible that internally it does something fancier and doesn't explicitly allocate or operate on the padding.

Thankfully, it's not too hard to work out the pushforwards and pullbacks for the non-square case by splitting the matrices into blocks and working out the rules for the individual blocks. So in this post, we'll review the pushforward for square \(A\), also working out its pullback, and then we'll do the same for wide and tall \(A\).

Square \(A\)

A pushforward for the LU decomposition for square \(A\) is already known ^[1]. For completeness, I've included its derivation, as well as one for the corresponding pullback.

Pushforward

We start by differentiating the defining equation:

We can solve both sides to get

\(\dot{L}\) and \(\dot{U}\) must be at least as sparse as \(L\) and \(U\), respectively. Hence, \(\dot{U}\) is upper triangular, and because the diagonal of \(L\) is constrained to be unit, \(\dot{L}\) will be lower triangular with a diagonal of zeros (strictly lower triangular). Note also that the inverse of a lower/upper triangular matrix is still lower/upper triangular. Likewise, the product of two lower/upper triangular matrices is still lower/upper triangular.

Hence the right-hand side is the sum of a strictly lower triangular matrix \(L^{-1} \dot{L}\) and an upper triangular matrix \(\dot{U} U^{-1}\). Let's introduce the triangularizing operators \(\operatorname{triu}(X)\), which extracts the upper triangle of the matrix \(X\), and \(\operatorname{tril}_-(X)\), which extracts its strict lower triangle (so that \(X = \operatorname{tril}_-(X) + \operatorname{triu}(X)\)).

Introducing an intermediate \(\dot{F}\), we can then solve for \(\dot{L}\) and \(\dot{U}\):

Pullback

The corresponding pullback is

Note that these expressions use the same elementary operations as solving a system of equations using the LU decomposition, as noted above. The pushforwards and pullbacks can then be computed in-place with no additional allocations.

Wide \(A\)

We can write wide \(A\) in blocks \(A = \begin{bmatrix}A_1 & A_2 \end{bmatrix}\), where \(A_1 \in \mathbb{C}^{m \times m}\) and \(A_2 \in \mathbb{C}^{m \times (n - m)}\). It will turn out to be very convenient that \(A_1\) is square. The LU decomposition in terms of these blocks is written

where \(U_1\) is square upper triangular. This is a system of two equations that we will address separately.

Pushforward

Introducing an intermediate \(\dot{H} = \begin{bmatrix} \dot{H}_1 & \dot{H}_2 \end{bmatrix}\) with the same block structure as \(U\), the complete pushforward is

Pullback

Introducing an intermediate \(\overline{H} = \begin{bmatrix} \overline{H}_1 & \overline{H}_2 \end{bmatrix}\) with the same block structure as \(U\), the corresponding pullback is

Tall \(A\)

The tall case is very similar to the wide case, except now we have the block structure

where now \(L_1\) is square unit lower triangular, and \(U\) is square upper triangular. This gives us the system of equations

Pushforward

The first equation is again identical to the square case, so we can use it to solve for \(\dot{L}_1\) and \(\dot{U}\). Likewise, the same approach we used to solve \(\dot{U}_2\) in the wide case can be applied here to solve for \(\dot{L}_2\).

Introducing an intermediate \(\dot{H} = \begin{bmatrix} \dot{H}_1 \\ \dot{H}_2 \end{bmatrix}\) with the same block structure as \(L\), the complete pushforward is

Pullback

Introducing an intermediate \(\overline{H} = \begin{bmatrix} \overline{H}_1 \\ \overline{H}_2 \end{bmatrix}\) with the same block structure as \(L\), the corresponding pullback is

Implementation

The product of this derivation is this pull request to ChainRules.jl, which includes tests of the rules using FiniteDifferences.jl.

Conclusion

The techniques employed here are general and can be used for differentiation rules for other factorizations of non-square matrices. A recent paper used a similar approach to derive the pushforwards and pullbacks of the \(QR\) and \(LQ\) decompositions^[2].

References

[1]	de Hoog F.R., Anderssen R.S., and Lukas M.A. (2011) Differentiation of matrix functionals using triangular factorization. Mathematics of Computation, 80 (275). p. 1585. doi: 10.1090/S0025-5718-2011-02451-8.

[2]	Roberts D.A.O. and Roberts L.R. (2020) QR and LQ Decomposition Matrix Backpropagation Algorithms for Square, Wide, and Deep – Real or Complex – Matrices and Their Software Implementation. arXiv: 2009.10071.