A Quick Introduction to Orthonormal Matrices

Photo by Possessed Photography on Unsplash

Setting the stage

Before dwelling into “orthonormal”, we must first understand what “orthogonal” means.

In simple terms, two vectors x and y are said to be “orthogonal” if they make an angle of 90 degrees with each other.

So how do we find if two vectors are orthogonal?

Well, that’s really simple! The vectors x and y are orthogonal if their dot product is equal to 0.

Consider two vectors of length n

Their dot product (also called inner product) is defined as follows:

Mathematically, the vectors x and y are orthogonal if

Now, what about “orthonormal”?

The vectors x and y are now orthonormal if they both are of unit length as well as orthogonal. Mathematically,

It is important to note that while both “orthogonal” and “orthonormal” pairs of vectors are perpendicular to each other, the “orthogonal” vectors need not be of length 1. Hence, any pair of orthonormal vectors are also orthogonal (but the vice versa isn’t true).

Moving from vector to matrix

An orthogonal matrix Q is a square matrix whose columns are all orthonormal i.e., orthogonal unit vectors.

Mathematically, Q is orthonormal if the following conditions are satisfied:

In simpler words, the magnitude of every column of the orthonormal matrix is 1, and each column is perpendicular to the other.

Examples of orthonormal matrices

→ An identity matrix is the simplest orthonormal matrix

→ The permutation of an identity matrix

→ Any rotation matrix is an orthonormal matrix. For example, the 3x3 rotation along Z-axis as shown below

Visual example of rotation matrices:

In three dimensions, rotation can be applied in three directions i.e., X, Y and Z directions.

The following matrices are used in order to apply rotation in the corresponding directions:

The figure below consists of the following plots:

1. Basis vectors in X, Y and Z directions
2. Basis vectors rotated along X direction by 45 degrees
3. Basis vectors rotated along Y direction by -15 degrees
4. Basis vectors rotated along Z direction by 30 degrees

While it may not be visibly obvious from the plots, the three colored lines are indeed perpendicular to each other. Also, the length of each of the colored lines in the rotated axes are still 1. Hence, the property that orthonormal matrix preserves the length of a vector is experimentally proved.

The Octave code used to generate the above figure is provided below for any further exploration.

Few properties of orthonormal matrices:

Some of the properties of orthonormal matrices are listed below, along with the proofs. It is important to know that this list is non-exhaustive, and the goal here is to give

The inverse of an orthonormal matrix is its transpose

As can be seen above, multiplying the orthonormal matrix Q with its transpose yielded the identity matrix, thereby proving that the transposed matrix is indeed the inverse of the original matrix Q.

The transpose of the orthonormal matrix is also orthonormal

Given that Q is orthonormal,

The columns of the matrix are orthonormal

This directly follows the definition.

The rows of the matrix are orthonormal

Since the transpose of the matrix Q is orthonormal, the columns of Q^T are orthonormal. However, these columns form the rows in Q, thereby proving that the rows of Q are orthonormal.

An orthonormal matrix is non-singular

It is already established that for any orthonormal matrix, the inverse exists and is the transpose of the matrix itself.

This proves the following properties:

1. The orthonormal matrix is non-singular
2. Determinant of an orthonormal matrix is NOT equal to 0
3. 0 is NOT an eigen values of the orthonormal matrix

Transforming a vector by an orthonormal matrix does not change the magnitude of the vector

Mathematically,

Proof:

The product of two orthonormal matrix is also orthonormal

Given two orthonormal matrices A and B

The determinant of an orthonormal matrix is +1 or -1

The eigenvalues of an orthonormal matrix are always +1 or -1

If Q is an orthonormal matrix,

Extending from real to complex matrices

So far so good. But what happens when Q is a complex matrix consisting of one or more complex numbers?

A complex matrix whose columns are orthonormal is called a Unitary matrix. Unitary matrix also satisfies the above-mentioned properties, albeit with a minor change as below:

i.e., the inverse of a unitary matrix is the conjugate transpose or Hermitian transpose of the same (One may recall that the inverse of a real orthonormal matrix is only the transpose).

The conjugate transpose or Hermitian transpose of a matrix is obtained by first transposing the matrix and then taking the complex conjugate of every element of the matrix. An example is shown below.

Note: In literature, one may also come across the following notation for the conjugate transpose:

Conclusion:

For a novice reader, any rotation matrix is the most obvious example or orthonormal matrix. However, orthonormal and unitary matrices find applications in various aspects of linear algebra such as eigenvalue decomposition, spectral decomposition, Principal Component Analysis (PCA) etc. which form the basis for several real-world applications.

References and Further Reading:

1. David W Lewis, “Matrix Theory”
2. R A Horn & C R Johnson, “Matrix Analysis”
3. Wikipedia article on orthonormal matrix: https://en.wikipedia.org/wiki/Orthogonal_matrix
4. Wikipedia article on rotation matrix: https://en.wikipedia.org/wiki/Rotation_matrix