Linear Algebra

Scalars, vectors, and matrices are fundamental structures of linear algebra, and understanding them is integral to unlock the concepts of deep learning.

• A scalar is a singular quantity like a number.
• A vector is an array of numbers (scalar values).
• A matrix is a grid of information with rows and columns.

They can all be represented in Python using the NumPy library.

In matrix addition, corresponding matrix entries are added together.

In matrix multiplication, the number of rows of the first matrix must be equal to the number of columns of the second matrix. The dot product between each row and column of the matrix and placed as an entry into the resulting matrix as shown in the image.

A matrix transpose switches the rows and columns of a matrix.

To calculate the magnitude of a vector, use the following formula:

$||v|| = \sqrt{v_{1}^2 + v_{2}^2 + \dots + v_{n}^2}$

For example, if we have the following 3D vector:

$v = \begin{bmatrix}
3 \\
6 \\
-6 
\end{bmatrix}$

To calculate the magnitude, we do the following:

$\begin{aligned}
||v|| = \sqrt{3^2 + 6^2 +(-6)^2} \\
||v|| = \sqrt{81} \\
||v|| = 9
\end{aligned}$

We can also calculate the magnitude of a vector using the np.linalg.norm() function from the NumPy library.

Vectors can be added and subtracted from each other when they are of the same dimension (same number of components). Doing so adds or subtracts corresponding elements, resulting in a new vector of the same dimension as the two being summed or subtracted. Any vector can also be multiplied by a scalar, which results in every element of that vector being multiplied by that scalar individually.

$\begin{bmatrix}
           x_{1} \\
           y_{1} \\
           z_{1}
    \end{bmatrix} 
+
2 \begin{bmatrix}
           x_{2} \\
           y_{2} \\
           z_{2}
    \end{bmatrix} 
-
3 \begin{bmatrix}
           x_{3} \\
           y_{3} \\
           z_{3}
    \end{bmatrix} 
=
 \begin{bmatrix}
           x_{1} + 2x_{2} - 3x_{3} \\
           y_{1} + 2y_{2} - 3y_{3}\\
           z_{1} + 2z_{2} - 3z_{3}
    \end{bmatrix}$

Using NumPy, we can add equally sized vectors and matrices together using built-in Python addition between NumPy arrays. We can also use built-in Python multiplication to perform scalar multiplication on NumPy arrays. The code example shows an example implementation of both of these.

The dot product of two vectors measures how much one vector “goes into” the other vector by summing the products of the vectors’ corresponding components. The formula for this is:

$a \cdot b = \sum\limits_{i=1}^{n} a_{i}b_{i}$

To recall how to use this formula, consider the following two vectors:

$a = \begin{bmatrix}
           4 \\
           0 \\
           -1
    \end{bmatrix},
b = \begin{bmatrix}
           2 \\
           8 \\
           -2
    \end{bmatrix}$

To find the dot product between these two vectors, we do the following:

$a \cdot b = 4*2 + 0*-8 + -1*-2 = 10$

We can use NumPy to compute vector dot products by using the np.dot() function. An example implementation is shown.

The inverse of a matrix, A^-1, is one where the following equation is true:

$AA^{-1} = A^{-1}A = I$

We can use Gauss-Jordan elimination to solve for the inverse of a square matrix by hand (if one exists). With NumPy, we can use np.linalg.inv() to solve for the inverse of a square matrix. An example implementation is shown in the code block.

The identity matrix is a square matrix of elements equal to 0 except for the elements along the diagonal that are all equal to 1.

$\begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{bmatrix}$

Any matrix multiplied by the identity matrix, either on the left or right side, will be equal to itself.

$\begin{bmatrix}
4 & 3 & -2 \\
-3 & 0 & 6 \\
1 & 2 & 8 \\
\end{bmatrix} \cdot 
\begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{bmatrix} = 
\begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{bmatrix} \cdot
\begin{bmatrix}
4 & 3 & -2 \\
-3 & 0 & 6 \\
1 & 2 & 8 \\
\end{bmatrix} =
\begin{bmatrix}
4 & 3 & -2 \\
-3 & 0 & 6 \\
1 & 2 & 8 \\
\end{bmatrix}$

Using NumPy, we can create any n x n identity matrix using the np.eye() function. It takes one argument which determines the size of the matrix. A code implementation example is shown.

A permutation matrix is a square matrix that allows us to flip rows and columns of a separate matrix.