Learn the fundamentals of Linear Algebra!

Introduction to Linear Algebra

Now that we have our system of linear equations in augmented matrix form, we can solve for the unknown variables using a technique called *Gauss-Jordan Elimination*. In regular algebra, we may try to solve the system by combining equations to eliminate variables until we can solve for a single one. Having one variable solved for then allows us to solve for a second variable, and we can continue that process until all variables are solved for.

The same goal can be used for solving for the unknown variables when in the matrix representation. We start with forming our augmented matrix.

```tex
\begin{bmatrix}
    a_{1} & b_{1} & c_{1} \bigm| d_{1} \\
    a_{2} & b_{2} & c_{2} \bigm| d_{2} \\
    a_{3} & b_{3} & c_{3} \bigm| d_{3} \\ 
\end{bmatrix} 
```

To solve for the system, we want to put our augmented matrix into something called *row echelon form* where all elements below the diagonal are equal to zero. This looks like the following:

```tex
\begin{bmatrix}
    a_{1} & b_{1} & c_{1} \bigm| d_{1} \\
    a_{2} & b_{2} & c_{2} \bigm| d_{2} \\
    a_{3} & b_{3} & c_{3} \bigm| d_{3} \\ 
\end{bmatrix} 
\rightarrow

\begin{bmatrix}
    1 & b_{1}' & c_{1}' \bigm| d_{1}' \\
    0 & 1 & c_{2}' \bigm| d_{2}' \\
    0 & 0 & 1 \hspace{1.25mm} \bigm| d_{3}' \\ 
\end{bmatrix}

```
Note that the values with apostrophes in the *row echelon form* matrix mean that they have been changed in the process of updating the matrix. Once in this form we can rewrite our original equation as:

```tex
\begin{aligned}
x + b_{1}' y + c_{1}'z = d_{1}' \\
y + c_{2}'z = d_{2}' \\
     z = d_{3}' \\ 
\end{aligned}
```
This allows us to solve the equations directly using simple algebra. But how do we get to this form?

To get to row echelon form we swap rows and/or add or subtract rows against other rows. A typical strategy is to add or subtract row 1 against all rows below in order to make all elements in column 1 equal to 0 under the diagonal. Once this is achieved, we can do the same with row 2 and all rows below to make all elements below the diagonal in column 2 equal to 0. 

Once all elements below the diagonal are equal to 0, we can simply solve for the variable values, starting at the bottom of the matrix and working our way up.

It’s important to realize that not all systems of equations are solvable! For example, we can perform Gauss-Jordan Elimination and end up with the following augmented matrix.

```tex
\begin{bmatrix}
1 & -2 & -2 \bigm| 5 \\
 \bf 0 & 3 & -1 \bigm| 3 \\
 \bf 0 &  \bf  0 &  \enspace\; {\bf 0 } \bigm| 2 
\end{bmatrix}
```

This final augmented matrix here suggests that 0*z* = 2, which is impossible!

Gauss-Jordan Elimination

An important vector operation in linear algebra is the *dot product*. A dot product takes two equal dimension vectors and returns a single scalar value by summing the products of the vectors’ corresponding components. This can be written out formulaically as:

```tex 
a \cdot b = \sum\limits_{i=1}^{n} a_{i}b_{i}
```

The dot product operation is both commutative *(a <span>&#183;</span> b = b <span>&#183;</span> a)* and distributive *(a <span>&#183;</span> (b+c) = a <span>&#183;</span> b + a <span>&#183;</span> c)*. 

The resulting scalar value represents how much one vector “goes into” the other vector. If two vectors are perpendicular (or orthogonal), their dot product is equal to 0, as neither vector "goes into the other." 

Let's take a look at an example dot product. Consider the following two vectors: 

```tex
a = \begin{bmatrix}
           3 \\
           2 \\
           -3
    \end{bmatrix},
b = \begin{bmatrix}
           0 \\
           -3 \\
           -6
    \end{bmatrix}
```
To find the dot product between these two vectors, we do the following:
```tex
a \cdot b = 3*0 + 2*-3 + -3*-6 = 12
```

The dot product can also be used to find the magnitude of a vector and the angle between two vectors. To find the magnitude, we can reference Exercise 2 and see that the magnitude of a vector is simply the square root of a vector’s dot product with itself. 

```tex
||a|| = \sqrt{a \cdot a}
```
To find the angle between two vectors, we rely on the dot product between the two vectors and use the following equation.

```tex
\theta = \arccos{ \frac{a \cdot b}{||a|| ||b||}}
```

Let's look at the same two vectors from above:

```tex
a = \begin{bmatrix}
           3 \\
           2 \\
           -3
    \end{bmatrix}
b = \begin{bmatrix}
           0 \\
           -3 \\
           -6
    \end{bmatrix}
```

To find the angle between these two vectors, we do the following:

```tex
\theta = \arccos{ \frac{3*0 + 2*-3 + -3*-6 }{\sqrt{(3)^2+(2)^2+(-3)^2}*\sqrt{(0)^2+(-3)^2+(-6)^2}}}
```
Solving this, we end up with:
```tex
\theta = 67.58 \degree
```

Vector Dot Products

Now that we know what a vector is and how to represent one, we can begin to perform various operations on vectors and between different vectors. As we’ve previously discussed, the basis of linear algebra is the linear combinations we can generate between vectors (and matrices).

#### Scalar Multiplication

Any vector can be multiplied by a scalar, which results in every element of that vector being multiplied by that scalar individually. 

```tex
k \begin{bmatrix}
           x \\
           y \\
           z
    \end{bmatrix} 
= 
\begin{bmatrix}
           kx \\
           ky \\
           kz
    \end{bmatrix} 
```

Multiplying vectors by scalars is an *associative operation*, meaning that rearranging the parentheses in the expression does not change the result. For example, we can say *2(a3) = (2a)3*.

#### Vector Addition and Subtraction

Vectors can be added and subtracted from each other when they are of the same dimension (same number of components). Doing so adds or subtracts corresponding elements, resulting in a new vector of the same dimension as the two being summed or subtracted. Below is an example of three-dimensional vectors being added and subtracted together.

```tex
\begin{bmatrix}
           x_{1} \\
           y_{1} \\
           z_{1}
    \end{bmatrix} 
+
2 \begin{bmatrix}
           x_{2} \\
           y_{2} \\
           z_{2}
    \end{bmatrix} 
-
3 \begin{bmatrix}
           x_{3} \\
           y_{3} \\
           z_{3}
    \end{bmatrix} 
=
 \begin{bmatrix}
           x_{1} + 2x_{2} - 3x_{3} \\
           y_{1} + 2y_{2} - 3y_{3}\\
           z_{1} + 2z_{2} - 3z_{3}
    \end{bmatrix} 
```

Vector addition is *commutative*, meaning the order of the terms does not matter. For example, we can say *(a+b = b+a)*. Vector addition is also associative, meaning that *(a + (b+c) = (a+b) + c)*.


Basic Vector Operations

Linear algebra is a branch of mathematics that studies linear transformations, including the use of vectors and matrices. Linear algebra enables us to solve problems like linear regression, as well as computational problems that concern large amounts of data (machine learning, computer graphics, etc.).

In this lesson, we will learn about the following topics:
* Vectors and their basic operations
* The dot product between vectors
* Matrices and their basic operations
* Matrix multiplication and special matrices
* Gauss-Jordan Elimination to solve systems of linear equations
* Gauss-Jordan Elimination to solve for inverse matrices

These topics will give us a solid foundation in linear algebra, providing a strong base to solve more complicated problems like linear regression in future content.


This image shows two intersecting planar graphs

Introduction

Now that we’ve learned about vector quantities, we can expand upon those lessons and focus on matrices. A *matrix* is a quantity with *m* rows and *n* columns of data. For example, we can combine multiple vectors into a matrix where each column of that matrix is one of the vectors. 

To the right, you can see a comparison between scalars, vectors, and matrices for context.

We can also think of vectors as single-column matrices in their own right. Matrices are helpful because they allow us to perform operations on large amounts of data, such as representing entire systems of equations in a single matrix quantity.

Matrices can be represented by using square brackets that enclose the rows and columns of data (elements). The shape of a matrix is said to be *mxn*, where there are *m* rows and *n* columns. When representing matrices as a variable, we denote the matrix with a capital letter and a particular matrix element as the matrix variable with an “*m,n*” determined by the element's location. Let's look at an example of this. Consider the matrix below.

```tex
A= \begin{bmatrix}
a & b & c \\
d & e & f \\
g & h & i

\end{bmatrix}
```
The value corresponding to the first row and second column is *b*. 

```tex
A_{1,2} = b
```

What is the value corresponding to the second row and third column?

```tex
A_{2,3} = ?
```

<details>
    <summary>Solution</summary>
<i>f</i>
</details>
<br>




This image outlines the relationship between a scalar value, a vector, and a matrix. The scalar value depicts a single digit. Next to the scalar, a vector shows a column of with 2 digits. Finally, on the far end, a matrix is labeled with a grid of 4 numbers. The 4 numbers are organized in 2 rows and 2 columns.

Matrices

There are a couple of important matrices that are worth discussing on their own.

#### Identity matrix

The *identity matrix* is a square matrix of elements equal to 0 except for the elements along the diagonal that are equal to 1. Any matrix multiplied by the identity matrix, either on the left or right side, will be equal to itself.

```tex
\begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1 \\
\end{bmatrix}
```

#### Transpose matrix

The *transpose* of a matrix is computed by swapping the rows and columns of a matrix. The transpose operation is denoted by a superscript uppercase “T” (*A<sup>T</sup>*).

```tex
\begin{bmatrix}
a & b & c \\
d & e & f \\
g & h & i \\
\end{bmatrix}
^T
=
\begin{bmatrix}
a & d & g \\
b & e & h \\
c & f & i \\
\end{bmatrix}
```

#### Permutation Matrix

A *permutation matrix* is a square matrix that allows us to flip rows and columns of a separate matrix. Similar to the identity matrix, a permutation matrix is made of elements equal to 0, except for one element in each row or column that is equal to 1. In order to flip rows in matrix *A*, we multiply a permutation matrix *P* on the left (*PA*). To flip columns, we multiply a permutation matrix *P* on the right (*AP*). 

```tex
PA = 
\begin{bmatrix}
0 & 1 & 0 \\
0 & 0 & 1 \\
1 & 0 & 0 \\
\end{bmatrix}
\begin{bmatrix}
a & b & c \\
d & e & f \\
g & h & i \\
\end{bmatrix}
=
\begin{bmatrix}
d & e & f \\
g & h & i \\
a & b & c \\
\end{bmatrix}

```
```tex
AP =
\begin{bmatrix}
a & b & c \\
d & e & f \\
g & h & i \\
\end{bmatrix}
\begin{bmatrix}
0 & 1 & 0 \\
0 & 0 & 1 \\
1 & 0 & 0 \\
\end{bmatrix}
=
\begin{bmatrix}
c & a & b \\
f & d & e \\
i & g & h \\
\end{bmatrix}
```




Special Matrices

The fundamental building blocks of linear algebra are *vectors*. Vectors are defined as quantities having both direction and magnitude, compared to *scalar* quantities that only have magnitude. In order to have direction and magnitude, vector quantities consist of two or more elements of data. The *dimensionality* of a vector is determined by the number of numerical elements in that vector. For example, a vector with four elements would have a dimensionality of four. 

Let's take a look at examples of a scalar versus a vector. A car driving at a speed of 40mph is a scalar quantity. Describing the car driving 40mph to the east would represent a two-dimensional vector quantity since it has a magnitude in both the *x* and *y* directions. 

Vectors can be represented as a series of numbers enclosed in parentheses, angle brackets, or square brackets. In this lesson, we will use square brackets for consistency. For example, a three-dimensional vector is written as:

```tex
v = \begin{bmatrix}
           x \\
           y \\
           z
    \end{bmatrix}
```
The magnitude (or length) of a vector, *||v||*, can be calculated with the following formula:

```tex
||v|| = \sqrt{\sum\limits_{i=1}^{n} v_{i}^2}
```

This formulates translates to the sum of each vector component squared, which can be also written out as:

```tex
||v|| = \sqrt{v_{1}^2 + v_{2}^2 + \dots + v_{n}^2}
```

Let's look at an example. We are told that a ball is traveling through the air and given the velocities of that ball in its *x*, *y*, and *z* directions in a standard Cartesian coordinate system. The velocity component values are:
* *x* = -12
* *y* = 8
* *z* = -2 

Convert the velocities into a vector, and find the total speed of the ball. (Hint: the speed of the ball is the magnitude of the velocity vector!)

Click the dropdown below to check your answers!

<details>
    <summary>Solution</summary>
<img src="https://static-assets.codecademy.com/skillpaths/ds-math/vector-solution.png" alt="The vector form is v = [-12, 8, -2], and the magnitude of the vector is v=14.56.">
</details>
<br>



Vectors

An extremely useful application of matrices is for solving systems of linear equations. Consider the following system of equations in its algebraic form.

```tex
\begin{aligned}
a_1 x + b_1 y + c_1 z = d_1 \\
a_2 x + b_2 y + c_2 z = d_2 \\
a_3 x + b_3 y + c_3 z = d_3
\end{aligned}
```

This system of equations can be represented using vectors and their linear combination operations that we discussed in the previous exercise. We combine the coefficients of the same unknown variables, as well as the equation solutions, into vectors. These vectors are then scalar multiplied by their unknown variable and summed.

```tex
x 
\begin{bmatrix}
a_1 \\
a_2 \\
a_3 \\
\end{bmatrix}

+
y
\begin{bmatrix}
b_1 \\
b_2 \\
b_3 \\
\end{bmatrix}
+
z
\begin{bmatrix}
c_1 \\
c_2 \\
c_3 \\
\end{bmatrix}
=
\begin{bmatrix}
d_1 \\
d_2 \\
d_3 \\
\end{bmatrix}
```

Our final goal is going to be to represent this system in form *Ax = b*, using matrices and vectors. As we learned earlier, we can combine vectors to create a matrix, which we’ll do with the coefficient vectors to create matrix *A*. We can also convert the unknown variables into vector *x*. We end up with the following *Ax = b* form:

```tex
\begin{bmatrix}
a_1 & b_1 & c_1 \\
a_2 & b_2 & c_2 \\
a_3 & b_3 & c_3 \\
\end{bmatrix}
\begin{bmatrix}
x \\
y \\
z \\
\end{bmatrix}
=
\begin{bmatrix}
d_1 \\
d_2 \\
d_3 \\
\end{bmatrix}
```

Click below if you would like to see how these two expressions are equivalent:
```tex
\begin{bmatrix}
a_1 & b_1 & c_1 \\
a_2 & b_2 & c_2 \\
a_3 & b_3 & c_3 \\
\end{bmatrix}
\begin{bmatrix}
x \\
y \\
z \\
\end{bmatrix} =
x 
\begin{bmatrix}
a_1 \\
a_2 \\
a_3 \\
\end{bmatrix}

+
y
\begin{bmatrix}
b_1 \\
b_2 \\
b_3 \\
\end{bmatrix}
+
z
\begin{bmatrix}
c_1 \\
c_2 \\
c_3 \\
\end{bmatrix}

```

<details>
  <summary>Click to see proof</summary>

  <h6> First Step: Matrix Multiplication </h6>
  <img src=https://static-assets.codecademy.com/skillpaths/ds-math/matrix_proof_step1.svg alt="The product of the 3 by 3 matrix [[a subscript 1, b subscript 1, c subscript 1], [a subscript 2, b subscript 2, c subscript 2], [a subscript 3, b subscript 3, c subscript 3]] multiplied by the 3 by 1 matrix [x, y, z] is the 3 by 1 matrix where row 1 is: a subscript 1 x plus b subscript 1 y plus c subscript 1 z.  Row 2 is:  a subscript 2 x plus b subscript 2 y plus c subscript 2 z.  Row 3 is: a subscript 3 x plus b subscript 3 y plus c subscript 3 z.">
  <br>

  <h6> Second Step: Separate Terms </h6>
  <img src=https://static-assets.codecademy.com/skillpaths/ds-math/matrix_proof_step2.svg alt="In this step, the 3 by 1 matrix is rewritten as the sum of 3 separate 3 by 1 matrices.  The 3 by 1 matrix with row 1 of: a subscript 1 x plus b subscript 1 y plus c subscript 1 z,   Row 2 of:  a subscript 2 x plus b subscript 2 y plus c subscript 2 z, and Row 3 of: a subscript 3 x plus b subscript 3 y plus c subscript 3 z is equal to the sum of 3 separate 3 by 1 matrices.  These 3 matrices are:  [a subscript 1 x, a subscript 2 x, a subscript 3 x] plus [b subscript 1 y, b subscript 2 y, b subscript 3 y] plus [c subscript 1 z, c subscript 2 z, c subscript 3 z].">
  <br>

  <h6> Third Step: Take out the Coefficients </h6>
  <img src=https://static-assets.codecademy.com/skillpaths/ds-math/matrix_proof_step3.svg
 alt="In this step, the coefficients x, y, and z are taken out of the three matrices that are the result of step 2.  The three initial 3 by 1 matrices are [a subscript 1 x, a subscript 2 x, a subscript 3 x] plus [b subscript 1 y, b subscript 2 y, b subscript 3 y] plus [c subscript 1 z, c subscript 2 z, c subscript 3 z].  These are rewritten as: the coefficient x times the matrix [a subscript 1, a subscript 2, a subscript 3], plus the coefficient y times the matrix [b subscript 1, b subscript 2, b subscript 3], plus the coefficient z times the matrix [c subscript 1, c subscript 2, c subscript 3].">
  <br>

  <h6>Therefore, we can say:</h6>
  <img src=https://static-assets.codecademy.com/skillpaths/ds-math/matrix_proof_step4.svg alt="The product of the 3 by 3 matrix [[a subscript 1, b subscript 1, c subscript 1], [a subscript 2, b subscript 2, c subscript 2], [a subscript 3, b subscript 3, c subscript 3]] multiplied by the 3 by 1 matrix [x, y, z] is the coefficient x times the matrix [a subscript 1, a subscript 2, a subscript 3], plus the coefficient y times the matrix [b subscript 1, b subscript 2, b subscript 3], plus the coefficient z times the matrix [c subscript 1, c subscript 2, c subscript 3].">
</details>
<br>

We can also write *Ax = b* in the following form:
```tex
Ax = b
\rightarrow
\begin{bmatrix}
A  \bigm|  b
\end{bmatrix}
```

This means we can write our *Ax = b* equation above as:

```tex
\begin{bmatrix}
a_1 & b_1 & c_1 \\
a_2 & b_2 & c_2 \\
a_3 & b_3 & c_3 \\
\end{bmatrix}
\begin{bmatrix}
x \\
y \\
z \\
\end{bmatrix}
=
\begin{bmatrix}
d_1 \\
d_2 \\
d_3 \\
\end{bmatrix}

\rightarrow

\begin{bmatrix}
    a_{1} & b_{1} & c_{1} \bigm| d_{1} \\
    a_{2} & b_{2} & c_{2} \bigm| d_{2} \\
    a_{3} & b_{3} & c_{3} \bigm| d_{3} \\ 
\end{bmatrix}
```

This is what is known as an *augmented matrix*, and we will use these in the following exercises to solve for linear systems!

Linear Systems in Matrix Form

Like with vectors, there are fundamental operations we can perform on matrices that enable the linear transformations needed for linear algebra. We can again both multiply entire matrices by a scalar value, as well as add or subtract matrices with equal shapes.

```tex
2 \begin{bmatrix}
a_{1} & b_{1} \\
a_{2} & b_{2}

\end{bmatrix}
+
3 \begin{bmatrix}
c_{1} & d_{1} \\
c_{2} & d_{2}

\end{bmatrix}
=
\begin{bmatrix}
2a_{1} + 3c_{1} & 2b_{1} + 3d_{1} \\
2a_{2} + 3c_{2} & 2b_{2} + 3d_{2}

\end{bmatrix}
```

A new and important operation we can now perform is matrix multiplication. Matrix multiplication works by computing the dot product between each row of the first matrix and each column of the second matrix. For example, in computing *AB = C*, element (1,2) of the matrix product *C* will be the dot product of row 1 of matrix *A* and column 2 of matrix *B*. The animation to the right walks through the multiplication of two matrices showing the dot product between each row and column.

An important rule about matrix multiplication is that the shapes of the two matrices *AB* must be such that the number of columns in *A* is equal to the number of rows in *B*. With this condition satisfied, the resulting matrix product will be the shape of *rows<sub>A</sub> x columns<sub>B</sub>*. For example, in the animation to the right, the product of a 2x3 matrix and a 3x2 matrix ends up being a 2x2 matrix.

Based on how we compute the product matrix with dot products and the importance of having correctly shaped matrices, we can see that matrix multiplication is not commutative, *AB <span>&#8800;</span> BA*. However, we can also see that matrix multiplication is associative, *A(BC) = (AB)C*. 

Matrix Operations

In this lesson, we learned about the fundamental building blocks of linear algebra: vectors and matrices, their important operations, and how to use them to solve important problems.

We discussed that vectors are quantities that include both direction and magnitude, and can hold two or more elements of data. These vectors can be multiplied by scalars and be added together with other vectors of the same dimension. Additionally, we found that the dot product, the amount that one vector goes into the other, can be calculated as the sum of the product of corresponding elements.

We then transitioned to matrices, which are quantities that hold rows and columns of data. Matrices can be constructed by combining vectors as the columns of a matrix. Matrices also have similar operations to vectors in terms of scalar multiplication and the addition of same-sized matrices. We were also introduced to matrix multiplication, where the product of two matrices is determined by taking the dot products of the rows and columns of the two matrices being multiplied.

Finally, we learned how to use both vectors and matrices to efficiently solve systems of linear equations. A traditional set of *n* equations for *n* unknown variables can be represented in the form *Ax = b*. Solving for *x* in this form is done by using Gauss-Jordan elimination. We additionally use Gauss-Jordan elimination to find the potential inverse of square matrices.

Review

The *inverse of a matrix*, *A<sup>-1</sup>*, is one where the following equation is true:
```tex
AA^{-1} = A^{-1}A = I
```
This means that the product of a matrix and its inverse (in either order) is equal to the identity matrix.

The inverse matrix is useful in many contexts, one of which includes solving equations with matrices. Imagine we have the following equation:
```tex 
xA = BC
``` 

If we are trying to solve for *x*, we need to find a way to isolate that variable. Since we cannot divide matrices, we can multiply both right sides of the equation by the inverse of *A*, resulting in the following:

```tex
 xAA^{-1} = BCA^{-1} \rightarrow x = BCA^{-1}
```

An important consideration to keep in mind is that not all matrices have inverses. Those matrices that do not have an inverse are referred to as *singular matrices*.

To find the inverse matrix, we can again use Gauss-Jordan elimination. Knowing that *AA<sup>-1</sup> = I*, we can create the augmented matrix [ *A* | *I* ], where we attempt to perform row operations such that [ *A* | *I* ] -> [ *I* | *A<sup>-1</sup>* ]. 

One method to find the necessary row operations to convert *A* -> *I* is to start by creating an upper triangular matrix first, and then work on converting the elements above the diagonal afterward (starting from the bottom right side of the matrix). If the matrix is invertible, this should lead to a matrix with elements equal to 0 except along the diagonal. Each row can then be multiplied by a scalar that leads to the diagonal elements equaling 1 to create the identity matrix. 


