Explore writing unit tests with Python's unittest framework. 

Unit Testing

In the last exercise, we saw how to check for equality between two values in the unittest framework using the `.assertEqual` method of the `TestCase` class. The framework relies on built-in assert methods instead of `assert` statements to track results without actually raising any exceptions. Specific assert methods take arguments instead of a condition, and like assert statements, they can take an optional message argument. 

Let’s go over three commonly used assert methods for testing equality and membership, their general syntax, and their `assert` statement equivalents.

- [assertEqual](https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertEqual): The `assertEqual()` method takes two values as arguments and checks that they are equal. If they are not, the test fails.

  ```
   self.assertEqual(value1, value2)
  ```
- [assertIn](https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertIn): The `assertIn()` method takes two arguments. It checks that the first argument is found in the second argument, which should be a container. If it is not found in the container, the test fails.
  
  ```
   self.assertIn(value, container)
  ```
- [assertTrue](https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertTrue): The `assertTrue()` method takes a single argument and checks that the argument evaluates to `True`. If it does not evaluate to `True`, the test fails.
  
  ```
   self.assertTrue(value)
  ```

The equivalent `assert` statements would be the following: 

 Method | Equivalent
 --- | ---
 `self.assertEqual(2, 5)` | `assert 2 == 5`
 `self.assertIn(5, [1, 2, 3])` | `assert 5 in [1, 2, 3]`
 `self.assertTrue(0)` | `assert bool(0) is True`

<br>

The full list for equality and membership can be seen in the [Python documentation](https://docs.python.org/3/library/unittest.html#unittest.TestCase.debug). Let’s put these methods into practice!

Assert Methods I: Equality and Membership

There is another group of assert methods related to exceptions and warnings. Note that while we haven’t covered [warnings](https://docs.python.org/3/library/warnings.html) in detail yet, they are a type of exception. Let’s go over two of these methods and their general syntax. 

- [assertRaises](https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertRaises): The `assertRaises()` method takes an exception type as its first argument, a function reference as its second, and an arbitrary number of arguments as the rest. 

  It calls the function and checks if an exception is raised as a result. The test passes if an exception is raised, is an error if another exception is raised, or fails if no exception is raised. This method can be used with custom exceptions as well!
  ```
  self.assertRaises(specificException, function, functionArguments...)
  ```

- [assertWarns](https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertWarns): The `assertWarns()` method takes a warning type as its first argument, a function reference as its second, and an arbitrary number of arguments for the rest. 

  It calls the function and checks that the warning occurs. The test passes if a warning is triggered and fails if it isn’t.
  ```
   self.assertWarns(specificWarningException, function, functionArguments...)
  ```

There are no particular concise ways to replicate these tests using the `assert` keyword so it is recommended to use these methods instead when possible!

The full list of exception and warning methods can be seen in the [Python documentation](https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertNotIsInstance). Let’s put these methods into practice!

Assert Methods III: Exception and Warning Methods

Awesome job! We’ve covered a lot of material related to unit testing in Python. We learned:
* The difference between manual and automated testing.
* What unit tests are.
* How to write simple tests with the `assert` keyword.
* How to create and run test cases with the `unittest` framework.
* Best practices for test fixtures, test parameterization, skipped tests and expected failures.

The world of software testing is vast and can take time to master, but the basic principles of unit testing will almost always be applicable to any language we work with. Incorporating testing into our software is the best way to prevent unexpected bugs from occurring. The sooner we write tests, the faster we can catch and fix bugs and make our software better!

Review

Sometimes we have a test that we know will fail. This could happen when a feature has a known bug or is designed to fail on purpose. In this case, we wouldn’t want an expected failure to cloud our test results. Rather than simply skipping the test, `unittest` provides a way to mark tests as expected failures. Expected failures are counted as passed in our test results. If the test passes when we expected it to fail, then it is marked as failed in test results. 

To setup a test to have an expected failure, we can use the [`expectedFailure` decorator](https://docs.python.org/3/library/unittest.html#unittest.expectedFailure). Let’s consider the following example:

```py
class FeatureTests(unittest.TestCase):

    @unittest.expectedFailure
    def test_broken_feature(self):
        raise Exception("This test is going to fail")
```

The `expectedFailure` decorator takes no arguments. The test in the example will always fail because an exception was raised during test execution. When run, we get the following output:

```
x
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK (expected failures=1)
```

The test failure did not cause any failures in our test results because it was marked as expected. Let’s do the same for some of our Small World Air tests. 

Expected Failures

One of the most important principles of testing is that tests need to occur in a known state. If the conditions in which a test runs are not controlled, then our results could contain false negatives (invalid failed results) or false positives (invalid passed results). 

This is where _test fixtures_ come in. A test fixture is a mechanism for ensuring proper test setup (putting tests into a known state) and test teardown (restoring the state prior to the test running). Test fixtures guarantee that our tests are running in predictable conditions, and thus the results are reliable. 

Let’s say we are testing a Bluetooth device. The device’s Bluetooth module can sometimes fail. When this happens, the device needs to be power cycled (shut off and then on) to restore Bluetooth functionality. We would not want tests to run if the device was already in a failed state because these results would not be valid. Furthermore, if our tests cause the Bluetooth module to fail, we want to restore it to a working state after the tests run. So, we add a test fixture to power cycle the device before and after each test. Here is how we might do it:

```py
def power_cycle_device():
  print('Power cycling bluetooth device...')

class BluetoothDeviceTests(unittest.TestCase):
  def setUp(self):
    power_cycle_device()

  def test_feature_a(self):
    print('Testing Feature A')

  def test_feature_b(self):
    print('Testing Feature B')

  def tearDown(self):
    power_cycle_device()
```

The `unittest` framework automatically identifies setup and teardown methods based on their names. A method named `setUp` runs before each test case in the class. Similarly, a method named `tearDown` gets called after each test case. Now, we can guarantee that our Bluetooth module is in a working state before and after every test. Here is the output when these tests are run:

```bash
Power cycling bluetooth device...
Testing Feature A
Power cycling bluetooth device...
.Power cycling bluetooth device...
Testing Feature B
Power cycling bluetooth device...
.
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK
```

Let’s consider another scenario. Perhaps our tests rely on working Bluetooth, but there is nothing in the tests that would cause the bluetooth to stop working. In this case, it would be inefficient to power cycle the device before and after every test. Let’s refactor the previous example so that setup and teardown only happen once - before and after all tests in the class are run:

```py
def power_cycle_device():
    print('Power cycling bluetooth device...')

class BluetoothDeviceTests(unittest.TestCase):
  @classmethod
  def setUpClass(cls):
    power_cycle_device()

  def test_feature_a(self):
    print('Testing Feature A')

  def test_feature_b(self):
    print('Testing Feature B')

  @classmethod
  def tearDownClass(cls):
    power_cycle_device()
```

We replaced our `setUp` method with the `setUpClass` method and added the [`@classmethod` decorator](https://docs.python.org/3/library/functions.html#classmethod). We changed the argument from `self` to `cls` because this is a class method. Similarly, we replaced the `tearDown` method with the `tearDownClass` class method. Now, we get the following output:

```
Power cycling bluetooth device...
Testing Feature A
Testing Feature B
Power cycling bluetooth device...

----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK
```

In addition to calling functions, we can also use setup methods to instantiate objects and or gather any other data needed. Anything stored in our class will be available throughout our test functions.

It’s generally good practice to create fixtures that run for every test. However, when a fixture has a large cost (i.e. it takes a long time), then it might make more sense to have it run once per test class rather than once per test. Let's practice setting up test fixtures!

Test Fixtures

In previous examples, we created test cases for the `times_ten()` function with various inputs. However, the actual logic of our tests really didn’t change. To decrease repetition, Python provides us a specific toolset for tests with only minor differences. This is known as _test parameterization_. By parameterizing tests, we can leverage the functionality of a single test to get a large amount of coverage of different inputs. 

To accomplish test parameterization, the `unittest` framework provides us with the [`subTest` context manager](https://docs.python.org/3/library/unittest.html#distinguishing-test-iterations-using-subtests). Let’s refactor our previous test class to utilize it and see it in action:

```py
import unittest

# The function we want to test
def times_ten(number):
    return number * 100

# Our test class
class TestTimesTen(unittest.TestCase):
    
    # A test method
    def test_times_ten(self):
        for num in [0, 1000000, -10]:
            with self.subTest():
                expected_result = num * 10
                message = 'Expected times_ten(' + str(num) + ') to return ' + str(expected_result)
                self.assertEqual(times_ten(num), expected_result, message)
```

Here, in our test method `test_times_ten()`, instead of writing individual test cases for each input of `0`, `10`, and `1000000`, we can test a collection of inputs by using a loop followed by a `with` statement and our `subTest` context manager. 

By using `subTest`, each iteration of our loop is treated as an individual test. Python will run the code inside of the context manager on each iteration, and if one fails, it will return the failure as a separate test case failure. 

Just like before, we are using the `assertEqual()` method to check the expected result, and we are expecting (due to an error in `times_ten()`) that the cases of using an input of `-10` and `1000000` will fail. 

Here is the new output:
```error
======================================================================
FAIL: test_times_ten (__main__.TestTimesTen) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "scratch.py", line 12, in test_times_ten
    self.assertEqual(times_ten(num), expected_result, message)
AssertionError: 100000000 != 10000000 : Expected times_ten(1000000) to return 10000000

======================================================================
FAIL: test_times_ten (__main__.TestTimesTen) (<subtest>)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "scratch.py", line 12, in test_times_ten
    self.assertEqual(times_ten(num), expected_result, message)
AssertionError: -1000 != -100 : Expected times_ten(-10) to return -100

----------------------------------------------------------------------
Ran 1 test in 0.000s

FAILED (failures=2)
```

If we want to expand our test coverage, we can simply modify the list that our loop iterates over. We can test a range of thousands of inputs simply by using the context manager setup to achieve test parameterization. 

Optionally, we can give our subtests better readability by making a small change in our code for the first argument of `self.subTest()`. The below code has most of our script omitted for brevity but uses the same script we executed above:  
```py
# ... more code above..

for num in [0, 1000000, -10]:
  with self.subTest(num):

# ... more code below ....
```

This makes our test clearer, because our test error message goes from: 
```error
FAIL: test_times_ten (__main__.TestTimesTen) (<subtest>)
```

to:
```error
FAIL: test_times_ten (__main__.TestTimesTen) [1000000]
```

When working with large amounts of test inputs, it is much easier to distinguish which case failed. We can actually use any message we want as the first argument, but using the tested case is usually the best way to increase readability for ourselves and other developers. 

By using test parameterization, we made our codebase much cleaner and more maintainable. Let’s get some practice by refactoring some of our previous tests!

Parameterizing Tests

When working with Python, or any programming language, there is a lot that can go wrong with our code. There are syntax errors and exceptions, but there are also mistakes in the program logic which cause it to behave in unexpected ways. 

For these reasons, testing is crucial to creating quality software. The goal of testing isn’t just to find bugs but to find them quickly. Leaving bugs unfound and unresolved can lead to massive consequences in the real world. Take a look at some of the [most infamous bugs that have ever occurred in software](https://en.wikipedia.org/wiki/List_of_software_bugs). 

Don’t worry though - by following some common practices and using the tools built into Python, we can start creating quality tests in no time. To dive in, first, let's talk about the different types of testing styles that exist.

The world of testing can generally be divided into two categories: 

- Manual Testing: 
  - With manual testing, a physical person interacts with software much as a user would. In fact, we have been manually testing our code any time we run it and observe the results!
- Automated Testing: 
  - With automated testing, tests are performed with code. Generally, automated testing is faster and less prone to human error. 

In this lesson, we'll be diving into the world of testing. For most of the exercises, let's imagine we've been hired to create tests for a new airline company called Small World Air. Before we start writing automated tests, let’s do some manual testing and see what kind of shape their software is in. 


Introduction to Testing

There are some problems with the approach to our previous unit tests that would make them difficult to maintain. First, we had to call each function specifically when a new test was created. We also didn’t have any way of grouping tests, which is necessary when the number of tests increases. Perhaps most importantly, if one test failed, the `AssertionError` would prevent any remaining tests from running!

Luckily, Python provides a framework that solves these problems and provides many other tools for writing unit tests. This framework lives in the `unittest` module which is included in the standard library. It can be imported like so: 

```py
import unittest

#The rest of our program….
```

The `unittest` module provides us with a _test runner_. A test runner is a component that collects and executes tests and then provides results to the user. The framework also provides many other tools for test grouping, setup, teardown, skipping, and other features that we’ll soon learn about. 

First, let’s refactor our tests for the `times_ten` function to use the `unittest` framework. There are several things we need to do:

First, we must create a class which inherits from `unittest.TestCase`, as follows:
```py
import unittest 

class TestTimesTen(unittest.TestCase):
    pass
```

This class will serve as the main storage of all our unit testing functions. Once we have the class, we need to change our test functions so that they are methods of the class. The `unittest` module requires that test functions begin with the word `'test'`, so our existing names work well:
```py
import unittest

class TestTimesTen(unittest.TestCase):
    def test_multiply_ten_by_zero(self):
        pass

    def test_multiply_ten_by_one_million(self):
        pass

    def test_multiply_ten_by_negative_number(self):
        pass
```

Lastly, we need to change our `assert` statements to use the `assertEqual` method of `unittest.TestCase`. The framework requires that we use special methods instead of standard `assert` statements. Don't worry we’ll cover these methods in the remainder of this lesson, for now, simply get used to the syntax. Here is what our class looks after the change:
```py
import unittest

class TestTimesTen(unittest.TestCase):
    def test_multiply_ten_by_zero(self):
        self.assertEqual(times_ten(0), 0, 'Expected times_ten(0) to return 0')

    def test_multiply_ten_by_one_million(self):
        self.assertEqual(times_ten(1000000), 10000000, 'Expected times_ten(1000000) to return 10000000')

    def test_multiply_ten_by_negative_number(self):
        self.assertEqual(times_ten(-10), -100, 'Expected add_times_ten(-10) to return -100')
```

That's it! Now we can run our tests by calling `unittest.main()`. The `unittest` framework will work its magic to detect any tests in the existing module, run them, and provide us results. Our final code would look like this:
```py
# Importing unittest framework
import unittest

# Function that gets tested
def times_ten(number):
    return number * 100

# Test class
class TestTimesTen(unittest.TestCase):
    def test_multiply_ten_by_zero(self):
        self.assertEqual(times_ten(0), 0, 'Expected times_ten(0) to return 0')

    def test_multiply_ten_by_one_million(self):
        self.assertEqual(times_ten(1000000), 10000000, 'Expected times_ten(1000000) to return 10000000')

    def test_multiply_ten_by_negative_number(self):
        self.assertEqual(times_ten(-10), -100, 'Expected add_times_ten(-10) to return -100')

# Run the tests
unittest.main()
```

When we run this code, we would see the following output:

```error
FF.
======================================================================
FAIL: test_multiply_ten_by_negative_number (__main__.TestTimesTen)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "scratch.py", line 16, in test_multiply_ten_by_negative_number
    self.assertEqual(times_ten(-10), -100, 'Expected add_times_ten(-10) to return -100')
AssertionError: -1000 != -100 : Expected add_times_ten(-10) to return -100

======================================================================
FAIL: test_multiply_ten_by_one_million (__main__.TestTimesTen)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "scratch.py", line 13, in test_multiply_ten_by_one_million
    self.assertEqual(times_ten(1000000), 10000000, 'Expected times_ten(1000000) to return 10000000')
AssertionError: 100000000 != 10000000 : Expected times_ten(1000000) to return 10000000

----------------------------------------------------------------------
Ran 3 tests in 0.001s

FAILED (failures=2)
```

In the test output, we can see that two of the tests failed (`test_multiply_ten_by_one_million` and `test_multiply_ten_by_negative_number`).

Let’s get some practice with `unittest` by refactoring our previous test cases for Small World Air and then we will dive into learning all the useful testing methods we can work with! 


Python's unittest Framework

Sometimes we have tests that should only run in a particular context. For example, we might have a group of tests that only runs on the Windows operating system but not Linux or macOS. For these situations, it’s helpful to be able to skip tests. 

The `unittest` framework provides two different ways to skip tests: 

1. The `@unittest` skip decorator 
2. The `skipTest()` method

First, let's examine the skip decorator option. There are two decorator options to accomplish the goal of skipping a test. Let's observe both of them in the example below: 

```py
import sys

class LinuxTests(unittest.TestCase):

    @unittest.skipUnless(sys.platform.startswith("linux"), "This test only runs on Linux")
    def test_linux_feature(self):
        print("This test should only run on Linux")

    @unittest.skipIf(not sys.platform.startswith("linux"), "This test only runs on Linux")
    def test_other_linux_feature(self):
        print("This test should only run on Linux")
```

Let's break down both skip decorator options:

- The `skipUnless` option skips the test if the condition evaluates to `False`. 

- The `skipIf` option skips the test if the condition evaluates to `True`. 

Both share common requirements. Firstly, both of these skip decorators are prefaced with `@unittest` to denote the decorator pattern. They both take a condition as a first argument, followed by a string message as the second. In this example, both decorators achieve the same goal: skipping the test if the operating system is not Linux. 

If we ran the tests on a macOS system, we would get the following output:
```
ss
----------------------------------------------------------------------
Ran 2 tests in 0.000s

OK (skipped=2)
```

The second way to skip tests is to call the `skipTest` method of the `TestCase` class, as in this example:

```py
import sys

class LinuxTests(unittest.TestCase):

    def test_linux_feature(self):
        if not sys.platform.startswith("linux"):
            self.skipTest("Test only runs on Linux")
```

Here we call `self.skipTest()` from within the test function itself. It takes a single string message as its argument and always causes the test to be skipped when called. When run on macOS we get the following output:

```
s
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK (skipped=1)
```

Skip decorators are slightly more convenient and make it easy to see under what conditions the test is skipped. When the conditions for skipping a test are too complicated to pass into a skip decorator, the `skipTest` method is the recommended alternative. 

Let's practice skipping tests in our Small World Air application!

Skipping tests

Often we need to test conditions related to numbers. The `unittest` module provides a handful of assert methods to achieve this. Let’s take a look at two common assert methods related to quantitative comparisons, their general syntax, as well as their `assert` statement equivalents.

- [assertLess](https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertLess): The `assertLess()` method takes two arguments and checks that the first argument is less than the second one. If it is not, the test will fail.
  
  ```py
   self.assertLess(value1, value2)
  ```
- [assertAlmostEqual](https://docs.python.org/3/library/unittest.html#unittest.TestCase.assertAlmostEqual): The `assertAlmostEqual()` method takes two arguments and checks that their difference, when rounded to 7 decimal places, is 0. In other words, if they are almost equal. If the values are not close enough to equality, the test will fail.
  
  ```py
   self.assertAlmostEqual(value1, value2)
  ```

The equivalent `assert` statements would be the following: 

 Method | Equivalent
 --- | ---
 `self.assertLess(2, 5)` | `assert 2 < 5`
 `self.assertAlmostEqual(.22, .225)` | `assert round(.22 - .225, 7) == 0`

<br>

The full list of quantitative methods can be seen in the [Python documentation](https://docs.python.org/3/library/unittest.html#unittest.TestCase.output). Let’s put these methods into practice!

Assert Methods II: Quantitative Methods

Assertion statements are a good start to ensuring our programs are being tested, but they don't necessarily tell us ***what*** we should test. Generally, we can start by testing the smallest unit of a program. 

For example, in the real world, if we were testing the functionality of a door, we could test a multitude of units. The handle could be an example of a single unit that we must check to make sure a door functions, followed by the hinges and maybe even the lock. 

In programming, these types of individual tests are called _unit tests_. Like our door handle, we can test a single unit of a program, such as a function, loop, or variable. A unit test validates a single behavior and will make sure all of the units of a program are functioning properly.

Let's say we wanted to test a single function (a single unit). To test a single function, we might create several _test cases_. A test case validates that a specific set of inputs produces an expected output for the unit we are trying to test. Let’s examine a test case for our `times_ten()` function from the previous exercise:
```py
# The unit we want to test
def times_ten(number):
    return number * 100

# A unit test function with a single test case
def test_multiply_ten_by_zero():
    assert times_ten(0) == 0, 'Expected times_ten(0) to return 0'
```

Great, now we have a simple test case that validates that `times_ten()` is behaving as expected for a valid input of `0`! We can improve our testing coverage of this function by adding some more test cases with different inputs. A common approach is to create test cases for specific edge case inputs as well as reasonable ones. Here is an example of testing two extreme inputs: 

```py
def test_multiply_ten_by_one_million():
    assert times_ten(1000000) == 10000000, 'Expected times_ten(1000000) to return 10000000

def test_multiply_ten_by_negative_number():
    assert times_ten(-10) == -100, 'Expected times_ten(-10) to return -100'
```

Now we have several test cases for a wide variety of inputs: a large number, a negative number, and zero. We can create as many test cases as we see fit for a single unit, and we should try to test all the unique types of inputs our unit will work with. 

Now, let’s create a variety of unit tests for another feature of Small World Air. 

While the previous example was simple to test, most code will be much more complex. It would be very tedious to have to perform these tests manually. Our time would be better spent writing automated tests.

Luckily, Python provides an easy way to perform simple tests in our code - the `assert` statement. An `assert` statement can be used to test that a condition is met. If the condition evaluates to `False`, an `AssertionError` is raised with an optional error message.

The general syntax looks like this:
```
assert <condition>, 'Message if condition is not met'
```

Consider the following example that demonstrates the `assert` statement paired with a function called `times_ten`. Note there is a bug in the function for demonstration purposes. 

```py
def times_ten(number):
    return number * 100

result = times_ten(20)
assert result == 200, 'Expected times_ten(20) to return 200, instead got ' + str(result)
```

Here, we want to test if our `times_ten()` function works as intended. We use the `assert` statement to evaluate the expression `result == 200` since we expect that our function would return `200` given an input of `20`. Since this is not the case, this expression evaluates to `False` (there is a bug in `times_ten` - it actually multiplies by `100`!), we get the following exception:

```error
AssertionError: Expected times_ten(20) to return 200, instead got 2000
```

An `assert` statement is a quick and powerful way to verify that a program is in the correct state. They can be used to catch mistakes early and make sure we avoid any catastrophes. Let's practice using `assert` to get a feel for automated testing!