3.1 Testing

3.1 Testing#

Testing is the act of evaluating software to see if it meets specified requirements and functions as expected. There are different types of tests that can be performed at different stages of the software development process. These tests can be classified into two main categories: functional and non-functional tests.

Functional testing assesses the core functionalities of a software system to verify that it behaves as expected according to specified requirements. It focuses on validating inputs, outputs, and the interactions between different components to ensure that the system performs its intended tasks correctly.

On the other hand, non-functional testing evaluates aspects of the system beyond its primary functions, such as performance, usability, reliability, and security. Non-functional testing aims to assess the quality attributes of the software, ensuring that it meets standards for efficiency, user experience, resilience, and protection against threats.

While functional testing confirms what the system does, non-functional testing explores how well it does it under various conditions, providing a comprehensive assessment of the software’s overall quality and suitability for its intended purpose.

flowchart TD A[Software Testing] --> B(Functional) A --> C(Non Functional) B --> D[Unit\nTesting] B --> E[Integration\nTesting] B --> F[System\nTesting] B --> G[Acceptation\nTesting] C --> H[Security\nTesting] C --> I[Performance\nTesting] C --> J[Usability\nTesting] C --> K[Compatibility\nTesting]

If we focus on functional testing, we can classify tests based on the level at which they are performed. Unit testing focuses on verifying the behavior of individual components or modules in isolation, while integration testing evaluates the interactions between different components to ensure that they work together as expected. System testing assesses the behavior of the entire system as a whole, and acceptance testing validates that the system meets the specified requirements and is ready for deployment.

We can also classify tests based on the level of automation. Automated testing uses software tools and scripts to execute test cases, validate system behavior, and provide rapid feedback. It’s efficient, repeatable, and ideal for regression testing, but requires initial setup and maintenance effort. While manual testing involves human intervention to execute test cases and verify software functionality by interacting directly with the application’s user interface. It’s intuitive but time-consuming and prone to human error.

In this chapter, we will focus on functional testing, and in particular automated unit testing. We will explore how to write and run tests using doctest and the pytest framework and how to integrate testing with Visual Studio Code.

Tests as documentation#

Although testing is crucial for verifying that the code behaves as intended and meets the specified requirements. It can also contribute to understanding the inner workings of the code. Through writing tests, developers often gain insights into the behavior of their code, its dependencies, and potential edge cases. Test code can serve as documentation, providing examples of how different parts of the system should behave under various conditions.

Testing inherently involves defining expected outcomes based on requirements and specifications. In this sense, writing tests requires a clear understanding of how the code should behave under different scenarios. By defining test cases that cover different aspects of the code’s functionality, developers articulate their understanding of the expected behavior, ensuring alignment between the code’s implementation and the desired outcomes.

Documentation and Testing with Doctest#

To start as lightweigth as possbile, we will first get familiar with doctest. Doctest is a module that comes with Python and allows you to write tests in the docstrings of your code.

Imagine that you are working on a Python file named addition.py and you developed the following add function that adds two numbers and returns the result:

def add(a, b):
    return a + b

To ensure that everyone understands what the function is supposed to do, you went ahead and added a docstring to the function:

def add(a, b):
    """Compute the sum of two numbers."""
    return a + b

To ensure that the function works as expected and since you know that a usage example always makes the documentation much more clear, you decide to add a test in the docstring.

def add(a, b):
    """Compute the sum of two numbers.

    Usage example:
    >>> add(4.0, 2.0)
    6.0
    """
    return a + b

The added doctest has two parts:

The first part is the test itself, it starts with >>> and contains the function call.
The second part is the expected output, it is the expected result of the function call.

Now to run the tests, you can use the following command in the terminal:

python -m doctest -v addition.py

Trying:
    add(4.0, 2.0)
Expecting:
    6.0
ok
1 items had no tests:
    addition
1 items passed all tests:
   1 tests in addition.add
1 tests in 2 items.
1 passed and 0 failed.
Test passed.

What to notice

Doctest automatillcally detected the test in the docstring and ran it.
It shows the expected output and if the test passed.

We can add as many tests as we want to the docstring. For example, we can add a test for the case when the two numbers are integers:

def add(a, b):
    """Compute the sum of two numbers.

    Usage examples:
    >>> add(4.0, 2.0)
    6.0
    >>> add(1, 3)
    4
    """
    return a + b

python -m doctest -v addition.py

Trying:
    add(4.0, 2.0)
Expecting:
    6.0
ok
Trying:
    add(1, 3)
Expecting:
    4
ok
1 items had no tests:
    addition
1 items passed all tests:
   2 tests in addition.add
2 tests in 2 items.
2 passed and 0 failed.
Test passed.

What to notice

We now see the two tests and their results.

As we add more tests, the docstring becomes longer and harder to read. We can remove the -v flag to avoid the verbose output.

python -m doctest addition.py

What to notice

If all tests pass we see no output.

But what if a test fails? Let’s see what happens if we change the second test to expect 5 instead of 4:

def add(a, b):
    """Compute the sum of two numbers.

    Usage examples:
    >>> add(4.0, 2.0)
    6.0
    >>> add(1, 3)
    5
    """
    return a + b

What to notice

We see the doctest telling us that the test failed.
We see the expected output and the actual output.

Testing with Pytest#

As our project grows, we will usually need a more powerful tool to write and run tests. Pytest is a popular testing framework that makes writing and running tests easier. Pytest is a third-party package, this means that we need to install it before we can use it.

We can use pip to install pytest:

pip install pytest

To explore how pytest works, let’s get back to our addition.py file and our original implementation of the add function without the doctests.

def add(a, b):
    """Compute the sum of two numbers.
    """
    return a + b

To write tests with pytest, we use the assert statement to check if the function returns the expected result. Although we can write the tests in the same file as we did when we explored the doctests, this time we will write the tests in a separate file named test_addition.py.

def test_add():
    from addition import add
    sum_result = add(4, 2)
    assert sum_result == 6

To run the tests, we use the pytest command in the terminal:

pytest test_addition.py

============================= test session starts ==============================
platform linux -- Python 3.7.4, pytest-7.4.4, pluggy-1.2.0
rootdir: /home/callaram/tds/home/ch3
collected 1 item                                                               

test_addition.py .                                                       [100%]

============================== 1 passed in 0.01s ===============================

What to notice

We see that in this case pytest automatically detected the test in the file and ran it.
The green dot at the side of the test name indicates that the test passed.

Arrange-Act-Assert#

In general, unit tests follow the Arrange-Act-Assert pattern. This means that we:

Arrange: prepare the environment for the test.
Act: perform the action that we want to test.
Assert: check if the result is as expected.

In the test we just wrote:

Arrange: we import the add function from the addition module.
Act: we call the add function with the arguments 4 and 2.
Assert: we check if the result is 6.

Integration with Visual Studio Code#

Pytest integrates very well with Visual Studio Code. Visual Studio Code will automatically look for files that start with the word test detect the tests in the file and show a Run Test button. If we click on it, we will see the test result in the Test Explorer tab.

3.1 Testing

Contents

3.1 Testing#

Tests as documentation#

Documentation and Testing with Doctest#

Testing with Pytest#

Arrange-Act-Assert#

Integration with Visual Studio Code#