Automated software testing with Python

Software testing is the process in which a developer ensures that the actual output of the software matches with the desired output by providing some test inputs to the software. Software testing is an important step because if performed properly, it can help the developer to find bugs in the software in very less amount of time. Software testing can be divided into two classes, Manual testing and Automated testing. Automated testing is the execution of your tests using a script instead of a human. In this article, we’ll discuss some of the methods of automated software testing with Python. Let’s write a simple application over which we will perform all the tests. 

Note: For more information about the function __repr__(), refer this article. Now that we have our software ready, let’s have a look at the directory structure of our project folder and after that, we’ll start testing our software.

---Software_Testing
   |--- __init__.py (to initialize the directory as python package)
   |--- app.py (our software)
   |--- tests (folder to keep all test files)
           |--- __init__.py

The ‘unittest’ module

One of the major problems with manual testing is that it requires time and effort. In manual testing, we test the application over some input, if it fails, either we note it down or we debug the application for that particular test input, and then we repeat the process. With unittest, all the test inputs can be provided at once and then you can test your application. In the end, you get a detailed report with all the failed test cases clearly specified, if any. The unittest module has both a built-in testing framework and a test runner. A testing framework is a set of rules which must be followed while writing test cases, while a test runner is a tool which executes these tests with a bunch of settings, and collects the results. Installation: unittest is available at PyPI and can be installed with the following command –

pip install unittest

Use: We write the tests in a Python module (.py). To run our tests, we simply execute the test module using any IDE or terminal. Now, let’s write some tests for our small software discussed above using the unittest module.

1. Create a file named tests.py in the folder named “tests”.
2. In tests.py import unittest.
3. Create a class named TestClass which inherits from the class unittest.TestCase. Rule 1: All the tests are written as the methods of a class, which must inherit from the class unittest.TestCase.
4. Create a test method as shown below. Rule 2: Name of each and every test method should start with “test” otherwise it’ll be skipped by the test runner. 

1. Rule 3: We use special assertEqual() statements instead of the built-in assert statements available in Python. The first argument of assertEqual() is the actual output, the second argument is the desired output and the third argument is the error message which would be displayed in case the two values differ from each other (test fails).
2. To run the tests we just defined, we need to call the method unittest.main(), add the following lines in the “tests.py” module. 

1. Because of these lines, as soon as you run execute the script “test.py”, the function unittest.main() would be called and all the tests will be executed.

Finally the “tests.py” module should resemble the code given below. 

Having written our test cases let us now test our application for any bugs. To test your application you simply need to execute the test file “tests.py” using the command prompt or any IDE of your choice. The output should be something like this.

.
----------------------------------------------------------------------
Ran 1 test in 0.000s

OK

In the first line, a .(dot) represents a successful test while an ‘F’ would represent a failed test case. The OK message, in the end, tells us that all the tests were passed successfully. Let’s add a few more tests in “tests.py” and retest our application. 

.F.F
======================================================================
FAIL: test_area_negative (__main__.TestSum)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests_unittest.py", line 11, in test_area_negative
    self.assertEqual(sq.area(), -1, f'Area is shown {sq.area()} rather than -1 for negative side length')
AssertionError: 9 != -1 : Area is shown 9 rather than -1 for negative side length

======================================================================
FAIL: test_perimeter_negative (__main__.TestSum)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "tests_unittest.py", line 19, in test_perimeter_negative
    self.assertEqual(sq.perimeter(), -1, f'Perimeter is {sq.perimeter()} rather than -1 for negative side length')
AssertionError: -24 != -1 : Perimeter is -24 rather than -1 for negative side length

----------------------------------------------------------------------
Ran 4 tests in 0.001s

FAILED (failures=2)

A few things to note in the above test report are –

• The first line represents that test 1 and test 3 executed successfully while test 2 and test 4 failed
• Each failed test case is described in the report, the first line of the description contains the name of the failed test case and the last line contains the error message we defined for that test case.
• At the end of the report you can see the number of failed tests, if no test fails the report will end with OK

Note: For further knowledge you can read the complete documentation of unittest.

The “nose2” module

The purpose of nose2 is to extend unittest to make testing easier. nose2 is compatible with tests written using the unittest testing framework and can be used as a replacement of the unittest test runner. Installation: nose2 can be installed from PyPI using the command,

pip install nose2

Use: nose2 does not have any testing framework and is merely a test runner which is compatible with the unittest testing framework. Therefore we’ll the run same tests we wrote above (for unittest) using nose2. To run the tests we use the following command in the project source directory (“Software_Testing” in our case),

nose2

In nose2 terminology all the python modules (.py) with name starting from “test” (i.e. test_file.py, test_1.py) are considered as test files. On execution, nose2 will look for all test files in all the sub-directories which lie under one or more of the following categories,

• which are python packages (contain “__init__.py”).
• whose name starts with “test” after being lowercased, i.e. TestFiles, tests.
• which are named either “src” or “lib”.

nose2 first loads all the test files present in the project and then the tests are executed. Thus, with nose2 we get the freedom to split our tests among various test files in different folders and execute them at once, which is very useful when dealing with large number of tests. Let’s now learn about different customisation options provided by nose2 which can help us during the testing process.

1. Changing the search directory – If we want to change the directory in which nose2 searches for test files, we can do that using the command line arguments -s or –start-dir as,

nose2 -s DIR_ADD DIR_NAME

1. here, DIR_NAME is the directory in which we want to search for the test files and, DIR_ADD is the address of the parent directory of DIR_NAME relative to the project source directory (i.e. use “./” if test directory is in the project source directory itself). This is extremely useful when you want to test only one feature of your application at a time.
2. Running specific test cases – Using nose2 we can also run a specific test at a time by using the command line arguments -s and –start-dir as,

nose2 -s DIR_ADD DIR_NAME.TEST_FILE.TEST_CLASS.TEST_NAME

• TEST_NAME: name of the test method.
• TEST_CLASS: class in which the test method is defined.
• TEST_FILE: name of the test file in which the test case is defined i.e. test.py.
• DIR_NAME: directory in which the test file exists.
• DIR_ADD: address of the parent directory of DIR_NAME relative to the project source.

1. Running tests in a single module – nose2 can also be used like unittest by calling the function nose2.main() just like we called unittest.main() in previous examples.

The “pytest” module

pytest is the most popular testing framework for python. Using pytest you can test anything from basic python scripts to databases, APIs and UIs. Though pytest is mainly used for API testing, in this article we’ll cover only the basics of pytest. Installation: You can install pytest from PyPI using the command,

pip install pytest

Use: The pytest test runner is called using the following command in project source,

py.test

Unlike nose2, pytest looks for test files in all the locations inside the project directory. Any file with name starting with “test_” or ending with “_test” is considered a test file in the pytest terminology. Let’s create a file “test_file1.py” in the folder “tests” as our test file. Creating test methods: pytest supports the test methods written in the unittest framework, but the pytest framework provides easier syntax to write tests. See the code below to understand the test method syntax of the pytest framework. 

Note: similar to unittest, pytest requires all test names to start with “test”. Unlike unittest, pytest uses the default python assert statements which make it further easier to use. Note that, now the “tests” folder contains two files namely, “tests.py” (written in unittest framework) and “test_file1.py” (written in pytest framework). Now let’s run the pytest test runner.

py.test

You’ll get a similar report as obtained by using unittest.

============================= test session starts ==============================
platform linux -- Python 3.6.7, pytest-4.4.1, py-1.8.0, pluggy-0.9.0
rootdir: /home/manthan/articles/Software_testing_in_Python
collected 6 items                                                              

tests/test_file1.py .F                                                   [ 33%]
tests/test_file2.py .F.F                                                 [100%]

=================================== FAILURES ===================================

The percentages on the right side of the report show the percentage of tests that have been completed at that moment, i.e. 2 out of the 6 test cases were completed at the end of the “test_file1.py”. Here are a few more basic customisations that come with pytest.

1. Running specific test files: To run only a specific test file, use the command,

py.test <filename>

1. Substring matching: Suppose we want to test only the area() method of our Square class, we can do this using substring matching as follows,

py.test -k "area"

1. With this command pytest will execute only those tests which have the string “area” in their names, i.e. “test_file1_area()”, “test_area()” etc.
2. Marking: As a substitute to substring matching, marking is another method using which we can run a specific set of tests. In this method we put a mark on the tests we want to run. Observe the code example given below, 

1. In the above code example test_file1_area() is marked with tag “area”. All the test methods which have been marked with some tag can be executed by using the command,

py.test -m <tag_name>

1. Parallel Processing: If you have a large number of tests then pytest can be customised to run these test methods in parallel. For that you need to install pytest-xdist which can be installed using the command,

pip install pytest-xdist

1. Now you can use the following command to execute your tests faster using multiprocessing,

py.test -n 4

1. With this command pytest assigns 4 workers to perform the tests in parallel, you can change this number as per your needs. If your tests are thread-safe, you can also use multithreading to speed up the testing process. For that you need to install pytest-parallel (using pip). To run your tests in multithreading use the command,

pytest --workers 4