XClose
Menu

Testing frameworks

Why use testing frameworks?

Frameworks should simplify our lives:

  • Should be easy to add simple test
  • Should be possible to create complex test:
    • Fixtures
    • Setup/Tear down
    • Parameterized tests (same test, mostly same input)
  • Find all our tests in a complicated code-base
  • Run all our tests with a quick command
  • Run only some tests, e.g. test --only "tests about fields"
  • Report failing tests
  • Additional goodies, such as code coverage

Common testing frameworks

py.test framework: usage

py.test is a recommended python testing framework.

We can use its tools in the notebook for on-the-fly tests in the notebook. This, happily, includes the negative-tests example we were looking for a moment ago.

In [1]:
def I_only_accept_positive_numbers(number):
    # Check input
    if number < 0: 
        raise ValueError("Input "+ str(number)+" is negative")

    # Do something
In [2]:
from pytest import raises
In [3]:
with raises(ValueError):
    I_only_accept_positive_numbers(-5)

but the real power comes when we write a test file alongside our code files in our homemade packages:

In [4]:
%%bash
mkdir -p saskatchewan
touch saskatchewan/__init__.py
In [5]:
%%writefile saskatchewan/overlap.py
def overlap(field1, field2):
    left1, bottom1, top1, right1 = field1
    left2, bottom2, top2, right2 = field2
    
    overlap_left=max(left1, left2)
    overlap_bottom=max(bottom1, bottom2)
    overlap_right=min(right1, right2)
    overlap_top=min(top1, top2)
    # Here's our wrong code again
    overlap_height=(overlap_top-overlap_bottom)
    overlap_width=(overlap_right-overlap_left)
    
    return overlap_height*overlap_width
Overwriting saskatchewan/overlap.py
In [6]:
%%writefile saskatchewan/test_overlap.py
from .overlap import overlap

def test_full_overlap():
    assert overlap((1.,1.,4.,4.),(2.,2.,3.,3.)) == 1.0

def test_partial_overlap():
    assert overlap((1,1,4,4),(2,2,3,4.5)) == 2.0
                 
def test_no_overlap():
    assert overlap((1,1,4,4),(4.5,4.5,5,5)) == 0.0
Overwriting saskatchewan/test_overlap.py
In [7]:
%%bash
cd saskatchewan
py.test
============================= test session starts ==============================
platform linux -- Python 3.6.3, pytest-4.1.1, py-1.5.2, pluggy-0.8.1
rootdir: /home/travis/build/alan-turing-institute/rsd-engineeringcourse/ch03tests/saskatchewan, inifile:
plugins: cov-2.6.1
collected 3 items

test_overlap.py ..F                                                      [100%]

=================================== FAILURES ===================================
_______________________________ test_no_overlap ________________________________

    def test_no_overlap():
>       assert overlap((1,1,4,4),(4.5,4.5,5,5)) == 0.0
E       assert 0.25 == 0.0
E        +  where 0.25 = overlap((1, 1, 4, 4), (4.5, 4.5, 5, 5))

test_overlap.py:10: AssertionError
====================== 1 failed, 2 passed in 0.06 seconds ======================
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
<ipython-input-7-40dce32fa8c8> in <module>
----> 1 get_ipython().run_cell_magic('bash', '', 'cd saskatchewan\npy.test\n')

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2321             magic_arg_s = self.var_expand(line, stack_depth)
   2322             with self.builtin_trap:
-> 2323                 result = fn(magic_arg_s, cell)
   2324             return result
   2325 

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/magics/script.py in named_script_magic(line, cell)
    140             else:
    141                 line = script
--> 142             return self.shebang(line, cell)
    143 
    144         # write a basic docstring:

<decorator-gen-109> in shebang(self, line, cell)

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/magics/script.py in shebang(self, line, cell)
    243             sys.stderr.flush()
    244         if args.raise_error and p.returncode!=0:
--> 245             raise CalledProcessError(p.returncode, cell, output=out, stderr=err)
    246 
    247     def _run_script(self, p, cell, to_close):

CalledProcessError: Command 'b'cd saskatchewan\npy.test\n'' returned non-zero exit status 1.

Note that it reported which test had failed, how many tests ran, and how many failed.

The symbol ..F means there were three tests, of which the third one failed.

Pytest will:

  • automagically finds files test_*.py
  • collects all subroutines called test_*
  • runs tests and reports results

Some options:

  • help: py.test --help
  • run only tests for a given feature: py.test -k foo # tests with 'foo' in the test name

Testing with floating points

Floating points are not reals

Floating points are inaccurate representations of real numbers:

1.0 == 0.99999999999999999 is true to the last bit.

This can lead to numerical errors during calculations: $1000 (a - b) \neq 1000a - 1000b$

In [8]:
1000.0 * 1.0 - 1000.0 * 0.9999999999999998
Out[8]:
2.2737367544323206e-13
In [9]:
1000.0 * (1.0 - 0.9999999999999998)
Out[9]:
2.220446049250313e-13

Both results are wrong: 2e-13 is the correct answer.

The size of the error will depend on the magnitude of the floating points:

In [10]:
1000.0 * 1e5 - 1000.0 * 0.9999999999999998e5
Out[10]:
1.4901161193847656e-08

The result should be 2e-8.

Comparing floating points

Use the "approx", for a default of a relative tolerance of $10^{-6}$

In [11]:
from pytest import approx
assert  0.7 == approx(0.7 + 1e-7) 

Or be more explicit:

In [12]:
magnitude = 0.7
assert 0.7 == approx(0.701 , rel=0.1, abs=0.1)

Comparing vectors of floating points

Numerical vectors are best represented using numpy.

In [13]:
from numpy import array, pi

vector_of_reals = array([0.1, 0.2, 0.3, 0.4]) * pi

Numpy ships with a number of assertions (in numpy.testing) to make comparison easy:

In [14]:
from numpy import array, pi
from numpy.testing import assert_allclose
expected = array([0.1, 0.2, 0.3, 0.4, 1e-12]) * pi
actual = array([0.1, 0.2, 0.3, 0.4, 2e-12]) * pi
actual[:-1] += 1e-6

assert_allclose(actual, expected, rtol=1e-5, atol=1e-8)

It compares the difference between actual and expected to atol + rtol * abs(expected).