Things to make writing tests easier

Chris Withers

Something to test

"Here's some data in a CSV..."

"...parse it and tell me who owes the most money!"

Standard Library vs Other Libraries

Standard Library

• unittest

Other Libraries

• testfixtures
• mock
• unittest2

Comparing things

class MyTests(TestCase):

    def test_something(self):
        self.assertEqual(..., ...)

• Has got much better with successive Python versions
• Not much help if you're stuck on 2.6/2.7

Rich comparison

>>> from testfixtures import compare

>>> compare(1, 2)
Traceback (most recent call last):
 ...
AssertionError: 1 != 2

>>> compare("1234567891011", "1234567789")
Traceback (most recent call last):
...
AssertionError:
'1234567891011'
!=
'1234567789'

Rich comparison: long strings

>>> from testfixtures import compare
>>> compare("""
...         This is line 1
...         This is line 2
...         This is line 3
...         """,
...         """
...         This is line 1
...         This is another line
...         This is line 3
...         """)
Traceback (most recent call last):
 ...
AssertionError:
@@ -1,5 +1,5 @@

         This is line 1
-        This is line 2
+        This is another line
         This is line 3

Rich comparison: sets

>>> from testfixtures import compare
>>> compare(set([1, 2]), set([2, 3]))
Traceback (most recent call last):
 ...
AssertionError: set not as expected:

in first but not second:
[1]

in second but not first:
[3]

Rich comparison: dicts

>>> from testfixtures import compare
>>> compare(dict(x=1, y=2, a=4), dict(x=1, z=3, a=5))
Traceback (most recent call last):
 ...
AssertionError: dict not as expected:

same:
['x']

in first but not second:
'y': 2

in second but not first:
'z': 3

values differ:
'a': 4 != 5

Rich comparison: sequences

• lists
• tuples

>>> from testfixtures import compare
>>> compare([1, 2, 3], [1, 2, 4])
Traceback (most recent call last):
 ...
AssertionError: Sequence not as expected:

same:
[1, 2]

first:
[3]

second:
[4]

Comparison Helpers

• not all objects support comparison
• nor should they
• even just to make things testable

class SomeModel:
    def __eq__(self, other):
        if isinstance(other, SomeModel):
            return True
        return False

Rich comparison: generators/iterators

>>> def my_gen(t):
...     i = 0
...     while i<t:
...         i += 1
...         yield i

• be careful of just comparing by id
• be careful when unwinding

>>> from testfixtures import generator
>>> compare(generator(1, 2, 3), my_gen(2))
Traceback (most recent call last):
 ...
AssertionError: Sequence not as expected:

same:
(1, 2)

first:
(3,)

second:
()

Unfriendly strings

• process ids
• thread ids

from testfixtures import compare, StringComparison as S

compare(S('Starting thread \d+'),'Starting thread 132356')

• USE SPARINGLY!

Registering your own comparers

DataRow = namedtuple('DataRow', ('x', 'y', 'z'))

>>> from testfixtures.comparison import register, compare_sequence
>>> register(DataRow, compare_sequence)
>>> compare(DataRow(1, 2, 3), DataRow(1, 2, 4))
Traceback (most recent call last):
 ...
AssertionError: Sequence not as expected:

same:
(1, 2)

first:
(3,)

second:
(4,)

Standard library has better support in 3.1+:

    def test_different(self):
        self.addTypeEqualityFunc(DataRow, self.assertSequenceEqual)
        self.assertEqual(DataRow(1, 2, 3), DataRow(1, 2, 4))

Strict comparison

• comparison is relaxed and useful by default:

>>> TypeA = namedtuple('A', 'x')
>>> TypeB = namedtuple('B', 'x')
>>> compare(TypeA(1), TypeB(1))
<identity>

• not always what you want, so you can be strict

>>> compare(TypeA(1), TypeB(1), strict=True)
Traceback (most recent call last):
 ...
AssertionError:
A(x=1) (<class '__main__.A'>)!= B(x=1) (<class '__main__.B'>)

What about some context?

• all this rich comparison is great
• no contextual information provided

>>> compare(1, 2, prefix='wrong number of orders')
Traceback (most recent call last):
 ...
AssertionError: wrong number of orders: 1 != 2

• works with all previous examples

Things that print

• Lots of code writes to stdout/stderr

def myfunction()
    # code under test
    print("Hello!")
    print("Something bad happened!", file=sys.stderr)

• We should test that:

from testfixtures import OutputCapture
with OutputCapture() as output:
    myfunction()

output.compare('''
Hello!
Something bad happened!
''')

• Whitespace is stripped before comparison

output.captured

Exceptions

from testfixtures import ShouldRaise

with ShouldRaise():
    ...

with ShouldRaise(ValueError):
    ...

with ShouldRaise(ValueError('Something went wrong!')) as s:
    ...
compare(str(s.raised), 'Something went wrong!')

Standard library provides some of this in 3.1+:

with self.assertRaises(SomeException) as cm:
   ...

the_exception = cm.exception
self.assertEqual(the_exception.error_code, 3)

Files and Directories

• annoying to set up
• have to remember to clean up
• difficult to make cross-platform

from testfixtures import TempDirectory

class MyTests(TestCase):

    def setUp(self):
        self.dir = TempDirectory()

    def tearDown(self):
        self.dir.cleanup()

Writing files

• gives you back the path written:

with TempDirectory as dir:
    path = dir.write('foo.txt', b'bar')

• can do the encoding for you:

with TempDirectory as dir:
    path = dir.write('foo.txt', 'bar', 'utf8')

• can take sequences of path segments:

with TempDirectory as dir:
    path = dir.write(('root', gethostname(), 'foo.txt'), 'bar')

• takes slash separated path, on all operating systems:

with TempDirectory as dir:
    path = dir.write('my_folder/foo.txt', 'bar')

Okay, but it's my code that's writing!

• get a path that doesn't yet exist:

with TempDirectory as dir:
    path = dir.getpath('foo.txt')

• can take some familiar options:

with TempDirectory as dir:
    path = dir.getpath(('root', gethostname(), 'foo.txt'))

with TempDirectory as dir:
    path = dir.getpath('my_folder/foo.txt')

Reading files

• gives you bytes by default:

with TempDirectory as dir:
    ...
    compare(b'expected', dir.read('foo.txt'))

• but can decode if you want:

with TempDirectory as dir:
    ...
    compare('expected', dir.read('foo.txt', 'utf8'))

• the usual suspects:

with TempDirectory as dir:
    actual = dir.read(('root', gethostname(), 'foo.txt'))

with TempDirectory as dir:
    actual = dir.read('my_folder/foo.txt')

Directories

• easy to create, including intermediate directories:

with TempDirectory as dir:
    dir_path = dir.makedir('root/userfolder')

with TempDirectory as dir:
    dir_path = dir.makedir(('root', gethostname()))

• check contents of a particular directory:

with TempDirectory as dir:
    ...
    dir.check_dir('root/userfolder',
                  'user1.txt', 'user2.txt')

Directories

• can recursively check the contents of the whole temporary directory:

with TempDirectory as dir:
    ...
    dir.check_all('',
                  'root/',
                  'root/userfolder/',
                  'root/userfolder/user1.txt',
                  'root/userfolder/user2.txt')

• stable, ordered and cross platform
• can be configured to ignore certain name patterns

Logging

Python has a great standard logging framework:

• separates log generation from log recording
• provides a good interface for libraries
• has gotten much easier to configure handlers
• rarely tested!

Only capturing certain logging

• capture above a certain level

with LogCapture(level=logging.INFO) as log:
    logger= getLogger()
    logger.debug('junk')
    logger.info('what we care about')

log.check(('root', 'INFO', 'what we care about'))

• capture a particular logger

with LogCapture('specific') as log:
    getLogger('something').info('junk')
    getLogger('specific').info('what we care about')
    getLogger().info('more junk')

log.check(('specific', 'INFO', 'what we care about'))

Only capturing certain logging

• capture during a particular piece of code

class JobTests(TestCase):

    def setUp(self):
        self.log = LogCapture(install=False)
        log.job = MyJob()
        self.log.install()

    def tearDown(self):
        self.log.uninstall()

    def test_1(self):
        ...

    def test_2(self):
        ...

    def test_2(self):
        self.log.uninstall()
        self.job.prepare()
        self.log.install()
        self.job.run()
        self.log.check(...)

Testing handler configuration

class LoggingConfigurationTests(TestCase):

    def setUp(self):
         self.logger = logging.getLogger()
         self.orig_handlers = self.logger.handlers
         self.logger.handlers = []
         self.level = self.logger.level

     def tearDown(self):
         self.logger.handlers = self.orig_handlers
         self.logger.level = self.level

     def test_basic_configuration(self):
         # do configuration
         logging.basicConfig(format='%(levelname)s %(message)s',
                             level=logging.INFO)
         # check results of configuration
         compare(self.logger.level, 20)
         compare([
             C('logging.StreamHandler',
               stream=sys.stderr,
               formatter=C('logging.Formatter',
                           _fmt='%(levelname)s %(message)s',
                           strict=False),
               level=logging.NOTSET,
               strict=False)
             ], self.logger.handlers)

Mocking

• complex functionality with a defined API
• simple mock that matches that API
• often make assertions about calls
• sometimes need side effects / behaviour

Where do you mock?

One place:

import datetime as dt

def my_code():
    if dt.datetime.now() > dt.datetime(2013, 3, 23, 14, 30):
        say_i_do()

Lots of places:

from datetime import datetime

def my_code():
    if datetime.now() > datetime(2013, 3, 23, 14, 30):
        say_i_do()

How do you mock?

Patch decorator from mock

>>> @patch('package.module.Class')
... def test(MockClass):
...     instance = MockClass.return_value
...     instance.method.return_value = 'foo'
...     from package.module import Class
...     assert Class() is instance
...     assert Class().method() == 'foo'

• returns a Mock instance for you
• can only mock one thing per instance

How do you mock?

Replacer from testfixtures

def test_function():
    with Replacer() as r:
        mock_method = Mock()
        mock_class = Mock()
        r.replace('module.AClass.method', mock_method)
        r.replace('module.BClass', mock_class)
        from module import BClass
        x = AClass()
        y = BClass()
        self.assertTrue(x.method() is mock_method.return_value)
        self.assertTrue(y is mock_class.return_value)

• you can insert any object of your choice
• it's simple to mock multiple things

Mocking other things

someDict = dict(
    x='value',
    y=[1, 2, 3],
    )

• dict keys

with Replacer() as r:
    r.replace('module.someDict.x', 'foo')
    pprint(someDict)

{'x': 'foo', 'y': [1, 2, 3]}

• elements in lists

with Replacer() as r:
    r.replace('module.someDict.y.1', 42)
    pprint(someDict)

{'x': 'value', 'y': [1, 42, 3]}

Removing things

• attributes

from testfixtures import Replacer, not_there

with Replacer() as r:
    r.replace('module.someDict.x', not_there)
    print(hasattr(module, 'someDict')

False

• keys in dictionaries

from testfixtures import Replacer, not_there

with Replacer() as r:
    r.replace('module.someDict.y', not_there)
    pprint(someDict)

{'x': 'value'}

Things that might be there

try:
    from guppy import hpy
    guppy = True
except ImportError:
    guppy = False

def dump(path):
    if guppy:
        hpy().heap().stat.dump(path)

from mock import Mock, call
from testfixtures import replace

class Tests(unittest.TestCase):

    @replace('module.guppy',True)
    @replace('module.hpy', Mock(), strict=False)
    def test_method(self, hpy):

        dump('somepath')

        compare([
                 call(),
                 call().heap(),
                 call().heap().stat.dump('somepath')
                ], hpy.mock_calls)

What mock to use?

• Michael Foord's mock library
• soon to be in standard library
• worthy of a talk of its own
• what about dates and times?

datetimes

from testfixtures import test_datetime

• simple, variable gap times by default:

datetime = test_datetime()
print(datetime.now())
print(datetime.now())
print(datetime.now())

2001-01-01 00:00:00
2001-01-01 00:00:10
2001-01-01 00:00:30

datetimes

from testfixtures import test_datetime

• can be specifically configured:

datetime = test_datetime(None)
datetime.add(1978, 6, 13, 16, 0, 1)
datetime.add(2013, 3, 23, 14, 30)
print(datetime.now())
print(datetime.now())

1978-06-13 16:00:01
2013-03-23 14:30:00

datetimes

from testfixtures import test_datetime

• can also just have deltas configured:

datetime = test_datetime(delta=2, delta_type='hours')
print(datetime.now())
print(datetime.now())

2001-01-01 00:00:00
2001-01-01 02:00:00

• good support for time zones.

dates

from testfixtures import test_date

• Non-sequential dates by default:

date = test_date(2013, 3, 23)
print(date.today())
print(date.today())
print(date.today())

2013-03-23
2013-03-24
2013-03-26

dates

from testfixtures import test_date

• specific dates can be configured:

date = test_date(None)
date.add(1978, 6, 13)
date.add(2013, 3, 23)
print(date.today())
print(date.today())

1978-06-13
2013-03-23

dates

from testfixtures import test_date

• deltas can be configured for dates too:

date = test_date(delta=2, delta_type='days')
print(date.today())
print(date.today())

2001-01-01
2001-01-03

times

from testfixtures import test_time

• sequential times with increasing gaps:

time = test_time()
print(time())
print(time())
print(time())

978307200.0
978307201.0
978307203.0

times

from testfixtures import test_time

• specific times can be configured:

time = test_time(None)
time.add(1978, 6, 13, 16, 1)
time.add(2013, 3, 23, 14, 30)
print(time())
print(time())

266601660.0
1364049000.0

times

from testfixtures import test_time

• sub-second deltas are supported:

time = test_time(delta=0.5, delta_type='seconds')
print(time())
print(time())

978307200.0
978307200.5

Common pitfalls

Symmetric testing

• proves nothing

Excessive mocking

• misses interface changes
• doesn't match real behaviour

Modes of operation

TempDirectory, Replacer and LogCapture can all be used:

• as a decorator
• as a context manager
• manual, as we have in these examples

Questions

?

Links

testfixtures

• https://pypi.python.org/pypi/testfixtures
• search for "testfixtures python"

mock

• http://docs.python.org/dev/library/unittest.mock
• search for "python mock"

Chris Withers

• chris@withers.org
• @chriswithers13