5.6 Continuous Integration#

Estimated time for this notebook: 15 minutes

Getting past “but it works on my machine…”#

Try running the code below:

%%bash
rm -rf continuous_int
mkdir continuous_int
touch continuous_int/__init__.py

%%writefile continuous_int/test_demo.py
import sys
import re

def test_platform():
    assert re.search("\d", sys.platform)

def test_replace():
    assert "".replace("", "A", 2) == "A"

Writing continuous_int/test_demo.py

%%bash
cd continuous_int
pytest || echo "tests complete"

============================= test session starts ==============================

platform linux -- Python 3.8.18, pytest-7.4.4, pluggy-1.3.0

rootdir: /home/runner/work/rse-course/rse-course/module05_testing_your_code/continuous_int

plugins: pylama-8.4.1, anyio-4.2.0, cov-4.1.0

collected 2 items



test_demo.py FF                                                          [100%]



=================================== FAILURES ===================================

________________________________ test_platform _________________________________



    def test_platform():

>       assert re.search("\d", sys.platform)

E       AssertionError: assert None

E        +  where None = <function search at 0x7fb09b1a75e0>('\\d', 'linux')

E        +    where <function search at 0x7fb09b1a75e0> = re.search

E        +    and   'linux' = sys.platform



test_demo.py:5: AssertionError

_________________________________ test_replace _________________________________



    def test_replace():

>       assert "".replace("", "A", 2) == "A"

E       AssertionError: assert '' == 'A'

E         - A



test_demo.py:8: AssertionError

=============================== warnings summary ===============================

test_demo.py:5

  /home/runner/work/rse-course/rse-course/module05_testing_your_code/continuous_int/test_demo.py:5: DeprecationWarning: invalid escape sequence \d

    assert re.search("\d", sys.platform)



-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

=========================== short test summary info ============================

FAILED test_demo.py::test_platform - AssertionError: assert None

 +  where None = <function search at 0x7fb09b1a75e0>('\\d', 'linux')

 +    where <function search at 0x7fb09b1a75e0> = re.search

 +    and   'linux' = sys.platform

FAILED test_demo.py::test_replace - AssertionError: assert '' == 'A'

  - A

========================= 2 failed, 1 warning in 0.10s =========================

tests complete

The example above is a trival, and deliberate, example of code that will behave differently on different computers.

Much more subtle instances can occur in real-life, which if allowed to propergate, they can result in bugs and errors that are difficult to trace, let alone fix.

One mitigation for this problem is to use a process of “Continuous Integration (CI)”. This is a process of drawing together all developer contributions as early as possible and freaquently running the automated tests. Typically this involves the use of CI servers, which provide a common and reliable environment to run our tests. (This is not the only use of CI servers - we will touch on other use cases in later modules)

Options for CI Servers#

There are many different open-source or propritory CI Servers available. In some cases it might be appropriate to have on-permise CI Servers at your organisation.

There are also a number of Continuous-Integration-Server-as-a-Service products that can be use free-of-charge for open source projects. Here we will expand on “GitHub Actions” which is a Continuous-Integration-Server-as-a-Service, which is one component of the wider GitHub ecosystem.

Objectives#

We would like to test our code on

  • different operating systems

  • different versions of python

  • each commit to a pull request

%%bash
mkdir -p continuous_int/.github/workflows

%%writefile continuous_int/.github/workflows/ci-tests.yml
# This workflow will install Python dependencies, run tests with a variety of Python versions, on Windows and Linux
# For more information see: https://help.github.com/actions/language-and-framework-guides/using-python-with-github-actions

name: Unit tests

on:
  pull_request:
    branches:
      - main
  push:

jobs:
  build:
    strategy:
      # We use `fail-fast: false` for teaching purposess. This ensure that all combinations of the matrix
      # will run even if one or more fail. 
      fail-fast: false
      matrix:
        python-version: [3.8, 3.9, "3.10"]
        os: [ubuntu-latest, windows-latest]
    
    runs-on: ${{ matrix.os }}

    steps:
    - uses: actions/checkout@v2
    - name: Set up Python ${{ matrix.python-version }}
      uses: actions/setup-python@v1
      with:
        python-version: ${{ matrix.python-version }}
    
    # Yes we have to explictly install pytest. In a "real" example this could be included in a 
    # requirement.txt or environment.yml to setup your environment
    - name: Install PyTest
      run: |
        python -m pip install pytest
    # Now run the tests
    - name: Test with pytest
      run: |
        pytest

Writing continuous_int/.github/workflows/ci-tests.yml

Apply this to the personal github repo you made in module 04”#

  • Create a new branch in your repo.

  • Copy the files in the continuous_int directory into your local clone. Note that the .yml file must exist in the directory .github/workflows, which must be in the root of your repo. (The . prefixed to the .github directory means that it is hidden by default).

  • Commit your changes and push them

  • Create a Pull Request to the main branch of your own repo.

When succesfully applied to your repo, you should see that a number of tests are completed on every commit pushed, on every pull request.

These tests have been designed that they will both pass only if they are run on Windows and on Python v3.9 or higher, in order to demostrate the matrix workings of GH Actions. In a more more realistic senario, you should aim to have your test pass in all contexts.

Futher reading:#

  • There can be cases where is it propriate to expect different behaviour on different platforms. PyTest has features that allow for cases.

  • GitHub Actions themselves can be difficult to debug because of the need to commit and push every minor change. “Act” provides a tool to help debug some GH Actions locally.