Packaging

Packaging

Once we’ve made a working program, we’d like to be able to share it with others.

A good cross-platform build tool is the most important thing: you can always have collaborators build from source.

Distribution tools

Distribution tools allow one to obtain a working copy of someone else’s package.

Language-specific tools: PyPI, Ruby Gems, CPAN, CRAN Platform specific packagers e.g. brew, apt/yum

Until recently windows didn’t have anything like brew install or apt-get You had to build an ‘installer’, but now there is https://chocolatey.org

Laying out a project

When planning to package a project for distribution, defining a suitable project layout is essential.

%%bash
#%%cmd (windows)
tree --charset ascii greetings -I "doc|build|Greetings.egg-info|dist|*.pyc"
greetings
|-- CITATION.md
|-- LICENSE.md
|-- README.md
|-- greetings
|   |-- __init__.py
|   |-- command.py
|   |-- greeter.py
|   `-- test
|       |-- __init__.py
|       |-- fixtures
|       |   `-- samples.yaml
|       `-- test_greeter.py
`-- setup.py

3 directories, 10 files

We can start by making our directory structure

%%bash
mkdir -p greetings/greetings/test/fixtures

Using setuptools

To make python code into a package, we have to write a setupfile:

%%writefile greetings/setup.py
from setuptools import setup, find_packages

setup(
    name="Greetings",
    version="0.1.0",
    packages=find_packages(exclude=["*test"]),
    entry_points={"console_scripts": ["greet = greetings.command:process"]},
)
Overwriting greetings/setup.py

We can now install this code with

cd greetings
pip install .

And the package will be then available to use everywhere on the system.

from greetings.greeter import greet

greet("James", "Hetherington")
---------------------------------------------------------------------------
ModuleNotFoundError                       Traceback (most recent call last)
Input In [4], in <cell line: 1>()
----> 1 from greetings.greeter import greet
      3 greet("James", "Hetherington")

ModuleNotFoundError: No module named 'greetings.greeter'
from greetings.greeter import *

And the scripts are now available as command line commands:

%%bash
greet --help
usage: greet [-h] [--title TITLE] [--polite] personal family

Generate appropriate greetings

positional arguments:
  personal
  family

optional arguments:
  -h, --help            show this help message and exit
  --title TITLE, -t TITLE
  --polite, -p
%%bash
greet James Hetherington
greet --polite James Hetherington
greet James Hetherington --title Dr
Hey, James Hetherington.
How do you do, James Hetherington.
Hey, Dr James Hetherington.

Installing from GitHub

We could now submit “greeter” to PyPI for approval, so everyone could pip install it.

However, when using git, we don’t even need to do that: we can install directly from any git URL:

pip install git+git://github.com/jamespjh/greeter
%%bash
greet Humphry Appleby --title Sir
Hey, Sir Humphry Appleby.

Convert the script to a module

Of course, there’s more to do when taking code from a quick script and turning it into a proper module:

%%writefile greetings/greetings/greeter.py
def greet(personal, family, title="", polite=False):
    """Generate a greeting string for a person.

    Parameters
    ----------
    personal: str
        A given name, such as Will or Jean-Luc
    family: str
        A family name, such as Riker or Picard
    title: str
        An optional title, such as Captain or Reverend
    polite: bool
        True for a formal greeting, False for informal.

    Returns
    -------
    string
        An appropriate greeting
    """

    greeting = "How do you do, " if polite else "Hey, "
    if title:
        greeting += title + " "

    greeting += personal + " " + family + "."
    return greeting
Overwriting greetings/greetings/greeter.py
import greetings

help(greetings.greeter.greet)
Help on function greet in module greetings.greeter:

greet(personal, family, title='', polite=False)
    Generate a greeting string for a person.
    
    Parameters
    ----------
    personal: str
        A given name, such as Will or Jean-Luc
    family: str
        A family name, such as Riker or Picard
    title: str
        An optional title, such as Captain or Reverend
    polite: bool
        True for a formal greeting, False for informal.
    
    Returns
    -------
    string
        An appropriate greeting

The documentation string explains how to use the function; don’t worry about this for now, we’ll consider this next time.

Write an executable script

%%writefile greetings/greetings/command.py
from argparse import ArgumentParser
from .greeter import greet  # Note python 3 relative import


def process():
    parser = ArgumentParser(description="Generate appropriate greetings")

    parser.add_argument("--title", "-t")
    parser.add_argument("--polite", "-p", action="store_true")
    parser.add_argument("personal")
    parser.add_argument("family")

    arguments = parser.parse_args()

    print(
        greet(arguments.personal, arguments.family, arguments.title, arguments.polite)
    )


if __name__ == "__main__":
    process()
Overwriting greetings/greetings/command.py

Specify dependencies

We use the setup.py file to specify the packages we depend on:

setup(
    name = "Greetings",
    version = "0.1.0",
    packages = find_packages(exclude=['*test']),
    install_requires = ['argparse', 'pyyaml']
)

Specify entry point

%%writefile greetings/setup.py

from setuptools import setup, find_packages

setup(
    name="Greetings",
    version="0.1.0",
    packages=find_packages(exclude=["*test"]),
    install_requires=["argparse", "pyyaml"],
    entry_points={"console_scripts": ["greet = greetings.command:process"]},
)
Overwriting greetings/setup.py

Write a readme file

e.g.:

%%writefile greetings/README.md

Greetings!
==========

This is a very simple example package used as part of the Turing
[Research Software Engineering with Python](https://alan-turing-institute.github.io/rse-course) course.

Usage:
    
Invoke the tool with greet <FirstName> <Secondname>
Overwriting greetings/README.md

Write a license file

e.g.:

%%writefile greetings/LICENSE.md

(C) The Alan Turing Institute 2021

This "greetings" example package is granted into the public domain.
Overwriting greetings/LICENSE.md

Write a citation file

e.g.:

%%writefile greetings/CITATION.md

If you wish to refer to this course, please cite the URL
https://alan-turing-institute.github.io/rse-course

Portions of the material are taken from Software Carpentry
http://swcarpentry.org
Overwriting greetings/CITATION.md

You may well want to formalise this using the codemeta.json standard - this doesn’t have wide adoption yet, but we recommend it.

Define packages and executables

%%bash
touch greetings/greetings/test/__init__.py
touch greetings/greetings/__init__.py

Write some unit tests

Separating the script from the logical module made this possible:

%%writefile greetings/greetings/test/test_greeter.py
import yaml
import os
from ..greeter import greet


def test_greeter():
    with open(
        os.path.join(os.path.dirname(__file__), "fixtures", "samples.yaml")
    ) as fixtures_file:
        fixtures = yaml.safe_load(fixtures_file)
        for fixture in fixtures:
            answer = fixture.pop("answer")
            assert greet(**fixture) == answer
Overwriting greetings/greetings/test/test_greeter.py

Add a fixtures file:

%%writefile greetings/greetings/test/fixtures/samples.yaml
- personal: James
  family: Hetherington
  answer: "Hey, James Hetherington."
- personal: James
  family: Hetherington
  polite: True
  answer: "How do you do, James Hetherington."
- personal: James
  family: Hetherington
  title: Dr
  answer: "Hey, Dr James Hetherington."
Overwriting greetings/greetings/test/fixtures/samples.yaml
%%bash
pytest
============================= test session starts ==============================
platform darwin -- Python 3.8.13, pytest-7.1.2, py-1.10.0, pluggy-1.0.0
rootdir: /home/turingdev/projects/research-software/rse-course/ch04packaging
plugins: cov-2.12.1, anyio-3.3.0
collected 1 item

greetings/greetings/test/test_greeter.py .                               [100%]

============================== 1 passed in 0.15s ===============================

Developer Install

If you modify your source files, you would now find it appeared as if the program doesn’t change.

That’s because pip install copies the file.

(On my system to /Library/Python/2.7/site-packages/: this is operating system dependent.)

If you want to install a package, but keep working on it, you can do

cd greetings
pip install -e .

Distributing compiled code

If you’re working in C++ or Fortran, there is no language specific repository. You’ll need to write platform installers for as many platforms as you want to support.

Typically:

  • dpkg for apt-get on Ubuntu and Debian

  • rpm for yum on Redhat and Fedora

  • homebrew on OSX (Possibly macports as well)

  • An executable msi installer for Windows.

Homebrew

Homebrew: A ruby DSL, you host off your own webpage

See my installer for the cppcourse example

If you’re on OSX, do:

brew tap jamespjh/homebrew-reactor
brew install reactor