XClose
Menu

Packaging

Packaging

Once we've made a working program, we'd like to be able to share it with others.

A good cross-platform build tool is the most important thing: you can always have collaborators build from source.

Distribution tools

Distribution tools allow one to obtain a working copy of someone else's package.

Language-specific tools: PyPI, Ruby Gems, CPAN, CRAN Platform specific packagers e.g. brew, apt/yum

Until recently windows didn't have anything like brew install or apt-get You had to build an 'installer', but now there is https://chocolatey.org

Laying out a project

When planning to package a project for distribution, defining a suitable project layout is essential.

In [1]:
%%bash
tree --charset ascii greetings -I "doc|build|Greetings.egg-info|dist|*.pyc"
greetings
|-- CITATION.md
|-- conf.py
|-- greetings
|   |-- command.py
|   |-- greeter.py
|   |-- __init__.py
|   `-- test
|       |-- fixtures
|       |   `-- samples.yaml
|       |-- __init__.py
|       `-- test_greeter.py
|-- index.rst
|-- LICENSE.md
|-- README.md
`-- setup.py

3 directories, 12 files

We can start by making our directory structure

In [2]:
%%bash
mkdir -p greetings/greetings/test/fixtures
mkdir -p greetings/scripts

Using setuptools

To make python code into a package, we have to write a setupfile:

In [3]:
%%writefile greetings/setup.py

from setuptools import setup, find_packages

setup(
    name = "Greetings",
    version = "0.1.0",
    packages = find_packages(exclude=['*test']),
)
Overwriting greetings/setup.py

We can now install this code with

python setup.py install

And the package will be then available to use everywhere on the system.

In [4]:
from greetings.greeter import greet
greet("James","Hetherington")
Out[4]:
'Hey, James Hetherington.'

And the scripts are now available as command line commands:

In [5]:
%%bash
greet --help
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.3/bin/greet", line 2, in <module>
    from greetings.command import process
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/greetings/command.py", line 14
    print greet(arguments.personal, arguments.family, 
              ^
SyntaxError: invalid syntax
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
<ipython-input-5-e306ac58e44f> in <module>
----> 1 get_ipython().run_cell_magic('bash', '', 'greet --help\n')

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2321             magic_arg_s = self.var_expand(line, stack_depth)
   2322             with self.builtin_trap:
-> 2323                 result = fn(magic_arg_s, cell)
   2324             return result
   2325 

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/magics/script.py in named_script_magic(line, cell)
    140             else:
    141                 line = script
--> 142             return self.shebang(line, cell)
    143 
    144         # write a basic docstring:

<decorator-gen-109> in shebang(self, line, cell)

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/magics/script.py in shebang(self, line, cell)
    243             sys.stderr.flush()
    244         if args.raise_error and p.returncode!=0:
--> 245             raise CalledProcessError(p.returncode, cell, output=out, stderr=err)
    246 
    247     def _run_script(self, p, cell, to_close):

CalledProcessError: Command 'b'greet --help\n'' returned non-zero exit status 1.
In [6]:
%%bash
greet James Hetherington
greet --polite James Hetherington
greet James Hetherington --title Dr
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.3/bin/greet", line 2, in <module>
    from greetings.command import process
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/greetings/command.py", line 14
    print greet(arguments.personal, arguments.family, 
              ^
SyntaxError: invalid syntax
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.3/bin/greet", line 2, in <module>
    from greetings.command import process
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/greetings/command.py", line 14
    print greet(arguments.personal, arguments.family, 
              ^
SyntaxError: invalid syntax
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.3/bin/greet", line 2, in <module>
    from greetings.command import process
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/greetings/command.py", line 14
    print greet(arguments.personal, arguments.family, 
              ^
SyntaxError: invalid syntax
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
<ipython-input-6-08bc2fc2a965> in <module>
----> 1 get_ipython().run_cell_magic('bash', '', 'greet James Hetherington\ngreet --polite James Hetherington\ngreet James Hetherington --title Dr\n')

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2321             magic_arg_s = self.var_expand(line, stack_depth)
   2322             with self.builtin_trap:
-> 2323                 result = fn(magic_arg_s, cell)
   2324             return result
   2325 

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/magics/script.py in named_script_magic(line, cell)
    140             else:
    141                 line = script
--> 142             return self.shebang(line, cell)
    143 
    144         # write a basic docstring:

<decorator-gen-109> in shebang(self, line, cell)

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/magics/script.py in shebang(self, line, cell)
    243             sys.stderr.flush()
    244         if args.raise_error and p.returncode!=0:
--> 245             raise CalledProcessError(p.returncode, cell, output=out, stderr=err)
    246 
    247     def _run_script(self, p, cell, to_close):

CalledProcessError: Command 'b'greet James Hetherington\ngreet --polite James Hetherington\ngreet James Hetherington --title Dr\n'' returned non-zero exit status 1.

Installing from GitHub

We could now submit "greeter" to PyPI for approval, so everyone could pip install it.

However, when using git, we don't even need to do that: we can install directly from any git URL:

pip install git+git://github.com/jamespjh/greeter
In [7]:
%%bash
greet Humphry Appleby --title Sir
Traceback (most recent call last):
  File "/home/travis/virtualenv/python3.6.3/bin/greet", line 2, in <module>
    from greetings.command import process
  File "/home/travis/virtualenv/python3.6.3/lib/python3.6/site-packages/greetings/command.py", line 14
    print greet(arguments.personal, arguments.family, 
              ^
SyntaxError: invalid syntax
---------------------------------------------------------------------------
CalledProcessError                        Traceback (most recent call last)
<ipython-input-7-b05193d8ac1e> in <module>
----> 1 get_ipython().run_cell_magic('bash', '', 'greet Humphry Appleby --title Sir\n')

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
   2321             magic_arg_s = self.var_expand(line, stack_depth)
   2322             with self.builtin_trap:
-> 2323                 result = fn(magic_arg_s, cell)
   2324             return result
   2325 

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/magics/script.py in named_script_magic(line, cell)
    140             else:
    141                 line = script
--> 142             return self.shebang(line, cell)
    143 
    144         # write a basic docstring:

<decorator-gen-109> in shebang(self, line, cell)

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
    185     # but it's overkill for just that one bit of state.
    186     def magic_deco(arg):
--> 187         call = lambda f, *a, **k: f(*a, **k)
    188 
    189         if callable(arg):

~/virtualenv/python3.6.3/lib/python3.6/site-packages/IPython/core/magics/script.py in shebang(self, line, cell)
    243             sys.stderr.flush()
    244         if args.raise_error and p.returncode!=0:
--> 245             raise CalledProcessError(p.returncode, cell, output=out, stderr=err)
    246 
    247     def _run_script(self, p, cell, to_close):

CalledProcessError: Command 'b'greet Humphry Appleby --title Sir\n'' returned non-zero exit status 1.

Convert the script to a module

Of course, there's more to do when taking code from a quick script and turning it into a proper module:

In [8]:
%%writefile greetings/greetings/greeter.py

def greet(personal, family, title="", polite=False):

    """ Generate a greeting string for a person.

    Parameters
    ----------
    personal: str
        A given name, such as Will or Jean-Luc
    family: str
        A family name, such as Riker or Picard
    title: str
        An optional title, such as Captain or Reverend
    polite: bool
        True for a formal greeting, False for informal.

    Returns
    -------
    string
        An appropriate greeting
    """

    greeting= "How do you do, " if polite else "Hey, "
    if title:
        greeting+=title+" "

    greeting+= personal + " " + family +"."
    return greeting
Overwriting greetings/greetings/greeter.py
In [9]:
import greetings
help(greetings.greeter.greet)
Help on function greet in module greetings.greeter:

greet(personal, family, title='', polite=False)
    Generate a greeting string for a person.
    
    Parameters
    ----------
    personal: str
        A given name, such as Will or Jean-Luc
    family: str
        A family name, such as Riker or Picard
    title: str
        An optional title, such as Captain or Reverend
    polite: bool
        True for a formal greeting, False for informal.
    
    Returns
    -------
    string
        An appropriate greeting

The documentation string explains how to use the function; don't worry about this for now, we'll consider this next time.

Write an executable script

In [10]:
%%writefile greetings/greetings/command.py
from argparse import ArgumentParser
from .greeter import greet # Note python 3 relative import

def process():
   parser = ArgumentParser(description = "Generate appropriate greetings")

   parser.add_argument('--title', '-t')
   parser.add_argument('--polite', '-p', action="store_true")
   parser.add_argument('personal')
   parser.add_argument('family')

   arguments= parser.parse_args()

   print(greet(arguments.personal, arguments.family, 
               arguments.title, arguments.polite))

if __name__ == "__main__":
    process()
Overwriting greetings/greetings/command.py
In [11]:
### Specify dependencies

We use the setup.py file to specify the packages we depend on:

setup(
    name = "Greetings",
    version = "0.1.0",
    packages = find_packages(exclude=['*test']),
    install_requires = ['argparse']
)

Specify entry point

In [12]:
%%writefile greetings/setup.py

from setuptools import setup, find_packages

setup(
    name = "Greetings",
    version = "0.1.0",
    packages = find_packages(exclude=['*test']),
    install_requires = ['argparse'],
    entry_points={
        'console_scripts': [
            'greet = greetings.command:process'
        ]})
Overwriting greetings/setup.py

Write a readme file

e.g.:

In [13]:
%%writefile greetings/README.md

Greetings!
==========

This is a very simple example package used as part of the UCL
[Research Software Engineering with Python](development.rc.ucl.ac.uk/training/engineering) course.

Usage:
    
Invoke the tool with greet <FirstName> <Secondname>
Overwriting greetings/README.md

Write a license file

e.g.:

In [14]:
%%writefile greetings/LICENSE.md

(C) University College London 2014

This "greetings" example package is granted into the public domain.
Overwriting greetings/LICENSE.md

Write a citation file

e.g.:

In [15]:
%%writefile greetings/CITATION.md

If you wish to refer to this course, please cite the URL
http://development.rc.ucl.ac.uk/training/engineering

Portions of the material are taken from Software Carpentry
http://swcarpentry.org
Overwriting greetings/CITATION.md

You may well want to formalise this using the codemeta.json standard - this doesn't have wide adoption yet, but we recommend it.

Define packages and executables

In [16]:
%%bash
touch greetings/greetings/test/__init__.py
touch greetings/greetings/__init__.py

Write some unit tests

Separating the script from the logical module made this possible:

In [17]:
%%writefile greetings/greetings/test/test_greeter.py
import yaml
import os
from ..greeter import greet

def test_greeter():
    with open(os.path.join(os.path.dirname(__file__),
            'fixtures','samples.yaml')) as fixtures_file:
        fixtures=yaml.load(fixtures_file)
        for fixture in fixtures:
            answer=fixture.pop('answer')
            assert greet(**fixture) == answer
Overwriting greetings/greetings/test/test_greeter.py

Add a fixtures file:

In [18]:
%%writefile greetings/greetings/test/fixtures/samples.yaml
- personal: James
  family: Hetherington
  answer: "Hey, James Hetherington."
- personal: James
  family: Hetherington
  polite: True
  answer: "How do you do, James Hetherington."
- personal: James
  family: Hetherington
  title: Dr
  answer: "Hey, Dr James Hetherington."
Overwriting greetings/greetings/test/fixtures/samples.yaml
In [19]:
%%bash
py.test
============================= test session starts ==============================
platform linux -- Python 3.6.3, pytest-4.1.1, py-1.5.2, pluggy-0.8.1
rootdir: /home/travis/build/alan-turing-institute/rsd-engineeringcourse/ch04packaging, inifile:
plugins: cov-2.6.1
collected 1 item

greetings/greetings/test/test_greeter.py .                               [100%]

=========================== 1 passed in 0.07 seconds ===========================

Developer Install

If you modify your source files, you would now find it appeared as if the program doesn't change.

That's because pip install copies the file.

(On my system to /Library/Python/2.7/site-packages/: this is operating system dependent.)

If you want to install a package, but keep working on it, you can do

python setup.py develop

Distributing compiled code

If you're working in C++ or Fortran, there is no language specific repository. You'll need to write platform installers for as many platforms as you want to support.

Typically:

  • dpkg for apt-get on Ubuntu and Debian
  • rpm for yum on Redhat and Fedora
  • homebrew on OSX (Possibly macports as well)
  • An executable msi installer for Windows.

Homebrew

Homebrew: A ruby DSL, you host off your own webpage

See my installer for the cppcourse example

If you're on OSX, do:

brew tap jamespjh/homebrew-reactor
brew install reactor