MAKE working with Ipython notebooks more convenient

I really enjoy using the ipython notebook for coding. It gives me the possibilty to have code, plots and documentation in the same place all wrapped up in a nice format.

annoying

There are only two things that started to annoy me:

1. Until now it is not possible to use pythons from notebook1 import function syntax to split and reuse notebooks
2. I often find myself wanting to take my notebooks to meetings but there is no convenient way to turn all notebooks into html files which are easily shared and viewed on every computer.

MAKE to the rescue

Make is a command line tool for automatization. By storing rules in a file called makefile in our notebook directory we can run a sequence of commands we need often to save us all the typing.

make is lazy (= fast?) by design as it will not run commands if their input is unchanged.

There are plenty of tutorials out there and the documentation of make is very good, so I won't go into the workings of make.

Lets see how we can use it for out notebooks:

Importing from other notebooks

Since we can't import from notebook.ipynb we can't reuse its code. One work aroud is to convert the notebook from .ipynb to a regular .py script which can then be imported regularly.

# find all notebooks in the directory
# and store their names in a variable
notebooks = $(wildcard *.ipynb)

# we wan't to have '.py's for all our notebooks
scripts : $(notebooks:.ipynb=.py)

# and this is how we are going to get them
%.py : %.ipynb
    ipython nbconvert --to python $<

Converting to html

Similarly to the example above we can convert our notebookt to .html by adding the following targets and rules to our makefile.

docs : $(notebooks:.ipynb=.html)
%.html : %.ipynb
    ipython nbconvert --to html $<

Wrapping it all up

# file: makefile
notebooks = $(wildcard *.ipynb)

# all in one target
all : docs scripts

scripts : $(notebooks:.ipynb=.py)
%.py : %.ipynb
    ipython nbconvert --to python $<

docs : $(notebooks:.ipynb=.html)
%.html : %.ipynb
    ipython nbconvert --to html $<

# easily removing all the clutter we generated
clean :
    -rm $(notebooks:.ipynb=.py) $(notebooks:.ipynb=.html)

We can now invoce make in the command-line by simply typing:

make

It will generate all scripts and docs (run make scripts or make docs if you don't need both) for all our notebooks.

To remove all the files we generated we can run make clean

Importing and documentation is now only a single make command away

More automatization?

To automatically re-run make whenever a file has changed check out this stackoverflow conversation.

Installing numpy with pip

I am working on a university cluster and getting software installed the way it is done on my personal system, using debian/ubuntu package management, is always a tedious process as is involves various steps of permissions from senior staff.

When I started using python more frequently it was pretty easy to set up a virtual environment using virtualenv. But when I ran pip install numpy and tried to import it in the interpreter a certain library (libopenblas) was not accessible from my user account. It was softlinked to the /usr/lib directory so pip found it and used it for compilation but it wasnt readable.

Only after setting up ~/site.cfg (not ~/.numpy-site.cfg as mentioned in the documentation) the false libopenblas wa ignored

[openblas]
libraries =
library_dirs =