Travis Continuous Integration

I have been writing a lot of codes for my research and for random side projects. To ensure that modification to existing codes don’t break the rest of the code, it is crucial to test codes. Part of the astronomical software development community has been pretty good at this. The use of continuous integration (e.g., travis) is also becoming more and more common. I finally put in a few hours to pick it up. I incoporate a very simple test in a repo that I recently created for a case study.

To begin, I decided that I might as well convert the codes into a package, because I need to import some codes from src/ for the test scripts. So I wrote a setup.py and included pip install -e . inside my .travis.yml file (which means install current directory in development mode). In principle, I could have instead just include the __init__.py and sys.path.append('src') in the right places.

I also restructured the codebase a fair bit, following the more standard practice.

Note that inside my setup.py, I have package_dir={'indigocase': 'src'} , so that I can actually do import indigocase.lib in my test scripts after the repo is being build as a package as part of travis.

Similarly, I could write an __init__.py file inside src/ that does, e.g., from src.script import function so that we can just do from src import function in other scripts. <src> would be replaced with the name of package after it’s installed as a package.

Making GDAL and Travis Work is Non-trivial

The code in this repo uses the python package GDAL, which cannot be installed simply using pip. I have certainly installed it using conda, so I will do the same for the travis build. Thus, I added the installation of miniconda as part of the travis build process. Alas, to install gdal, it requires a separate libary and package (libspatialindex) that needed to be compiled from source. So I had to include some curl, tar, ./configure, make in the .travis.yml file.

A few things that require special attention and took me a few commits to figure out how to build the travis “environment” properly when we need to compile and install a package from source.

1. sudo is needed for make in .travis.yml
2. subdirctories listed in pytest.ini are automatically searched through following pytest discoveries rules. Put indigocase and not indigocase/tests/ as testpaths in the pytest.ini file
3. make sure to use pushd and popd when installing a package from source. Otherwise we won’t find the pytest.ini file.

Code structure before:

indigocase
├── MLutils.py
├── README.md
├── Vegation-Multispectral-Bands-Green-Red-Red-Edge-NIR.jpg
├── case.ipynb
├── case.py
├── config.py
├── config.yaml
├── imSeg.ipynb
├── indigocase
│   ├── data
│   │   ├── test.tif
│   │   ├── train.tif
│   │   ├── train_ground_truth.dbf
│   │   ├── train_ground_truth.prj
│   │   ├── train_ground_truth.qpj
│   │   ├── train_ground_truth.shp
│   │   └── train_ground_truth.shx
│   ├── plotField
│   ├── plotML
│   └── tests
├── main.py
├── parser.py
├── plant_analytics.py
├── present.pptx
├── scikitimage.png
├── split_fields_to_DF.py
├── utils.py
└── visutil.py

Code structure after:

indigocase
├── README.md
├── Vegation-Multispectral-Bands-Green-Red-Red-Edge-NIR.jpg
├── indigocase
│   ├── __init__.py
│   ├── case.ipynb
│   ├── case.py
│   ├── config.yaml
│   ├── data
│   │   ├── test.tif
│   │   ├── train.tif
│   │   ├── train_ground_truth.dbf
│   │   ├── train_ground_truth.prj
│   │   ├── train_ground_truth.qpj
│   │   ├── train_ground_truth.shp
│   │   └── train_ground_truth.shx
│   ├── imSeg.ipynb
│   ├── imSeg.py
│   ├── plotField
│   ├── plotML
│   ├── src
│   │   ├── MLutils.py
│   │   ├── __init__.py
│   │   ├── config.py
│   │   ├── main.py
│   │   ├── parser.py
│   │   ├── plant_analytics.py
│   │   ├── split_fields_to_DF.py
│   │   ├── utils.py
│   │   └── visutil.py
│   └── tests
│       ├── __init__.py
│       └── test_input.py
├── present.pptx
├── pytest.ini
├── requirements.txt
├── scikitimage.png
└── setup.py