PyFAI is a library to deal with diffraction images for data reduction. This chapter describes the project from the computer engineering point of view.

PyFAI is an open source project licensed under the GPL mainly written in Python (v2.6, 2.7) and heavily relying on the python scientific ecosystem: numpy, scipy and matplotlib. It provides high performances image treatment thanks to cython and OpenCL... but only a C-compiler is needed to build that.

Programming language

PyFAI is a Python project but uses many programming languages:

  • 15000 lines of Python (plus 5000 for the test)
  • 11000 lines of Cython which are converted into ... C (about a million lines)
  • 5000 lines of OpenCL kernels

The OpenCL code has been tested using:

  • Nvidia OpenCL v1.1 on Linux, Windows (GPU device)
  • Intel OpenCL v1.2 on Linux and Windows (CPU and ACC (Phi) devices)
  • AMD OpenCL v1.2 on Linux and Windows (CPU and GPU device)
  • Apple OpenCL v1.2 on MacOSX (CPU and GPU)
  • Beignet OpenCL v1.2 on Linux (GPU device)
  • Pocl OpenCL v1.2 on Linux (CPU device)

Repository:

The project is hosted by GitHub: https://github.com/pyFAI/pyFAI

Which provides the issue tracker in addition to Git hosting. Collaboration is done via Pull-Requests in github’s web interface:

Everybody is welcome to fork the project and adapt it to his own needs: CEA-Saclay, Synchrotrons Soleil, Desy and APS have already done so. Collaboration is encouraged and new developments can be submitted and merged into the main branch via pull-requests.

Getting help

A mailing list: pyfai@esrf.fr is publicly available. To subscribe, send an email to sympa@esrf.fr with “subscribe pyfai” as subject. On this mailing list, you will have information about release of the software, new features available and meet experts to help you solve issues related to azimuthal integration. This mailing list is archived and can be consulted at: http://www.edna-site.org/lurker

Run dependencies

  • Python version 2.6, 2.7 3.2 or newer
  • NumPy
  • SciPy
  • Matplotlib
  • FabIO
  • h5py (optional)
  • pyopencl (optional)
  • fftw (optional)
  • pymca (optional)
  • PyQt4 or PySide (for the graphical user interface)

Build dependencies:

In addition to the run dependencies, pyFAI needs a C compiler. There is an issue with MacOSX (v10.8 onwards) where the default compiler (Xcode5 or 6) switched from gcc 4.2 to clang which dropped the support for OpenMP (clang v3.5 supports OpenMP under linux but not directly under MacOSX). Multiple solution exist, pick any of those:

  • Install a recent version of GCC (>=4.2)
  • Use Xcode without OpenMP, using the –no-openmp flag for setup.py.

C files are generated from cython source and distributed. Cython is only needed for developing new binary modules. If you want to generate your own C files, make sure your local Cython version supports memory-views (available from Cython v0.17 and newer), unless your Cython files will not be compiled or used.

Building procedure

As most of the python projects:

python setup.py build install

There are few specific options to setup.py:

  • –no-cython: do not use cython (even if present) and use the C source code provided by the development team
  • –no-openmp: if you compiler lacks OpenMP support, like Xcode on MacOSX
  • –with-testimages: build the source distribution including all test images. Download 200MB of test images to create a self consistent tar-ball.

Test suites

To run the test an internet connection is needed as 200MB of test images will be downloaded.

Setting the environment variable http_proxy can be necessary (depending on your network):

export http_proxy=http://proxy.site.org:3128

Especially at ESRF, the configuration of the network proxy can be obtained by phoning on the hotline: 24-24.

PyFAI comes with 30 test-suites (172 tests in total) representing a coverage of 67%. This ensures both non regression over time and ease the distribution under different platforms: pyFAI runs under Linux, MacOSX and Windows (in each case in 32 and 64 bits). Test may not pass on computer featuring less than 2GB of memory or 32 bit computers.

Test suite coverage
Name Stmts Miss Cover
pyFAI/__init__ 14 5 64%
pyFAI/_version 31 1 97%
pyFAI/azimuthalIntegrator 1209 311 74%
pyFAI/blob_detection 520 323 38%
pyFAI/calibrant 360 112 69%
pyFAI/detectors 1147 287 75%
pyFAI/directories 30 8 73%
pyFAI/geometry 815 208 74%
pyFAI/geometryRefinement 477 304 36%
pyFAI/gui_utils 66 41 38%
pyFAI/io 457 214 53%
pyFAI/massif 189 60 68%
pyFAI/multi_geometry 66 4 94%
pyFAI/ocl_azim 269 78 71%
pyFAI/ocl_azim_csr 225 46 80%
pyFAI/ocl_azim_lut 219 45 79%
pyFAI/opencl 191 52 73%
pyFAI/peak_picker 732 537 27%
pyFAI/spline 397 249 37%
pyFAI/third_party/__init__ 0 0 100%
pyFAI/third_party/six 393 184 53%
pyFAI/units 42 6 86%
pyFAI/utils 744 315 58%
test_all 81 1 99%
test_azimuthal_integrator 241 67 72%
test_bilinear 82 8 90%
test_bispev 66 16 76%
test_blob_detection 54 5 91%
test_bug_regression 41 5 88%
test_calibrant 126 35 72%
test_convolution 54 6 89%
test_csr 88 23 74%
test_detector 171 13 92%
test_distortion 56 8 86%
test_dummy 27 6 78%
test_export 87 9 90%
test_flat 112 9 92%
test_geometry 91 6 93%
test_geometry_refinement 64 7 89%
test_histogram 234 19 92%
test_integrate 139 12 91%
test_io 108 30 72%
test_marchingsquares 42 9 79%
test_mask 218 55 75%
test_multi_geometry 79 20 75%
test_openCL 225 27 88%
test_peak_picking 89 11 88%
test_polarization 57 6 89%
test_saxs 105 31 70%
test_sparse 44 5 89%
test_split_pixel 74 6 92%
test_utils 97 6 94%
test_watershed 40 6 85%
TOTAL 11585 3857 67%

Note that the test coverage tool does not count lines of Cython, nor those of OpenCL Updated 27/05/2014

Continuous integration is made by a home-made scripts which checks out the latest release and builds and runs the test every night. Nightly builds are available for debian6-64 bits in: http://www.edna-site.org/pub/debian/binary/

List of contributors in code

$ git log  --pretty='%aN##%s' | grep -v 'Merge pull' | grep -Po '^[^#]+' | sort | uniq -c | sort -rn
As of 05/2015:
  • Jérôme Kieffer (ESRF)
  • Aurore Deschildre (ESRF)
  • Frédéric-Emmanuel Picca (Soleil)
  • Giannis Ashiotis (ESRF)
  • Dimitrios Karkoulis (ESRF)
  • Jon Wright (ESRF)
  • Zubair Nawaz (Sesame)
  • Amund Hov (ESRF)
  • Dodogerstlin @github
  • Gunthard Benecke (Desy)
  • Gero Flucke (Desy)
  • Vadim Dyadkin (ESRF)

List of other contributors (ideas or code)

  • Peter Boesecke (geometry)
  • Manuel Sanchez del Rio (histogramming)
  • Armando Solé (masking widget + PyMca plugin)
  • Sebastien Petitdemange (Lima plugin)

List of supporters

  • LinkSCEEM project: porting to OpenCL
  • ESRF ID11: Provided manpower in 2012 and 2013 and beamtime
  • ESRF ID13: Provided manpower in 2012, 2013, 2014, 2015 and beamtime
  • ESRF ID29: provided manpower in 2013 (MX-calibrate)
  • ESRF ID02: provided manpower 2014
  • ESRF ID15: provide manpower 2015