Skip to content

FSL conda packages

This page describes:

  • how to write a conda recipe for a FSL project

  • how to build a conda package from a FSL conda recipe

All FSL project repositories have an associated recipe repository, which contains metadata describing how to build a conda recipe from the project repository.

When creating a conda recipe for a new FSL project, the easiest way to start is to copy the recipe for an existing similar project. For example, if you are creating a recipe for a Python project, you could copy the eddy_qc recipe. Or, if you are creating a recipe for a C++ project, you could copy the avwutils recipe.

Conda recipes for all FSL projects, and some internally packaged/managed third-party dependencies, are hosted in the GitLab fsl/conda namespace. The name of a FSL conda recipe repository is usually the same as the name of the FSL conda package - for example, the conda package for the fsl/avwutils project is named fsl-avwutils, and is hosted at fsl/conda/fsl-avwutils.

FSL conda package naming conventions

FSL conda package names must follow the conda package naming conventions, and be comprised solely of lowercase alpha characters, numeric digits, underscores, hyphens, or dots.

Furthermore, all FSL conda packages are prefixed with fsl-. An FSL project with name <project> will have a corresponding conda package name of fsl-<project>. For FSL projects with a name that begins with fsl (e.g. fslvbm, fsl_deface), the leading fsl will be dropped in the construction of the corresponding conda-package name. For example:

FSL project name Conda package name
avwutils fsl-avwutils
fslvbm fsl-vbm
fsl_deface fsl-deface
fsl-mrs fsl-mrs
NewNifti fsl-newnifti

There are a small number of exceptions to the above conventions. For example, the fdt project is built into two conda packages - the fsl-fdt package, providing CPU-only executables, and the fsl-fdt-cuda package, providing GPU/CUDA-enabled executables.

Conda recipes for FSL CUDA projects

Most FSL CUDA projects usually provide both GPU-enabled and CPU-only executables. For example, the fsl/fdt provides a range of CPU-only executables, including dtifit and vecreg, in addition to providing GPU-enabled executables such as xfibres_gpu.

To accommodate this convention, multiple conda recipes are used for these "hybrid" projects. For example, packages for the fsl/fdt project are built from two separate recipes:

  • fsl/conda/fsl-fdt, which builds the CPU-only executables - these recipes are built as linux and macos packages.
  • fsl/conda/fsl-fdt-cuda, which builds the GPU-enabled executables - these recipes are built as linux-cuda packages.

Creating a conda recipe for a FSL project

We recommend copying an existing FSL conda recipe to generate an initial version of the recipe for your FSL project. This has the advantage that recipes for all FSL projects will follow a few simple conventions and standards.

However, it is possible to create a conda recipe by hand. The FSL build system supports two types of conda recipes:

The purpose of this section is not to provide full instructions on writing a conda recipe, but rather on the specific issues that need to be considered when writing a conda recipe for a FSL project. More general information on creating conda recipes can be found in the conda build and rattler-build documentation.

A FSL conda recipe is simply a flat directory containing the following files:

  • meta.yaml or recipe.yaml: A YAML file which contains a description of the conda package, including its name, current version, URL to the project repository, and list of dependencies.

  • build.sh: A bash script which is executed in order to build the package. For FSL C++ projects, this script essentially just calls ${FSLDIR}/etc/fslconf/fsl-devel.sh, then runs make and make install. For FSL Makefile-based projects, build.sh is required, but for Python-based projects, build.sh may or may not be necessary.

  • post-link.sh: For projects which provide executables that are installed into ${FSLDIR}/bin/, this script is used to conditionally create entry points/wrapper scripts in ${FSLDIR}/share/fsl/bin/, as outlined in the configuration page. This script is not required for projects which do not provide any executables.

  • pre-unlink.sh: This script is called when a package is uninstalled - its job is to remove any entry points that were created by post-link.sh. This script is not required for projects which do not provide any executables.

FSL projects are broadly divided into one of the following categories:

  • Makefile-based project: FSL projects which use a FSL-style Makefile to compile and install their deliverables (shared libraries, scripts, compiled executables, and/or data files).

  • Makefile-based CUDA project: FSL projects which use a FSL-style Makefile, and which provide GPU-accelerated executables linked against CUDA.

  • Python project: FSL projects which are written in Python, and which have a setup.py or pyproject.toml file which is used to build the project as a Python package.

The mechanisms by which projects in these categories are built are slightly different and, therefore, the conda recipe for projects from different category will look slightly different. Some important details are highlighted below.

Deciding between conda build and rattler-build

The FSL build system supports two different recipe formats:

  • Recipes containing a meta.yaml file will be built with conda build.

  • Recipes containing a recipe.yaml file will be built with rattler-build.

Most FSL recipes are based on a meta.yaml file, and so are built with conda build. However, rattler-build is more performant (i.e. builds packages more quickly), and so is recommended for new recipes.

Writing the meta.yaml/recipe.yaml file

The meta.yaml/recipe.yaml file contains metadata about your project, including:

  • The conda package name
  • The URL of the project git repository
  • The version number
  • The build number (used in case the conda package for a single version needs to be re-built for some reason).
  • A list of the package build- and run-time dependencies.

In order to allow for automatic maintenance of FSL conda recipes, you should define the essential metadata (name, version, etc) using the following syntax:

As jinja2 variables at the top of the file, e.g.:

{% set name       = '<conda-package-name>' %}
{% set version    = '<version-number>'     %}
{% set repository = '<repository-url>'     %}
{% set build      = '0'                    %}

Within the context section at the top of the file, e.g.:

context:
  name:       "<conda-package-name>"
  version:    "<version-number>"
  repository: "<repository-url>"
  build:      "0"

Then use these variables within the recipe YAML, e.g.:

package:
  name:    {{ name    }}
  version: {{ version }}
package:
  name:    ${{ name    }}
  version: ${{ version }}

By following this convention, FSL conda recipes can be automatically updated when a new version of a project is released.

Project repository and git revision

The automated CI rules defined in fsl-ci-rules allow the project repositoy and git revision (tag, branch, etc) used to build a conda package to be overridden via the FSLCONDA_REPOSITORY and FSLCONDA_REVISION environment variables. To facilitate this, the project source repository and git revision must be specified in the meta.yaml like so (remembering that we have defined repository and version variables as described above):

source:
  # the FSLCONDA_REPOSITORY and FSLCONDA_REVISION
  # environment variables can be used to override
  # the repository/revision for development purposes.
  git_url: {{ os.environ.get("FSLCONDA_REPOSITORY", repository) }}
  git_rev: {{ os.environ.get("FSLCONDA_REVISION",   version)    }}
source:
  # the FSLCONDA_REPOSITORY and FSLCONDA_REVISION
  # environment variables can be used to override
  # the repository/revision for development purposes.
  git: ${{ env.get("FSLCONDA_REPOSITORY", default=repository) }}
  rev: ${{ env.get("FSLCONDA_REVISION",   default=version)    }}

Following this convention allows conda packages for development and testing to be built, both automatically, and when developing/testing locally, simply by setting the FSLCONDA_REPOSITORY and FSLCONDA_REVISION variables.

run_exports (C/C++ projects only)

When compiling a C/C++ project, any shared library dependencies of the project must be present at the time of compilation, and at run time. This means that the dependencies of your project may need to be listed twice within the requirements section - once under host (or build), and again under run. We can avoid having to list dependencies twice by specifying run_exports in the dependency recipes.

The run_exports section is a trick which can be used within the meta.yaml/recipe.yaml file, which essentially allows us to define a dependency as a build-time dependency, and have it automatically propagated as a run-time dependency.

run_exports also allows us to specify the ABI compatibilty guarantees of a project. Maintainers of a C/C++ library must consider ABI compatibility across different versions of the library. For FSL C/C++ projects which follow the YYMM.B versioning scheme (described in the FSL project management page), different releases within a single YYMM series must preserve ABI compatibility.

The ABI compatibility guarantee that a particular project promises can be encoded in the run_exports section of the project conda recipe, by using the pin_subpackages(name, max_pin) macro function. As an example, for a project A which follows the major.minor.patch versioning scheme, setting max_pin to 'X.X' specifies that different releases of project A with the same major.minor version are ABI-compatible. For another package B which is dependent on A, conda will ensure that, when B is installed, an ABI-compatible version of A will be installed alongside B.

For FSL projects which follow the YYMM.B versioning scheme, max_pin should be set to 'X', which indicates that different releases with the same YYMM prefix are ABI-compatible.

So if you are writing a conda recipe for a C/C++ project which provides shared library files that will be used by other projects, add a run_exports section to the recipe like so:

Add run_exports to the build section, and set max_pin according to the versioning scheme used for the project, like so:

build:
  number: {{ build }}
  run_exports:
    strong:
      - {{ pin_subpackage(name, max_pin='x') }}

Add run_exports to the requirements section, and set upper_bound according to the versioning scheme used for the project, like so:

requirements:
  run_exports:
    strong:
      - {{ pin_subpackage(name, upper_bound='x') }}

noarch and script (Python based projects only)

Conda recipes for Python projects are built slightly differently to native projects, as the Python package build machinery is integrated into the conda build process. For Python projects, the build command is usually specified within the build section of the meta.yaml/recipe.yaml file. Furthermore, if you are packging a pure Python project, with no natively compiled code or extensions, you must label your recipe as being of type noarch: python - this is also specified within the build section. So a typical build section for a Python project will resemble the following:

build:
  number: {{ build }}
  noarch: python
  script: {{ PYTHON }} -m pip install . --no-deps --ignore-installed --no-cache-dir -vvv
build:
  number: ${{ build }}
  noarch: python
  script: ${{ PYTHON }} -m pip install . --no-deps --ignore-installed --no-cache-dir -vvv

Requirements

The fsl/base project provides the fundamental elements of a FSL installation, including FSL initialisation scripts and the Makefile machinery. As such it must be installed in order to build Makefile-based FSL projects, and must be present at run time for most FSL commands to function. All FSL conda recipes must therefore list fsl-base as a host requirement, which will cause it to be automatically added as both a build and run requirement1.

C/C++ projects will also need to have a C++ compiler installed at build time. For example:

requirements:
  host:
    - fsl-base
  build:
    - {{ compiler('cxx') }}
    - make
requirements:
  host:
    - fsl-base
  build:
    - ${{ compiler('cxx') }}
    - make

1 This is because the fsl-base recipe uses the run_exports trick described above.

Tests

Some FSL projects contain unit tests - these can be automatically executed as part of the conda package build process. For example, the znzlib project contains tests that can be executed with pyfeeds (the FSL testing framework). These can be executed during the package build by adding the following:

Under the top-level test section:

test:
  # Command(s) to run the tests
  commands:
    - export FSLDIR=${CONDA_PREFIX}
    - pyfeeds run -v -k -p .

  # Files/directories to copy from the project repository
  source_files:
    - test

  # Test dependencies
  requires:
    - fsl-pyfeeds
    - fslpy
    - numpy
    - bzip2
    - zlib
    - zstd

Under the top-level tests section:

tests:
    # Command(s) to run the tests
  - script:
      - export FSLDIR=${CONDA_PREFIX}
      - pyfeeds run -v -k -p .

    # Files/directories to copy from the project repository
    files:
      source:
        - test

    # Test dependencies
    requirements:
      run:
        - fsl-pyfeeds
        - fslpy
        - numpy
        - bzip2
        - zlib
        - zstd

Defining the build proceess

If you are writing a recipe for a Makefile-based FSL project, you need to write a build.sh script which uses the FSL Makefile machinery to build your project. A typical build.sh script will resemble the following:

#!/usr/bin/env bash

# $PREFIX is the installation destination
# when a conda package is being built
export FSLDIR=${PREFIX}

# Configure the build environment
. ${FSLDIR}/etc/fslconf/fsl-devel.sh

# Inject FSL copyright boilerplate
# into project source code
make insertcopyright

# Install project source code into
# ${PREFIX}/src/ (a.k.a. ${FSLDIR}/src/)
mkdir -p     ${PREFIX}/src/
cp -r $(pwd) ${PREFIX}/src/${PKG_NAME}
rm -rf       ${PREFIX}/src/${PKG_NAME}/.git

# Build and install the project
make
make install

post-link.sh and pre-unlink.sh scripts.

Official/full FSL installations contain entry points/wrapper scripts for every FSL executable in ${FSLDIR}/share/fsl/bin/ - this is so that a user can add FSL commands to their ${PATH} via this directory, rather than the ${FSLDIR}/bin/ directory, and avoid adding all of the other executables in ${FSLDIR}/bin/ to their ${PATH} (e.g. python).

These wrapper scripts are created/removed by two utility commands which are installed by the fslinstaller.py script - createFSLWrapper and removeFSLWrapper. The conda recipes for any FSL projects which provide executables need to call these scripts at the time of installation/uninstallation to ensure that wrapper scripts for the executables are created/removed.

This can be achieved by using post-link.sh and pre-unlink.sh scripts, which are conda-specific mechanisms allowing custom logic to be executed when a conda package is installed or uninstalled.

For a FSL project which provides executables called fsl_command1, fsl_command2 and fsl_command3, the post-link.sh for the project recipe should contain:

if [ -e ${FSLDIR}/share/fsl/sbin/createFSLWrapper ]; then
    ${FSLDIR}/share/fsl/sbin/createFSLWrapper fsl_command1 fsl_command2 fsl_command3
fi

The corresponding pre-unlink.sh script should contain:

if [ -e ${FSLDIR}/share/fsl/sbin/removeFSLWrapper ]; then
    ${FSLDIR}/share/fsl/sbin/removeFSLWrapper fsl_command1 fsl_command2 fsl_command3
fi

Notes on CUDA projects

Some FSL projects (e.g. eddy, fdt, mmorf, ptx2) provide both executables that run on a CPU, and executables that require a CUDA-capable GPU. These projects are published as two separate conda packages; for example, for the fdt project:

  • The fsl-fdt package provides CPU executables, libraries, and scripts.
  • The fsl-fdt-cuda package provides GPU executables.

A critical requirement of "hybrid" FSL CUDA projects, which provide both CPU-only and GPU-capable executables, is that the project Makefile must be able to conditionally compile only the CPU components, or the GPU components, provided by the project.

The specific make invocations that need to be made depend on how the project Makefile is written. For example, the fdt Makefile accepts cpu and gpu flags, to control which parts of the project are compiled - make cpu=1 will compile only the CPU components of the fdt project, whereas make cpu=0 gpu=1 will compile only the GPU components.

The build.sh scripts for the fsl-fdt and fsl-fdt-cuda take advantage of this mechanism, so that the fsl-fdt recipe will only compile the CPU components, and the fsl-fdt-cuda recipe will only compile the GPU components

The FSL Makefile-based build system allows CUDA binaries to either statically or dynamically link against the CUDA Toolkit. Static linking is the default behaviour - this allows binaries to be used on an end user's system without requiring the CUDA Toolkit to be installed. However, dynamic linking may be preferable for local development - it can be enabled by passing the CUDA_DYNAMIC=1 variable to the make command.

The build.sh script for the fsl-fdt recipe therefore looks the same as the template build.sh script above, except the make invocations are as follows:

make cpu=1
make cpu=1 install

And the make invocations in the build.sh script for the fsl-fdt-cuda recipe are as follows:

make cpu=0 gpu=1
make cpu=0 gpu=1 install

Conda recipes for CUDA projects

When a CUDA package is built, the CUDA toolkit and nvcc compiler must be installed in the build environment independently of conda. For the official FSL packages, a Docker image with the CUDA toolkit pre-installed, is used to build CUDA packages.

However, the C++ compiler should be installed via conda. One complication which needs to be addressed is that different versions of nvcc are compatible with different versions of gcc, so the version of gcc that should be installed depends on the version of CUDA against which the package is being built.

The meta.yaml/recipe.yaml file for a CUDA project therefore needs to ensure that the correct version of gcc is installed at build time. The FSL build system sets an environment variable called CUDA_GCC_VERSION which contains a suitable gcc version - this can be used by recipes like so:

requirements:
  build:
    - {{ compiler("cxx") }} {{ os.environ.get("CUDA_GCC_VERSION", "*") }}
requirements:
  build:
    - ${{ compiler("cxx") }} ${{ env.get("CUDA_GCC_VERSION", default="*") }}

The FSL build system also sets an environment variable called NVCC, which contains the path to the nvcc CUDA compiler. This environment variable needs to be passed through to the build.sh script, and therefore should be added to a set of white-listed environment variables in the build section, i.e.:

build:
  script_env:
    - NVCC
build:
  script:
    env:
      NVCC: ${{ env.get("NVCC") }}

Building a FSL conda package locally

This section describes how to build a conda package for an FSL project from the corresponding conda recipe.

Normally there should be no need to build FSL conda packages by hand, as packages are automatically built and published using Gitlab CI. However the need may arise for development, testing, or debugging purposes, and hence the process is described here.

In order to build a FSL conda package locally, all you need is a (mini)conda environment with conda-build or rattler-build installed. You do not need to have a compiler, or even FSL, installed. If you are working with internal-only FSL conda packages, you may also need a username and password to access the internal FSL conda channel.

Note: If you are building a CUDA package, you need to have a CUDA Toolkit installed, and the nvcc executable available on your ${PATH}.

Step 1: Clone the recipe repository

To start, you need to create a local clone of the recipe repository. For example, if you would like to build a package for fsl/avwutils:

git clone https://git.fmrib.ox.ac.uk/fsl/conda/fsl-avwuitls

Step 2: Clone the project repository (optional)

By default, the fsl-avwutils recipe will build a package from the fsl/avwutils gitlab repository. If you would like to build a package from a local clone, or a personal fork, of fsl/avwutils, you can use the FSLCONDA_REPOSITORY variable to override the default setting (which is to build from the gitlab fsl/avwutils project):

git clone https://git.fmrib.ox.ac.uk/fsl/avwuitls.git
export FSLCONDA_REPOSITORY=$(pwd)/avwutils

Note: It is not currently possible to build a conda package from a local copy of a project which is not a git repository, nor from a dirty working tree (i.e. the changes you want to build must be committed). This functionality may be added in the future if it is deemed necessary.

Step 3: Choose the revision you want to build (optional)

By default, the fsl-avwutils conda recipe will build a package from the last released version (tag) on the fsl/avwutils gitlab repository - this is specified in in the recipe meta.yaml. If you would like to build a package for a different release, or for a specific branch, you can set the FSLCONDA_REVISION variable to override the default setting. For example, if you want to build a package for the main branch:

export FSLCONDA_REVISION=main

Step 4: Run conda build or rattler-build to build the package

Now you can run conda build or rattler-build to build the conda package. The channel order is very important, and must be:

  1. The public FSL conda channel has the highest priority.
  2. conda-forge Packages from conda-forge have lower priority.
export FSL_CHANNEL=https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/releases/public/
conda build            \
  -c ${FSL_CHANNEL}    \
  -c conda-forge       \
  --output-folder=dist \
  ./fsl-avwutils
export FSL_CHANNEL=https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/releases/public/
rattler-build       \
  -c ${FSL_CHANNEL} \
  -c conda-forge    \
  --output-dir dist \
  --recipe ./fsl-avwutils

If your project depends on internal-only FSL packages, add the internal FSL conda channel before the public channel, like so (set the ${FSLCONDA_USERNAME} and ${FSLCONDA_PASSWORD} environment variables to the username/password for the internal FSL conda channel):

export FSL_CHANNEL=https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/public/
export FSL_INT_CHANNEL=https://${FSLCONDA_USERNAME}:${FSLCONDA_PASSWORD}@fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/internal/
conda build             \
  -c ${FSL_INT_CHANNEL} \
  -c ${FSL_CHANNEL}     \
  -c conda-forge        \
  --output-folder=dist  \
  ./fsl-avwutils
export FSL_CHANNEL=https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/public/
export FSL_INT_CHANNEL=https://${FSLCONDA_USERNAME}:${FSLCONDA_PASSWORD}@fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/internal/
rattler-build           \
  -c ${FSL_INT_CHANNEL} \
  -c ${FSL_CHANNEL}     \
  -c conda-forge        \
  --output-dir dist     \
  --recipe ./fsl-avwutils

Similarly, if you wish to build your package using development versions of other FSL conda packages, specify the FSL development conda channel before the public channel, e.g.:

export FSL_CHANNEL=https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/public/
export FSL_DEV_CHANNEL=https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/development/
conda build             \
  -c ${FSL_DEV_CHANNEL} \
  -c ${FSL_CHANNEL}     \
  -c conda-forge        \
  --output-folder=dist  \
  ./fsl-avwutils
export FSL_CHANNEL=https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/public/
export FSL_DEV_CHANNEL=https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/development/
rattler-build           \
  -c ${FSL_DEV_CHANNEL} \
  -c ${FSL_CHANNEL}     \
  -c conda-forge        \
  --output-dir dist     \
  --recipe ./fsl-avwutils

If the build succeeds, the built package will be saved to the dist directory - binary packages will be saved in either the osx-64/osx-arm64/linux-64/linux-aarch64 sub-directories, and platform-independent (e.g. python) packages will be saved in the noarch sub-directory.

Building multiple FSL conda packages locally

The need may arise to build conda packages for multiple FSL projects at once. For example, you might be making changes to one project which depends on changes you have made to another project (a dependency of the first project). In this case it makes sense to set up a local conda channel, and to build/install the dependencies into that channel.

Let's say we are working on the randomise project, and are simultaneously making changes to the newimage project. We will need local clones of all the project and recipe repositories:

git clone https://git.fmrib.ox.ac.uk/fsl/randomise.git
git clone https://git.fmrib.ox.ac.uk/fsl/newimage.git
git clone https://git.fmrib.ox.ac.uk/fsl/conda/fsl-randomise.git
git clone https://git.fmrib.ox.ac.uk/fsl/conda/fsl-newimage.git

We also need to create a directory which will be used as conda channel for our local builds:

mkdir my_local_conda_channel

export LOCAL_CHANNEL=file://$(pwd)/my_local_conda_channel
export FSL_CHANNEL=https://fsl.fmrib.ox.ac.uk/fsldownloads/fslconda/public/

Before we can build a conda package for randomise, we need to build a conda package from our local development version of newimage. We use FSLCONDA_REPOSITORY and FSLCONDA_REVISION to direct the build to use our local newimage repository, and direct the build to our local channel directory:

export FSLCONDA_REPOSITORY=$(pwd)/newimage
export FSLCONDA_REVISION=enh/my_local_newimage_development_branch

conda build                        \
  -c ${FSL_CHANNEL}                \
  -c conda-forge                   \
  --output-folder=${LOCAL_CHANNEL} \
  ./fsl-newimage
export FSLCONDA_REPOSITORY=$(pwd)/newimage
export FSLCONDA_REVISION=enh/my_local_newimage_development_branch

rattler-build                   \
  -c ${FSL_CHANNEL}             \
  -c conda-forge                \
  --output-dir ${LOCAL_CHANNEL} \
  --recipe ./fsl-newimage

Now we can build our development version of randomise, using the development conda package we just built for newimage, simply by adding our local channel directory as a conda channel, making sure to list it before the FMRIB conda channel URL, so that it takes precedence:

export FSLCONDA_REPOSITORY=$(pwd)/randomise
export FSLCONDA_REVISION=enh/my_local_randomimse_development_branch

conda build                        \
  -c ${LOCAL_CHANNEL}              \
  -c ${FSL_CHANNEL}                \
  -c conda-forge                   \
  --output-folder=${LOCAL_CHANNEL} \
  ./fsl-randomise
export FSLCONDA_REPOSITORY=$(pwd)/randomise
export FSLCONDA_REVISION=enh/my_local_randomimse_development_branch

rattler-build                   \
  -c ${LOCAL_CHANNEL}           \
  -c ${FSL_CHANNEL}             \
  -c conda-forge                \
  --output-dir ${LOCAL_CHANNEL} \
  --recipe ./fsl-randomise