Adding a pipeline to Nipoppy

This guide explains how to generate and customize a pipeline configuration for use with Nipoppy.

Create a Boutiques descriptor file

Nipoppy pipeline execution relies on Boutiques. We recommend creating (or finding) a descriptor for the desired pipeline as the first step for adding a new pipeline to Nipoppy.

A Boutiques descriptor is a JSON file that describes:

  • the command-line interface of your tool ("command-line"),

  • required inputs and their types ("inputs"),

  • outputs and how to find them ("output-files"),

  • and execution environment (e.g., Docker/Singularity container via "container-image").

Example: descriptor for FSL SIENA

  1{
  2    "author": "Oxford Centre for Functional MRI of the Brain (FMRIB)",
  3    "name": "FSL SIENA",
  4    "description": "SIENA is a package for both single-time-point (\"cross-sectional\") and two-time-point (\"longitudinal\") analysis of brain change, in particular, the estimation of atrophy (volumetric loss of brain tissue)",
  5    "tool-version": "6.0.4",
  6    "schema-version": "0.5",
  7    "command-line": "/opt/fsl-6.0.4/bin/siena [INPUT1] [INPUT2] [OUTPUT_DIR] [DEBUG] [BET_OPTS] [TWO_CLASS_SEGMENTATION] [T2_WEIGHTED] [STANDARD_SPACE_MASKING] [IGNORE_UPWARDS] [IGNORE_DOWNWARDS] [SIENA_DIFF_OPTS] [VIENA] [VENTRICLE_MASK_IMAGE]",
  8    "container-image": {
  9        "image": "mathdugre/fsl:6.0.4",
 10        "index": "docker://",
 11        "type": "singularity"
 12    },
 13    "inputs": [
 14        {
 15            "id": "input1",
 16            "name": "First T1 weighted image",
 17            "optional": false,
 18            "type": "String",
 19            "value-key": "[INPUT1]"
 20        },
 21        {
 22            "id": "input2",
 23            "name": "Second T1 weighted image",
 24            "optional": false,
 25            "type": "String",
 26            "value-key": "[INPUT2]"
 27        },
 28        {
 29            "command-line-flag": "-o",
 30            "id": "output_dir",
 31            "name": "set output directory (default output is <input1>_to_<input2>_siena)",
 32            "optional": true,
 33            "type": "String",
 34            "value-key": "[OUTPUT_DIR]"
 35        },
 36        {
 37            "command-line-flag": "-d",
 38            "id": "debug",
 39            "name": "debug (don't delete intermediate files)",
 40            "optional": true,
 41            "type": "Flag",
 42            "value-key": "[DEBUG]"
 43        },
 44        {
 45            "command-line-flag": "-B",
 46            "id": "bet_opts",
 47            "name": "options to pass to BET brain extraction (inside double-quotes), e.g. -B \"-f 0.3\"",
 48            "optional": true,
 49            "type": "String",
 50            "value-key": "[BET_OPTS]"
 51        },
 52        {
 53            "command-line-flag": "-2",
 54            "id": "two_class_segmentation",
 55            "name": "two-class segmentation (don't segment grey and white matter separately)",
 56            "optional": true,
 57            "type": "Flag",
 58            "value-key": "[TWO_CLASS_SEGMENTATION]"
 59        },
 60        {
 61            "command-line-flag": "-t2",
 62            "id": "t2_weighted",
 63            "name": "T2-weighted input image (default T1-weighted)",
 64            "optional": true,
 65            "type": "Flag",
 66            "value-key": "[T2_WEIGHTED]"
 67        },
 68        {
 69            "command-line-flag": "-m",
 70            "id": "standard_space_masking",
 71            "name": "use standard-space masking as well as BET",
 72            "optional": true,
 73            "type": "Flag",
 74            "value-key": "[STANDARD_SPACE_MASKING]"
 75        },
 76        {
 77            "command-line-flag": "-t",
 78            "id": "ignore_upwards",
 79            "name": "ignore from t (mm) upwards in MNI152/Talairach space",
 80            "optional": true,
 81            "type": "Number",
 82            "value-key": "[IGNORE_UPWARDS]"
 83        },
 84        {
 85            "command-line-flag": "-b",
 86            "id": "ignore_downwards",
 87            "name": "ignore from b (mm) downwards in MNI152/Talairach space (b should probably be negative)",
 88            "optional": true,
 89            "type": "Number",
 90            "value-key": "[IGNORE_DOWNWARDS]"
 91        },
 92        {
 93            "command-line-flag": "-S",
 94            "id": "siena_diff_opts",
 95            "name": "options to pass to siena_diff timepoint differencing (inside double-quotes), e.g. -S \"-s -i 20\"",
 96            "optional": true,
 97            "type": "String",
 98            "value-key": "[SIENA_DIFF_OPTS]"
 99        },
100        {
101            "command-line-flag": "-V",
102            "id": "viena",
103            "name": "run ventricle analysis (VIENA)",
104            "optional": true,
105            "type": "Flag",
106            "value-key": "[VIENA]"
107        },
108        {
109            "command-line-flag": "-v",
110            "id": "ventricle_mask_image",
111            "name": "optional user-supplied ventricle mask (default is /opt/fsl-6.0.4/data/standard/MNI152_T1_2mm_VentricleMask)",
112            "optional": true,
113            "type": "String",
114            "value-key": "[VENTRICLE_MASK_IMAGE]"
115        }
116    ],
117    "output-files": [
118        {
119            "id": "output1",
120            "name": "Output directory",
121            "optional": false,
122            "path-template": "[OUTPUT_DIR]"
123        }
124    ],
125    "tags": {
126        "domain": [
127            "neuroimaging",
128            "mri",
129            "fmri"
130        ],
131        "toolbox": "fsl"
132    }
133}

Helpful commands

bosh is the CLI from Boutiques. It comes packaged with Nipoppy.

Searching for a descriptor:

$ bosh search

Validating a descriptor:

$ bosh validate descriptor.json

Generating an invocation template:

$ bosh example descriptor.json > invocation.json

Simulating a run:

$ bosh exec simulate descriptor.json invocation.json

References

Create the remaining configuration files

Use the nipoppy pipeline create command to generate a sample configuration. Nipoppy supports three pipeline types: bidsification, processing, and extraction.

nipoppy pipeline create \
    --type processing \
    --source-descriptor <EXISTING_DESCRIPTOR_FILE> \
    <OUTPUT_PIPELINE_DIR>

Note

It is also possible to run this command without the --source-descriptor option, but that will require more downstream manual editing since the files will not be pre-populated with information from the descriptor.

This will create the following structure:

$ tree ./pipelines/howto
pipelines/howto
├── config.json
├── descriptor.json
├── hpc.json
├── invocation.json
└── tracker.json

Edit the configuration files

Attention

Nipoppy uses string substitutions to inject dataset-specific information (e.g., paths, participant IDs) into configuration files at runtime. See this page for lists of available substitutions for each type of configuration file.

File common for all pipeline types

config.json

Edit general pipeline metadata and settings. See the schemas for BIDSification, processing and extraction pipelines.

Key fields

  • "NAME" and "VERSION" (if no source descriptor was used)

  • "CONTAINER_CONFIG" -> "BIND_PATHS": paths to volumes to be mounted to the container (format: local_path[:path_inside_container[:mode]])

  • "STEPS" -> "ANALYSIS_LEVEL": to control the looping (if any) performed by Nipoppy

    • "participant_session": iterate over all participant-session pairs (default)

    • "participant”: iterate over participants only

    • "session”: iterate over sessions only

    • "group”: single iteration, for pipelines that do group analysis or handle all looping internally

  • "GENERATE_PYBIDS_DATABASE”: only set to true if the pipeline accepts a PyBIDS database path as input. Nipoppy will then index the raw BIDS data and create a database that is constrained to the participant and/or session being run. The [[NIPOPPY_DPATH_PIPELINE_BIDS_DB]] substitution can be used to inject the path to this database into the invocation file.

    • Indexing can further be controlled by the user via the "PYBIDS_IGNORE_FILE” field

  • "VARIABLES”: for pipelines that require information (typically file/directory paths) for which there is no good default (e.g. path to a configuration file or a FreeSurfer license file). This should be a dictionary with variable names as keys and descriptions as values, e.g., {"REQUIRED_FILE": "This file is for running the pipeline"}

Example for the FSL SIENA pipeline
 1{
 2    "NAME": "fsl_siena",
 3    "VERSION": "6.0.4",
 4    "CONTAINER_INFO": {
 5        "FILE": "[[NIPOPPY_DPATH_CONTAINERS]]/[[PIPELINE_NAME]]_[[PIPELINE_VERSION]].sif",
 6        "URI": "docker://mathdugre/fsl:[[PIPELINE_VERSION]]"
 7    },
 8    "CONTAINER_CONFIG": {
 9        "ENV_VARS": {},
10        "BIND_PATHS": [],
11        "ARGS": []
12    },
13    "STEPS": [
14        {
15            "INVOCATION_FILE": "invocation.json",
16            "DESCRIPTOR_FILE": "descriptor.json",
17            "ANALYSIS_LEVEL": "participant_session",
18            "TRACKER_CONFIG_FILE": "tracker.json",
19            "HPC_CONFIG_FILE": "hpc.json",
20            "GENERATE_PYBIDS_DATABASE": false,
21            "PYBIDS_IGNORE_FILE": null
22        }
23    ],
24    "VARIABLES": {
25        "FILEPATH_T1W_1": "File path of the first T1w image to be processed",
26        "FILEPATH_T1W_2": "File path of the second T1w image to be processed"
27    },
28    "PIPELINE_TYPE": "processing",
29    "SCHEMA_VERSION": "1.0"
30}

Note

[[NIPOPPY_DPATH_CONTAINERS]]: will be replaced dynamically based on the Nipoppy layout.

[[PIPELINE_NAME]] and [[PIPELINE_VERSION]] are replaced dynamically during execution using the value from the "NAME" and "VERSION" fields, respectively.

descriptor.json

Refer to Creating a Boutiques descriptor above if you did not use a source descriptor.

invocation.json

Defines default input arguments to the pipeline. The keys in the invocation file should match the "id" field of input items in the descriptor. At pipeline runtime, the descriptor and invocation files will be used to generate the command to be executed.

1{
2    "input1": "[[FILEPATH_T1W_1]]",
3    "input2": "[[FILEPATH_T1W_2]]"
4}

Note

You can automatically generate an example invocation.json file with all the required keys from the descriptor.

bosh example ./pipelines/howto/descriptor.json > ./pipelines/howto/invocation.json

hpc.json (Optional)

Defines HPC job submission parameters. See Parallelizing pipeline runs on HPC systems for more information on configuring HPC schedulers with Nipoppy.

{
    "ACCOUNT": "[[HPC_ACCOUNT_NAME]]",
    "TIME": "1:00:00",
    "CORES": "1",
    "MEMORY": "16G",
    "ARRAY_CONCURRENCY_LIMIT": ""
}

Files specific to processing pipelines

tracker.json

Defines output tracking paths. See here for more information about pipeline tracking.

{
    "PATHS": [
        "[[NIPOPPY_BIDS_PARTICIPANT_ID]]/[[NIPOPPY_BIDS_SESSION_ID]]/anat/[[NIPOPPY_BIDS_PARTICIPANT_ID]]_[[NIPOPPY_BIDS_SESSION_ID]]*_example.txt"
    ],
    "PARTICIPANT_SESSION_DIR": "[[NIPOPPY_BIDS_PARTICIPANT_ID]]/[[NIPOPPY_BIDS_SESSION_ID]]"
}

Note

Tracking is always done at the participant-session level, no matter the value of "ANALYSIS_LEVEL" in config.json.

The tracked paths are expected to be relative to <NIPOPPY_PROJECT_ROOT>/derivatives/<PIPELINE_NAME>/<PIPELINE_VERSION>/output.

Assuming a participant ID 001 and session ID A, the template strings will resolve to:

  • [[NIPOPPY_PARTICIPANT_ID]]: 001

  • [[NIPOPPY_BIDS_PARTICIPANT_ID]]: sub-001

  • [[NIPOPPY_SESSION_ID]]: A

  • [[NIPOPPY_BIDS_SESSION_ID]]: ses-A

Upload to the Nipoppy pipeline store (optional)

Nipoppy provides an easy way to upload and install community-developed pipelines via Zenodo.

Important

Before uploading a pipeline to the catalog via the Nipoppy CLI, you must generate a Zenodo token.

nipoppy pipeline upload \
  --password-file <PASSWORD_FILE> \
  <PIPELINE_DIR>

Note

To update an existing Zenodo record, use the --zenodo-id flag.

zenodo.json (Optional)

By default, Nipoppy infers the value using the metadata from the user’s Zenodo account.

To provide custom metadata for your Zenodo record, you must specify it in the optional zenodo.json file. This file is not part of the config.json file.

 1{
 2    "title": "",
 3    "description": "",
 4    "creators": [
 5        {
 6            "person_or_org": {
 7                "given_name": "",
 8                "family_name": "",
 9                "type": "personal"
10            }
11        }
12    ]
13}