Adding a pipeline to Nipoppy

This guide explains how to generate and customize a pipeline configuration for use with Nipoppy.

Create a Boutiques descriptor file

Nipoppy pipeline execution relies on Boutiques. We recommend creating (or finding) a descriptor for the desired pipeline as the first step for adding a new pipeline to Nipoppy.

A Boutiques descriptor is a JSON file that describes:

  • the command-line interface of your tool ("command-line"),

  • required inputs and their types ("inputs"),

  • outputs and how to find them ("output-files"),

  • and execution environment (e.g., Docker/Singularity container via "container-image").

Example: descriptor for FSL SIENA

  1{
  2    "author": "Oxford Centre for Functional MRI of the Brain (FMRIB)",
  3    "name": "FSL SIENA",
  4    "description": "SIENA is a package for both single-time-point (\"cross-sectional\") and two-time-point (\"longitudinal\") analysis of brain change, in particular, the estimation of atrophy (volumetric loss of brain tissue)",
  5    "tool-version": "6.0.4",
  6    "schema-version": "0.5",
  7    "command-line": "/opt/fsl-6.0.4/bin/siena [INPUT1] [INPUT2] [OUTPUT_DIR] [DEBUG] [BET_OPTS] [TWO_CLASS_SEGMENTATION] [T2_WEIGHTED] [STANDARD_SPACE_MASKING] [IGNORE_UPWARDS] [IGNORE_DOWNWARDS] [SIENA_DIFF_OPTS] [VIENA] [VENTRICLE_MASK_IMAGE]",
  8    "container-image": {
  9        "image": "mathdugre/fsl:6.0.4",
 10        "index": "docker://",
 11        "type": "singularity"
 12    },
 13    "inputs": [
 14        {
 15            "id": "input1",
 16            "name": "First T1 weighted image",
 17            "optional": false,
 18            "type": "String",
 19            "value-key": "[INPUT1]"
 20        },
 21        {
 22            "id": "input2",
 23            "name": "Second T1 weighted image",
 24            "optional": false,
 25            "type": "String",
 26            "value-key": "[INPUT2]"
 27        },
 28        {
 29            "command-line-flag": "-o",
 30            "id": "output_dir",
 31            "name": "set output directory (default output is <input1>_to_<input2>_siena)",
 32            "optional": true,
 33            "type": "String",
 34            "value-key": "[OUTPUT_DIR]"
 35        },
 36        {
 37            "command-line-flag": "-d",
 38            "id": "debug",
 39            "name": "debug (don't delete intermediate files)",
 40            "optional": true,
 41            "type": "Flag",
 42            "value-key": "[DEBUG]"
 43        },
 44        {
 45            "command-line-flag": "-B",
 46            "id": "bet_opts",
 47            "name": "options to pass to BET brain extraction (inside double-quotes), e.g. -B \"-f 0.3\"",
 48            "optional": true,
 49            "type": "String",
 50            "value-key": "[BET_OPTS]"
 51        },
 52        {
 53            "command-line-flag": "-2",
 54            "id": "two_class_segmentation",
 55            "name": "two-class segmentation (don't segment grey and white matter separately)",
 56            "optional": true,
 57            "type": "Flag",
 58            "value-key": "[TWO_CLASS_SEGMENTATION]"
 59        },
 60        {
 61            "command-line-flag": "-t2",
 62            "id": "t2_weighted",
 63            "name": "T2-weighted input image (default T1-weighted)",
 64            "optional": true,
 65            "type": "Flag",
 66            "value-key": "[T2_WEIGHTED]"
 67        },
 68        {
 69            "command-line-flag": "-m",
 70            "id": "standard_space_masking",
 71            "name": "use standard-space masking as well as BET",
 72            "optional": true,
 73            "type": "Flag",
 74            "value-key": "[STANDARD_SPACE_MASKING]"
 75        },
 76        {
 77            "command-line-flag": "-t",
 78            "id": "ignore_upwards",
 79            "name": "ignore from t (mm) upwards in MNI152/Talairach space",
 80            "optional": true,
 81            "type": "Number",
 82            "value-key": "[IGNORE_UPWARDS]"
 83        },
 84        {
 85            "command-line-flag": "-b",
 86            "id": "ignore_downwards",
 87            "name": "ignore from b (mm) downwards in MNI152/Talairach space (b should probably be negative)",
 88            "optional": true,
 89            "type": "Number",
 90            "value-key": "[IGNORE_DOWNWARDS]"
 91        },
 92        {
 93            "command-line-flag": "-S",
 94            "id": "siena_diff_opts",
 95            "name": "options to pass to siena_diff timepoint differencing (inside double-quotes), e.g. -S \"-s -i 20\"",
 96            "optional": true,
 97            "type": "String",
 98            "value-key": "[SIENA_DIFF_OPTS]"
 99        },
100        {
101            "command-line-flag": "-V",
102            "id": "viena",
103            "name": "run ventricle analysis (VIENA)",
104            "optional": true,
105            "type": "Flag",
106            "value-key": "[VIENA]"
107        },
108        {
109            "command-line-flag": "-v",
110            "id": "ventricle_mask_image",
111            "name": "optional user-supplied ventricle mask (default is /opt/fsl-6.0.4/data/standard/MNI152_T1_2mm_VentricleMask)",
112            "optional": true,
113            "type": "String",
114            "value-key": "[VENTRICLE_MASK_IMAGE]"
115        }
116    ],
117    "output-files": [
118        {
119            "id": "output1",
120            "name": "Output directory",
121            "optional": false,
122            "path-template": "[OUTPUT_DIR]"
123        }
124    ],
125    "tags": {
126        "domain": [
127            "neuroimaging",
128            "mri",
129            "fmri"
130        ],
131        "toolbox": "fsl"
132    }
133}

Helpful commands

bosh is the CLI from Boutiques—already installed with Nipoppy.

Searching for a descriptor:

$ bosh search

Validating a descriptor:

$ bosh validate descriptor.json

Generating an invocation template:

$ bosh example descriptor.json > invocation.json

Simulating a run:

$ bosh exec simulate descriptor.json invocation.json

References

Create the remaining configuration files

Use the nipoppy pipeline create command to generate a sample configuration. Nipoppy supports three pipeline types: bidsification, processing, and extraction.

nipoppy pipeline create \
    --type processing \
    --source-descriptor <EXISTING_DESCRIPTOR_FILE> \
    <OUTPUT_PIPELINE_DIR>

Note

It is also possible to run this command without the --source-descriptor option, but that will require more downstream manual editing since the files will not be pre-populated with information from the descriptor.

This will create the following structure:

$ tree ./pipelines/howto
pipelines/howto
├── config.json
├── descriptor.json
├── hpc.json
├── invocation.json
└── tracker.json

Edit the configuration files

File common for all pipeline types

config.json

Edit general pipeline metadata and settings. See the schemas for BIDSification, processing and extraction pipelines.

Example for the FSL SIENA pipeline:

 1{
 2    "NAME": "FSL SIENA",
 3    "VERSION": "6.0.4",
 4    "CONTAINER_INFO": {
 5        "FILE": "[[NIPOPPY_DPATH_CONTAINERS]]/[[PIPELINE_NAME]]_[[PIPELINE_VERSION]].sif",
 6        "URI": "docker://mathdugre/fsl:[[PIPELINE_VERSION]]"
 7    },
 8    "CONTAINER_CONFIG": {
 9        "ENV_VARS": {},
10        "ARGS": []
11    },
12    "STEPS": [
13        {
14            "INVOCATION_FILE": "invocation.json",
15            "DESCRIPTOR_FILE": "descriptor.json",
16            "ANALYSIS_LEVEL": "participant_session",
17            "TRACKER_CONFIG_FILE": "tracker.json",
18            "HPC_CONFIG_FILE": "hpc.json",
19            "GENERATE_PYBIDS_DATABASE": false,
20            "PYBIDS_IGNORE_FILE": null
21        }
22    ],
23    "PIPELINE_TYPE": "processing",
24    "SCHEMA_VERSION": "1"
25}

Note

[[NIPOPPY_DPATH_CONTAINERS]]: will be replaced dynamically based on the Nipoppy layout.

[[PIPELINE_NAME]] and [[PIPELINE_VERSION]] are replaced dynamically during execution using the value from the "NAME" and "VERSION" fields, respectively.

Warning

If not using a source descriptor, be sure to update the "NAME" and "VERSION" fields, and the <OWNER> placeholder of the "CONTAINER_INFO"’s "URI". You may need to replace the entire "URI" field if the container name does not follow the <OWNER>/[[PIPELINE_NAME]]:[[PIPELINE_VERSION]] naming convention.

descriptor.json

Refer to Creating a Boutiques descriptor above if you did not use a source descriptor.

invocation.json

Defines default input arguments to the pipeline. The keys in the invocation file should match the "id" field of input items in the descriptor. At pipeline runtime, the descriptor and invocation files will be used to generate the command to be executed.

1{
2    "input1": "str_input1_PV",
3    "input2": "str_input2_Yh"
4}

Note

Optionally, you can regenerate the invocation.json file to match the descriptor arguments.

bosh example ./pipelines/howto/descriptor.json > ./pipelines/howto/invocation.json

hpc.json (Optional)

Defines HPC job submission parameters. See Parallelizing pipeline runs on HPC systems for more information on configuring HPC schedulers with Nipoppy.

{
    "ACCOUNT": "[[HPC_ACCOUNT_NAME]]",
    "TIME": "1:00:00",
    "CORES": "1",
    "MEMORY": "16G",
    "ARRAY_CONCURRENCY_LIMIT": ""
}

Files specific to processing pipelines

tracker.json

Defines output tracking paths. See here for more information about pipeline tracking.

{
    "PATHS": [
        "[[NIPOPPY_BIDS_PARTICIPANT_ID]]/[[NIPOPPY_BIDS_SESSION_ID]]/anat/[[NIPOPPY_BIDS_PARTICIPANT_ID]]_[[NIPOPPY_BIDS_SESSION_ID]]*_example.txt"
    ],
    "PARTICIPANT_SESSION_DIR": "[[NIPOPPY_BIDS_PARTICIPANT_ID]]/[[NIPOPPY_BIDS_SESSION_ID]]"
}

Note

Tracking is always done at the participant-session level, no matter the value of "ANALYSIS_LEVEL" in config.json.

The tracked paths are expected to be relative to <NIPOPPY_PROJECT_ROOT>/derivatives/<PIPELINE_NAME>/<PIPELINE_VERSION>/output.

Assuming a participant ID 001 and session ID A, the template strings will resolve to:

  • [[NIPOPPY_PARTICIPANT_ID]]: 001

  • [[NIPOPPY_BIDS_PARTICIPANT_ID]]: sub-001

  • [[NIPOPPY_SESSION_ID]]: A

  • [[NIPOPPY_BIDS_SESSION_ID]]: ses-A

Upload to the Nipoppy pipeline store (optional)

Nipoppy provides an easy way to upload and install community-developed pipelines via Zenodo.

Important

Before uploading a pipeline to the catalog via the Nipoppy CLI, you must generate a Zenodo token.

nipoppy pipeline upload \
  --password-file <PASSWORD_FILE> \
  <PIPELINE_DIR>

Note

To update an existing Zenodo record, use the --zenodo-id flag.

zenodo.json (Optional)

By default, Nipoppy infers the value using the metadata from the user’s Zenodo account.

To provide custom metadata for your Zenodo record, you must specify it in the optional zenodo.json file. This file is not part of the config.json file.

 1{
 2    "title": "",
 3    "description": "",
 4    "creators": [
 5        {
 6            "person_or_org": {
 7                "given_name": "",
 8                "family_name": "",
 9                "type": "personal"
10            }
11        }
12    ]
13}