Converting a dataset to BIDS

Tip

BIDS conversion with Nipoppy can be skipped if you already have BIDSified data.

Organizing imaging data following the Brain Imaging Data Structure standard can greatly facilitate downstream processing and sharing of data. However, BIDS conversion can be a tricky process, especially for retrospective and/or messy datasets. Some manual work and trial-and-error process is usually needed to create an accurate configuration file to map the raw DICOMs (or NIfTIs) to valid BIDS paths.

Currently, the default Nipoppy global configuration file (as created with nipoppy init) allows users to run any of the following BIDS converters:

  • dcm2bids, a user-friendly DICOM (or NIfTI) converter that is configured with a JSON file

  • HeuDiConv, a flexible DICOM converter that is configured with a heuristic Python file

  • BIDScoin, a user-friendly DICOM (or NIfTI) converter with a graphical user interface (GUI) for editing the configuration file

Note

Nipoppy uses the Boutiques framework to run pipelines. Other BIDS converters can be run by adding the appropriate Boutiques descriptor and invocation files default invocation files.

Summary

Prerequisites

  • A Nipoppy dataset with a valid global configuration file and an accurate manifest

  • Organized (but not BIDS) imaging data in <DATASET_ROOT>/sourcedata/imaging/post_reorg/sub-<PARTICIPANT_ID>/ses-<SESSION_ID> directories

If using the default configuration file:

  • The Apptainer (formerly Singularity) container platform installed on your system

    • See here for installation instructions

    • Note: Apptainer is only natively supported on Linux systems

  • The container image file for the pipeline you wish to use

    • This can be downloaded by e.g., running apptainer pull <URI> inside your container directory (see the configuration file for URI). Make sure the file path is the same as what is specified in the configuration file!

Caution

Although it is possible to use Nipoppy without containers by modifying the default invocation files, we highly recommend using containerized pipelines to make your workflow as reproducible as possible. Using containers instead of locally installed software can also help avoid conflicts or unwanted interactions between different software/versions.

Data directories

Directory

Content description

<DATASET_ROOT>/sourcedata/imaging/post_reorg

Input – Raw imaging data (DICOMs or NIfTIs) organized in a way that facilitates BIDS conversion

<DATASET_ROOT>/bids

Output – Raw imaging data (NIfTIs) organized according to the BIDS standard

Commands

Workflow

  1. Nipoppy BIDS conversion runners will loop over all participants/sessions that have data in <DATASET_ROOT>/sourcedata/imaging/post_reorg but do not have BIDS data in <DATASET_ROOT>/bids according to the curation status file

  2. For each participant-session pair:

    1. The pipeline’s invocation will be processed such that template strings related to the participant/session and dataset paths (e.g., [[NIPOPPY_PARTICIPANT_ID]]) are replaced by the appropriate values

    2. The pipeline is launched using Boutiques, which will be combine the processed invocation with the pipeline’s descriptor file to produce and run a command-line expression

    3. The curation status file is updated to indicate that this participant-session pair now has BIDS data

Configuring the BIDS conversion

Most BIDS conversion tools are designed to be run in steps, with some manual work expected between steps to create/edit a file that helps the converter map DICOM header information onto BIDS file names. The default global config splits BIDS conversion pipelines into the following steps:

  • dcm2bids

    • Step prepare: run dcm2bids_helper, which will convert DICOM files to NIfTI files with JSON sidecars and store them in a temporary directory

      • The JSON configuration file is expected to be created based on information in the sidecars

    • Step convert: run the actual conversion using the configuration file

  • heudiconv

    • Step prepare: extract information from DICOM headers that can be used to write/test the heuristic file

    • Step convert: run the actual conversion

  • bidscoin

    • Step prepare: run the bidsmapper, which creates a mapping between input data and output BIDS filenames

    • Step edit: run the bidseditor, a graphical user interface (GUI) for checking/editing the bidsmap created in the previous step (note that by default the bidseditor is launched at the end of a bidsmapper run)

    • Step convert: run the bidscoiner, which will convert the data to BIDS using the bidsmap

    • Note: By default, Nipoppy will run BIDScoin through a container, just like it does for the other BIDS converters. However, since BIDScoin has a GUI window, some additional commands may need to be executed before/after running nipoppy bidsify. See the BIDScoin documentation for more information.

Note

These step names prepare and convert (and edit) are a Nipoppy convention based on the general BIDS conversion process. The BIDS conversion tools themselves do not use these names.

Customizing BIDS pipeline invocations

Understanding Boutiques descriptors and invocations

Boutiques descriptors have an inputs field listing all available parameters for the tool being described. As a simple example, let’s use the following descriptor for a dummy “pipeline”:

{
    "name": "example",
    "description": "An example tool",
    "tool-version": "0.1.0",
    "schema-version": "0.5",
    "command-line": "echo [PARAM1] [PARAM2] [FLAG1]",
    "inputs": [
        {
            "name": "The first parameter",
            "id": "basic_param1",
            "type": "File",
            "optional": true,
            "value-key": "[PARAM1]"
        },
        {
            "name": "The second parameter",
            "id": "basic_param2",
            "type": "String",
            "optional": false,
            "value-key": "[PARAM2]",
            "value-choices": [
                "choice1",
                "choice2"
            ]
        },
        {
            "name": "The first flag",
            "id": "basic_flag1",
            "type": "Flag",
            "optional": true,
            "command-line-flag": "-f",
            "value-key": "[FLAG1]"
        }
    ]
}

Each key in the invocation file should match the id field in an input described in the descriptor file. The descriptor contains information about the input, such as its type (e.g., file, string, flag), whether it is required or not, etc.

Here is a valid invocation file for the above descriptor:

{
    "basic_param1": ".",
    "basic_param2": "choice1",
    "basic_flag1": true
}

If we pass these two files to Boutiques (or rather, bosh, the Boutiques CLI tool), it will combine them into the following command (and run it):

echo . choice1 -f

Hence, Boutiques allows Nipoppy to abstract away pipeline-specific parameters into JSON text files, giving it the flexibility to run many different kinds of pipelines!

See also

See the Boutiques tutorial for a much more comprehensive overview of Boutiques.

The default pipeline invocation files (in <DATASET_ROOT>/pipelines/<PIPELINE_NAME>-<PIPELINE_VERSION>) can be modified by changing existing values or adding new key-value pairs.

Tip

Run the pipeline on a single participant and session with the --simulate flag to check/debug custom invocation files.

Note

To account for invocations needing to be different for different participants and sessions (amongst other things), Nipoppy invocations are actually templates that need to be slightly processed at runtime to replace template strings by actual values. Recognized template strings include:

  • [[NIPOPPY_PARTICIPANT_ID]]: the participant ID without the sub- prefix

  • [[NIPOPPY_SESSION_ID]]: the session ID without the ses- prefix

  • [[NIPOPPY_BIDS_PARTICIPANT_ID]]: the participant ID with the sub- prefix

  • [[NIPOPPY_BIDS_SESSION_ID]]: the session ID with the ses- prefix

  • [[NIPOPPY_<LAYOUT_PROPERTY>]], where <LAYOUT_PROPERTY> is a property in the Nipoppy dataset layout configuration file (all uppercase): any path defined in the Nipoppy dataset layout

Running the BIDS conversion

Using the command-line interface

To convert all participants and sessions in a dataset, run:

$ nipoppy bidsify \
    --dataset <DATASET_ROOT> \
    --pipeline <PIPELINE_NAME> \
    --pipeline-step <PIPELINE_STEP_NAME>

where <PIPELINE_NAME> and <PIPELINE_STEP_NAME> correspond to the pipeline name and the step name as specified in the global configuration file.

Note

If --pipeline-step is not specified, the first step defined in the global configuration file will be used.

The BIDS conversion can also be run on a single participant and/or session at a time:

$ nipoppy bidsify \
    --dataset <DATASET_ROOT> \
    --pipeline <PIPELINE_NAME> \
    --pipeline-step <PIPELINE_STEP_NAME> \
    --participant-id <PARTICIPANT_ID> \
    --session-id <SESSION_ID>

Hint

The --simulate argument will make Nipoppy print out the command to be executed with Boutiques (instead of actually executing it). It can be useful for checking runtime parameters or debugging the invocation file.

See the CLI reference page for more information on additional optional arguments.

Note

Log files for this command will be written to <DATASET_ROOT>/logs/bids_conversion

Using the Python API

from nipoppy.workflows import BidsConversionRunner

# replace by appropriate values
dpath_root = "<DATASET_ROOT>"
pipeline_name = "<PIPELINE_NAME>"
pipeline_step = "<PIPELINE_STEP>"

workflow = BidsConversionRunner(
    dpath_root=dpath_root,
    pipeline_name=pipeline_name,
    pipeline_step=pipeline_step,
)
workflow.run()

See the API reference for nipoppy.workflows.BidsConversionRunner for more information on optional arguments (they correspond to the ones for the CLI).

Next steps

Now that the imaging data is in BIDS, it is time to run image processing pipelines on it!