Creating a manifest from data directories on disk for a study with different imaging datatypes

In this example, we have a longitudinal study with imaging visits, but not all participants have the same imaging datatypes for all visits.

We do not have a tabular file indicating which datatypes are available for which participants and visits. However, this information can be obtained by looking at the data directories on disk:

data/
├── ABC001/
│   ├── BL/
│   │   ├── T1w/
│   │   │   └── ...
│   │   └── diffusion/
│   │       └── ...
│   └── M12/
│       └── T1w/
│           └── ...
└── ABC002/
    ├── BL/
    │   ├── T1w/
    │   │   └── ...
    │   └── diffusion/
    │       └── ...
    └── M12/
        ├── T1w/
        │   └── ...
        └── diffusion/
            └── ...

Here is a script that creates a Nipoppy manifest for the directory structure above:

Attention

The script below was written for Python 3.11 with pandas 2.2.3. It may not work with older/different versions.

#!/usr/bin/env python
"""Manifest-generation script for Example 3."""

from pathlib import Path

import pandas as pd

if __name__ == "__main__":
    # get the path to the data directory
    # we assume that it is in the same directory as this script
    path_data = Path(__file__).parent / "data"

    data_for_manifest = []
    for path_participant in sorted(path_data.iterdir()):
        for path_participant_visit in sorted(path_participant.iterdir()):
            # participant_id and visit_id are the names of the directories
            participant_id = path_participant.name
            visit_id = path_participant_visit.name

            # use the visit_id as session_id
            session_id = visit_id

            # check which datatypes are present
            datatype = []
            if (path_participant_visit / "T1w").exists():
                datatype.append("anat")
            if (path_participant_visit / "diffusion").exists():
                datatype.append("dwi")

            # create the manifest entry
            data_for_manifest.append(
                {
                    "participant_id": participant_id,
                    "visit_id": visit_id,
                    "session_id": session_id,
                    "datatype": datatype,
                }
            )

    df_manifest = pd.DataFrame(data_for_manifest)

    # write the manifest in the same directory as this script
    df_manifest.to_csv(
        Path(__file__).parent / "example3-manifest.tsv", sep="\t", index=False
    )

Running this script creates a manifest that looks like this:

participant_id	visit_id	session_id	datatype
ABC001	BL	BL	[‘anat’, ‘dwi’]
ABC001	M12	M12	[‘anat’]
ABC002	BL	BL	[‘anat’, ‘dwi’]
ABC002	M12	M12	[‘anat’, ‘dwi’]