ENH: Use BIDS URIs to track Sources in sidecars

[tsalo]

Closes #3252. This is just a first pass- it only modifies the RawSources for now. I can add the immediate Sources in another PR (this'll require a lot more effort).

Changes proposed in this pull request

• Add DatasetLinks field to the dataset_description.json
  • The input dataset is called raw
  • Any other datasets supplied through --derivatives are automatically named deriv-<index>. We might want to support named derivatives at some point.
• Replace BIDS-relative paths in RawSources fields with BIDS-URIs.
• Change RawSources to Sources in the sidecar files, since RawSources is deprecated.

Documentation that should be reviewed

I'll probably need to update the documentation, but haven't yet.

[codecov]

Codecov Report

Attention: Patch coverage is 93.33333% with 6 lines in your changes are missing coverage. Please review.

Project coverage is 73.60%. Comparing base (c46c893) to head (7e73f04).
Report is 1 commits behind head on master.

Files	Patch %	Lines
fmriprep/config.py	80.00%	2 Missing ⚠️
fmriprep/utils/bids.py	88.88%	2 Missing ⚠️
fmriprep/cli/run.py	0.00%	1 Missing ⚠️
fmriprep/interfaces/bids.py	96.77%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3255      +/-   ##
==========================================
+ Coverage   73.24%   73.60%   +0.36%     
==========================================
  Files          54       56       +2     
  Lines        4164     4240      +76     
==========================================
+ Hits         3050     3121      +71     
- Misses       1114     1119       +5     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[tsalo]

I'm stumped. I can't figure out why config.execution.dataset_links isn't working.

[mgxd]

hey @tsalo - took a quick look through to try to help get you unstuck

[tsalo]

It's working! Thanks @mgxd!

I have a few ideas for next steps:

1. Unit test of BIDSURI, as you recommended.
2. Modify BIDSURI to accept a numinputs parameter and populate in[i] inputs, similar to Nipype's Merge interface.
   • This way, we can provide filenames from different sources without having to merge them.
3. Return DerivativesDataSink out_files from some output workflows to pass along to others.
   • For example, take the output file from ds_bold_native_wf and pass it along to ds_volumes_wf.
   • Actually tracking down each proximal source is probably going to take a bit of effort.

[tsalo]

I have the unit test implemented, as well as a more flexible BIDSURI class. I think tracking immediate source files should happen in a future PR.

[mgxd]

Nice, looks pretty good - just one question on the BIDURI implementation

[effigies]

From the artifacts, full-run/sub-01/func/sub-01_task-mixedgamblestask_run-01_desc-coreg_boldref.json has:

{
  "Sources": [
    "/scratch/fmriprep_24_0_wf/sub_01_wf/bold_task_mixedgamblestask_run_01_wf/bold_fit_wf/hmc_boldref_wf/gen_avg/sub-01_task-mixedgamblestask_run-01_bold_average.nii.gz"
  ]
}

Possibly we were already screwing this up, but this seems worth fixing.

[tsalo]

@effigies I took a crack at it. I'll fix any bugs that come up in the CI.

[tsalo]

Aha!

{
  "Sources": [
    "bids::sub-01/func/sub-01_task-mixedgamblestask_run-01_desc-hmc_boldref.nii.gz"
  ]
}

[effigies]

One last one:

sub-02/func/sub-02_task-cuedSGT_run-01_desc-brain_mask.json

{
  "Sources": [
    "bids:raw:sub-02/func/sub-02_task-cuedSGT_run-01_echo-1_bold.nii.gz",
    "bids:raw:sub-02/func/sub-02_task-cuedSGT_run-01_echo-2_bold.nii.gz",
    "bids:raw:sub-02/func/sub-02_task-cuedSGT_run-01_echo-3_bold.nii.gz",
    "/scratch/fmriprep_24_0_wf/sub_02_wf/bold_task_cuedSGT_run_01_echo_1_wf/bold_fit_wf/bold_hmc_wf/fsl2itk/mat2itk.txt",
    "bids::sub-02/func/sub-02_task-cuedSGT_run-01_from-boldref_to-auto00000_mode-image_xfm.txt"
  ]
}

[effigies]

This is awesome. Thanks so much for taking the time to work on this.

[tsalo]

I can't figure out what's going on with ds005. It's failing but I'm not seeing any crash files in the artifacts.

[effigies]

Fun:

Traceback (most recent call last):
  File "/opt/conda/envs/fmriprep/lib/python3.11/multiprocessing/forkserver.py", line 274, in main
    code = _serve_one(child_r, fds,
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/multiprocessing/forkserver.py", line 313, in _serve_one
    code = spawn._main(child_r, parent_sentinel)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/multiprocessing/spawn.py", line 131, in _main
    prepare(preparation_data)
  File "/opt/conda/envs/fmriprep/lib/python3.11/multiprocessing/spawn.py", line 246, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/opt/conda/envs/fmriprep/lib/python3.11/multiprocessing/spawn.py", line 297, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/opt/conda/envs/fmriprep/bin/fmriprep", line 5, in <module>
    from fmriprep.cli.run import main
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/fmriprep/cli/run.py", line 25, in <module>
    from .. import config
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/fmriprep/config.py", line 378, in <module>
    class execution(_Config):
  File "/opt/conda/envs/fmriprep/lib/python3.11/site-packages/fmriprep/config.py", line 442, in execution
    work_dir = Path('work').absolute()
               ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/pathlib.py", line 979, in absolute
    return self._from_parts([self.cwd()] + self._parts)
                             ^^^^^^^^^^
  File "/opt/conda/envs/fmriprep/lib/python3.11/pathlib.py", line 907, in cwd
    return cls(os.getcwd())
               ^^^^^^^^^^^
FileNotFoundError: [Errno 2] No such file or directory

Let's hope this is just a weird one-off. Rerunning.

[effigies]

I think the error is actually #3256. Not sure why the error text is hidden, but nothing in this PR seems to be causing any error.