babs-merge: Merge results and provenance

Command-Line Arguments

babs-merge merges results and provenance from all successfully finished jobs.

usage: babs-merge [-h] --project_root PROJECT_ROOT [--chunk-size CHUNK_SIZE]

Named Arguments

--project_root, --project-root

Absolute path to the root of BABS project. For example, '/path/to/my_BABS_project/'.

--chunk-size, --chunk_size

Number of branches in a chunk when merging at a time. We recommend using default value.

Default: 2000

--trial-run, --trial_run

Whether to run as a trial run which won't push the merge back to output RIA. This option should only be used by developers for testing purpose. Users: please don't turn this on!

Default: False

Example commands

babs-merge \
    --project-root /path/to/my_BABS_project

It's usually not necessary to add the other two arguments (--chunk-size, --trial-run), as those two arguments are mainly for developers to use.

Detailed description

babs-merge will generate a folder called merge_ds to perform merging. This folder is also a DataLad dataset and a sibling of output RIA.

What if babs-merge fails?

The folder merge_ds generated by babs-merge won't be automatically removed by babs-merge, as users may want to check files regarding warnings saved in this folder. If you have fixed the issue based on the error message, and hope to rerun babs-merge, you need to remove folder merge_ds first:

# 1. go to `merge_ds` folder:
cd /path/to/my_BABS_project/merge_ds

# 2. save any new files:
datalad save -m "save any untracked files generated by babs-merge"

# 3. ask DataLad not to track here:
git annex dead here

# 4. remove `merge_ds` DataLad dataset:
cd ..
datalad remove -d merge_ds --reckless availability

What if babs-merge takes a long time?

Usually babs-merge won't take a long time. For example, merging FreeSurfer results (got from fMRIPrep) for about n=2300 successful jobs took around 10-15 minutes. However if you are concerned that babs-merge takes a long time, you can:

  • Either use screen so that even you are disconnected from the cluster login node, babs-merge command keeps running;

  • Or submit babs-merge as a job.