Data management

Data ownership and local storage are managed by DMagic, which retrieves user information from the APS scheduling system and updates the user-info PVs:

tomo_user

Run a user-PV update:

(base) [2bmb@arcturus]$ bash
(base) [2bmb@arcturus]$ conda activate dm
(dm)   [2bmb@arcturus]$ dmagic show
(dm)   [2bmb@arcturus]$ dmagic tag

You can also enter the user last name, email address, and YYYY-MM manually on the MEDM screen if needed.

Upload

To upload data from tomo1:/data to Voyager:

(base) [2bmb@arcturus]$ bash
(base) [2bmb@arcturus]$ source /home/dm_bm/etc/dm.setup.sh
(dm-user) [2bmb@arcturus]$ dm-station-gui

Click Start New:

dm

Then select Add Experiment:

dm

Select the GUP associated with the data set, or choose Continue Manually if there is no GUP (e.g., internal time):

dm

Set the experiment Name, Dates, Type, Storage Root Path, etc., then click Save:

dm

After the experiment is created, specify the data location on tomo1 (e.g. /data/...) in the Data Directory or Single file field:

dm

Delete

To delete a folder of data already uploaded to Voyager:

(base) [2bmb@arcturus]$ source /home/dm_bm/etc/dm.setup.sh
(dm-user) [2bmb@arcturus]$ dm-delete-files --experiment 2023-03-Xu --path-pattern LCO15v18b_232_rec

There are 75 files that match specified criteria. If you continue:

  1) Experiment files will be removed from storage.
  2) Experiment files will be removed from the catalog.

Proceed (yes|no)? [no]: yes

To delete a subset of files, use a pattern such as .*. For example, to delete all TIFF files in the rkd10_024_rec folder:

(dm-user) [2bmb@arcturus]$ dm-delete-files --experiment test-delete-rec --path-pattern rkd10_024_rec/.*.tiff

There are 1852 files that match specified criteria. If you continue:
  1) Experiment files will be removed from storage.
  2) Experiment files will be removed from the catalog.

Proceed (yes|no)? [no]:

To apply this to all folders ending with _rec:

(dm-user) [2bmb@arcturus]$ dm-delete-files --experiment Stock-2020-11 --path-pattern .*_rec/.*.tiff
There are 154085 files that match specified criteria. If you continue:
  1) Experiment files will be removed from storage.
  2) Experiment files will be removed from the catalog.

The dm-delete-files command is recursive. For example, to delete all *.tiff files in all subfolders of an experiment:

(dm-user) [2bmb@arcturus]$ dm-delete-files --experiment Parejiya-2022-04 --path-pattern /.*.tiff

Or to delete all recon_*.tiff files in all subfolders:

(dm-user) [2bmb@arcturus]$ dm-delete-files --experiment Drummond-2022-09 --path-pattern /recon_.*.tiff

Download

To download an experiment (e.g. Finfrock-2022-12) from Voyager to tomodata1:/data/:

(base)   [2bmb@tomo1 ~]$ source /home/dm_bm/etc/dm.setup.sh
(dm-user) [2bmb@tomo1 ~]$ dm-download --experiment Finfrock-2022-12 --destination-directory /data/

For additional details see the DM instructions:

Tape restore

To restore an experiment (e.g. Finfrock-2023-03) from tape to its original location on Voyager:

dm

Run:

(base)   [2bmb@arcturus]$ source /home/dm_bm/etc/dm.setup.sh
(dm-user) [2bmb@arcturus]$ dm-restore-experiment --experiment Finfrock-2023-03
id=0293f99b-c724-402f-af94-1f2606499d96 name=restoreArchive experimentName=Finfrock-2023-03 status=pending

Check the status of the restore process with:

(dm-user) [2bmb@arcturus]$ dm-restore-experiment --experiment Finfrock-2023-03
Unfinished archive task with id 0293f99b-c724-402f-af94-1f2606499d96 already exists for experiment Finfrock-2023-03

or:

(dm-user) [2bmb@arcturus]$ dm-get-archive-task --id 0293f99b-c724-402f-af94-1f2606499d96 --display-keys ALL

To list all tasks:

(dm-user) [2bmb@arcturus]$ dm-list-archive-tasks

Tape archive

To archive an experiment (e.g. Finfrock-2023-03) from Voyager to tape:

(base)   [2bmb@arcturus]$ source /home/dm_bm/etc/dm.setup.sh
(dm-user) [2bmb@arcturus]$ dm-archive-experiment --experiment Finfrock-2023-03

Pin / extend archive-to-tape date

If you receive a notification email like:

Experiment Lu-2023-03 can be archived in 1 days, 22 hours (168205.33 seconds).

and want to extend the archival to tape date, run:

(base)   [2bmb@arcturus]$ source /home/dm_bm/etc/dm.setup.sh
(dm-user) [2bmb@arcturus]$ dm-pin-experiment --experiment Lu-2023-03 --n-days 90
id=12322 name=Lu-2023-03 experimentTypeId=32 experimentStationId=23 updateDate=2026-02-06 13:21:24.826141-06:00 startDate=2023-03-24 00:00:00-05:00 endDate=2023-03-27 00:00:00-05:00

To verify the pinned status of an experiment:

(dm-user) dmadmin@s2bmdm> dm-get-experiment --experiment Lu-2023-03 -a -pp | grep pinned
  'pinnedAtTime': '1771565030.0576963',
  'pinnedAtTimestamp': '2026/02/19 23:23:50',
  'pinnedByUser': 'user2bm',
  'pinnedUntilTime': '1779341030.0576963',
  'pinnedUntilTimestamp': '2026/05/21 00:23:50',

Add users

To grant a user read access to datasets in an experiment:

(base)   [2bmb@arcturus]$ source /home/dm_bm/etc/dm.setup.sh
(dm-user) [2bmb@arcturus]$ dm-add-user-experiment-role --experiment expName --username d<badge> --role User

Find the d<badge> identifier with:

(dm-user) dmadmin@s2bmdm> dm-list-users | grep -i decarlo

Example:

dm-add-user-experiment-role --experiment McDowell-2023-03 --username d240474 --role User
dm-add-user-experiment-role --experiment Finfrock-2022-12 --username d240474 --role User
dm-add-user-experiment-role --experiment Finfrock-2023-03 --username d240474 --role User