Data Management

Data ownership and local storage location is managed by Dmagic by automatically retrieving user information from the APS scheduling system and updating the user info PVs at:

tomo_user

To run a user PV adptate:

[user2bmb@arcturus]$ bash
[user2bmb@arcturus]$ source /home/dm_bm/etc/dm.setup.sh
[user2bmb@arcturus]$ dmagic show
[user2bmb@arcturus]$ dmagic tag

you can also enter manually the user last name/email address/YYYY-MM.

Data management and sharing with users is manged by Globus. The Globus servers available at Argonnne are petrel#tomography and aps#data both accessible from the Globus portal.

You can select with server to use with:

[user2bmb@arcturus]$ bash
[user2bmb@arcturus]$ globus-server-name voyager (or petrel)

To automatically create YYYY-MM/PI_lastName/ on the globus server:

[user2bmb@arcturus]$ bash
[user2bmb@arcturus]$ globus dirs

and you can share this folder will all users listed in the scheduled proposal with:

[user2bmb@arcturus]$ globus email --schedule

To upload data from the beamline to the Globus server

If using petrel#tomography, make a Globus copy between:

[user2bmb@pg10ge]$  /local/data/YYYY-MM/PI_lastName/

and the same YYYY-MM/PI_lastName/ on petrel.

If using the aps#tomography:

[user2bmb@arcturus]$ bash
[user2bmb@arcturus]$ source /home/dm_bm/etc/dm.setup.sh
[user2bmb@arcturus]$ dm-station-gui

For more details see the DM instruction.

Petrel#tomography status is avaialble here.

Example

Upload data from tomo1:/data to voyager:

(base) [user2bmb@handyn]$ bash
(base) [user2bmb@handyn]$ source /home/dm_bm/etc/dm.setup.sh
(base) [user2bmb@handyn]$ dm-station-gui

and presss Start New:

dm

then select Add Experiment

dm

Select the GUP associated with the data set or Continue Manually if there is no GUP (e.g. internal time):

dm

Set the experiment Name, Dates, Type, Storage Root Path etc., then press Save

dm

Once the experiment is created you can enter in the Data Directory or single file path the location of the data on tomo1 as /data/ …

dm

Delete

To delete a folder of data already uploaded to voyager use:

(base) [user2bmb@handyn]$ dm-delete-files --experiment 2023-03-Xu --path-pattern LCO15v18b_232_rec

There are 75 files that match specified criteria. If you continue:

  1) Experiment files will be removed from storage.
  2) Experiment files will be removed from the catalog.

Proceed (yes|no)? [no]: yes

To delete a subset of files use .*, for example to delete all tiff contained in the rkd10_024_rec folder:

(base) [user2bmb@handyn]$ dm-delete-files --experiment test-delete-rec --path-pattern rkd10_024_rec/.*.tiff

There are 1852 files that match specified criteria. If you continue:
  1) Experiment files will be removed from storage.
  2) Experiment files will be removed from the catalog.

Proceed (yes|no)? [no]:

to apply this to all _rec folders:

dm-delete-files --experiment Stock-2020-11 --path-pattern .*_rec/.*.tiff
There are 154085 files that match specified criteria. If you continue:
  1) Experiment files will be removed from storage.
  2) Experiment files will be removed from the catalog.

The dm-delete-files is recursive, so to delete all *.tiff files part of an experiment, in all subfolder:

dm-delete-files --experiment Parejiya-2022-04 --path-pattern /.*.tiff

or to delete all recon_*.tiff files part of an experiment, in all subfolder:

dm-delete-files --experiment Drummond-2022-09 --path-pattern /recon_.*.tiff