Move data

HPC@PoliTO filesystems

The High Performance Computing infrastructure is supported by two complementary storage systems:

$HOME: a high-capacity Network Attached Storage (NAS) to store user homes and data; data are permanent (cancelled 6 months after account expiration). Default quota = 1.5 TB per user.
$SCRATCH: a temporary, high performance and lower-capacity BeeGFS parallel filesystem that provides a space for IO intensive workloads; files older than 30 days are automatically cancelled. Default quota = 5 TB per user.

Copy files to your HPC@PoliTO home ($HOME)

To copy files to and from the Legion cluster we suggest to use the scp protocol. In order to copy a file or a folder from your home folder on Legion to your local computer use the following command:

scp -r username@hpc-legionlogin.polito.it:/home/username/file /path/local/

where /home/username is your account home (also addressable as $HOME), file is the actual file (or folder) that you want to copy and /path/local/ is the folder where you want to copy the file on your local computer.

Copy files from your HPC@PoliTO home ($HOME)

To copy files to and from the Legion cluster we suggest to use the scp protocol. In order to copy a file or a folder from your local computer to your home folder on Legion use the following command:

scp -r /path/local/file username@hpc-legionlogin.polito.it:/home/username/folder

where username is your account username, /path/local/file is the file (or folder) that you want to copy and /home/username/folder is where you want to copy the file on the cluster.

Use the BeeGFS high performance filesystem ($SCRATCH)

$SCRATCH is a temporary, high performance, storage space that users can access to run their jobs (further info about the $SCRATCH filesystem and BeeGFS on our manual). In order to exploit the higher I/O performances of the BeeGFS filesystem ($SCRATCH space) you should first move your files (e.g.: datasets) from your home folder to a dedicated folder on $SCRATCH. Your jobs will typically access these data and produce new files containing your simulation results. When your simulations are done you should copy back relevant data (typically only the needed ones, avoiding temporary files) to your home for longer and safer storage. You can simply copy data (files and folders) from $HOME to $SCRATCH using cp as:

cp -r $HOME/job_folder/file $SCRATCH/job_folder/

and viceversa from $SCRATCH to $HOME as:

cp -r $SCRATCH/job_folder/file $HOME/job_folder/

or using rsync if you just need to copy newly produced data back to home. In the manual it is also described how to embed data movement from $HOME to $SCRATCH and back directly within a SLURM submission script.