Data Transfer¶
Moving files between local computer and remote server can be done via terminal (console) using scp,rsync, and sftp commands, see examples below. Another popular option how to transfer files is to use MobaXterm or WinSCP Windows clients. Examples to those to commands can be found below.
Transfer data between local/remote computer¶
Using terminal (console)¶
These options are available without additional software instalation on Mac OS, Linux and Windows 11 (Powershell). The commands scp,rsync, and sftp must be started from local computer. For more information about these commands type**man scp**, man rsync, or man sftp in terminal (console) or visit web page for scp, rsync, or sftp.
scp commands: basic syntax¶
scp (secure copy protocol) is a network protocol to securely copy files and directories. scp allows you to transfer files/directories between local/remote host or between two remote hosts. If you want to copy files to the directory in which you are currently located, use the dot at the end of the command.
General syntax for scp command
scp [options] [user@local_host]:<source_path> [user@remote_host]:<destination_path>
Copying data files and directories from a local computer to a remote server.
scp -P 5522 <path-to-destination-file-on-local-computer> <username>@login.devana.nscc.sk:<path-to-destination-file-on-cluster>
scp -r -P 5522 <path-to-destination-folder-on-local-computer> <username>@login.devana.nscc.sk:<path-to-destination-folder-on-cluster>
Copying data files and directories from a remote server to a local computer.
scp -P 5522 <username>@login.devana.nscc.sk:<path-to-destination-file-on-devana-computer> <path-to-destination-file-on-local-computer>
scp -r -P 5522 <username>@login.devana.nscc.sk:<path-to-destination-folder-on-devana-computer> <path-to-destination-folder-on-local-computer>
scp -r -P 5522 <username>@login.devana.nscc.sk:<path-to-destination-folder-on-devana-computer> .
rsync commands: basic syntax¶
rsync can be used for mirroring data, copying files between systems, backups and as a replacement for scp, sfpt and cp comands. Its advantage is reduction of amount of sent data by sending only the differences between the source files and existing files in the target destination. If you would like to transfer more than a few small files it is recommended to use rsync command, since there is always a chance of failing connections on both side or occurence of other unspecified issues. Rsync can be use together with SSH protocol via "-e".
General syntax of rsync is:
rsync [options] [user@local_host]:<source_path> [user@remote_host]:<destination_path>
Copy a file from a local computer to a remote server.
To show the progress while transfering the data from remote to local computer use "--progress" option or "-P".
rsync -avhe "ssh -p 5522" <path-to-destination-file-on-local-computer> <username>@login.devana.nscc.sk:<path-to-destination-file-on-devana-computer>
rsync -avhe "ssh -p 5522" --progress <path-to-destination-file-on-local-computer> <username>@login.devana.nscc.sk:<path-to-destination-folder-on-devana-computer>
rsync -avhe "ssh -p 5522" -P <path-to-destination-file-on-local-computer> <username>@login.devana.nscc.sk:<path-to-destination-folder-on-devana-computer>
Copy a file from a remote server to a local computer using SSH.
rsync -avhe "ssh -p 5522" <username>@login.devana.nscc.sk:<path-to-destination-file-on-devana-computer> <path-to-destination-file-on-local-computer>
rsync -avhe "ssh -p 5522" --progress <username>@login.devana.nscc.sk:<path-to-destination-folder-on-devana-computer> <path-to-destination-file-on-local-computer>
rsync -avhe "ssh -p 5522" -P <username>@login.devana.nscc.sk:<path-to-destination-folder-on-devana-computer> <path-to-destination-file-on-local-computer>
rsync -avhPe "ssh -p 5522" <username>@login.devana.nscc.sk:<path-to-destination-folder-on-devana-computer> .
As in case of scp, rsync also provides options to control over variety of other features. The most widely used options are:
-
-a - Archive mode, preserves file permission and does not follows symlinks. The archive mode does not preserve hard links, because finding multiply-linked files is expensive. A -H options must be explicitly specified for hard links.
-
-v - enable verbose mode, display or get extended information.
-
-h - output numbers in a human-readable format.
-
-P or --progress - show progress during transfer.
-
-e - for using ssh protocol. "-e" options has to be place before "ssh -p 5522" the end of the option string, or can be used also separately, rsync -avhe "ssh -p 5522" or rsync -avh -e "ssh -p 5522"
-
-z - enable compression, compress file data during the transfer.
sftp commands: basic syntax¶
sftp (secure file transfer protocol) is an encrypted version of FTP designed for a file transfer over network connection. It performs all operations over an encrypted ssh session using its features, such as publick key authentication and data compression.
- sftp can be run in an interactive session that logs onto specified host, where commands can be written at the prompt.
Launch an interactive session on Devana or in specific directory.
sftp -P 5522 <username>@login.devana.nscc.sk
sftp -P 5522 <username>@login.devana.nscc.sk:<path-to-destination-folder-on-devana-computer>
- sftp can be also used to extract files automatically without any prompted interaction or in a batch mode (running in scripted fashion).
Copy a file from a remote server to a local computer with sftp.
sftp -P 5522 <username>@login.devana.nscc.sk:<path-to-destination-folder-file-on-devana-computer> /<path-to-destination-file-on-local-computer>
sftp -b batchfile -P 5522 <username>@login.devana.nscc.sk:/home/<username> /<path-to-destination-folder-on-local-computer>
where batchfile contains:
cd <path-to-destination-folder-on-devana-computer>
put file*.txt
As in case of scp and rsync, sftp also provides options to control over variety of other features. The most widely used options are:
-
-B buffer_size - Specify the size of the buffer that sftp uses when transferring files. Larger buffers require fewer round trips, but require more memory to do so. The default is 32768 bytes.
-
-b batchfile - Enables batch mode.
-
-C - Enables compression.
-
-i identity_file - Selects the file from which the identity (private key) for public key authentication is read. This option is directly passed to ssh.
-
-P port - Specifies the port to connect to on the remote host.
-
-q - Quiet mode: disables the progress meter and warning and diagnostic messages from ssh.
-
-r - Recursively copy entire directories when uploading and downloading. Note that sftp does not follow symbolic links encountered in the tree traversal.
-- -v - Verbose. This option is also passed to ssh.
Description
- path-to-destination-file-on-local-computer - path to the files what you want to copy from your personal (local) computer to remote server (Devana)
- username - your login username on Devana cluster
Warning
If your ssh key is located outside default folder (/.ssh) you need to specify path to the key by using -i flag
Mounting remote directory locally using SSHFS¶
You can mount a directory from the Devana infrastructure as a file system on your local computer via SSHFS.
On Linux clients install fuse-sshfs (RedHat distributions) or sshfs (Debian distributions).
Mount home directory from Devana locally.
sshfs -p 5522 -oIdentityFile=/<path-to-identity-key> <username>@login.devana.nscc.sk:<path-to-destination-folder-on-devana-computer> /<path-to-destination-folder-on-local-comupter>
After you run this command the directory /<path-to-destination-folder-on-local-computer> should show the contents of <path-to-destination-folder-on-devana-computer> on the HPC. You don’t need to mount your entire home directory, it’s best to just mount the directory that you require access to.
Unmounting the filesystem.
When you have finished working on your files you should unmount the directory, otherwise when you shutdown your PC you could get corrupted files.
To unmount the filesystems issue following command:
fusermount -u /<path-to-destination-folder-on-local-computer>
sudo to run the sshfs command and you might need to add yourself to the "fuse" group."
Download and install the latest stable installer of "WinFSP" from here. Download and install the latest stable 64-bit installer of "SSHFS-Win" from here.
After these are installed you can setup the SSHFS connection as described below.
Open File Explorer, right-click on This PC and choose Map Network Drive. In the Drive field, choose a drive letter that is not already mapped. Already mapped drives will have a location shown after the drive letter. In the Folder field enter:
Mount home directory from Devana with File Explorer.
\\sshfs.k\<username>@login.devana.nscc.sk!5522\<path-to-destination-folder-on-devana-computer>
where <path-to-destination-folder-on-devana-computer> is the directory that you wish to mount.
Info
sshfs.k uses the SSH key stored in <local_user>/.ssh/id_rsa
Do not select "Reconnect at sign-in". You should only mount your HPC home directory when you need to. It should be “disconected” when you no longer need to access it. Click Finish.
Finally when you have finished your work you should “disconnect” your mapped drive from the HPC. From File Explorer, right click the mounted drive and select “Disconnect”.
Download the latest stale releases of "Fuse for macOS" and the "SSHFS" modules from here. Create a folder on your Mac to use as a mount point.
In Terminal, run following command to access your directory on the remote system and mount it at the local folder you created:
Mount home directory from Devana with File Explorer.
sshfs <username>@login.devana.nscc.sk:/<path-to-destination-folder-on-devana-computer> /<path-to-destination-folder-on-local-computer> -i <path-to-identity-key>
Unmounting the filesystem.
When you have finished working on your files you should unmount the directory, otherwise when you shutdown your PC you could get corrupted files.
Windows clients¶
These options are accessible on computers with Windows OS after installation of the specific clients (MobaXterm or WinSCP) and work on the principle of providing GUI so scp/sftp commands explained above.
Info
Following procedures assumes working connection on MobaXterm to target computer, in this case Devana cluster. If you have not set up MobaXterm yet, see section How to connect?
[1] Using MobaXterm, you can easily view and access the remote machine using the window to the left of the terminal. You can see its current location displayed in the top bar and navigate through the machine by clicking on the folders as shown in the screenshot below.
You can also navigate the machine via the command-line in the window to the right and see the changes reflected in the folder to the left if you check the Follow terminal folder box.
[2] To download a file on to your local machine, first locate and select the file on the remote machine that you wish to download using one of the methods explained in the above step. (To download a file or folder is the same process)
[3] Next, click on the blue download arrow. A window displaying your local machine will pop up prompting you to select the location where you wish to download the files. After doing so, click the OK button to confirm the download.
The newest version of WinSCP client can be downloaded and installed from here.
[1] Connect to Devana cluster with the authentication key.
[2] The WinSCP file transfer window is displayed after connection. On the left presents the local computer while on the right presents the files of the remote computer. Drag the target file you want to transfer and drop it to another side.
[3] Choose the file location in the pop-up window, and then click OK to achieve the file transfer.
Open OnDemand¶
To transfer the files using Open OnDemand see section file transfer.
Transfer data from internet to cluster¶
Internet access
Currently only login nodes have an access to internet.
One of the most straightforward ways to download files is to use wget. Any file that can be downloaded in your web browser with an accessible link can be downloaded using wget. This is a quick way to download datasets or source code.
For example, download the lesson sample files using the following command:
wget <URL>
Sometimes the data, pipeline or software you need is stored in a repository on GitHub or GitLab. In this case you either download individual (“raw”) files using wget or the whole repository with git clone.
For example, download this test repository with:
git clone https://github.com/test/HelloWorld.git
It will be saved into the current directory with the new folder having the name of the repository, so “HelloWorld” in this case.