Transferring files between local machine and HPC cluster

Users need to have the data and application related to their project/research work on PARAM Shakti.

To store the data special directories have been made available to the users with name “scratch and home” the path to this directory is “/scratch” and “/home”. Whereas these directories are common to all the users, a user will get his own directory with their username in /scratch/ as well as /home/ directories where they can store their data.

/home/<username>/: ! This directory is generally used by the user to install applications.

/scratch/<username>/: ! This directory is user to store the user data related to the project/research.

However, there is limit to the storage provided to the users, the limits have been defined according to quota over these directories, all users will be allotted same quota by default. When a user wishes to transfer data from their local system (laptop/desktop) to HPC system, they can use various methods and tools.

A user using ‘Windows’ operating system will get methods and tools that are native to Microsoft windows and tools that could be installed on your Microsoft windows machine. Linux operating system users do not require any tool. They can just use “scp” command on their terminal, as mentioned below.

Users are advised to keep a copy of their data with themselves, once the project/research work is completed by transferring the data in from PARAM Shakti to their local system (laptop/desktop). The command shown below can be used for file transfers (In all the tools):

[localhost]$scp –r <path to the local data> saurabh@paramsahkti.iitkgp.ac.in:<path to directory on HPC>  

Example:

[localhost]scp  –r  /mydata/dir1  saurabh@paramsahkti.iitkgp.ac.in:/home/saurabh  

Same Command could be used to transfer data from HPC system to your local system (laptop/desktop).

[localhost]$scp –r saurabh@paramsahkti.iitkgp.ac.in:<path to directory on HPC>  <path to the local data directory>

 Example:

[localhost]scp  –r   saurabh@paramsahkti.iitkgp.ac.in:/home/saurabh  /mydata/dir1  

File transfer tools

WinSCP (Windows installable application):

This popular tool is freely available and is used very often to transfer data from Windows machine to Linux machine. This tool is GUI based which makes it very user-friendly.

Link for this tool is :   https://winscp.net/eng/download.php 

Command Prompt (Linux native application):

SCP is available as a native tool for All type of Unix/Linux machine which can be used to transfer data  To/From PARAM Shakti system through your local systems (laptop/desktop).

Please use scp command option “-o UserKnownHostsFile=/dev/null” to avoid the SSH key error , "-r" option is to copy the directory recursively.

File transfer when working from outside IIT Kharagpur campus

Latest Windows 10 Desktop, Linux and Mac systems provide a built-in SCP client, so there is no need to install any additional package. Open the terminal and provide the port number “4422” in scp command line.

scp –P [port_no] -o UserKnownHostsFile=/dev/null -r [source_folder_path][username]@[hostname]:[destination_Path]

For example, to transfer of folder from local machine to home directory in Paramshakti use below command. 

user1@login01$scp -P 4422 -o UserKnownHostsFile=/dev/null -r iitkgp user1@paramshakti.iitkgp.ac.in:/home/user1/

You will be prompted for a string followed by password, and then folder will be transferred to destination.

*Note:  -P option should be in capital. A windows 10 screenshot given below.