Command Line Interface Tools

Compute- or data-heavy research often requires moving data to and from remote locations. While some of storage locations have click-and-drag web browser file transfer options (e.g. Google Drive), others require the use of dedicated tools or command line programs to move or download data.

In addition to basic file transfer commands (scp, sftp, rync), many cloud storage providers, such as Amazon Web Services, Google Cloud Platform and Microsoft Azure, provide local CLI tools to transfer data to and from their servers. Open source tools like RClone can be used to manage data on a variety of cloud storage systems, and can help you mirror and migrate data between cloud systems, or from local storage to the cloud.

Generally speaking, it is best practice, whenever possible, to compress files into a single archive file before transferring data between systems, especially for directories with large numbers of files.

• For information on transferring data securely over the web, please see OIT’s guidance.
• For more details on transferring data to or from NC State’s Henry2 HPC cluster, please see OIT’s file transfer documentation.

Common CLI file transfer tools

Scp (Winscp)

The most basic of remote file transfer tools, scp allows a user to transfer files over an ssh connection. If you are already using the command line and would like to carry out a single, one time transfer to and from UNIX based systems (MacOS, linux), scp will help you do just that. It comes pre-installed on most UNIX based systems, though Windows users will have to install a packaged windows alternative (WinSCP) or use the windows subsystem for linux.

Example: scp -r unityID@login.hpc.ncsu.edu:/home/unityID/my/labdata ~/Downloads/

Rsync

Rsync has slightly more built-in functionality than scp, but typically requires installation. Where scp copies over all files contained in a directory and must start over if it fails during a transfer, rsync only copies over new or changed files from source to destination and will restart after certain types of failures that would cause scp to fail completely. For these reasons, rsync is often used to execute scheduled transfers and backups, even when the source and destination are accessible on the same system (e.g. backing up folder contents to external drive.

Note: rsync does not ignore trailing backslashes (/), which can be tricky for new users. When in doubt, don’t let your paths end with a slash. For certain storage options (e.g. NCSU Research Storage), remote storage must be mounted to your local machine before rsync can be used effectively.

Example: rsync -avzh /path/to/source/folder username@:/path/to/destination

RClone

Rclone is a powerful file transfer tool that is designed for transfers from a wide range of remote or cloud storage resources. Using a combination of a command line, text based interface and web browser authentication, rclone allows you to access and copy files from Amazon S3, Box, Dropbox, Google Drive, Microsoft OneDrive, traditional ftp/ssh accessible servers, and more. However, Rclone can also be installed on NCSU’s HPC cluster, Hazel.