Skip to content

Data Transfer from/to PERUN cluster located at TUKE

Introduction

Data transfers between SAS and the TUKE site of the Perun cluster are performed using SSH-based tools. The supported utilities are:

  • rsync – recommended for most use cases
  • scp – suitable for simple, one-time transfers

Transfers must be executed via dedicated data transfer nodes:

  • data01
  • data02

These nodes are optimized for high-throughput data movement and are connected via a 2 × 100 Gb/s link to TUKE. Filesystems such as /home and /project are directly mounted on these nodes, allowing efficient direct transfers. The service is available to all integrated users without additional setup.

Both transfer nodes are accesible via ssh command from Perun login nodes: 

ssh data01

On the TUKE side, the corresponding transfer endpoints are:

  • 10.101.40.16
  • 10.101.40.17

In the following examples <TUKE_HOST> refers to either one of these addresses.

Login nodes

Use of login nodes for large data transfers is discouraged. Always use data01 or data02 to ensure performance and system stability.

Prerequisites

To use the transfer service:

  • Your account must be integrated with TUKE
  • You must have valid SSH access
  • You must connect through data01 or data02

If your account is not yet integrated, please follow the procedure described in section User Account Integration

rsync over scp

For most scenarios, rsync is preferred due to its efficiency and ability to resume transfers. Use scp only for small or quick operations where simplicity is sufficient.

Using rsync

rsync is a robust tool for synchronizing files and directories. It transfers only differences between source and destination, making it efficient for large datasets.

Basic syntax

rsync [options] SOURCE DESTINATION

Examples

Transfer local directory to TUKE:

rsync -avz </path/to/source/>  <TUKE_HOST>:</path/to/destination/>

Transfer from TUKE to local:

rsync -avz <TUKE_HOST>:</path/to/source/> </path/to/destination/>

Resume interrupted transfer:

rsync -avz --partial --progress </path/to/source/> <TUKE_HOST>:</path/to/destination/>

Synchronize and remove obsolete files:

rsync -avz --delete </path/to/source/> <TUKE_HOST>:</path/to/destination/>

Common options:

  • -a archive mode (preserves metadata)
  • -v verbose output
  • -z compression
  • --progress transfer progress

Copy directory content

Use a trailing / on the source to copy contents rather than the directory itself.

Using scp

scp provides a simple way to transfer files over SSH but lacks the efficiency and robustness of rsync.

Basic syntax

scp [options] SOURCE DESTINATION

Examples

Copy file to TUKE:

scp </path/to/file> <TUKE_HOST>:</path/to/file>

Copy directory to TUKE:

scp -r </path/to/source/> <TUKE_HOST>:</path/to/destination/>

Copy from TUKE: 

scp -r <TUKE_HOST>:</path/to/source/> </path/to/destination/>

Limitations:

  • No resume capability
  • Transfers full files only
  • Inefficient for large datasets
Created by: Andrej Sec, Marek Štekláč