Dataclone trial build

This document describes the installation process of dataclone on DEB-based or RPM-based operating system and guides the first steps of dataclone usage.

Minimal System Requirements

  • CPU multicore, at the best case at least 4 cores. This is a soft requirement, however, the performance of data transmission strictly depends on CPU frequency and ability to use multiple cores
  • RAM: 4GB of free RAM is required to achieve higher performance.
  • System Architecture: Linux x64
  • By default following ports must be allowed for inbound connections on the firewall:
    • TCP: 3205 (for control communication)
    • UDP: 2312 (for data transmission)

Installation guide

To install the software:

tar -xf dataclone_bundle*.tar.gz -C dataclone
cd dataclone

depending on operating system and version of rmdt run:

./install-deb.sh for Debian and Ubuntu;

./install-rpm.sh for RedHat and CentOS;

This is it! Now you have dataclone installed on your system.

Note: dataclone will be installed systemwide.

Note: superuser rights are required for installation.

Quickstart

First of all, make sure that the software has been installed on all hosts which planned to be involved in data transmission.

Make sure that the sender host is able to connect to counterparts via ssh.

Now we are ready to perform first run.

dccli <source_path> <remote_host>:<destination>

for example:

dccli /tmp/myfile awshost:/tmp/

for more details run dccli --help

What exactly will be installed

The installation script will install three packages:

  • dataclone – the core of data transport
  • dccli – a command-line interface for communication with the dataclone
  • codemeter – a third party software protection module. You may want to download it by yourself, or learn more about it.

The dccli and dataclone are using TLS channel for communication with each other. Therefore it is necessary to give the correct certificates to the software. Upon installation, we already created necessary certificates for the current machine. The certificates are signed by our demo CA which is used for demo or test purposes only.

Consider using your own TLS/SSL certificates and under no circumstances use it for transmission of sensible data.

The demo ca certificate and the corresponding key are located in /etc/dexor/ssl/ (the key is accessible by superuser only).

The user certificates were signed with this CA and include all found IP-addresses, aliases and FQDN of the current machine. user certificates are located in $HOME/.dexor/ssl/

Paths to certificates may be changed in the configuration file: $HOME/.dexor/dataclone.ini for dataclone and $HOME/.dexor/dccli.yaml for dccli

How does it work

Reliable Multi-Destination Transport (RMDT) protocol, which is used in dataclone is UDP based. The protocol implemented as a user software, and therefore does not require any kernel modifications and can be easily installed on nearly every distribution of Linux.

From a network point of view the data traffic, produced by dataclone, is a simple stream of UDP datagrams. The entire logic, which ensures the reliability of data, congestion control, etc, is covered by the software.

Dataclone architecture

dataclone is a core of accelerated data transmission from Dexor. This application may be run as a daemon and serve incoming requests or maybe started upon request over SSH and then perform the transmission. The last approach is implemented within this trial version.

According to this scenario, the dataclone is installed on all involved hosts. dccli serves as a communicator to dataclones. It runs dataclone over SSH, makes requests on data transmission and receives back the progress of the job. As dccli communicates with dataclone over the network, it is not necessary to run dccli on the same host with sender or receiver. It may be run on a machine, which is even not involved in data transmission. Consider for this case flag -H of dccli (see dccli --help for details).

Point-to-multipoint

dataclone is a file transfer application based on RMDT protocol. RMDT natively supports data transmission to multiple destinations simultaneously. dataclone does as well. To check out the feature, simply pass many destinations to the dccli:

dccli <source_path> <remote_host_1>:<destination> <remote_host_2>:<destination>

Performance tips

There are many different aspects, which may improve the performance of data transmission. Here are some of them:

  • Depending on RTT between communication hosts, it may be necessary to increase the RMDT buffer size. The default size is 400MBytes. The value may be changed in $HOME/.dexor/dataclone.ini
  • Best performance (beyond 10 Gbps) may be achieved if the network allows using Jumbo Frames. It is always better to use the maximum possible MTU size of connection for data transmission. MTU may be identified using e.g. tracepath utility. MTU size for dataclone may be set in $HOME/.dexor/dataclone.ini. The Default value is 1500 Bytes.
  • Extend queue size of your NIC from 1000 to, e.g. 3000
  • Extend receive and send buffers of system connections:
sysctl -w net.core.rmem_max=8388608;
sysctl -w net.core.wmem_max=8388608
  • Network performance sometimes is not the bottleneck of file transmission. To be able to send data with some certain rate, make sure, that sender’s storage is able to read on such a rate and receiver’s storage is able to write on such a rate.

Trial version limitation

This trial version does not perform any encryption of data. Only control channels between dccli and dataclone are TLS encrypted.

The trial version will become unusable in 30 days from the first run.

Support

In case of any questions or issues please contact support@dexor.io

Copyright notation

Copyright © 2019, Dexor S&M GmbH. All rights reserved.

We in Dexor believe that information in this document is accurate as of its publication date. We will keep the right to change this information without notice.

The information in this document is provided “as is”. Dexor S&M GmbH makes no warranties of any kind. Dexor S&M GmbH assumes no liability for damages, caused by errors or inaccuracies, or errors resulting from the use of the document, or resulting from the application described herein.

The information in this document is the property of Dexor S&M GmbH. It may not be used, reproduced or disclosed without the written approval of Dexor S&M GmbH.

version of document: 1.0.2.0

publication date: 18.10.2019