Dataclone trial build
This document describes the installation process of dataclone on DEB-based or RPM-based operating system and guides the first steps of dataclone usage.
Minimal System Requirements
- CPU multicore, at the best case at least 4 cores. This is a soft requirement, however, the performance of data transmission strictly depends on CPU frequency and ability to use multiple cores
- RAM: 4GB of free RAM is required to achieve higher performance.
- System Architecture: Linux x64
- By default following ports must be allowed for inbound connections on the firewall:
- TCP: 3205 (for control communication)
- UDP: 2312 (for data transmission)
Installation guide
To install the software:
tar -xf dataclone_bundle*.tar.gz -C dataclone
cd dataclone
depending on operating system and version of rmdt run:
./install-deb.sh
for Debian and Ubuntu;
./install-rpm.sh
for RedHat and CentOS;
This is it! Now you have dataclone
installed on your system.
Note: dataclone
will be installed systemwide.
Note: superuser rights are required for installation.
Quickstart
First of all, make sure that the software has been installed on all hosts which planned to be involved in data transmission.
Make sure that the sender host is able to connect to counterparts via ssh
.
Now we are ready to perform first run.
dccli <source_path> <remote_host>:<destination>
for example:
dccli /tmp/myfile awshost:/tmp/
for more details run dccli --help
What exactly will be installed
The installation script will install three packages:
dataclone
– the core of data transportdccli
– a command-line interface for communication with thedataclone
codemeter
– a third party software protection module. You may want to download it by yourself, or learn more about it.
The dccli
and dataclone
are using TLS channel for communication with each other. Therefore it is necessary to give the correct certificates to the software. Upon installation, we already created necessary certificates for the current machine. The certificates are signed by our demo CA which is used for demo or test purposes only.
Consider using your own TLS/SSL certificates and under no circumstances use it for transmission of sensible data.
The demo ca certificate and the corresponding key are located in /etc/dexor/ssl/
(the key is accessible by superuser only).
The user certificates were signed with this CA and include all found IP-addresses, aliases and FQDN of the current machine. user certificates are located in $HOME/.dexor/ssl/
Paths to certificates may be changed in the configuration file: $HOME/.dexor/dataclone.ini
for dataclone
and $HOME/.dexor/dccli.yaml
for dccli
How does it work
Reliable Multi-Destination Transport (RMDT) protocol, which is used in dataclone is UDP based. The protocol implemented as a user software, and therefore does not require any kernel modifications and can be easily installed on nearly every distribution of Linux.
From a network point of view the data traffic, produced by dataclone
, is a simple stream of UDP datagrams. The entire logic, which ensures the reliability of data, congestion control, etc, is covered by the software.
Dataclone architecture
dataclone
is a core of accelerated data transmission from Dexor. This application may be run as a daemon and serve incoming requests or maybe started upon request over SSH and then perform the transmission. The last approach is implemented within this trial version.
According to this scenario, the dataclone
is installed on all involved hosts. dccli
serves as a communicator to dataclone
s. It runs dataclone
over SSH, makes requests on data transmission and receives back the progress of the job. As dccli
communicates with dataclone
over the network, it is not necessary to run dccli
on the same host with sender or receiver. It may be run on a machine, which is even not involved in data transmission. Consider for this case flag -H
of dccli
(see dccli --help
for details).
Point-to-multipoint
dataclone
is a file transfer application based on RMDT protocol. RMDT natively supports data transmission to multiple destinations simultaneously. dataclone
does as well. To check out the feature, simply pass many destinations to the dccli
:
dccli <source_path> <remote_host_1>:<destination> <remote_host_2>:<destination>
Performance tips
There are many different aspects, which may improve the performance of data transmission. Here are some of them:
- Depending on RTT between communication hosts, it may be necessary to increase the RMDT buffer size. The default size is 400MBytes. The value may be changed in
$HOME/.dexor/dataclone.ini
- Best performance (beyond 10 Gbps) may be achieved if the network allows using Jumbo Frames. It is always better to use the maximum possible MTU size of connection for data transmission. MTU may be identified using e.g.
tracepath
utility. MTU size fordataclone
may be set in$HOME/.dexor/dataclone.ini
. The Default value is 1500 Bytes. - Extend queue size of your NIC from 1000 to, e.g. 3000
- Extend receive and send buffers of system connections:
sysctl -w net.core.rmem_max=8388608;
sysctl -w net.core.wmem_max=8388608
- Network performance sometimes is not the bottleneck of file transmission. To be able to send data with some certain rate, make sure, that sender’s storage is able to read on such a rate and receiver’s storage is able to write on such a rate.
Trial version limitation
This trial version does not perform any encryption of data. Only control channels between dccli
and dataclone are TLS encrypted.
The trial version will become unusable in 30 days from the first run.
Support
In case of any questions or issues please contact support@dexor.io
Copyright notation
Copyright © 2019, Dexor S&M GmbH. All rights reserved.
We in Dexor believe that information in this document is accurate as of its publication date. We will keep the right to change this information without notice.
The information in this document is provided “as is”. Dexor S&M GmbH makes no warranties of any kind. Dexor S&M GmbH assumes no liability for damages, caused by errors or inaccuracies, or errors resulting from the use of the document, or resulting from the application described herein.
The information in this document is the property of Dexor S&M GmbH. It may not be used, reproduced or disclosed without the written approval of Dexor S&M GmbH.
version of document: 1.0.2.0
publication date: 18.10.2019