rsync Remote Synchronization

preamble

rsync (Remote Sync) is an open source fast backup tool , can be mirrored between different hosts to synchronize the entire directory tree , support for incremental backups , and to maintain links and permissions , and the use of optimized synchronization algorithms , compression is performed prior to the transfer , so it is very suitable for offsite backup , mirroring servers and other applications .

I. Introduction to Rsync

(1) Introduction to rsync

rsync is an open source, fast, multi-functional, full and incremental local or remote data synchronization backup excellent tool. And can not change the original data attribute information.Realization of data backup migration features。

In the Remote Synchronization task, the person responsible forThe client that initiates the rsync synchronization operation is called the initiatorAnd responsible for the soundThe server that responds to an rsync synchronization operation from a client is called the synchronization source. In the synchronization process, the synchronization source is responsible for providing the original location of the file, and the initiator should have read access to that location.

Rsync isLinux Data mirroring backup tool under the system, using fast incremental backup tool Remote Sync can be synchronized remotely, can be synchronized between different hosts, full and incremental backups can be achieved, maintaining links and permissions, and the use of optimized synchronization algorithms, compression is performed before the transfer, so it is very suitable for the architecture of centralized backup or off-site backup and other applications. Rsync also supports local replication or synchronization with other SSH or rsync hosts.rsync listening port:873, rsync operation mode:C/S

Official website: https://rsync.samba.org/

(2) rsync synchronization method

1. Full Backup: Each backup backs up all files or directories from the backup source to the destination.

2. Differential Backup: Backs up data that has changed since the last full backup (he targets the last full backup, he does not clear the archive attributes during the backup process).

3. Incremental backup: backs up data that has changed since the last backup (he doesn't care what type of backup it is, if there is a change in the data, it is backed up and he will clear the archive properties)

II. rsync synchronization source

• rsync synchronization source
• Refers to the remote server where the backup operation takes place, also known as the backup source

(1) Configure the rsync source

①Basic Idea

• Create profiles, separate account files
• Enable --daemon mode for rsync

②Profile

Authentication configuration auth users, secrets file, do not add the anonymous

③Independent account password

• Username:Password
• One user record per line
• Independent account data, not dependent on system accounts

④Enable rsync service

• Service alone via --daemon rsync --daemon
• Execute kill $(cat /var/run/) to shut down the service

Common Rsync commands

Command Syntax
rsync [Options] Original location Destination location

• 1
• 2

• Common Options

The role of -delete is simply to delete discrepant files and retain consistency

IV. rsync vs. cp,scpcomparison

• The cp command isA typical copy of a file in its entirety to a single location. And rsync is, for the first copy, when the target location doesn't have aThe full amount is copied over, but on the second copy, only the differences are copied synchronously.The rsync is faster than the cp if you do a second backup of the same file. All if you do a second backup of the same file, rsync will be faster compared to cp.
• cp only supports local, while rsync supports remote
• scp is based on the principle of cp, also belongs to the integrity copy file. Assuming that rsync and scp copy files are the first, the target address are not to synchronize the file, at this point, the difference between the two lies in the first, the file to be transmitted is not big, the second depends on the transmission process, with the bandwidth is not big. If the file is not large, scp is to extract the data from the disk block storage, encapsulation, the network to pass, at this time scp faster, if it is a larger file, for example, 40G, the bandwidth only supports 100M bandwidth, scp want to transmit, you need to split the data, a section of the transfer. And rsync will be based on a logical sense of the space, the data is divided out, the data is compressed before transmission, so this way, the bandwidth school, the file is large, this time the first compression and then transmission will be faster. At this time it is suitable to use rsync remote synchronization.

V. Two ways of expressing configuration sources

Format I:

Username@host address::Shared module name
Example:
rsync -avz [email protected]::wwwroot /opt/

• 1
• 2
• 3

Format II:

rsync://username@host address/shared module name
Example:
rsync -avz rsync://[email protected]/wwwroot /opt/

• 1
• 2
• 3

VI. Configuring server-side and client-side experiments

To disable firewalls and enhancements
systemctl stop firewalld
setenforce 0

• 1
• 2
• 3

1. Turn off firewalls and installation mechanisms

(1) Configure the environment

Configure the rsync source server (192.168.159.230)
rpm -q rsync #Rsync is usually installed by default.

• 1
• 2

(2) Configure the client side

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
#Disable the firewall and setup mechanism

yum install -y httpd rsync
#The rsync system is usually installed by default, and httpd is installed to generate the /var/www/html directory (which will be used as a shared directory later).

vim /etc/
#Edit the rsync configuration file
uid = root
gid = root
use chroot = yes
address = 192.168.159.230
port 873
log file = /var/log/
pid file = /var/run/
hosts allow = 192.168.159.0/24
[wwwroot]
path = /var/www/html
comment = Document Root of
read only = yes
dont comperss = *.gz *.bz2 *.tgz *.zip *.rar *.z
auth users = backuper
secrets file = /etc/rsyncd_users.db
---- detail ----
uid = root #User id, indicates the identity of the user who can execute with shared permissions.
gid = root #Group id
use chroot = yes #Enabled, confined to the source directory, means that when you access the directory or file I backed up, you are allowed to use the role of root, and you have root privileges when you access the local directory.
address = 192.168.159.320 # Listening address.
port 873 #Default port number is 873.
log file = /var/log/ #location of the log file
pid file = /var/run/ #location of the process id file
hosts allow = 192.168.159.0/24 #Allowed host segments, kind of like a black/white list.
[wwwroot] #Name of the shared module that rsync calls by default, and the default path I call is the path specified by that module
path = /var/www/html # source directory path
comment = Document Root of #
read only = yes # If it is read only
dont comperss = *.gz *.bz2 *.tgz *.zip *.rar *.z #File types that are no longer compressed during synchronization, since -avz is already compressed during synchronization.
auth users = backuper #Authorized users, which user is using the wwwroot module, multiple accounts separated by spaces
secrets file = /etc/rsyncd_users.db #Data file for account information, one line at a time.
----
#############################################
uid = nobody
gid = nobody
use chroot = yes
address = 192.168.159.230
port 873
log file = /var/log/
pid file = /var/run/
hosts allow = 192.168.159.0/24
[wwwroot]
path = /var/www/html
comment = Document Root of
read only = yes
dont comperss = *.gz *.bz2 *.tgz *.zip *.rar *.z
auth users = kiki
secrets file = /etc/rsyncd_users.db
###############################################
vim /etc/rsyncd_users.db
kiki:123123
# Edit the user accounts file in the fixed format [name:password], one line at a time.

chmod 600 /etc/rsyncd_users.db
#Official request, it's better to just empower 600!

rsync --daemon
# Turn on the service
netstat -natp | grep rsync
# Check the port number to make sure the service is successfully turned on

cd /var/www/html
# Switch to the shared directory
touch
cd /var/www/html

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34
• 35
• 36
• 37
• 38
• 39
• 40
• 41
• 42
• 43
• 44
• 45
• 46
• 47
• 48
• 49
• 50
• 51
• 52
• 53
• 54
• 55
• 56
• 57
• 58
• 59
• 60
• 61
• 62
• 63
• 64
• 65
• 66
• 67
• 68
• 69
• 70
• 71
• 72
• 73
• 74
• 75

1. Turn off firewalls and installation mechanisms

2, rsync system has generally been installed by default, the installation of httpd is to generate /var/www/html directory (subsequently will be used as a shared directory)

3. Edit the rsync configuration file

4, edit the user account file, fixed format [name: password], a line a

5. Empowerment, official recommendation best 600

6, open the service, detect the port number, confirm whether the service is successfully opened

7. Switch to the shared directory to create the file

(3) Configurationserverextremity

systemctl stop firewalld
systemctl disable firewalld
setenforce 0
#Disable the firewall and setup mechanism

yum install -y rsync
#Install rsync

mkdir /opt/haha

#Create a directory to synchronize

rsync -avz [email protected]::wwwroot /abc
#Use rsync to synchronize the synchronized files of the backup source

ls
#Check if the synchronization was successful


vim /etc/
123123
#Edit the no-interaction key file, the first line is the passphrase


chmod 600 /etc/
#Assign 600 to the key file
rsync -avz [email protected]::wwwroot /opt/haha
#Synchronize data

rsync -az --delete --password-file=/etc/ [email protected]::wwwroot /opt/haha
#rsync, using the key file /etc/server/pass corresponding to the backuper user, IP address 192.168.159.230 shared module files are compressed and archived and synchronized to the /abc directory of the current server, and at the same time delete the difference content, if the original target has it, it will be added, if the original target does not have it, it will be deleted. Maintain consistency.

ls /abc
# Check if the downstream synchronization was successful

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34

1. Turn off firewalls and enhancement mechanisms

2. Install rsync

3. Create a directory synchronization directory

4. Use rsync to synchronize the synchronized files of the backup source and see if it is complete


5. Synchronization of data

6. Synchronize using rsync -az --delete

(4) rsync+inotify real-time synchronization

①Introduction to inotify

Can monitor file system changes and respond with notifications

#Tuning inotify kernel parameters (optimization)
letc/ (kernel parameter configuration file)
inotifywait: #Use for continuous monitoring, output results in real time.
inotifywatch: #Use for short-term monitoring, output results after the task is completed.
max_queue_events #Monitoring event queue size.
max_user instances #Maximum number of instances to monitor.
max_user_watches #Maximum number of watched files per instance.

inotifywait Format Parameters
Common Parameters Description
-m Continuous monitoring
-r Recursive monitoring of all child objects
-q Simplify output
-e Specify which event types to monitor (*)

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14

②Configure Rsync+Inotify real-time synchronization

• The inotify notification interface can be used to monitor various changes in the file system, such as file access, deletion, movement, modification and so on. Using this mechanism, it is very convenient to realize file movement alerts, incremental backups, and timely response to changes in directories or files.
• Combining the inotify mechanism with the rsync tool enables triggered backups (real-time synchronization), i.e., as long as the document in the original location has changed, an incremental backup operation is immediately initiated; otherwise it is in a silent waiting state.
• Because the inotify notification mechanism is provided by the Linux kernel, it is mainly used for local monitoring and is better suited for upstream synchronization when used in triggered backups.

③Master disables read-only mode and empowers the shared directory

vim /etc/
read only = no
# Turn off read-only mode, otherwise it will be unwritable


kill `cat /var/run/'
#Restart the service after modifying the configuration file, here we use the direct kill process number method

netstat -natp | grep rsync
#Check if the service has been terminated

rsync --daemon
netstat -natp | grep rsync
#Start the service again and check the port number to confirm

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14

1. Turn off read-only mode, otherwise it is not writable

2, after modifying the configuration file need to restart the service, here using the direct kill process number, check whether the service has been suspended, and open again!

④ Optimize Slave kernel parameters

cat /proc/sys/fs/inotify/max_queued_events #Monitored event queue
cat /proc/sys/fs/inotify/max_user_instances #Maximum number of monitored instances
cat /proc/sys/fs/inotify/max_user_watches #Maximum number of monitored files per instance

vim /etc/ #increase each parameter
.max_queued_events = 16384
.max_user_instances = 1024
.max_user_watches = 1048576
# Increase the value of the parameter when the directory or file to be monitored has a large amount of data or changes frequently.

sysctl -p
#Refresh

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12

1、View the parameters

2. Increase each parameter

3、Refresh

⑤ Compile and install inotify-tools.

yum install -y gcc gcc-c++
#mountinggcc gcc-c++

cd /opt
#Switch to/optuploadinotify-toolsmounting包

tar zxf inotify-tools-3.
#decompression (in digital technology)

cd /opt/inotify-tools-3.14/
./configure
make && make install
#编译mounting

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13

1、Install the compilation dependency packages, upload the installation package and decompression compilation

2. Installation

⑥ Write trigger synchronization scripts

vim /opt/
/bin/bash /bin/bash
INOTIFY_CMD="inotifywait -mrq -e create,delete,move,modify,attrib /opt/abc/"
RSYNC_CMD="rsync -azH --delete --password-file=/etc/ /opt/abc/ [email protected]::wwwroot"

$INOTIFY_CMD | while read DIRECTORY EVENT FILE
if [ $(pgrep-file/etc/ /opt/abc/ )
    if [ $(pgrep rsync | wc -l) -le 0 ] ; then
        $RSYNC_CMD
do while read DIRECTORY EVENT FILE
done
---- detail ----
/bin/bash /bin/bash
INOTIFY_CMD="inotifywait -mrq -e create,delete,move,modify,attrib /opt/abc/"
#INOTIFY_CMD variable: Continuously monitor the /opt/abc directory for creation, deletion, movement, modification, and time change operations.
RSYNC_CMD="rsync -azH --delete --password-file=/etc/ /opt/abc/ [email protected]::wwwroot"
#RSYNC_CMD variable: enable xixi user, /etc/ key file, zip files in /opt/xcf1 directory, archive them, keep hardlinked files synchronized to the directory /var/www/html defined by the shared module on 192.168.159.230, and remove discrepancies to maintain consistency

$INOTIFY_CMD | while read DIRECTORY EVENT FILE # Continuous monitoring...
do
    if [ $(pgrep rsync | wc -l) -le 0 ] ; then #If the service is not started, perform the synchronization
        $RSYNC_CMD
If the service is not started, perform synchronization.
done
----

cd /opt/
chmod +x
#Authorize the script

chmod +x /etc//
echo "/opt/" >> /etc//
#Set up bootup

sh -x
#Execute the script

cd /opt/abc
touch
rm -rf
#Create a new html file and delete the previous qwe
ls
#Check again

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34
• 35
• 36
• 37
• 38
• 39
• 40
• 41
• 42
• 43

1. Writing scripts

2. Empowering scripts and empowering execution

3, test results in the server-side operation, create a file and delete a file

4. View the results on the server side

5, before the above figure server synchronization will report an error, the reason is that we logged in as an anonymous user, the following is the solution:

VII. rsync application scenarios

1. Incremental synchronization

2. Backup/migration: It can be used as an auxiliary tool (mysql Master-slave replication rsync +inotify -- "convergence Bidirectional synchronization -- MM, but only when the concurrency is not too large)

Assuming mysql migration from host A to host B (one master, two slaves mode)

One-master-twoslave migration scenario

① Determine the time to migrate, the time to stop the line of business, to write a document describing the specific operation, to send an email for approval

② confirm the migration of the node after the environmental issues (resource environment + system dependency environment + mysql internal format environment ------1) such as self-growth, self-growth may lead to inconsistent master-slave replication 2) version migration issues, such as the migration of the low version to the high version. The differences in the two versions are relatively large, data migration is more important, you need to communicate with the R & D, so that the R & D to develop a tool, so that the low version can be converted to the high version of the tool format, using the tool migration)

③ If the database is the same version, you can use rsync + inotify to synchronize continuously

④ Test and write rollback scenarios beforehand.

⑤ Submission of overhaul applications

3, the use of rsync to achieve rapid deletion of a large number of files

Suppose you want to delete a large number of files under linux, say 1 million, 10 million, like /usr/local/nginx/proxy_temp's nginx cache, etc., then rm -rf * may not work well because of the long wait. In this case we can use rsync to handle it smartly. rsync actually uses the replacement principle.

Start by creating an empty folder:
mkdir /home/blank

Delete the target directory with rsync:
rsync --delete-before -a -H -v --progress --stats /home/blank /usr/local/nginx/proxy_temp
This way the target directory is quickly emptied
Option Description:
-delete-before The receiver is transmitting a delete operation.
-a Archive mode, which indicates that files are transferred recursively, maintaining all file attributes
-H Maintain hardwired files
-v Detailed output mode
-progress Displays the progress of a transfer while it is in progress.
-stats gives the status of certain file transfers

VIII. Summary

rsync is an open source, fast, versatile and excellent tool for full and incremental local or remote data synchronization and backup. remote Sync, remote synchronization, support for local replication, or synchronization with other SSH, rsync host.