rsync Remote Synchronization ~ Offsite Backup Policy

---

One,rsyncpresent (sb for a job etc)

• rsync (Remote Sync, Remote Sync) is an open source fast backup tool that can mirror and synchronize the entire directory tree between different hosts, supports incremental backups and maintains links and permissions, and uses optimized synchronization algorithms and performs compression prior to transmission, so it is very suitable for offsite backups, mirroring servers and other applications.
• The official rsync site is located at / and is maintained by Wayne Davison. As one of the most commonly used file backup tools, rsync is often one of the basic components installed by default on Linux and UNIX systems.

II. rsync source server

In a remote synchronization task, the client that initiates the rsync synchronization operation is called the initiator, and the server that responds to the rsync synchronization operation from the client is called the source.

In the synchronization process, the synchronization source is responsible for providing the original location of the file, and the initiator should have read access to that location, as shown in the figure below.

III. Configuring the rsync source server

Steps to configure the rsync source server

Environmental Preparation:

• Creating the /etc/ configuration file

rpm -q rsync								#Usually rsync is already installed by default

#build up/etc/rsyncd.conf configuration file

vim /etc/rsyncd.conf			  	 		#Add the following configuration items
uid = nobody					    		#It can also be root
gid = nobody					    		#It can also be root
use chroot = yes							#Confinement to the source directory
address = 192.168.100.10						#Listen to the address, listen to the local address
port 873						   		    #Listening ports tcp/udp 873, which can be accessed via cat /etc/services | grep rsync view
log file = /var/log/rsyncd.log				#Log file location
pid file = /var/run/rsyncd.pid				#depository process ID The location of the document
hosts allow = 192.168.100.0/24				# Client network segments allowed to be synchronized
[wwwroot]					        		#Shared Module Name
path = /var/www/html						#Actual path to the source directory (the directory to be synchronized)
comment = Document Root of www.clj.com
read only = yes					    		#Read-only or not
dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z		#File types that are no longer compressed during synchronization
auth users = backuper						#Authorized accounts, multiple accounts separated by spaces
secrets file = /etc/rsyncd_users.db		    #Data files that hold account information
-------------------------------------------------------------------------------
uid = nobody
gid = nobody                                    
use chroot = yes                                        
address = 192.168.100.10
port 873                                                
log file = /var/log/rsyncd.log
pid file = /var/run/rsyncd.pid                          
hosts allow = 192.168.100.0/24
[wwwroot]                                       
path = /var/www/html                            
comment = Document Root of www.clj.com
read only = yes                                 
dont compress = *.gz *.bz2 *.tgz *.zip *.rar *.z                
auth users = backuper                           
secrets file = /etc/rsyncd_users.db 
----------------------------------------------------------------------------------
#In case of anonymization, simply replace the "auth" in the users" and "secrets". The "file" configuration item can be removed.

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34
• 35
• 36
• 37
• 38

• Creating Data Files for Backup Accounts

# Create data files for backup accounts
vim /etc/rsyncd_users.db
backuper:abc123					#No need to create system users with the same name

chmod 600 /etc/rsyncd_users.db
# Supplementary: SSH -i Key File Location root@192.168.200.1   # Authorize remote login
The permissions on the #key file need to be600

# Ensure that all users have access to the source directory /var/www/html Both have read access.

#Installing the http service
yum -y install httpd
systemctl start httpd
systemctl enable httpd

mkdir -p /var/www/html
echo “this is gcc” > /var/www/html/gcc.txt

chmod +r /var/www/html/
ls -ld /var/www/html/    #Display file directory permissions in long format
drwxr-xr-x. 2 root root 6 7moon  28 09:01 /var/www/html/

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22

• Start the rsync service program

#Start rsync service program
rsync --daemon					#activate (a plan) rsync service, running as a standalone listening service (daemon) 

netstat -anpt | grep rsync

#cloture rsync Approach to services
kill $(cat /var/run/rsyncd.pid)
rm -rf /var/run/rsyncd.pid

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8

Fourth, use rsync backup tool

1. Basic usage of the rsync command

• Most backup programs require that you specify the original and destination locations, and so does the rsync command. In its simplest form, rsync is similar to the cp command.
  • For example, you can synchronize the backup of the file /etc/fstab and the directory /boot/grub to the /opt directory, with the "-r" option recursing through the entire directory tree, and the "-l" option backing up the linked files.

[root@localhost ~]# rsync /etc/fstab /opt 
[root@localhost ~]# rsync -rl /etc/fstab /boot/grub /opt

• 1
• 2

(1) Command Format and Commonly Used Backup Options From the above operation, we can see that the basic format of backup is "rsync [option] original location target location", and some of the commonly used command options are shown below, which should be selected according to the actual needs (e.g. -avz).

(2) Configuring the Representation of the Source When performing a remote synchronization task, the rsync command needs to specify the location of the resources in the synchronization source server.

• The resource representation of the rsync synchronization source is either "username@host address::shared module name" or "rsync://username@host address/shared module name", with the former in the form of a two-colon separation and the latter in the form of a URL address. The former is in the form of two colons and the latter is in the form of a URL address.
  • Let's say you do the following to access the rsync synchronization source and download the specified resources to the local /root directory for backup.

[root@localhost ~]# rsync -avz backuper@192.168.100.10::wwwroot /root
[root@localhost ~]# rsync -avz rsync://backuper@192.168.100.10/wwwroot /root
##Both ways are fine

• 1
• 2
• 3

2, rsync backup operation example

Prepare the environment:

• Download the specified resource to the local /opt directory for backup.

Format I:
rsync -avz backuper@192.168.100.10::wwwroot /opt/		#Password abc123

Format II:
rsync -avz rsync://backuper@192.168.100.10/wwwroot /opt/

• 1
• 2
• 3
• 4
• 5

• Viewing synchronized acquired files

cat /opt/index.html
this is www.txp.com

• 1
• 2

• Interaction-Free Format Configuration

cd /opt
rm -rf gcc.txt

echo "abc123" > /etc/server.pass
chmod 600 /etc/

• 1
• 2
• 3
• 4
• 5

• Interaction-free test synchronization

# interaction-free synchronization
/usr/bin/rsync -avz --delete --password-file=/etc/server.pass backuper@192.168.100.10::wwwroot /opt/

ls /opt/	#Check if the synchronization was successful
	
cat /opt/index.html 		#Check if the file contents are synchronized successfully

• 1
• 2
• 3
• 4
• 5
• 6

• Add to programmatic tasks

crontab -e
30 22 * * * /usr/bin/rsync -avz --delete --password-file=/etc/server.pass backuper@192.168.100.10::wwwroot /opt/

systemctl restart crond
systemctl enable crond

• 1
• 2
• 3
• 4
• 5

V. inotify + rsync real-time synchronization

• Since version 2.6.13, the Linux kernel has provided an inotify notification interface to monitor file system changes, such as file accesses, deletions, moves, modifications, etc. This mechanism makes it easy to implement file alerts, incremental backups, and timely responses to directory or file changes. Using this mechanism, it is very easy to implement file movement alerts, incremental backups, and timely response to changes in directories or files.
• By combining the inotify mechanism with the rsync tool, it is possible to achieve triggered backups (real-time synchronization) - incremental backups are started as soon as the documents in the original location change; otherwise they are in a silent waiting state, as shown in the following figure. This avoids the problems of latency and overcrowding that exist when backing up on a fixed schedule.

Because the inotify notification mechanism is provided by the Linux kernel, it is mainly used for local monitoring and is more suitable for upstream synchronization when used in triggered backups.

Six, configure inotify + rsync real-time synchronization

Steps to configure inotify+rsync real-time synchronization

Environmental preparedness:

| | |

| | |

• Modify the rsync source server (192.168.100.10) configuration file

vim /etc/rsyncd.conf
......
read only = no			         	#Turn off read-only, uplink synchronization needs to be writable
	
kill `cat /var/run/rsyncd.pid`		#Kill the rsyncd process
netstat -natp | grep rsync			#To see if a process has been killed

rsync --daemon						#reopen
netstat -natp | grep rsync			#Check if the service is turned on
 
chmod 777 /var/www/html				#Give permission

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11

• Adjust inotify kernel parameters (configured on the client: 192.168.100.20)

In the Linux kernel, the default inotify mechanism provides three regulatory parameters:
max_queue_events (monitor event queue, default value is 16384),
max_user_instances (maximum number of monitoring instances, default value is 128),
max_user_watches (maximum number of monitored files per instance, default value is 8192).
When the number of directories and files to be monitored is large or changes frequently, it is recommended to increase the values of these three parameters.

cat /proc/sys/fs/inotify/max_queued_events      #Monitoring Event Queues
cat /proc/sys/fs/inotify/max_user_instances     #Maximum number of monitoring instances
cat /proc/sys/fs/inotify/max_user_watches       #Maximum number of monitored files per instance

vim /etc/sysctl.conf         #Increase each parameter
fs.inotify.max_queued_events = 16384
fs.inotify.max_user_instances = 1024
fs.inotify.max_user_watches = 1048576

sysctl -p

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10

• Install inotify-tools (configured on the client: 192.168.100.20)

1. The inotify mechanism also requires the installation of inotify-tools, which provides the inotifywait and inotifywatch utility programs.
2. notifywait: can monitor modify (modify), create (create), move (move), delete (delete), attribute change (attribute change) and other events, a change immediately output results.
3. inotifywatch: can be used to collect filesystem changes and output summarized changes at the end of the run.

# Dependent environment
mount /dev/cdrom /mnt     #Mounting Disks
yum install gcc gcc-c++ make -y
#Include the package inotify-tools-3.14.tar.gz uploaded to/opt directory and then unpack
tar zxvf inotify-tools-3.14.tar.gz -C /opt/

cd /opt/inotify-tools-3.14
./configure
make && make install

#Execute the "inotifywait" command, and then in another terminal, send a message to the/var/www/html directory to add files, move files, and track screen output results.
#After executing the following command, it will enter the listening mode and cannot be operated, so you need to open another terminal page to do it.
inotifywait -mrq -e modify,create,move,delete /opt

#Option "-e": Used to specify which events to monitor.
#Option "-m": for continuous monitoring
#Option "-r": recurses through the entire directory.
#Option "-q": simplifies the output message

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18

• Write a trigger synchronization script on the client (192.168.100.20)

vim /opt/inotify.sh
#!/bin/bash
INOTIFY_CMD="inotifywait -mrq -e modify,create,attrib,move,delete /var/www/html/"     # Continuous monitoring
RSYNC_CMD="rsync -apzH --delete --password-file=/etc/ /var/www/html/ [email protected]::wwwroot/"     #Synchronize
$INOTIFY_CMD       #Output monitoring content
$INOTIFY_CMD | while read DIRECTORY EVENT FILE     #(math.) ergodic
##While determining whether a monitoring record has been received
do
    if [ $(pgrep rsync | wc -l) -le 0 ] ; then    #Determine if synchronization is being performed
        $RSYNC_CMD
    fi
done

chmod +x /opt/inotify.sh

chmod +x /etc/rc.d/rc.local
echo '/opt/' >> /etc/rc.d/rc.local				#Add power-on auto-execution

mkdir -p /var/www/html/     #Create a directory for synchronization in the script

#Attention!192.168.100.10 is the remote synchronization server ip(source IP)！！

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21

• Verification experiment, the client to open the monitoring

cd /opt/
./inotify.sh &

• 1
• 2

Create directories test1 and test2 in the client (192.168.100.10)/var/www/html/ directory.

cd /var/www/html/ 
mkdir test1 test2

• 1
• 2

• Check in the source server process to see if real-time synchronization is complete

ls /opt

• 1

Seven, the use of rsync to achieve rapid deletion of a large number of files.

If you want to delete a large number of files under Linux, say 1 million, 10 million, like the nginx cache in /usr/local/nginx/proxy_ temp, etc., then rm -rf *It may not work as well, because it takes a long time. In this case we can use rsync to handle it in a clever way. rsync actually uses the substitution principle.

1. Start by creating an empty folder.
   mkdir /home/blank
2. Delete the target directory with rsync:

rsync --delete-before -a -H -v --progress --stats /tmp/linuxany /usr/local/nginx/proxy_temp

• 1

• This way the target directory is quickly emptied

Option Description.
-delete-before : The receiver performs the delete operation while transmitting.
-a : archive mode, which means transferring files recursively and maintaining all file attributes
-H: File to keep hardwired
-v : Detailed output mode
-progress : displays the progress of the transfer while it is in progress
-stats : Gives the status of certain file transfers.