rsync+inotify configuration, scripting, and demo examples

rsync+inotify

Compared with the traditional cp, tar backup method, rsync has the advantages of high security, rapid backup, support for incremental backup, etc. Through rsync can solve the real-time requirements of the data backup needs are not high, such as regular backup file server data to a remote server, the local disk regularly do data mirroring.
With the continuous expansion of the scale of the application system, the security and reliability of the data also put forward better requirements, rsync in the high-end business system also gradually exposed a lot of shortcomings, first of all, rsync synchronization of data, you need to scan all the files after the comparison, the difference in the amount of transmission. If the number of files reaches millions or even tens of millions, scanning all the files will be very time-consuming. And what is changing is often a very small part of it, which is very inefficient way. Secondly, rsync can't monitor and synchronize data in real time, although it can do so through thelinuxDaemon way to trigger synchronization, but the two trigger action must have a time difference, which leads to the server and client data may be inconsistent, can not be in the application failure to restore data completely. For these reasons, the rsync+inotify combination appears!

Inotify is a powerful, fine-grained, asynchronous file system event monitoring mechanism, linux kernel from 2.6.13 onwards, joined the Inotify support, through the Inotify can monitor the file system to add, delete, modify, move, and other subtle events, the use of the kernel interface, the third-party software will be able to monitor a variety of changes in the file system files under the situation. Using this kernel interface, third-party software can monitor file changes under the file system, and inotify-tools is such a third-party software.
As mentioned earlier, rsync can achieve triggered file synchronization, but triggered by the crontab daemon, the synchronized data and the actual data will be different, and inotify can monitor various changes in the file system, when there is any change in the file, it triggers the rsync synchronization, which just solves the real-time synchronization data problem.

Demonstration

Environmental Description:

demand (economics)

Deploy rsync+inotify to synchronize the /runtime directory to the target server under /NAME/. Here NAME is your name, for example, if your name is tom, synchronize the /runtime directory to the target server under /tom/.

Target server operation.

Disable firewall and selinux

[root@target ~]# systemctl stop firewalld
[root@target ~]# setenforce 0

• 1
• 2

Install rsync and set up configuration files

[root@target ~]# dnf install -y rsync
[root@target ~]# vi /etc/
[root@target ~]# cat /etc/
log file = /var/log/    # The location of the log file, which is automatically generated when rsync is started, so there is no need to create it in advance.
pidfile = /var/run/     # where the pid file is stored
lock file = /var/run/   # Lock files that support the max connections parameter
secrets file = /etc/    # User authentication profile, which holds user names and passwords and must be created manually

[yangzhenyu]     # Customized synchronization names
path = /yzy/          # rsync server data storage path, client data will be synchronized to this directory
comment = sync etc from client
uid = root        # Set rsync runtime privileges to root
gid = root        # Set rsync runtime privileges to root
port = 873        # Default port
ignore errors     # Indicates an error Ignore error
use chroot = no       # Default is true, change to no to add backup of catalog file softlinks
read only = no    # Set the rsync server to read and write permissions
list = no     # Do not show the rsync server resource list
max connections = 200     # of maximum connections
timeout = 600     # Set the timeout period
auth users = admin        # Usernames to perform data synchronization, can be set more than one, separated by commas in English state
hosts allow = 192.168.10.40   # Client IP addresses allowed for data synchronization, more than one can be set, separated by commas in English state
hosts deny = 192.168.1.1      # Client IP addresses that prohibit data synchronization, multiple can be set, separated by commas in the English state

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23

Create user authentication files and set file permissions

[root@target ~]# echo 'admin:1' > /etc/ 
[root@target ~]# cat /etc/ 
admin:1

[root@target ~]# chmod 600 /etc/rsync*
[root@target ~]# ll /etc/rsync*
-rw-------. 1 root root 1389 6moon7 06:19 /etc/
-rw-------. 1 root root   13 6moon7 03:18 /etc/

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8

Newly created and written documents (becausecentos (This file is not in 8, but comes with centos 7.)

[root@target ~]# vim /usr/lib/systemd/system/
[root@target ~]# cat /usr/lib/systemd/system/ 
[Unit]
Description=fast remote file copy program daemon

[Service]
User=root
Group=root
EnvironmentFile=/etc/sysconfig/rsyncd
ExecStart=/usr/bin/rsync --daemon --config=/etc/ --no-detach
ExecReload=/bin/kill -HUP $MAINPID
KillMode=process
Restart=on-failure
RestartSec=30s

[Install]
WantedBy=

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17

Start the rsync service and set it to start on boot

[root@target ~]# systemctl daemon-reload
[root@target ~]# systemctl status rsyncd
●  - fast remote file copy program daemon
   Loaded: loaded (/usr/lib/systemd/system/; disabled; vendor prese>
   Active: inactive (dead)
[root@target ~]# systemctl enable --now rsyncd
Created symlink /etc/systemd/system// → /usr/lib/systemd/system/.
[root@target ~]# ss -antl
State    Recv-Q   Send-Q     Local Address:Port     Peer Address:Port   Process   
LISTEN   0        5                0.0.0.0:873           0.0.0.0:*                
LISTEN   0        128              0.0.0.0:22            0.0.0.0:*                
LISTEN   0        5                   [::]:873              [::]:*                
LISTEN   0        128                 [::]:22               [::]:*             

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13

Source Server Operation.

Disable firewall and selinux

[root@source ~]# systemctl stop firewalld 
[root@source ~]# setenforce 0

• 1
• 2

Install rsync, just install, no startup required

[root@source ~]# dnf install -y rsync

• 1

Create an authentication password file, set the file permissions, and only set the file owner to have read and write permissions.

[root@source ~]# echo '1' > /etc/
[root@source ~]# cat /etc/
1

[root@source ~]# chmod 600 /etc/ 
[root@source ~]# ll /etc/ 
-rw-------. 1 root root 2 6moon7 07:03 /etc/

• 1
• 2
• 3
• 4
• 5
• 6
• 7

beta (software)

Create a directory on the target server

[root@target ~]# cd /
[root@target /]# mkdir yzy

• 1
• 2

Create a test directory on the source server and then do the following on the source server

[root@source ~]# cd /
[root@source /]# mkdir runtime
[root@source /]# cd runtime
[root@source runtime]# touch aa bb
[root@source runtime]# rsync -avH --port 873 --progress --delete /runtime/ [email protected]::yangzhenyu --password-file=/etc/
sending incremental file list
./
aa
              0 100%    0.00kB/s    0:00:00 (xfr#1, to-chk=1/3)
bb
              0 100%    0.00kB/s    0:00:00 (xfr#2, to-chk=0/3)

sent 170 bytes  received 65 bytes  470.00 bytes/sec
total size is 0  speedup is 0.00

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15

View test results on the target server

[root@target /]# ls /yzy
aa  bb

• 1
• 2

Build an epel repository on the source server, install inotify-tools

[root@source ~]# vim /etc//
[root@source ~]# cat /etc// 
[epel]
name=EPEL
baseurl=/epel/$releasever/Everything/$basearch
enabled=1
gpgcheck=0

[root@source ~]# yum install -y inotify-tools
a1                                                258 MB/s | 2.4 MB     00:00    
a2                                                208 MB/s | 6.3 MB     00:00    
EPEL                                              153 kB/s | 9.5 MB 01:03
Dependency resolution.
==================================================================================
 Package Architecture Version Repository Size
==================================================================================
Installation: inotify-tools x86_64
 inotify-tools x86_643.14-19.el8           epel           57 k

Summary of transactions
==================================================================================
mounting1 software package

Total downloads: 57 k
Installation size: 120 k
Download package:
inotify-tools-3.14-19.el8.x86_64.rpm145 kB/s |  57 kB 00:00
----------------------------------------------------------------------------------
Total143 kB/s |  57 kB 00:00
Running a transaction check
The transaction check was successful.
Running Transaction Test
Transaction test succeeded.
Running Transaction
  Preparing:                                                                   1/1
  Installation: inotify-tools-3.14-19.el8.x86_64                                  1/1
  Run script: inotify-tools-3.14-19.el8.x86_641/1
  validate (a theory): inotify-tools-3.14-19.el8.x86_64                                  1/1
Installed products updated.

installed:
  inotify-tools-3.14-19.el8.x86_64

end！

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22
• 23
• 24
• 25
• 26
• 27
• 28
• 29
• 30
• 31
• 32
• 33
• 34
• 35
• 36
• 37
• 38
• 39
• 40
• 41
• 42
• 43
• 44

Writing synchronization scripts on the source server

[root@source ~]# mkdir /scripts
[root@source ~]# touch /scripts/
[root@source ~]# chmod 755 /scripts/
[root@source ~]# ll /scripts/
-rwxr-xr-x. 1 root root 0 6moon7 08:58 /scripts/
[root@source ~]# vim /scripts/
[root@source ~]# cat /scripts/ 
host=192.168.10.20      # ip of target server (backup server)
src=/runtime        # Backup directory to be monitored on the source server (this can be customized, but make sure it exists)
des=yangzhenyu     # Customized module name, which needs to match the synchronization name defined on the target server
password=/etc/        # Password file to perform data synchronization
user=admin          # Username to perform data synchronization
inotifywait=/usr/bin/inotifywait

$inotifywait -mrq --timefmt '%Y%m%d %H:%M' --format '%T %w%f%e' -e modify,delete,create,attrib $src \
| while read files;do
    rsync -avzP --delete  --timeout=100 --password-file=${password} $src $user@$host::$des
    echo "${files} was rsynced" >>/tmp/ 2>&1
done

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19

View files in the yzy directory under the target server

[root@target /]# ls yzy
aa  bb

• 1
• 2

Start the script and create a new file in the runtime directory to trigger the script

[root@source ~]# nohup bash /scripts/ &
[5] 136777
[root@source ~]# nohup: ignoring input and appending output to ‘’
//If you are using finalshell, it will display Chinese.
[root@source ~]# nohup: ignore input and append output to ''

[root@source ~]# ps -ef|grep inotify
root       81788    5908  0 08:45 pts/0    00:00:00 /usr/libexec/platform-python /usr/bin/dnf install -y inotify-tools
root       92486    5908  0 08:52 pts/0    00:00:00 /usr/libexec/platform-python /usr/bin/yum install -y inotify-tools
root      136777    5908  0 09:11 pts/0    00:00:00 bash /scripts/
root      136778  136777  0 09:11 pts/0 00:00:00 /usr/bin/inotifywait -mrq --timefmt %Y%m%d %H:%M --format %T %w%f%e -e modify,delete,create,attrib /runtime // Seeing this is success!
root136779  136777  0 09:11 pts/0    00:00:00 bash /scripts/
root      137730    5908  0 09:11 pts/0    00:00:00 grep --color=auto inotify

[root@source ~]# ls /runtime/
aa  bb
[root@source ~]# touch /runtime/ttt

//View log files
[root@source ~]# tail /tmp/ 
20210607 09:17 /runtime/tttCREATE was rsynced
20210607 09:17 /runtime/tttATTRIB was rsynced

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15
• 16
• 17
• 18
• 19
• 20
• 21
• 22

Setting up script boot-up on the source server

[root@source ~]# chmod +x /etc//
[root@source ~]# ll /etc//
-rwxr-xr-x. 1 root root 474 12moon1 2020 /etc//
[root@source ~]# echo 'nohup /bin/bash /scripts/' >> /etc//
[root@source ~]# tail /etc//
#
# In contrast to previous versions due to parallel execution during boot
# this script will NOT be run after all other services.
#
# Please note that you must run 'chmod +x /etc//' to ensure
# that this script will be executed during boot.

touch /var/lock/subsys/local

nohup /bin/bash /scripts/

• 1
• 2
• 3
• 4
• 5
• 6
• 7
• 8
• 9
• 10
• 11
• 12
• 13
• 14
• 15

Check on the target server to see if the script was successfully triggered and the files were synchronized

[root@target /]# ls yzy
aa  bb  runtime
[root@target /]# ls /yzy/runtime
aa  bb  ttt

• 1
• 2
• 3
• 4