Summary of HADOOP Tuning and Ops Issues

I. Hadoop Tuning

1. Adjust the upper limit of the operating system to open file descriptors

The command "ulimit -a" allows you to see all of thesystem resourceParameters, here you need to focus on setting "open files" and "max user processes", you can set other discretionary settings.

To permanently set the resource parameters, the main way to do this is through the following files:

/etc/security/limits.conf 




/etc/security/limits.d/90-(centos6.x)




/etc/security/limits.d/20-(centos7.x)

2、Modify the parameters

This kernel parameter corresponds to the specificfile pathis /proc/sys/net/core/somaxconn, which is used to set the backlog limit for socket listening (listen).

What is backlog?

A backlog issocketWhen a request has not yet been processed or established, it enters the backlog. socketserverAll requests in the backlog can be processed at once, and the processed requests are no longer in the listening queue. If the server is slow in processing requests, new requests will be rejected when the listener queue is full. So this value must be increased. The default value of this parameter is 128.

As a basic optimization of the network parameters, it is recommended to modify them to the following values:

echo 4096 >/proc/sys/net/core/somaxconn

3. Adjust the proportion of the operating system using swap

swap is intended to be used as an extension of physical memory, but in today's abundant memory, the use of swap is becoming less and less scenarios, mainly because the use of swap will greatly reduce the performance of the application, in hadoop, if the data is exchanged to swap, it will lead to the operation of timeout, which very much affects the read and write of hadoop, as well as the performance of data analysis.

The percentage of swap usage can be adjusted with the kernel parameter /proc/sys/vm/swappiness. swappiness = 0 indicates that physical memory is used to the maximum extent possible before swap space, and swappiness = 100 indicates that the swap partition is used aggressively, and that the data in memory is moved to the swap space in a timely manner. to swap space in a timely manner.

linuxThe basic default setting of 60 indicates that your physical memory is used up to 100-60=40% when you start to see the use of having swap partitions. This value needs to be set small enough on some servers with high memory requirements, such as hadoop, redis, and hbase machines, and should be set between 0-10 to indicate maximum physical memory usage.

4. Disable THP (Transparent Huge Pages) function.

THP is intended to improve the performance of memory, but it is found in hadoop environments that this feature can bring about an increase in CPU usage, affecting hadoop performance, so it is recommended to turn it off.

First check the THP enable status:

[root@localhost ~]# cat /sys/kernel/mm/transparent_hugepage/defrag



[always] madvise never



[root@localhost ~]# cat /sys/kernel/mm/transparent_hugepage/enabled



[always] madvise never

Always is displayed here, indicating that THP is currently enabled.

To disable THP, open the /etc// file and add the following:

if test -f /sys/kernel/mm/transparent_hugepage/enabled; then




echo never > /sys/kernel/mm/transparent_hugepage/enabled



fi




if test -f /sys/kernel/mm/transparent_hugepage/defrag; then




echo never > /sys/kernel/mm/transparent_hugepage/defrag



fi

Then save and exit, and finally, give the file execute permission and execute the following command:

[root@localhost ~]# chmod +x /etc// 



[root@localhost ~]# source /etc/

5, big data platform network architecture design

Hadoop base network architecture:

6. Hardware selection for big data platforms

In a typical Hadoop architecture, there are usually five roles, namely NameNode (and Standby NameNode), ResourceManager, NodeManager, DataNode, and peripheral machines. Among them, NameNode is responsible for coordinating data storage on the cluster, ResourceManager is responsible for coordinating compute analysis, and Standby NameNode belongs to the hot backup of NameNode, which belongs to the management roles, and generally need to be deployed on independent servers, while NodeManager and DataNode roles are mainly used for computation and storage, NodeManager and DataNode are usually deployed together for better performance.

It is recommended to choose a unified hardware configuration for NameNode, ResourceManager and its Standby NameNode nodes, and the recommended base configuration is as follows:

• CPU: Recommended 2-way 8-core, 2-way 10-core or 2-way 12-core, etc., with a main frequency of at least 2-2.5GHz;
• Memory: 64-256GB recommended;
• Disk: Divided into 2 groups, system disk and data disk, system disk 2T * 2, do raid1, data disk 2-4T or so, the number of data disks depends on the number of copies of metadata you want to redundantly backup;
• NIC: 10Gb/s NIC (fiber optic card);
• Power supply: All configured with redundant power supply;

Here are some more common and mainstream enterprise NodeManager and DataNode node server hardware configurations, as shown in the table below:

• CPU: Recommended 2-way 10-core, 2-way 12-core or 2-way 14-core, etc., with a main frequency of at least 2-2.5GHz;
• Memory: 64-512GB recommended;
• Disk: divided into 2 groups, system disk and data disk, system disk 2T * 2, do raid1, data disk 4-8T or so, data disk single disk use, no need to do raid;
• Network card: 10 Gigabit network card (fiber optic card), the more storage, the higher the network throughput requirements;
• Power supply: It is best to configure a redundant power supply, or use a single power supply if the budget is insufficient;

7, big data platform architecture design points

Typical big data platform topology:

Before building a big data platform, the first thing to consider is the required storage capacity, computing power, whether there is a need for real-time analysis, and the storage cycle of the data, and then the architectural design of the platform is based on these needs.

At the same time, the robustness of the platform should also be considered, for example, any one node downtime will not affect the normal use of the platform, any one disk damage will not lead to data loss.

8, big data platform storage, computing node planning

Here to one of our previous practical case to introduce, we currently have a data volume of 500TB, the daily data volume growth of about 2T, the data block copy of 3, all the data storage cycle of 2 years, according to this demand, you can calculate the number of storage nodes required.

• Storage space required for 2 years of data volume: (2*3)*(365*2)=4380TB;
• Total storage space required: 4380TB + (500*3) TB = 5880TB;

If you take a storage node with 12 4T hard disks, you need about (5880TB/48TB=147) 123 storage nodes, while if you use a storage node with 10 8T hard disks, you need about (5880TB/80TB=147) 74 storage nodes can be used.

The planning principle of the big data platform for hardware is: if you can know exactly what the storage and computing resources need, then just configure according to this demand, and if you can't accurately assess the storage and computing resource demand, then you must leave room for expansion, such as leaving enough cabinet locations, network interfaces, disk interfaces, etc. In practice, the storage capacity is generally very good at estimating but the computing resources are hard to predict. In practical applications, storage capacity is generally well predicted, but computing resources are difficult to predict, so leaving enough expansion interfaces is an issue that must be considered.

Second, HADOOP daily operation and maintenance issues summary

1, how to downsize a datanode node (cluster downsizing)

1. Modification of documentation

Locate the namenode node configuration file /etc/hadoop/conf/ file with the following options:

<property>    




<name></name>     




<value>/etc/hadoop/conf/hosts-exclude</value> 




</property> 

2. Modify the hosts-exclude file

existhosts-exclude add the hostname of the datanode that needs to be taken offline, do the following:

vi /etc/hadoop/conf/hosts-exclude 




172.16.213.188 

3. Refresh configuration

Execute the following command on the namenode as the hadoop user to refresh the hadoop configuration:

[hadoop@namenodemaster ~]$hdfs dfsadmin -refreshNodes

4. Checking for completion of offline

Check if the downlink is complete by executing the following command:

[hadoop@cdh5master ~]$hdfs dfsadmin -report

The web interface can also be accessed by looking at the NameNode's port 50070 to view the HDFS status, and you need to focus on the number of nodes retired as well as the number of blocks replicated and their progress.

2, a datanode node disk is bad how to do

If the disk of a datanode node fails, the node will not be able to write operations and cause the datanode process to exit. To address this problem, first check the corresponding parameter settings in the /etc/hadoop/conf/ file on the failed node, and remove the directory mount point corresponding to the failed disk.

Then, check the corresponding -dirs parameter setting in the /etc/hadoop/conf/ file at the failed node and remove the directory mount point corresponding to the failed disk. Finally, restart the datanode service and nodemanager service on this node.

3、Namenode server failure how to do

In an HDFS cluster, the Namenode host stores all the metadata information, if this information is lost, then the entire HDFS above the data will be unavailable, and if the Namenode server fails and can not be started, the solution is divided into two cases: if the Namenode does a highly available service, then after the failure of the main Namenode, the Namenode service will automatically switch to the standby Namenode, this process is automatic, without manual intervention.

If your Namenode does not do high availability service, then you can also resort to the SecondaryNameNode service, find the metadata information on the SecondaryNameNode host, and then start the Namenode service directly on this node can be, of course, this way may lose part of the data, because the SecondaryNameNode realizes the cold backup of Namenode.

As you can see, it is crucial to perform disaster recovery and backup for Namenode. In the production environment, it is recommended to realize highly available hot backup for Namenode through standby Namenode.

4, datanode storage node data tilt how to solve the problem

In HDFS clusters, disk corruption is commonplace, disk failure, our general strategy is to replace the new hard disk, and when the new hard disk is replaced, only the new data will be written to this hard disk, and the old data will not automatically balance the data over, so on and so forth, the more hard disks are replaced, the more unbalanced the data will be between the nodes, between the disks of each node.

How to solve this problem? At this point, you can use the Balancer program provided by hadoop to achieve a balanced state of the HDFS cluster, the execution of the command is as follows:

[hadoop@namenodemaster sbin]$ $HADOOP_HOME/bin/ –t 5%

Or execute the following command:

[hadoop@namenodemaster sbin]$ hdfs balancer -threshold 5

The -t parameter in this command is followed by the disk utilization deviation value for HDFS to reach a balanced state. If the node-to-node disk utilization deviation is less than 5%, then we consider the HDFS cluster to have reached a balanced state.

5、hadoop cluster how to achieve expansion

1. Deployment of hadoop environment on new nodes

After the new node is installed on the system, a series of operations have to be carried out, such as the basic system optimization settings, the deployment and installation of the hadoop environment, the installation of jdk, etc., and these basic tasks need to be completed beforehand.

2. Modification of documentation

Look at the /etc/hadoop/conf/ file on the namenode and find the following:

<property>




  <name></name>




  <value>/etc/hadoop/conf/hosts</value>





</property>

3. Modify the hosts file

Modify the /etc/hadoop/conf/hosts file on the namenode to add the hostname of the added node as follows:

vi /etc/hadoop/conf/hosts



slave0191. 

Finally, synchronize the configuration to the machines of all datanode nodes.

4. Bringing the configuration into effect

After adding a new node, to make namenode recognize the new node, you need to refresh the configuration on the namenode by performing the following:

[hadoop@namenodemaster ~]$hdfs dfsadmin -refreshNodes 

5. Starting dn services on new nodes

After completing the configuration on the namenode, you finally need to start the datanode service on the added node by performing the following:

[hadoop@  ~]$ hdfs --daemon start datanode

In this way, a new node is added to the cluster, hadoop this mechanism can be in the existing cluster does not affect the operation of the state, arbitrarily add or delete a node, very convenient.

6, found under the HDFS missing blocks, how to solve the

This problem also often occurs, and there will be data loss, because once the HDFS cluster missing blocks error, it means that there is metadata loss or corruption, to recover the difficulty, or basically can not be recovered, for this situation, the solution is to first execute the following command:

[hadoop@namenodemaster sbin]$ hdfs fsck /blocks-path/

This command will check the status of all the blocks under HDFS and give those files which have block loss or corruption, finally just execute the following command to delete those files:

[hadoop@namenodemaster sbin]$ hdfs fsck -fs hdfs://bigdata/logs/  -delete

The above is deleting this file on HDFS, so the metadata of this file is lost and cannot be recovered, so it can only be deleted.