Author(s)

Wei Wang, Xin Liu, Yong Shi, Ning Tao, Chong Xu

Corresponding Author

Wei Wang

Abstract

In order to better verify that Hadoop performance can be improved through optimization of parameters, we can use the following test methods: benchmarking, stability testing, high availability testing, scalability testing, and security testing. In this paper, the benchmark test method is used to verify the optimization of parameters and to optimize the performance of Hadoop. This article mainly focuses on the 15 parameters in Tab.1. The optimization results are shown in Tab.3. The optimization of the parameters was verified by the execution time of the WordCount algorithm in the benchmark test. During the experiment, the CPU and memory utilization rate, disk IO and network IO throughput and other indicators were collected. Fig.4-6 fully illustrates the comparison between Hadoop and WordCount algorithm after parameter default value and parameter adjustment. The experimental results show that after the Hadoop parameters are adjusted and optimized, the Hadoop performance tuning is achieved under certain conditions.

Keywords

Hadoop, Word Count, Parameter Optimization, Performance Tuning