Optimization Analysis of Hadoop

查看全文 作　　者：Jinglun [1]Li;Shengfei [1]Shi;Hongzhi [1]Wang 机构地区：[1]School of Computer Science and Technology,Harbin Institute of Technology,Harbin,China 出　　处：《国际计算机前沿大会会议论文集》索引2016年第1期,共2页 摘　　要：Hadoop is a distributed data processing platform supporting MapReduce parallel computing framework. In order to deal with general problems, there is always a need of accelerating Hadoop under certain circumstance such as Hive jobs. By outputting current time to logs at specially selected points, we traced the workflow of a typical MapReduce job generated by Hive and making time statistics for every phase of the job. Using different data quantities, we compared the proportion of each phase and located the bottleneck points of Hadoop. We make two major optimization advices: (1) focus on using combine and optimizing Net Work and Disk IO when dealing with big jobs having a large number of intermediate results; (2) optimizing map function and Disk IO when dealing with short jobs. 关 键 词：HADOOP Hive OPTIMIZATION