Search. Read. Cite.

Easy to search. Easy to read. Easy to cite with credible sources.

Information Technology Journal

Year: 2014  |  Volume: 13  |  Issue: 3  |  Page No.: 501 - 507

A Job Scheduling Policy based on the Job-classification and Dynamic Replica Mechanism

Yan Qiang, Yue Li, Wei Wei, Bo Pei, Juanjuan Zhao and Hui Zhang

Abstract

In order to guarantee the stability of the system performance and the high Qos(quality of service) of users, a new method based on the HDFS (Hadoop Distributed File System) was proposed which including a job type classification method and a dynamic replica manage mechanism. The method uses the job type classification method to select the I/O intensive job, in order to achieve more accuracy of the classification taken the heterogeneity of the jobs into consideration. For the classified jobs, a dynamic replica manage mechanism was used to determine whether to increase or decrease the number of copies on the specific data node. For a test of a cluster with 1 namenode and 20 data nodes, the method has a high performance. The theoretical and experimental analyses in this paper prove that the proposed method has the ability to improve the performance of HDFS effectively.

Cited References Fulltext