Constructing Data Nodes of the China-VO with the MapReduce

The MapReduce is a distributed parallel processing model and execution environment for processing large data sets. It was initially applied to handle massive data in web service, but its applications have been extended to a variety of areas. A current project of Virtual Observatory may face an increasingly massive amount of astronomical data from ground-based and space telescopes. In order to improve the processing capacity of the astronomical data center in the China Virtual Observatory, this paper proposes a new approach to construct data nodes using the MapReduce. It translates an astronomical query to a standard SQL query, and then turns the query into a MapReduce job. It finally outputs the results in the standard formats of astronomical data. The MapReduce is integrated into the China Virtual Observatory by using the above three steps. Because cross-identifying between object catalogs takes place only once,the main consumed time in the MapReduce is in indexing and calculating data. We implement object cross-identification based on the MapReduce framework and our performance evaluation shows that the MapReduce-based cross-identification outperforms the traditional approach based on DBMS. Our results also show that the MapReduce-based framework achieves not only good performance but also scalability and low cost.