A Pre-research on GWAC Massive Catalog Data Storage and Processing System

GWAC (Ground Wide Angle Camera) poses huge challenges in large-scale catalogue storage and real-time processing of quick search of transients among wide field-of-view time-series data. Firstly, this paper proposes a concept to employ databases' functions such as fast data processing and parallelism, which will improve system performance and availability through the integration of data storage and computing platform. To understand the feasibility of Column-store MonetDB in vast catalogue management, we carry out a variety of pilot experiments on key technologies. We conduct TPC-H benchmark, data loading benchmark and optimization, and key algorithm testing of astronomical source association, all compared with the traditional row store database. Then, we use MonetDB to realize cross-match Zone algorithm. UDF function is developed for customizable data loading. Test results show that MonetDB database has a remarkable performance in big data management and it is efficient in real-time data processing: it has the ability to deal with 2.5T catalog data.In the end we propose a wide field of view massive time serial observation data processing solution using the in-memory column store database MonetDB. The experimental results confirm the feasibility of this scheme. The design plan of MonetDB-based massive catalogue data processing solution is an efficient astronomical database solution that combines data processing and data management.