The Study of the Data Storage and Retrieval for the Massive Data of MUSER Based on Cassandra

Shi Yue; Wang Feng; Li Pengcheng; Liu Yingbo

Astronomical Research and Technology > 2018 > 15(3): 361-368.

Shi Yue, Wang Feng, Li Pengcheng, Liu Yingbo. The Study of the Data Storage and Retrieval for the Massive Data of MUSER Based on Cassandra[J]. Astronomical Research and Technology, 2018, 15(3): 361-368.

Citation:

PDF (3660 KB)

The Study of the Data Storage and Retrieval for the Massive Data of MUSER Based on Cassandra

1. Key Laboratory of Applications of Computer Technology of the Yunnan Province, Kunming University of Science and Technology, Kunming 650500, China;
2. Yunnan Observatories, Chinese Academy of Sciences, Kunming 650011, China

More Information

Received Date: November 26, 2017
Revised Date: January 04, 2018
Available Online: November 20, 2023

Graphical Abstract

Abstract

Abstract

Mingantu Ultrawide Spectral Radioheliograph produces massive observational data every day. When the data is stored and managed by traditional relational database, problems appear, such as high latency in reading and writing, limited expansion capacity in performance and storage, and weak usability. In order to solve the problems, in this paper, researches on the application of MUSER data storage based on NoSQL has been carried out. First of all, we thoroughly analyze the characteristics of MUSER data, the requirement of storage and relating problems in detail. Secondly, the MUSER data model is given as well as the MUSER column non-relational data model. The metadata and data are synchronized in NoSQL based on Cassandra. The MUSER massive astronomical data storage management system (MBDMS) is realized based on Cassandra. The experiment validates the efficiency, expansibility and feasibility of data storage and retrieval of the system. As a result, MBDMS can well meet MUSER data management needs, is an effective program to solve the current MUSER data storage problem.
- Non-relational database,
- Storage,
- Searching

FullText(HTML)

References (11)

References

[1]	梅盈, 刘东浩, 王锋, 等. 中国频谱射电日像仪FITS-IDI文件格式研究[J]. 天文研究与技术——国家天文台台刊, 2014, 11(4):388-395. Mei Ying, Liu Donghao, Wang Feng, et al. A study of the FITS-IDI Format for the Chinese Spectral Radio Heliograph[J]. Astronomical Research & Techonolgy——Publications of National Astronomical Observatories of China, 2014, 11(4):388-395.
[2]	梅盈. MUSER海量数据预处理关键技术研究[D]. 昆明:昆明理工大学, 2015.
[3]	Shi C M, Wang F, Deng H, et al. High performance negative database for massive data management system of the Mingantu Spectral Radioheliograph[J]. Publications of the Astronomical Society of the Pacific, 2017, 129(978):1-12.
[4]	刘应波, 王锋, 季凯帆, 等. 基于NoSQL的FITS文件头元数据存储和查询研究[J]. 计算机应用研究, 2015, 32(2):461-465. Liu Yingbo, Wang Feng, Ji Kaifan, et al. Research on metadata storage and querying of FITS file based on NoSQL[J]. Application Research of Computers, 2015, 32(2):461-465.
[5]	陈亚杰, 王锋, 邓辉, 等. ElasticSearch分布式搜索引擎在天文大数据检索中的应用研究[J]. 天文学报, 2016, 57(2):241-251. Chen Yajie, Wang Feng, Deng Hui, et al. The application of ElasticSearch in the massive astronomical data retrieval[J]. Acta Astronomica Sinica, 2016, 57(2):241-251.
[6]	唐李洋, 倪志伟, 李应. 基于Cassandra的可扩展分布式反向索引的构建[J]. 计算机科学, 2011, 38(6):187-190. Tang Liyang, Ni Zhiwei, Li Ying. Scalable distributed inverted index built on Cassandra[J]. Computer Science, 2011, 38(6):187-190.
[7]	侯朋朋. 一种高性能NoSQL存储系统的设计与实现[D]. 北京:中国科学院大学, 2013.
[8]	潘洪志. 高性能NOSQL存储系统的研究与实现[D]. 吉林:吉林大学, 2014.
[9]	林茂强. 基于NoSQL技术的MUSER数据归档与发布系统的设计与实现[D]. 昆明:昆明理工大学, 2015.
[10]	梁波, 陈腾达, 于孔林, 等. 分布式实时存储环境下的FITS数据一致性研究[J]. 天文研究与技术, 2016, 13(4):489-497. Liang Bo, Chen Tengda, Yu Konglin, et al. Research on the consistency of FITS data in the process of distributed real-time storage[J]. Astronomical Research & Techonolgy, 2016, 13(4):489-497.
[11]	胡超晔. 基于Cassandra数据库集群的高并发读写系统的分析和应用研究[D]. 上海:上海交通大学, 2013.