A Data Processing Acceleration Method and System for FAST Petabyte Pulsar Data Processing

Zhang Hui; Xie Xiaoyao; Li Di; Liu Zhijie; Wang Pei; Yu Xuhong; You Shanping; Xu Yuyun; Jiang Jiatao

doi:10.14005/j.cnki.issn1672-7673.20200628.001

Astronomical Research and Technology > 2021 > 18(1): 129-137. > DOI: 10.14005/j.cnki.issn1672-7673.20200628.001 CSTR: 32083.14.j.cnki.issn1672-7673.20200628.001

Zhang Hui, Xie Xiaoyao, Li Di, Liu Zhijie, Wang Pei, Yu Xuhong, You Shanping, Xu Yuyun, Jiang Jiatao. A Data Processing Acceleration Method and System for FAST Petabyte Pulsar Data Processing[J]. Astronomical Research and Technology, 2021, 18(1): 129-137. DOI: 10.14005/j.cnki.issn1672-7673.20200628.001

Citation:

PDF (2944 KB)

A Data Processing Acceleration Method and System for FAST Petabyte Pulsar Data Processing

Zhang Hui^1,3,4,
Xie Xiaoyao^1,4, ,,
Li Di^2,5,
Liu Zhijie^1,4,
Wang Pei²,
Yu Xuhong^1,4,
You Shanping^1,4,
Xu Yuyun⁶,
Jiang Jiatao^1,3,4

1. Guizhou Key Laboratory of Information and Computing Science, Guizhou Normal University, Guiyang 550001, China;
2. National Astronomical Observatories, Chinese Academy of Sciences, Beijing 100101, China;
3. School of Mathematics and Sciences, Guizhou Normal University, Guiyang 550001, China;
4. FAST Early Science Data Center, Guiyang 550001, China;
5. University of Chinese Academy of Sciences, Beijing 100049, China;
6. Institute of Management Engineering, Guizhou Vocational and Technical College of Water Resources and Hydropower, Guiyang 551416, China

More Information

Received Date: May 05, 2020
Revised Date: May 18, 2020
Available Online: November 20, 2023

Graphical Abstract

Abstract

Abstract

The Five-hundred-meter Aperture Spherical radio Telescope (FAST) has started normal science operation. Data collected by drift scan pulsar survey has exceeded 1 PB, and it is expected to further increase by at least 5 PB per year. Existing pulsar search software, such as PRESTO, SIGPROC, and etc., cannot meet the real-time data analysis and management requirements. How to efficiently process PB volume of data has become a new challenge in the field of radio astronomy. In order to tackle the problems of PB data analysis and data management encountered by FAST, we, the joint team from Guizhou Normal University (GZNU) and the National Astronomical Observatories (NAOC), designed and implemented a PRESTO-based, distributed-parallel-computing system, named Craber, which integrated network technology, database, and cross-regional hardware computing resources. Craber performed well on data sets both from the Parkes Multibeam Pulsar Survey (PMPS) and the Commensal Radio Astronomy FAST Survey (CRAFTS). A 100 MB Parkes data file took ~36 seconds by 55 computing nodes in sub-cluster D of Craber, while a 128 MB data file from CRAFTS cost ~22 seconds. Up to date, Craber processed more than 66 000 data files from FAST, helped FAST detect more than 140 high-quality candidates, 114 of which have been confirmed. All resulting data products were then stored into the integrated Oracle database or dedicated file server, ready for further candidates selection with AI. Craber has already helped FAST speed up its data processing substantially and discovered a number of new pulsars.
- Pulsar,
- Acceleration system,
- Search database,
- Data processing,
- FAST

FullText(HTML)

References (14)

References

[1]	HEWISH A, BELL S J, PILKINGTON J D H, et al. Observation of a rapidly pulsating radio source[J]. Nature, 1968, 217:709-713.
[2]	KHOO J, HOBBS G, MANCHESTER R N, et al. Using the Parkes Pulsar Data Archive[J]. Astronomical Research & Techonolgy——Publications of National Astronomical Observatories of China, 2012, 9(3):229-236.
[3]	MANCHESTER R N, LYNE A G, CAMILO F, et al. The Parkes multi-beam pulsar survey-I. observing and data analysis systems, discovery and timing of 100 pulsars[J]. Monthly Notices of the Royal Astronomical Society, 2001, 328(1):17-35.
[4]	LI D, WANG P, QIAN L, et al. FAST in space:considerations for a multibeam, multipurpose survey using China's 500-m Aperture Spherical Radio Telescope (FAST)[J]. IEEE Microwave Magazine, 2018, 19(3):112-119.
[5]	LI D, PAN Z C. The Five-hundred-meter Aperture Spherical Radio Telescope project[J]. Radio Science, 2016, 51(7):1060-1064.
[6]	LORIMER D R, KRAMER M. Handbook of pulsar astronomy[M]. UK:Cambridge University Press, 2004:126-164.
[7]	潘之辰, 钱磊, 岳友岭. 脉冲星搜索技术及FAST望远镜脉冲星搜索展望[J]. 天文研究与技术, 2017, 14(1):8-16. PAN Z C, QIAN L, YUE Y L. Pulsar searching techniques and their applications to FAST pulsar search[J]. Astronomical Research & Technology, 2017, 14(1):8-16.
[8]	RANSOM S M, CORDES J M, EIKENBERRY S S. A new search technique for short orbital period binary pulsars[J]. The Astrophysical Journal, 2002, 589(2):911-920.
[9]	LORIMER D. SIGPROC:pulsar signal processing programs[EB/OL]. (2008-06-02). http://sigproc.sourceforge.net/sigproc.pdf.
[10]	黄玉祥, 汪敏, 郝龙飞, 等. 脉冲星信号相干消色散与非相干消色散的比较研究[J]. 天文研究与技术, 2019, 16(1):16-24. HUANG Y X, WANG M, HAO L F, et al. Comparative study between the coherent de-dispersion and the incoherent de-dispersion of pulsar signal[J]. Astronomical Research & Technology, 2019, 16(1):16-24.
[11]	中国科学院国家天文台. 国家天文台举办FAST首批成果新闻发布会[EB/OL]. (2017-10-10). http://www.bao.ac.cn/xwzx/zyxw/201710/t20171010_4871382.html. National Astronomical Observatories, Chinese Academy of Sciences. National Astronomical Observatories holds press conference on the first batch of FAST achievements[EB/OL]. (2017-10-10). http://www.bao.ac.cn/xwzx/zyxw/201710/t20171010_4871382.html.
[12]	QIAN L, PAN Z C, LI D, et al. The first pulsar discovered by FAST[J]. Science China:Physics, Mechanics & Astronomy, 2019, 62(5):959508.
[13]	CAMERON A D, LI D, HOBBS G, et al. An in-depth investigation of 11 pulsars discovered by FAST[J]. Monthly Notices of the Royal Astronomical Society, 2020, 495(3):3515-3530.
[14]	ZHANG L, LI D, HOBBS G, et al. PSR J1926-0652:a pulsar with interesting emission properties discovered at FAST[J]. The Astrophysical Journal, 2019, 877(1):55.