Real-Time Data Processing in Mingantu Ultrawide Spectral Radio Heliograph Based on Spark Streaming

There is a growing demand for real-time processing in astronomical observations in recent years, meanwhile, the requirement for performance is also increasing dramatically. Mingantu Ultrawide Spectral Radio Heliograph (MUSER) is a synthetic aperture radio interferometer with high temporal, spatial and spectral resolution. In daily observation of low frequency, MUSER contains two aspects of data processing, historical data processing and sampling observational data which is produced every 5 seconds and processed in real-time mode. The procedure of raw data processing contains validation, correction, clean and other processing steps, then the results need to be transmitted in real-time mode to monitoring end without user constantly refreshing or sending a request. The traditional stand-alone processing mode has been unable to meet the requirements of large amounts of data in real-time mode. In this paper, we explored the use of Spark Streaming in a new approach for MUSER real-time calculations across multiple machines and evaluated its effectiveness and efficiency. A customized receiver was created for real-time binary stream of MUSER. We also extended the Spark cluster by adding multiple GPU's nodes. The experiments have shown that Spark Streaming can significantly improve MUSER real-time processing performance for its memory-based execution engine. We might look forward to optimize the algorithm through experiments and configurations so as to obtain better results, and apply it to the actual environment of MUSER finally.