Automation and Parallelization Scheme to Accelerate Pulsar Observation Data Processing

Previous studies aiming to accelerate data processing have focused on enhancement algorithms, using the graphics processing unit to speed up programs, and thread-level parallelism. These methods overlook maximizing the utilization of existing central processing unit (CPU) resources and reducing human and computational time costs via process automation. Accordingly, this paper proposes a scheme, called SSM, that combines “Srun job submission mode,” “Sbatch job submission mode,” and “Monitor function.” The SSM scheme includes three main modules: data management; command management; and resource management. Its core innovations are command splitting and parallel execution. The results show that this method effectively improves CPU utilization and reduces the time required for data processing. In terms of CPU utilization, the average value of this scheme is 89%. In contrast, the average CPU utilizations of “Srun job submission mode” and “Sbatch job submission mode” are significantly lower, at 43% and 52%, respectively. In terms of the data-processing time, SSM testing on FAST data requires only 5.5 hours, compared with 8 hours in the “Srun job submission mode” and 14 hours in the “Sbatch job submission mode.” In addition, tests on the FAST and Parkes datasets demonstrate the universality of the SSM scheme, which can process data from different telescopes. The compatibility of the SSM scheme for pulsar searches is verified using 2 days of observational data from the globular cluster M2, with the scheme successfully discovering all published pulsars in M2.