A Study on Strong Consistency Replication Mechanism of Large-Size Astronomical Image Distributed Storage

The new generation of observational technology in astronomy is driving a growth explosion in observational data. However, the traditional direct attaching storage device can hardly meet the needs of real-time storage and maga data processing. Although the distributed storage system based on inexpensive hardware is an effective approach, the high failure rates of the hardware seriously limit the application of these distributed storage devices in scientific data storage. Therefore, the method of generating two or more full copies is becoming a practical method to guarantee the reliability of the data files. In this paper, aiming to the replication of the large-size (>1MB) flexible image transmission formats (FITS) astronomical data file, we present a replication algorithm for the file size from one to dozens of MB range. The algorithm sacrifices the storage delay for the acquirement of the strong consistency with multiple copies in the conditions of multiple hardware and software exceptions. The experimental result shows the validity of the algorithm.