Abstract: To address the challenges of high-precision optical surface defect detection, we propose a novel design for a wide-field and broadband light field camera in this work. The proposed system can achieve a 50° field of view and operates at both visible and near-infrared wavelengths. Using the principles of light field imaging, the proposed design enables 3D reconstruction of optical surfaces, thus enabling vertical surface height measurements with enhanced accuracy. Using Zemax-based simulations, we evaluate the system’s modulation transfer function, its optical aberrations, and its tolerance to shape variations through Zernike coefficient adjustments. The results demonstrate that this camera can achieve the required spatial resolution while also maintaining high imaging quality and thus offers a promising solution for advanced optical surface defect inspection.