Piecing the Masterpiece Together: A Framework for Cross-Modality Art Alignment and Registration

A masterpiece painting is a complex structure of historical layers, from underdrawings to surface pigments. Studying the key stages of artwork’s creation requires combining complementary data from diverse modalities, including Hyperspectral (HSI), Infrared (IR), and X-ray Fluorescence (XRF). However, acquisition is often restricted to localized patches requiring image stitching. Moreover, severe cross-modality variations - such as contrast inversions, varying perspectives, divergent fields of view, and modality-specific artifacts (e.g., underpainting detected solely via XRF) - make precise registration challenging.

To overcome these obstacles, we propose a two-stage multimodal registration framework. First, we achieve robust parametric alignment under illumination changes by maximizing a photometrically invariant directional correlation measure. Second, we leverage self-supervised deep learning architecture for robust geometric matching across divergent sensor perspectives. We showcase our methodology’s ability to mosaic and register several 14th-century masterpieces from the National Gallery of Washington.