Award

3D models are commonly generated from both multiview satellite and FMV sources using photogrammetric methods. However, such models lack semantic labels (e.g. segmentation of buildings, roads, vegetation, vehicles, etc.) needed for further analytics. Prior work relies largely on discriminative models to classify 3D surface points using local context, but they also have difficulty learning complex relationships between objects that generalize to new domains. Recent work in Vision-Language Foundation Models (VLMs) trained on a combination of imagery and language has shown great power in modeling and generating large-scale imagery consistently and meaningfully. It has also been shown that generative models used in semantic segmentation can generalize better to new domains than discriminative models. Kitware proposes to leverage these recent findings to provide semantic segmentation of 3D surfaces via the fusion of multiple generated 2D segmentation maps. We will leverage the fact that these 3D models are derived from 2D imagery to adapt and fine-tune existing VLMs to generate semantically meaningful segmentation in 2D before fusing into consistent 3D surface labels. The result will be 3D segmentation and detection results that generalize far better to new environments.