You are here

Deriving Uncertainty Estimates for Automated Observations


TECHNOLOGY AREA(S): Info Systems, Human Systems 

OBJECTIVE: Develop novel ways to assess uncertainty in observations of objects in overhead imagery that were identified by automated systems. Design aggregation techniques that can identify important spatial groupings while considering the uncertainty estimates for the primitives of those groupings. 

DESCRIPTION: Recent advances in deep learning have made object recognition and localization systems applicable to a wider variety of tasks. These techniques have started to be applied to overhead imagery; this provides great promise for automated recognition of objects for analysts. As such, this topic has two research components: (1) to enhance an object detection network to be able to describe its own certainty in its identifications; and (2) to develop methods to aggregate observations based on models that consider individual objects’ uncertainty. For (1), the standard technique for these recognition systems is to use algorithms to classify objects into a pre-defined set of categories. For instance, a network that has been trained to identify cats and dogs, given an image of a chair, will categorize it as a cat or a dog without providing a measure of certainty in its judgment. However, analysts always provide measures of their confidence in their judgments. Automated identification of objects in overhead imagery will require methods to produce measures of uncertainty and to process multiple observations to reduce uncertainty. For (2), the research will need to provide example methods and models to identify salient groups of objects and classify activities based on a reasoned, a priori knowledge-base of a given environment codified in the model. Object configurations can suggest a larger class of objects. For instance, rows of cars in a linear formation indicate a parking lot. Configurations of objects can also indicate activity; e.g., a crane over a cargo container ship indicates active on- or off-loading. Spatial data aggregation that employs certainty estimates should be able to ingest models that are easily represented in order to increase accuracy and applicability of judgments suggested by automated systems. The National Geospatial-Intelligence Agency searches for novel approaches that combine these two concepts: uncertainty estimates and spatial object grouping. An object recognition system requires certainty estimates in order to provide the necessary context for future reasoning tasks that analysts currently conduct as a uniquely human task. Spatial data aggregation from an automated recognition system must understand how certain that system is of a particular judgment in order to make its own assessment of the grouping or activity of those objects. 

PHASE I: Demonstrate an ability to obtain uncertainty estimates in neural architectures applied to overhead imagery. Design proof-of-concept for algorithmic approaches to object aggregation and/or group activity assessment to include uncertainty information. 

PHASE II: Demonstrate an aggregation/activity identification algorithm applied to datasets of images and pre-defined object identifications with certainty estimates on the identifications. Jointly with the government develop test metrics for performance evaluation of uncertainty measures and aggregation techniques. 

PHASE III: This system could be used for a broad range of military and civilian applications where the presence of particular objects in groups indicate a particular activity – for example, in monitoring of shipping or commercial business operations. 


1: Yarin Gal and Zoubin Ghahramani. Bayesian convolutional neural networks with Bernoulli approximate variational inference. arXiv 1506.02142v5, 2016.

2:  Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation appendix. arXiv 1506.02157, 2015.

3:  Alex Graves. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems 24 25th Annual Conference on Neural Information Processing Systems 2011.

4:  D.J.C. MacKay. A practical Bayesian framework for backpropagation networks. Neural Computation, 4(3):448-472, May 1992.

5:  John William Paisley, David M. Blei, and Michael I. Jordan. Variational Bayesian inference with stochastic search. In Proceedings of the 29th International Conference on Machine Learning, ICML, 2012.

KEYWORDS: Computer Vision; Overhead Imagery; Machine Learning; Spatial Data; Aggregation 


Samuel Dooley 

(571) 557-7312 

US Flag An Official Website of the United States Government