You are here

SHAPE-BASED GENERALIZATION BOUNDS FOR DEEP LEARNING

Award Information
Agency: Department of Defense
Branch: National Geospatial-Intelligence Agency
Contract: HM047620C0075
Agency Tracking Number: NGA-P1-20-19
Amount: $99,923.00
Phase: Phase I
Program: STTR
Solicitation Topic Code: NGA20A-001
Solicitation Number: 20.A
Timeline
Solicitation Year: 2020
Award Year: 2020
Award Start Date (Proposal Award Date): 2020-09-30
Award End Date (Contract End Date): 2021-07-04
Small Business Information
636 Rock Creek Road
Chapel Hill, NC 27514-1111
United States
DUNS: 078652742
HUBZone Owned: No
Woman Owned: No
Socially and Economically Disadvantaged: No
Principal Investigator
 Jay Hineman
 (623) 261-7734
 jay.hineman@geomdata.com
Business Contact
 Megan Hohenstein
Phone: (919) 670-0808
Email: megan.hohenstein@geomdata.com
Research Institution
 Duke University
 Keith Owen
 
2200 West Main Street
Durham, NC 27705-4677
United States

 (919) 681-8687
 Nonprofit college or university
Abstract

We propose to develop a theoretical understanding of the relationship between intrinsic geometric structure in both training and latent data and characteristics of functions learned from that data for deep neural network (DNN) architectures. Along the way we propose to also understand the structure of the neural networks that are best trained on a given data set. Both of these theories will lead to generalization bounds in the form of theorems that will be tested rigorously and at scale, using both synthetic datasets and real imagery datasets. NGA’s mission is to provide timely, relevant, and accurate geospatial intelligence in support of national security. We will develop partnerships that will test the theory at the points of greatest value to NGA, as well as to potential commercial partners that is, on deep architecture at a scale not explored in the research literature. \n\n Theoretical tools to measure generalization in deep neural network lag behind practical cross-validation techniques that simply apply trained models to unseen data. In many cases generalization bounds are vacuous or only apply to simple architectures that are not used in practice. Recent principled work has provided improved generalization bounds, but there remains a gap between practical evaluation and theoretic bound. A common thread that has appeared in recent work is the resurfacing of compression as inference. There is a clear opportunity to measure compression geometrically, and our proposed work is a key step in that direction. \n\n More specifically, in Phase I we propose to: 1) leverage techniques from stratified space theory, persistent homology, and tropical geometry to investigate how data shape changes as it moves through the layers of a deep network, enabling us to prove theorems constraining the function classes learnable by such networks, and thus prove generalization bounds relating to predictions that can be made; 2) study the relationship between geometric structure on training data and the structure of the neural networks best trained on that data in order to reduce the amount of time it takes to analyze a class of data; 3) determine the class of functions that are appropriate to train with a given training dataset; 4) empirically validate our theorems on modern architectures by measuring estimates of network generalization on data drawn from previously unseen distributions. We will focus on models trained on imagenet.

* Information listed above is at the time of submission. *

US Flag An Official Website of the United States Government