You are here

Rapid Terrain/Map Generation for Robotic and Autonomous Vehicle Simulations

Description:

TECHNOLOGY AREA(S): Ground Sea

OBJECTIVE: The objective is the rapid generation of Real World Landscapes (Geo Specific) with features (e.g. ground material, bushes/trees, roads, buildings) for Unreal Engine 4 using GIS Data (Digital elevation model), LIDAR data, satellite images, Photogrammetry so that the Landscape map in Unreal Engine 4 is high fidelity enough to use for Robotic and Autonomous Vehicle Simulations

DESCRIPTION: The Ground Vehicle System Center (GVSC) performs research using interactive ground vehicle simulations akin to realistic video games. In these simulations we measure the performance of soldiers at tasks such as planning and driving. The GVSC interactive ground vehicle simulations are also used for the development of robotic and autonomous systems and the evaluation of these autonomous driving algorithms.It presently takes months to create a geospecific terrain model for use in the simulation largely due to the quantity of area that needs manual creation and touch-up. This is primarily due to our unique requirement for high visual quality (and rendering performance) but only landmark level accuracy for geotypical features. We are using Unreal Engine for rendering our camera views which put some requirements on the structure of the data that we use. The digital elevation model of the terrain surface is a rectangular grid of altitudes in UTM-style coordinate system. Accompanying the elevation data is any number of layers which we use, one each, for grass, dirt, road, etc. Each layer has a single graphical “material” which affects how it drawn and a blend weight image (of the same dimensions as the elevation data) which represents the “weight” of coverage of that material on that part of the map. On top of this terrain surface we place 3D polygon models of bushes, trees, fences, rocks, buildings, etc. Outside of particular areas of interest, we repeatedly use the same models as we only care about the location of a tree not if every branch is in the right place. Specifically we model the terrain type and elevation with a post spacing of .5m-1.0m which we cover with “typical” imagery of grass, dirt, asphalt, etc. at <1cm/pixel resolution. In addition to the re-use of 3D models, large non-navigable areas outside our areas of interest, such as forests and buildings, will modeled as solid volumes with imagery on the sides. While these models don’t necessarily visually match the real objects in the area they must be located in the exact spots as we sometimes replay position data recorded from real vehicles. Existing solutions for generating this data are either too detailed, preventing rendering at interactive speeds, too specific, or accurate only when viewed from a distance. A complicating factor is that we are viewing the scene from the ground at a height of 1m-4m which is close to perpendicular to the viewpoint of commonly available imagery. Aerial imagery looks blurry when shown from a viewpoint that close. This difference in viewing angle leads to a deficiency in 3D model generation algorithms that use aerial imagery (e.g. Google Maps 3D view). The resulting models are misshapen, misaligned, and of insufficient detail when viewed up close and from the ground but otherwise look fine when viewed from above. A secondary failing of areal imagery in our use case is that it is blind to the area under a tree. Drawing a tree canopy on the ground surface is clearly wrong when viewed from low angles. While we can never truly know what is under the tree we need to back fill those areas with plausible terrain. There is a lot of existing research into terrain classification/segmentation which we can use to generate terrain layers. However, these algorithms are unable to look under the trees where we don’t have data yet but need plausible terrain. Classifying the terrain is also important from a driving perspective. While we don’t need to model soil properties, our simplified tire models need to know, broadly, if we’re on asphalt vs. dirt vs. grass. We also don’t need to model geometric features as small as a pot hole. A possible solution to filling in the gaps is inpainting or image healing, akin to Photoshop, but such tools require manual intervention. There is no present tool to automatically identify obscured areas in the terrain and fill them in with plausible data. In addition, some these areas, such as a single tree, will need to be identified and replaced with a 3D model that is not part of the terrain. This SBIR is seeking innovative solutions to the development of realistic terrain models with dimensions up to 8km x 8km with rectangular regions of 5-10 km2 being typical. The solution should be standalone software which accepts as input elevation data and satellite imagery and then generates a visually rich terrain suitable for use in the Unreal Engine.The solution must be highly automated employing classification to generate specific terrain types to include roads, fields, forests, water features and buildings as well as detect, remove, and inpaint the terrain covered by individual trees, cars, people, etc. The solution must provide methods for human-guided interpretation and classification of areas that are ambiguous by means of selective labeling and or morphing.The tool shall allow an adept user to convert convincingly real terrain at a rate of 1 km2/hr.

PHASE I: The vendor shall deliver a prototype system demonstrating the classification/inpainting of terrain. From a 2-D elevation data (height-map) (.5m/px resolution) and aerial imagery aligned to the elevation data, classify/segment the image into terrain types; one lossless greyscale image per type (png, tiff, etc). The input imagery may be at a different resolution than the elevation data with the edges of the data aligned. The output images must be pixel aligned with the elevation data, where white represents 100% coverage or confidence. Considering the low resolution, multiple terrain types for a given spot are possible, for example a patch of thin grass with dirt showing through could be marked as 50% each on their respective layers. Adjusting the weights for improved appearance encouraged. All-or-none rounding to create hard edges is discouraged. (We acknowledge that this is sub-optimal for paved roads.) For certain types representing covered areas, the unknown area underneath will be inpainted with a different terrain type surmised from the surrounding area and similar features elsewhere. For example, the area under a tree will be filled in with grass, dirt, etc. or a combination. The algorithm should also indicate areas that may need to be represented with a new terrain type or otherwise require manual intervention. A user shall have access to intermediate products which can then be adjusted and the following stages of the process re-ran. We expect that solutions will combine and modify existing image processing and classification tools to accomplish the goals. Developing a custom neural net is outside of the scope for Phase I. The essence of Phase I is the interaction between the classifier, the inpainter, and the user. Although we discuss a segmentation/inpainting process, other classifying/fill-in-the-gaps algorithms along with other pre/post processing steps is permitted.

PHASE II: The vendor shall deliver a plugin for Unreal Editor which will import the elevation data and aerial imagery, classify and heal it, and generate terrain elevation model, terrain type layers, unreal rendering blueprints, etc. as necessary to represent the terrain. As part of this process, a user may manually override classification in certain areas and the software must rework the final output based on the changes. For certain movable objects such as people, vehicles will be erased. For areas that represent large objects (trees, rocks, road signs), it will place a representative model in the scene. These 3D models will be user provided but the software must automate the placement.For areas that represent impassible terrain (forests, buildings), a 3D model representing the footprint of the area extruded to a best-guess height will be generated. (This model will be used by an artist as a basis for detailed model which is outside the scope of this project.). Hard edged roads shall be detected, erased, and replaced with a polygonal model. Physics material properties may be provided by the user and if so, the software will attach them to the corresponding terrain. (Software does not need to understand what these represent.)

PHASE III: Military application is for the development and testing of automated vehicles or vehicle functions which are highly dependent on a modeling and simulation environments with geospecific high fidelity terrain/maps that can be generated rapidly.It is for the development and testing of such systems.The U.S. Army GVSC has a critical need for geospecific high fidelity terrain/maps.Commercial application is in the automated driving technology testing and assessment.This applies to both the automotive industry as well as military OEMs and suppliers.

KEYWORDS: Modeling & Simulation, Autonomous Systems, 3D Terrain, three-dimensional models. geospatial intelligence, Digital Terrain Models, Digital Surface Model

References:

[1] Dmitry Ulyanov, Andrea Vedaldi, Victor Lempitsky, “Deep Image Prior”, https://dmitryulyanov.github.io/deep_image_prior; [2] Guilin Liu, Fitsum A. Reda, Kevin J. Shih, Ting-Chun Wang, Andrew Tao, Bryan Catanzaro, “Image Inpainting for Irregular Holes Using Partial Convolutions”, https://arxiv.org/abs/1804.07723 https://news.developer.nvidia.com/new-ai-imaging-technique-reconstructs-photos-with-realistic-results/, ECCV 2018; [3] Steven D. Fleming, Ryan McAlinden, Matt S. O’Banion, Christopher Oxendine, William Wright, Ian Irmischer,“Rapid Terrain Generation, Geospecific 3D terrain representation is revolutionizing geovisualization, simulation, and engineering practices around the world”,https://trajectorymagazine.com/rapid-terrain-generation/ Trajectory Magazine the Official Magazine of USGIF, 25 January 2019.; [4] TensorFlow Core, Image Segmentation, https://www.tensorflow.org/tutorials/images/segmentation; [5] Wolfram Language & System Documentation Center, Inpainting, https://reference.wolfram.com/language/ref/Inpaint.html; [6] MATLAB Computer Vision Toolbox, Semantic Segmentation of Multispectral Images Using Deep Learning, https://www.mathworks.com/help/images/multispectral-semantic-segmentation-using-deep-learning.html

US Flag An Official Website of the United States Government