Ease of use and deployment for a fast, scalable data movement infrastructure

In recent decades, High Performance Computing (HPC) has risen as one of the key enablers of innovation in science and engineering. However, industrial adoption has been impeded by the complexity and cost of deploying applications. Problems include the quality, speed, and flexibility of software tools and the specialized expertise required to adapt solutions given the profusion of hardware configurations. Efficient data reading and writing remains a key bottleneck in standard approaches, with poor I/O often limiting the scalability of application codes, reducing the scale and scope of problems that can be addressed with simulation. This SBIR Phase I will support the deployment of PIDX, a high- performance parallel I/O library that has been shown to scale efficiently to a million cores on DoE flagship HPC systems. The key innovation enabling high-performance I/O is a proprietary hierarchical space-filling data layout that delivers optimal performance for memory hierarchies. Furthermore, the ViSUS framework enables fast visualization and analysis, even for petabytes of data stored on remote systems. However, achieving this performance, until now, has been a complex process requiring deep integration of the PIDX library, custom tuning of parameters for HPC systems, and designed for the open data typical of scientific environments. This SBIR will make this technology accessible for commercial use by reducing the time and complexity of installing and using the PIDX library and ViSUS framework. The first component of the work will be to harden the existing PIDX library and develop a simple API that is easy to use and integrate in simulation software. Next, software tools will be developed to simplify the installation and tuning of the library to achieve high performance on commercial HPC systems. Furthermore, the library will be integrated with mainstream data analysis and visualization clients in the form of plugins to enable fast remote visualization and simulation monitoring without disrupting existing workflows. Finally, The ViSUS data server will be extended to account for restrictions and permissions required for private use of data. Make the data generated immediately available for remote streaming. Commercial impact and societal benefit: A simple and efficient I/O ecosystem is a prerequisite component of engineering and industrial use of HPC systems. Removing the I/O bottleneck will enable more efficient use of expensive HPC resources, allow larger and higher-fidelity simulations for more accurate modeling, and reduce the cost and complexity of writing scaling simulation codes. The kind of data addressed by this project is already being produced and utilized commercially, such as reservoir simulation in oil and gas exploration, fluid dynamics studies in aerospace industries, and computational electromagnetics in manufacturing processes. The industrial HPC market saw a 10% growth in the first quarter of 2015, which is further expected to expand as additional applications are adopted. The I/O library supported by this SBIR will furthermore ease the burden of scaling, as today’s scientific research capacity runs become tomorrow’s commercial capability processing.