USA flag logo/image

An Official Website of the United States Government

Web-Based Infrastructure for Comparison and Validation of Image Computing…

Award Information

Agency:
Department of Health and Human Services
Branch:
N/A
Award ID:
Program Year/Program:
2010 / STTR
Agency Tracking Number:
R41EB011796
Solicitation Year:
2010
Solicitation Topic Code:
NIBIB
Solicitation Number:
PA09-081
Small Business Information
Kitware Inc.
28 Corporate Drive Clifton Park, NY 12065-8688
View profile »
Woman-Owned: No
Minority-Owned: No
HUBZone-Owned: No
 
Phase 1
Fiscal Year: 2010
Title: Web-Based Infrastructure for Comparison and Validation of Image Computing Methods
Agency: HHS
Contract: 1R41EB011796-01A1
Award Amount: $237,264.00
 

Abstract:

DESCRIPTION (provided by applicant): Validation of computing algorithms has been a challenging topic over the last few years. In fact, several international workshops in the medical imaging field started to involve the community through grand challenges .A grand challenge involves selecting driving biological/scientific problem and asking experts to submit their best results and methods to solve it. Grand challenges often use blind verification in order to provide an unbiased validation. Validation is critical to science because it imposes to scientists a rigorous protocol before claiming the validity of their algorithms. Validation also ensures that algorithms are clinically viable and will perform with the same robustness and accuracy in the clinic. There is a clear consensus among the scientific community that careful validation is needed. However, validation still remains a challenge and can become a laborious task for several reasons. First, the overall design of the validation experiment should followstrict rules in order to be consistent with the scientific reasoning. For instance, if a registration algorithm uses landmarks as a base for registration, these same landmarks should not be involved during the validation process. Second, the testing and training datasets should be clearly identified and separated. The testing datasets should be used only for testing purposes and not to tune the algorithm. Third and last, the metrics used to measure the error of the algorithm should be relevant to the scientific goal of the research. For instance, only measuring the mean value of the resulting error of segmentation could have critical impact in the clinic if the maximum error is a very high value. Validation remains a difficult task and several tools have emerged to help scientists with validation tasks. The open source Insight Toolkit and Visualization Toolkit provide off the shelf algorithms for medical imaging, making comparison with other methods easier. Grand challenges for segmentation and registration, like the ones hosted at the Medical Image Computing and Computer Assisted Intervention, invite researchers to test their algorithms against each other providing a level of validation. However, no complete infrastructure is currently being offer to the research validation for collection and hosting validation tools. The aim of this proposal is to develop an infrastructure to help scientists to perform validation tasks. While considered an important element towards full clinical validation, the system doesnot aim to perform a full clinical validation, but rather help research choose the best tools for their clinical application. The proposed system, named COVALIC, provides an online repository of testing and training datasets, an open source framework forvalidation metrics and an infrastructure for hosting grand challenges and publishing validation results. Through the online system, researchers can perform validation tasks from the convenience of a web browser. Furthermore, COVALIC is built upon open access and open source, thus engaging the community in the effort and encouraging researchers to share their data, algorithms, metrics and results. We propose to develop and test the system with the help of six experts in the field: clinical researchers, surgeon, computer scientist, and scientific researchers, thus creating a system designed by the end user community. PUBLIC HEALTH RELEVANCE: Validation is a critical component of the development of computing methods and often present major challenges. The main difficulty in comparing performance of algorithms is to define a common reference for the training and testing datasets as well as validation metrics. The other challenge is to access other researchers' results and algorithms. We propose to developan intuitive web-based system for collecting, distributing and processing validation algorithms. Additionally, we propose to develop an open-source framework for the validation of image processing algorithms.

Principal Investigator:

Stephen R. Aylward
518-371-3971
Stephen.Aylward@Kitware.com

Business Contact:

William J. Schroeder
518-371-3971
will.schroeder@kitware.com
Small Business Information at Submission:

KITWARE, INC.
28 Corporate Drive SUITE 204 Clifton Park, NY -

EIN/Tax ID: 114180269
DUNS: N/A
Number of Employees: N/A
Woman-Owned: No
Minority-Owned: No
HUBZone-Owned: No
Research Institution Information:
University Of Utah
75 South 2000 East
SALT LAKE CITY, UT 84112-
Contact: