Case Study

National Institutes of Health, NIH Blueprint for Neuroscience: Neuroimaging Informatics Tools & Resources Clearinghouse (NITRC) – All Together Now

Key Findings

  • A successful collaboration environment should blend together subject matter expertise in the science domain and strong technical and management capabilities such as the Capability Maturity Model Integration (CMMI) and Project Management Body of Knowledge (PMBoK).
  • Architecting and building a system will not automatically draw users. Concerted, organized outreach drives user adoption and is essential to the viability and success of an initiative.
  • Strong program governance yields cost savings and accelerates success.
  • Use of open source technologies for application and web development can, where appropriate, yield significant cost and schedule savings over the long term.
  • O&M must continue to evolve by listening to the customer base and broaden services to meet the ever changing challenges facing the user base.

Challenge

Neuroimaging tools and resources developed under NIH grants were often abandoned when the grant ended, the responsible graduate students graduated, and the postdocs moved onto other projects. Consequently, important resources were frequently forgotten or left to go stale, or the broader research community was unaware of their existence. NIH wanted to:

  • Capture the work and investments already made through grant funding
  • Allow other researchers to take advantage of the tools developed under grants
  • Support open distribution of the resources to the community as a whole
  • Fund teams to “harden” the tools for use by scientists outside of the originating laboratory, and refresh stale tools that still have scientific merit
  • Level the research playing field by publicly sharing previously funded data sets
  • Enable reproducible research through the sharing of research pipelines, software tools, and data
  • Create an open collaboration environment in which all of this activity could occur

From these desires came the idea for the Neuroimaging Informatics Tools & Resources Clearinghouse (NITRC). Dr. Elias Zerhouni, former director of the NIH, described NITRC’s mission: “By building upon significant investments already made in the informatics area, NITRC will provide a coordinated, coherent resource for the neuroimaging resource community.” NITRC is a web site, big data repository, and cloud services collaboratory where researchers can see what tools and resources are available, obtain use of them, get involved in improving them for their own research, share them back with the community as a whole, and create reproducible research.

Solution

Our tailored solution for NITRC included several key aspects:

  • Create a team that included key figures in the scientific community: Experts who had the academic reputation necessary to secure support from the broader neuroimaging community were an intrinsic part of our NITRC project team.
  • Utilize project management and systems development best practices (PMBoK and CMMI): NITRC was released early and well within the budget, based on our management approach and a careful adherence to planning, validating, and then acting. The use of the Capability Maturity Model Integration (CMMI) and Project Management Body of Knowledge (PMBoK) were central to our management approach.
  • Focus on customer service and outreach: We provided professional marketing expertise to help establish the NITRC brand in neuroimaging. This attention to the importance of reaching and engaging our audience helped to establish credibility in the community. We developed and implemented a comprehensive marketing and outreach plan that identified the universities and laboratories developing relevant tools and resources, and laid out the tactics necessary to encourage them to share their work with “competing” research institutions around the world. These tactics included lecturing and exhibiting at neuroscience conferences, traveling to universities’ neuroscience laboratories and meeting with graduate students, creating symposiums and panel discussions at conferences, and publishing articles in scholarly journals.
  • Leverage open source: NITRC is powered by a customized suite of open-source applications.  The open-source solution met 80% of the contract requirements out of the box; TCG built the remaining 20% mostly by reconfiguring open-source packages. The NIH saved approximately $5 million over the cost of a comparable custom health-information system. All NITRC functions, except its menu system, are driven by these open-source applications (99.97% of NITRC’s lines of code are open source). This platform can now be utilized on other government projects at no additional cost to the government.

TCG’s commitment to documented processes, and its quality-control procedures, ensured that NITRC met all requirements the first time. Such preparation work saves time and money.  In terms of return on investment (ROI) and saving taxpayer dollars, NITRC has delivered the goods.  For example:

  • Tools and resources developed for specific projects may now be shared by many projects, tools and resources developed for sharing may be shared more effectively, designers and creators have a storefront and a warehouse, and neuroimaging researchers have a mega-mall of options at one site. If only 5% of the tools on NITRC today  are utilized by other laboratories rather than have those laboratories request new government funding to build their own tools, this project will have more than paid for itself.
  • In 2006, NIH spent approximately $4.2 billion taxpayer dollars on neuroscience research. NIH has funded neuroscience research for decades; other agencies, such as the National Science Foundation, also fund neuroscience initiatives. Therefore the total amount of government funding for neuroscience is far larger than $4.2 billion per year.  The NITRC project cost roughly 0.1% of NIH’s 2006 neuroscience budget.
  • By the end of 2009 120,800 files—tools and data—had been downloaded from NITRC by researchers. The site had a monthly average of 785,000 hits. NITRC has allowed the government to transparently reach the target community.
  • 253  software tools and resources, a portion or all of whose development was underwritten by the NIH, are now available for use by the broader scientific community. Another 43 software tools have restricted access.
  • NITRC helps the government avoid waste by encouraging research laboratories to share their NIH-funded neuroimaging tools and resources, which means more money for basic research with the same tax dollar. The valuable time and money saved can be redirected towards basic scientific research to understand key health concerns, giving rise to scientific discoveries and tools that otherwise would not have come about.

Results

Our tailored solution for NITRC included several key aspects:

  • Create a team that included key figures in the scientific community: Experts who had the academic reputation necessary to secure support from the broader neuroimaging community were an intrinsic part of our NITRC project team.
  • Utilize project management and systems development best practices (PMBoK and CMMI): NITRC was released early and well within the budget due to TCG’s management approach and a careful adherence to planning, validating, and then acting. The use of CMMI and PMBoK were central to our management approach.
  • Focus on customer service and outreach: We provided professional marketing expertise to establish the NITRC brand in neuroimaging. This attention to the importance of reaching and engaging our audience helped to establish credibility in the community. We developed and implemented a comprehensive marketing and outreach plan that identified the universities and laboratories developing relevant tools and resources, and laid out the tactics necessary to encourage them to share their work with “competing” research institutions around the world. These tactics included lecturing and exhibiting at neuroscience conferences, traveling to universities’ neuroscience laboratories and meeting with graduate students, creating symposiums and panel discussions at conferences, and publishing articles in scholarly journals.
  • Leverage open source: NITRC is powered by a customized suite of open-source applications. The open-source solution met 80% of the contract requirements out of the box; TCG built the remaining 20% mostly by reconfiguring open-source packages. The NIH saved approximately $5 million over the cost of a comparable custom health-information system. All NITRC functions, except its menu system, are driven by these open-source applications (99.97% of NITRC’s lines of code are open source). This platform can now be utilized on other government projects at no additional cost to the government.
  • Stay ahead of the community’s needs: NITRC provided a software repository a decade before Code.gov, openly distributed brain imaging data four years before the White House Open Data Policy, and fulfilled OMB’s Cloud First Policy by distributing the first science cloud compute service on AWS

TCG’s commitment to documented processes and their standardized quality-control procedures ensured that NITRC met all requirements. Such preparation work saves time and money. In terms of return on investment (ROI) and saving taxpayer dollars, NITRC has delivered the goods. For example:

  • Tools and resources developed for specific projects may now be shared by many projects, tools and resources developed for sharing may be shared more effectively, designers and creators have a storefront and a warehouse, and neuroimaging researchers have a mega-mall of options at one site. If only 2% of the tools on NITRC today are utilized by other laboratories, rather than have those laboratories request new government funding to build their own tools, at an average cost of $52,000 per grant, this project has paid for itself many times over.
  • In 2015, NIH spent $5.7 billion taxpayer dollars on neuroscience research. NIH has funded neuroscience research for decades; other agencies, such as the National Science Foundation, also fund neuroscience initiatives. The total amount of government funding for neuroscience is far larger than $5.7 billion per year. Since 2006, NITRC and its related grants cost approximately 0.1% of NIH’s annual neuroscience budget.
  • By the end of 2016, 9 million files—tools and data—had been downloaded from NITRC by researchers. The site had 17,860 registered users. NITRC has allowed the government to transparently reach the target community.
  • 870 software tools and resources, a portion or all of whose development was underwritten by the NIH, are now available for use by the broader scientific community. Another 85 software tools have restricted access.
  • NITRC helps the government avoid waste by encouraging research laboratories to share their NIH-funded neuroimaging tools and resources, which means more money for basic research with the same tax dollar. The valuable time and money saved can be redirected towards basic scientific research to understand key health concerns, giving rise to scientific discoveries and tools that otherwise would not have come about.

Accolades

NITRC won the coveted “Overall Best” 2009 Excellence.Gov Award, which aims to identify the very best IT projects across the entire government. NITRC was lauded as the best example of how to use IT to support government collaboration.

In 2015, NITRC was the recipient of the HHS Innovates Award. The program was recognized for delivering the “Biggest Bang for the Buck” by providing “significant savings or return-on-investment for HHS resources expended.”

NITRC testimonials include: “It is very clear that NITRC...is standing at the bleeding edge on seeing how science is being done. You’ve done a wonderful job.” NIH NIDA Program Officer. “I’ve quadrupled the number of of peer reviewed publications since NITRC came into my life.” MIT Professor. “(I) needed a solution that would quickly and reliably conduct image processing for Parkinson’s research. I found NITRC-CE on AWS Marketplace, reducing time required to process neuroimaging data by 85%...allowing me to complete a critical stage of my research in 2 days, instead of 2 weeks.” University of Leige, Researcher.

Taxpayer Savings: $186.87 million

For every project, TCG tracks the savings generated by innovation in methods and technologies. Just by publicly sharing neuroscience researchers’ neuroimaging datasets, NIH documented that NITRC Image Repository alone represents a cost savings of over $36.55 million. Here are some of the additional ways that NIH demonstrates cost savings from our work:

  • Sharing software and data online reduces or eliminates duplicative software development, new studies to gather new brain images, and other effort. Grantees no longer need to redevelop software tools stored on NITRC.
  • Providing virtual machines with pipelines of software applications allow researchers to save system administrative time and leverage the massive compute power of cloud services such as Amazon Web Services (AWS).
  • Matrix managing a team of 100% off-site staff saves government resources by eliminating unnecessary travel and maximizing personnel productivity across multiple projects.
  • Use of Concurrent Versions System (CVS) branches allows us to simplify the development cycle.
  • Meeting online, using web-based tools, is faster and more efficient for many kinds of topics.
  • By modifying open source software, we save development time compared to custom development for equivalent functionality.