The Government Analytics Breakfast (GAB) Forum was pleased to welcome Dr. Jason Matheny, Director of the Intelligence Advanced Research Projects Activity (IARPA), and learn about the methods used to evaluate IARPA's research investments.
IARPA is a U.S. Government organization that invests in high-risk, high-payoff research in support of national intelligence. Approximately one quarter of IARPA's budget is devoted to measuring and evaluating the quality of the research it funds, a very difficult task. For example, Dr. Matheny noted that scientists often learn a lot from what doesn't work. IARPA devotes time and resources to recording failures so future scholars can learn from them (and thereby avoid the "file drawer problem").
Two metrics commonly used to measure research success are (1) the number or impact of publications and (2) the transfer of technologies that result from the research undertaken. Dr. Matheny, explained, however, that these are certainly not the only measures of success available, but it's not clear which measures are "best", particularly when determinig future funding.
Increasingly IARPA is using prize money (instead of traditional contracts) to entice a wider breadth of researchers to investigate solutions that can be applied in the intelligence field. Under this compensation structure, teams of researchers need not submit proposals to receive funding -- they simply submit their solutions and IARPA then judges which solution(s) is best.
Dr. Matheny concluded by noting the importance of scientists learning how to communicate complex findings to wide audiences. He suggested the researchers leverage tools like videos and podcasts as means to convey the importance of their work to nontechnical audiences, including high-level policymakers.
You can view the entire talk here.