One of the continuing challenges faced by science funders relates to designing and to implementing effective approaches to measuring grantmaking progress.  This issue is especially acute for discovery science funders since the value of such work may not be apparent until years or even decades after its completion.  Despite these difficulties, funders in this space have developed a variety of creative approaches to determine the impact of their work.

In 2021, the Science Philanthropy Alliance (Alliance) created a Shared Interest Group (SIG) on the topic of Measurement, Evaluation and Learning (MEL) enabling members to engage with each other by sharing questions, expertise and approaches.  These regular gatherings, both in-person and virtual, featured expert presentations, case studies and lively discussion.  Since the MEL SIG’s formation, the program has engaged 67 individuals from 26 institutions. The lessons learned from these engagements were captured in a cross-cutting summary, Measurement, Evaluation, and Learning: Practices, Perspectives, and Pathways Forward from the Science Philanthropy Alliance, authored by the SIG’s co-chairs, Julia Klebanov of Chan Zuckerberg Initiative (CZI) and Evan Michelson, Alfred P. Sloan Foundation.

To share this publication and to exchange knowledge with a robust community of funders with a MEL interest in the UK and Europe, the Alliance co-hosted an event with its member Wellcome at their headquarters in London.  A private tour of the Francis Crick Institute, a discussion between David Harding of the David and Claudia Harding Foundation, Andy Parker from Cambridge University and Sue Merrilees of the Alliance was followed by a reception and opening dinner before the day-long workshop on September 8.  The workshop attracted 54 program and evaluation staff from Alliance member organizations, evaluation professionals, science policy scholars, and representatives from 37 philanthropic organizations based in the United States, Canada, the United Kingdom, and Europe.

“We need to shift focus in evaluation from what we have done, to what does it matter?” Fiona Gatty, Templeton World Charity Foundation

The workshop kicked off with opening remarks from Lesley Alborough from Wellcome and Sue Merrilees from the Alliance. These introductory remarks explained the context of the workshop and the hoped-for outcomes.  They were followed by a detailed overview of Wellcome’s own evaluation processes led by Peter O’Donovan.

“It’s a complex process so there’s no simple answer.  Wellcome is moving toward a portfolio assessment framework that encourages context-specific data gathering, analysis, and decision making.” Peter O’Donovan, Wellcome

Next up was a panel addressing the special challenges of conducting MEL in the international grantmaking space. In a discussion moderated by Elizabeth Weiss from the Alliance, panelists included Hannah Cameron from the Bill & Melinda Gates Foundation, Marc Malandro from CZI and Matthew Walhout, John Templeton Foundation.  All agreed on the importance of taking multiple perspectives into account.

“Having grantees with lived, first-hand knowledge of global health concerns and local context is very important if you want to deliver meaningful impact.” Hannah Cameron, Bill & Melinda Gates Foundation

“We’re hoping to integrate a wide range of perspectives from across different domains—including cultural wisdom, indigenous traditions, religions, and philosophies—so that science funders can more fully grasp how their grantmaking builds on cultural understanding, as well as scientific knowledge of the natural world.”
Matthew Walhout, John Templeton Foundation

The workshop’s morning session concluded with a joint presentation by Evan Michelson and Julia Klebanov, who focused on illuminating selected insights from the compendium. Key points underscored   included the importance of adopting mixed methods approaches that leverage quantitative and qualitative data by acknowledging how both the purpose and the approach to MEL are evolving in the broader field of evaluation in philanthropy.

The panel discussion following the report’s presentation included representatives from three science philanthropies based in the United States, all of whom had participated in the MEL SIG and have been working to advance MEL practices within their organizations. Stephanie Albin from the Kavli Foundation, Richard Wiener from the Research Corporation for Science Advancement (RCSA) and Ted Hodapp from the Gordon and Betty Moore Foundation addressed experiments around measuring collaboration, randomization grant awards and embedding MEL practices early in the process. All reflected on the emerging MEL challenges they foresaw and what they learned from participating in the MEL SIG discussions.

“Our project was designed to better understand how novel scientific collaborations are catalyzed at conferences, and also to understand the pros and cons of virtual vs in-person modalities for scientific convenings.” Richard Wiener, RCSA

After a busy networking lunch, Wellcome introduced four case studies that demonstrated how different funders, researchers, and practitioners from across the United Kingdom and Europe are working to strengthen MEL across philanthropy and government.

“Funders need to shift from understanding impact based on a linear model of research production and dissemination to one that moves toward a more relational understanding of the linkage between research and practice.” Annette Boaz, Co-Lead for the Transforming Evidence Network

James Wilsdon, from the Research on Research Institute, outlined the growing metascience movement to better understand the culture of science and technology.

“We use a variety of measurement tools, including qualitative interviews and innovative data science techniques, such as graphing academic connections and textual analysis of publications.” Tim Killick, Wellcome

Justine Karpusheff and Amanda Watt at the Health Foundation laid out their organization’s three-year MEL practice, noting the use of both in-house staff and external consultants, and the necessity of regular access to evaluation expertise.

“If we are never failing, are we ever really learning?”  Rafael Lang, Klaus Tschira Stiftung

At the conclusion of the workshop, participants were invited to reflect on lessons learned through small breakout discussions. Many groups commented on the consensus that had emerged throughout the day, both around challenges associated with conducting MEL in science philanthropy, as well as the commonalities of effective practices and approaches that different funders have adopted.  All valued discovering a community of peers dealing with the same challenges and reassurance as to the magnitude of those challenges.

“I was missing a group of experts and peers, then I found it here.” Samuel Kembou, Jacobs Foundation