The TPA strategy evaluation and landscape scan: narrowing our scope for data collection

OTT Consulting would like to thank everyone who commented on the guiding questions for the Hewlett Foundation transparency, accountability and participation (TPA) evaluation and the landscape scan. It was really encouraging to receive such thoughtful responses.

Balancing the ambition to explore and learn as much as we can, with what is feasible within a reasonable time frame and budget is always the biggest challenge for a project of this nature. And prioritising questions and focus areas inevitably involves difficult decisions and saying no to relevant and interesting lines of enquiry. But it’s an important and necessary part of the process.

Over the past couple of months, we’ve been exploring with the foundation where to narrow down our focus. In this post, we wanted to share with you the outcomes of those discussions.

The evaluation

Prioritising questions
If you read our post inviting comments on the questions, you’ll know that there were a lot of them. Indeed, some of you pointed this out in your feedback. And so, we have organised questions into two groups: eight priority evaluation areas for interview data collection and eight ‘nice to have’ areas that will be covered in a lighter-touch way, primarily through the document review.

We arrived at these questions through a series of one-to-one and group meetings with the foundation’s TPA team, to get a better understanding of where there were gaps in knowledge, and where there was particular interest to learn more or deepen understanding.

We also invited comments from you, to find out what you’re interested in, whether we were duplicating knowledge that exists elsewhere, or whether we were missing something significant. Your comments have helped us to refine, nuance and deepen some of the questions and sub-questions.

You can see the full final list of guiding questions here.

Selecting focus countries and themes for interviews
A huge amount of work took place over the recent five-year TPA strategic period; the foundation made over 370 grants to 107 grantees, at the international, regional and national level, in 10 countries and 14 thematic areas.

As much as we’d love to speak to all 107 grantees, it’s sadly not feasible and so we had to narrow our scope. To do this, we divided grants into three groups: international, regional and national and identified a number of focus countries and thematic areas.

At the national level, we will be reaching out to grantees in three countries: Kenya, Senegal and Uganda. Why these three? Firstly, these are three countries in which the foundation has disbursed a significant number of grants, or has been engaged for a long time, and so there is sufficient data for the evaluation to draw upon. Secondly, none of these countries have been the subject of a stand-alone evaluation during the strategic period, and so we will not be duplicating efforts but rather generating new knowledge to contribute to our understanding. Thirdly, we tried to pay attention to context, choosing countries that represent a cross-section of governance and political contexts, as well as both francophone and anglophone African countries.

For grantees working at the international and regional levels, we will be reaching out to those working in eight focus areas (with a further emphasis on those who support country programmes or local partners within the three focus countries): budget and expenditure transparency; field learning; gender equality and social inclusion (GESI); legal empowerment; multi-theme (grants working across more than one thematic area); natural resource governance, procurement reform; and tax. Thematic areas were selected in consultation with the foundation as they represent areas where the TPA team wants to better understand the contribution its investment is making.

The landscape scan

Prioritising questions
For the landscape scan, we now have four priority question areas and three ‘nice to have’ areas. Much like the evaluation questions, we arrived at this final list through a series of one-to-one and group meetings with the foundation, as well as inviting comments from you. You can see a full list here.

Selecting focus themes and countries
The landscape scan will look at broad global trends and the status of current knowledge, focusing on seven themes: budget and expenditure transparency; GESI; natural resource governance; procurement reform; responsible investing and corporate behaviour; social accountability; and tax. Again, these areas were selected in consultation with the foundation as areas where they would like to deepen their understanding of the state of the field.

As part of the scan, we will be e commissioning a special study on GESI. Over the past five years, the foundation has sought to better understand how gender and social inclusion dynamics manifest in the TPA field, and then to identify grantmaking opportunities to address them. But the foundation’s TPA team feels that it has a lot more to learn in this area. GESI is also a cross cutting focus area for the evaluation, through which we aim to capture and learn from the innovative work that some of the foundation’s grantees are already doing in this area.

The landscape scan will deep dive into five countries: Burkina Faso, Ghana, Kenya, Senegal and Tanzania. The aim here is not to provide detailed country analysis. Rather, we aim to capture salient political economy factors, both barriers and enablers, that are representative of the range of contexts in which the foundation is working. This includes francophone and anglophone Africa, competitive and authoritarian political contexts, and those that are performing well, and not so well, on formal governance measures.

We will also identify and assess the influence of key Africa-based regional and international forums and institutions that influence grantees and other stakeholders in our countries and themes of interest.

Final thoughts

For the foundation it’s important to emphasise that the strategy refresh evaluation is an assessment of the foundation’s TPA strategy and what the foundation can learn from its efforts to implement it; it’s not an evaluation of individual organisations that the foundation funds.

Equally important to note is that if a country or thematic area isn’t included in the focus list, it doesn’t necessarily mean that it won’t be included in the new strategy. The goal here is to generate new knowledge to complement what is already known, to learn more about where and how the foundation can add value and contribute in the most meaningful way looking ahead.

Data collection takes place this month. We’ll be reaching out to some of you for interviews. We’re also planning to create online spaces for all grantees and external stakeholder to participate in, should you wish to. We’ll keep you posted on our progress and opportunities to participate here, on the TPA Medium blog.

We look forward to going on this journey with you!