Research and evaluation programme grant management

REC Management Note No. 2

Large funders often contract out grant management to another organization, which then makes subgrants according to the criteria for that grant. 3ie was primarily a grant making organisation, and I have also managed grant programmes at the Campbell Collaboration and CEDIL. This note reflects on those experiences, as well as my experience applying for grants (see REC Management Note 3 Getting Funded), with lessons for grant makers.

Open or closed calls? 

Should you request proposals from anyone at all – of course with some eligibility criteria – or should it be restricted to those you invite, or who have been pre-qualified? The advantage of the latter is that there are fewer applications to manage, and they are likely all relevant. If you decide to simply invite one organization to apply you can save a lot of time. The disadvantage of closed calls is of course that you may miss some excellent proposals, and run the risk of being accused of only commissioning ‘the usual suspects’.  So, if you are looking for innovation, new ideas, and time is not a pressing concern, then open calls are best. If you have a very specific need with limited time,  then a closed call, including possibly a single supplier,  can be the way to go. Your organization should have a sole source policy in place to make single supplier possible, and document that decision for transparency and audit purposes.

Disseminate, disseminate, disseminate  

You want the people you want to apply to know about the opportunity. And there may be people you don’t know who will submit excellent applications. In the first round of grants from 3ie, several peer reviewers commented that they would have applied had they known about it. That first grant round received 79 applications. Our second received 255 applications and the third 366.  So getting the information out there will give you a much larger pool to choose from. 

There is an adage in communications that you have to tell someone something seven times in seven different ways to get the message across. My response to that is ‘only seven?’. Social media helps of course. But spend some time in targeted dissemination to the people you want to reply. And any dissemination should include the line ‘please share this opportunity with colleagues who may be interested’ or simply ‘Please repost’.

Be clear what on what you want, from whom and by when 

As an applicant to an RFP, the first questions I have are ‘am I eligible to apply’ and ‘have I missed the deadline already’?. So the request for proposal (RFP, other acronyms are also used), should have clearly headed sections on eligibility and timeline. Indeed, put these at the top in a box, as they are the first thing people should know. I have been asked to advised on draft proposals only to have to tell the team they don’t appear to be eligible. 

And be clear on what you want, and in what timeframe. I read many RFPs where it simply isn’t clear what they want. Avoid unduly long background sections which simply serve as a barrier to getting to the point.  Before any background simply state ‘We are seeking applications for  XXX’.

So a good RFP could look the template given in the annex at the end of this note.

Indicate the budget 

Some commissioners believe they can increase value for money by not giving a grant amount, as researchers will always ask for the maximum amount available. I find this approach very frustrating as the applicant needs an idea of the scale of the work which is intended.  I suggest giving an actual or indicative maximum grant amount in the RFP but also include value for money in the assessment criteria. I do not suggest simply accepting the cheapest proposal meeting some threshold on technical quality, as some organizations do, as this will likely mean that you don’t fund excellence.

Provide an application form 

Provide an application form with sections for all required information. But don’t collect information you don’t need!.

Align the application form with the assessment criteria 

Amongst the things to be clear about are the assessment criteria for proposals, including the weights given to each criterion. It is extremely useful to align the application form with these assessment criteria. At worst you may have an assessment criterion, such as the role of local researchers or addressing intersectionality, for which you have not explicitly requested information in the application.  

But, more generally, for peers reviewers to review the proposal it is very helpful if the headings in the application correspond to the assessment criteria. For example, you may include a question such as ‘How will the proposed research add to the existing body of evidence about X?’, ‘Explain what is innovative about the proposed research’, or ‘How will equity considerations be addressed?’

Explain the assessment criteria

An explanation of the assessment criteria can help both applicants and peer reviewers. But don’t get over-detailed which will take away room for judgement by peer reviewers. 

To EOI or not to EOI?  

Using a first round of an expression of interest (EOI) means that you do not need to spend a lot of time going through a large number of full proposals. Reviewing EOIs can likely be done in-house so reducing the burden on external reviewers. At 3ie we used EOI in the third round of our Open Window. I calculated the time needed for reviewing, assuming that if we had gone straight to full proposal we would have received 2/3 as many applications. From our point of view, the time saving from an EOI stage was not that great. The real time saving is to applicants, and that is a benefit which should not be undervalued.  If you expect a substantial number of applications then I would suggest using an EOI round.

Use opportunities to explain the RFP to applicants

It has become common to have one or two online sessions early in the RFP process to explain about the call and for Q&A.  There is also usually a timeline by which questions may be submitted in writing. It is good practice to have an FAQ section on the RFP page to post all received questions and the answers. You should not reply to questions received online only to the organization submitting the question as that may create an unfair advantage.  Many potential applicants will only learn about the RFP only after the online sessions and deadline for questions has passed, making it important to get the FAQ up quickly and to make sure it is easily discoverable.

Allow sufficient time

My rule of thumb is that it takes one year between launching a call and the commissioned research starting. Experienced grant makers I have worked with agree with this figure of one year. 

But most funders look askance at that timeline.  It can be shortened but there is a cost to doing so.  If you want to reach all those who should know about it allow 3-4 months for that. And they likely need 1-2 months to prepare a proposal, more for complex proposals (e.g. multi-country). I have been involved in applications recently where the application was meant to have a letter of agreement from government. Such letters can take months to get, but the RFP was allowing less than three months from launch to submission.

Once submitted there should be an internal screening for completeness, and possibly for quality so only the top, say, 50% go for external peer review. Peer reviewers are typically given one month, and then they need meet to agree. Then it may go for Board approval.

And then there is contracting…. You’d think organizations would be happy to be given money, but contracting can take up to six months (it can also take just a week). The usual sticking points are intellectual property (IP) and the country for arbitration of disputes. On a visit to Gates in Seattle I met a lawyer who had done his PhD on IP in research contracts! So expect some discussion on that. You may be or may not be liable for VAT depending on the wording of the contract, and the country in which services will be provided.

Having a realistic timeline will allow applicants to arrange their time accordingly.

Don’t extend the deadline

Around 90% of grant applications come in on the last day, mostly in the last hour. Do not worry that there are few applications a few days before and extend the deadline.

What deadline?

Don’t make the deadline a Friday. You won’t read it over the weekend anyway. Sunday or Monday are good days for a deadline.  

Give a time and a time zone.  We generally used 11.59 PM EST.  We used 11.59 as it really is not commonly agreed if midnight is 12 AM or 12 PM.

How to submit

If you do not expect many proposals then email submissions, and an Excel sheet to manage the process, are fine.

But once you are above 50 submissions then you are likely better off using a grant management platform.  These are available off the peg or you can have one built specifically for you. 3ie went the latter route as grant making was our prime function. Like most IT projects it was over budget and late. But it was ours.  It is quite likely that off the peg platforms are now more customizable than they were 15 years ago, so think very carefully before commissioning your own (i.e. probably don’t).

But have an unannounced 24 hour extension (a soft deadline)

Some people will just miss the deadline for reasons beyond their control. They will write you pleading emails.

If you have an online submission platform it may well crash with the weight of the last hour submissions – it happened to 3ie. So allow submissions for up to 24 hours after the deadline. But don’t announce this otherwise you will have the same problem again 24 hours later.

The peer review process 

The peer review process will normally include: (i) an initial internal review which rejects incomplete proposals and possibly those of insufficient quality, (ii) peer review comments by a selected number of peer reviewers, (iii) a peer review panel, and (iv) approval of the proposed set of grants by the Board.  The selection of peer reviewers should reflect the nature of the type of research you want to commission. Inhouse expertise is needed to manage this process.

How may peer reviewers? 

There is an academic literature which points to the need for a large number of peer reviewers. In natural and medical sciences it is not uncommon to have 10-12 reviewers. In social sciences it remains common to have just two. Two is definitely too few, but 10 or more would be very difficult to obtain and manage.  We used 4 or 5 peer reviewers, taking the average after dropping the lowest score to rank proposals. We dropped the lowest score as some peer reviewers will just not like a proposal for no apparent reason, or just be a ‘low marker’ (a standardization process could also be used to account for peer reviewers using different scales).

Who should be peer reviewers? 

It is most common to use other researchers as peer reviewers. Many researchers rate geekiness over relevance. Policy-makers can be brought into the peer review process to improve relevance.  At 3ie we tried mixed panels, which didn’t work well as there was no ground for a common conversation. We also tried first a technical review, selecting a set of studies for review by policy makers on relevance. It provided hard to find policy makers who could see beyond their sector or geography – so the approach can work for sector and geographic-focused grants but not really otherwise. The funder will have their own view on relevance and that should be built into the RFP.

I make a distinction between BYTs (bright young things) and WOSs (wise old sages). You want a combination of the two. BYTs will get into the weeds and give you detailed comments on design. WOSs won’t spend time doing that but are more likely to raise big picture issues.

Should you pay reviewers? 

On balance I think no. It is an additional administrative burden (for the reviewer as well) and the amount you will pay will be insufficient to affect the decision to review or its timing.  At the Campbell Collaboration I abolished payment to reviewers  – it didn’t affect reviewer availability. You should publicly acknowledge their contribution by naming all peer reviewers on the website, say each year.

Do be conscious of the academic calendar in giving work to peer reviewers i.e. don’t give peer review assignments at the start of the academic year.

Create a roster of reviewers in advance

Do not wait until you need peer reviewers to start contacting peer reviewers. You will lose 1-2 months by doing so. Create a roster of reviewers in advance, inviting potential reviewers and telling them clearly what is expected of them and when.  Keep notes of who is a reliable reviewer and who is not.  When I was Managing Editor of the Journal of Development Studies our Administrative Editor had a black list of reviewers not to use.

What acceptance rate? 

There is a sort of consensus that 10% acceptance rate is a sign of a healthy grant programme, though it likely means rejecting some perfectly good proposals. A lot higher than that means you are not getting sufficient quality proposals. A lot lower means that your RFP is properly insufficient focused, or clearly explained.

Assessing value for money 

At 3ie we developed two approaches to assessing value for money. The first was ‘are the unit costs reasonable’? The second was ‘it is worth paying this much to know the answer to the question to be addressed by this proposal?’  Both of these criteria worked well.

Role of Board in the peer review process

It is common to have the Board sign off on the set of approved grants. It may be that the organization’s procurement matrix requires Board approval for the level of expenditure that will be incurred. But the Board’s role is to ensure that due process was followed in determining which proposals to select. They should not be involved in accepting or rejecting specific proposals. There may be exceptions at the margins e.g. the Board is told that we can spend X and fund these five proposals, or the Board can agree an additional Y amount to include a sixth highly ranked proposal which is also excellent and different to those we’d otherwise fund.

Inhouse capacity

All the above needs a lot of inhouse capacity. The organization needs to employ people who could do the study themselves. When I was still at the World Bank, many bilateral agencies asked for advice on commissioning impact evaluations. My main advice was hire someone who understands impact evaluation – and two of my own team left for such roles in DANIDA and USAID. At both 3ie and CEDIL we ended up putting a lot of time into improving study designs.

Monitoring the grant management process 

Especially if you will be doing more grant rounds then monitor your applications: how many, how many from developing country organizations, PI demographics, grant size applied for etc.  These data can be analysed to inform future rounds. At 3ie we wanted to give more money to developing country teams but they were never going to outperform J-PAL on technical quality, so we increased the weight given to substantive engagement of developing country PIs.

• Background
• Research requirements [include subsections on specific points such as equity]
• Timeline for the research
• Eligibility
• Expected qualifications of applicants [any preferences, eg for female headed teams , can come here].
• Proposal assessment criteria
• How to apply
• Timeline for applications
• Learning more about this RFP (with link to FAQ)

*Acknowledgement: Image generated by gencraft.com.