Today’s post is by Dr. John W. (Jack) Williams, a paleoecologist and climate scientist in the Geography department at the University of Wisconsin. Jack is the Bryson Professor of Climate, People & Environment, and the Director of the Center for Climatic Research. You can follow him on Twitter as @IceAgeEcologist.
Hi everyone. At Jacquelyn’s invitation, I’m writing a guest post about the National Science Foundation’s Division of Environmental Biology (NSF-DEB) new pre-proposal system. (Why is NSF doing this? The short version: NSF and the scientific community faces a workload crisis created by the ever increasing rates of proposal submission. For a good review of the new rules, NSF’s rationale, and why the pre-proposal system giving heartburn to many scientists, go here.) I recently served on one of the panels, and am sharing here some experiences and thoughts. The goal here is to demystify the process, and to throw in a bit of commentary from the perspective of a scientist who has both submitted and reviewed pre-proposals in this first experimental round.
Before I get started, two quick comments:
1) It is very unlikely that I saw your proposal. By voluntarily disclosing that I was a panelist, there’s a slight but non-zero chance of getting heat from disgruntled colleagues with unsuccessful pre-proposals. So let me hasten to add: NSF-DEB comprises multiple programs and each program operates multiple grant-review panels, each of which sees only a subset of all the proposals coming in. It’s unlikely that I was part of the process for your pre-proposal.
2) All opinions expressed here are my own and do not represent the National Science Foundation, the University of Wisconsin (my employer), or any of the professional societies to which I belong.
First, a quick description of the panel-review process. The panel format was pretty similar to NSF’s review process for full proposals. Panelists are invited to participate by NSF program officers, who try to get a representative mix of expertise from across their program’s domain. Each panelist had to read and submit reviews for roughly 20 pre-proposals in advance of the panel, with rigorous screening for conflicts of interest. Each pre-proposal was reviewed in advance by three panelists. Our panel had ~160 pre-proposals to discuss, and we reviewed these preproposals in 2 days. Yes, for those keeping score at home, that’s 80 proposals per day!
The NSF program officers encouraged panels to identify the top 20% or so of pre-proposals. This target was a recommendation only and panels had discretion about how closely to meet it, so the actual percentages varied.
These recommendations surprised me at first, because the pre-proposal filter was more stringent than I’d expected. But the rationale makes sense: NSF wants to give decent odds of funding to the scientists who will be invited to write full proposals. Specifically, NSF wants to at least double recent rates of funding for the full-proposals, so that there is more return on the massive amounts of time invested in writing, reviewing, and processing these proposals. But still, this is a bit sobering, because the recent rate of funding has been so low recently for many DEB programs, often <10%. Let’s assume that the rate of funding for full proposals will now be 20%. That means that even if you get past the pre-proposal hurdle, you have only a one-in-five chance of getting funded. Ugh.
What makes a successful 5-page pre-proposal?
Successful pre-proposals had to convince the panel that 1) it addressed an interesting topic(s) that would advance our understanding at critical scientific frontiers, 2) the research plan was well-thought-out and had a reasonable chance of successfully addressing that question, and 3) that the research team had the necessary expertise to carry out the proposed work.
Rocks on which pre-proposals foundered:
- Too system-specific, with not a clear enough connection drawn to broader ecological theory and research questions. This was a big one, and a major challenge for ecologists with research questions focused on particular systems.
- Vague questions and hypotheses with unclear connections to the proposed work.
- Failure to connect the dots.
- Research plan not well described or suffering from an apparent flaw.
- Lack of integration among the parts of a proposed research plan.
Overall, I was struck by how similar the review process was for the 5-page proposals and 15-page proposals. Going into this process, one of my big questions was whether a panel could adequately assess the scientific merit of a short 5-page proposal. (NSF is also very interested in this question; we held extensive debriefing discussions about this point). For me at least, the answer was yes. However, not all panelists agreed with this point.The biggest difference: there simply wasn’t enough space for pre-proposals to describe all methodological details. This tended to upweight the importance of well-posed research questions. Research plans had to pass a plausibility test, but weren’t held to the standard expected of full proposals.
The toughest cases that required the most panel deliberation occurred when pre-proposals had successfully convinced the panelists that they were tackling an interesting research question, but the research methods were too briefly described. There were no obvious flaws in the proposed methods, but there were omissions or vagueness that left the panel unable to evaluate some critical aspect. Sometimes the panel was inclined to give the pre-proposal the benefit of the doubt (under the theory that an assessment of methods could take place during the review of full proposals), and sometimes the panel wasn’t. In these borderline cases, the reputation of research team sometimes came into play. However, in general, I was impressed by how rarely the reputation of the investigators directly affected panel decisions and how consistently the review process focused on the merits of the pre-proposal in question.
Commentary on NSF-DEB’s Pre-Proposal Experiment
Given my experience on the panel, I now accept the premise that scientific merit can be (at least initially) assessed with a five-page format. That’s important; I was not convinced about this before the panel. But there are still at least three disadvantages of the pre-proposal system:
1) Workload is increased for the best proposals. The whole point of NSF-DEB’s new system is to reduce workload, and for most scientists, this will be the outcome. The burden is massively reduced for panelists and reviewers, and for the ~80% of us who will have unsuccessful pre-proposals, we only had to write 5 pages prior to rejection instead of 15. (Yes, it’s cold comfort.) But for the 20% or so of us with successful pre-proposals, our workload has increased (we wrote a 5-page proposal, and now we have to write a 15-page proposal, just like before). Of course, much of the content can be transferred from one to the other, but writing a tight and compelling proposal takes work at any length. So in a sense the pre-proposal system taxes the best science while reducing the burden for others. This seems like a perverse outcome.
2) It slows down the scientific rate of discovery. Under the old system the turnaround time between proposal submission and receiving funding (if so fortunate) might have been on the order of 5-8 months. Now it will be on the order of 11-14 months. This is a major issue, both because it slows scientific progress and because of the uncertainty created for early career scientists, grad students needing funding, etc.
3) Early-career investigators have fewer chances to succeed, and to learn. With only one funding cycle per year and 2 chances per cycle, early-career investigators have fewer chances.
Given the above concerns, I’m still not wild about the pre-proposal system, yet clearly the status quo was unsustainable. What to do instead?
I’d suggest a hybrid model that keeps the limits on the proposals submitted per investigator (there are issues here too, but some kind of limit is needed), but replaces the pre-proposal system with a shorter-format full-proposal system (8 pages? 10?). After all, there’s nothing magic about 15 pages and the current round convinced me that a shorter-format system could work well. I’d love it if NSF-DEB could return to having two submission deadlines per year, perhaps modifying the limits rule so that any single scientist could be a PI or co-PI on only 4 (3? 5?) proposals per calendar year. These limits could be relaxed for early career investigators, giving them more chances to apply.
Of course, when we take another step back, all these details about rules and proposal procedures are just rearranging deck chairs. The larger systemic issue is that we have more good scientists, doing good science, than we have resources allocated to support their research.
NSF is a model agency, and has done a remarkably effective job of distributing limited resources. Even in these tough economic times, Congress has continued to support NSF’s mission of advancing the progress of science and thereby promoting the nation’s health, prosperity, and welfare. But given all the above, it sure wouldn’t hurt to send a note sometime to your Representative or Senator, thanking them for their support of scientific research, and outlining the benefits that science (either writ large, or your own personal work) brings to our country.
Categories: Academia Commentary Guest Posts
Any difference in success rates for the “Small Grant” option?
It would help if instead of funding a small number very large budget proposals, NSF were to favor those of more modest costs–spread the wealth, so to speak. I recognize that some research is exceedingly expensive, but as a country we seemed to have evolved (devolved) towards supporting a handful of labs with teams of post-docs and PhD students all beavering away for a few very successful PIs…that approach leaves a huge number of worthy proposals and PIs unfunded.
Could not agree more. Spreading funding out some is a very good solution — more scientists should be funded. Enough feast then famine. Thanks for the discussion.
I very much like the idea of writing and reviewing 6-page proposals, thereby skipping the pre proposal stage and saving time. Surely a compelling argument for the science can be made in six pages. There should be two cycles per year with a limit of two submissions per year as a PI and two as a co-PI. The limit on submissions will improve the quality of the proposals. If you are only writing two per year you can put much more time and thought into those proposals as opposed to having the ability to submit 10 per year.
Thanks, Jack! This is an interesting summary that I think will be useful to many as our community tries to adjust to a new system.
Note, I want to correct one incorrect perception in Jack’s summary. Junior faculty still have an opportunity to submit proposals at two different times in the year to the same panel. They can submit CAREER grants via the mid-summer deadline in addition to pre-proposals for the mid-winter deadline. The CAREER proposals are full length proposals–essentially bypassing the pre-proposal stage.
Good point and thanks for the clarification. CAREER proposals remain an important separate venue for early career scientists that aren’t being held to the pre-proposal system.
Both here and at a recent NSF conference, I’ve heard the comment that junior faculty can always apply to the CAREER program and thus have two venues for proposals. I’m somewhat puzzled by this. The CAREER program is quite different in its emphasis, with both innovative research and teaching as core elements, and it’s billed as one of NSF’s most prestigious awards. I wonder how many junior faculty would seriously consider themselves for this prestigious award and how many are actively developing innovative ways to integrate teaching and research. Without the latter, junior faculty absolutely won’t be competitive for such a grant. So, unless you’ve been seriously thinking about a CAREER, you might not have the teaching component developed along with your research question. Also, PIs can only apply three times to the CAREER program. Yes, the CAREER is available to junior faculty, but I find it disingenuous to imply that it is a straightforward alternative. And, if more and more junior faculty are encouraged to write CAREERs, won’t this have the same outcome as NSF is trying to avoid with the pre-propsoals, that of increasing the number of submitted CAREER proposals, increasing demand on reviewers, and decreasing success rate (assuming funding stays static, which is probably a fair assumption)?
This is actually not accurate. The CAREER always existed – the CAREER was not added as a new RFP after scaling back from 2 proposal calls per year to one, so it was always an option for new investigators. Additionally, the CAREER is one of the most prestigious awards that the NSF funds, with a very low funding rate and a much greater emphasis on broader impact and educational outcomes than standard DEB proposals. Finally, if one were to receive a CAREER, it is very unlikely that the same investigator would receive a DEB proposal as well during the tenure of the CAREER grant. I don’t think they can be treated equivalently at all.
It has been common for someone in the genomics and informatics business, like me, where project teams are typically large, to be co-P.I. or to lead a subaward on multiple proposals for specific, relatively utilitarian parts of the research. This change at NSF is very punitive to me, and I’d argue damaging to progress in the field, to have such a strict limit on on the number of annual proposals one can make in these roles. Secondly, this new system further exacerbates the shortcomings of the all-or-nothing “tournament” model for research success, in which a good scientist can lose his or her career by little more than sampling bias in the few proposals permitted in the time span it takes to reach failure. Lastly, I suspect that there is a substantial difference among projects and among investigators in the likelihood of making a strong impression in such a short pre-proposal versus having more space to lay out the challenges and research plan.
Thanks, Jack, for your thoughts on this process. You didn’t mention it, but I presume one of the issues that came up was an avalanche of pre-proposals, so that pre-proposals might further strain the system they were designed to help. The point about the relative weight put on the attractiveness of ideas vs. proof of competence in methods was, I thought, obvious from the get-go. I do NOT like to hear about the degree of arbitrariness used in judging methods. The methods simply can NOT be held to the same rigorous standards if we go to a five-page format, and to hold some proposals hostage to details of the methods while not doing that for others seems unfair and prejudicial on its face.
I worry that the new system is going to wind up destroying many labs, led by both new and established investigators. You simply can NOT have a lab survive for, say, three years without funding. Yet I believe that with only one cycle per year, and only a 20% chance that any given pre-proposal will be funded, it will be relatively easy to miss a few times in a row … and then goodbye cutting edge research, goodbye recruiting top graduate students, goodbye tenure and career in many cases.
Yes, the current system is unsustainable. But the replacement system appears, to me, to be deeply flawed. What is to be done?
I would recommend ELIMINATING 15-page proposals AND 5-page pre-proprosals, and going to 6-page full proposals, no more than 2 per cycle in DEB (or maybe 3 per year), and KEEPING two review cycles per year. Advantages of this approach would be (1) lowering the chance of killing labs and investigators; (2) greatly reducing the review load and writing load from the previous system; (3) avoiding the perverse “tax” on successful proposals that you mentioned; and (4) reducing emphasis on finding truly picayune details with which to sink proposals and focusing on the key issues of innovative research and general demonstration (supported by publications). I think the importance of the last point cannot be over-emphasized. A bit less information for the reviewers might make some take their role a bit more humbly, and stop the process of panels fine-tuning proposals for reviewers.
What do you think? Would it be worthwhile suggesting such a system to NSF?
Thanks for this! Appreciate the insights into the process.
One advantage I see of this system from the investigator point of view is that budgets and mountain of paperwork and time associated with them aren’t required for preproposals. Your proposal’s scope obviously has to be within the realm of reasonable given the funding conventions of the program, but you don’t have to account for every $1 that you will spend, complete with internal university paperwork, justification and headache. Only if you have a “real” shot at getting funding (i.e., invitation for full proposal) do you put yourself through it. (Disclosure that I haven’t gone through the DEB process, but I have done other competitive preproposals.)
In the UK we have something of a hybrid. Proposals for the principal environment funding body (NERC) are ‘only’ 8 pages (for the science case) but all proposals are first screened by College members so that only c. 50% get sent out for full external review. Of course of that ‘lucky’ 50%, success rates are still very low.