Today’s post is by Dr. John W. (Jack) Williams, a paleoecologist and climate scientist in the Geography department at the University of Wisconsin. Jack is the Bryson Professor of Climate, People & Environment, and the Director of the Center for Climatic Research. You can follow him on Twitter as @IceAgeEcologist.

Jack Williams is Professor of Geography at UW Madison

Hi everyone.  At Jacquelyn’s invitation, I’m writing a guest post about the National Science Foundation’s Division of Environmental Biology (NSF-DEB) new pre-proposal system.  (Why is NSF doing this?  The short version: NSF and the scientific community faces a workload crisis created by the ever increasing rates of proposal submission.  For a good review of the new rules, NSF’s rationale, and why the pre-proposal system giving heartburn to many scientists, go here.)  I recently served on one of the panels, and am sharing here some experiences and thoughts.  The goal here is to demystify the process, and to throw in a bit of commentary from the perspective of a scientist who has both submitted and reviewed pre-proposals in this first experimental round.

Before I get started, two quick comments:

1) It is very unlikely that I saw your proposal.  By voluntarily disclosing that I was a panelist, there’s a slight but non-zero chance of getting heat from disgruntled colleagues with unsuccessful pre-proposals.  So let me hasten to add:  NSF-DEB comprises multiple programs and each program operates multiple grant-review panels, each of which sees only a subset of all the proposals coming in.  It’s unlikely that I was part of the process for your pre-proposal.

2) All opinions expressed here are my own and do not represent the National Science Foundation, the University of Wisconsin (my employer), or any of the professional societies to which I belong.

First, a quick description of the panel-review process.  The panel format was pretty similar to NSF’s review process for full proposals.  Panelists are invited to participate by NSF program officers, who try to get a representative mix of expertise from across their program’s domain.  Each panelist had to read and submit reviews for roughly 20 pre-proposals in advance of the panel, with rigorous screening for conflicts of interest.  Each pre-proposal was reviewed in advance by three panelists.  Our panel had ~160 pre-proposals to discuss, and we reviewed these preproposals in 2 days.  Yes, for those keeping score at home, that’s 80 proposals per day!

The NSF program officers encouraged panels to identify the top 20% or so of pre-proposals.  This target was a recommendation only and panels had discretion about how closely to meet it, so the actual percentages varied.

These recommendations surprised me at first, because the pre-proposal filter was more stringent than I’d expected.  But the rationale makes sense:  NSF wants to give decent odds of funding to the scientists who will be invited to write full proposals.  Specifically, NSF wants to at least double recent rates of funding for the full-proposals, so that there is more return on the massive amounts of time invested in writing, reviewing, and processing these proposals.  But still, this is a bit sobering, because the recent rate of funding has been so low recently for many DEB programs, often <10%.  Let’s assume that the rate of funding for full proposals will now be 20%.  That means that even if you get past the pre-proposal hurdle, you have only a one-in-five chance of getting funded.  Ugh.

What makes a successful 5-page pre-proposal?

Successful pre-proposals had to convince the panel that 1) it addressed an interesting topic(s) that would advance our understanding at critical scientific frontiers, 2) the research plan was well-thought-out and had a reasonable chance of successfully addressing that question, and 3) that the research team had the necessary expertise to carry out the proposed work.

Rocks on which pre-proposals foundered:

• Too system-specific, with not a clear enough connection drawn to broader ecological theory and research questions.  This was a big one, and a major challenge for ecologists with research questions focused on particular systems.
• Vague questions and hypotheses with unclear connections to the proposed work.
• Failure to connect the dots.
• Research plan not well described or suffering from an apparent flaw.
• Lack of integration among the parts of a proposed research plan.

National Science Foundation

Overall, I was struck by how similar the review process was for the 5-page proposals and 15-page proposals.  Going into this process, one of my big questions was whether a panel could adequately assess the scientific merit of a short 5-page proposal.  (NSF is also very interested in this question; we held extensive debriefing discussions about this point).  For me at least, the answer was yes.  However, not all panelists agreed with this point.The biggest difference:  there simply wasn’t enough space for pre-proposals to describe all methodological details.  This tended to upweight the importance of well-posed research questions.  Research plans had to pass a plausibility test, but weren’t held to the standard expected of full proposals.

The toughest cases that required the most panel deliberation occurred when pre-proposals had successfully convinced the panelists that they were tackling an interesting research question, but the research methods were too briefly described.  There were no obvious flaws in the proposed methods, but there were omissions or vagueness that left the panel unable to evaluate some critical aspect.  Sometimes the panel was inclined to give the pre-proposal the benefit of the doubt (under the theory that an assessment of methods could take place during the review of full proposals), and sometimes the panel wasn’t.  In these borderline cases, the reputation of research team sometimes came into play.  However, in general, I was impressed by how rarely the reputation of the investigators directly affected panel decisions and how consistently the review process focused on the merits of the pre-proposal in question.

Commentary on NSF-DEB’s Pre-Proposal Experiment

Given my experience on the panel, I now accept the premise that scientific merit can be (at least initially) assessed with a five-page format.  That’s important; I was not convinced about this before the panel.  But there are still at least three disadvantages of the pre-proposal system:

1) Workload is increased for the best proposals.  The whole point of NSF-DEB’s new system is to reduce workload, and for most scientists, this will be the outcome.  The burden is massively reduced for panelists and reviewers, and for the ~80% of us who will have unsuccessful pre-proposals, we only had to write 5 pages prior to rejection instead of 15.  (Yes, it’s cold comfort.)  But for the 20% or so of us with successful pre-proposals, our workload has increased (we wrote a 5-page proposal, and now we have to write a 15-page proposal, just like before).  Of course, much of the content can be transferred from one to the other, but writing a tight and compelling proposal takes work at any length.  So in a sense the pre-proposal system taxes the best science while reducing the burden for others.  This seems like a perverse outcome.

2) It slows down the scientific rate of discovery.  Under the old system the turnaround time between proposal submission and receiving funding (if so fortunate) might have been on the order of 5-8 months.  Now it will be on the order of 11-14 months.  This is a major issue, both because it slows scientific progress and because of the uncertainty created for early career scientists, grad students needing funding, etc.

3) Early-career investigators have fewer chances to succeed, and to learn.  With only one funding cycle per year and 2 chances per cycle, early-career investigators have fewer chances.

Given the above concerns, I’m still not wild about the pre-proposal system, yet clearly the status quo was unsustainable.  What to do instead?

I’d suggest a hybrid model that keeps the limits on the proposals submitted per investigator (there are issues here too, but some kind of limit is needed), but replaces the pre-proposal system with a shorter-format full-proposal system (8 pages?  10?).  After all, there’s nothing magic about 15 pages and the current round convinced me that a shorter-format system could work well.  I’d love it if NSF-DEB could return to having two submission deadlines per year, perhaps modifying the limits rule so that any single scientist could be a PI or co-PI on only 4 (3? 5?) proposals per calendar year.  These limits could be relaxed for early career investigators, giving them more chances to apply.

Of course, when we take another step back, all these details about rules and proposal procedures are just rearranging deck chairs.  The larger systemic issue is that we have more good scientists, doing good science, than we have resources allocated to support their research.

NSF is a model agency, and has done a remarkably effective job of distributing limited resources.  Even in these tough economic times, Congress has continued to support NSF’s mission of advancing the progress of science and thereby promoting the nation’s health, prosperity, and welfare.  But given all the above, it sure wouldn’t hurt to send a note sometime to your Representative or Senator, thanking them for their support of scientific research, and outlining the benefits that science (either writ large, or your own personal work) brings to our country.

Categories: Academia Commentary Guest Posts

Tagged as: ecology funding nsf politics research

Jacquelyn Gill