Agency Information Collection Activities: Comment Request; Merit Review Survey-2021 and 2023 Assessment of Applicant and Reviewer Experiences

AGENCY:

National Science Foundation.

ACTION:

Submission for OMB review; comment request.

SUMMARY:

The National Science Foundation (NSF) has submitted the following information collection requirement to OMB for review and clearance under the Paperwork Reduction Act of 1995. This is the second notice for public comment; the first was published in the Federal Register, and no comments were received. NSF is forwarding the proposed submission to the Office of Management and Budget (OMB) for clearance simultaneously with the publication of this second notice.

DATES:

Written comments and recommendations for the proposed information collection should be sent within 30 days of publication of this notice to www.reginfo.gov/public/do/PRAmain. Find this particular information collection by selecting “Currently under 30-day Review—Open for Public Comments” or by using the search function.

FOR FURTHER INFORMATION CONTACT:

Suzanne H. Plimpton, Reports Clearance Officer, National Science Foundation, 2415 Eisenhower Avenue, Alexandria, VA 22314, or send email to splimpto@nsf.gov. Individuals who use a telecommunications device for the deaf (TDD) may call the Federal Information Relay Service (FIRS) at 1-800-877-8339, which is accessible 24 hours a day, 7 days a week, 365 days a year (including federal holidays).

Copies of the submission may be obtained by calling 703-292-7556.

SUPPLEMENTARY INFORMATION:

NSF may not conduct or sponsor a collection of information unless the collection of information displays a currently valid OMB control number and the agency informs potential persons who are to respond to the collection of information that such persons are not required to respond to the collection of information unless it displays a currently valid OMB control number.

Title of Collection: Merit Review Survey—2021 & 2023 Assessment of Applicant and Reviewer Experiences.

OMB Number: 3145-NEW.

Type of Request: Request for approval to establish an information collection.

Proposed Project: The National Science Foundation (NSF) receives close to 50,000 proposals for funding annually, each of which undergoes a rigorous merit review process that is designed to ensure all proposals are fairly and thoroughly reviewed. The merit review process comprises three phases:

1. NSF announces funding opportunities on the NSF website and Grants.gov. Applicants prepare proposals in response to these opportunities and submit their proposals via FastLane (NSF's web-based system for proposal submission and review) or Grants.gov.

2. Proposals are assigned to the appropriate program(s) for review. Each proposal is assigned a Program Officer (PO) who selects external reviewers to evaluate the proposal according to the two NSF merit review criteria, Intellectual Merit and Broader Impacts. The Intellectual Merit criterion encompasses the potential to advance knowledge. The Broader Impacts criterion encompasses the potential to benefit society and contribute to the achievement of specific, desired societal outcomes. Programs may have additional review criteria particular to the goals and objectives of the program. The NSF guidelines for the selection of reviewers are designed to ensure selection of experts who can give program officers the proper information needed to make a recommendation in accordance with the merit review criteria. POs utilize the proposal's reference list, the investigator's suggested reviewers, and personal knowledge of individual reviewers to identify a pool of diverse experts with respect to type of organization represented, demographics, experience, and geographic balance, selecting appropriate reviewers with no apparent potential conflicts. Most proposals are reviewed by three to ten content expert reviewers who provide written feedback on the proposal through FastLane. POs synthesize reviewer comments and issue a recommendation to either decline or award funding based on reviewer feedback, panel discussions, the amount of available funding, and portfolio balances (i.e., the diversity of a portfolio, including factors such as award type, career stage, demographic characteristics, geographic location, institution type, research topic, laboratory funding status, and intellectual risk). The proposal and PO recommendation is then forwarded to the appropriate Division Director or other NSF official for additional review and action to either decline or award.

3. Each proposal recommended for award undergoes an administrative review conducted by NSF's Office of Budget, Finance, and Award Management. If it passes this review, the proposal is awarded.

Through this review process, NSF aims to identify the highest quality proposals to receive funding. The success of this process hinges on the assumptions that applicants will continue to submit to NSF their ideas for cutting-edge research and that experts in their respective fields will continue to provide high-quality reviews of those proposals.

The goal of this data collection is to assess the experiences of applicants and reviewers and their satisfaction with NSF's merit review process. The data collection for which this OMB approval is requested includes a Web-based survey that will be administered to all applicants and reviewers who participated in the merit review process between fiscal years (FY) 2018 and FY 2020 (2021 survey) and between FY 2020 and FY 2022 (2023 survey).

The specific research objectives are to—

1. Assess applicant and reviewer perceptions of, and satisfaction with, various aspects of the merit review process.

2. Document the time burden the merit review process places on reviewers and applicants.

3. Examine applicant and reviewer perceptions of the quality of reviews and of proposals.

4. Assess the changes in applicant and reviewer perceptions of burden, satisfaction, and quality between the 2019 and 2021 surveys and the 2021 and 2023 surveys.

5. Examine the variation of applicant and reviewer perception of satisfaction, burden, and quality by key population subgroups, including race/ethnicity, gender, and disability.

6. Describe the extent to which NSF's reviewer orientation video is correlated with awareness of different types of cognitive biases and the use of strategies to reduce cognitive bias and to provide constructive feedback.

7. Describe the extent to which the elimination of annual proposal deadlines affected reviewer and applicant burden, perceptions of proposal and review quality, and satisfaction with the merit review process.

8. Describe applicants and reviewers experiences with student support programs as well as what NSF application and funding support is associated with the receipt of financial support from NSF as an undergraduate or graduate student.

Data from the survey will be used to improve NSF's implementation of the merit review process.

Use of the information: The primary purpose of collecting this information is program evaluation. The data collected will enable NSF to assess the satisfaction, including perceptions of burden and quality, of applicants and reviewers who participate in the merit review process in order to monitor and improve the program and assess its implementation. Findings will inform continual improvement activities related to the merit review process.

Expected respondents: All applicants who have submitted proposals and reviewers who have reviewed NSF proposals between FY 2018 and 2020 will be invited to participate in the 2021 survey and comparable individuals who participated between FY 2020 and FY 2022 will be invited to participate in the 2023 survey. This is estimated to be approximately 87,000 individuals per survey round.

Average time per reporting: The online survey is comprised primarily of close-ended questions and is designed to be completed by respondents in approximately 20 minutes.

Frequency: Eligible applicants and reviewers will be asked to the complete the 2021 Merit Review survey one time in fall 2021. For the 2023 survey, eligible applicants and reviewers will be asked to complete the survey one time in fall 2023.

Estimate of burden: It is estimated the survey will require approximately 20 minutes (on average) to complete. The anticipated universe size for each survey cycle is 87,000 individuals, which includes all applicants who submitted proposals and all reviewers between FY 2018 and FY 2020 (for the 2021 survey) and between FY 2020 and FY 2022 (for the 2023 survey). The estimated survey response rate for each the 2021 and 2023 survey rounds is 40 percent. Therefore, the total burden is 23,200 hours; this is a respondent burden of 11,600 hours per survey year (2021 and 2023).

Based on 2019 merit review survey data, it is anticipated that most survey respondents will be working at an academic institution, likely in a teaching and/or research capacity. Therefore, for the purpose of burden estimates, we have used the annual mean wage for postsecondary teachers from Bureau of Labor Statistics, which is $79,540. Assuming a 40-hour workweek over the course of 52 weeks annually, the hourly wage for this occupation is approximately $38.00. Therefore, the overall cost to survey respondents for each survey year (2021 and 2023) would be approximately $440,800 (11,600 burden hours × $38.00 per hour), as shown in table A.12.1 below.

Comments: Comments are invited on (a) whether the proposed collection of information is necessary for the proper performance of the functions of the Agency, including whether the information shall have practical utility; (b) the accuracy of the Agency's estimate of the burden of the proposed collection of information; (c) ways to enhance the quality, utility, and clarity of the information on respondents, including through the use of automated collection techniques or other forms of information technology; and (d) ways to minimize the burden of the collection of information on those who are to respond, including through the use of appropriate automated, electronic, mechanical, or other technological collection techniques or other forms of information technology.