HUD Program Evaluation Policy-Policy Statement

AGENCY:

Office of the Assistant Secretary for Policy Development and Research, HUD.

ACTION:

Notice.

SUMMARY:

This policy statement of HUD's Office of Policy Development and Research articulates the core principles and practices of the office's evaluation and research activities. This policy reconfirms the Department's commitment to conducting rigorous, relevant evaluations and to using evidence from evaluations to inform policy and practice.

DATES:

December 6, 2016.

FOR FURTHER INFORMATION CONTACT:

Mark D. Shroder, Associate Deputy Assistant Secretary, Office of Research, Evaluation, and Monitoring, Office of Policy Development and Research, U.S. Department of Housing and Urban Development, 451 7th Street SW., Washington, DC 20410, telephone (202) 402-5922. The listed telephone number is not a toll-free number. Persons with hearing- or speech-impairments may access this number through TTY by calling Federal Relay Service at 1-800-877-8339 (this is a toll-free number).

SUPPLEMENTARY INFORMATION:

I. Background

The mission of HUD's Office of Policy Development and Research (PD&R) is to inform HUD policy development and implementation to improve life in American communities through conducting, supporting, and sharing research, surveys, demonstrations, program evaluations, and best practices. Within HUD, PD&R is responsible for nearly all program evaluations. The office provides reliable and objective data and analysis to help inform policy decisions. Program evaluation has been a core activity of PD&R since its formation in 1974.

In July 2016, the Government Accountability Office (GAO) issued a report entitled “Department of Housing and Urban Development: Actions Needed to Incorporate Key Practices into Management Functions and Program Oversight,” (GAO 16-497) in which GAO presented a broad assessment of HUD's management of its operations and programs. In the report, GAO examined HUD efforts to: (1) Meet Federal requirements and implement key practices for management functions, including performance planning and reporting, human capital, financial, acquisition, and information technology (IT) management; and (2) oversee and evaluate programs.

PD&R is the primary office within HUD responsible for data analysis, research, program evaluations, and policy studies that inform the development and implementation of programs and policies across HUD offices. PD&R undertakes program evaluations, often by using a process that includes convening expert panels. However, GAO found that PD&R had not developed agency-wide, written policies for its program evaluations, nor documented the criteria used to select the expert panels and review the quality of program evaluations.

This policy statement responds to the GAO report by setting out the core principles and practices of PD&R's evaluation and research activities. This statement incorporates some language from a policy statement by the Office of Policy, Research, and Evaluation of the Administration for Children and Families of the U.S. Department of Health and Human Services.

II. HUD Program Evaluation Policy

PD&R has identified the following core principles and practices as fundamental to ensuring high-quality and consistent evaluation results: rigor, relevance, transparency, independence, ethics, and technical innovation. This policy applies to all PD&R-sponsored evaluations and economic analyses of regulations; they apply as well to the selection of projects, contractors, and PD&R staff that is involved in evaluations.

Rigor

PD&R is committed to using the most rigorous methods that are appropriate to the evaluation questions and feasible within budget and other constraints. Rigor is not restricted to impact evaluations, but is also necessary in implementation or process evaluations, descriptive studies, outcome evaluations, and formative evaluations; and in both qualitative and quantitative approaches. Rigor requires ensuring that inferences about cause and effect are well founded (internal validity); requires clarity about the populations, settings, or circumstances to which results can be generalized (external validity); and requires the use of measures that accurately capture the intended information (measurement reliability and validity).

In assessing the effects of programs or services, PD&R evaluations use methods that isolate to the greatest extent possible the impacts of the programs or services from other influences such as trends over time, geographic variation, or pre-existing differences between participants and non-participants. For such causal questions, experimental approaches are preferred. When experimental approaches are not feasible, PD&R uses the most rigorous approach that is feasible. PD&R ensures that contractors and grantees conducting evaluations have appropriate expertise through emphasizing the capacity for rigor in requests for proposal and funding opportunity announcements.

PD&R also employs a strategic human capital development plan to hire, train, and retain a workforce that ensures the staff has the tools and resources to accomplish the mission.

Relevance

The PD&R evaluation agenda reflects the legislative requirements and policy issues related to HUD's mission. PD&R solicits input from stakeholders, both internal and external, on the selection of programs to be evaluated, initiatives, demonstrations, and research questions. For new initiatives and demonstrations in particular, evaluations will be more feasible and useful when planned in advance, in concert with the development of the initiative or demonstration, rather than as an afterthought.

PD&R disseminates findings in ways that are accessible and useful to policy-makers and practitioners. PD&R partners with other HUD program offices to inform internal and external stakeholders through disseminating evidence from PD&R-sponsored evaluations.

Transparency

PD&R will release methodologically valid evaluations without regard to the findings. Evaluation reports must describe the methods used, including strengths and weaknesses, and discuss the generalizability of the findings. Evaluation reports must present comprehensive results, including favorable, unfavorable, and null findings.

PD&R publishes a 5-year Research Roadmap that outlines the research and evaluation that we believe would be of greatest value to public policy. PD&R lists all ongoing evaluation projects at the HUDUSER.gov Web site, and updates it monthly. PD&R will release evaluation results timely, usually within 4 months of receiving the final report.

PD&R will, where possible, archive evaluation data for secondary use by interested researchers. PD&R typically builds requirements into contracts to prepare data sets for secondary use.

Independence

Independence and objectivity are core principles of evaluation. Agency and program leadership, program staff, service providers, and others participate actively in setting evaluation priorities, identifying evaluation questions, and assessing the implications of findings. However, it is important to insulate evaluation functions from undue influence and from both the appearance and the reality of bias. To promote objectivity, PD&R protects independence in the design, conduct, and analysis of evaluations. To this end:

• PD&R conducts evaluations through the competitive award of grants and contracts to external experts who are free from conflicts of interest.
• PD&R also conducts evaluations in-house and supports unsolicited external evaluation proposals with funding, data, or both.
• The Assistant Secretary for PD&R has authority to approve the design of evaluation projects and analysis plans; and has authority to approve, release, and disseminate evaluation reports. The Assistant Secretary does so, in consultation with career staff.

Ethics

PD&R-sponsored evaluations must be conducted in an ethical manner and safeguard the dignity, rights, safety, and privacy of participants. PD&R-sponsored evaluations must comply with both the spirit and the letter of relevant requirements such as regulations governing research involving human subjects. In particular, PD&R protects the privacy of HUD-assisted households and HUD-insured borrowers through the Rule of Eleven; that is, PD&R allows no disclosure of information about the characteristics of any group of individuals or households numbering less than eleven by PD&R staff, contractors, grantees, or licensees.

Technical Innovation

PD&R supports and employs new methods of data collection and analysis that more reliably and efficiently answer research questions than old methods do.

Application of These Principles to Economic Analysis of Regulations

Economic analysis of regulations, properly conducted, is a critical tool in improving public policy. In any PD&R Regulatory Impact Analysis:

• PD&R analyzes whether the issues addressed by the regulation stem from a market failure, government failure, or other systemic problem, and whether the regulation addresses the root causes of those problems.
• PD&R uses and as necessary produces the best objective estimates of the benefits, costs, and transfers resulting from the regulation, taking into account gaps and uncertainties in the available data.
• Where clear alternatives to the regulatory actions exist, PD&R objectively estimates the benefits, costs, and transfers of those alternatives as well.