Proposed Information Collection Submitted for Public Comment and Recommendations: Data Validation Requirement for Employment and Training Programs

AGENCY:

Employment and Training Administration (ETA), Labor.

ACTION:

Notice.

SUMMARY:

The Department of Labor, as part of its continuing effort to reduce paperwork and respondent burden, conducts a pre-clearance consultation program to provide the general public and Federal agencies with an opportunity to comment on proposed and/or continuing collections of information in accordance with the Paperwork Reduction Act of 1995 (PRA95) (44 U.S.C. 3506(c)(2)(A)). This program helps to ensure that requested data can be provided in the desired format, the reporting burden (time and financial resources) is minimized, the collection instruments are clearly understood, and the impact of collection requirements on respondents can be properly assessed. The Employment and Training Administration (ETA) is soliciting comments on the establishment of a data validation requirement for the following employment and training programs: Workforce Investment Act (WIA) Title IB, Labor Exchange, Trade Adjustment Assistance (TAA), Migrant and Seasonal Farmworkers (MSFW), Native American Employment and Training, and Senior Community Service Employment Program (SCSEP).

DATES:

Submit comments on or before March 16, 2004.

ADDRESSES:

Send comments to: Gail Eulenstein, Performance and Results Office, Employment and Training Administration, U.S. Department of Labor, 200 Constitution Avenue, NW., Room N-5309, Washington, DC 20210; telephone: (202) 693-3013 (this is not a toll-free number); fax: (202) 693-3991; e-mail: Eulenstein.Gail@dol.gov.

FOR FURTHER INFORMATION CONTACT:

Gail Eulenstein, Performance and Results Office, Employment and Training Administration, U.S. Department of Labor, 200 Constitution Avenue, NW., Room N-5309, Washington, DC 20210; telephone: (202) 693-3013 (this is not a toll-free number); fax: (202) 693-3991; e-mail: Eulenstein.Gail@dol.gov.

SUPPLEMENTARY INFORMATION:

I. Background

The accuracy and reliability of program reports submitted by States and grantees using Federal funds are fundamental elements of good public administration, and are necessary tools for maintaining and demonstrating system integrity. The President's Management Agenda to improve the management and performance of the Federal government has emphasized the importance of complete information for program monitoring and improving program results.

States and grantees receiving funding under WIA Title I, Wagner-Peyser Act, TAA, and SCSEP are required to maintain and report accurate program and financial information (WIA section 185 (29 U.S.C. 2935) and WIA Regulations 20 CFR 667.300(e)(2); Wagner-Peyser Act section 10 (29 U.S.C. 49i), Older Americans Act section 503(f)(3) and (4) (42 U.S.C. 3056a(f)(3) and (4)), and TAA Regulations 20 CFR 617.57). Further, all States and grantees receiving funding from ETA and the Veterans' Employment and Training Service are required to submit reports or participant records and attest to the accuracy of these reports and records.

Recent performance audits conducted by the Department of Labor's Office of the Inspector General, however, found that the accuracy of reported performance outcomes cannot be assured due to insufficient local, State, and Federal oversight. To address this concern and meet the Agency's goal for accurate and reliable data, ETA committed to the development and implementation of a data validation process in order to ensure the accuracy of data collected and reported on program activities and outcomes.

Data Validation. The data validation requirement for employment and training programs will strengthen the workforce system by ensuring that accurate and reliable information on program activities and outcomes is available. Data validation is intended to accomplish the following goals:

• Ensure that critical performance data are accurate.
• Detect and identify specific problems with a State's or grantee's reporting process, including software and data issues, to enable the State or grantee to correct the problems.
• Help States and grantees analyze the causes of performance successes and failures by displaying participant data organized by performance outcomes. In addition, the process will allow States and grantees to select appropriate validation samples necessary to compute statistically significant error rates.

Data validation consists of two parts:

(1) Report validation evaluates the validity of aggregate reports submitted to ETA by checking the accuracy of the reporting software used to calculate the reports. Report validation is accomplished by processing an entire file of participant records into validation counts and comparing the validation counts to those reported by the State or grantee.

(2) Data element validation assesses the accuracy of participant data records. Data element validation is performed by reviewing samples of participant records against source documentation to ensure compliance with Federal definitions.

Data Validation Pilot Test. Two States participated in a pilot test of the validation process in the summer and fall of 2002. Grantees in the MSFW program, Native American Employment and Training program, and SCSEP will begin pilot tests by the end of CY 2003. The pilot States conducted validation for the WIA Title IB, Labor Exchange, and TAA programs. The States received preparatory training prior to beginning validation and technical assistance throughout the pilot from ETA's validation contractor. The pilot test indicated the following:

• States and grantees will generally be able to conduct data validation with a reasonable but sustained level of effort.
• The validation process allows States and grantees to identify and address reporting errors.
• States and grantees do make reporting errors which need detecting and fixing.
• The average staff requirements for a State to complete validation for the WIA Title IB, Labor Exchange, and TAA programs will be about 882 hours per year. The average annual time required by grantees operating MSFW programs, Native American Employment and Training programs, and SCSEP to complete validation is approximately 102 hours. Start-up activities in the initial year of validation will require an additional 264 hours on average per State and 74 hours on average per grantee.

II. Desired Focus of Comments

Currently, ETA is soliciting comments about the proposed new collection of information on the validity of data that States and grantees report to the Agency. ETA is seeking Office of Management and Budget (OMB) approval under PRA95 to establish a data validation requirement for the following employment and training programs: WIA Title IB, Labor Exchange, TAA, MSFW, Native American Employment and Training, and SCSEP. Data validation will increase the reporting burden by requiring States and grantees to submit reports on data validity to ETA.

A copy of the proposed information collection request can be obtained by contacting the office listed above in the ADDRESSES section of this notice. The Department of Labor is particularly interested in comments which:

• Evaluate whether the proposed collection of information is necessary for the proper performance of the functions of the Agency, especially whether the information will have practical utility;
• Evaluate the accuracy of the Agency's estimate of the burden of the proposed collection of information, including the validity of the methodology and assumptions used;
• Discuss how to enhance the quality, utility, and clarity of the information to be collected; and
• Suggest how to minimize the burden of the collection of information on those who are to respond, including through the use of appropriate automated, electronic, mechanical, or other technological collection techniques or other forms of information technology (e.g., permitting electronic submissions of responses).

III. Current Actions

The Department proposes the following plan for implementing and operating data validation:

• In order to ensure the accuracy of reported information throughout the workforce investment system, States and grantees will be required to conduct data validation and submit validation output reports to ETA. States will initiate data validation for WIA Title IB, Labor Exchange, and TAA by the end of CY 2003, and grantees operating MSFW programs, Native American Employment and Training programs, and SCSEP will initiate validation during CY 2004.
• Data validation will be required annually. States and grantees will be required to send validation output reports to ETA within 120 days after the submission of required annual reports or participant records to ETA. Report validation will be performed prior to the submission of reports. Data element validation will be completed within 120 days after required annual reports or participant records are due to ETA.
• ETA has developed a set of validation tools discussed below—instructional handbooks, software, and user guides—that States and grantees can use to validate data. States and grantees may use an alternative methodology and tools as long as the methodology meets standards for sampling methods and confidence intervals. States or grantees that do not use the validation tools provided by ETA will be required to document that the alternative methodology is statistically valid.
• In addition to performing validation, the ETA software can be used to generate the aggregate information required in reports submitted to ETA. States or grantees that use the software provided by ETA to generate this aggregate information will not be required to perform report validation.
• ETA will establish acceptable levels for the accuracy of reports and data elements. These accuracy standards will be established in phases. The initial validation year will focus on detecting and resolving any issues with State and grantee data and reporting systems. Error rates collected in the second year will be analyzed, and, based on this information, standards for accuracy will be established prior to the third year of validation.
• Once accuracy standards are established, States and grantees will be held accountable for meeting those standards and will be required to address any issues concerning data accuracy. States and grantees that fail to meet accuracy standards will receive technical assistance from ETA and will develop and implement a corrective action plan. Data that does not meet accuracy standards will not be acceptable for measuring performance, and may keep the State or grantee from being eligible for incentives that are awarded based on performance data. Significant or unresolved deviation from accuracy standards may be deemed a failure to report.

Resources. The requirement to perform validation derives from States' and grantees' responsibility to provide accurate information on program activities and outcomes to ETA. States and grantees are expected to provide resources for conducting validation from their administrative funds. Validation of program performance is a basic responsibility of grantees, who are required to report on program performance, under Department of Labor regulations (29 CFR 95.51 and 97.40). ETA has taken a number of steps to minimize the resources needed for data validation, including developing tools that States and grantees can use to perform validation. The estimates provided below, which are based on state pilot experiences, indicate that annual staff requirements for continuing operations of data validation will be on average 882 hours (or less than 1/2 of a staff year) for a State and 102 hours (or about 1/20 of a staff year) for a grantee.

Data Validation Tools. To reduce startup costs related to implementing data validation, ETA has developed standardized software, instructional handbooks, and user guides that States and grantees can use to perform data validation:

• Software developed by ETA generates samples, worksheets, and reports on data accuracy. For report validation, the software will validate the accuracy of aggregate reports that are generated by the State's or grantee's reporting software and will produce an error rate for each reported count. For data element validation, the software generates a sample of the participant records and data elements for the state or grantee to validate. The software produces worksheets on which the validator records information after checking the source documentation in the sampled case files. The software calculates error rates for each data element, with confidence intervals of 3.5 percent for large States/grantees and 4 percent for small States/grantees.
• Handbooks provide detailed information on the validation methodology, including sampling specifications and data element validation instructions for each data element to be validated.
• User guides developed for each ETA validation software application guide States and grantees through the process of installing the application, building and loading a validation file, and completing report and data element validation.

Data Recording and Reports. States and grantees will record the results of their validation on spreadsheet software prepared as an accompaniment to their handbooks. Initially, the spreadsheets can be transmitted by e-mail to ETA. Eventually, the results will be submitted in the same manner as other reports. The results will be stored in a dataset in the National Office in Washington, DC, and compiled in an annual validation accuracy report.

Training and Technical Assistance. ETA provided validation training to States in regional sessions during the summer of 2003. Training for grantees of the MSFW and Native American Employment and Training programs will be held during winter 2003/04, and training will be provided for SCSEP grantees during spring 2004. States and grantees may obtain technical assistance on validation procedures and the use of the validation tools by contacting ETA's data validation contractor.

Type of Review: New.

Agency: Employment and Training Administration.

Title: Data Validation Requirement for Employment and Training Programs.

OMB Number: 1205-0NEW.

Recordkeeping: States and grantees must maintain complete records of all validation activities for three years. The retention requirement will apply to records of all validation activities, including files, worksheets, reports, and source documentation.

Affected Public: State, local, and tribal government entities and private non-profit organizations.

Total Respondents: 317 (53 states will perform validation for the WIA Title IB, Labor Exchange, and TAA programs annually. 264 grantees operating MSFW programs, Native American Employment and Training programs, and SCSEP will perform validation annually).

Frequency: Complete data validation annually.

Total Responses: 317 (53 responses from states annually and 264 responses from grantees annually).

Estimated Time per Response: 882 hours per year on average for a state to complete validation of the WIA Title IB, Labor Exchange, and TAA programs. 102 hours per year on average for a grantee operating a MSFW program, Native American Employment and Training program, or SCSEP to perform validation.

Total Burden Hours: An estimated 46,732 hours per year will be required for all states to complete validation for the WIA Title IB, Labor Exchange, and TAA programs. An estimated 13,992 hours will be necessary by all states for startup activities in the initial year of validation. An estimated 26,830 hours per year will be required for all grantees operating MSFW programs, Native American Employment and Training programs, and SCSEP to perform validation. An estimated 19,552 hours will be necessary by all grantees for startup activities in the initial year of validation.

Total Burden Cost (startup): The start-up cost is estimated to be $454,740 for all states in the initial year of validation for the WIA Title IB, Labor Exchange, and TAA programs ($8,580 on average per state). The start-up cost is estimated to be $312,322 for all grantees in the initial year of validation for MSFW, Native American Employment and Training, and SCSEP ($1,183 on average per grantee).

Total Burden Cost (operating): The cost is estimated to be $1,518,791 per year for all states to complete validation for the WIA Title IB, Labor Exchange, and TAA programs ($28,656 on average per state). The cost is estimated to be $495,767 per year for all grantees operating MSFW programs, Native American Employment and Training programs, and SCSEP to perform validation ($1,878 on average per grantee).

Summary of Burden

Data validation is estimated to require an annual burden of 73,562 hours and $2,015,000 for all six programs subject to the validation requirement. An additional 33,544 hours and $767,000 will be required for startup activities for all six programs in the initial year of validation.

Comments submitted in response to this comment request will be summarized and/or included in the request for OMB approval of the information collection request; they will also become a matter of public record.