Labor Exchange Performance Measurement System

AGENCY:

Employment and Training Administration, Labor.

ACTION:

Notice and request for comments.

SUMMARY:

This notice announces and requests comments on two components of a proposed labor exchange performance measurement system. A set of performance measures are proposed for measuring the performance of the public employment service in providing effective labor exchange services to employers and job seekers as part of the One-Stop delivery systems established by the States. A set of procedures also are proposed for State agencies and ETA to employ in establishing expected levels of performance to assure the delivery of high quality labor exchange services. These proposed labor exchange performance measures and procedures for establishing expected levels of performance will be key components of a comprehensive performance accountability system being developed for the employment service.

DATES:

Comments on these proposed labor exchange performance measures and procedures for establishing expected levels of performance must be received by the U.S. Department of Labor on or before October 13, 2000. Late-filed comments will be considered to the extent possible.

ADDRESSES:

Send comments to Timothy F. Sullivan, Chief, Division of United States Employment Service & ALMIS, U.S. Department of Labor, 200 Constitution Avenue NW., Room C-4514, Washington, DC 20210, Facsimile: 202-208-5844, E-mail: tsullivan@doleta.gov.

FOR FURTHER INFORMATION CONTACT:

Timothy F. Sullivan, 202-219-5257, E-mail: tsullivan@doleta.gov.

SUPPLEMENTARY INFORMATION:

I. Authority

Components of a labor exchange performance measurement system are proposed under the following authority:

A. Wagner-Peyser Act Sec. 3(a), 29 U.S.C. 49b(a)

The Secretary shall assist in coordinating the State public employment services throughout the country and in increasing their usefulness by developing and prescribing minimum standards of efficiency, assisting them in meeting problems peculiar to their localities, promoting uniformity in their administrative and statistical procedures, furnishing and publishing information as to opportunities for employment and other information of value in the operation of the system, and maintaining a system for clearing labor between the States.

B. Wagner-Peyser Act Sec. 3(c)(2), 29 U.S.C. 49b(c)

The Secretary shall—(2) assist in the development of continuous improvement models for such nationwide system that ensure private sector satisfaction with the system and meet the demands of job seekers relating to the system.

C. Wagner-Peyser Act Sec. 7(b), 29 U.S.C. 49f(b)

Ten percent of the sums allotted to each State pursuant to section 6 shall be reserved for use in accordance with this subsection by the Governor of each such State to provide—(1) performance incentives for public employment service offices and programs, consistent with performance standards established by the Secretary, taking into account direct or indirect placements (including those resulting from self-directed job search or group job search activities assisted by such offices or programs), wages on entered employment, retention, and other appropriate factors.

D. Wagner-Peyser Act Sec. 10(c), 29 U.S.C. 49i(c)

Each State receiving funds under this Act shall—

(1) make such reports concerning its operations and expenditures in such form and containing such information as shall be prescribed by the Secretary, and

(2) establish and maintain a management information system in accordance with guidelines established by the Secretary designed to facilitate the compilation and analysis of programmatic and financial data necessary for reporting, monitoring and evaluating purposes.

E. Wagner-Peyser Act Sec. 13(a), 29 U.S.C. 49l(a)

The Secretary is authorized to establish performance standards for activities under this Act which shall take into account the differences in priorities reflected in State plans.

F. Wagner-Peyser Act Sec. 15(e)(2)(I), 29 U.S.C. 49l-2(e)

(e) State responsibilities.—

(2) Duties.—In order to receive Federal financial assistance under this section, the State agency shall—

(I) utilize the quarterly records described in section 136(f)(2) of the Workforce Investment Act of 1998 (29 U.S.C. 2871(f)(2)) to assist the State and other States in measuring State progress on State performance measures.

II. Labor Exchange Performance Measurement System

The Employment and Training Administration (ETA) is establishing a comprehensive performance measurement system for the public labor exchange. This process consists of three tasks: (1) Developing a set of labor exchange performance measures, (2) developing procedures for establishing expected levels of performance that State agencies and ETA can use for assuring the delivery of high quality labor exchange services, and (3) revising the data collection procedures and reporting requirements applicable to the public labor exchange.

In February 2000, ETA convened a workgroup to begin the development of a comprehensive performance measurement system for the public labor exchange. This workgroup was formed in collaboration with the Interstate Conference of Employment Security Agencies (ICESA), and is comprised of representatives from fifteen State agencies, ICESA, the Veterans' Employment and Training Service (VETS), and the ETA regional and national offices. Representatives from America's Workforce Technology Solutions (AWTS) and Social Policy Research (SPR) Associates, Inc. provided technical support to the workgroup, but did not participate in the process of making final recommendations. The workgroup met once in the winter and once in the spring of 2000.

By the conclusion of the spring 2000 meeting, the workgroup had identified and defined a set of recommended performance measures for the public labor exchange. It also had developed recommended procedures for State agencies and ETA to employ in establishing expected levels of performance for the labor exchange and for assuring the delivery of high quality labor exchange services.

This notice announces and requests comments on a proposed set of performance measures to be used to measure the performance of the public employment service in providing effective labor exchange services to employers and job seekers as part of the One-Stop delivery systems. It also announces and requests comments on a proposed set of procedures for State agencies and ETA to use in establishing expected levels of performance.

A. Labor Exchange Performance Measures

Based on recommendations of the labor exchange performance measurement system workgroup, ETA proposes the following performance measures for the public labor exchange:

• Employer Customer Satisfaction
• Job Seeker Customer Satisfaction
• Employment Rate
• Entered Employment Rate
• Employment Retention Rate at Six Months

1. Identification and Selection of Performance Measures

During the first meeting, the workgroup followed a methodological approach in developing performance measures to recommend for the labor exchange. This consisted of describing the value that implementation of a performance measurement system would have for the public labor exchange and identifying concerns that might arise out of such a system; identifying the labor exchange outcomes that should be measured; and identifying qualities that are characteristic of good performance measures.

The workgroup identified a number of sound reasons for establishing a performance measurement system for the labor exchange. Key among these are that performance measures are essential for program managers to monitor the effectiveness of service delivery, and that performance information is of paramount importance to the Congress, State legislatures, the business community, and the general public as a means of assessing the value of the public labor exchange. The workgroup also noted that funds for the labor exchange are budgeted and appropriated, in part, based on such information and the message it conveys regarding the effectiveness of labor exchange service delivery. Additionally, performance reporting is required under the Government Performance and Results Act (GPRA) and it is important that a common system of measurement be developed so that performance reports of the various State agencies can be aggregated for reporting at the Federal level in support of GPRA requirements.

The workgroup recommended that the labor exchange performance measurement system contain procedures for setting expected levels of performance. These procedures should take into account the many differences between the States, such as labor market conditions and variations in how the States administer their programs under the Wagner-Peyser Act. The workgroup decided that demographic characteristics of the population served, such as age, race, ethnicity, and sex should not be considered when negotiating expected levels of performance. This is consistent with the requirement of providing universal access to job seekers. The workgroup also observed that timely and reliable data are essential to an effective performance measurement system. Finally, the group recognized that the labor exchange performance measurement system must not be developed in a vacuum. The performance measurement systems developed for related workforce development programs, such as the one currently being implemented for the Workforce Investment Act of 1998 (WIA) and those currently in use by the various States, should be taken into account in the development process.

The events that naturally result from the labor exchange carrying out its roles and responsibilities in providing services to its customers can be termed outcomes. There are two types of outcomes: end outcomes and intermediate outcomes. End outcomes represent the final objectives of the labor exchange, while intermediate outcomes represent accomplishments that lead to achieving the final objectives. The workgroup identified the following as key end outcomes for customers of the labor exchange:

• Job placements
• Entry into employment
• Shorter duration of unemployment
• Steady employment and income
• Customer satisfaction

The workgroup identified the following as key intermediate outcomes for primary customers:

• Quality job matches
• Knowledge of career and labor market information
• Qualified applicants
• Access to qualified applicants
• Access to job openings

The workgroup also identified characteristics of a good performance measurement system and used these as it considered and then recommended performance measures for the labor exchange. A good performance measurement system must be comprehensive and, to the extent possible, measure the primary end outcomes of the labor exchange. It must consist of a limited number of simple and easy to understand measures, and must yield timely information for management purposes. The performance measurement system must be developed such that it is objective and non-manipulative in order to avert unintended consequences. Data necessary for input to the performance measures must be readily available and collected at a reasonable cost. The system must take into account the uniqueness of the States and how each operates somewhat differently, while also allowing for uniform measurement across the States so that the aggregation of State performance information will be meaningful at the national level. The system also should allow for the measures to be applied at the sub-State level if so desired by the various States. Finally, the system should be consistent, to the extent possible, with related workforce development programs.

2. Proposed Labor Exchange Performance Measures

Five performance measures are proposed for the public labor exchange based on recommendations of the labor exchange performance measurement system workgroup. In its deliberations, the workgroup considered a wide range of options as potential measures of performance. The workgroup agreed by consensus to recommend two customer satisfaction measures, an employment rate measure and an entered employment rate measure. A substantial majority of the workgroup also supported the employment retention rate at six months measure. These recommended labor exchange performance measures are consistent with the aforementioned characteristics of good performance measures.

What follows are operational definitions of the proposed labor exchange employer and job seeker performance measures, and the rationale for recommending them:

a. Employer Measure

Employer Customer Satisfaction

It is proposed that the results of the American Customer Satisfaction Index (ACSI) which will be used to measure employer customer satisfaction under WIA also be used to measure employers' satisfaction with labor exchange services. Under this proposal, one survey will be conducted by the States to measure employer customer satisfaction with both WIA services and Wagner-Peyser Act labor exchange services. Specifications for the employer customer satisfaction survey are described in Training and Employment Guidance Letter (TEGL) 7-99, pp. 36-40, issued by ETA on March 3, 2000.

Adopting the WIA employer employer customer satisfaction measure for the labor exchange is proposed because the employer population from which the sample is drawn consists of employers who received a substantial service involving personal contact with One-Stop staff. Labor exchange staff provide a substantial portion of such services and the WIA employer satisfaction measure depicts, to a great extent, the satisfaction of employers with labor exchange services. Using a common measure to obtain information on employer customer satisfaction for both WIA and the labor exchange supports the integration of the labor exchange into the One-Stop delivery system. It also emphasizes the importance of providing high quality services to employers, a focus of the One-Stop delivery system. State Wagner-Peyser Act agencies will need to coordinate with State WIA agencies to obtain the results from the employer customer satisfaction survey and, if they so desire, to add additional questions to the survey instrument.

Using a uniform telephone methodology, each State must survey up to 1000 employers each year to obtain at least 500 completed surveys (except for States that serve less than 1000 employers, in which case, all employers served must be surveyed). The surveys should be conducted on a rolling basis throughout the program year. To obtain sufficient numbers, smaller States will need to survey on an ongoing basis. Employers should be contacted within 60 days of the completion of the service or 30-60 days after a job order has been listed where no referrals have been made. The employer customer satisfaction score is a weighted average of employer ratings on each of three questions regarding overall satisfaction, and is reported on a 0-100 point scale. The score is a weighted average, not a percentage.

What Questions Will Be Asked in the Survey?

The survey will be conducted by telephone. The proposed lead-in can be modified to suit the individual needs of the State and the program names recognizable for their population. The lead-in provided below is a model to be used as guidance. However, the numbered questions must remain as stated.

My name is _____ with XXXXX and I am conducting a survey for the XXXX XXXXX. I would like to speak to Ms./Mr. _____.

Are you the Ms./Mr. _____ who (describe the service received).

I would like to ask you some questions about your recent experience with _____. Our purpose is to learn from you how to improve programs and services offered to employers. The survey should take about XX minutes to complete.

(1) Utilizing a scale of 1 to 10 where “1” means “Very Dissatisfied” and “10” means “Very Satisfied” what is your overall satisfaction with the service(s) provided from _____?

(2) Considering all of the expectations you may have had about the services, to what extent have the services met your expectations? “1” now means “Falls Short of Your Expectations” and “10” means “Exceeds Your Expectations.”

(3) Now think of the ideal service(s) for people in your circumstances. How well do you think the service(s) you received compare with the ideal service(s)? “1” now means “Not Very Close to Ideal” and “10” now means “Very Close to the Ideal.”

Definition of Terms

Sample. A group of cases selected from a population by a random process where everyone has an equal probability of being selected.

Response rate. The percentage of people who have valid contact information who are contacted and respond to all the questions on the survey.

DK. Don't Know.

REF. Refused to answer.

The Calculation

The overall score for the American Customer Satisfaction Index (ACSI) Measure is accomplished by calculating the weighted average of the raw scores for each of the customer satisfaction questions given by each respondent. The weighted average score is then transformed to an index reported on a 0-100 scale. The aggregate index score is simply the weighted average of each case's index score.

The ACSI trademark is proprietary property of the University of Michigan. The Department has established a license agreement with the University of Michigan that will allow States the use of the ACSI for a Statewide sample of employers (and WIA participants). States that want to use the ACSI for measuring customer satisfaction for each local area will have to establish an independent contract with the University of Michigan. States may also contract with CFI Group for additional assistance in measuring, analyzing, and understanding ACSI data. Procedures for contracting with the CFI Group are being developed and will be issued when finalized.

Notes:

CFI Group will provide the actual weights given for (W1), (W2), and (W3) below. (It has yet to be determined how the weights will be distributed to the States). In calculating respondent level index scores, round to two decimal points.

When calculating the average index score, round to the nearest whole number. For any case, the general formula for calculating the index score is given as:

Index Score = {[(Q1)(W1) + (Q2)(W2) + (Q3)(W3)] −1} × 11.111 where:

Q1 = raw score on question #1

Q2 = raw score on question #2

Q3 = raw score for question #3

W1 = weight for question #1

W2 = weight for question #2

W3 = weight for question #3

Example:

If the respondent answers were 5, 8, and 9 respectively for each of the three customer satisfaction questions, and the weights for each of the three questions were .4, .2, and .4* respectively the calculation for the respondent's index score would be as follows:

{[(5)(.4) + (8)(.2) + (9)(.4)]—1} × 11.111 = {[7.2]−1} × 11.111 = 68.89

If two more respondents whose raw scores on the three questions were 6, 10, and 6 and 9, 6, and 7 respectively, using the same weights listed above, those two respondent's index scores would be: 64.44 and 73.33. To calculate the aggregate index score, simply average the individual respondent's index scores and round to the nearest whole number as follows:

(68.89 + 64.44 + 73.33) /3 = 69

* These weights are examples only, CFI group will provide the actual weights.

The workgroup considered other options for employer measures including a job order or job opening fill rate, an employer market penetration rate, and a measure of employer use of labor exchange services. The workgroup was unable to identify definitions for these types of employer performance measures that were consistent with the identified characteristics of a good performance measure. While employer customer satisfaction is the only proposed performance measure for employers, ETA will research the development of possible additional employer performance measures for future consideration.

b. Job Seeker Measures

For job seekers, measures will be used to account for performance of the public labor exchange with respect to all applicants who register, subject to the criteria contained in the definition of each measure. The universe of job seekers will consist of an unduplicated count of job seekers who register, or who renew or reactivate their registration, during the applicable program year.

Including all registrants in the universe is proposed because it maintains consistency with the concept of providing universal access to labor exchange services. The measurement system is designed to capture the employment outcomes of all those who request access to labor exchange services through registration. This allows for measuring the outcomes of all labor exchange services that are made available to job-seeking applicants. It also maintains consistency with the criteria described in section 7(b) of the Wagner-Peyser Act for performance standards to be established by the Secretary that take into account entry into employment resulting from either self-directed job search activities or staff-assisted job search activities.

The job seeker customer satisfaction measure will rely on telephone survey data for outcome information, while the employment rate, entered employment rate, and employment retention rate at six months outcome measures will rely on unemployment insurance (UI) wage records as a primary data source. State agencies also will retain the option to use data obtained from administrative follow-up, the method of data collection currently used by many State agencies, to supplement the wage record information. The use of wage record information will allow for more reliable and comprehensive collection of employment outcome data at a lower cost than methods currently used by many State agencies. The advent of the Wage Record Interchange System (WRIS) will provide State agencies with an additional resource for obtaining wage records from other State agencies to use in tracking the outcomes of job seekers who have migrated across State lines.

The measurement period will consist of the four quarters comprising a program year. Performance outcomes will be attributable to the program year in which the outcome occurs, whether the job seeker registered with the labor exchange in that program year or the previous program year. This will require reporting in the numerator, the total number of job seekers who achieve the expected outcomes during the appropriate measurement quarters, and reporting in the denominator, the total number of registered job seekers who could have achieved the expected outcomes (i.e. employment or retention) during the measurement quarters.

The aforementioned criteria apply, as appropriate, to the following job seeker labor exchange performance measures:

Job Seeker Customer Satisfaction

ETA proposes to implement a job seeker customer satisfaction measure that mirrors the WIA participant customer satisfaction survey and uses the ACSI methodology. Specifications for the labor exchange job seeker customer satisfaction survey are as follows:

The job seeker customer satisfaction score is a weighted average of participant ratings on each of three questions regarding overall satisfaction, and is reported on a 0-100 point scale. The score is a weighted average, not a percentage.

Who Will Be Surveyed?

All labor exchange applicants who register with the labor exchange are eligible to be chosen for inclusion in the random sample.

How Many Must Be Surveyed?

A sample of 250 will be taken from the job seeker applicants who register with the labor exchange in each quarter. Five hundred completed job seeker surveys must be obtained each year for calculation of the measure. A completed job seeker survey is defined as a survey in which all three questions regarding overall satisfaction have been answered. The response rate from the sample with valid contact information must be a minimum of 50 percent. The standard of 500 from a sample of the whole population of customers provides accuracy such that there is only a 5 in 100 chance that the results would vary by more than ±5 points from the score obtained from surveying the whole population.

How Should the Survey Be Conducted?

The responses are obtained using a uniform telephone methodology. The rationale for only using telephone surveys include: the comparability of the measure for assessing performance levels is most reliably obtained with a telephone survey; telephone surveys are easily and reliably administered; and defining procedures for mailed surveys is more difficult than defining procedures for telephone surveys. Estimates of the cost of telephone surveys nationwide average $15 per completed survey. Since States will need to complete 500 job seeker surveys, the cost is estimated at about $7,500 per State per year.

When Should the Survey Be Conducted?

The surveys should be conducted on a rolling basis within the timeframe indicated below for job seekers. To obtain sufficient numbers, smaller States will need to survey on an ongoing basis. Job seekers should be contacted within 60-90 days of the date of registration, or renewal or reactivation.

What Questions Will Be Asked in the Survey?

The survey will be conducted by telephone and the following lead-in will be used at the beginning of the interview. The lead-in can be modified to suit the individual needs of the State and the names for program services recognizable for their population. The lead-in provided below is a model to be used as guidance. The numbered questions must remain as stated.

My name is _____ with XXXXX and I am conducting a survey for the XXXX XXXXX. I would like to speak to Ms./Mr. _____.

Are you the Ms./Mr. _____ who was looking for a job a few months ago?

I would like to ask you some questions about your recent experience looking for a job. Our purpose is to learn from you how to improve programs and services offered to people in XXX. The survey should take about XX minutes to complete. First I am going to read a list of services you may have received. Indicate as I read them those you recall receiving during the period in which you were seeking employment and/or training at the XX center.

• A thorough assessment of your needs
• Assistance about finding a job
• Assistance to develop an individual employment plan
• Assistance to decide about the best training to take
• Assistance from someone to support you during your job search or training
• Use of electronic job search tools (e.g. America's Job Bank, Internet tools)

(States may modify the list of services as appropriate for the labor exchange)

Did you get any other help or services that I have not mentioned? (specify)

(1) Utilizing a scale of 1 to 10 where “1” means “Very Dissatisfied” and “10” means “Very Satisfied” what is your overall satisfaction with the services provided from _____?

(2) Considering all of the expectations you may have had about the services, to what extent have the services met your expectations? “1” now means “Falls Short of Your Expectations” and “10” means “Exceeds Your Expectations.”

(3) Now think of the ideal program for people in your circumstances. How well do you think the services you received compare with the ideal set of services? “1” now means “Not very close to the Ideal” and “10” means “Very close to the Ideal.”

The same ASCI calculation is used for the job seeker customer satisfaction measure as was described above for the employer customer satisfaction measure. The Department is currently engaging in discussions with the University of Michigan to make arrangements to use the ACSI for the labor exchange job seeker customer satisfaction measure. As the population of job seekers registering with the labor exchange is different from the population of participants exiting WIA services, a separate survey is required to adequately gauge the satisfaction of job seeker customers.

State agencies will have flexibility in modifying the lead-in to the questionnaire to suit their particular needs and also may add additional questions, as long as the three questions presented above remain the same and are the initial three questions in the survey. Since there likely will be a number of individuals who both register with the labor exchange and who exit WIA, State agencies are requested to coordinate these survey efforts to eliminate the possibility of individuals being surveyed twice.

Employment Rate

The proposed employment rate performance measure is defined as:

All Wagner-Peyser Act labor exchange applicants who registered in quarter Q₀ and who earned wages in quarter Q₁ or Q₂ after registration, divided by the number of Wagner-Peyser Act labor exchange applicants who registered in quarter Q₀.

This performance measure reports on the employment outcomes that may be attributable to the labor exchange services made available to all applicants. Including all applicants in the measurement population supports the concept of providing universal access to labor exchange services by establishing accountability for the employment outcomes of all job seekers provided access to labor exchange services. This includes new entrants to the labor market, job seekers who are not employed, and incumbent workers.

This performance measure uses a period of two quarters to look for entry into employment because two quarters can be considered an appropriate length of time in which to expect a positive employment outcome for those provided access to labor exchange services. This period of time is also similar to the 26-week maximum period of eligibility for unemployment insurance (UI) benefits, which is deemed to be an appropriate period of time for UI claimants, a key labor exchange customer group, to obtain suitable employment.

In addition to entry into employment with a new employer, this measure also recognizes as a positive outcome, the job seeker who is employed at the time of registration with the labor exchange, and who during the next two quarters remains employed with the same employer. In such instances, if this job seeker registered with the labor exchange, he or she had some inclination to continue being employed, either with the same or a different employer. For the job seeker who remained with the same employer, available labor exchange services, such as job counseling and labor market information, may have enabled that job seeker to assess his or her employment situation by surveying the labor market (i.e. available jobs, availability of transportation, wage rates, training requirements, etc.). That job seeker's continued employment, with the same employer, may in part be attributable to a rational decision to maintain his or her employment situation based on such labor exchange services.

The workgroup considered including only as a positive outcome for this measure, entry into employment by those not employed and entry into employment with a different employer by those currently employed. While such a definition was a strong candidate for a labor exchange performance measure, members of the workgroup deemed the difficulty of distinguishing employment with one employer from that with another as too burdensome for performance measurement purposes, and thus this option was not recommended to ETA. Such a performance measure would have required each State agency to conduct a match to determine whether the Federal Employer Identification Number (FEIN) on a job seeker's wage records was with a different employer in the measurement quarters (Q₁ or Q₂) than it was in the registration quarter (Q₀). Several members of the workgroup expressed concern that this was not feasible at a reasonable cost. Others pointed out that it would also exclude job seekers who obtained a new position with the same employer.

Finally, a considerable number of workgroup members suggested limiting the measurement period to a single quarter following registration, rather than two quarters as proposed for the job seeker measures. While this would enhance consistency with the WIA core measures and might support the more timely delivery of services to job seekers, the workgroup ultimately decided to use two quarters, acknowledging that the benefits of allowing two quarters to record employment outcomes outweighed these other concerns.

Entered Employment Rate

The proposed entered employment rate performance measure is defined as:

Of those Wagner-Peyser Act labor exchange applicants who were not employed upon registration in quarter Q₀: The number who earned wages in quarter Q₁ or Q₂ after registration, divided by the number who registered in quarter Q₀.

Within the universe of all applicants, this performance indicator measures the employment outcomes of the job seeker population that is not employed at the time of registration. The rationale for using a time period of two quarters for this performance measure is the same as that described above for the employment rate measure. The entered employment rate measure is proposed out of recognition that it is important to obtain employment outcome information specifically on job seekers who are not employed when registering with the labor exchange.

It is acknowledged that there are some subtle distinctions between the entered employment rate measure for labor exchange job seekers and the entered employment rate measure for WIA participants, (i.e., using registration as the trigger for the measurement period for the labor exchange, rather than exit, as is done for WIA; and using two quarters as the measurement period for the labor exchange, rather than one, as is the case for WIA). The entered employment rate measure for the labor exchange is recommended, as defined above, since the nature of the labor exchange services provided to job seekers are different than the services provided under WIA. Many WIA participants need core, intensive, and training services in order to become job ready, while the preponderance job seekers who avail themselves of labor exchange services are at the stage of actively seeking work.

The workgroup also considered including in the definition of the entered employment rate measure, the outcomes of currently employed job seekers who enter into new employment with a different employer. For the same reasons as indicated for the employment rate measure, the workgroup did not recommend defining the entered employment rate measure to include this group of job seekers.

Employment Retention Rate at Six Months

The proposed employment retention rate at six months performance measure is defined as:

Of those Wagner-Peyser Act labor exchange applicants who registered in quarter Q₀ and who earned wages in quarter Q₁ or Q₂ after registration: the number who also earned wages in the second quarter following the quarter in which earned wages were first recorded, divided by the number who earned wages in quarter Q₁ or Q₂.

This performance measure recognizes as a positive employment outcome employment in any job two quarters following the employment that is recorded in an initial job during quarter Q₁ or Q₂ after registration. In recommending this performance measure, the workgroup acknowledged that while many job seekers register with the labor exchange to search for and find a job that results in lasting employment, others may use labor exchange services to assist them in acquiring temporary employment or a series of short-term jobs. Services such as job search workshops, resume assistance, job finding clubs, job counseling, and even self-services are activities that have a lasting effect on job seekers and can contribute to a job seeker retaining employment in his or her current job, or entering and retaining employment in a subsequent job.

The labor exchange employment retention rate measure provides a degree of consistency with the WIA performance measurement system, which also includes an employment retention measure. Both rely on the assumption that the vast majority of individuals seeking the services provided by the respective programs possess an inherent desire to maintain employment during the Short- and medium-term. However, a small number of seasonal workers, such as students and some farmworkers, may desire to work only sporadically throughout the year. Recognizing that such workers are only a small fraction of all job seekers and that it is important for the labor exchange to be able to monitor the employment outcomes of job seekers beyond their initial entry into employment, the employment retention rate at six months measure is recommended as one that provides valuable information on the medium-term employment outcomes of the job seekers who register with the labor exchange.

B. Procedures for Establishing Expected Levels of Performance

In accordance with the recommendation from the workgroup, the WIA Title I framework will be used for negotiating and setting expected performance levels for labor exchange services. This means that States will develop baseline data for the measures, analyze the baseline data, and propose performance levels for each measure based on that analysis. After providing the required information to the appropriate ETA Regional Office, States will negotiate with the region to obtain mutually agreed upon expected levels of performance. In developing baseline data, States should use two years of data if possible, but not less than one year in determining trends for performance and factors which may influence performance. For the customer satisfaction measures, States should look at experience thus far under WIA and any other survey instruments they have previously used. In establishing expected performance levels for each measure, factors beyond the control of the State are also to be considered. The specific steps for setting expected levels are as follows:

Baseline Performance

Baselines for each of the measures will be developed by each State and will be a key factor used to determine the expected level of performance that is negotiated with ETA. Baselines are intended to give an indication of the past outcomes of a performance measure. For performance negotiations to be data-driven and reality based, the development of baselines is a critical aspect of the negotiation process.

State Expected Levels of Performance

As part of the 5 year State Plan submitted to the Department of Labor, each State will propose expected levels of performance for the next three program years (PY 2001-2003). States should be prepared to provide support for their proposed levels by providing information on how baseline performance levels were developed and providing other information they believe may affect performance. States will include in their plan expected levels of performance levels for each measure. In addition, States will provide the baseline performance data and a description of any other factors, such as economic conditions, that contributed to the establishment of the expected performance levels. States also should include the methodology for developing the baseline data, a description of data sources and appropriate factors used to project expected levels of performance.

In recommending factors to be considered, the workgroup explicitly excluded applicant characteristics and types of services provided. The labor exchange is viewed as an agency offering universal access to all job seekers, with basically the same set of services provided across all States. Allowing for differences in applicant characteristics might have the unintended consequence of favoring service provision to some applicant groups over others. With respect service mix, this is not an appropriate factor for adjusting expected levels of performance, because this is within the full control of the State agency.

Examples of possible factors to consider in negotiating expected levels of performance are: economic conditions such as the unemployment rate, the rate of job creation/loss, new business start-ups; community factors such as availability of transportation and daycare; pursuit of new or enhanced employer partnerships; other factors such as State legislation or policies which might impact performance; and natural disasters. This list is not intended to be prescriptive or exhaustive, but to suggest the kinds of information that might be considered in the negotiation process.

Negotiation of Expected Levels of Performance

The Regional Office will review the information contained in the State plan and will compare the expected performance levels with the national averages, baseline information from other States, and the negotiated levels of performance established for other States, taking into account factors including differences in economic conditions and other factors as discussed above. In addition, the Regional Office will analyze the quality of the data presented by States, including the relevance of the data, the source of the data, the time period from which the data were drawn, and if the data are part of a trend or anomalous. Established GPRA Annual Performance Plan goals for relevant measures will also be an important part of regional review and negotiation of performance levels. When the Regional Office finalizes its analysis, there will be negotiations with the State to obtain mutually agreed upon expected levels of performance.

Similar to WIA, provision will be made for renegotiation of performance levels if circumstances arise that result in a significant change in the factors used to establish the original levels. It is understood that either a State or the regional office may elect to renegotiate performance as new information becomes available. Factors which will be considered for making changes include those discussed above.

Expected performance levels may, depending on the factors to be considered, be renegotiated for any one or all three years of the performance period. States initiating the renegotiation will prepare a modification to the approved State plan and submit it to the regional office. The negotiation process described above will then be followed. In cases where the change is initiated by the region, States will be asked to prepare an amendment to the approved plan. Once the amendment is submitted to the regional office, the established negotiation process will then be followed.

C. Rules for Application

Actual performance for each program year will be compared to negotiated performance levels. For a State to be designated as “exemplary,” expected levels for all measures must be achieved or exceeded for all measures. Actions that may be taken in the case of “exemplary” performance by States include: formal recognition by the Department of Labor through letters to Governors and publication of results in an annual report; recognizing and publicizing practices that foster good performance through publication on Internet websites; and the election by States to use their Wagner-Peyser Act 7(b) funds to provide performance incentives for public employment service offices and programs. States and regional offices should analyze performance information on an ongoing basis and, where performance is not achieving expected levels, work together to develop corrective action plans—including the provision of any training or technical assistance that may be required.