Confidentiality in Federal Alternative Dispute Resolution Programs; Evaluation of Federal Alternative Dispute Resolution Programs

AGENCY:

Department of Justice/Federal Alternative Dispute Resolution Council.

ACTION:

Notice.

SUMMARY:

This notice solicits public comment on two documents designed to assist Federal agencies in developing alternative dispute resolution (ADR) programs: “Confidentiality in Federal Alternative Dispute Resolution Programs” and “Evaluation of Federal Alternative Dispute Resolution Programs.” These documents were created by the Federal ADR Steering Committee, a group of subject matter experts from federal agencies with active ADR programs. They were approved for publication in draft form by the Federal ADR Council, a group of high level government officials chaired by the Attorney General. The first document contains detailed guidance on the nature and limits of confidentiality in Federal ADR programs and also includes a statement on these issues for Federal neutrals to use in ADR proceedings. The second document contains detailed recommendations for agencies to follow when evaluating their ADR programs.

All interested individuals or organizations are invited to submit comments on these documents for the consideration of the Federal ADR Council before they are published in final form at the end of this year.

DATES:

All comments must be postmarked by November 1, 2000, in order to receive consideration.

ADDRESSES:

Address all comments to Jeffrey M. Senger, Deputy Senior Counsel for Dispute Resolution, United States Department of Justice, 950 Pennsylvania Ave. NW, Room 4328, Washington, D.C., 20530

SUPPLEMENTARY INFORMATION:

In recent years, the government and the private sector increasingly have been using techniques known as alternative dispute resolution (ADR). Our experience has shown that ADR can resolve disputes in a manner that is quicker, cheaper, and less adversarial than traditional processes such as litigation. In ADR, parties meet with each other directly, under the guidance of a neutral professional who is trained and experienced in handling disputes. They talk about the problems that led to the complaint and the resolution that will work best for them in the future. While litigation often silences the parties and severely restricts their control over the outcome of their own dispute, ADR allows them instead to work collaboratively to find creative, effective solutions that are agreeable to all sides.

The Administrative Dispute Resolution Act of 1996 (ADRA), 5 U.S.C. 571-584, requires each Federal agency to promote the use of ADR and calls for the establishment of an interagency committee to assist agencies in the use of ADR. Pursuant to this Act, a Presidential Memorandum dated May 1, 1998, created the Interagency ADR Working Group, chaired by the Attorney General, to “facilitate, encourage, and provide coordination” for Federal agencies. In the Memorandum, the President charged the Working Group with assisting agencies with training in “how to use alternative means of dispute resolution” and evaluation “to ascertain the benefits of alternative means of dispute resolution.” The following two documents are designed to serve these goals.

The first document describes the nature and limits of confidentiality in Federal ADR proceedings. Confidentiality is vital for the success of ADR for several reasons. Parties must be free to engage in candid, informal discussions of their interests in order to reach the best possible settlement of their claims. Guarantees of confidentiality permit parties to speak openly, without fear their statements will be used against them later. Confidentiality also facilitates ADR by encouraging parties to avoid the posturing that often occurs when proceedings are on the record. Further, confidentiality gives parties the ability to trust the mediator because they are assured he or she will not later take sides and talk publicly in favor of one party or the other. At the same time, members of the public have a general right to know what happens in government proceedings and do not want ADR to be used to shield improper activity that involves public business. The ADRA is designed to strike the appropriate balance between the public interest in access to government decision-making and the necessity for certain guarantees of confidentiality in ADR in order for the process to be effective.

Understandably, there has been a great deal of interest in understanding what statements made in the context of a Federal ADR proceeding are confidential and what statements are not. This document is designed to give a detailed explanation of the reasonable expectations of confidentiality for parties who participate in ADR involving the government. The first section of the report reprints the confidentiality provisions of the ADRA. Next, the report contains a section-by-section analysis of these confidentiality provisions. Then the report sets forth, in question-and-answer format, an expanded analysis of the issues likely to arise in practice. Finally, the report presents a model confidentiality statement suitable for use by neutrals in Federal ADR proceedings.

The second document contains detailed guidance for agencies to use when conducting evaluations of their ADR programs. Proponents of ADR have described many benefits from its use, including savings of time and money, increased party satisfaction with the process and its outcome, increased settlement rates, and improved relationships. In order to ensure the growth of ADR programs, these benefits must be rigorously documented and communicated to the public. If evaluations determine problems with ADR programs, these must be remedied. Evaluation is a vital part of any ADR program, and it is consistent with the obligations of all Federal agencies under the Government Performance and Results Act (Pub. L. 103-62).

The first part of this document is a two-page description of general evaluation recommendations for Federal ADR programs. It sets forth specific data that agencies should capture and gives a brief introduction to other important concepts, such as validity, reliability, and presentation of data. The remainder of the report is a twenty-page detailed description of evaluation, including planning and design, methodology, and communicating results. The report concludes with a bibliography of additional resources in this area.

The Federal ADR Council encourages all interested parties to submit comments on these documents. The Council will consider all comments in connection with its review of the final versions of these documents at the end of 2000.

Nothing in these guidance documents shall be construed to create any right or benefit, substantive or procedural, enforceable at law or in equity, by a party against the United States, its agencies, its officers or any other person.

The Federal ADR Council

Chair: Janet Reno, Attorney General, Department of Justice

Vice Chair: Erica Cooper, Deputy General Counsel, Federal Deposit Insurance Corporation

Members: Leigh A. Bradley, General Counsel, Department of Veterans Affairs; Meyer Eisenberg, Deputy General Counsel, Securities and Exchange Commission; Mary Anne Gibbons, General Counsel, U.S. Postal Service; Gary S. Guzy, General Counsel, Environmental Protection Agency; Jeh C. Johnson, General Counsel, Department of the Air Force; Harold Kwalwasser, Deputy General Counsel, Department of Defense; Rosalind Knapp, Acting General Counsel, Department of Transportation; Anthony N. Palladino, Director, Office of Dispute Resolution, Federal Aviation Administration, Department of Transportation; Janet S. Potts, Counsel to the Secretary, Department of Agriculture; Harriett S. Rabb, General Counsel, Department of Health and Human Services; Henry L. Solano, Solicitor, Department of Labor; John Sparks, Principal Deputy General Counsel, Department of the Navy; Peter R. Steenland, Jr., Senior Counsel for Dispute Resolution, U.S. Department of Justice; Mary Ann Sullivan, General Counsel, Department of Energy; Robert Ward, Dispute Resolution Specialist, Environmental Protection Agency.

Report on the Reasonable Expectations of Confidentiality Under the Administrative Dispute Resolution Act of 1996

Table of Contents

I. Introduction

II. Administrative Dispute Resolution Act

III. Section-By-Section Analysis of Confidentiality Provisions

IV. Questions & Answers on Confidentiality under the Administrative Dispute Resolution Act (ADR Act)

V. Model Confidentiality Statement for Use by Neutrals

I. Introduction

The Administrative Dispute Resolution Act of 1996 (“ADR Act”) contains provisions that affect the confidentiality of administrative ADR proceedings. Neutrals and participants in federal dispute resolution proceedings need to have an accurate understanding of these provisions. The Federal ADR Council directed the Interagency ADR Working Group Steering Committee to review the ADR Act confidentiality provisions and provide the Council with a report outlining reasonable expectations of confidentiality for parties in federal dispute resolution. This report, the product of that effort, describes the ADR Act confidentiality provisions principally located at 5 U.S.C. Section 574.

The report has four sections: (1) A reprint of the confidentiality provisions of the ADR Act; (2) a section-by-section analysis of the confidentiality provisions; (3) a set of questions and answers designed to expand upon the analysis and address issues likely to arise in practice; and (4) a model confidentiality statement suitable for use by neutrals in federal ADR proceedings.

During preparation of this report, several issues emerged regarding implementation of the ADR Act that are not fully addressed in this report. These issues are important to the practice of federal ADR and would benefit from further investigation and study. As federal sector experience with ADR evolves, some issues addressed in this report will be refined and new issues are likely to arise. It is also important to note that the ADR Act is not the only means of maintaining confidentiality and other laws, regulations, and agency policies may impact confidentiality. A complete analysis of all such authorities is beyond the scope of this report.

II. Administrative Dispute Resolution Act

Definitions (5 U.S.C. 571)

For the purposes of this subchapter, the term—

(1) “agency” has the same meaning as in section 551(1) of this title;

(2) “administrative program” includes a Federal function which involves protection of the public interest and the determination of rights, privileges, and obligations of private persons through rule making, adjudication, licensing, or investigation, as those terms are used in subchapter II of this chapter;

(3) “alternative means of dispute resolution” means any procedure that is used to resolve issues in controversy, including, but not limited to, conciliation, facilitation, mediation, factfinding, minitrials, arbitration, and use of ombuds, or any combination thereof;

(4) “award” means any decision by an arbitrator resolving the issues in controversy;

(5) “dispute resolution communication” means any oral or written communication prepared for the purposes of a dispute resolution proceeding, including any memoranda, notes or work product of the neutral, parties or nonparty participant; except that a written agreement to enter into a dispute resolution proceeding, or final written agreement or arbitral award reached as a result of a dispute resolution proceeding, is not a dispute resolution communication;

(6) “dispute resolution proceeding” means any process in which an alternative means of dispute resolution is used to resolve an issue in controversy in which a neutral is appointed and specified parties participate;

(7) “in confidence” means, with respect to information, that the information is provided—

(A) with the expressed intent of the source that it not be disclosed; or

(B) under circumstances that would create the reasonable expectation on behalf of the source that the information will not be disclosed;

(8) “issue in controversy” means an issue which is material to a decision concerning an administrative program of an agency, and with which there is disagreement—

(A) between an agency and persons who would be substantially affected by the decision; or

(B) between persons who would be substantially affected by the decision;

(9) “neutral” means an individual who, with respect to an issue in controversy, functions specifically to aid the parties in resolving the controversy;

(10) “party” means—

(A) for a proceeding with named parties, the same as in section 551(3) of this title; and

(B) for a proceeding without named parties, a person who will be significantly affected by the decision in the proceeding and who participates in the proceeding;

(11) “person” has the same meaning as in section 551(2) of this title; and

(12) “roster” means a list of persons qualified to provide services as neutrals.

Confidentiality (5 U.S.C.574)

(a) Except as provided in subsections (d) and (e), a neutral in a dispute resolution proceeding shall not voluntarily disclose or through discovery or compulsory process be required to disclose any dispute resolution communication or any communication provided in confidence to the neutral, unless—

(1) all parties to the dispute resolution proceeding and the neutral consent in writing, and, if the dispute resolution communication was provided by a nonparty participant, that participant also consents in writing;

(2) the dispute resolution communication has already been made public;

(3) the dispute resolution communication is required by statute to be made public, but a neutral should make such communication public only if no other person is reasonably available to disclose the communication; or

(4) a court determines that such testimony or disclosure is necessary to—

(A) prevent a manifest injustice;

(B) help establish a violation of law; or

(C) prevent harm to the public health or safety, of sufficient magnitude in the particular case to outweigh the integrity of dispute resolution proceedings in general by reducing the confidence of parties in future cases that their communications will remain confidential.

(b) A party to a dispute resolution proceeding shall not voluntarily disclose or through discovery or compulsory process be required to disclose any dispute resolution communication, unless—

(1) the communication was prepared by the party seeking disclosure;

(2) all parties to the dispute resolution proceeding consent in writing;

(3) the dispute resolution communication has already been made public;

(4) the dispute resolution communication is required by statute to be made public;

(5) a court determines that such testimony or disclosure is necessary to—

(A) prevent a manifest injustice;

(B) help establish a violation of law; or

(C) prevent harm to the public health and safety, of sufficient magnitude in the particular case to outweigh the integrity of dispute resolution proceedings in general by reducing the confidence of parties in future cases that their communications will remain confidential;

(6) the dispute resolution communication is relevant to determining the existence or meaning of an agreement or award that resulted from the dispute resolution proceeding or to the enforcement of such an agreement or award; or

(7) except for dispute resolution communications generated by the neutral, the dispute resolution communication was provided to or was available to all parties to the dispute resolution proceeding.

(c) Any dispute resolution communication that is disclosed in violation of subsection (a) or (b), shall not be admissible in any proceeding relating to the issues in controversy with respect to which the communication was made.

(d)(1) The parties may agree to alternative confidential procedures for disclosures by a neutral. Upon such agreement the parties shall inform the neutral before the commencement of the dispute resolution proceeding of any modifications to the provisions of subsection (a) that will govern the confidentiality of the dispute resolution proceeding. If the parties do not so inform the neutral, subsection (a) shall apply.

(2) To qualify for the exemption established under subsection (j), an alternative confidential procedure under this subsection may not provide for less disclosure than the confidential procedures otherwise provided under this section.

(e) If a demand for disclosure, by way of discovery request or other legal process, is made upon a neutral regarding a dispute resolution communication, the neutral shall make reasonable efforts to notify the parties and any affected nonparty participants of the demand. Any party or affected nonparty participant who receives such notice and within 15 calendar days does not offer to defend a refusal of the neutral to disclose the requested information shall have waived any objection to such disclosure.

(f) Nothing in this section shall prevent the discovery or admissibility of any evidence that is otherwise discoverable, merely because the evidence was presented in the course of a dispute resolution proceeding.

(g) Subsections (a) and (b) shall have no effect on the information and data that are necessary to document an agreement reached or order issued pursuant to a dispute resolution proceeding.

(h) Subsections (a) and (b) shall not prevent the gathering of information for research or educational purposes, in cooperation with other agencies, governmental entities, or dispute resolution programs, so long as the parties and the specific issues in controversy are not identifiable.

(i) Subsections (a) and (b) shall not prevent use of a dispute resolution communication to resolve a dispute between the neutral in a dispute resolution proceeding and a party to or participant in such proceeding, so long as such dispute resolution communication is disclosed only to the extent necessary to resolve such dispute.

(j) A dispute resolution communication which is between a neutral and a party and which may not be disclosed under this section shall also be exempt from disclosure under section 552(b)(3).

III. Section-by-Section Analysis of Confidentiality Provisions (5 U.S.C. 574)

Section 574(a)

In general, a neutral in a dispute resolution proceeding is prohibited from disclosing any dispute resolution communication or any communication provided to him or her in confidence. Unless the communication falls within one of the exceptions listed below, the neutral cannot voluntarily disclose a communication and cannot be forced to disclose a communication through a discovery request or by any other compulsory process.

The exceptions to this general rule are found in subsections 574(a)(1)-(4), 574(d) and 574(e).

Section 574(a)(1)

A neutral may disclose a communication if all parties and the neutral agree in writing to the disclosure. If a nonparty provided the communication, then the nonparty must also agree in writing to the disclosure.

Section 574(a)(2)

A neutral may disclose a communication if the communication has already been made public.

Section 574(a)(3)

A neutral may disclose a communication if there is a statute which requires it to be made public. However, the neutral should not disclose the communication unless there is no other person available to make the disclosure.

Section 574(a)(4)

A neutral may disclose a communication if a court finds that the neutral's testimony, or the disclosure, is necessary to:

A. prevent a manifest injustice;

B. help establish a violation of law; or

C. prevent harm to the public health and safety.

In order to require disclosure, a court must determine that the need for disclosure is of sufficient magnitude to outweigh the detrimental impact on the integrity of dispute resolution proceedings in general. The need for the information must be so great that it outweighs a loss of confidence among other potential parties that their dispute resolution communications will remain confidential in future proceedings.

Section 574(b)

Unless the communication falls within one of the exceptions listed below, the party cannot voluntarily disclose a communication and cannot be forced to disclose a communication through a discovery request or by any other compulsory process.

Section 574(b)(1)

The party who makes a statement or communication is free to disclose it.

Section 574(b)(2)

A party may disclose a communication if all the parties agree in writing to the disclosure.

Section 574(b)(3)

A party may disclose a communication if the communication has already been made public.

Section 574(b)(4)

A party may disclose a communication if there is a statute which requires it to be made public.

Section 574(b)(5)

A party may disclose a communication if a court finds that the party's testimony, or the disclosure, is necessary to:

A. prevent a manifest injustice;

B. help establish a violation of law; or

C. prevent harm to the public health and safety.

In order to require disclosure, a court must determine that the need for disclosure is of sufficient magnitude to outweigh the detrimental impact on the integrity of dispute resolution proceedings in general. The need for the information must be so great that it outweighs a loss of confidence among other potential parties that their dispute resolution communications will remain confidential in future proceedings.

(1) Parties may use dispute resolution communications to show that a settlement agreement was in fact reached or to show what the terms of this agreement mean.

(2) Parties may also use communications in connection with later issues regarding enforcing the agreement.

(3) Communications may only be revealed to the extent that they meet the above purposes.

(1) There is no confidentiality protection for parties' dispute resolution communications that were available to everyone in the proceeding. For example, in a joint mediation session with all parties present, statements made and documents provided by parties are not confidential.

(2) Communications coming from the neutral are confidential. For example, early neutral evaluations or settlement proposals from the neutral are protected.

(3) A party may not use this provision to gain protection for a communication by providing it to the neutral who then provides it to the other party.

Section 574(c)

No one may use any dispute resolution communication in a related proceeding, if that communication was disclosed in violation of Section 574 (a) and (b).

(1) Parties may agree to alternative confidentiality procedures to limit disclosure by a neutral.

(2) Parties must inform the neutral of the alternative procedures before the dispute resolution proceeding begins.

(3) If parties do not inform the neutral of the alternative procedures, the procedures outlined in Section 574(a) will apply.

(1) Dispute resolution communications covered by alternative confidentiality procedures may be protected from disclosure under FOIA.

(2) To qualify for this protection, the alternative procedures must provide for as much, or more, disclosure than the procedures provided in Section 574.

(3) Dispute resolution communications covered by alternative confidentiality procedures do not qualify for protection from disclosure under FOIA if they provide for less disclosure than those outlined in Section 574.

(1) A neutral who receives a demand for disclosure, in the form of a discovery request or other legal process, must make reasonable efforts to notify the parties and any affected non-party participants of the demand.

(2) Parties and non-party participants who receive a notice of a demand for disclosure from a neutral:

a. must respond within 15 days and offer to defend a refusal to disclose the information; or

b. if they do not respond within 15 days, will have waived their objections to disclosure of the information.

Section 574(f)

Evidence that is otherwise discoverable or admissible is not protected from disclosure under this Section merely because the evidence was presented during a dispute resolution proceeding.

Section 574(g)

The provisions of Section 574 (a) and (b) do not affect information and data that are necessary to document agreements or orders resulting from dispute resolution proceedings.

Section 574(h)

Information from and about dispute resolution proceedings may be used for educational and research purposes as long as the parties and specific issues in controversy are not identifiable.

(1) Dispute resolution communications may be used to resolve disputes between the neutral in a dispute resolution proceeding and a party or non-party participant.

(2) Dispute resolution communications may be disclosed only to the extent necessary to resolve a dispute between a neutral and party or non-party participant.

Section 574(j)

A dispute resolution communication between a neutral and a party that is protected from disclosure under this section is also protected from disclosure under FOIA (Section 552(b)).

IV. Questions and Answers on Confidentiality under the Administrative Dispute Resolution Act of 1996 (ADR Act)

General Confidentiality Rules

1. What communications are confidential?

Subject to certain exceptions, the following two types of communications are potentially confidential under the ADR Act:

A. A dispute resolution communication. A dispute resolution communication is any oral or written statement made by a party or a neutral that occurs during a dispute resolution proceeding and any writing prepared specifically for the purposes of a dispute resolution proceeding. Written agreements to enter into a dispute resolution proceeding and any written final agreement reached as a result of the proceeding are not dispute resolution communications. Citation: 5 U.S.C. 571(5).

B. A “communication provided in confidence to the neutral.” A “communication provided in confidence to the neutral” is any oral statement or document provided to a neutral during a dispute resolution proceeding. The communication must be made: (1) With the express intent that it not be disclosed, or (2) provided under circumstances that would create a reasonable expectation that it not be disclosed. Citation: 5 U.S.C. 571(7) and 574(a).

2. What confidentiality protection is provided for dispute resolution communications?

Generally, neutrals and parties may not voluntarily disclose or be compelled to disclose dispute resolution communications. The ADR Act contains specific exceptions to the general rule. Citation: 5 U.S.C. 574(a), (b).

3. What confidentiality protection applies to a “communication provided in confidence” by a party to a neutral?

A neutral may not disclose any communication provided in confidence. Citation: 5 U.S.C. 574(a).

4. What is a dispute resolution proceeding?

A dispute resolution proceeding is any process involving the services of a neutral that is used to resolve an issue in controversy arising from an agency's program, operations, or actions. A dispute resolution proceeding includes any stage of such a dispute resolution process. Citation: 5 U.S.C. 571(6) and (8). See also, Question 10.

5. Who is a Neutral?

A neutral is anyone who functions specifically to aid the parties during a dispute resolution process. A neutral may be a private person or a federal government employee who is acceptable to the parties. There may be more than one neutral during the course of a dispute resolution process (e.g., an “intake” neutral, a “convener” neutral, as well as the neutral who facilitates a face-to-face proceeding). It is important that agencies clearly identify neutrals to avoid misunderstanding.

The ADR Act supports a broad reading of the term “neutral.” An intake or convening neutral is included in this definition as “an individual who * * * functions specifically to aid the parties in resolving the controversy” because such neutrals take the necessary first steps toward a potential resolution of a dispute.

In situations where an intake neutral is identified by an agency, a party's willingness to contact and/or work with the intake neutral to initiate an ADR process is an indication that the intake neutral is acceptable to the party. Citation: 5 U.S.C 571(9), 571(6), 571(3), 573(a).

Example:

An employee contacts an agency ADR program and describes a dispute to an intake person. The conversation is confidential only if the intake person has been appropriately identified as a neutral by the agency to aid parties in resolving such disputes.

6. Who Is a Party?

A party is any person or entity who participates in a dispute resolution proceeding and is named in a legal proceeding or will be affected significantly by the outcome of the proceeding. The obligations of parties extend to their representatives and agents. Citation: 5 U.S.C. 571(10).

7. What Constitutes Disclosure?

Disclosure is not defined in the ADR Act. Disclosure occurs when a neutral, a party, or a non-party participant makes a communication available to some other person by any method.

8. May a Party or Neutral Disclose Dispute Resolution Communications in Response to Discovery or Compulsory Process?

In general, neither a neutral nor a party can be required to disclose dispute resolution communications through discovery or compulsory process. Compulsory processes include any administrative, judicial or regulatory process that compels action by an individual. (See also Question 15) Citation: 5 U.S.C. 574(a) & 574(b).

9. What Confidentiality Protection Is Provided for Communications by a Nonparty Participant in a Dispute Resolution Proceeding?

A nonparty participant in a dispute resolution proceeding is an individual other than a party, agent or representative of a party, or the neutral. This could be an individual who is asked by the neutral to present information for use of the neutral or parties. A nonparty participant has an independent right to protect his or her communications from disclosure by a neutral. A neutral needs to obtain the consent of all parties and the nonparty participant to disclose such a communication. Citation: 5 U.S.C. 574(a)(1).

10. When in an ADR Process do the Confidentiality Protections of ADR Act Apply?

Confidentiality applies to communications when a person seeking ADR services contacts an appropriate neutral. A communication made by a party to a neutral is covered even if made prior to a face-to-face ADR proceeding. Confidentiality does not apply to communications made after a final written agreement is reached, or after resolution efforts aided by the neutral have otherwise ended. Citation: 5 U.S.C. 571(6), 574(a) and (b).

Exceptions to Confidentiality Protection

11. What Communications Are Not Protected by the ADR Act?

A. A party's own communications made during a dispute resolution proceeding. A party may disclose any oral or written communication which the party makes or prepares for a dispute resolution proceeding. Citation: 5 U.S.C. 574(b)(1).

B. A dispute resolution communication that has “already been made public.” The ADR Act's confidentiality protections do not apply to a communication that has already been made public. Examples of communications that have “already been made public” include:

1. The communication has been discussed in a Congressional hearing;

2. The communication has been posted on the Internet;

3. The communications has been released to the media;

4. The communication has been placed in a court filing or testified about in a court in a proceeding not under seal;

5. The communication has been reported in the newspapers;

6. The communication has been discussed in an open meeting;

7. The communication has been released under FOIA.

Citation: 5 U.S.C. 574(a)(2) & 574(b)(3).

C. Communications required by statute to be made public. FOIA is an example of a federal statute which requires agency records to be made public under certain circumstances. NOTE: A protected dispute resolution communication which is between a neutral and a party is exempt from disclosure under FOIA. (See Question 23) Citation: 5 U.S.C. 574(a)(3), 574(b)(4), & 574(j).

D. When a court orders disclosure. A federal court may override the confidentiality protections of ADR Act in three limited situations. In order to override the confidentiality protections, a court must determine that testimony or disclosure of a communication is necessary to either (1) prevent a manifest injustice, (2) help establish a violation of law, or (3) prevent harm to the public health or safety. The court must also determine that the need for the information is of a sufficient magnitude in the particular case to outweigh the integrity of dispute resolution proceedings in general by reducing the confidence of parties in future cases that their communications will remain confidential. There are no cases as of August 2000 that have interpreted these provisions. Citation: 5 U.S.C. 574(a)(4) & (b)(5).

E. In order to resolve a dispute over the existence or meaning of a settlement arrived at through a dispute resolution proceeding. The ADR Act creates an exception to the general rule of nondisclosure for the limited purpose of determining the existence or meaning of an agreement arrived at through a dispute resolution proceeding. Parties may also disclose communications as required to enforce an agreement arrived at through a dispute resolution proceeding. Citation: 5 U.S.C. 574(b)(6).

Example:

Parties may disclose dispute resolution communications as required to show that a settlement agreement was reached or to show what the terms of this agreement were.

F. Parties' communications in joint session, with all parties present. A neutral may not disclose communications made in joint session. However, there is no prohibition against a party disclosing communications available to everyone in the proceeding. Citation: 5 U.S.C. 574(b)(7).

G. Information sought for specific purposes. The ADR Act allows for the disclosure of information for educational and research purposes, in cooperation with agencies, governmental entities, or dispute resolution programs. It is essential that the parties and specific issues in controversy not be identifiable, however. Citation: 5 U.S.C. 574(h).

Example:

An individual who has served as a neutral in a number of agency ADR proceedings may share collected experiences when participating in a training program provided that the parties and specific issues are not identifiable.

Example:

An ADR program administrator may collect statistics to monitor the results of the program.

H. Communications required to resolve disputes that arise between the neutral and a party. If there is a dispute between a neutral and a party regarding the conduct of a dispute resolution proceeding, both may disclose information to the extent necessary to resolve the dispute. Citation: 5 U.S.C. 574(i)

Example:

If a party refuses to pay the neutral for services, the neutral can disclose communications to the extent necessary to establish that payment is due.

12. Are a neutral's communications to parties in joint session or provided to all parties confidential?

Yes. ADR Act protects communications by a neutral.

Example:

Early neutral evaluations or settlement proposals provided to the parties by a neutral are protected.

Note:

A party, however, may not use this provision to gain protection for a communication by providing it to the neutral who then provides it to the other party. The statute says that the communication must be “generated” by the neutral, not just passed along by the neutral. Citation: 5 U.S.C. 574 (b)(7). (See H. Rept. 104-841,142 Cong. Rec. H11108-11 (September 25, 1996).

13. Can confidentiality attach to communications that are provided to or available to fewer than all of the parties?

Yes. The ADR Act does not prohibit disclosure of dispute resolution communications that are “provided to or * * * available to all parties to the dispute resolution proceeding.” Under a plain reading of the statute, communications are not protected when provided to, or available to, all parties; thus, they remain protected if they are provided to, or are available to, some (but not all) of the parties in a dispute.

The legislative history states, “A dispute resolution communication originating from a party to a party or parties is not protected from disclosure by the ADR Act.” H.R. Rep. No. 104-841, 142 Cong. Rec. H11, 110 (Sept. 25, 1996). The plain language of the statute is not inconsistent with this piece of legislative history, in that it can be interpreted to mean both parties in a two-party (“party to the other party”) or all parties in a multi-party dispute (“party to all other parties”). Citation: 5 U.S.C. 574(b)(7).

14. Does ADR Act provide confidentiality protection for all evidence used in the course of a dispute resolution proceeding?

No. All evidence that is otherwise discoverable is not protected merely because it was presented at a dispute resolution proceeding. Citation: 5 U.S.C. 574(f).

15. Does the ADR Act protect against the disclosure of dispute resolution communications in response to requests by federal entities for such information?

Section 574 of the ADR Act prohibits a neutral or a party from disclosing, voluntarily or in response to discovery or compulsory process, any protected communication. The ADR Act further states that neutrals and parties shall not “be required” to disclose such communications. However, a number of federal entities have statutory authority to request disclosure of documents from federal agencies and employees. Examples of such statutes include, but are not limited to, The Inspector General Act (5 U.S.C. App.); The Whistleblower Protection Act (5 U.S.C. Section 1212(b)(2)); and the Federal Service Labor-Management Relations Act (5 U.S.C. Section 7114(4)). None of the exceptions to the ADR Act's confidentiality provisions directly applies to requests for disclosure of information from federal entities. For example, these statutes do not require information to be made public under ADR Act Section 574 (a)(3) & (b)(4). In addition, the judicial override procedure outlined in Section 574 (a)(4) & (b)(5) is not always available to federal entities with authority to access information. Some federal entities may lack jurisdiction to seek a court order to compel disclosure. Other federal entities may have such jurisdiction, but may seek disclosure under other statutory authority.

In summary, a tension between these statutory authorities exists. The issues of statutory interpretation of these differing authorities have not yet been considered in an appropriate forum. We do not anticipate that there will be many occasions when such requests will be directed to neutrals or participants. However, it is important for agencies, neutrals and participants to be aware of the potential for requests.

In order to prevent unnecessary disputes over requests for information pursuant to an access statute and to mitigate damage to ADR programs, we recommend:

• Agency ADR programs should enter into a dialogue with potential requesting entities so that each may be educated about their respective missions.
• Procedures should be established for access to information that recognize the importance of confidentiality in dispute resolution processes and protect the integrity of the agency's ADR program.
• ADR programs should identify classes of information that are not confidential.
• Requesting entities should use non-confidential information as a basis for information requests.
• Requesting entities should seek confidential information only after other potential sources have been exhausted.
• Requesting entities should seek information from a neutral only as a last resort.
• The ADR program and requesting entities should agree to procedures to resolve specific disagreements that arise with regard to the disclosure of information.
• If a federal employee party or neutral receives a request for disclosure, he or she should contact the agency's ADR program as soon as possible to discuss appropriate courses of action. Neutrals must also notify parties of any such request (See Question 19).

Alternative Procedures To Establish Confidentiality Protection

16. May parties agree to confidentiality procedures which are different from those contained in ADR Act?

Yes. Parties may agree to more, or less, confidentiality protection for disclosure by the neutral or themselves than is provided for in the Act.

Subsection 574(d)(1) provides that the parties can agree to alternative confidential procedures for disclosures by a neutral. While there is no parallel provision for parties, the exclusive wording of this subsection should not be construed as indicating Congressional intent to limit alternative procedures by parties. Parties have a general right to sign confidentiality agreements, and there is no reason this should change in a mediation context.

If the parties agree to alternative confidentiality procedures regarding disclosure by a neutral, they must so inform the neutral before the dispute resolution proceeding begins or the confidentiality procedures in the ADR Act will apply. An agreement providing for alternative confidentiality procedures is binding on anyone who signs the agreement. (See Questions 23 and 24 for potential FOIA implications.)

Example:

Parties to an ADR proceeding can agree to authorize the neutral to use his or her judgment about whether to voluntarily disclose a protected communication, as long as the neutral is informed of this agreement before the ADR proceeding commences.

Example:

Parties to an ADR proceeding can agree that they, and the neutral, will keep everything they say to each other in joint session confidential.

Issues Regarding the Disclosure of Protected Communications

17. What restrictions are put on the use of confidential communications disclosed in violation of the ADR Act?

If the neutral or any participant discloses a confidential communication in violation of Sections 574(a) or (b), that communication may not be used in any proceeding that is related to the subject of the dispute resolution proceeding in which the protected communication was made. A dispute resolution communication that was improperly disclosed may not be protected from use in an unrelated proceeding. Citation: 5 U.S.C. 574(c).

18. What is the penalty for disclosing confidential communications in violation of the statute?

The ADR Act does not specify any civil or criminal penalty for the disclosure of a protected communication in violation of the Act. However, such disclosure may violate other laws, regulations or agreements of the parties.

19. What must a neutral do when he or she receives a “demand for disclosure” of confidential communications?

A demand for disclosure is a formal request for confidential information. The demand must be made by a discovery request or some other legal process. Upon receiving a demand for disclosure of a confidential communication, a neutral must make a reasonable effort to notify the parties and any affected non-party participants of the demand. Notice must be provided even if the neutral believes that there is no basis for refusing to disclose the communication.

Notice should be delivered to the last address provided by a party. Parties have fifteen days, from the date they receive the notice, in which to offer to defend the neutral against disclosure. Therefore, notice should be sent by a process that provides certification of delivery. For example, delivery could be by registered mail, by any carrier that provides tracking and certification of delivery, or by courier. Use of telephone or email communications as notice could be problematic. Since the parties must respond within 15 days or waive their right to object to disclosure, there must be a written record of when the notice was sent and when it was received. Citation: 5 U.S.C. 574(e).

Example:

A colleague asks a neutral what happened in a mediation. The neutral must simply refuse to discuss the matter. The neutral does not need to notify the parties of the request.

Example:

A neutral receives a formal discovery request for information on what happened in a mediation. The neutral must notify the parties of this demand for disclosure using the procedures described above.

20. What can/must parties do when they receive notice of a demand for disclosure from the neutral?

If a party has no objection to the disclosure of confidential communications, it need not respond to the notice. On the other hand, if a party believes that the sought-after communications should not be disclosed, it should notify the neutral and make arrangements to defend the neutral. Where the party is a federal agency, it should develop departmental procedures for processing the notice.

21. What responsibilities do agencies have for ensuring that the notification requirement is met?

In some federal ADR programs, the neutral may be a federal employee performing collateral duty. Imposing an obligation upon these neutrals to keep records of parties to dispute resolution proceedings may be unduly onerous and ineffective. Agencies should develop administrative procedures to assure that the notification functions are fulfilled.

22. May a neutral refuse to disclose communications even when the parties have failed to agree to defend the neutral?

Yes. The ADR Act permits, but does not compel, a neutral to disclose if the parties have waived objections to disclosure under Section 574(e). While the statute is clear that a neutral “shall not” disclose where a party objects, the statute does not say that a neutral must disclose if a party does not object.

The effectiveness and integrity of mediation and other ADR processes is largely dependent on the credibility and trustworthiness of neutrals. In order to safeguard the integrity of ADR programs and to eliminate the potential for eroding confidence in future ADR proceedings, neutrals should be allowed to rely on established codes of ethics and confidentiality standards to support a decision not to disclose. Citation: 5 U.S.C. 574(a) & (e).

Issues Related to the Freedom of Information Act (FOIA)

23. What dispute resolution communications are protected from disclosure under FOIA?

Dispute resolution communications between a neutral and a party that are covered by the confidentiality protections of the ADR Act are specifically exempted from disclosure under the Freedom of Information Act. This includes communications that are generated by a neutral and provided to all parties, such as an Early Neutral Evaluation. In addition, other FOIA exemptions may apply.

Since only federal records are subject to FOIA, dispute resolution communications that are not federal records are not subject to the disclosure requirements of FOIA. Therefore, this subsection would not apply to oral dispute resolution communications. Citation: 5 U.S.C. 574(j).

24. If parties agree to alternative confidentiality procedures, are dispute resolution communications subject to FOIA?

Parties may agree to confidentiality procedures that differ from those provided for in the ADR Act. Parties should be aware, however, that the FOIA exemption may not apply to all the communications protected under their agreement.

If the agreement provides for the same or more disclosure than provided by the Act, dispute resolution communications are exempt from disclosure under FOIA. If the agreement provides for less disclosure, communications are not exempt from disclosure under FOIA. The ADR Act, in effect, establishes a ceiling on the extent to which confidential communications will be exempt. Parties cannot contract for more FOIA protection than the ADR Act provides.

V. Model Confidentiality Statement for Use by Neutrals

The confidentiality provisions of the Administrative Dispute Resolution Act (ADR Act) apply to this process. Generally, if you tell me something during this process, I will keep it confidential. The same is true for written documents you prepare for this process and give to me. [Similarly, you are generally required to keep information confidential that you receive during conversations with other parties or me and from writings prepared for this process. *

Be advised, there are limits on our ability to keep information confidential. If you say something or provide documents to all the other parties it is not confidential. Under rare circumstances, a judge can order disclosure of confidential information. Even though not required by the ADR Act, information about a violation of criminal law, or an act of fraud, waste, or abuse, or an imminent threat of serious harm may have to be disclosed to appropriate authorities by a participant, but not necessarily by me.

You can agree to more confidentiality if you want to. For example, you can agree to keep confidential things you share with all the parties. If you want to do any of that, it will require the agreement of all parties and should be memorialized in writing. You should be aware that if you agree to more confidentiality, written documents may still be available to others, for example, through the Freedom of Information Act. Confidentiality provisions other than those in the ADR Act may also apply to this process.

ADR Program Evaluation Recommendations

I. Introduction

The alternative dispute resolution (ADR) field has long promoted the various benefits of using non-traditional methods to resolve disputes, such as savings of time and money, party satisfaction with the ADR process and outcomes, high settlement rates, and improved relationships. The ADR Council recognizes that ADR has the potential to produce these results, and notes the value of hard data to back up the assertion that ADR really delivers these benefits to agencies. The Council's Core Principles for Non-binding Workplace ADR Programs [and if approved, the ADR Pledge] identify evaluation as a key component of successful ADR program management. Up-front and thorough evaluation initiatives allow ADR program managers to ensure the quality of their programs, to identify programmatic successes and difficulties, and to make necessary improvements. Therefore, it is important that all federal ADR programs engage in a rigorous evaluation of ADR's use and benefits to ensure quality ADR programs and to provide the necessary information to sustain and increase support of ADR.

As the use of ADR becomes institutionalized within federal agencies, the government has a heightened interest in evaluating the benefits and impact of these dispute resolution initiatives. This type of formal evaluation is consistent with the legal obligations of all federal programs, under the Government Performance and Results Act (Pub. L. 103-62) which requires that agencies create a performance plan, define goals, and track the extent to which they achieve their desired outcomes. ADR program management best practices emphasize the importance of an evaluation component in program design as well as practice, and some federal agencies have initiated evaluations of their ADR programs. However, the federal sector will benefit from agencies' coordinated and uniform efforts at ADR program evaluation.

II. Recommendations

The Council acknowledges that throughout the government, ADR program goals and services differ dramatically among Federal agencies. Consequently, it is appropriate to tailor evaluation plans and methods to meet the needs of a particular program. Even with agency-specific tailoring, effective evaluations will include certain common elements. Therefore, to promote consistency and coordination among Federal ADR evaluation efforts, the Council makes the following recommendations to agencies:

1. Importance of Evaluation. Each agency should engage in an up-front and ongoing evaluation of its ADR programs.

2. Data to be Captured. At a minimum, evaluators should attempt to capture and analyze in a timely manner the following information:

a. Usage: the extent to which ADR is considered and used.

b. Time Savings: the time it takes for a case to be resolved through ADR as compared to traditional dispute resolution processes.

c. Cost Avoidance: the amount of financial savings (or costs) to the agency, including staff time, dollars, or other quantifiable factors, by resolving cases through ADR as compared to traditional dispute resolution processes.

d. Customer Satisfaction: parties' satisfaction with the process and outcomes, including the quality of the neutral.

e. Improved Relationships: where ongoing relationships are important, to what extent relationships are improved.

f. Other Appropriate Indicators: in line with the agency's strategic goals and objectives.

3. Validity and Reliability of Data. Methodologies should be valid and reliable. ADR program results should be compared to results from alternate or previously existing dispute resolution methods.

4. Presentation of Data. ADR Program Managers should present a realistic, accurate and complete picture of the results of their program.

5. Use of Data. ADR success stories should be summarized and publicized, to help foster a culture in which ADR is accepted as beneficial to Federal agencies and their customers. If areas for improvement are identified, that information should be used to enhance the ADR program.

6. Reporting. Federal ADR Program Managers are encouraged to report the results of their evaluations to the Federal Interagency ADR Working Group.

7. Potential Resources. In undertaking ADR activities, agencies should consult: (1) The Federal ADR Program Manager's Resource Manual, Chapter 8: Evaluating ADR Programs, and (2) The Electronic Guide to Federal Procurement ADR. Both of these resources, as well as other valuable information are available electronically at: www.financenet.gov/iadrwg

Evaluating ADR Programs

I. Introduction

For the past ten years the practice of ADR, the creation of ADR programs, and the discipline of ADR evaluation have been developing in tandem. We have learned that organizations best design and develop ADR programs by knowing an organization's conflict resolution culture, we see that evaluation can and should be a reflective feedback mechanism for ADR program development, and that evaluation belongs at the beginning of ADR program design. While evaluation is ideally present at the beginning of ADR program development, we recognize that there are many ADR programs already up and running that do not have evaluation components. This chapter will address ADR programs at any stage along the way of program development.

II. Planning and Designing the Evaluation

Traditional ADR program evaluation is a way to determine whether an ADR program is meeting its goals and objectives. Evaluation data are useful in finding out what works and what does not work and may be a critical factor in decisions to modify or expand a program.

When planning and designing a federal ADR program evaluation, it is important to understand what components of the program are essential to comply with federal statutes and initiatives. To the extent that an ADR program maintains compliance with federal ADR requirements, it fulfills a necessary and useful function for your organization or agency. A good design will build upon an existing program structure and will establish an evaluation methodology for each program “core” area, core areas being defined by statute or initiative. Overall program effectiveness can then be determined by combining data from all function areas, with consideration being given to intangible benefits and consumer satisfaction.

Evaluation is an art as well as a science, even, perhaps, a state of mind. It is almost never a linear process. Decisions made early in the evaluation planning and design process will almost certainly need to be reconsidered and modified as your ADR program grows and develops. In addition, traditional cost/benefit analysis does not capture many of the benefits derived from ADR service programs because these benefits are often intangible and not easily quantifiable. With all of this in mind, evaluators need to strive for a workable balance between the need for defensible results and practical limitations.

Key questions to ask when planning and designing an ADR program evaluation are:

• What are your goals and objectives for your ADR program evaluation?
• How will you pay for your ADR evaluation?
• Who will evaluate your ADR program?
• Who is your audience for this evaluation?
• What is your evaluation design strategy?
• What are your measures of success?

A. What Are Your Goals and Objectives for Your ADR Program Evaluation?

The goals and objectives of an evaluation should link closely with the goals and objectives of the ADR program being evaluated, should reflect the needs and interests of those requesting the evaluation, and should be sensitive to the needs and interests of the expected audiences for the results. Ideally, the ADR program's goals and objectives will have been established early on. Sometimes, however, these goals may not have been clearly articulated, may not be measurable as stated, or may have changed. Evaluators may need to ask program managers and other stakeholders to provide input (and hopefully arrive at a consensus) on the program's goals, while addressing questions such as, how well is the program working, should changes be made, should the program be continued or expanded, and how well is the ADR program working in a particular federal context?

B. How Will You Pay For Your ADR Evaluation?

The cost of conducting an ADR program evaluation depends upon a number of factors, such as the number and complexity of success measures, the type of ADR program selected, the level of statistical significance required of the results, the availability of acceptable data, and who is selected to carry out the evaluation. Costs can be controlled, however, by careful planning, appropriate adjustments in the design phase, and a creative use of outside evaluators, from universities, for example.

C. Who Will Evaluate Your ADR Program?

When selecting an evaluator, or a team of evaluators, a number of qualifications should be considered. Objectivity (i.e. no stake in the outcome) is essential for your results to be seen as credible. An evaluator should have sufficient knowledge of the ADR process as well as program expertise to design the evaluation, perform the data collection process and data analysis as well as present your results to your audience if you chose to have the evaluator present your results. Such expertise may be found inside some agency policy and program evaluation offices, at the U. S. General Accounting Office, or at various outside evaluation consulting firms and university departments specializing in social science research. Some understanding of the organization or the context in which the program operates can be helpful to the evaluator, as are good interpersonal and management skills.

Evaluations can be conducted by people outside the agency, within the agency but outside the program being evaluated, or by people involved with the ADR program. There are advantages and disadvantages to each option. An outside evaluator has the potential for the greatest impartiality, lending credibility and validity to your results. In addition, depending upon the expertise available in a particular agency, an outside evaluator may have more technical knowledge and experience. Outside evaluation may be relatively expensive, however, depending upon the affiliation of the evaluators (e.g. colleges or universities, other non-profit groups, or private sector entities such as management consulting or social science research firms). If the agency has evaluation capacity inside the organization where the ADR program is being implemented, the requisite neutrality may be available at a potentially lower cost. An inside evaluator involved in ADR program implementation or design may be the least expensive, and offer the best understanding of program context, but it also carries with it potential perceptions of a lack of impartiality. One way to avoid some of the disadvantages of each of these approaches is to use a team of people, representing internal and external groups.

Regardless of who does the evaluation (outside or inside), it is useful to have someone in the ADR program who can serve as a liaison with the evaluator to ensure access to the necessary information. The liaison might be the person responsible for planning the evaluation.

D. Who Is Your Audience For This Evaluation?

There are usually a variety of people who have an interest in the results of a program evaluation. These audiences may be interested in different issues and seek different types of information. Potential audiences should be identified as early as possible, and kept in mind while planning the evaluation, so that their questions will be addressed.

Possible audiences for an ADR program evaluation include ADR program officials, other agency officials, program users, members of Congress, the general public, and others. Agency program officials may be interested in finding out how the ADR program is working, and how it might be improved. Their interests might focus, for example, on the program's impact on case inventory (backlogs), the effects of ADR use on long-term relationships among disputants, or how well information about the program is being disseminated. Program officials involved in the day-to-day operation may have different interests than those at higher levels.

Other agency officials such as budget officers, staff within offices of General Counsel and Inspector General, or managers from other programs may also have an interest in evaluation results. Budget officials may be interested in whether cost savings have been achieved through implementation of the program. The Inspector General may be interested in the nature of the settlements and whether ADR use promotes long-term compliance. General Counsels may care about how long it takes to resolve cases or the nature of outcomes; other managers may want to know how effectively the program was implemented.

Members of Congress and their staffs may be interested in how ADR use affects budgets and how related laws, such as the Administrative Dispute Resolution Act, are being implemented. Members of the public may be interested in how efficiently the agency is resolving its disputes, and how satisfied participants are with ADR processes. Disputants may be interested in finding out how typical their experience was compared to other users. Officials in other federal agencies may find evaluation results helpful as they plan or modify their own ADR programs. There may be other audiences whose interests or desire for information should be considered.

Although terminology differs, evaluations are commonly characterized as either: (1) Program effectiveness (also known as impact, outcome, or summative) evaluations, which focus on whether a program is meeting its goals and/or having the desired impact; or (2) program design and administration (also known as process or formative) evaluations, which examine how a program is operating. Program effectiveness evaluations may be useful in determining whether a program should be continued or expanded; program design/administration evaluations often focus on how a continuing program can be improved.

Remember that decisions on the future of programs (or even how they could be improved) are usually not made solely on the basis of program evaluation results. Agency priorities, other institutional concerns, budget limitations, and other factors will also affect program decisions.

While it is not possible to satisfy every audience by answering all potential questions, it is useful to figure out what the possible questions are and then focus the evaluation on the most important ones. Talking to members of the various potential audiences can help identify the issues they are interested in, and may help develop consensus about which issues to address. Such discussions also improve the likelihood that evaluation results will be a useful and meaningful part of future decision making processes.

E. What Is Your Evaluation Design Strategy?

ADR program design is based on an understanding that certain components of a program are essential to comply with federal statutes and initiatives. Program effectiveness evaluations are conducted to answer fundamental questions about a program's utility, e.g., does the program provide a necessary or useful function, is the program accomplishing its goals, and is the program being administered effectively. A comprehensive evaluation system measures tangible and intangible benefits, including customer satisfaction, using both quantitative and qualitative data. To be a useful and effective management and planning tool, an evaluation system must do more than provide comparison data. It also must provide a flexible process for reevaluating the goals of the program, modifying the evaluation methodology, and implementing necessary changes.

Development of an evaluation design might include the following steps:

1. Identification and Clarification of ADR Program Goals

Clear goals and objectives mean that useful conclusions can be drawn from the data collected.

2. Development of an Appropriate Evaluation Methodology

It is necessary to determine what is to be measured and how, what the sources of the data are, and how the data will be collected. To do this most effectively, core functional areas of ADR program practice need to be identified, as do quantitative and qualitative sources of data.

3. Development of an Analysis Plan and Research Methodologies

Traditionally-based experimental designs (time-cost benefit analysis) provide statistically reliable results. Program analysis, while producing quantifiable results, must go beyond a bare assessment of program outcomes to explain the outcomes and to offer suggestions for program improvement.

4. Collection Data Mechanisms

Status reports, case studies, time series collections, agency databases, logs, surveys, and evaluation forms are all sources of information, as are personal interviews.

F. What Are Your Measures of Success?

1. Program Effectiveness (Impact)

Program effectiveness measures are aimed at assessing the impact of the program on users/participants, overall mission accomplishment, etc.

The indicators of program effectiveness can be further divided into three categories: efficiency, effectiveness, and customer satisfaction.

• Efficiency

□ Cost to the Government of using alternative dispute resolution vs. traditional dispute resolution processes:

Is the use of ADR more or less costly than the use of traditional means of dispute resolution? (Cost may be measured in staff time, dollars, or other quantifiable factors.)

□ Cost to disputants of using alternative dispute resolution vs. traditional dispute resolution processes:

Is the use of ADR more or less costly than the use of traditional means of dispute resolution? (Cost may be measured in terms of staff time, dollars, or other quantifiable factors.)

□ Time required to resolve disputes using alternative dispute resolution vs. traditional means of dispute resolution:

Are disputes resolved more or less quickly using ADR, compared to traditional means of dispute resolution? Such factors as administrative case processing, participant preparation, dispute resolution activity timeframes, and/or days to resolution may be considered.

• Effectiveness

□ Dispute Outcomes

Number of settlements achieved through the use of mediation vs. traditional dispute resolution processes:

Does the use of alternative dispute resolution result in a greater or a fewer number of settlements?

Number of cases going beyond mediation steps:

Does the use of alternative dispute resolution result in a greater/fewer number of investigations, further litigation activities, etc.?

Nature of outcomes:

What impact does the use of alternative dispute resolution have on the nature of outcomes, e.g. do settlement agreements “look different”? Do settlement agreements reflect more “creative” solutions? Do outcomes vary according to the type of alternative dispute resolution process used?

Correlations for cases selected for alternative dispute resolution, between dispute outcomes and such factors as complexity or number of issues, or number of parties:

Is there any correlation, where ADR is used, between the complexity and/or number of parties/issues in a case and the outcome of the case?

□ Durability of Outcomes

Rate of compliance with settlement agreements:

Does the use of alternative dispute resolution result in greater or lesser levels of compliance with settlement agreements?

Rate of dispute recurrence:

Does the use of alternative dispute resolution result in greater or lesser levels of dispute recurrence, i.e. recurrence of disputes among the same parties?

□ Impact on Dispute Environment

Size of case inventory:

Does the use of alternative dispute resolution result in an increase/decrease in case inventory?

Types of disputes:

Does the use of alternative dispute resolution have an impact on the types of disputes that arise?

Negative impacts:

Does the use of alternative dispute resolution have any negative consequences, e.g. an inability to diagnose and correct systemic problem/issues?

Timing of dispute resolution:

Does the use of alternative dispute resolution affect the stage at which disputes are resolved?

Level at which disputes are resolved:

Does the use of alternative dispute resolution have any impact on where and by whom disputes are resolved?

Management perceptions:

What are the quantitative and qualitative effects of using alternative dispute resolution on management, e.g. how does the use of ADR impact upon allocation and use of management time and resources? Does the use of ADR ease the job of managing?

Public perceptions:

Is the public satisfied with alternative dispute resolution outcomes? Is there any perceived impact of use of ADR on effectiveness of the underlying program? “Public” may be defined differently, depending on the particular program/setting involved.

• Customer Satisfaction

□ Participants' Satisfaction with Process

Participants' perceptions of fairness:

What are participant perceptions of access to alternative dispute resolution, procedural fairness, fair treatment of parties by neutrals, etc.?

Participants' perceptions of appropriateness:

What are participant perceptions of appropriateness of matching decisions (i.e. matching of particular process to particular kinds of disputes or specific cases)?

Participants' perceptions of usefulness:

What are participant perceptions of the usefulness of alternative dispute resolution in the generation of settlement options, the quantity and reliability of information exchanged, etc.?

Participants' perceptions of control over their own decisions:

Do participants feel a greater or lesser degree of control over dispute resolution process and outcome through the use of alternative dispute resolution? Is greater control desirable?

□ Impact on Relationships Between Parties

Nature of relationships among the parties:

Does the use of alternative dispute resolution improve or otherwise change the parties' perceptions of one another? Is there a decrease or increase in the level of conflict between the parties? Are the parties more or less likely to devise ways of dealing with future disputes? Are the parties able to communicate more directly or effectively at the conclusion of the ADR process and/or when new problems arise?

□ Participants' Satisfaction with Outcomes

Participants' satisfaction with outcomes:

Are participants satisfied or unsatisfied with the outcomes of cases in which alternative dispute resolution has been used?

Participants' willingness to use alternative dispute resolution in the future:

Would participants elect to use alternative dispute resolution in future disputes?

2. Program Design and Administration (Structure and Process)

How a program is implemented will have an impact on how effective a program is in meeting its overall goals. Program design and administration measures are used to examine this relationship and to determine how a program can be improved.

The indicators of program design and administration are further divided into three categories: program organization, service delivery, and program quality.

• Program Organization

□ Program structure and process:

Are program structure and process consistent with underlying laws, regulations, executive orders, and/or agency guidance? Do program structure and process adequately reflect program design? Are program structure and process adequate to permit appropriate access to and use of the program?

□ Directives, guides, and standards:

Do program directives, guides, and standards provide staff/users with sufficient information to appropriately administer/use the program?

□ Delineation of responsibilities:

Does the delineation of staff/user responsibilities reflect program design? Is the delineation of responsibilities such that it fosters smooth and effective program operation?

□ Sufficiency of staff (number/type):

Is the number/type of program staff consistent with program design and operational needs?

□ Coordination/working relationships:

Is needed coordination with other relevant internal and external individuals and organizations taking place? Have effective working relationships been established to carry out program objectives?

• Service Delivery

□ Access and Procedure

Participant access to alternative dispute resolution:

Are potential participants made aware of the program? Is the program made available to those interested in using ADR?

Relationship between participant perceptions of access and usage of alternative dispute resolution:

What impact do participants' perceptions about the availability of the program have on the levels of program usage?

Participant understanding of procedural requirements:

Do program users understand how the program works? Did they feel comfortable with the process in advance?

Relationship between procedural understanding and rates of usage:

Is there any relationship between the level of participant understanding and the degree of program use, e.g. is a lack of participant understanding serving as a disincentive to using the program?

□ Case Selection Criteria

Participants' perceptions of fairness, appropriateness:

Do participants feel that appropriate types of cases are being handled in the program? Do participants or non-participants feel that the criteria for which cases are eligible for alternative dispute resolution are fair? Are cases being sent to the program at the appropriate dispute stages?

Relationship between dispute outcomes and categories of cases:

Is there a correlation between the nature (size, types of disputants, and/or stage of the dispute) of cases and the outcome of the dispute? Are certain types of cases more likely to be resolved through alternative dispute resolution than other types?

• Program Quality

□ Training

Participants' perceptions of the appropriateness of staff and user training:

Do participants feel that they were provided with sufficient initial information and/or training on how to use the program? Do they feel that program staff had sufficient training and/or knowledge to appropriately conduct the program?

Relationship between training variable and dispute outcomes:

Is there a relationship between the type/amount of training (for participant and/or staff) and dispute outcomes?

□ Neutrals

Participants' views of the selection process:

Are participants satisfied with the manner in which neutrals were selected and assigned to cases? Were they involved in the selection decision? If not, did they feel they should be?

Relationship between participants' views of the selection process, perceptions of neutral competence and objectivity, and dispute outcomes:

Is there any relationship between participant views about the neutrals selection process and dispute outcomes? How do these views affect participants' assessment of the competence and neutrality of neutrals?

Participants' perceptions of competence (including appropriateness of skill levels/training):

Do participants feel that neutrals were sufficiently competent or trained? Do participants feel that more or less training was needed?

Participants' perceptions of neutrality/objectivity:

Do participants feel that neutrals were sufficiently objective? Do participants feel that neutrals were fair in their handling of the dispute?

G. Other Specific Program Features

Every dispute resolution program is unique. Those requesting and/or conducting an evaluation may want to consider examining other aspects of the program. These unique features may relate to the design of a program, who was and continues to be involved in program design and administration, etc. Each is likely to have at least some impact on service delivery and the quality of the program, and should be considered for inclusion in either a comprehensive or selected evaluation of the program, as appropriate.

II. Presentation, Dissemination, and Use of Results

Results should be communicated in ways that will allow meaningful decisionmaking by program administrators and decisionmakers.

It is easier to make decisions about the best way to present and disseminate results if the people who will use the results (the audience) have been consulted during the initial and subsequent evaluation processes. Such consultation can avoid costly or embarrassing errors; e.g., omission of a key area for analysis, and can ensure the report meets the needs of those who will be using it.

A. What Is the Best Method for Communicating Your Findings?

There are a variety of ways that evaluators can communicate results to potential audiences. Evaluators or program staff may provide briefings, hold meetings with users, and/or prepare a written report.

Briefings and presentations allow evaluators or program staff to convey important evaluation information quickly and selectively. In selecting material to be presented, care should be taken to avoid bias or presentation of material out of context. Some discussion of methodology is important, as are appropriate cautions about the limits and appropriate use of evaluation data. Providing for interaction with or feedback from the audience may allow issues and potential problems to be identified.

Written reports typically take a great deal of time to prepare, but allow evaluators to provide considerably more detail on both methodology and results. Legislation or executive decisions often require a final, written report. If it is important to ensure that there is one “official” source of information on evaluation methodology and results, a formal, written report may be an important and/or required format in addition to briefings and presentations by evaluators or staff.

B. What Kind of Information Needs to Be Communicated?

Although the potential audiences, program content, and evaluation objectives will vary for each ADR program evaluation, it is generally helpful to include the following kinds of information in a report or other type of presentation:

• Description of the ADR program and how it operates;
• Goals and objectives of the evaluation;
• Description of the evaluator's methodology;
• Presentation of evaluation findings;
• Discussion of program strengths and weaknesses;
• Implications for program administration (e.g., training, budget, staff.); and
• Recommendations as appropriate.

Presentation style is entirely a matter of what works for whom. It is always important, however, to make sure that evaluation data are presented accurately and completely, to prevent charges of misrepresentation or overreaching, and to avoid misuse of results.

B. How Can You Enhance the Effectiveness of Your Presentation?

Variations in presentation format and style aside, we offer the following suggestions for making the presentation of evaluation results as effective as possible.

• Involve potential users as early as possible in determining presentation format and style:

Evaluation data should be organized and communicated in a way that is useful for potential audiences and users.

• Tailor presentation method, format, and style to audience needs:

Select the method of presentation (e.g., oral briefing, written report), format, and style of presentation (e.g., formal vs. informal, briefing vs. discussion) based on who your audience is and what their needs are. There may be multiple audiences with multiple needs. Be flexible and willing to adapt material as appropriate.

• Be clear and accurate:

Evaluation information must be presented clearly and accurately. Always keep the audience in mind as you prepare to describe your ADR program and present evaluation data. Avoid any gaps in describing the program or presenting the results. A clear and accurate portrayal of the program and evaluation results will allow the audience to draw appropriate conclusions about program effectiveness and any need for change.

• Be honest and direct:

Sharing evaluation findings with potential users and involving them in key decisions concerning presentation format and style does not mean publishing only those findings that reflect well on the program or those affiliated with it. Evaluators must present the story objectively; too heavy an emphasis on the positive may cast doubt on the integrity of the results as well as the integrity of the evaluators. Data that suggest weaknesses in program design or administration or that reveal failure to accomplish program goals or objectives should be reported and can be used as a basis for suggesting appropriate changes. Honest analysis and thoughtful consideration of the information will enhance both the credibility and usefulness of the results.

• Keep the body of the report or the bulk of the presentation simple: Reduce complex data to understandable form, use graphic illustrations where appropriate. Evaluation results must be presented so that the most essential data are available, understandable, and useful. Too complex a format or over-reliance on narrative may detract from evaluation results and analysis. Organize the presentation or report for multiple uses. Use headings and subheadings to help the audience identify useful information quickly.

Limit the use of technical jargon. Prevent misinterpretation or misuse by considering how the data will look if lifted from the context of the presentation or report. Use simple graphics to illustrate results and call attention to key findings. Use footnotes and make technical data available in handouts or appendices so that the body of the presentation or report is as uncomplicated as possible.

• Provide an executive summary or abstract:

Evaluators should provide an overview. The “quick take” should be supplemented by more detailed discussion later in the report.

• Make survey instruments and other data collection tools available: Materials can be made available as handouts, at an oral presentation or face-to-face meeting, or as appendices to a written report. The availability of such material enhances both understanding and credibility. It also allows other ADR program evaluators to learn from the experiences of their peers.
• Note limitations on the interpretation and use of evaluation data, where appropriate: Limitations on the interpretation of the data, such as those that might relate to the ability to study results, should be communicated to the audience. Evaluators need to exercise caution in expressing their own views and conclusions. Where conclusions are not an objective reflection of the data, they need to be labeled appropriately; i.e., as the views of the evaluators and not necessarily of officials responsible for the program.
• Expect the need for follow-up; be flexible and responsive:

Have extra copies of reports and presentation handouts available. Keep materials accessible. Provide addresses and telephone numbers for follow-up discussion or questions. Be available for consultation. Stay abreast of how results are being used; provide clarification or added direction in the case of misinterpretation or misuse. Prepare additional materials as needed. Tailor subsequent releases to customer needs.

B. Who Is Responsible for Making Decisions Regarding the Dissemination of Evaluation Results?

It is important to think about dissemination of the results at two points: early in the planning process, and again as results become available. Decisions about dissemination may be made solely by the evaluator, solely by program officials or other entity that has requested the evaluation, or, more typically, cooperatively. Such decisions may be circumscribed by contract or agreement, or may be discussed and resolved informally by evaluators and decisionmakers.

C. When Should Evaluation Results Be Made Available?

Decisionmakers need to consider the implications of releasing evaluation results at different times. For example, if you want publicity for the results, select slower news days. The timing of data release may be defined by contract or agreement, or may otherwise be discussed and resolved by evaluators and decisionmakers. Releasing preliminary data before all data are collected or analyzed may be risky.

D. How Widely Will Evaluation Results Be Disseminated?

Evaluation results may be disseminated widely or narrowly. Cost, convenience, and level of interest are likely to play a role. It is rare that either the evaluator or program officials will have complete control over dissemination of the results.

E. How Will Evaluation Results Be Disclosed Initially?

Evaluation results can be initially disclosed in different ways, with more or less fanfare. They may be made available to the selected audiences by memorandum, by press release, by press conference, etc. Typically, such decisions will be made at the executive level, by those who have the authority to make the disclosure.

Evaluation Checklist

✔ Is your ADR program ongoing or in the formative stage?

✔ What are your goals and objectives for your ADR program evaluation?

✔ How will you pay for your ADR program evaluation?

✔ Who will do the evaluation?

✔ Who is your audience?

✔ What is your evaluation design strategy?

✔ What are your measures of success?

✔ What do you need to know about your program effectiveness (impact)?

✔ What do you need to know about your program structure and administration?

✔ How and when will you disseminate your evaluation results?

Resources

Administrative Conference of the United States. (1995). Dispute Systems Design Working Group. Evaluating ADR Programs: A Handbook for Federal Agencies. Washington, D.C.: Administrative Conference of the United States.

Brett, J. M., Barsness, Z. I., & Goldberg, S. B. (1996). The Effectiveness of Mediation: An Independent Analysis of Cases Handled by Four Major Service Providers. Negotiation Journal, 12 (3), 259-269.

Costantino, Cathy and Sickles-Merchant, Christine. (1996). Designing Conflict Management Systems: A Guide to Creating Productive and Healthy Organizations. Jossey-Bass.

Empowerment Evaluation: http://www.stanford.edu/davidf/empowermentevaluation.html

Federal Deposit Insurance Corporation. (1999). Checklist for Evaluation of Federal Agency ADR Programs: Short and Long Term. Attorney General's ADR Working Group, Workplace Session Notes, 5/18/99.

Federal Deposit Insurance Corporation. (1997). ADR Program Evaluation Project, Annual Report.

Galanter, M. (1989). Compared to What? Assessing the Quality of Dispute Processing. Denver University Law Review, 66 (3), xi-xiv.

Honeyman, C. (1990). On Evaluating Mediators. Negotiation Journal, 23-36.

Honeyman, C. (1995). Financing Dispute Resolution. Madison, WI: Wisconsin Employment Relations Commission.

McEwen, C. A. (1991). Evaluating ADR Programs. In F. E. A. Sander, Emerging ADR Issues in State and Federal Courts. Washington, D.C.

Patton, Michael. (1990). Qualitative Evaluation and Research Methods. Sage: Beverly Hills, CA.

Posovac, Emil J. and Raymond B. Carey. (1997). Program Evaluation: Methods and Case Studies, 5th Edition. Prentice Hall Humanities/Social Sciences.

Rossi, Peter and Howard Freeman. (1993). Evaluation: A Systematic Approach. Sage: Beverly Hills, CA.

Scher, E. (1996). Evaluations: What for, by Whom, Who Pays? Consensus, October 5, 7-8.

Susskind, L. E. (1986). Evaluating Dispute Resolution Experiments. Negotiation Journal, April, 135-139.

Tyler, T. (1989). The Quality of Dispute Resolution Procedures and Outcomes. Denver University Law Review, 66, 419-436.

Wholey, Joseph S., Harry P. Hatry, and Kathryn E. Newcomer, Eds. (1994). Handbook of Practical Program Evaluation. Jossey-Bass.

Worthen, B.R., J.R. Sanders, and J. Fitzpatrick. (1997). Program Evaluation: Alternative Approaches and Practical Guidelines. Addison, Wesley, Longman.

This document was written by Lee Scharf, ADR Specialist at the Environmental Protection Agency, and draws from the work of Cathy Costantino and Christine Sickles-Merchant as well as that of the Administrative Conference of the United States. See the Resources section for cites.