Notice of Republication of Standard Services Grants Announcement

AGENCY:

Substance Abuse and Mental Health Services Administration, HHS.

ACTION:

Notice of republication of standard services grants announcement.

SUMMARY:

On November 21, 2003, the Substance Abuse and Mental Health Services Administration published standard grant announcements for Services Grants, Infrastructure Grants, Best Practices Planning and Implementation Grants, and Service to Science Grants. The primary purpose of this republication is to revise the criteria used to screen out applications from peer review. Motivated by the need to assure equitable opportunity and a “level playing field” to all applicants, SAMHSA believes the screening criteria in these announcements will not best serve the public unless revised and republished. This is a republication of the Services Grants announcement. This republication makes those criteria more lenient, permitting a greater number of applications to be reviewed. The revisions to the criteria can be found, in their entirety, in: Section IV, Application and Submission Information; and Appendix A, Checklist for Formatting Requirements and Screenout Criteria for SAMHSA Grant Applications. Additional references to the criteria elsewhere in the text have been changed to be consistent with the revised criteria in Section IV and Appendix A.

Authority: Sections 509, 516, and 520A of the Public Health Service Act.

In addition, this republication includes an additional award criterion in Section V, updated agency contact information in Section VII, and minor technical changes to comply with the formatting requirements for announcement of Federal funding opportunities, as specified by the Office of Management and Budget.

This notice provides the republished text for SAMHSA's standard Services Grants announcement.

DATES:

Use of the republished standard Services Grants announcement will be effective March 8, 2004. The standard Services Grants announcement must be used in conjunction with separate Notices of Funding Availability (NOFAs) that will provide application due dates and other key dates for specific SAMHSA grant funding opportunities.

ADDRESSES:

Questions about SAMHSA's standard Services Grants announcement may be directed to Cathy Friedman, M.A., Office of Policy, Planning and Budget, 5600 Fishers Lane, Room 12C-26, Rockville, Maryland 20857. Fax: (301-594-6159) E-mail: cfriedma@samhsa.gov.

FOR FURTHER INFORMATION CONTACT:

Cathy Friedman, M.A., Office of Policy, Planning and Budget, 5600 Fishers Lane, Room 12C-26, Rockville, Maryland 20857. Fax: (301-594-6159) E-mail: cfriedma@samhsa.gov. Phone: (301) 443-6902.

SUPPLEMENTARY INFORMATION:

SAMHSA is republishing its standard Services Grants announcement to make the criteria used to screen out applications from peer review more lenient, permitting a greater number of applications to be reviewed. This republication also includes an additional award criterion in Section V, updated agency contact information in Section VII, and minor technical changes to comply with the formatting requirements for announcement of Federal funding opportunities, as specified by the Office of Management and Budget. The text for the republished standard Services Grants announcement is provided below.

The standard Services Grants announcement will be posted on SAMHSA's web page ( http://www.samhsa.gov ) and will be available from SAMHSA's clearinghouses on an ongoing basis. The standard announcements will be used in conjunction with brief Notices of Funding Availability (NOFAs) that will announce the availability of funds for specific grant funding opportunities within each of the standard grant programs (e.g., Homeless Treatment grants, Statewide Family Network grants, HIV/AIDS and Substance Abuse Prevention Planning Grants, etc.).

Department of Health and Human Services

Substance Abuse and Mental Health Services Administration

Services Grants—SVC 04 PA (MOD)

(Modified Announcement)

Catalogue of Federal Domestic Assistance (CFDA) No.: 93.243 (unless otherwise specified in a NOFA in the Federal Register and on http://www.grants.gov )

Key Dates

Application Deadline—This Program Announcement provides general instructions and guidelines for multiple funding opportunities. Application deadlines for specific funding opportunities will be published in Notices of Funding Availability (NOFAs) in the Federal Register and on http://www.grants.gov.

Intergovernmental Review (E.O. 12372)—Letters from State Single Point of Contact (SPOC) are due no later than 60 days after application deadline.

Public Health System Impact Statement (PHSIS)/Single State Agency Coordination—Applicants must send the PHSIS to appropriate State and local health agencies by application deadline. Comments from Single State Agency are due no later than 60 days after application deadline.

Table of Contents

I. Funding Opportunity Description

1. Introduction

2. Expectations

II. Award Information

1. Award Amount

2. Funding Mechanism

III. Eligibility Information

1. Eligible Applicants

2. Cost Sharing

3. Other

IV. Application and Submission Information

1. Address to Request Application Package

2. Content and Form of Application Submission

3. Submission Dates and Times

4. Intergovernmental Review (E.O. 12372) Requirements

5. Funding Limitations/Restrictions

6. Other Submission Requirements

V. Application Review Information

1. Evaluation Criteria

2. Review and Selection Process

VI. Award Administration Information

1. Award Notices

2. Administrative and National Policy Requirements

3. Reporting Requirements

VII. Agency Contacts

Appendix A—Checklist for Formatting Requirements and Screenout Criteria for SAMHSA Grant Applications

Appendix B—Glossary

Appendix C—National Registry of Effective Programs

Appendix D—Center for Mental Health Services Evidence-Based Practice Toolkits

Appendix E—Effective Substance Abuse Treatment Practices

Appendix F—Statement Of Assurance

Appendix G—Logic Model Resources

I. Funding Opportunity Description

1. Introduction

The Substance Abuse and Mental Health Services Administration (SAMHSA) announces its intent to solicit applications for Services Grants. These grants will expand and strengthen effective, culturally appropriate substance abuse and mental health services at the State and local levels. The services implemented through SAMHSA's Services Grants must incorporate the best objective information available regarding effectiveness and acceptability. In general, the services implemented through SAMHSA's Services Grants will have strong evidence of effectiveness. However, because the evidence base is limited in some areas, SAMHSA may fund some services for which the evidence base, while limited, is sound. SAMHSA expects that the services funded through these grants will be sustained by the grantee beyond the term of the grant.

SAMHSA also funds grants under three other standard grant announcements:

• Infrastructure Grants support identification and implementation of systems changes but are not designed to fund services.
• Best Practices Planning and Implementation Grants help communities and providers identify practices to effectively meet local needs, develop strategic plans for implementing/adapting those practices and pilot-test practices prior to full-scale implementation.
• Service to Science Grants document and evaluate innovative practices that address critical substance abuse and mental health service gaps but that have not yet been formally evaluated.

This announcement describes the general program design and provides application instructions for all SAMHSA Services Grants. The availability of funds for specific Services Grants will be announced in supplementary Notices of Funding Availability (NOFAs) in the Federal Register and at http://www.grants.gov —the Federal grant announcement web page.

SAMHSA's Services Grants are authorized under Section 509, 516 and/or 520A of the Public Health Service Act, unless otherwise specified in a NOFA in the Federal Register and on http://www.grants.gov.

Typically, funding for Services Grants will be targeted to specific populations and/or issue areas, which will be specified in the NOFAs. The NOFAs will also:

• Specify total funding available for the first year of the grants and the expected size and number of awards;
• Provide the application deadline;
• Note any specific program requirements for each funding opportunity; and
• Include any limitations or exceptions to the general provisions in this announcement (e.g., eligibility, allowable activities).

It is, therefore, critical that you consult the NOFA as well as this announcement in developing your grant application.

2. Expectations

The Services Grant program is designed to address gaps in substance abuse and mental health services and/or to increase the ability of States, units of local government, Indian tribes, tribal organizations and governments, and community- and faith-based organizations to help specific populations or geographic areas with serious, emerging mental health and substance abuse problems. SAMHSA intends that its Services Grants result in the delivery of services as soon as possible and no later than 4 months after award. SAMHSA's Services Grants may include substance abuse prevention, substance abuse treatment and/or mental health services. Throughout this announcement, SAMHSA will use the term “services” to refer to all three types of services. The NOFA will provide guidance on the particular type of service to be provided through each funding opportunity.

2.1 Documenting the Evidence-Base for Services To Be Implemented

The services implemented through SAMHSA's Services Grants must incorporate the best objective information available regarding the effectiveness and acceptability of the services to be implemented. In general, the services implemented through SAMHSA's Services Grants will have strong evidence of effectiveness. However, because the evidence base is limited in some areas, SAMHSA may fund some services for which the evidence of effectiveness is based on formal consensus among recognized experts in the field and/or evaluation studies that have not been published in the peer reviewed literature.

Applicants must document in their applications that the services/practices they propose to implement are evidence-based services/practices. In addition, applicants must justify use of the proposed services/practices for the target population along with any adaptations or modifications necessary to meet the unique needs of the target population or otherwise increase the likelihood of achieving positive outcomes. Further guidance on each of these requirements is provided below.

Documenting the Evidence-Based Practice/Service

SAMHSA has already determined that certain services/practices are solidly evidence-based services/practices and encourages applicants to select services/practices from following sources (though this is not required):

• SAMHSA's National Registry of Effective Programs (NREP) (see Appendix C)
• Center for Mental Health Services (CMHS) Evidence Based Practice Tool Kits (see Appendix D)
• List of Effective Substance Abuse Treatment Practices (see Appendix E)
• Additional practices identified in the NOFA for a specific funding opportunity, if applicable

Applicants proposing services/practices that are not included in the above-referenced sources must provide a narrative justification that summarizes the evidence for effectiveness and acceptability of the proposed service/practice. The preferred evidence of effectiveness and acceptability will include the findings from clinical trials, efficacy and/or effectiveness studies published in the peer-reviewed literature.

In areas where little or no research has been published in the peer-reviewed scientific literature, the applicant may present evidence involving studies that have not been published in the peer-reviewed research literature and/or documents describing formal consensus among recognized experts. If consensus documents are presented, they must describe consensus among multiple experts whose work is recognized and respected by others in the field. Local recognition of an individual as a respected or influential person at the community level is not considered a “recognized expert” for this purpose.

In presenting evidence in support of the proposed service/practice, applicants must show that the evidence presented is the best objective information available.

Justifying Selection of the Service/Practice for the Target Population

Regardless of the strength of the evidence-base for the service/practice, all applicants must show that the proposed service/practice is appropriate for the proposed target population. Ideally, this evidence will include research findings on effectiveness and acceptability specific to the proposed target population. However, if such evidence is not available, the applicant should provide a justification for using the proposed service/practice with the target population. This justification might involve, for example, a description of adaptations to the proposed service/practice based on other research involving the target population.

Justifying Adaptations/Modifications of the Proposed Service/Practice

SAMHSA has found that a high degree of faithfulness or “fidelity” (see Glossary) to the original model for an evidence-based service/practice increases the likelihood that positive outcomes will be achieved when the model is used by others. Therefore, SAMHSA encourages fidelity to the original evidence-based service/practice to be implemented. However, SAMHSA recognizes that adaptations or modifications to the original model may be necessary for a variety of reasons:

• To allow implementers to use resources efficiently
• To adjust for specific needs of the client population
• To address unique characteristics of the local community where the service/practice will be implemented

All applicants must describe and justify any adaptations or modifications to the proposed service/practice that will be made.

2.2 Services Delivery

SAMHSA's Services Grant funds must be used primarily to support direct services, including the following types of activities:

• Conducting outreach and pre-service strategies to expand access to treatment or prevention services to underserved populations. If you propose to provide only outreach and pre-service strategies, you must show that your organization is an effective and integral part of a network of service providers.
• Purchasing or providing direct treatment (including screening, assessment, and care management) or prevention services for populations at risk. Treatment must be provided in outpatient, day treatment or intensive outpatient, or residential programs.
• Purchasing or providing “wrap-around” services (see Glossary) (e.g., child care, vocational, educational and transportation services) designed to improve access and retention.
• Collecting data using specified tools and standards to measure and monitor treatment or prevention services and costs. (No more than 20% of the total grant award may be used for data collection and evaluation.)

2.3 Infrastructure Development (Maximum 15% of Total Grant Award)

Although SAMHSA expects that its Services Grant funds will be used primarily for direct services, SAMHSA recognizes that infrastructure changes may be needed to support service delivery expansion in some instances. You may use up to 15% of the total Services Grant award for the following types of infrastructure development, if necessary to support the direct service expansion of the grant project.

• Building partnerships to ensure the success of the project and entering into service delivery and other agreements.
• Developing or changing the infrastructure to expand treatment or prevention services.
• Training to assist treatment or prevention providers and community support systems to identify and address mental health or substance abuse issues.

2.4 Data and Performance Measurement

The Government Performance and Results Act of 1993 (P.L.103-62, or “GPRA”) requires all Federal agencies to set program performance targets and report annually on the degree to which the previous year's targets were met.

Agencies are expected to evaluate their programs regularly and to use results of these evaluations to explain their successes and failures and justify requests for funding.

To meet the GPRA requirements, SAMHSA must collect performance data (i.e., “GPRA data”) from grantees. Grantees are required to report these GPRA data to SAMHSA on a timely basis.

Specifically, grantees will be required to provide data on a set of required measures, as specified in the NOFA. The data collection tools to be used for reporting the required data will be provided in the application kits distributed by SAMHSA's clearinghouses and posted on SAMHSA's Web site along with each NOFA. In your application, you must demonstrate your ability to collect and report on these measures, and you may be required to provide some baseline data.

The terms and conditions of the grant award also will specify the data to be submitted and the schedule for submission. Grantees will be required to adhere to these terms and conditions of award.

Applicants should be aware that SAMHSA is working to develop a set of required core performance measures for each of SAMHSA's standard grants (i.e., Services Grants, Infrastructure Grants, Best Practices Planning and Implementation Grants, and Service-to-Science Grants). As this effort proceeds, some of the data collection and reporting requirements included in SAMHSA's NOFAs may change. All grantees will be expected to comply with any changes in data collection requirements that occur during the grantee's project period.

2.5 Grantee Meetings

You must plan to send a minimum of two people (including the Project Director) to at least one joint grantee meeting in each year of the grant, and you must include funding for this travel in your budget. At these meetings, grantees will present the results of their projects and Federal staff will provide technical assistance. Each meeting will be 3 days. These meetings will usually be held in the Washington, DC, area, and attendance is mandatory.

2.6 Evaluation

Grantees must evaluate their projects, and you are required to describe your evaluation plans in your application. The evaluation should be designed to provide regular feedback to the project to improve services. The evaluation must include both process and outcome components. Process and outcome evaluations must measure change relating to project goals and objectives over time compared to baseline information. Control or comparison groups are not required. You must consider your evaluation plan when preparing the project budget.

Process components should address issues such as:

• How closely did implementation match the plan?
• What types of deviation from the plan occurred?
• What led to the deviations?
• What effect did the deviations have on the planned intervention and evaluation?
• Who provided (program, staff) what services (modality, type, intensity, duration), to whom (individual characteristics), in what context (system, community), and at what cost (facilities, personnel, dollars)?

Outcome components should address issues such as:

• What was the effect of treatment on participants?
• What program/contextual factors were associated with outcomes?
• What individual factors were associated with outcomes?
• How durable were the effects?

No more than 20% of the total grant award may be used for evaluation and data collection, including GPRA.

II. Award Information

1. Award Amount

The expected award amount for each funding opportunity will be specified in the NOFA. Typically, SAMHSA's Services Grant awards are expected to be about $500,000 per year in total costs (direct and indirect) for up to 5 years. Awards may range as high as $3.0 million per year in total costs (direct and indirect) for up to 5 years. Regardless of the award amount specified in the NOFA, the actual award amount will depend on the availability of funds.

Proposed budgets cannot exceed the allowable amount specified in the NOFA in any year of the proposed project. Annual continuation awards will depend on the availability of funds, grantee progress in meeting project goals and objectives, and timely submission of required data and reports.

2. Funding Mechanism

The NOFA will indicate whether awards for each funding opportunity will be made as grants or cooperative agreements (see the Glossary in Appendix B for further explanation of these funding mechanisms). For cooperative agreements, the NOFA will describe the nature of Federal involvement in project performance and specify roles and responsibilities of grantees and Federal staff.

III. Eligibility Information

1. Eligible Applicants

Eligible applicants are domestic public and private nonprofit entities. For example, State, local or tribal governments; public or private universities and colleges; community- and faith-based organizations; and tribal organizations may apply. The statutory authority for this program prohibits grants to for-profit organizations. The NOFA will indicate any limitations on eligibility.

2. Cost Sharing

Cost sharing (see Glossary) is not required in this program, and applications will not be screened out on the basis of cost sharing. However, you may include cash or in-kind contributions (see Glossary) in your proposal as evidence of commitment to the proposed project.

3. Other

3.1 Additional Eligibility Requirements

Applications must comply with the following requirements, or they will be screened out and will not be reviewed: Use of the PHS 5161-1 application; application submission requirements in Section IV-3 of this document; and formatting requirements provided in Section IV-2.3 of this document. Applicants should be aware that the NOFA may include additional requirements that, if not met, will result in applications being screened out and returned without review. These requirements will be specified in Section III-3 of the NOFA.

You also must comply with any additional program requirements specified in the NOFA, such as signature of certain officials on the face page of the application and/or required memoranda of understanding with certain signatories.

3.2 Evidence of Experience and Credentials

SAMHSA believes that only existing, experienced, and appropriately credentialed organizations with demonstrated infrastructure and expertise will be able to provide required services quickly and effectively. Therefore, in addition to the basic eligibility requirements specified in this announcement, applicants must meet three additional requirements related to the provision of treatment or prevention services.

The three requirements are:

• A provider organization for direct client services (e.g., substance abuse treatment, substance abuse prevention, mental health services) appropriate to the grant must be involved in each application. The provider may be the applicant or another organization committed to the project. More than one provider organization may be involved;
• Each direct service provider organization must have at least 2 years experience providing services in the geographic area(s) covered by the application, as of the due date of the application; and
• Each direct service provider organization must comply with all applicable local (city, county) and State/tribal licensing, accreditation, and certification requirements, as of the due date of the application.

[Note: The above requirements apply to all service provider organizations. A license from an individual clinician will not be accepted in lieu of a provider organization's license.]

In Appendix 1 of the application, you must: (1) Identify at least one experienced, licensed service provider organization; (2) include a list of all direct service provider organizations that have agreed to participate in the proposed project, including the applicant agency if the applicant is a treatment or prevention service provider organization; and (3) include the Statement of Assurance (provided in Appendix F of this announcement), signed by the authorized representative of the applicant organization identified on the face-page of the application, that all participating service provider organizations:

• Meet the 2-year experience requirement,
• Meet applicable licensing, accreditation, and certification requirements, and
• If the application is within the funding range, will provide the Government Project Officer (GPO) with the required documentation within the time specified.

If Appendix 1 of the application does not contain items (1)-(3), the application will be considered ineligible and will not be reviewed.

In addition, if, following application review, an application's score is within the fundable range for a grant award, the GPO will call the applicant and request that the following documentation be sent by overnight mail:

• A letter of commitment that specifies the nature of the participation and what service(s) will be provided from every service provider organization that has agreed to participate in the project;
• Official documentation that all participating organizations have been providing relevant services for a minimum of 2 years before the date of the application in the area(s) in which the services are to be provided; and
• Official documentation that all participating service provider organizations comply with all applicable local (city, county) and State/tribal requirements for licensing, accreditation, and certification or official documentation from the appropriate agency of the applicable State/tribal, county, or other governmental unit that licensing, accreditation, and certification requirements do not exist.

If the GPO does not receive this documentation within the time specified, the application will be removed from consideration for an award and the funds will be provided to another applicant meeting these requirements.

IV. Application and Submission Information

To ensure that you have met all submission requirements, a checklist is provided for your use in Appendix A of this document.

1. Address To Request Application Package

You may request a complete application kit by calling one of SAMHSA's national clearinghouses:

• For substance abuse prevention or treatment grants, call the National Clearinghouse for Alcohol and Drug Information (NCADI) at 1-800-729-6686.
• For mental health grants, call the National Mental Health Information Center at 1-800-789-CMHS (2647).

You also may download the required documents from the SAMHSA Web site at http://www.samhsa.gov. Click on “grant opportunities.”

Additional materials available on this Web site include:

• A technical assistance manual for potential applicants;
• Standard terms and conditions for SAMHSA grants;
• Guidelines and policies that relate to SAMHSA grants (e.g., guidelines on cultural competence, consumer and family participation, and evaluation); and
• Enhanced instructions for completing the PHS 5161-1 application.

2. Content and Form of Application Submission

2.1 Required Documents

SAMHSA application kits include the following documents:

• PHS 5161-1 (revised July 2000)—Includes the face page, budget forms, assurances, certification, and checklist. Use the PHS 5161-1, unless otherwise specified in the NOFA. Applications that are not submitted on the required application form will be screened out and will not be reviewed.
• Program Announcement (PA)—Includes instructions for the grant application. This document is the PA.
• Notice of Funding Availability (NOFA)—Provides specific information about availability of funds, as well as any exceptions or limitations to provisions in the PA. The NOFAs will be published in the Federal Register, as well as on the Federal grants Web site ( http://www.grants.gov ).

You must use all of the above documents in completing your application.

2.2 Required Application Components

To ensure equitable treatment of all applications, applications must be complete. In order for your application to be complete, it must include the required ten application components (Face Page, Abstract, Table of Contents, Budget Form, Project Narrative and Supporting Documentation, Appendices, Assurances, Certifications, Disclosure of Lobbying Activities, and Checklist).

—Face Page—Use Standard Form (SF) 424, which is part of the PHS 5161-1. [Note: Beginning October 1, 2003, applicants will need to provide a Dun and Bradstreet (DUNS) number to apply for a grant or cooperative agreement from the Federal Government. SAMHSA applicants will be required to provide their DUNS number on the face page of the application. Obtaining a DUNS number is easy and there is no charge. To obtain a DUNS number, access the Dun and Bradstreet Web site at http://www.dunandbradstreet.com or call 1-866-705-5711. To expedite the process, let Dun and Bradstreet know that you are a public/private nonprofit organization getting ready to submit a Federal grant application.]

—Abstract—Your total abstract should not be longer than 35 lines. In the first five lines or less of your abstract, write a summary of your project that can be used, if your project is funded, in publications, reporting to Congress, or press releases.

—Table of Contents—Include page numbers for each of the major sections of your application and for each appendix.

—Budget Form—Use SF 424A, which is part of the PHS 5161-1. Fill out Sections B, C, and E of the SF 424A.

—Project Narrative and Supporting Documentation—The Project Narrative describes your project. It consists of Sections A through E. Sections A-E together may not be longer than 30 pages. More detailed instructions for completing each section of the Project Narrative are provided in “Section V—Application Review Information” of this document.

The Supporting Documentation provides additional information necessary for the review of your application. This supporting documentation should be provided immediately following your Project Narrative in Sections F through I. There are no page limits for these sections, except for Section H, the Biographical Sketches/Job Descriptions.

• Section F—Literature Citations. This section must contain complete citations, including titles and all authors, for any literature you cite in your application.
• Section G—Budget Justification, Existing Resources, Other Support. You must provide a narrative justification of the items included in your proposed budget, as well as a description of existing resources and other support you expect to receive for the proposed project. Be sure to show that no more than 15% of the total grant award will be used for infrastructure development and that no more than 20% of the total grant award will be used for data collection and evaluation, including GPRA.
• Section H—Biographical Sketches and Job Descriptions.
• Include a biographical sketch for the Project Director and other key positions. Each sketch should be 2 pages or less. If the person has not been hired, include a letter of commitment from the individual with a current biographical sketch.
• Include job descriptions for key personnel. Job descriptions should be no longer than 1 page each.
• Sample sketches and job descriptions are listed on page 22, Item 6 in the Program Narrative section of the PHS 5161-1.
• Section I—Confidentiality and SAMHSA Participant Protection/Human Subjects. Section IV-2.4 of this document describes requirements for the protection of the confidentiality, rights and safety of participants in SAMHSA-funded activities. This section also includes guidelines for completing this part of your application.

—Appendices 1 through 5—Use only the appendices listed below. Do not use more than 30 pages for Appendices 1, 3, and 4. There are no page limitations for Appendices 2 and 5. Do not use appendices to extend or replace any of the sections of the Project Narrative unless specifically required in the NOFA. Reviewers will not consider them if you do.

• Appendix 1: Letters of commitment/support. Identification of at least one experienced, licensed service provider organization. A list of all direct service provider organizations that have agreed to participate in the proposed project, including the applicant agency, if it is a treatment or prevention service provider organization. The Statement of Assurance (provided in Appendix F of this announcement) signed by the authorized representative of the applicant organization identified on the face page of the application, that assures SAMHSA that all listed providers meet the 2-year experience requirement, are appropriately licensed, accredited, and certified, and that if the application is within the funding range for an award, the applicant will send the GPO the required documentation within the specified time.
• Appendix 2: Data Collection Instruments/Interview Protocols
• Appendix 3: Sample Consent Forms
• Appendix 4: Letter to the SSA (if applicable; see Section IV-4 of this document)
• Appendix 5: A copy of the State or County Strategic Plan, a State or county needs assessment, or a letter from the State or county indicating that the proposed project addresses a State-or county-identified priority.

—Assurances—Non-Construction Programs. Use Standard Form 424B found in PHS 5161-1. Some applicants will be required to complete the Assurance of Compliance with SAMHSA Charitable Choice Statutes and Regulations Form SMA 170. If this assurance applies to a specific funding opportunity, it will be posted on SAMHSA's Web site with the NOFA and provided in the application kits available at SAMHSA's clearinghouse (NCADI).

—Certifications—Use the “Certifications” forms found in PHS 5161-1.

—Disclosure of Lobbying Activities—Use Standard Form LLL found in the PHS 5161-1. Federal law prohibits the use of appropriated funds for publicity or propaganda purposes, or for the preparation, distribution, or use of the information designed to support or defeat legislation pending before the Congress or State legislatures. This includes “grass roots” lobbying, which consists of appeals to members of the public suggesting that they contact their elected representatives to indicate their support for or opposition to pending legislation or to urge those representatives to vote in a particular way.

—Checklist—Use the Checklist found in PHS 5161-1. The Checklist ensures that you have obtained the proper signatures, assurances and certifications and is the last page of your application.

2.3 Application Formatting Requirements

Applicants also must comply with the following basic application requirements. Applications that do not comply with these requirements will be screened out and will not be reviewed.

—Information provided must be sufficient for review.

—Text must be legible.

• Type size in the Project Narrative cannot exceed an average of 15 characters per inch, as measured on the physical page. (Type size in charts, tables, graphs, and footnotes will not be considered in determining compliance.)
• Text in the Project Narrative cannot exceed 6 lines per vertical inch.

—Paper must be white paper and 8.5 inches by 11.0 inches in size.

—To ensure equity among applications, the amount of space allowed for the Project Narrative cannot be exceeded.

• Applications would meet this requirement by using all margins (left, right, top, bottom) of at least one inch each, and adhering to the 30-page limit for the Project Narrative.
• Should an application not conform to these margin or page limits, SAMHSA will use the following method to determine compliance: The total area of the Project Narrative (excluding margins, but including charts, tables, graphs and footnotes) cannot exceed 58.5 square inches multiplied by 30. This number represents the full page less margins, multiplied by the total number of allowed pages.
• Space will be measured on the physical page. Space left blank within the Project Narrative (excluding margins) is considered part of the Project Narrative, in determining compliance.

—The 30-page limit for Appendices 1, 3 and 4 cannot be exceeded.

To facilitate review of your application, follow these additional guidelines. Failure to adhere to the following guidelines will not, in itself, result in your application being screened out and returned without review. However, following these guidelines will help reviewers to consider your application.

—Pages should be typed single-spaced with one column per page.

—Pages should not have printing on both sides.

—Please use black ink and number pages consecutively from beginning to end so that information can be located easily during review of the application. The cover page should be page 1, the abstract page should be page 2, and the table of contents page should be page 3. Appendices should be labeled and separated from the Project Narrative and budget section, and the pages should be numbered to continue the sequence.

—Send the original application and two copies to the mailing address in Section IV-6.1 of this document. Please do not use staples, paper clips, and fasteners. Nothing should be attached, stapled, folded, or pasted. Do not use heavy or lightweight paper or any material that cannot be copied using automatic copying machines. Odd-sized and oversized attachments such as posters will not be copied or sent to reviewers. Do not include videotapes, audiotapes, or CD-ROMs.

2.4 SAMHSA Confidentiality and Participant Protection Requirements and Protection of Human Subjects Regulations

Applicants must describe procedures relating to Confidentiality, Participant Protection and the Protection of Human Subjects Regulations in Section I of the application, using the guidelines provided below. Problems with confidentiality, participant protection, and protection of human subjects identified during peer review of the application may result in the delay of funding.

Confidentiality and Participant Protection

All applicants must describe how they will address requirements for each of the following elements relating to confidentiality and participant protection.

1. Protect Clients and Staff from Potential Risks

• Identify and describe any foreseeable physical, medical, psychological, social and legal risks or potential adverse effects as a result of the project itself or any data collection activity.
• Describe the procedures you will follow to minimize or protect participants against potential risks, including risks to confidentiality.
• Identify plans to provide guidance and assistance in the event there are adverse effects to participants.
• Where appropriate, describe alternative treatments and procedures that may be beneficial to the participants. If you choose not to use these other beneficial treatments, provide the reasons for not using them.

2. Fair Selection of Participants

• Describe the target population(s) for the proposed project. Include age, gender, and racial/ethnic background and note if the population includes homeless youth, foster children, children of substance abusers, pregnant women, or other targeted groups.
• Explain the reasons for including groups of pregnant women, children, people with mental disabilities, people in institutions, prisoners, and individuals who are likely to be particularly vulnerable to HIV/AIDS.
• Explain the reasons for including or excluding participants.
• Explain how you will recruit and select participants. Identify who will select participants.

3. Absence of Coercion

• Explain if participation in the project is voluntary or required. Identify possible reasons why participation is required, for example, court orders requiring people to participate in a program.
• If you plan to compensate participants, state how participants will be awarded incentives (e.g., money, gifts, etc.).
• State how volunteer participants will be told that they may receive services intervention even if they do not participate in or complete the data collection component of the project.

4. Data Collection

• Identify from whom you will collect data (e.g., from participants themselves, family members, teachers, others). Describe the data collection procedures and specify the sources for obtaining data (e.g., school records, interviews, psychological assessments, questionnaires, observation, or other sources). Where data are to be collected through observational techniques, questionnaires, interviews, or other direct means, describe the data collection setting.
• Identify what type of specimens (e.g., urine, blood) will be used, if any. State if the material will be used just for evaluation or if other use(s) will be made. Also, if needed, describe how the material will be monitored to ensure the safety of participants.
• Provide in Appendix 2, “Data Collection Instruments/Interview Protocols,” copies of all available data collection instruments and interview protocols that you plan to use.

5. Privacy and Confidentiality:

• Explain how you will ensure privacy and confidentiality. Include who will collect data and how it will be collected.
• Describe:
• How you will use data collection instruments.
• Where data will be stored.
• Who will or will not have access to information.
• How the identity of participants will be kept private, for example, through the use of a coding system on data records, limiting access to records, or storing identifiers separately from data.

Note:

If applicable, grantees must agree to maintain the confidentiality of alcohol and drug abuse client records according to the provisions of Title 42 of the Code of Federal Regulations, Part II.

6. Adequate Consent Procedures:

• List what information will be given to people who participate in the project. Include the type and purpose of their participation. Identify the data that will be collected, how the data will be used and how you will keep the data private.
• State:
• Whether or not their participation is voluntary.
• Their right to leave the project at any time without problems.
• Possible risks from participation in the project.
• Plans to protect clients from these risks.
• Explain how you will get consent for youth, the elderly, people with limited reading skills, and people who do not use English as their first language.

Note:

If the project poses potential physical, medical, psychological, legal, social or other risks, you must obtain written informed consent.

• Indicate if you will obtain informed consent from participants or assent from minors along with consent from their parents or legal guardians. Describe how the consent will be documented. For example: Will you read the consent forms? Will you ask prospective participants questions to be sure they understand the forms? Will you give them copies of what they sign?
• Include, as appropriate, sample consent forms that provide for: (1) Informed consent for participation in service intervention; (2) informed consent for participation in the data collection component of the project; and (3) informed consent for the exchange (releasing or requesting) of confidential information. The sample forms must be included in Appendix 3, “Sample Consent Forms”, of your application. If needed, give English translations.

Note:

Never imply that the participant waives or appears to waive any legal rights, may not end involvement with the project, or releases your project or its agents from liability for negligence.

• Describe if separate consents will be obtained for different stages or parts of the project. For example, will they be needed for both participant protection in treatment intervention and for the collection and use of data?
• Additionally, if other consents (e.g., consents to release information to others or gather information from others) will be used in your project, provide a description of the consents. Will individuals who do not consent to having individually identifiable data collected for evaluation purposes be allowed to participate in the project?

7. Risk/Benefit Discussion:

Discuss why the risks are reasonable compared to expected benefits and importance of the knowledge from the project.

Protection of Human Subjects Regulations

Depending on the evaluation and data collection requirements of the particular funding opportunity for which you are applying or the evaluation design you propose in your application, you may have to comply with the Protection of Human Subjects Regulations (45 CFR 46). The NOFA will indicate whether all applicants for a particular funding opportunity must comply with the Protection of Human Subject Regulations.

Applicants must be aware that even if the Protection of Human Subjects Regulations do not apply to all projects funded under a given funding opportunity, the specific evaluation design proposed by the applicant may require compliance with these regulations.

Applicants whose projects must comply with the Protection of Human Subjects Regulations must describe the process for obtaining Institutional Review Board (IRB) approval fully in their applications. While IRB approval is not required at the time of grant award, these applicants will be required, as a condition of award, to provide the documentation that an Assurance of Compliance is on file with the Office for Human Research Protections (OHRP) and the IRB approval has been received prior to enrolling any clients in the proposed project.

Additional information about Protection of Human Subjects Regulations can be obtained on the web at http://ohrp.osophs.dhhs.gov. You may also contact OHRP by e-mail (ohrp@osophs.dhhs.gov) or by phone (301/496-7005).

3. Submission Dates and Times

Deadlines for submission of applications for specific funding opportunities will be published in the NOFAs in the Federal Register and posted on the Federal grants Web site ( http://www.grants.gov ).

Your application must be received by the application deadline. Applications received after this date must have a proof-of-mailing date from the carrier dated at least 1 week prior to the due date. Private metered postmarks are not acceptable as proof of timely mailing.

You will be notified by postal mail that your application has been received.

Applications not received by the application deadline or not postmarked by a week prior to the application deadline will be screened out and will not be reviewed.

4. Intergovernmental Review (E.O. 12372) Requirements

Executive Order 12372, as implemented through Department of Health and Human Services (DHHS) regulation at 45 CFR part 100, sets up a system for State and local review of applications for Federal financial assistance. A current listing of State Single Points of Contact (SPOCs) is included in the application kit and can be downloaded from the Office of Management and Budget (OMB) Web site at http://www.whitehouse.gov/omb/grants/spoc.html.

• Check the list to determine whether your State participates in this program. You do not need to do this if you are a federally recognized Indian tribal government.
• If your State participates, contact your SPOC as early as possible to alert him/her to the prospective application(s) and to receive any necessary instructions on the State's review process.

—For proposed projects serving more than one State, you are advised to contact the SPOC of each affiliated State.

—The SPOC should send any State review process recommendations to the following address within 60 days of the application deadline: Substance Abuse and Mental Health Services Administration, Office of Program Services, Review Branch, 5600 Fishers Lane, Room 17-89, Rockville, Maryland 20857, Attn: SPOC—Funding Announcement No. [fill in pertinent funding opportunity number from the NOFA].

In addition, community-based, non-governmental service providers who are not transmitting their applications through the State must submit a Public Health System Impact Statement (PHSIS) (approved by OMB under control no. 0920-0428; see burden statement below) to the head(s) of appropriate State or local health agencies in the area(s) to be affected no later than the pertinent receipt date for applications. The PHSIS is intended to keep State and local health officials informed of proposed health services grant applications submitted by community-based, non-governmental organizations within their jurisdictions. State and local governments and Indian tribal government applicants are not subject to these requirements.

The PHSIS consists of the following information:

• A copy of the face page of the application (SF 424); and
• A summary of the project, no longer than one page in length, that provides: (1) A description of the population to be served, (2) a summary of the services to be provided, and (3) a description of the coordination planned with appropriate State or local health agencies.

For SAMHSA grants, the appropriate State agencies are the Single State Agencies (SSAs) for substance abuse and mental health. A listing of the SSAs can be found on SAMHSA's Web site at http://www.samhsa.gov. If the proposed project falls within the jurisdiction of more than one State, you should notify all representative SSAs.

Applicants who are not the SSA must include a copy of a letter transmitting the PHSIS to the SSA in Appendix 4, “Letter to the SSA.” The letter must notify the State that, if it wishes to comment on the proposal, its comments should be sent not later than 60 days after the application deadline to: Substance Abuse and Mental Health Services Administration, Office of Program Services, Review Branch, 5600 Fishers Lane, Room 17-89, Rockville, Maryland 20857. Attn: SSA—Funding Announcement No. [fill in pertinent funding opportunity number from NOFA].

In addition:

• Applicants may request that the SSA send them a copy of any State comments.
• The applicant must notify the SSA within 30 days of receipt of an award.

[Public reporting burden for the Public Health System Reporting Requirement is estimated to average 10 minutes per response, including the time for copying the face page of SF 424 and the abstract and preparing the letter for mailing. An agency may not conduct or sponsor, and a person is not required to respond to, a collection of information unless it displays a currently valid OMB control number. The OMB control number for this project is 0920-0428. Send comments regarding this burden to CDC Clearance Officer, 1600 Clifton Road, MS D-24, Atlanta, GA 30333, Attn: PRA (0920-0428).]

5. Funding Limitations/Restrictions

Cost principles describing allowable and unallowable expenditures for Federal grantees, including SAMHSA grantees, are provided in the following documents:

• Institutions of Higher Education: OMB Circular A-21
• State and Local Governments: OMB Circular A-87
• Nonprofit Organizations: OMB Circular A-122
• Appendix E Hospitals: 45 CFR Part 74

In addition, SAMHSA Services Grant recipients must comply with the following funding restrictions:

• No more than 15% of the total grant award may be used for developing the infrastructure necessary for expansion of services.
• No more than 20% of the total grant award may be used for evaluation and data collection, including GPRA.

Service Grant funds must be used for purposes supported by the program and may not be used to:

• Pay for any lease beyond the project period.
• Provide services to incarcerated populations (defined as those persons in jail, prison, detention facilities, or in custody where they are not free to move about in the community).
• Pay for the purchase or construction of any building or structure to house any part of the program. (Applicants may request up to $75,000 for renovations and alterations of existing facilities, if necessary and appropriate to the project.)
• Provide residential or outpatient treatment services when the facility has not yet been acquired, sited, approved, and met all requirements for human habitation and services provision. (Expansion or enhancement of existing residential services is permissible.)
• Pay for housing other than residential mental health and/or substance abuse treatment.
• Provide inpatient treatment or hospital-based detoxification services. Residential services are not considered to be inpatient or hospital-based services.
• Pay for incentives to induce individuals to enter treatment. However, a grantee or treatment provider may provide up to $20 or equivalent (coupons, bus tokens, gifts, child care, and vouchers) to individuals as incentives to participate in required data collection follow-up. This amount may be paid for participation in each required interview.
• Implement syringe exchange programs, such as the purchase and distribution of syringes and/or needles.
• Pay for pharmacologies for HIV antiretroviral therapy, sexually transmitted diseases (STD)/sexually transmitted illnesses (STI), TB, and hepatitis B and C, or for psychotropic drugs.

6. Other Submission Requirements

6.1 Where To Send Applications

Send applications to the following address: Substance Abuse and Mental Health Services Administration, Office of Program Services, Review Branch, 5600 Fishers Lane, Room 17-89, Rockville, Maryland 20857.

Be sure to include the funding announcement number from the NOFA in item number 10 on the face page of the application. If you require a phone number for delivery, you may use (301) 443-4266.

6.2 How To Send Applications

Mail an original application and 2 copies (including appendices) to the mailing address provided above. The original and copies must not be bound. Do not use staples, paper clips, or fasteners. Nothing should be attached, stapled, folded, or pasted.

You must use a recognized commercial or governmental carrier. Hand carried applications will not be accepted. Faxed or e-mailed applications will not be accepted.

V. Application Review Information

1. Evaluation Criteria

Your application will be reviewed and scored according to the quality of your response to the requirements listed below for developing the Project Narrative (Sections A-E). These sections describe what you intend to do with your project.

• In developing the Project Narrative section of your application, use these instructions, which have been tailored to this program. These are to be used instead of the “Program Narrative” instructions found in the PHS 5161-1.
• The Project Narrative (Sections A-E) together may be no longer than 30 pages.
• You must use the five sections/headings listed below in developing your Project Narrative. Be sure to place the required information in the correct section, or it will not be considered. Your application will be scored according to how well you address the requirements for each section of the Project Narrative.
• Reviewers will be looking for evidence of cultural competence in each section of the Project Narrative. Points will be assigned based on how well you address the cultural competence aspects of the evaluation criteria. SAMHSA's guidelines for cultural competence can be found on the SAMHSA Web site at http://www.samhsa.gov. Click on “Grant Opportunities.”
• The Supporting Documentation you provide in Sections F-I and Appendices 1-5 will be considered by reviewers in assessing your response, along with the material in the Project Narrative.
• The number of points after each heading is the maximum number of points a review committee may assign to that section of your Project Narrative. Bullet statements in each section do not have points assigned to them. They are provided to invite the attention of applicants and reviewers to important areas within the criterion.

Section A: Statement of Need (10 points)

• Describe the target population (see Glossary) as well as the geographic area to be served, and justify the selection of both. Include the numbers to be served and demographic information. Discuss the target population's language, beliefs, norms and values, as well as socioeconomic factors that must be considered in delivering programs to this population.
• Describe the nature of the problem and extent of the need for the target population based on data. The statement of need should include a clearly established baseline for the project. Documentation of need may come from a variety of qualitative and quantitative sources. The quantitative data could come from local data or trend analyses, State data (e.g., from State Needs Assessments), and/or national data (e.g., from SAMHSA's National Household Survey on Drug Abuse and Health or from National Center for Health Statistics/Centers for Disease Control reports). For data sources that are not well known, provide sufficient information on how the data were collected so reviewers can assess the reliability and validity of the data.
• Non-tribal applicants must show that identified needs are consistent with priorities of the State or county that has primary responsibility for the service delivery system. Include, in Appendix 5, a copy of the State or County Strategic Plan, a State or county needs assessment, or a letter from the State or county indicating that the proposed project addresses a State- or county-identified priority. Tribal applicants must provide similar documentation relating to tribal priorities.
• Check the NOFA for any additional requirements.

Section B: Proposed Evidence-Based Service/Practice (30 points)

• Clearly state the purpose, goals and objectives of your proposed project. Describe how achievement of goals will produce meaningful and relevant results (e.g., increase access, availability, prevention, outreach, pre-services, treatment, and/or intervention).
• Identify the evidenced based service/practice that you propose to implement. Describe the evidence-base for the proposed service/practice and show that it incorporates the best objective information available regarding effectiveness and acceptability. Follow the instructions provided in #1, #2 or #3 below, as appropriate:

1. If you are proposing to implement a service/practice included in NREP (see Appendix C), one of the CMHS tool-kits on evidence-based practices (see Appendix D), the list of Effective Substance Abuse Treatment Practices (see Appendix E), or the NOFA (if applicable), simply identify the practice and state the source from which it was selected. You do not need to provide further evidence of effectiveness.

2. If you are providing evidence that includes scientific studies published in the peer-reviewed literature or other studies that have not been published, describe the extent to which:

—The service/practice has been evaluated and the quality of the evaluation studies (e.g., whether they are descriptive, quasi-experimental studies, or experimental studies)

—The services/practice has demonstrated positive outcomes and for what populations the positive outcomes have been demonstrated

—The service/practice has been documented (e.g., through development of guidelines, tool kits, treatment protocols, and/or manuals) and replicated

—Fidelity measures have been developed (e.g., no measures developed, key components identified, or fidelity measures developed)

3. If you are providing evidence based on a formal consensus process involving recognized experts in the field, describe:

—The experts involved in developing consensus on the proposed service/practice (e.g., members of an expert panel formally convened by SAMHSA, NIH, the Institute of Medicine or other nationally recognized organization). The consensus must have been developed by a group of experts whose work is recognized and respected by others in the field. Local recognition of an individual as a respected or influential person at the community level is not considered a “recognized expert” for this purpose.

—The nature of the consensus that has been reached and the process used to reach consensus

—The extent to which the consensus has been documented (e.g., in a consensus panel report, meeting minutes, or an accepted standard practice in the field)

—Any empirical evidence (whether formally published or not) supporting the effectiveness of the proposed service/practice

—The rationale for concluding that further empirical evidence does not exist to support the effectiveness of the proposed service/practice

• Justify the use of the proposed service/practice for the target population. Describe and justify any adaptations necessary to meet the needs of the target population as well as evidence that such adaptations will be effective for the target population.
• Identify and justify any additional adaptations or modifications to the proposed service/practice.
• Describe how the proposed project will address issues of age, race, ethnicity, culture, language, sexual orientation, disability, literacy, and gender in the target population, while retaining fidelity to the chosen practice.
• Demonstrate how the proposed service/practice will meet your goals and objectives. Provide a logic model (see Glossary) that links need, the services or practice to be implemented, and outcomes.
• Check the NOFA for any additional requirements.

Section C: Proposed Implementation Approach (25 points)

• Describe how the proposed service or practice will be implemented. Provide a realistic time line for the project (chart or graph) showing key activities, milestones, and responsible staff. [Note: The time line should be part of the Project Narrative. It should not be placed in an appendix.]
• Clearly state the unduplicated number of individuals you propose to serve (annually and over the entire project period) with grant funds, including the types and numbers of services to be provided and anticipated outcomes. Describe how the target population will be identified, recruited, and retained.
• Describe how members of the target population helped prepare the application, and how they will help plan, implement, and evaluate the project.
• Describe how the project components will be embedded within the existing service delivery system, including other SAMHSA-funded projects, if applicable. Identify any other organizations that will participate in the proposed project. Describe their roles and responsibilities and demonstrate their commitment to the project. Include letters of commitment from community organizations supporting the project in Appendix 1. Identify any cash or in-kind contributions that will be made to the project by the applicant or other partnering organizations.
• Show that the necessary groundwork (e.g., planning, consensus development, development of memoranda of agreement, identification of potential facilities) has been completed or is near completion so that the project can be implemented and service delivery can begin as soon as possible and no later than 4 months after grant award.
• Describe the potential barriers to successful conduct of the proposed project and how you will overcome them.
• Provide a plan to secure resources to sustain the proposed project when Federal funding ends.
• Check the NOFA for any additional requirements.

Section D: Staff and Organizational Experience (20 points)

• Discuss the capability and experience of the applicant organization and other participating organizations with similar projects and populations, including experience in providing culturally appropriate/competent services.
• Provide a list of staff who will participate in the project, showing the role of each and their level of effort and qualifications. Include the Project Director and other key personnel, such as the evaluator and treatment/prevention personnel.
• Describe the racial/ethnic characteristics of key staff and indicate if any are members of the target population/community. If the target population is multi-linguistic, indicate if the staffing pattern includes bilingual and bicultural individuals.
• Describe the resources available for the proposed project (e.g., facilities, equipment), and provide evidence that services will be provided in a location that is adequate, accessible, compliant with the Americans with Disabilities Act (ADA), and amenable to the target population.
• Check the NOFA for any additional requirements.

Section E: Evaluation and Data (15 points)

• Document your ability to collect and report on the required performance measures as specified in the NOFA. Specify and justify any additional measures you plan to use for your grant project.
• Describe plans for data collection, management, analysis, interpretation and reporting. Describe the existing approach to the collection of data, along with any necessary modifications. Be sure to include data collection instruments/interview protocols in Appendix 2.
• Discuss the reliability and validity of evaluation methods and instrument(s) in terms of the gender/age/culture of the target population.
• Describe the process and outcome evaluation, including assessments of implementation and individual outcomes. Show how the evaluation will be integrated with requirements for collection and reporting of performance data, including data required by SAMHSA to meet GPRA requirements.
• Describe how the evaluation will be used to ensure the fidelity to the practice.
• Provide a per-person or unit cost of the project to be implemented, based on the applicant's actual costs and projected costs over the life of the project.
• Check the NOFA for any additional requirements.

Note:

Although the budget for the proposed project is not a review criterion, the Review Group will be asked to comment on the appropriateness of the budget after the merits of the application have been considered.

2. Review and Selection Process

SAMHSA applications are peer-reviewed according to the review criteria listed above. For those programs where the individual award is over $100,000, applications must also be reviewed by the appropriate National Advisory Council.

Decisions to fund a grant are based on:

• The strengths and weaknesses of the application as identified by peer reviewers and, when applicable, approved by the appropriate National Advisory Council;
• Availability of funds;
• Equitable distribution of awards in terms of geography (including urban, rural and remote settings) and balance among target populations and program size; and
• After applying the aforementioned criteria, the following method for breaking ties: When funds are not available to fund all applications with identical scores, SAMHSA will make award decisions based on the application(s) that received the greatest number of points by peer reviewers on the evaluation criterion in Section V-1 with the highest number of possible points (Proposed Evidence-Based Service/Practice—30 points). Should a tie still exist, the evaluation criterion with the next highest possible point value will be used, continuing sequentially to the evaluation criterion with the lowest possible point value, should that be necessary to break all ties. If an evaluation criterion to be used for this purpose has the same number of possible points as another evaluation criterion, the criterion listed first in Section V-1 will be used first.

VI. Award Administration Information

1. Award Notices

After your application has been reviewed, you will receive a letter from SAMHSA through postal mail that describes the general results of the review, including the score that your application received.

If you are approved for funding, you will receive an additional notice, the Notice of Grant Award, signed by SAMHSA's Grants Management Officer. The Notice of Grant Award is the sole obligating document that allows the grantee to receive Federal funding for work on the grant project. It is sent by postal mail and is addressed to the contact person listed on the face page of the application.

If you are not funded, you can re-apply if there is another receipt date for the program.

2. Administrative and National Policy Requirements

• You must comply with all terms and conditions of the grant award. SAMHSA's standard terms and conditions are available on the SAMHSA Web site at http://www.samhsa.gov/grants/2004/useful_info.asp.
• Depending on the nature of the specific funding opportunity and/or the proposed project as identified during review, additional terms and conditions may be identified in the NOFA or negotiated with the grantee prior to grant award. These may include, for example:
• Actions required to be in compliance with human subjects requirements;
• Requirements relating to additional data collection and reporting;
• Requirements relating to participation in a cross-site evaluation; or
• Requirements to address problems identified in review of the application.
• You will be held accountable for the information provided in the application relating to performance targets. SAMHSA program officials will consider your progress in meeting goals and objectives, as well as your failures and strategies for overcoming them, when making an annual recommendation to continue the grant and the amount of any continuation award. Failure to meet stated goals and objectives may result in suspension or termination of the grant award, or in reduction or withholding of continuation awards.
• In an effort to improve access to funding opportunities for applicants, SAMHSA is participating in the U.S. Department of Health and Human Services “Survey on Ensuring Equal Opportunity for Applicants.” This survey is included in the application kit for SAMHSA grants. Applicants are encouraged to complete the survey and return it, using the instructions provided on the survey form.

3. Reporting Requirements

3.1 Progress and Financial Reports

• Grantees must provide annual and final progress reports. The final report must summarize information from the annual reports, describe the accomplishments of the project, and describe next steps for implementing plans developed during the grant period.
• Grantees must provide annual and final financial status reports. These reports may be included as separate sections of annual and final progress reports or can be separate documents. Because SAMHSA is extremely interested in ensuring that treatment or prevention services can be sustained, your financial reports should explain plans to ensure the sustainability (see Glossary) of efforts initiated under this grant. Initial plans for sustainability should be described in year 01. In each subsequent year, you should describe the status of your project, as well as the successes achieved and obstacles encountered in that year.
• SAMHSA will provide guidelines and requirements for these reports to grantees at the time of award and at the initial grantee orientation meeting after award. SAMHSA staff will use the information contained in the reports to determine the grantee's progress toward meeting its goals.

3.2 Government Performance and Results Act (GPRA)

The Government Performance and Results Act (GPRA) mandates accountability and performance-based management by Federal agencies. To meet the GPRA requirements, SAMHSA must collect performance data (i.e., “GPRA data”) from grantees. These requirements will be specified in the NOFA for each funding opportunity.

3.3 Publications

If you are funded under this grant program, you are required to notify the Government Project Officer (GPO) and SAMHSA's Publications Clearance Officer (301-443-8596) of any materials based on the SAMHSA-funded grant project that are accepted for publication.

In addition, SAMHSA requests that grantees:

• Provide the GPO and SAMHSA Publications Clearance Officer with advance copies of publications.
• Include acknowledgment of the SAMHSA grant program as the source of funding for the project.

—Include a disclaimer stating that the views and opinions contained in the publication do not necessarily reflect those of SAMHSA or the U.S. Department of Health and Human Services, and should not be construed as such.

SAMHSA reserves the right to issue a press release about any publication deemed by SAMHSA to contain information of program or policy significance to the substance abuse treatment/substance abuse prevention/mental health services community.

VII. Agency Contacts

The NOFAs provide contact information for questions about program issues.

For questions on grants management issues, contact:

Gwendolyn Simpson (CMHS), Office of Program Services, Division of Grants Management, Substance Abuse and Mental Health Services Administration, 5600 Fishers Lane, Room 13-103, Rockville, MD 20857, (301) 443-4456, gsimpson@samhsa.gov.

Edna Frazier (CSAP), Office of Program Services, Division of Grants Management, Substance Abuse and Mental Health Services Administration, 5600 Fishers Lane, Rockwall II, Suite 630, Rockville, MD 20857, (301) 443-6816, efrazier@samhsa.gov.

Kathleen Sample (CSAT), Office of Program Services, Division of Grants Management, Substance Abuse and Mental Health Services Administration, 5600 Fishers Lane, Rockwall II, Suite 630, Rockville, MD 20857, (301) 443-9667, ksample@samhsa.gov.

Appendix A—Checklist for Formatting Requirements and Screenout Criteria for SAMHSA Grant Applications

SAMHSA's goal is to review all applications submitted for grant funding. However, this goal must be balanced against SAMHSA's obligation to ensure equitable treatment of applications. For this reason, SAMHSA has established certain formatting requirements for its applications. If you do not adhere to these requirements, your application will be screened out and returned to you without review. In addition to these formatting requirements, programmatic requirements (e.g., relating to eligibility) may be stated in the specific NOFA and in Section III of the standard grant announcement. Please check the entire NOFA and Section III of the standard grant announcement before preparing your application.

—Use the PHS 5161-1 application.

—Applications must be received by the application deadline. Applications received after this date must have a proof of mailing date from the carrier dated at least 1 week prior to the due date. Private metered postmarks are not acceptable as proof of timely mailing. Applications not received by the application deadline or not postmarked at least 1 week prior to the application deadline will not be reviewed.

—Information provided must be sufficient for review.

—Text must be legible.

• Type size in the Project Narrative cannot exceed an average of 15 characters per inch, as measured on the physical page. (Type size in charts, tables, graphs, and footnotes will not be considered in determining compliance.)
• Text in the Project Narrative cannot exceed 6 lines per vertical inch.

—Paper must be white paper and 8.5 inches by 11.0 inches in size.

—To ensure equity among applications, the amount of space allowed for the Project Narrative cannot be exceeded.

• Applications would meet this requirement by using all margins (left, right, top, bottom) of at least one inch each, and adhering to the page limit for the Project Narrative stated in the specific funding announcement.
• Should an application not conform to these margin or page limits, SAMHSA will use the following method to determine compliance: The total area of the Project Narrative (excluding margins, but including charts, tables, graphs and footnotes) cannot exceed 58.5 square inches multiplied by the page limit. This number represents the full page less margins, multiplied by the total number of allowed pages.
• Space will be measured on the physical page. Space left blank within the Project Narrative (excluding margins) is considered part of the Project Narrative, in determining compliance.

—The page limit for Appendices stated in the specific funding announcement cannot be exceeded.

To facilitate review of your application, follow these additional guidelines. Failure to adhere to the following guidelines will not, in itself, result in your application being screened out and returned without review. However, the information provided in your application must be sufficient for review. Following these guidelines will help ensure your application is complete, and will help reviewers to consider your application.

—The 10 application components required for SAMHSA applications should be included. These are:

• Face Page (Standard Form 424, which is in PHS 5161-1).
• Abstract.
• Table of Contents.
• Budget Form (Standard Form 424A, which is in PHS 5161-1).
• Project Narrative and Supporting Documentation.
• Appendices.
• Assurances (Standard Form 424B, which is in PHS 5161-1).
• Certifications (a form in PHS 5161-1).
• Disclosure of Lobbying Activities (Standard Form LLL, which is in PHS 5161-1).
• Checklist (a form in PHS 5161-1).

—Applications should comply with the following requirements:

• Provisions relating to confidentiality, participant protection and the protection of human subjects specified in Section IV-2.4 of the FY 2004 standard funding announcements.
• Budgetary limitations as specified in Section I, II, and IV-5 of the FY 2004 standard funding announcements.
• Documentation of nonprofit status as required in the PHS 5161-1.

—Pages should be typed single-spaced with one column per page.

—Pages should not have printing on both sides.

—Please use black ink and number pages consecutively from beginning to end so that information can be located easily during review of the application. The cover page should be page 1, the abstract page should be page 2, and the table of contents page should be page 3. Appendices should be labeled and separated from the Project Narrative and budget section, and the pages should be numbered to continue the sequence.

—Send the original application and two copies to the mailing address in the funding announcement. Please do not use staples, paper clips, and fasteners. Nothing should be attached, stapled, folded, or pasted. Do not use heavy or lightweight paper or any material that cannot be copied using automatic copying machines. Odd-sized and oversized attachments such as posters will not be copied or sent to reviewers. Do not include videotapes, audiotapes, or CD-ROMs.

Appendix B—Glossary

Best Practice: Best practices are practices that incorporate the best objective information currently available regarding effectiveness and acceptability.

Catchment Area: A catchment area is the geographic area from which the target population to be served by a program will be drawn.

Cooperative Agreement: A cooperative agreement is a form of Federal grant. Cooperative agreements are distinguished from other grants in that, under a cooperative agreement, substantial involvement is anticipated between the awarding office and the recipient during performance of the funded activity. This involvement may include collaboration, participation, or intervention in the activity. HHS awarding offices use grants or cooperative agreements (rather than contracts) when the principal purpose of the transaction is the transfer of money, property, services, or anything of value to accomplish a public purpose of support or stimulation authorized by Federal statute. The primary beneficiary under a grant or cooperative agreement is the public, as opposed to the Federal Government.

Cost Sharing or Matching: Cost sharing refers to the value of allowable non-Federal contributions toward the allowable costs of a Federal grant project or program. Such contributions may be cash or in-kind contributions. For SAMHSA grants, cost sharing or matching is not required, and applications will not be screened out on the basis of cost sharing. However, applicants often include cash or in-kind contributions in their proposals as evidence of commitment to the proposed project. This is allowed, and this information may be considered by reviewers in evaluating the quality of the application.

Fidelity: Fidelity is the degree to which a specific implementation of a program or practice resembles, adheres to, or is faithful to the evidence-based model on which it is based. Fidelity is formally assessed using rating scales of the major elements of the evidence-based model. A toolkit on how to develop and use fidelity instruments is available from the SAMHSA-funded Evaluation Technical Assistance Center at http://tecathsri.org or by calling (617) 876-0426.

Grant: A grant is the funding mechanism used by the Federal Government when the principal purpose of the transaction is the transfer of money, property, services, or anything of value to accomplish a public purpose of support or stimulation authorized by Federal statute. The primary beneficiary under a grant or cooperative agreement is the public, as opposed to the Federal Government.

In-Kind Contribution: In-kind contributions toward a grant project are non-cash contributions (e.g., facilities, space, services) that are derived from non-Federal sources, such as State or sub-State non-Federal revenues, foundation grants, or contributions from other non-Federal public or private entities.

Logic Model: A logic model is a diagrammatic representation of a theoretical framework. A logic model describes the logical linkages among program resources, conditions, strategies, short-term outcomes, and long-term impact. More information on how to develop logics models and examples can be found through the resources listed in Appendix G.

Practice: A practice is any activity, or collective set of activities, intended to improve outcomes for people with or at risk for substance abuse and/or mental illness. Such activities may include direct service provision, or they may be supportive activities, such as efforts to improve access to and retention in services, organizational efficiency or effectiveness, community readiness, collaboration among stakeholder groups, education, awareness, training, or any other activity that is designed to improve outcomes for people with or at risk for substance abuse or mental illness.

Practice Support System: This term refers to contextual factors that affect practice delivery and effectiveness in the pre-adoption phase, delivery phase, and post-delivery phase, such as (a) community collaboration and consensus building, (b) training and overall readiness of those implementing the practice, and (c) sufficient ongoing supervision for those implementing the practice.

Stakeholder: A stakeholder is an individual, organization, constituent group, or other entity that has an interest in and will be affected by a proposed grant project.

Sustainability: Sustainability is the ability to continue a program or practice after SAMHSA grant funding has ended.

Target Population: The target population is the specific population of people whom a particular program or practice is designed to serve or reach.

Wraparound Service: Wraparound services are non-clinical supportive services—such as child care, vocational, educational, and transportation services—that are designed to improve the individual's access to and retention in the proposed project.

Appendix C—National Registry of Effective Programs

To help SAMHSA's constituents learn more about science-based programs, SAMHSA's Center for Substance Abuse Prevention (CSAP) created a National Registry of Effective Programs (NREP) to review and identify effective programs. NREP seeks candidates from the practice community and the scientific literature. While the initial focus of NREP was substance abuse prevention programming, NREP has expanded its scope and now includes prevention and treatment of substance abuse and of co-occurring substance abuse and mental disorders, and psychopharmacological programs and workplace programs.

NREP includes three categories of programs: Effective Programs, Promising Programs, and Model Programs. Programs defined as Effective have the option of becoming Model Programs if their developers choose to take part in SAMHSA dissemination efforts. The conditions for making that choice, together with definitions of the three major criteria, are as follows.

Promising Programs have been implemented and evaluated sufficiently and are scientifically defensible. They have positive outcomes in preventing substance abuse and related behaviors. However, they have not yet been shown to have sufficient rigor and/or consistently positive outcomes required for Effective Program status. Nonetheless, Promising Programs are eligible to be elevated to Effective/Model status after review of additional documentation regarding program effectiveness. Originated from a range of settings and spanning target populations, Promising Programs can guide prevention, treatment, and rehabilitation.

Effective Programs are well-implemented, well-evaluated programs that produce consistently positive pattern of results (across domains and/or replications). Developers of Effective Programs have yet to help SAMHSA/CSAP disseminate their programs, but may do so themselves.

Model Programs are also well-implemented, well-evaluated programs, meaning they have been reviewed by NREP according to rigorous standards of research. Their developers have agreed with SAMHSA to provide materials, training, and technical assistance for nationwide implementation. That helps ensure the program is carefully implemented and likely to succeed.

Programs that have met the NREP standards for each category can be identified by accessing the NREP Model Programs Web site at http://www.modelprograms.samhsa.gov.

Appendix D—Center for Mental Health Services Evidence-Based Practice Toolkits

SAMHSA's Center for Mental Health Services and the Robert Wood Johnson Foundation initiated the Evidence-Based Practices Project to: (1) Help more consumers and families access services that are effective, (2) help providers of mental health services develop effective services, and (3) help administrators support and maintain these services. The project is now also funded and endorsed by numerous national, State, local, private and public organizations, including the Johnson & Johnson Charitable Trust, the MacArthur Foundation, and the West Family Foundation.

The project has been developed through the cooperation of many Federal and State mental health organizations, advocacy groups, mental health providers, researchers, consumers and family members. A Web site ( http://www.mentalhealthpractices.org ) was created as part of Phase I of the project, which included the identification of the first cluster of evidence-based practices and the design of implementation resource kits to help people understand and use these practices successfully.

Basic information about the first six evidence-based practices is available on the Web site. The six practices are:

1. Illness Management and Recovery

2. Family Psychoeducation

3. Medication Management Approaches in Psychiatry

4. Assertive Community Treatment

5. Supported Employment

6. Integrated Dual Disorders Treatment

Each of the resource kits contains information and materials written by and for the following groups:

— Consumers

— Families and Other Supporters

— Practitioners and Clinical Supervisors

— Mental Health Program Leaders

— Public Mental Health Authorities

Material on the Web site can be printed or downloaded with Acrobat Reader, and references are provided where additional information can be obtained.

Once published, the full kits will be available from National Mental Health Information Center at http://www.health.org or 1-800-789-CMHS (2647).

Appendix E—Effective Substance Abuse Treatment Practices

To assist potential applicants, SAMHSA's Center for Substance Abuse Treatment (CSAT) has identified the following listing of current publications on effective treatment practices for use by treatment professionals in treating individuals with substance abuse disorders. These publications are available from the National Clearinghouse for Alcohol and Drug Information (NCADI); Tele: 1-800-729-6686 or http://www.health.org and http://www.samhsa.gov/centers/csat2002/publications.html.

CSAT Treatment Improvement Protocols (TIPs) are consensus-based guidelines developed by clinical, research, and administrative experts in the field.

• Integrating Substance Abuse Treatment and Vocational Services. TIP 38 (2000) NCADI # BKD381
• Substance Abuse Treatment for Persons with Child Abuse and Neglect Issues. TIP 36 (2000) NCADI # BKD343
• Substance Abuse Treatment for Persons with HIV/AIDS. TIP 37 (2000) NCADI # BKD359
• Brief Interventions and Brief Therapies for Substance Abuse. TIP 34 (1999) NCADI # BKD341
• Enhancing Motivation for Change in Substance Abuse Treatment. TIP 35 (1999) NCADI # BKD342
• Screening and Assessing Adolescents for Substance Use Disorders. TIP 31 (1999) NCADI # BKD306
• Treatment for Stimulant Use Disorders. TIP 33 (1999) NCADI # BKD289
• Treatment of Adolescents with Substance Use Disorders. TIP 32 (1999) NCADI # BKD307
• Comprehensive Case Management for Substance Abuse Treatment. TIP 27 (1998) NCADI # BKD251
• Continuity of Offender Treatment for Substance Use Disorders From Institution to Community. TIP 30 (1998) NCADI # BKD304
• Naltrexone and Alcoholism Treatment. TIP 28 (1998) NCADI # BKD268
• Substance Abuse Among Older Adults. TIP 26 (1998) NCADI # BKD250
• Substance Use Disorder Treatment for People With Physical and Cognitive Disabilities. TIP 29 (1998) NCADI # BKD288
• A Guide to Substance Abuse Services for Primary Care Clinicians. TIP 24 (1997) NCADI # BKD234
• Substance Abuse Treatment and Domestic Violence. TIP 25 (1997) NCADI # BKD239
• Treatment Drug Courts: Integrating Substance Abuse Treatment With Legal Case Processing. TIP 23 (1996) NCADI # BKD205
• Alcohol and Other Drug Screening of Hospitalized Trauma Patients. TIP 16 (1995) NCADI # BKD164
• Combining Alcohol and Other Drug Abuse Treatment With Diversion for Juveniles in the Justice System. TIP 21 (1995) NCADI # BKD169
• Detoxification From Alcohol and Other Drugs. TIP 19 (1995) NCADI # BKD172
• LAAM in the Treatment of Opiate Addiction. TIP 22 (1995) NCADI # BKD170
• Matching Treatment to Patient Needs in Opioid Substitution Therapy. TIP 20 (1995) NCADI # BKD168
• Planning for Alcohol and Other Drug Abuse Treatment for Adults in the Criminal Justice System. TIP 17 (1995) NCADI # BKD165
• Assessment and Treatment of Cocaine-Abusing Methadone-Maintained Patients. TIP 10 (1994) NCADI # BKD157
• Assessment and Treatment of Patients With Coexisting Mental Illness and Alcohol and Other Drug Abuse. TIP 9 (1994) NCADI # BKD134
• Intensive Outpatient Treatment for Alcohol and Other Drug Abuse. TIP 8 (1994) NCADI # BKD139

Other Effective Practice Publications

CSAT Publications—

• Anger Management for Substance Abuse and Mental Health Clients: A Cognitive Behavioral Therapy Manual (2002) NCADI # BKD444
• Anger Management for Substance Abuse and Mental Health Clients: Participant Workbook (2002) NCADI # BKD445
• Multidimensional Family Therapy for Adolescent Cannabis Users. CYT Cannabis Youth Treatment Series Vol. 5 (2002) NCADI # BKD388
• Navigating the Pathways: Lessons and Promising Practices in Linking Alcohol and Drug Services with Child Welfare. TAP 27 (2002) NCADI # BKD436
• The Motivational Enhancement Therapy and Cognitive Behavioral Therapy Supplement: 7 Sessions of Cognitive Behavioral Therapy for Adolescent Cannabis Users. CYT Cannabis Youth Treatment Series Vol. 2 (2002) NCADI # BKD385
• Family Support Network for Adolescent Cannabis Users. CYT Cannabis Youth Treatment Series Vol. 3 (2001) NCADI # BKD386
• Identifying Substance Abuse Among TANF-Eligible Families. TAP 26 (2001) NCADI # BKD410
• Motivational Enhancement Therapy and Cognitive Behavioral Therapy for Adolescent Cannabis Users: 5 Sessions. CYT Cannabis Youth Treatment Series Vol. 1 (2001) NCADI # BKD384
• The Adolescent Community Reinforcement Approach for Adolescent Cannabis Users. CYT Cannabis Youth Treatment Series Vol. 4 (2001) NCADI # BKD387
• Substance Abuse Treatment for Women Offenders: Guide to Promising Practices. TAP 23 (1999) NCADI # BKD310
• Addiction Counseling Competencies: The Knowledge, Skills, and Attitudes of Professional Practice. TAP 21 (1998) NCADI # BKD246
• Bringing Excellence to Substance Abuse Services in Rural and Frontier America. TAP 20 (1997) NCADI # BKD220
• Counselor's Manual for Relapse Prevention with Chemically Dependent Criminal Offenders. TAP 19 (1996) NCADI # BKD723
• Draft Buprenorphine Curriculum for Physicians (Note: the Curriculum is in DRAFT form and is currently being updated) http://www.buprenorphine.samhsa.gov
• CSAT Guidelines for the Accreditation of Opioid Treatment Programs http://www.samhsa.gov/centers/csat/content/dpt/accreditation.htm
• Model Policy Guidelines for Opioid Addiction Treatment in the Medical Office http://www.samhsa.gov/centers/csat/content/dpt/model_policy.htm

NIDA Manuals—Available Through NCADI

• Brief Strategic Family Therapy. Manual 5 (2003) NCADI # BKD481
• Drug Counseling for Cocaine Addiction: The Collaborative Cocaine Treatment Study Model. Manual 4 (2002) NCADI # BKD465
• The NIDA Community-Based Outreach Model: A Manual to Reduce Risk HIV and Other Blood-Borne Infections in Drug Users. (2000) NCADI # BKD366
• An Individual Counseling Approach to Treat Cocaine Addiction: The Collaborative Cocaine Treatment Study Model. Manual 3 (1999) NCADI # BKD337
• Cognitive-Behavioral Approach: Treating Cocaine Addiction. Manual 1 (1998) NCADI # BKD254
• Community Reinforcement Plus Vouchers Approach: Treating Cocaine Addiction. Manual 2 (1998) NCADI # BKD255

• ^* Alcohol Problems in Intimate Relationships: Identification and Intervention. A Guide for Marriage and Family Therapists (2003) NIH Pub. No. 03-5284
• ^* Helping Patients with Alcohol Problems: A Health Practitioner's Guide. (2003) NIH Pub. No. 03-3769
• Cognitive-Behavioral Coping Skills Therapy Manual. Project MATCH Series, Vol. 3 (1995) NIH Pub. No. 94-3724
• Motivational Enhancement Therapy Manual. Project MATCH Series, Vol. 2 (1994) NIH Pub. No. 94-3723

Appendix F—Statement of Assurance

As the authorized representative of the applicant organization, I assure SAMHSA that if {insert name of organization} application is within the funding range for a grant award, the organization will provide the SAMHSA Government Project Officer (GPO) with the following documents. I understand that if this documentation is not received by the GPO within the specified timeframe, the application will be removed from consideration for an award and the funds will be provided to another applicant meeting these requirements.

• a letter of commitment that specifies the nature of the participation and what service(s) will be provided from every service provider organization, listed in Appendix 1 of the application, that has agreed to participate in the project;
• official documentation that all service provider organizations participating in the project have been providing relevant services for a minimum of 2 years prior to the date of the application in the area(s) in which services are to be provided. Official documents must definitively establish that the organization has provided relevant services for the last 2 years; and
• official documentation that all participating service provider organizations are in compliance with all local (city, county) and State/tribal requirements for licensing, accreditation, and certification or official documentation from the appropriate agency of the applicable State/tribal, county, or other governmental unit that licensing, accreditation, and certification requirements do not exist. (Official documentation is a copy of each service provider organization's license, accreditation, and certification. Documentation of accreditation will not be accepted in lieu of an organization's license. A statement by, or letter from, the applicant organization or from a provider organization attesting to compliance with licensing, accreditation and certification or that no licensing, accreditation, certification requirements exist does not constitute adequate documentation.)

(Signature of Authorized Representative)

(Date)

Appendix G—Logic Model Resources

Chen, W.W., Cato, B.M., & Rainford, N. (1998-9). Using a logic model to plan and evaluate a community intervention program: A case study. International Quarterly of Community Health Education, 18(4), 449-458.

Edwards, E.D., Seaman, J.R., Drews, J., & Edwards, M.E. (1995). A community approach for Native American drug and alcohol prevention programs: A logic model framework. Alcoholism Treatment Quarterly, 13(2), 43-62.

Hernandez, M. & Hodges, S. (2003). Crafting Logic Models for Systems of Care: Ideas into Action. [Making children's mental health services successful series, volume 1]. Tampa, FL: University of South Florida, The Louis de la Parte Florida Mental Health Institute, Department of Child & Family Studies. http://cfs.fmhi.usf.edu or phone (813) 974-4651

Hernandez, M. & Hodges, S. (2001). Theory-based accountability. In M. Hernandez & S. Hodges (Eds.), Developing Outcome Strategies in Children's Mental Health, pp. 21-40. Baltimore: Brookes.

Julian, D.A. (1997). Utilization of the logic model as a system level planning and evaluation device. Evaluation and Planning, 20(3), 251-257.

Julian, D.A., Jones, A., & Deyo, D. (1995). Open systems evaluation and the logic model: Program planning and evaluation tools. Evaluation and Program Planning, 18(4), 333-341.

Patton, M.Q. (1997). Utilization-Focused Evaluation (3rd Ed.), pp. 19, 22, 241. Thousand Oaks, CA: Sage.

Wholey, J.S., Hatry, H.P., Newcome, K.E. (Eds.) (1994). Handbook of Practical Program Evaluation. San Francisco, CA: Jossey-Bass Inc.