Proposed Issuance of Policy Memorandum, Material Qualification and Equivalency for Polymer Matrix Composite Material Systems

AGENCY:

Federal Aviation Administration, DOT.

ACTION:

Notice of policy statement; request for comments.

SUMMARY:

This document announces an FAA proposed general statement of policy applicable to the type certification of normal, utility, acrobatic, and commuter category airplanes. This document advises the public, in particular manufacturers of normal, utility, acrobatic, and commuter category airplanes, of additional information related to material qualification and equivalency for polymer matrix composite material systems. This notice is necessary to advise the public of FAA policy and give all interested persons an opportunity to present their views on the policy statement.

DATES:

Comments submitted must be received no later than July 13, 2000.

ADDRESSES:

Send all comments on this policy statement to the individual identified under FOR FURTHER INFORMATION CONTACT at Federal Aviation Administration, Small Airplane Directorate, ACE-111, Room 301, 901 Locust, Kansas City, Missouri 64106.

FOR FURTHER INFORMATION CONTACT:

Lester Cheng, Federal Aviation Administration, Small Airplane Directorate, ACE-111, Room 301, 901 Locust, Kansas City, Missouri 64106; telephone (816) 329-4120; fax 816-329-4090; e-mail: lester.cheng@faa.gov.

SUPPLEMENTARY INFORMATION:

Comments Invited

Interested persons are invited to comment on this proposed policy statement, ACE-00-23.613-01, by submitting such written data, views, or arguments as they desire. Comment should be marked, “Comments to policy statement ACE-00-23.613-01,” and be submitted in duplicate to the above address. The Manager, Small Airplane Directorate, will consider all communications received on or before the closing date for comments.

Background

This notice announces the availability of the following proposed policy memorandum, ACE-00-23.613-01, for review and comment. The purpose of this memorandum is to address certification projects initiated after the final date of the memorandum. Certification projects already in work do not necessarily need to comply.

Effect of General Statement of Policy

The FAA is presenting this information as a set of guidelines appropriate for use. However, this document is not intended to establish a binding norm; it does not constitute a new regulation and the FAA would not apply or rely upon it as a regulation. The FAA Aircraft Certification Offices (ACO's) that certify normal, utility, acrobatic, and commuter category airplanes should generally attempt to follow this policy when appropriate. Applicants should expect that the certificating officials would consider this information when making findings of compliance relevant to new certificate actions.

Also, as with all advisory material, this statement of policy identifies one means, but not the only means, of compliance.

Because this proposed general statement of policy only announces what the FAA seeks to establish as policy, the FAA considers it to be an issue for which public comment is appropriate. Therefore, the FAA requests comment on the following proposed general statement of policy relevant to compliance with 14 CFR part 23, § 23.613, and other related regulations.

General Statement of Policy

1.1 General

In the decades since the introduction of advanced composite materials for use in aircraft, the material qualification has been a costly burden to the airframe manufacturer. For each manufacturer, extensive qualification testing has often been performed to develop the base material properties and allowables at operating environmental conditions, which are used as part of an aircraft's design data, regardless of whether this material system had been previously certificated by other manufacturers. In addition to the use of such data in design, qualification also provides a population basis (e.g., in mean and variability statistics) to continuously ensure stable material production practices by the material supplier. The practice of qualification when performed by each manufacturer for an identical material system represents a massive duplication of effort.

In recent years, NASA, Industry, and the FAA have worked together to develop a cost-effective method of qualifying composite material systems by the sharing of a central material qualification database. This method is built on the existing sections of MIL-HDBK-17-1E, and allows credit for FAA witnessed materials testing performed by third parties such as material vendors or industry consortia. During the development process, the Small Airplane Directorate worked closely with members of the NASA Advanced General Aviation Transport Experiment (AGATE) research consortium to ensure the acceptability of this method of compliance to the applicable airworthiness regulations. Furthermore, the FAA and AGATE have maintained a good communication with the appropriate MIL-HBDK-17 Working Groups by participating in their regular meetings. Valuable thoughts have been shared for the development of this method.

This effort creates a new way of conducting business with airframe manufacturers and material suppliers. It enables composite material suppliers to work with the FAA to qualify their composite material system and receive approval (i.e., material qualification). An airframe manufacturer can then select this approved composite material system to fabricate aircraft parts and perform a smaller subset of testing to substantiate their control of material and fabrication processes tailored to a specific application. The terms “material equivalency” will be used in the current context to describe the sampling process for a subset of testing used to confirm equivalent mechanical, physical and chemical properties for a particular material or one undergoing minor changes. For purposes of example, a minor change would be a new material production line, which uses identical raw materials, processes and equipment. Another example of a minor change is the substitution of a new supplier for the same chemical constituent used to fabricate a given fiber or matrix type. A major change would involve more significant differences in the fiber type, matrix resin, and pre-impregnated fabrication process. It is anticipated, significant cost saving can be realized for both the industry and the FAA by sharing the approved central database and standardizing engineering protocol to demonstrate material equivalency.

As a precursor, efforts to establish protocol for shared material databases were documented in a letter, which was disseminated by the Small Airplane Directorate to both FAA certification field offices and industry in 1998. In that letter, the essential concepts of this method have been outlined both in terms of regulatory and technical considerations. As a follow-up, the current memorandum is intended to serve as a policy and guidance for the implementation of this newly developed methodology of qualifying the material systems. It is noted that currently this method pertains only to part 23 aircraft.

1.2 Substantiation of Composite Structures

It has been well recognized that analysis and base material data alone is generally not adequate for substantiation of composite structural designs. The “building-block approach” of testing, in concert with analysis, is typically used to fulfill the certification requirement. As outlined in Section 2.1 of MIL-HDBK-17-1E for Polymer Matrix Composites, the building-block approach consists of several levels of activities from both the “structural complexity” and “data application” considerations. The structural complexity is geometry or form-based, and may include levels of “constituent,” “lamina,” “laminate,” “structural element,” and “structural sub-component.” On the other hand, the data application is a specific activity performed within the design development and certification process. The specific levels of structural complexity required depend on the distinct purpose of the data application. For example, structural substantiation may use tests and analysis at many different levels of structural complexity, whereas material acceptance may only rely on the lowest levels (i.e., base material properties).

The material qualification and equivalency method discussed in this memorandum is a data application intended to be at the lower-levels of the structural complexity consideration. It includes testing to get mechanical and physical properties at the lamina level. Such tests are performed using laminates with simple ply stacking sequences to characterize the response of the composite material. At this level, the key properties represent un-notched and un-damaged base material strength allowables for loading in tension, compression, and shear. Other important results are the lamina moduli for these load cases. This material qualification testing provides quantitative assessment of the variability of key base material properties, leading to various statistics that are used to establish material acceptance, equivalence, quality control, and design basis.

For clarification purposes, tests at higher levels (i.e., structural laminate, element and sub-component) are typically needed to fulfill the remaining parts of the structural substantiation requirement. As the design moves closer to application specific, the testing program proceeds to a higher level.

Additional structural laminate specimen and element testing is intended to evaluate the ability of the material to tolerate common discontinuities. Key properties include open/filled hole tensile/compression strengths, cutouts, joint bearing and bearing/bypass strengths, bonded joint element and attachments strengths, and impact-damaged element strengths. These strength tests are used to derive the design values of the notched, bolted, bonded, and damaged features. These design values, in general, would be lower than that of the base material strength allowables established via the material qualification testing program. However, as the test element size and complexity increases, it is more costly to generate variability data. As a result, conservative engineering practices are typically applied to utilize statistics collected at the lower (specimen) level of tests.

Furthermore, the structural sub-component (or full scale) testing is typically required to confirm load paths (i.e., validate analyses) and evaluate the behavior and failure mode of increasingly more complex structural assemblies that are considered application specific. At this scale, it is unreasonable to think of shared databases due to unique features in the design of a given product.

2.0 Related Regulatory and Guidance Materials

2.1 Federal Regulations

This new method for material qualification and equivalency has been developed as a means of showing compliance with 14 CFR part 23 requirements for the field of application defined. The regulations that are directly related to this method include:

Section 23.601 General

Section 23.603 Materials and workmanship

Section 23.605 Fabrication and methods

Section 23.613 Material strength properties and design values

Section 23.613 contains specific requirements for material strength properties and design values. Presented below are the requirements that, in particular, are tied to this method:

• “Material strength properties must be based on enough tests of material meeting specifications to establish design values on a statistical basis.” [§ 23.613(a)]
• “Design values must be chosen to minimize the probability of structural failure due to material variability.” [§ 23.613(b)]. Section 23.613(b) requires that the design values selected to ensure structural integrity need to be characterized by the probability depending on the design configurations. That is, A-Basis for single-load-path design and B-Basis for multiple-load-path.
• “The effect of temperature on allowable stresses used for design in an essential component or structure must be considered where thermal effects are significant under normal operating conditions.” [§ 23.613(c)]. Similarly, § 23.603(a)(3) requires “Take into account the effects of environmental conditions such as temperature and humidity, expected in service.”

As discussed in Section 1.2, the database from the qualification program includes the base material strength allowables, which represent the design basis at the lamina level at appropriate environmental conditions. Design values utilized for any specific application still need to be established via some combination of additional testing programs, rationale engineering assumptions, and validated analyses. Nevertheless, the qualification database serves as a foundation upon which the material can be controlled and design values for higher-level application are derived. For certification purposes, the base material allowable is a subset of the aircraft's type design data.

2.2 Advisory Circulars

The following two FAA advisory circulars (AC's) present recommendations for showing compliance with FAA regulations associated with composite materials:

AC 20-107A—Composite Aircraft Structure

AC 21-26—Quality Control for the Manufacture of Composite Structures

AC 20-107A sets forth an acceptable, but not the only, means of showing compliance with the provisions of 14 CFR parts 23, 25, 27, and 29 regarding airworthiness type certification requirements for composite aircraft structures. Guidance information is also presented on associated quality control and repair aspects.

AC 21-26 provides information and guidance pertaining to an acceptable, but not the only, means of demonstrating compliance with the requirements of 14 CFR part 21 regarding quality control systems for the manufacture of composite structures. This AC also provides guidance regarding the essential features of quality control systems for composites as mentioned in AC 20-107A.

2.3 MIL-HDBK-17

The MIL-HDBK-17 has been developed and is maintained as a joint effort of the Department of Defense (DOD) and the Federal Aviation Administration (FAA). This handbook provides guidance in the development of base material properties (allowables) and design values acceptable to the FAA. This new methodology is derived based on the MIL-HDBK-17-1E (Polymer Matrix Composites Volume 1: Guidance). The sections that are closely related to this method include:

Section 2.3.2 Material qualification test matrices

Section 2.3.3 Material acceptance test matrices

Section 2.3.4 Alternate material equivalence test matrices

Section 8.4.3 Alternate material statistical procedures

For the simplicity of this memorandum, the MIL-HDBK-17-1E can also serve as a reference for most of the terminology used in this document.

For standardization purposes, guidance for material database presentation, both in terms of format and content, has been well outlined in MIL-HDBK-17-2E (Polymer Matrix Composites Volume 2: Materials Properties). Presentation of material data per the guidance set forth in the MIL-HDBK-17 is highly recommended.

2.4 AGATE Document (DOT/FAA Technical Report)

The specific methodology outlined in this memorandum has been developed through the effort of Work Package 3 (Integrated Design and Manufacturing Tasks) of the AGATE program. Technical works have been conducted mainly at the National Institute for Aviation Research (NIAR) facility affiliated with Wichita State University at Wichita, Kansas. Throughout the process, close coordination between the FAA [the Small Airplane Directorate, Technical Center and National Resource Specialist (NRS)] and the NIAR has been maintained to ensure this method is in compliance with the applicable airworthiness regulations.

Application of this method has been demonstrated for the epoxy-based pre-impregnated carbon or fiberglass material systems cured at 250 °F with low-pressure curing/processing cycles. This effort has resulted in an AGATE technical document entitled “Material Qualification and Equivalency for Polymer Matrix Composite Material Systems” where details of this methodology are presented. To enhance the accessibility of this document to the industry in general, an effort is underway by the FAA Technical Center to edit and publish it as a DOT/FAA Report.

3.0 Material Qualification

3.1 Field of Application

The developed material qualification methodology is intended, in general, for polymer matrix material systems. The purposes of this method include:

• To solidify and finalize material and process (M&P) specifications, including specific acceptance criteria for sampling relative to the qualification database
• To quantify base material variability
• To provide a central database with stabilized material processes

Application of this method has been conducted/demonstrated via the effort of the AGATE program. The AGATE program has applied this method to material systems that are characterized by the following specifics:

• Epoxy-based pre-impregnated carbon or fiberglass
• Unidirectional tape or woven fabric
• Cure temperature at 240 °F or higher
• Low-pressure curing/processing cycles (i.e., autoclave and vacuum bagging)

Testing requirements and data reduction procedures needed to certify the composite material system for complying with airworthiness regulations are presented in the AGATE document. The testing defined in the AGATE document represents the minimum requirement. In some cases, unique characteristics of a material system or its application may require testing beyond that defined by this method (i.e., more rigorous procedures and larger qualification databases). In these situations, Aircraft Certification Offices (ACO's) may require additional testing to demonstrate compliance with the applicable airworthiness regulations.

3.2 Qualification Approval Procedures

Material qualification bears the objective of establishing the FAA approved base material properties of an “original” material system. Test materials are fabricated using “original” process specifications. This effort may be part of ongoing certification programs and can be managed by the appropriate project ACO. In some cases, such as a consortium crossing geographic boundaries, the Small Airplane Directorate may manage this effort.

All specimen shall be fabricated according to the appropriate process specification to the geometry described in the AGATE document. Prior to testing, conformity of the test specimen must be performed by Manufacturing District Inspection Office (MIDO) inspectors at the request of ACO engineers. The MIDO inspector may elect to delegate this responsibility to a Designated Manufacturing Inspection Representative (DMIR) or Designated Airworthiness Representative (DAR).

Testing must be witnessed by the FAA. Witnessing can be performed by ACO engineers, or they may delegate this responsibility to a Designated Engineering Representative (DER) or MIDO inspector.

3.3 Environmental Conditions

In order to substantiate the environmental effects with respect to the material properties, several environmental conditions are defined to represent extreme cases of exposure. The selection of these conditions shall be based on the nature of the material system and its intended application as well.

To illustrate, the conditions defined as extreme cases for the AGATE program are as follows:

Properties for less extreme temperature conditions are determined through documented interpolation procedures.

3.4 Material Quality Control

As part of material qualification, physical and chemical property tests are recommended for each batch of material received from the material vendor. These tests should be traceable to each referenced test. Prior to a significant investment in material qualification testing, the quality control procedures of the material vendor should be reviewed to ensure that quality control programs are in place for the fiber and neat resin, as well as pre-impregnation of the material form (e.g., tape or fabric). The recommended testing items (e.g., resin content, fiber areal weight, and gel time), along with the test methods, are presented in the AGATE document.

In order to support the maximum operational temperature (MOT) limit of the material system and the specific data to be used in the statistical design allowable generation, cured lamina physical property tests (e.g., glass transition temperature, fiber/resin volume, and void content) are also required. These tests, along with the test methods, are defined in the AGATE document.

3.5 Batch-to-Batch Variability

For a composite material system base properties (allowables), several batches of material must be characterized to establish the statistically-based material property for each of the material systems. For this qualification method, a minimum of three (3) batches of material are required to establish a B-basis design allowable. For an A-basis design allowable, three (3) batches may also be used, but five (5) batches of material are highly recommended to establish more statistically stable properties. It is noted that the minimum number of batches used in AGATE methodology is less than that recommended in MIL-HDBK-17-1E.

In order to account for processing and panel-to-panel variability, the material system being qualified must also be representative of multiple processing cycles. For this qualification method, each batch of material must be represented by a minimum of two independent processing/curing cycles (e.g., low-pressure autoclave and vacuum bagging). One engineering observation, which led to this AGATE methodology, was that the variation from composite panel processing can be as important as batch-to-batch material variability.

3.6 Property Testing Requirement

The required material property tests are specified in the AGATE document, along with the recommended test method and the required number of batches/replicates per environmental condition (i.e., CTD, RTD, ETW and ETD). In the AGATE document, a format has been defined to represent the required number of batches and replicates per batch. The format reads: #×#, where the first # represents the required number of batches and the second # represents the required number of replicates per batch. For example, “3×6” refers to three batches of material and six specimen per batch for a total requirement of 18 specimen.

To illustrate, the tests required by the AGATE document for qualification at the environmental condition of “Room Temperature Dry (RTD)” are listed as follows:

3.7 Base Material Allowable Generation

Upon completion of the property testing, the statistical base material allowable can be generated for each mechanical strength property per the data reduction procedure described in the AGATE document. Software for the data reduction procedure has been made available in the form of a disk-file as an attachment to the AGATE document. Raw test values are normalized to a specified fiber volume as the fibers are the primary load-carrying component of the composite material. This provides a consistent basis for property comparisons and generally reduces variability in fiber-dominated properties. The procedure used for this is consistent with that recommended by MIL-HDBK-17-1E.

Proper consideration of the inherent material property variability in composite materials needs to be addressed in assigning design basis value to each mechanical property. Although the statistical procedures presented in the AGATE document may account for most common types of variability, these procedures may not account for all sources of variability.

B-basis and A-basis material allowables are determined for each strength property using the statistical procedures outlined in the AGATE document. The specific procedures used assume a normal distribution for the population and take advantage of pooling of data between environments in calculating statistical variations. The latter is dependent on the assumptions that the failure mode for a given type of test does not vary significantly between environments and that the material variability across environments is comparable. The AGATE document describes the additional statistical tests and engineering data analysis needed to ensure all assumptions are not violated for a given material system. If evidence of deviations from the assumptions exists, more general procedures in MIL-HDBK-17-1E should be followed. For the moduli and Poisson's ratio, the average value of all corresponding tests for each environmental condition is used.

If maximum strain material allowables are required, simple one-dimensional linear stress-strain relationships may be employed. The linear assumption works well for tensile and compressive strain behavior but may produce rather conservative strain values in shear due to nonlinear behavior. More realistic engineering guidelines to derive shear strain allowables are given in MIL-HDBK-17-1E (Section 5.7.6).

3.8 Material Performance Envelope

Referring back to the discussions in Sections 1.2, 2.1, and 3.1, base material strength allowables and elastic moduli generated by the procedures given in the AGATE document serve a purpose in stable composite material control within the industry and certification of specific aircraft products. Standard test methods and accepted statistical data treatment facilitate their use for the former, where a wide segment of the material supplier and aircraft manufacturing industry can share in the cost of generating the database. When it comes to the use of this data for the development and certification of structure for a specific aircraft, complementary test data and analysis is needed to account for the effects of design detail, structural scale, and damage.

Using the statistical allowables, a base material performance envelope can be generated for a material system by plotting these values as a function of temperature. Each specific aircraft application of the qualified material may have a different maximum operational temperature (MOT) limit than those tested for the material qualification. Some applications may require a reduced MOT. For these cases, interpolation may be used to obtain the corresponding basis values at the new application MOT.

Interpolation schemes and examples are presented in the AGATE document. The schemes provided in the document are practical for materials obeying typical mechanical behavior. In most cases, some minimal amount of testing may also be required to verify the interpolated values.

Since unforeseen material property drop-offs with respect to temperature and environment can occur, extrapolation to a higher MOT should not be attempted without additional testing and verification.

4.0 Material Equivalency

For clarification purposes, the terms “material equivalency” used in the current memorandum refer to the process of substantiating material properties for purposes of sharing a composite material qualification database and/or demonstrating that minor changes in material production processes have a negligible effect. This is achieved by test sampling and passing the acceptance criteria, which were derived from a larger population of material data.

4.1 Field of Application

Composite material equivalence testing, which constitutes reduced data sampling (e.g., a single batch), may be performed by a manufacturer to establish a link with the original qualification database and associated specifications. Depending on the manufacturer's use of the qualification database, specifications for processing a particular product and the associated design data may even change significantly after establishing the link. For example, if the only intent of a link with the qualification database is to establish a population from which acceptance criteria are derived for standard tests performed in base material control, then significant changes in processing for a particular product may be allowed. On the other hand, if the base material qualification database has greater use in design (e.g., applied in deriving design values), then additional testing may be needed to show equivalency with the process variations. In short, the role of material equivalency testing in certification will depend on details of the particular project.

For example, consider the use of a given material in sandwich construction, which may have process variations (e.g., lower autoclave pressures) and changes in laminate characteristics resulting from the sandwich panel design configuration (e.g., dimpling of the face-sheets on honeycomb cells). In such a case, standard tests for base material properties in the AGATE approach use flat laminates, which may yield different properties than occur in sandwich panels. If the manufacturer's intended use of the qualification database is limited to control of the base material as purchased, the manufacturer may elect to demonstrate equivalency using original specifications. On the other hand, if the qualification database will have greater use in design, then equivalency testing should expand to consider the effects of product process and design variations on the base material properties. Alternatively, subsequent tests within the building block approach used for certification may also be defined to account for such differences. Again, the role of material equivalency testing in certification will depend on details of the particular project.

The material equivalence testing may also be used to assess the effects of minor changes in constituent(s), the constituent manufacturing process, and/or the resin pre-impregnation process, for the purpose of utilizing the existing material qualification database. This testing evaluates the key properties for test populations large enough to provide a definitive conclusion but small enough to provide significant cost savings as compared to establishing a new database.

Note that MIL-HDBK-17-1E goes beyond the discussions in this memorandum to describe methods for demonstrating alternate material acceptance. The discussion can be found in Section 2.3.4. Although the term equivalence is used in this section of MIL-HDBK-17-1E, the test matrices presented are much more extensive, highlighting additional issues for the problems being addressed (i.e., changes in fiber type, fiber tow size, resin, and pre-impregnated manufacturer). Table 2.3.4.1.3 of this volume covers a wide variety of changes to a material system and highlights the fact that the performance of a material system is determined by both the materials and processes used in its manufacture.

The AGATE methodology of demonstrating material equivalency is derived from MIL-HDBK-17-1E. This methodology only applies to situations with minor changes to the “original” material system in terms of material constituents and/or manufacturing processes. These situations may include:

• Identical materials, processed by same manufacturer using identical fabrication process at different locations;
• Identical materials, processed by different manufacturer using a “follow-on” process that is equivalent to the “original” fabrication process;
• Identical materials, processed by different manufacturer using a “follow-on” process that is slightly different to the “original” fabrication process;
• Minor changes in constituent(s) and/or constituent manufacturing process, processed by same/different manufacturer using a “follow-on” process that is slightly different to the “original” fabrication process;
• Combinations of the above.

In summary, the purposes of this equivalency method include:

• To share and make use of the central database by a new user (i.e., original material qualification);
• To continue surveillance of material and process (e.g., Section 5.0 as applied in material quality control);
• To show that minor changes to material and processes do not affect base material properties;
• To make final adjustment on material and process specifications for specific application and demonstrate that it has little affect on base material properties.

4.2 Equivalency Approval Procedures

For the “follow-on” applicants to use the database, they need to develop their own material and process specifications based on the “original” material and process specifications. The applicants submit these specifications along with the necessary test plans to their geographically responsible ACO for review. In all cases of material equivalency, an “original” should exist that contains base material mechanical properties and strength allowables, as well as the chemical and physical properties, for the initially qualified material system.

As is the procedure on any certification program, the ACO reviews the test plans and the updated material/process specifications prior to the initiation of testing. The review of the applicants' specifications should determine if they meet the application limitations outlined in Section 4.1, and are, therefore, candidates for material equivalency testing. Since the basis properties of a composite material system are sensitive to both its material constituents and manufacturing process, vigilant engineering judgement must be exercised during the evaluation process. The fabrication methods of the applicants' structure must meet the applicable airworthiness regulations including, but not limited to, §§ 23.603 and 23.605.

Testing is required to qualify the “follow-on” material system by demonstrating material equivalency to the “original” material system. Testing must be witnessed by the FAA. Testing requirements, data reduction procedures, and material equivalency criteria/guidance are presented in the AGATE document.

In addition to the base material level coupon testing, certification programs may require some element or sub-component testing in demonstrating equivalency for minor changes in the material production processes over time, which are suspected to have some effect on part manufacturing processes. These requirements will depend on the degree of change as well as on the application (e.g., complexity of the components or parts to be manufactured).

4.3 Equivalency Testing Requirement

As described in Section 4.1, the AGATE material equivalency methodology is derived based on the most compatible situations existing, as discussed in MIL-HDBK-17-1E (i.e., an identical material is used or changes in the material are minor). Based upon the batch-to-batch variability established in the original qualification database, material equivalency testing should be conducted to investigate the processing or panel-to-panel variability inherent in the follow-on manufacturer or location. As a minimum requirement to initiate such an exercise, the material and process controls used to generate the initial database must be known (i.e., the “original” material and process specifications or “pedigree” must be known). This issue has come up relative to some of the data that has been published in MIL-HDBK-17-2E, and a plan has been initiated to ensure such information is available for data utilization.

The equivalency tests required are presented in the AGATE document along with the recommended test methods and the required number of batches/replicates per environmental condition (i.e., RTD and ETW). One (1) batch of material is the minimum required for this testing program. As with material qualification, two separately processed panels are used in obtaining specimen for strength tests.

To illustrate, the tests required by the AGATE document to demonstrate equivalency under the environmental condition of “Room Temperature Dry (RTD)” are listed as follows:

4.4 Success Criteria for Equivalency

Results derived from the equivalency testing are compared with the original qualification database. The statistical procedures and the success criteria for equivalency are presented in the AGATE document. As with qualification, the acceptance criteria adopted by AGATE to demonstrate equivalency assumes a normal distribution. If a normal distribution was not confirmed by checks performed as part of the “original” material qualification, the acceptance criteria will need to change to reflect the statistical distribution that was adopted for the population. In such a case, the more general procedures in MIL-HDBK-17-1E should be followed.

First, the qualification database shall present the property of interest in terms of “mean” and “standard deviation.” For base material strength properties, the qualification database also provides B-basis and/or A-basis values, which can be used for purposes of comparison in establishing specific acceptance criteria. In addition, two statistical parameters for sampling need to be defined, and they are: “α” (probability of rejecting a good material) and “n” (number of specimen to be tested for the property of interest).

A selection of α = 0.01, for example, represents 1% of the chance of wrongly rejecting a good material. A higher “α” value represents a more conservative criteria, yet at the expense of a higher chance of rejecting a good material. Also, as the number of specimen increases, the chance for the mean of the specimen (tests sample) to appear different from the original qualification data decreases. Statistically, the two parameters reflect the Type I errors in test on either means or minimum individual values. The Type I error refers to the situation of rejecting the null hypothesis when it is true. The B-basis and A-basis values, which were derived in population testing, have limited statistical meaning when assessing the equivalency from a small sample size. However, they may have some engineering value in setting the α for a particular application.

For strength properties, material equivalency is established by using both the means and the minimum individual values as the acceptance criteria. The material equivalence is not acceptable when either one of the two comparisons fails. The “α” represents the probability of failing either one of the two, or both, comparisons.

Based on a limited “round robin” testing program, the AGATE method currently recommends an “n” value of “8”, and an “α” value of “0.05” for material equivalency tests to link with the complete material qualification database. As the exposure and experience increase through time, the values for these two parameters may be revised from lessons learned. Also, considering the intrinsic difference both in terms of the nature of material system and the specific of application, the certification offices (ACO's) may adjust this set of values reflecting their unique circumstances.

Although specific criteria are not given, strength properties from equivalency testing should also not be excessively higher than those obtained for the original qualification database. Engineering judgement should be used to detect such increases in base strength, which may affect structural failure modes or reductions in untested strength properties. For example, un-notched (or small notch) tensile strength properties have been found to be inversely related to the tensile residual strength of composite structure with larger flaws.

For modulus, a simple comparison of means is used. The criterion is not satisfied when either the test sample mean is too high or too low in reference to the original maximum/minimum mean of the qualification database.

There are also statistical tests that interrogate the new samples as to their equivalency to the baseline sample qualification database. These can be used as an alternative to the test on means and minimum individual values described above. MIL-HDBK-17-1E recommends the k-sample Anderson-Darling (A-D) statistical test (Section 8.3.2.2) or the ANOVA (analysis of variance) method described in Section 8.4.3.1. The k-sample A-D test can be used for unequal sample sizes that will be encountered when comparing the baseline data to the new data. Discussion on the use of a significance level of (α = 0.05 is given in MIL-HDBK-17. The value chosen should be agreed upon by the particular application and should be the same if the ANOVA method is used.

Other alternate tests (if normal distribution is assumed) are to use the F-test to show equivalency of the means (Section 8.3.5.2.2) and Levene's test to show equivalency of the variances (Section 8.3.5.2.1). An “α” value for these tests must also be selected.

Successful completion of the equivalency testing allows the applicant to use the properties contained in the original qualification database. In the case when the testing of the first batch fails, a second opportunity using a different batch of material can be allowed for this equivalency testing. In order to limit the undesirable, statistically termed as the Type II error, only permission of retest to the 2nd batch is recommended. The Type II error refers to the situation of accepting the null hypothesis when it is false.

Should the applicant fail criteria for equivalency testing of the second batch, the original base material allowable database can no longer be used, and a new base material allowable database needs to be established per material qualification procedures. Such a scenario requires engineering to identify material and/or processing differences, which led to changes in the base material properties, and the associated update to specifications (i.e., a new material qualification). In addition, careful planning of material procurement, panel fabrication and testing may be considered at the start of a material equivalency exercise to ensure that equivalency testing of a first and second batch can be expanded to be part of a new qualification if required. For example, the material order and panel sizes fabricated for a particular batch of material may be sufficiently large enough to yield additional specimens, as needed for the larger test matrix in a qualification effort.

5.0 Continuous Quality Control

Material supplier and purchaser tests performed as part of a continuous quality control process may be considered a special case of material equivalency testing. In this case, the sample size is typically smaller than recommended for the material equivalence exercise described in Section 4.0. Nevertheless, the tests are typically performed on a per batch basis and a link with the qualification database can be developed using the same statistical methods (Section 4.4).

For purposes of continuous quality control, a recommended “α” value of 0.01 (i.e., 1% probability of rejecting “good” material) and an “n” value of 3 to 5 are appropriate. Note the less stringent requirement here than for obtaining access to the “original” qualification database discussed in Section 4.4. In the latter case, all future batches of material are being admitted while in the former case only one batch is under scrutiny. As the exposure and experience along this line increase through time, a new set of values for these two parameters may be provided. Also, considering the intrinsic difference both in terms of the nature of the material system and the specifics of application, the certification offices (ACO's) may adjust this set of values reflecting their unique circumstances.

If quality control testing fails, engineering evaluation can be performed to justify a retest of the same batch of material. As part of this effort, engineers should search for other reasons to believe the material is “bad” or identify a problem in specimen fabrication and/or testing. The number of “retests” should be limited to one which, from a purely statistical perspective, yields a probability of rejecting good material in two sets of receiving inspection tests for the same batch is only 0.01% for the recommended “α”.