This document is a review draft. This information is distributed solely for the purpose of pre-dissemination review under applicable information quality guidelines. It has not been formally disseminated by EPA. It does not represent and should not be construed to represent any Agency determination or policy. It is being circulated for review of its technical accuracy.

                                                        EPA Classification No.:
DRAFT FINAL
(2106-S-02.0)
                                                             CIO Approval Date:

                                                           CIO Transmittal No.:

                                                                   Review Date:
12/01/12 
Issued by the EPA Chief Information Officer,
                 Pursuant to Delegation 1-19, dated 07/07/2005
                     U.S. Environmental Protection Agency
                                       
  QUALITY STANDARD FOR ENVIRONMENTAL DATA COLLECTION, PRODUCTION, AND USE BY
                       NON-EPA (EXTERNAL) ORGANIZATIONS
                                       
   1. PURPOSE
This Quality Standard for Environmental Data Collection, Production, and Use by Non-EPA (External) Organizations (2106-S-02.0), including its Annexes:

   * defines and establishes the quality requirements and management controls for non-EPA organizations that collect, produce, and use environmental data;
   * defines how EPA quality requirements in CIO Standard 2106-S-01.0, Quality Standard for Environmental Data Collection, Production, and Use by EPA Organizations (EPA 2013a), shall be applied to non-EPA organizations under authorized agreements governed by Federal regulations;
   * reaffirms the applicability of and conformance to ANSI/ASQ E4 (Current Edition) for the planning, implementation, documentation, and assessment of extramural programs that collect, produce, and use environmental data;
   * provides the basis for non-EPA organizations to define effective processes, procedures, and management controls for planning, implementing, and assessing quality activities defined in their approved Quality Management Plans (QMPs) for environmental data collection, production, and use;
   * requires the use of Quality Assurance Project Plans (QAPPs) or equivalent documents to establish quality specifications in individual project activities that collect, produce, and use environmental data; identifies expected quality responsibilities for managers and staff in external organizations for activities involving environmental data collection, production, and use; and
   * replaces EPA QA/R-2, EPA Requirements for Quality Management Plans, March 2001 (Reissued May 2006) and EPA QA/R-5, EPA Requirements for Quality Assurance Project Plans, March 2001 (Reissued May 2006) in their entirety with exceptions as specified under Clause 2.3.

This Standard includes three annexes which contain detailed specifications and guidance to support the requirements presented in the Standard. Normative annexes are additional requirements beyond those requirements in the body of the Standard and shall be regarded as part of the requirements of the Standard. Informative annexes (or parts thereof) may become requirements if they are cited and/or stated as such in agreements or if so decided by the external organization using the Standard and this decision is documented in the external organization's QMP.



2. SCOPE AND APPLICABILITY
0.1 SCOPE

The scope of this Quality Standard is consistent with EPA Quality policies and procedures, and encompasses EPA expectations for all program activities conducted by external organizations that involve the collection, production, and use of environmental data. It applies to environmental data produced or used through approved external agreements.

Environmental data may be:
   * collected from environment media directly in the form of physical samples or from existing electronic databases;
   * produced from analytical methods or instrumentation, and mathematical or electronic computational models or programs;
   * collected from the literature or other existing data sources (e.g., web sites on the Internet); and
   * used in a variety of analyses and evaluations to document environmental conditions and to support decisions pertaining to such conditions.

Activities involving environmental data encompassed by this Standard include, but are not limited to:
   * direct and indirect field or laboratory measurements;
   * evaluating the operation and performance of environmental technology (e.g., pollution control);
   * inspections;
   * survey development or application;
   * enforcement and compliance monitoring or assessments;
   * application of environmental management systems;
   * environmental safety and health monitoring;
   * scientific research;
   * regulatory development;
   * use of statistical or economic analyses using environmental data;
   * use of information technology (e.g., the development and use of models such as pollutant transport and ground water migration, databases) supporting Agency programs;
   * use of information sources outside of direct EPA management controls or authority (e.g., academic institutions); and
   * use of data obtained from other sources (e.g., literature, Internet).

2.2 APPLICABILITY

This Standard applies to all non-EPA organizations having external agreements with EPA and shall be explicitly applied to all external agreements involving the collection, production, and use of environmental data.

The terms and conditions of applicable external agreements shall include this Standard by reference as the requirements for quality management processes, applications, and personnel responsibilities.





2.3 QUALITY MANAGEMENT REQUIREMENTS APPLICABLE TO NON-EPA ORGANIZATIONS

Prior to this Standard, the quality management requirements for non-EPA organizations were given in two extramural reference documents:
   * EPA QA/R-2, EPA Requirements for Quality Management Plans (March 2001 Edition, re-authorized in May 2006); and
   * EPA QA/R-5, EPA Requirements for Quality Assurance Project Plans (March 2001 Edition, re-authorized in May 2006).

These documents are deleted as requirements for use in new or revised external agreements effective with this Standard, and are replaced by this Standard.

2.3.1 Existing External Agreements

The use of QA/R-2 and QA/R-5 by non-EPA organizations in existing external agreements may be allowed by EPA organizations under one or more of the following circumstances:
   * the documents have been explicitly incorporated or referenced into an extramural agreement or enforcement/consent agreement and, as a result, are legal requirements;
   * current Federal and/or EPA regulations explicitly cite the documents;
   * other EPA policies and orders explicitly cite the documents; or
   * use of the Uniform Federal Policy for Quality Assurance Project Plans (UFP-QAPP) is explicitly required.

Under these conditions, the requirements of QA/R-2 and QA/R-5 shall replace those of CIO Standard 2106-S-02 (EPA 2013b) for the term of the particular external agreement, but shall be limited to the defined period of performance of the agreement. EPA will confirm the appropriate continued use of these deleted documents in existing external agreements.

2.3.2 New and Revised External Agreements.

All new work performed through external agreements shall comply with this Standard.

When the use of QA/R-2 and QA/R-5 is allowed for an external agreement, it is the responsibility of EPA to monitor the implementation of the agreement to ensure that criteria for their continued use are met. Such monitoring will be accomplished in accordance with this Standard (see Clause 7.8). If an existing external agreement (i.e., contract Statement of Work, assistance terms and conditions, etc.) is modified by the Agency during its period of performance and the external agreement includes the use of EPA QA/R-2 and EPA QA/R-5, the Agency shall determine if it is beneficial to the Government to invoke the quality requirements as included under CIO Standard 2106-S-02. If it is determined to be beneficial to the Government, the use of EPA QA/R-2 and EPA QA/R-5 shall cease and the requirements of CIO Standard 2106-S-02 shall apply and shall be invoked in the modification to the agreement.

Existing external or other agreements may be revised by EPA before their end dates to require the elements of this Standard in lieu of QA/R-2 and QA/R-5. Other Federal/EPA regulations will be revised when practicable to replace QA/R-2 and QA/R-5 with the requirements of this Standard.





3.	AUDIENCE
The principal audience for this external Quality Standard is all non-EPA organizations and their employees (managers and staff) responsible for planning, collecting, assessing, and/or using environmental data and personnel who are assigned specific quality management duties and responsibilities within the non-EPA organization.

4.	BACKGROUND
This Standard, along with other applicable EPA standards, and with other relevant regulations, statutes, and external agreement terms and conditions, provides the foundation for the implementation of the EPA Quality Policy among non-EPA organizations performing work through external agreements. Such external agreements include contracts, assistance agreements, and enforcement agreements. This Standard conforms to the current editions of EPA Quality Policy CIO Policy 2106 (EPA 2006a), and Procedure for Quality Policy CIO Procedure 2106-P-01 (EPA 2006b).

Since 1979, EPA has required all extramural organizations performing work that collects, produces, and uses environmental data to comply with quality requirements defined by EPA Orders. Originally issued in 1984 and reauthorized in 2000, EPA Order 5360.1 A2, Policy and Program Requirements for the Mandatory Agency-Wide Quality System, required application of the Quality Program to environmental data collected and/or used by EPA and the design, construction, and operation by EPA organizations of environmental technology systems such as pollution control and abatement systems; treatment, storage and disposal systems; and remediation systems. EPA Order 5360.1 A2, which was later re-designated CIO Policy 2105.0, also clarified that the scope of environmental data includes, in addition to information collected directly from measurements, data compiled from other sources (i.e., databases, scientific literature), environmental models, and the performance of environmental technology. EPA Order 5360 A1, EPA Quality Manual for Environmental Programs, was developed to provide instructions for implementing the requirements of this Policy. This order was subsequently re-designated CIO Procedure 2105-P-01.0. The relevant content of CIO Procedure 2105-P-01.0 for extramural agreement holders was communicated using two documents that when cited required quality documentation: EPA QA/R-2 (for QMPs) and EPA/QA R-5 (for QAPPs).

These previous policy and procedure documents required non-EPA organizations having extramural agreements with EPA to develop, document, and implement a Quality Management System (QMS) that conformed to all applicable EPA policies and procedures, with American National Standard ANSI/ASQC E4-1994. Specifications and Guidelines for Quality Systems for Environmental Data and Technology Programs and with applicable Federal regulations on quality in extramural agreements. ANSI/ASQ E4 is a national consensus standard authorized by the American National Standards Institute (ANSI) and was developed by the American Society for Quality (ASQ). The requirement for QMS conformance with ANSI/ASQ E4 (Current Edition) is continued in this Standard.

5.	AUTHORITY
Authorities applicable to this Standard include the current editions of:
   * Quality Policy, CIO Policy 2106;
   * Procedure for Quality Policy, CIO Procedure 2106-P-01; and
   * Quality Standard for the Collection, Production, and Use of Environmental Data by EPA Organizations, CIO Standard 2106-S-01.

6.	RELATED REGULATIONS, POLICIES, AND PROCEDURES
In addition to the documents listed in Clause 5 above, the following documents may be helpful for the application of this Standard. They contain provisions that, through reference in this text, constitute provisions of this Standard when invoked as part of a statement of work or required by an applicable policy or directive. Since regulations, policy documents, and standards are subject to periodic revision, users of this Standard should apply the current editions of the documents listed below as well as all citations provided throughout this Standard:

   * 40 CFR 30, "Grants and Agreements with Institutions of Higher Education, Hospitals, and Other Non-Profit Organizations";
   * 40 CFR 31, "Uniform Administrative Requirements for Grants and Cooperative Agreements to State and Local Governments";
   * 40 CFR 35, "State and Local Assistance";
   * 48 CFR 46, Federal Acquisition Regulations (FAR), "Quality Assurance";
   * U.S. EPA, Office of Grants and Debarment, Implementation of Quality Assurance Requirements for Organizations Receiving EPA Financial Assistance ;
   * U.S. EPA, Office of Acquisition Management, Contracts Management Manual  ;
   * EPA Manual 1610, Interagency Agreement Policies, Procedures and Guidance Manual  ;
   * ANSI/ASQ E4, Quality Systems for Environmental Data and Technology Programs  -  Requirements with Guidance for Use, American National Standard, American National Standards Institute and American Society for Quality  ;
   * Circular A-119, Federal Participation in the Development and Use of Voluntary Consensus Standards and in Conformity Assessment Activities, Office of Management and Budget, 1998;
   * National Technology Transfer and Advancement Act of 1995, PL104-113, March 1996;
   * 44 U.S.C. Chapter 31 - Records Management by Federal Agencies (Federal Records Act);
   * 44 U.S.C. Chapter 33 - Disposal of Records (Federal Records Disposal Act);
   * EPA Policy Peer Review and Peer Involvement at the U.S. Environmental Protection Agency  ;
   * U.S. EPA Science Policy Council Peer Review Handbook  ;
   * U.S. EPA Science Policy Council, A Summary of General Assessment Factors for Evaluating the Quality of Scientific and Technical Information, EPA 100/B-03/001, June 2003  ;
   * U.S. EPA Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity, of Information Disseminated by the Environmental Protection Agency  ; 
   * CIO Policy 2155.1, Records Management  ;
   * U.S. EPA, Assuring the Competency of Environmental Protection Agency Laboratories, Agency Policy Directive, EPA Forum on Environmental Measurements (February 2004);
   * U.S. EPA, Ensuring the Validity of Agency Methods: Methods Validation and Peer Review Guidelines (chemical methods), Agency Policy Directive Number FEM-2005-01, EPA Forum on Environmental Measurements;
   * U.S. EPA, Ensuring the Validity of Agency Methods: Methods Validation and Peer Review Guidelines (radiochemical methods), Agency Policy Directive Number FEM-2006-01, EPA Forum on Environmental Measurements;
   * U.S. EPA, Ensuring the Validity of Agency Methods: Methods Validation and Peer Review Guidelines: Sampling Methods for Chemical and Radiochemical Parameters, Agency Policy Directive Number FEM-2007-02, EPA Forum on Environmental Measurements;
   * U.S. EPA, Ensuring the Validity of Agency Methods. Validation and Peer Review Guidelines: Microbiological Methods of Analysis, Agency Policy Directive Number FEM-2009-01, EPA Forum on Environmental Measurements;
   * U.S. EPA, Ensuring the Validity of Agency Methods, Validation and Peer Review Guidelines: Methods of Analysis Developed for Emergency Response Situations, Agency Policy Directive Number FEM-2010-01, EPA Forum on Environmental Measurements;
   * U.S. EPA, Policy to Assure Competency of' Laboratories, Field Sampling, and Other Organizations Generating Environment Measurement Data Under Agency-Funded Acquisitions, Agency Policy Directive FEM-2011-01,Forum on Environmental Measurements;
   
   * U.S. EPA, CIO Data Standards Policy (DSP), Delegation 1-19, Environmental Data Standards (EDS), Final Environmental Sampling, Analysis, and Results: Project, Exchange Network Leadership Council Standard EX000002.2;
   * U.S. EPA, CIO Data Standards Policy (DSP), Delegation 1-19, Environmental Data Standards (EDS), Final Environmental Sampling, Analysis, and Results: Field Activity Data Standard, Exchange Network Leadership Council Standard EX000004.2;
   * U.S. EPA, CIO Data Standards Policy (DSP), Delegation 1-19, Environmental Data Standards (EDS), Final Environmental Sampling, Analysis, and Results: Analysis and Results, Exchange Network Leadership Council Standard EX000005.2; and
   * U.S. EPA, CIO Data Standards Policy (DSP), Delegation 1-19, Environmental Data Standards (EDS), Final Quality Assurance and Quality Control Data Standard, Exchange Network Leadership Council Standard EX000012.2.
   
7.	REQUIREMENTS
Each organization shall establish and maintain an effective Quality Management System (QMS) for the environmental programs it conducts or directs under the applicable external agreement. The QMS shall include the quality management controls and management elements needed for planning, implementing, and assessing the effectiveness of quality processes and procedures applied to the collection, production, and use of environmental data. These elements shall be used in conjunction with the other applicable clauses of this Standard to formulate a complete QMS. Many of the management elements contained in this Standard are consistent with requirements found in other consensus standards (e.g., ISO 9001 [Current Edition]). Each organization must document their QMS in an approved QMP which shall:

   * identify internal and external factors that are relevant to the organization's purpose and that affect its ability to achieve the expected outcomes of the QMS;
   * describe how the organization plans to achieve the expected outcomes of work to be performed, including prevention of undesired results;
   * describe and affirm management commitment to the QMS;
   * describe and chart organizational roles, responsibilities, and authorities;
   * describe how the competence of persons doing work shall be determined and maintained;
   * describe the activities expected to be performed within the scope of the QMS, including documentation of the work and quality procedures to be applied in a QAPP or equivalent document, and the process for its approval and implementation;
   * describe the process for addressing QA issues and concerns;
   * describe how communications related to the QMS scope will be ensured;
   * describe the control of required quality documentation;
   * describe how performance will be measured and reported, including internal assessments and management reviews;
   * discuss how problems and nonconformities will be corrected and documented; and
   * describe how work conducted through sub-agreements with other organizations will be evaluated.

The QMS shall be planned, established, documented, implemented, and assessed as an integral part of a management system for the organization's environmental programs.

The QMS shall be documented in the organization's QMP and shall be established and operational before the initiation of the work defined in the external agreement. In some circumstances, a combined QMP and QAPP may be utilized to document these requirements (see Clause 7.5.2).

The QMP shall be reviewed and approved for implementation by the external organization and then shall be reviewed and approved by EPA.



7.1 MANAGEMENT REPRESENTATIVE FOR QUALITY

The organization's senior management shall designate or appoint a management representative for quality such as Quality Assurance Manager (QA Manager) or Director of Quality with defined authority that includes determining that the approved QMS is implemented and maintained in accordance with the requirements of this Standard, and who reports to top management on the performance effectiveness of the QMS. This QA Manager functions independently of environmental data operations or model development and has sufficient technical and management expertise and authority to conduct independent oversight in the implementation of the organization's quality system in the environmental programs of the organization. The roles and responsibilities for all QA personnel must be documented in the organization's QMP. Additional information about these roles can be found in Clause 8.

7.2 QUALITY MANAGEMENT PLAN (QMP)

The QMP (or equivalent document) for the organization shall describe when, how, and by whom controls shall be applied to the specific technical or project efforts required in the external agreement, and shall outline how these efforts are planned, implemented, and assessed. The QMP shall identify and describe activities which directly or indirectly affect quality and shall include as a minimum:

   * an identification of policies and standards with which the QMS conforms;
   * a description of the QMS;
   * a description of processes to assess the effectiveness of the QMS;
   * an identification of all organizations and business lines and programs to which the QMP applies;
   * a description of the organization, management and staff responsibilities and authorities, and quality management personnel responsibilities and authorities;
   * a description of processes to ensure personnel competency;
   * description of documents and records management processes and procedures;
   * description of processes involving the development and use of information technology methods;
   * description of general processes for planning, implementing, and assessing environmental data collection, production, and use activities; and
   * description of processes for including appropriate quality requirements in sub-agreements.

The QMP shall reflect the current procedures of the organization for its QMS and shall be reviewed and approved by the organization and by EPA for implementation. The QMP shall address all applicable parts of this Standard and shall identify why any parts are not applicable.

Organizations should apply a "graded approach" to identify and define other types of acceptable quality documents to be used. A graded approach generally involves basing the level of managerial controls applied to an item or work according to the intended use of the results and the degree of confidence needed in the quality of the results. Organizations that use a graded approach should describe the basis for their approach in the organization's QMP.

QMPs may be approved by EPA for no longer than the term of the agreement or five years. If the QMS is modified as a result of a reorganization or assessment corrective action, the QMP must be revised and resubmitted for approval by the organization's senior management or delegated representative and the EPA CIO or delegated representative.

The specific requirements for the content of QMPs are contained in Annex A of this Standard and must be addressed by the organization. Advice on how to satisfy the QMP requirements may be found in Handbook for Preparing Quality Management Plans (EPA 2013c).





7.3 MANAGEMENT REVIEW AND REPORTING

At regular intervals or as required by the terms and conditions of the external agreement, management shall evaluate the QMS to confirm its ongoing effectiveness for the work to be performed and its adherence to policies, procedures, and standards included or referenced in the external agreement. The QMS shall be revised to reflect organizational, operational, and policy changes identified during this review or as otherwise determined by management. The changes shall be documented in a revised QMP and shall be reviewed and approved by EPA in accordance with this Standard.

7.4 PERSONNEL COMPETENCE

Personnel shall have the competence (i.e., technical skills, demonstrated knowledge, and experience) needed to perform assigned duties. Objective evidence of personnel job proficiency shall be documented and maintained for the duration of the project or activity affected (or longer if required by statute or policy) and documented in the QMP or QAPP as appropriate.

7.5 QUALITY ASSURANCE PROJECT PLAN (QAPP)

The QAPP or equivalent document shall document the project goals and objectives, and shall define the QA procedures, QC specifications, and other technical activities that must be implemented to ensure that the results of the project or task to be performed will meet project objectives. The organization shall prepare a QAPP, or equivalent quality document, for all applicable projects and tasks involving environmental data collection, production, and use as listed previously in Clause 2.1.

7.5.1 General Requirements

All QAPPs and equivalent documents shall be reviewed and approved by the organization's QA Manager and their management (as applicable) before submission to EPA. The EPA QA Manager, or authorized representative as defined in the QMP, must then review and approve the QAPPs.

Moreover, all QAPPs must be approved prior to any data collection, production, or use except under circumstances explicitly defined in the terms and conditions of the external agreement. Under these circumstances, the organization will use or modify existing standard operating procedures (SOPs) and other quality-related documents available to support such activities. This complete collection of QA documents must be submitted to the EPA QA Manager together with a detailed explanation of the circumstances under which this emergency quality documentation was produced. This explanation must also address potential limitations in the interpretation of resulting data and information.

The organization shall ensure that QAPPs are kept current for the term of the agreement. When modifications are needed, the changes shall be made in a timely manner and communicated to all appropriate personnel. For multi-year projects, the organization's project officer shall confirm at least annually the continued relevance and applicability of the QAPP(s). If a revision is needed, the review and approval shall be conducted in the same manner as the original documents, including review and approval by EPA.

Other forms of project-level quality planning documents which are equivalent to an EPA QAPP shall be described with their review and approval processes in the organization's QMP. Other QAPP formats (e.g., UFP-QAPP) which satisfy the requirements of this Standard and its Annex B, and are approved by EPA, should also be described in the organization's QMP.

The QAPP shall describe the project-specific activities and proposed timeframe to accomplish the work defined in the terms and conditions of the agreement. The QAPP's period of approval shall be defined by EPA but not exceed the life of the agreement or five years. The specific requirements for QAPPs are contained in Annex B of this Standard and must be addressed by the organization in all individual QAPPs. Advice on how to satisfy the QAPP requirements in this Standard may be found in Handbook for Developing Quality Assurance Project Plans (EPA 2013d).
7.5.2 Combined QMP-QAPP

In some circumstances, EPA may recommend the preparation of a combined QMP and QAPP as a more appropriate means to document organizational level and project-level environmental data activities in external agreements, particularly when a full QMP is not needed. The specific elements to be addressed are included in the terms and conditions of the external agreement and are typically taken from Clause 7.2, Clause 7.5.1, and Annex B; however, the EPA organization shall determine the final list of elements to be addressed in the combined QMP-QAPP. As a combined QMP-QAPP is essentially an expanded QAPP, it is subject to the same review as any other QAPP. As it contains QMP elements, it must also be approved by the organization's QA Manager. The combined QMP-QAPP is subject to EPA review and approval prior to the start of work.

A combined QMP-QAPP should not be used for external agreements containing multiple projects to be determined over the life of the agreement (e.g., mission support contracts or program grants, etc.).

7.6 USE OF INFORMATION TECHNOLOGY METHODS AND SOURCES

7.6.1 General Requirements

Environmental data encompassed by this Standard may include results from the use of information technology (IT) methods and sources, such as services that collect and provide access to data and information. Such sources of these environmental data may include:

    * computer and mathematical models and the use of other electronic methods;
    * information systems; and
    * the Internet. 

Organizations shall develop and document in their QMP the general processes, procedures, and personnel responsibilities for evaluating the suitability of environmental data obtained from information technology sources. Project-specific processes, procedures, and evaluation criteria shall be developed and documented in a QAPP or other equivalent quality documentation as defined in the organization's QMP. Environmental data from the use of IT methods and sources shall be assessed by the user to confirm its suitability. Any limitations or restrictions on the use of such data shall be documented and reported with the data (see Clause 7.9).

Electronic media upgrades or use of new software shall further require that all historical information is carried forward to meet retention schedule requirements and that original entries are maintained (e.g., results of calculations are consistent), information is not lost or corrupted, and that the information remains accessible.

7.6.2 Computer Models, Mathematical Models, and Use of Other Electronic Methods

The development, evaluation, and application of mathematical or computerized models of environmental processes and conditions require QAPPs. Environmental data produced from computer models, mathematical models, and produced by using other electronic methods (or software) shall be reviewed, evaluated, verified, and validated in order to confirm their suitability for use (see Clause 7.9 and Annex B). The specific process and procedure shall be documented in applicable quality documentation and shall be reviewed and approved for use.

7.6.3 Other Information Sources, Systems, and the Internet

Quality principles applied to environmental data collected from other information sources (including the Internet) shall require special attention (e.g., due diligence) to ensure that the data can be used for their intended purpose. Project specific evaluation criteria shall be developed and documented in the applicable QAPP. Data standards and metadata shall be used where applicable and documented to ensure that supplied data retain their meaning for appropriate interpretation by the end-user.
Any limitations, such as inconsistency with EPA environmental data standards requirements, or restrictions on the use of such data shall be documented and reported with the data.

7.6.4 Control of Computer Software and Hardware

The organization shall ensure that computer software and hardware relevant to the mission and objectives of the organization are controlled appropriately and satisfy the organization's objectives. The organization shall describe in its QMP the processes and the roles, responsibilities, and authorities of management and staff for the control of computer software and hardware. (See also Annex A Clause A3.9).

7.7 SYSTEMATIC PLANNING

Any activities involving the collection, production, or use of environmental data shall be planned, implemented, controlled and documented using a systematic planning process to develop acceptance or performance criteria for all work (a plan-do-check-act approach). At a minimum, this process shall:

   * identify all relevant customers and their expectations;
   * identify the technical and quality goals that meet the needs and expectations of the customer;
   * translate the technical and quality goals into specifications that will produce the desired result;
   * consider any cost and schedule constraints within which project activities are required to be performed; identify measurable acceptance criteria or performance measures by which the results will be evaluated and the suitability of the environmental data for their intended use is determined; and
   * result in a documented plan for collection, production, or use of environmental data.

The general planning processes (including applicable design steps) shall be described in the organization's QMP. All planning documentation shall be reviewed and approved for implementation by authorized personnel before the affected planned work commences. Such documentation includes confirmation of conformance with EPA environmental data standards, work plans, schedules, SOPs (or references to where they may be obtained), and QAPPs. In some cases, it may be appropriate to subject general planning documents to independent review (e.g., internal or external peer review) in order to confirm their suitability. The reviews of other planning documents do not replace the required reviews of quality documents (e.g., QMPs, QAPPs). Additional information about the systematic planning process can be found in Annex C.

7.8 ASSESSMENT, OVERSIGHT, AND RESPONSE

Management shall determine during planning the level of routine oversight appropriate for the particular project and shall assign oversight responsibilities to project personnel qualified to perform this oversight function. Personnel having authority to issue stop-work orders shall be identified and documented in the applicable QAPP. During this oversight attention should be given to interim assessments of data quality and the QA/QC used to ensure the integrity of the data.

Assessment findings shall be communicated to the QA Manager of the organization in a written report. Nonconforming methods or instruments or unsatisfactory interim assessments of data quality shall be reported to the organization responsible for them and on a timely basis to ensure the implementation of effective corrective actions.

When warranted by findings of deficiency or nonconformity with requirements (e.g., deviation from approved protocols, poor performance of the measurement systems), appropriate corrective actions shall be taken promptly. The adequacy and effectiveness of the corrective actions shall be confirmed, verified, communicated to participants, and documented. Results obtained from nonconforming methods or instruments shall be evaluated to determine the impact of the nonconformity on the quality of the data. The adequacy and effectiveness of corrective actions taken shall be confirmed, verified, and documented.

Clause 7.3 requires review of the QMS. Internal assessment and response should include any results of project-level oversight that affect programs covered by the QMS.

7.9 DATA REVIEW, VERIFICATION AND VALIDATION, AND DATA USABILITY REPORTING

7.9.1 General Requirements

Results obtained from products or services involving environmental data shall be reviewed, verified, validated, and qualified according to their intended use. The general processes and roles and responsibilities of personnel relative to the assessment and verification of data usability shall be described in the organization's QMP. Specific processes and roles and responsibilities of personnel pertaining to assessment of individual projects shall be described in the applicable QAPPs.

The data shall be evaluated according to approved and documented procedures and shall be qualified appropriately before use. Any limitations or restrictions on the use of the data shall be expressed quantitatively to the extent practicable and shall be documented in any reporting of the data.

Environmental data shall satisfy applicable data reporting requirements for data definition and source reporting and for review and certification (when necessary).

Any project reports that contain data, or report the results of environmental data operations (e.g., data summaries), shall be reviewed independently (i.e., by others than those who produced the data or the reports) to confirm that the data or results are presented correctly. These project reports must have a readily identifiable section that describes the applied QA and QC and discusses the usability of the data.

The reports shall be reviewed internally by the responsible project manager and QA Manager (or authorized representative defined in the QMP) to confirm that the report's conclusions are supported by sound science and that the data quality objectives have been met. This review shall be completed and any issues resolved prior to submission to EPA or any applicable external peer review, release, publication, or dissemination.

7.9.2 Peer Review

Organizations may use peer review as an assessment tool. Peer review is a documented critical review of a specific scientific or technical work product. When applied, peer review shall be planned, implemented, and documented in accordance with the organization's peer review policies and procedures (EPA may apply its peer review process at interim or final stages of the project).

For modeling projects, peer review is an important tool. To maximize its value to modeling projects, peer review and advice should begin as early in the conceptual development, modification, or application phase for models as possible (EPA 2009). Peer review, and in particular external peer review, involves significant time and resources so allocations must be incorporated into components of the project planning and any contracts.

Peer review in the early stages of model development, modification, or application helps to evaluate the conceptual basis of models and potentially save time by redirecting misguided initiatives, identifying alternative approaches, or providing strong technical support for EPA positions. Peer review in the later stages of model development is useful as an independent external review of model code (i.e., model verification).

7.9.3 Review of Data for Potential Use

Environmental data collected from sources that did not use a QMS equivalent to this Standard or whose quality is unknown or undocumented shall be reviewed and evaluated to confirm their suitability for use for this particular project. Data collected for purposes other than that of the current project shall also be reviewed and evaluated for their suitability for use.
These data, which sometimes have been called "data for secondary use" or simply (but inaccurately) "secondary data," shall be assessed according to approved and documented procedures and shall be qualified appropriately before use.

Some of the elements in the review of environmental data for potential use include:
      * the need to satisfy requirements for objectivity, utility, and integrity;
      * the principal traditional indicators of environmental quality (e.g., precision, bias, representativeness, comparability, completeness, and sensitivity); and
      * the general usability factors (e.g., relevance, temporal compatibility, spatial inclusivity, credibility, aggregation level, compatibility, reproducibility, and degree of augmentation).

These elements shall be compared to the requirements established by the acceptance criteria defined in the approved QAPP (see Annex B and EPA 2013d). The documentation of how well these elements met the acceptance plan then becomes part of the metadata that would accompany the data under consideration if accepted for use. If the data are not acceptable, the information regarding that determination shall be documented. If the data are only marginally acceptable, the rationale for any decisions derived from the use of such data shall be documented.

7.10 DOCUMENTS AND RECORDS MANAGEMENT

Sufficient records shall be specified, prepared, reviewed, authenticated, and maintained to reflect the achieved level of quality for completed work and/or to fulfill all records requirements as needed by EPA.

The processes for maintenance of records shall be documented and shall include provisions for retention, protection, preservation, traceability, and retrieval. Where evidentiary records are involved, the maintenance of records shall also include establishing and implementing appropriate management controls, tracking, chain of custody, and confidentiality procedures (including the handling of information identified as sensitive or confidential for the affected records). Retention times for records shall be determined based on external agreement and statutory requirements and protected from damage and deterioration.

8.	ROLES AND RESPONSIBILITIES
8.1 GENERAL MANAGEMENT ROLES AND RESPONSIBILITIES
THE LEADERSHIP (OR senior management) of the organization is responsible for implementing a QMS that covers all environmental programs for which the management is accountable. Senior management includes the principal executive responsible and accountable for mission accomplishment and overall operations of the organization.

The QMP must be approved by the senior manager of the organization and be approved and signed by the QA Manager of the organization. The senior manager may require the concurrence on the QMP by appropriate subordinate line managers to encourage the acceptance and implementation of the QMS. The QMP and the QAPP shall be approved by EPA before the start of work.

The following clauses list specific roles and responsibilities for quality management personnel and managers.

8.2 QUALITY MANAGEMENT PERSONNEL

Quality management personnel (e.g., QA Manager, Quality Assurance Officer, Quality Assurance Coordinator) refers to individuals within the organization who are assigned specific quality management duties and are delegated authority for quality management as defined in the organization's QMP.



The organization may assign and deploy other quality management personnel to support the day-to-day quality functions listed below. In some situations, such quality staff may be delegated specific quality management responsibilities as described in this Standard, including approval of quality documentation such as QMPs, QAPPs, SOPs, etc. A description of the coordination and deployment of all quality management personnel, including their roles and responsibilities, shall be documented in the organization's QMP.

The functions of the quality management personnel may be totally related to quality management system activities or be in conjunction with other functions and responsibilities within the organization. If these personnel have other concurrent functions to perform, there must be no conflict of interest. Specific duties and responsibilities of all quality management personnel shall be documented in the organization's QMP. Quality management personnel shall:
   * review and approve QMPs, QAPPs, and other quality documents prior to submission to EPA;
   * provide oversight of QA and QC implementation in the environmental programs conducted by or for the organization; and
   * perform assessments of implemented environmental programs when needed to confirm the effective application of QC and QA practices, report results to management, and confirm the effectiveness of corrective actions.

8.3 TECHNICAL MANAGERS AND STAFF

Technical managers at all levels shall:
   * ensure that all organizational components and programs comply fully with the requirements of this Standard;
   * ensure that all work specified in the applicable external agreement for which the manager is responsible complies fully with the requirements of this Standard; and
   * perform all other applicable quality management roles and responsibilities assigned to them in their organization's QMP.

Staff members shall:
   * ensure that all work specified in the applicable external agreement for which the staff member is responsible complies fully with the requirements of this Standard;
   * ensure that the results of environmental programs are of sufficient quantity and adequate quality for their intended use; and
   * perform all other quality management roles and responsibilities assigned to them in their organization's QMPs.

9.	DEFINITIONS
Approved: The documented determination that the proposed quality document is suitable for the intended purpose and meets the requirements specified in the applicable Quality Standard.

Assessment: The evaluation process used to measure the performance or effectiveness of a system and its elements. As used here, assessment is an all-inclusive term used to denote any of the following: audit, performance evaluation, management review, peer review, inspection, or surveillance.

Chief Information Officer (CIO): The Assistant Administrator for the OEI (AA/OEI) who is also the EPA Senior Management Official for Quality.

CIO Procedure: The required steps, course of action, or processes needed to accomplish or satisfy a policy.

Competence: The ability to apply knowledge, skills, and experience to an activity to achieve intended results.

Conformity: The fulfillment of a requirement.

Corrective Action: An action to eliminate the cause of a nonconformity or undesirable situation and to prevent recurrence.

Data Standard: A documented consensus-based agreement on the format and definition of data.

Data Usability: The process of determining and ensuring that the quality of the data produced meets the intended use of the data.

Environmental Data: Any data or information pertaining to the environment that describe measured outputs from processes; environmental conditions in a specific location; ecological effects and consequences; health effects and consequences; biological, chemical, and radiological conditions; or the performance of environmental technology. For EPA, environmental data include information collected directly from measurements, produced from models, and compiled from other sources such as databases, information systems, literature, or the Internet.

Environmental Data Collection: The process of acquiring or gathering environmental data through various means including (but not limited to) sampling and analysis activities, retrieval from information systems, literature, and received from EPA partners and the regulated community.

Environmental Data Operations: The work performed to collect, produce, or use environmental data.

Environmental Data Production: The process of generating environmental data through various means including (but not limited to) the use of measurement instrumentation, information technology, computer models, and data analysis tools (e.g., statistics, risk assessment methods).

Environmental Programs: The activities involving the environment, including, but not limited to: characterization of environmental processes and conditions; environmental monitoring; environmental research and development; the design, construction, and operation of environmental technologies; and laboratory operations on environmental samples.

Equivalent Document: A set of documents that contains all the information and management controls (signatures) as the required documents used in the Standard.

Extramural Agreement: A legal agreement between EPA and a non-EPA organization for the acquisition of items or services by EPA or financial assistance to a non-EPA organization. Such agreements include acquisition agreements (e.g., contracts, work assignments, delivery orders, task orders), assistance agreements (e.g., cooperative agreements, research grants, State and local grants), and EPA-funded Interagency Agreements (IAs) with other governmental entities.

Graded Approach: The process of basing the level of application of managerial controls applied to an item or work according to the intended use of the results and the degree of confidence needed in the quality of the results.

Guidance: A non-mandatory compilation of advice and example procedures and documentation, based on past experience. Guidance may supplement CIO procedures.

Handbook: A non-mandatory book of instruction, guidance, or information reference manual.

Information: For purposes of this Standard, information means any communication or representation of knowledge such as facts or data, in any medium or form, including, but not limited to, textual, numerical, graphic, cartographic, narrative, or audiovisual forms.

Information System: An organized collection, storage, and presentation system of data for decision making, progress reporting, and for planning and evaluation of programs.
Information Technology: The study, design, development, implementation, support, or management of computer-based information systems, particularly software applications and computer hardware.

Integrity (Information): Assurance that the information is protected from unauthorized access or change and is not compromised through corruption or falsification.

Life Cycle: The life span of a product or service from its initial planning and development, to its use and maintenance, and to its final closure or disposal.

Management Controls: A system of management functions to enable managers to determine that the operations of an organization satisfy predetermined goals and objectives, that performance is in line with standards and specifications, and to implement any remedial actions needed to ensure that human and other resources are being used in the most effective and efficient way.

Management System: A system to establish policy and objectives and to achieve those objectives (ISO 9001). A management system may describe the policies, objectives, principles, authority, responsibilities, accountability, and implementation plan of an organization for conducting work and producing products and services. Management systems include ISO 9001 on quality management, ISO 14001 on environmental management, and OHSAS 18000 on occupational health and safety.

Material Transfer Agreement (MTA): A contract that governs the transfer of tangible research materials between two organizations when the recipient intends to use it for its own research purposes.

Metadata: The information about data required to facilitate its use, understanding and management. For example, metadata should answer questions about data such as why they were collected, how they were collected and by whom, what was done to the data, what they were used for, what were their limitations, and what were the acceptance criteria.

Normative Annex: A part of a Standard that gives provisions (requirements) additional to those in the body of the Standard. A normative annex is considered to be a part of the Standard.

Normative Element: An element that describes the scope of a Standard and which sets out provisions (requirements) of the Standard.

Organization: A company, corporation, firm, enterprise, or institution, or part thereof, incorporated or not, public or private, that has its own functions and administration.

Objectivity (Information): The assurance that information is presented in an accurate, clear, complete, and unbiased manner, and, as a matter of substance, is accurate, reliable, and unbiased.

Policy: A high-level statement about an Agency requirement designed to influence and determine decisions, actions, and other matters.

Prime Recipient: An organization that directly receives funding, support, or authorization to implement a project or program.

Procedure: The required steps, course of action, or processes needed to accomplish a task.

Process: A set of interrelated resources and activities which transforms inputs into outputs. Examples of processes include analysis, design, data collection, operation, fabrication, and calculation.

Product: The intended result or final output of an activity or process that is disseminated or distributed among EPA organizations or outside of EPA.

Quality: The totality of features and characteristics of a product or service that bear on its ability to meet the stated or implied needs and expectations of the user.

Quality Assurance (QA): A management or oversight function that deals with setting policy and running an administrative system of management controls that cover planning, implementation, review, and maintenance to ensure products and services are meeting their intended use.

Quality Assurance Manager (QA Manager): The individual designated as the principal manager within the organization having management oversight and responsibilities for planning, documenting, coordinating, and assessing the effectiveness of the QMS for the organization. NOTE: Other personnel having QA or QC duties may be referred to as QA Officer and QA Coordinator.

Quality Assurance Project Plan (QAPP): A document describing in comprehensive detail the necessary QA, QC and other technical activities that must be implemented to ensure that the results of the work performed will satisfy the stated performance objectives and criteria.

Quality Control (QC): The overall system of technical activities that measure the attributes and performance of a process, item, or service against defined standards to verify that they meet the stated requirements established by the customer; operational techniques and activities that are used to fulfill requirements for quality.

Quality Management: That aspect of the overall quality management system of the organization that determines and implements the quality policy. Quality management typically includes strategic planning, allocation of resources, and other systematic quality activities for the organization.

Quality Management Plan (QMP): A formal document or manual that describes the QMS in terms of the organizational structure, functional responsibilities of management and staff, lines of authority, and required interfaces for those planning, implementing, and assessing all activities conducted.

Quality Management System (QMS): A structured and documented management system describing the policies, objectives, principles, organizational authority, responsibilities, accountability, and implementation plan of an organization for ensuring quality. The QMS provides the framework for planning, implementing, documenting, and assessing work for carrying out required QA and QC activities.

Required Element: An element whose presence is obligatory in a Standard.

Requirement: An expression of the content of a Standard conveying a criterion to be fulfilled if compliance is to be claimed and from which no deviation is permitted.

Standard: An accepted, consensus-based specification which defines systems, processes, methodologies, or practices. It provides a basis for assuring consistent and acceptable minimum levels of quality, performance, safety, and reliability. Standards usually are included in or accompany procedures.

Standard Operating Procedure (SOP): A written document that details the method for an operation, analysis, or action with thoroughly prescribed techniques and steps, and that is officially approved as the method for performing certain routine or repetitive tasks.

Sub-award: A legal instrument that provides support for the performance of any portion of the project or program award in an external agreement that the recipient awards to an eligible sub-recipient. A sub-award may be provided through any legal external agreement.

Sub-recipient (Sub): Generally an external organization that receives a sub-award from a prime recipient and that is accountable to the prime for products and/or services under that agreement.


Transparency (Information): The assurance that information is supported by sufficient metadata.

Usability Assessment: The evaluation of data based upon the results of data verification and validation for the decision being made. Reviewers assess whether the process execution and resulting data meet quality objectives based on the criteria given in the QAPP.

User: The customer or organization for whom the results or products were collected or created.

Utility (Information): The assurance that information is useful for its intended purpose.

Validation (Information): The confirmation by examination and provision of objective evidence that the particular requirement for which the information is intended is fulfilled; the process of determining whether the specifications were appropriate and that the verified results will meet the data user's needs.

Verification (Information): The confirmation by examination and provision of objective evidence that validated information fulfills specified requirements; the process of checking whether the information met the project's specifications.

10.	WAIVERS
Statutory requirements for quality may supersede the specifications in this Standard or be more rigorous. In such cases, affected programs shall be exempt from certain requirements in this Standard. EPA organizations conducting exempted activities shall comply with this Standard in all other respects. The following exemptions from these requirements apply:

   * The collection of environmental data under the authority of Good Laboratory Practices as defined by 40 CFR 792, for the Toxic Substances Control Act.
   * The collection of environmental data under the authority of Good Laboratory Practices as defined by 40 CFR 160, for the Federal Insecticide, Fungicide, and Rodenticide Act.
Requests for exemptions should be directed to the Office of Environmental Information (OEI) Quality Staff. Exemptions shall be made by the CIO (EPA Senior Management Official for Quality).
Approved waivers will be documented by the requesting organization and the waiver will be added to their Quality Management Plan.

11.	REFERENCES
EPA 2002a. Guidelines for Ensuring and Maximizing the Quality, Objectivity, Utility, and Integrity of Information Disseminated by the Environmental Protection Agency. EPA/260R-02-008 (October 2002 with Addenda in 2004 and 2005). www.epa.gov/quality/informationguidelines

EPA 2002b. Guidance for Choosing a Sampling Design for Environmental Data Collection (EPA QA/G-5S), Office of Environmental Information, U.S. Environmental Protection Agency. http://www.epa.gov/quality

EPA 2002c. Environmental Data Verification and Validation (EPA QA/G-8), Office of Environmental Information, U.S. Environmental Protection Agency. http://www.epa.gov/quality

EPA 2005. Uniform Federal Policy for Quality Assurance Projects Plan Workbook, Intergovernmental Data Quality Task Force, U.S. Environmental Protection Agency and U.S. Department of Defense. http://www.epa.gov/fedfac/documents/qualityassurance.htm



EPA 2006a. EPA Quality Policy, CIO Policy 2106. U.S. Environmental Protection Agency, Washington, D.C. http://www.epa.gov/irmpoli8/policies/21060.pdf

EPA 2006b. EPA Procedure for Quality Policy, CIO Procedure 2106-P-01.0. U.S. Environmental Protection Agency, Washington, D.C. http://www.epa.gov/irmpoli8/policies/21060.pdf

EPA 2006c. Peer Review Handbook. 3rd ed. EPA-100-B-06-002. Prepared for the U.S. Environmental Protection Agency by members of the Peer Review Advisory Group, for EPA's Science Policy Council. Washington, D.C: U.S. Environmental Protection Agency. http://epa.gov/peerreview/pdfs/Peer%20Review%20HandbookMay06.pdf

EPA 2006d. Data Quality Assessment: Statistical Methods for Practitioners, (EPA QA/G-9S), Office of Environmental Information, U.S. Environmental Protection Agency. http://www.epa.gov/quality

EPA 2006e. Guidance on Systematic Planning Using the Data Quality Objectives Process, (EPA QA/G-4), Office of Environmental Information, U.S. Environmental Protection Agency. http://www.epa.gov/quality

EPA 2009. Guidance on the Development, Evaluation, and Application of Environmental Models. Council for Regulatory Environmental Modeling, EPA/100/K-09/003, U.S. Environmental Protection Agency, Washington, DC. http://www.epa.gov/crem

EPA 2013a. Quality Standard for Environmental Data Collection, Production, and Use by EPA  Organizations, CIO Standard 2106-S-01 U.S. Environmental Protection Agency, Washington, DC. http://www.epa.gov/quality

EPA 2013b. Quality Standard for Environmental Data Collection, Production, and Use by Non-EPA (External) Organizations, CIO Standard 2106-S-02 U.S. Environmental Protection Agency, Washington, DC. http://www.epa.gov/quality

EPA 2013c. Handbook for Preparing Quality Management Plans, U.S. Environmental Protection Agency, Washington, DC. http://www.epa.gov/quality

EPA 2013d. Handbook for Developing Quality Assurance Project Plans, U.S. Environmental Protection Agency, Washington, DC. http://www.epa.gov/quality

Gilbert 1987. Statistical Methods for Environmental Pollution Monitoring, John Wiley, New York.

Ginevan, Michael E. & Splitstone, Douglas E., 2003. Statistical Tools for Environmental Quality Measurement, Chapman & Hall/CRC, Boca Raton, FL.

Manly, Bryan F.J., 2001. Statistics for Environmental Science and Management. Chapman & Hall/CRC, Boca Raton, FL.

      
12.	MATERAL SUPERSEDED
This Standard replaces EPA/QA R-2 and EPA/QA R-5 in their entirety.




13. ADDITIONAL INFORMATION
Other information and documentation on the EPA Quality Program may be found at http://www.epa.gov/quality.
The following Annexes contain detailed specifications and information that clarify the requirements and use of this Standard.

These are:
   *          Annex A (Normative)  -  Requirements for EPA Quality Management Plans for Non-EPA Organizations
   *          Annex B (Normative)  -  Requirements for EPA Quality Assurance Project Plans in External Agreements
   *          Annex C (Informative)  -  Guidelines on Planning for Projects in External Agreements
--------------------------------------------------------------------------------
                  Malcolm D. Jackson, Assistant Administrator and Chief Information Officer, Office of Environmental Information


                                                        EPA Classification No.:
DRAFT FINAL
(2106-S-02.0 Annex A)
                                                             CIO Approval Date:

                                                           CIO Transmittal No.:

                                                                   Review Date:
09/10/12 

                              ANNEX A (NORMATIVE)
      REQUIREMENTS FOR QUALITY MANAGEMENT PLANS FOR NON-EPA ORGANIZATIONS
 
 A1. Introduction

Clause 7.2 in CIO Standard 2106-S-02 (EPA 2013b) requires external organizations that collect, produce, or use environmental data to describe and document their quality management system (QMS) in a Quality Management Plan (QMP) or equivalent document acceptable to EPA.

The QMP is unique to an organization and shall describe the management and technical processes and procedures necessary to plan, implement, document, and assess the effectiveness of the organization's QMS applied to the work prescribed by the external agreement. Since non-EPA organizations may participate in more than one external agreement with EPA, the organization's QMP may contain common management information that may be applied to several agreements and not require an individual QMP for each agreement. The specific quality assurance (QA) and quality control (QC) procedures will be found in the Quality Assurance Project Plan (QAPP) for each external agreement (See Annex B).

The QMP defines an organization's quality-related elements that represent the core of the QMS including:

   * management controls, policies, and objectives;
   * processes for and areas of applying quality procedures; and
   * roles, responsibilities, and authorities of managers and staff.

The QMP, therefore, is a management document that should be appropriately tailored to the needs of the organization. The QMP must be sufficiently inclusive, explicit, and readable to enable managers, supervisors, and staff to understand the priority that senior management places on QA, the established QA policies and procedures, and their respective QA roles. The QMP must be constructed and written so that an assessment can be made of the QMS's effectiveness following implementation. This enables managers to determine whether the QMS is being implemented in a way that ensures successful results can be obtained. The QMP shall focus on the processes used to plan, implement, document, and assess the programs to which it is applied. The level of detail should be based on a common sense, graded approach that establishes QA and QC requirements commensurate with the importance of the work, the available resources, and the unique needs of the organization.

QMPs shall be modified as the requirements change. This Annex describes the quality management elements that are normally considered to be critical to an effective QMS. A non-EPA organization implementing an external agreement shall evaluate these core elements to determine their applicability to its QMS for environmental data collection, production, and use. If additional quality management elements are useful or necessary to adequately describe the QMS, these elements shall be developed and discussed in the QMP.

Guidelines for satisfying these requirements may be found in Handbook for Preparing Quality Management Plans (EPA 2013c).

 A2. Preparation, Submission, Review, and Approval

 A2.1 Preparation Responsibility

The senior management of the non-EPA organization is responsible for the preparation of a QMP that covers all work specified by the applicable external agreement and for which the organization's management is accountable. Senior management includes the senior executive responsible and accountable for mission accomplishment and overall operations. Senior management shall ensure that the QMS documented in the QMP conforms to the requirements of this Standard.

 A2.2 Non-EPA Organization QMP Submission and Approval

A QMP must be approved and signed by the senior manager in the organization prior to being submitted to EPA for review and approval. The QMP must also be approved and signed by the QA manager (or equivalent position) in the organization.

The senior manager may require the concurrence on the QMP by appropriate subordinate line managers to encourage the acceptance and implementation of the QMS.

 A2.3 EPA Review and Approval of the QMP

Following internal organization approval, the non-EPA QMP must be submitted to the EPA organization responsible for the external agreement for review and approval. Each QMP will be reviewed to determine compliance with general EPA Quality requirements including conformance with this Standard. QMPs that adequately address all program-level elements described in Clause 7 of this Standard should be approved and be valid for a period not to exceed the term of the external agreement.

 A2.4 QMP Revisions

A non-EPA QMS must be reviewed at least annually by the organization to confirm its continued accuracy and relevance in describing the QMS. This review must include an evaluation of the effectiveness of the QMP. When necessary, the organization shall revise its QMP to incorporate changes to the QMS. Revisions to a QMP may become necessary due to:

   * a change in the scope (i.e., statement of work) of the external agreement,
   * significant changes in the non-EPA organization's mission or structure, or
   * any major change to the organization's QMS.

A revised QMP may be submitted to EPA at any time and is subject to the same review and approval processes as the original QMP. All personnel performing work for the organization shall be notified of changes to the QMS and the QMP.

 A3. QUALITY MANAGEMENT PLAN REQUIREMENTS

 A3.1 General Requirements

The QMP shall address all of the requirements pertaining to environmental data collection, production, and use as given in Clause 7 of this Standard.

The QMP shall document all quality-related terms and conditions specified in the external agreement and describe their implementation.

The QMP shall contain descriptions of the processes by which the organization plans, implements, documents, and determines the effectiveness of its quality management procedures, including QA and QC activities, to help management to obtain results of its technical work that are of the type and quality needed for their intended use. 
Information about the organization's structure and mission should be included in the QMP to provide context for the external agreements. A brief description of the organization's background and history may be included in the QMP for completeness.

The QMP should reflect the organization's commitment to quality management principles, and may be tailored by the organization to meet its needs.

To reduce duplication between QMPs and QAPPs, the QMP shall include discussion of those activities, policies, and procedures that are common to projects governed by other external agreements (e.g., multiple contracts, work assignments, assistance agreements). Such discussions provide an "umbrella" under which individual project activities may be performed.

The remainder of Annex A, Clause A3, provides the elements that must be addressed in the organization's QMP. If the organization believes that an element does not apply to its QMS and should be omitted, the QMP must contain an explanation of why it does not apply.

The QMP shall always include:

   * title page;
   * table of contents;
   * evidence of the review, approvals page, and revision history of the QMP; and
   * applicable references.

A3.2 Conformance with Policies, Procedures, and Standards

The QMP shall identify all EPA Policies, Procedures, and Standards (and any internal organization policies and procedures to which the QMS conforms).

 A3.3 Organizational and Program Field of Application

The QMP shall identify and describe:

   * all parts of the organization to which the QMP applies, and
   * all business lines, programs, or actions involving the collection, production, and use of environmental data to which the QMS applies.

A3.4 Organization, Responsibility, and Authority

The QMP shall include the following:
   * a statement of the organization's policy on QA, including the importance of quality in its environmental data products and services to the organization and its mission, and why; the general objectives/goals of the QMS; and the policy for resource allocation for the QMS, including personnel, intramural and extramural funding, and QA-related travel funding for EPA-related work;
   * an organization chart that identifies all of the components of the organization and, in particular, the organizational position and lines of reporting for the QA Manager (and any QA staff) that confirm and document the independence of the QA Manager from groups collecting, generating, using, and evaluating environmental data; 
   * a discussion of the organizational structure of the QMS and the responsibilities and authorities of the QA Manager and any other QA staff;
   * a discussion of the specific quality management controls required by the QMS and the work to which the controls apply; where oversight of delegated, contracted, or other external programs is needed to ensure that objectives for data quality and usability are attained; and where internal coordination of QA and QC activities among organizational units needs to occur;

   * a discussion of the QA and QC roles and responsibilities of line management, technical staff, and any other staff, and how accountability is defined;
   * a discussion of the organization's process for resolving disputes regarding QMS requirements, QA and QC procedures, assessments, or corrective actions;
   * a discussion of how management shall ensure that applicable elements of the QMS are understood and implemented in all programs involving environmental data; and
   * an approval page for the signatures of the senior manager, senior line management (as appropriate), and the QA manager of the organization. This approval page may be part of a title page or a separate sheet following the title page.

 A3.5 QMS Description

The QMP shall provide a description of the QMS and its implementation in the organization. The QMP shall describe the principal components in the QMS and how they are used to implement the QMS.

These components include, but are not limited to,
   * QMS documents (e.g., Division or subordinate organization QMPs);
   * management assessments (self and independent); 
   * staff competence; 
   * systematic planning processes; 
   * project-specific quality documentation (e.g., QAPPs or equivalent documentation);
   * standard operating procedures (SOPs);
   * technical audits/assessments (self and independent); and
   * data assessments.
   
This discussion shall also identify how and when the components of the QMS are to be applied to individual projects and tasks in the external agreements.

A3.6 Personnel Competence

The QMP shall include a statement of the organization's policy for ensuring the competence of managers and staff to implement the QMS effectively and to provide training to ensure continued effectiveness by personnel.

The QMP discussion shall include the processes and the staff responsible for:
   * identifying statutory, regulatory, or professional certifications that may be required to perform certain operations; and
   * identifying, designing, performing, and documenting technical, quality, and project management training.

The QMP shall also describe how staff proficiency in critical technical disciplines is maintained and documented.

 A3.7 External Agreements

The QMP shall describe the organization's process for ensuring that appropriate quality requirements are included and implemented in all external agreements subject to this Standard. The QMP shall also discuss the organization's process for sub-agreements with other entities (e.g., subcontractors) for support to the statement of work in applicable external agreements.

 A3.8 Documents and Records Management

The QMP shall describe (or provide a reference to) the processes by which the requirements contained in Clause 7.10 of this Standard are met, including the responsibilities and authorities of management and staff.
In general, the QMP shall address the documents and records processes:
   * for identifying quality-related documents and records requiring control;
   * for handling documents and records to ensure their accessibility, protection from damage and deterioration, and means of retention, including discussion of the roles and responsibilities for management and staff;
   * by which all technical guidance documents are prepared, reviewed, approved, issued, used, and revised;
   * by which all planning documents (e.g., QAPPs, Sampling and Analysis Plans) are prepared, reviewed, approved, issued, used, and revised;
   * that ensure compliance with all statutory, contractual, and assistance agreement requirements for records from environmental programs and that provide adequate preservation of key records necessary to support the mission of the organization;
   * for formal and informal reviews of products (e.g., reports, peer reviewed journal articles); and
   * for reviews to confirm that performance and acceptance criteria were established and documented.

The QMP shall describe how documents and records, including revisions, are reviewed for conformance with QMS requirements and with the terms and conditions of external agreements, and are approved by authorized personnel before general use.

The QMP shall describe or provide a reference to the management process that ensures that records accurately reflect completed work and/or fulfill statutory and contractual requirements, including any specific record keeping requirements defined in applicable EPA policies and procedures. The maintenance of records includes defining requirements and responsibilities for record transmittal, distribution, retention, protection, preservation, traceability, disposition, and retrieval.

The QMP shall identify how the disposition of records, in accordance with regulatory requirements, schedules, or directives from senior management, is accomplished.

 A3.9 Use of Information Technology Methods and Sources

The QMP shall describe how the organization satisfies the requirements of Clause 7.6 of this Standard and shall include the responsibilities of management and staff (Clause 8).

 A3.10 Planning

The QMP shall describe how the organization satisfies the planning requirements given in Clause 7.7 of this Standard, including the responsibilities and authorities of management and staff. 

The descriptions shall address:
   * planning and scoping;
   * design of environmental data collection or acquisition;
   * design of model development, evaluation, and application; and
   * documentation of environmental data collection or acquisition and model design.

The QMP shall describe the process for communications between the organization and any suppliers to ensure that there is a clear understanding by all participants of the needs and expectations of the organization and the products or results to be provided by the supplier. The QMP shall describe how the results of planning are documented in a QAPP or equivalent document and are approved by authorized personnel for implementation.
The QMP shall also describe or discuss the process for how information or data obtained from sources outside EPA that did not use an EPA-approved QAPP (or equivalent planning document) for data collection shall be evaluated and qualified for use.
The QMP shall describe the review process for qualifying such data, including the use of data for purposes other than for which the data were originally collected.

 A3.11 Implementation of Work

The QMP shall describe how the organization satisfies the implementation of work requirements given in Clause 7.7 of this Standard, including the responsibilities and authorities of management and staff.

This description shall address and include processes for:
   * identification of operations needing procedures;
   * documentation of implementation procedures (e.g., reference methods, SOPs);
   * testing and evaluation of procedures to confirm their acceptable performance;
   * ensuring that work is performed according to approved plans;
   * deviations and waivers from approved procedures;
   * use of measurement and testing equipment and models;
   * use of data obtained from other sources;
   * ensuring the integrity of samples and data; and 
   * monitoring of performance.

The QMP shall also describe how appropriate management controls for the release, change, and use of planned procedures are implemented. Such management controls provide for the necessary approvals, specific times and points for implementing changes, removal of obsolete documentation from work areas, and verification that the changes are made as prescribed.

The QMP shall contain the organization's process for identifying the need for SOPs, the process for developing SOPs, and the policy for using SOPs. The QMP shall also describe the process by which SOPs are reviewed for initial and subsequent use.

The QMP shall also describe the organization's process for deviations and waivers from approved procedures and the necessary reviews and approvals.

 A3.12 Assessment, Oversight, and Response

The QMP shall describe how the organization satisfies the requirements for assessment, oversight, and response as given in Clause 7.3 of this Standard, including the responsibilities and authorities of management and staff (Clause 8).

This description shall address and include processes for:
   * routine oversight;
   * identifying appropriate assessment methods for use;
   * ensuring the competence of personnel conducting oversight and assessments;
   * ensuring the objectivity, independence, and competence of assessment personnel;
   * assessment planning and implementation;
   * assessment reporting; and
   * corrective action and verification.

The QMP shall describe how assessment methods are chosen and the expected frequency of their application to environmental programs. In particular, the QMP shall document how senior management assesses the adequacy of the QMS.

The QMP shall describe how the competence of personnel conducting assessments is determined. Personnel conducting assessments shall be competent, based on project-specific requirements, to perform the assigned assessment.

Management is responsible for choosing the assessors, defining acceptance criteria, approving audit procedures and checklists, and identifying goals prior to initiation of an assessment. Assessors shall be technically knowledgeable with no real or perceived conflict of interest. If the assessors are chosen from within the organization, they must have no direct involvement or responsibility for the work being assessed. The exception to this requirement is for self-assessments, which must be identified as such.

The QMP should also discuss the authority for and conditions under which "stop work" orders may be needed. It should also discuss when and how such decisions shall be made, and with whom any coordination will be needed.

The QMP shall describe how management will respond to the results (or findings) and recommendations from assessments. The QMP will indicate how follow-up action shall be taken and documented to confirm the implementation and effectiveness of the response action. In addition, processes for identifying, trending, and correcting common non-conformances found in different parts of the organization will be described to ensure QMS improvement. The QMP shall also describe how disputes, if encountered, as a result of assessments are resolved and by whom.

 A3.13 Quality Improvement
 
 The QMP will describe:
   * the organization's process for ensuring continual quality improvement, including the management process for determining, planning, implementing, and evaluating the effectiveness of quality improvement activities including:
         o  who (organizationally) is responsible for quality improvement; and
         o the corrective action program to ensure that conditions adverse to quality are
               #  identified promptly and
               # corrected as soon as practical.
   * how the organization shall detect and prevent quality problems, and
   * how staff at all levels are encouraged to identify and establish communications among customers and suppliers, identify process improvement opportunities, identify problems, and offer solutions to those problems.

 Corrective actions will include:
   * identification of root causes of problems,
   * determination of whether the problem is unique or has more generic implications, and a recommendation of procedures to prevent recurrence.
                                                        EPA Classification No.:
DRAFT FINAL
(2106-S-02.0 Annex B)
                                                             CIO Approval Date:

                                                           CIO Transmittal No.:

                                                                   Review Date:
09/10/12 

                              ANNEX B (NORMATIVE)
    REQUIREMENTS FOR QUALITY ASSURANCE PROJECT PLANS IN EXTERNAL AGREEMENTS

 B1. INTRODUCTION

This Quality Standard for Environmental Data Collection, Production, and Use by Non-EPA (External) Organizations (see Clauses 7. and 7.5) requires that all work performed by non-EPA organizations on behalf of EPA involving the collection, production, or use of any environmental data shall be implemented in accordance with an Agency-approved Quality Assurance Project Plan or approved equivalent planning document (hereafter referred to as a QAPP). The QAPP defines and documents how specific activities involving environmental data shall be planned, implemented, and assessed during a particular project. Therefore, it is the "road map" of how to achieve satisfactory results in such external agreements.

This Annex presents the minimum specifications for quality assurance (QA) and for quality control (QC) procedures that must be addressed in a QAPP prepared by non-EPA organizations for external agreements, and describes the process for its preparation, review, and approval. Supplemental guidelines on how to meet these QAPP requirements can be found in Handbook for Developing Quality Assurance Project Plans (EPA 2013d).

The QAPP is a critical planning document for any environmental data operation since it documents how environmental data operations are planned, implemented, documented, and assessed during the life cycle of a program, project, or task. The ultimate success of an environmental program or project depends on the adequacy and sufficiency of the quality of the environmental data collected, produced, and used in estimation or decision-making. This depends significantly on the adequacy of the QAPP and its effective implementation. Systematic planning (Clause 7.7) is an absolutely essential component of project management and the QAPP provides the mechanism for documenting the results of the planning process. Additional guidance on planning for projects may be found in Annex C of this Standard.

 B2. QAPP RESPONSIBILITIES AND APPLICATION

 B2.1 QAPP Preparation Responsibilities and Approvals

The external (non-EPA) organization's Quality Management Plan (QMP) (see Clause 7 and Annex A of this Standard) establishes how, when, and by whom development, review, approval, and effective oversight of QAPPs occur. In some cases, it may be necessary to add special requirements to the QAPP. The EPA organization sponsoring the work shall define any specific requirements beyond those listed in this Annex.

Each QAPP or similar document must be approved prior to any data collection, production, or use, with the possible exception of circumstances requiring immediate action to protect human health and the environment or operations conducted under police powers.

Under these circumstances, the organization will use or modify existing standard operating procedures (SOPs) and other quality-related documents available to support such activities. This complete collection of QA documents must be submitted to the QA Manager together with a detailed explanation of the circumstances under which this emergency quality documentation was produced. This explanation must also address potential limitations in the interpretation of resulting data and information.

As noted in Clause 7.5.2 of this Standard, EPA may elect to require a combined QMP and QAPP (e.g., for small environmental research projects). A combined QMP-QAPP incorporates some elements of the QMP and some elements of the QAPP. The general elements to be addressed are typically taken from Clauses 7.2 and 7.5.1 of this Standard; however, the EPA organization shall determine the final elements to be addressed in the combined QMP-QAPP.

As such, all of these elements must be addressed by EPA organizations and non-EPA organizations holding external agreements. The combined QMPQAPP is subject to the same review and approval process (both by the external organization and by EPA) as any other QMP and QAPP.

 B2.2 QAPP Implementation and Revision

The organization performing the work shall implement the EPA approved QAPP and ensure that all personnel involved in the work have copies of the approved QAPP and all other necessary documents and procedures, and fully understand the project needs prior to the start of activities.

Due to the complex and diverse nature of many programs involving environmental data collection, production, and use, changes to original QAPPs are often needed. The organization's project manager, with the assistance of the QA Manager as appropriate, must determine the impact of such changes on the technical and quality objectives of the project. There should be agreement as to what changes would have minimal impact on the work and can be authorized in the field by the project manager (e.g., change in sample location) and what changes would require a revision to an approved QAPP (e.g., change in chemical analysis method). When a substantive change is warranted, the organization shall modify the QAPP or prepare an addendum to document the change and submit the revision for approval by the same authorities that performed the original review. The change shall be implemented only after the revision has been approved and received (at least verbally with written follow-up) by project personnel and EPA.

It is essential that the QAPP be kept current and that all personnel involved in the work have easy access to a current version of the QAPP. For programs or projects of long duration, such as multi-year monitoring programs, the QAPPs should be reviewed at least annually by the organization. If revisions are necessary to reflect current needs, the QAPP shall be revised and resubmitted for review and approval by the same authorities that performed the original review.

 B2.3 Applicability of QAPPs

The QAPP requirements in this Annex apply to all activities for EPA that acquire, generate, or analyze environmental data that are performed on behalf of EPA through external agreements. Such agreements may include:

   * acquisition of services and items through contracts, task orders, and other acquisition agreements;
   * assistance agreements such as grants and cooperative agreements;
   * interagency agreements with other Federal, State, local, or international governments or entities;
   * enforcement agreements (e.g., administrative orders, consent agreements); and 
   * voluntary partner programs (e.g., voluntary monitoring).

Where specific Federal regulations require QA and QC activities, QAPPs shall be prepared, reviewed, and approved in accordance with the specifications contained in this document for the data collection activity unless superseded by the regulation. QA and QC requirements are negotiated into the applicable interagency and international agreements, including sub-agreements, since EPA cannot unilaterally impose its QA and QC requirements in these agreements.

 B3. QAPP ELEMENTS AND REQUIREMENTS

B3.1 Overview and General Requirements

This Clause describes the minimum elements and information that are generally required for a QAPP for most operations involving environmental data collection, production, and use by external organizations.  Due to the diversity of Agency programs, some elements described in this Annex may not be applicable to all programs. The final decision on the applicability or use of any or all of these elements for QAPPs shall be made by EPA and may be specified in the statement of work for the external agreement.

The content and level of detail in each QAPP may vary according to the nature of the work being performed and the intended use of the data. This is called "the graded approach" and allows for a flexible design of the QAPP content. The intent of the "graded approach" is to achieve sufficient and adequate detail in the QAPP to satisfy the scope and objectives of the project and the intended use of the data.  The final decision on QAPP content and level of detail belongs to EPA.

 B3.2 General Content Requirements

Most activities involving environmental data require the coordinated efforts of many individuals, possibly including managers, engineers, scientists, statisticians, and others. The QAPP must integrate the contributions and requirements of everyone involved into a clear, concise statement of what needs to be accomplished, how it shall be done, and by whom. It must provide understandable instructions to those who must approve or implement the QAPP, including the field sampling team, the analytical laboratory, and the data reviewers. The use of SOPs and, when possible, national standards are encouraged in all aspects of the QAPP.  When these are used, they must be cited in the QAPP.

The QAPP must include standardized, recognizable management elements that cover the entire scope of the project from planning, implementation, verification and validation, assessment, and data management to final reporting. The QAPP must also recognize that there are different types of projects that may involve environmental data including, but not limited to:

   * monitoring and use of direct measurements and laboratory results by EPA or its extramural partners;
   * collection or use of environmental data from other sources (e.g., obtained from databases, Internet, literature) that were collected originally for other purposes;
   * sampling and analytical methods development or for validation purposes; and
   * generating or producing environmental data from the use of mathematical or probabilistic models and computer software.

In order to be effective, the QAPP must specify the level or degree of QA and QC activities needed for the particular environmental data or model operations. The QA and QC technical requirements of a project should be commensurate with:

   * the purpose of the environmental data collection (e.g., enforcement action, research and development studies) or model use;
   * the type of work to be done (e.g., monitoring, site characterization, model simulation, bench level proof of concept); and
   * how the results shall be used (e.g., regulatory enforcement, Total Maximum Daily Load [TMDL] development, permit approval, research publications and journal articles).

The QAPP elements that follow are presented in an order reflecting the logical flow (Plan, Do, Check, Act) of the project life cycle and have been arranged, for convenience, into four general groups. The groups and their intent are summarized as follows:
   * Project Management (Plan) - These elements cover the basic area of project management, including the project history, project objectives, technical and quality objectives (e.g., Data Quality Objectives), and roles and responsibilities of the participants. These elements document that the project has a defined goal and that the participants understand the goal and the approach to be used.
   * Data Acquisition (Do) - These elements cover all aspects of measurement and data acquisition systems design and implementation. It includes the intended measurements, data collection, or acquisition methods appropriate for achieving project objectives. Elements are presented for the different types of projects discussed above (i.e., monitoring, acquisition from other sources, and modeling).
   * Assessments (Check) - These elements address the activities for assessing the effectiveness of the implementation of the project and associated QA and QC activities. The purpose of assessment is to ensure that the QAPP is implemented as prescribed and that project actions are implemented as expected.
   * Review, Evaluation of Usability, and Reporting Requirements (Act) - These elements cover the QA activities that occur after the data collection (or use) phase of the project is completed.  Implementation of these elements will help to determine that the data conform to the specified criteria, thus achieving the project objectives and ensuring that data usability is documented in a final project report. Included are any limitations or restrictions on the use of the data that will have to be identified and documented.

All applicable elements must be addressed in the QAPP.  For some projects, only part of the QAPP elements discussed may apply to work being performed. If so, the elements that do not apply to the work should be noted in the QAPP.

Documentation, such as an approved Work Plan, SOPs, model calibration reports, etc., may be referenced in response to a particular required QAPP element to reduce the size of the QAPP and the time required for preparation and review. All referenced documents must be attached to the QAPP itself or be placed on file with EPA and available to users when needed. Such references must be kept current by the submitter. The QAPP shall also identify and describe related QA planning documentation (e.g., QMPs) from subcontractors or suppliers of services critical to the technical and quality objectives of the project or task.

The format of the QAPP is not critical except that all required elements must be addressed. The organization should present the content of the QAPP in a logical, easily understood manner. The following items may be helpful in making the QAPP more user-friendly and useful:
   * table of contents;
   * document control format;
   * applicable references (e.g., methods, SOPs, handbooks); and
   * embedded Internet links.

 B3.3 Project Management

The QAPP must describe the project adequately. Elements that address the basic project management and objectives of the work include:
   * title, version number, and approval/sign-off sheets;
   * document format and table of contents;
   * distribution list;
   * project organization and schedule;
   * project background, overview, and intended use of data;
   * project quality objectives (Data Quality Objectives) and measurement performance criteria; 
   * special training requirements/certification; and
   * documentation and records requirements.

Refer to EPA 2013d for specifics on satisfying requirements in these areas.
 B3.4 DATA ACQUISITION

The collection, production, and use of environmental data may include:
   * monitoring/direct measurements - traditional sampling and analysis activities, taxonomic observations, or testing (e.g., ecological monitoring, biological studies);
   * collection of data from other sources - data retrieved from data bases, literature, or other electronic sources (e.g., internet or provided by other organizations);
   * sampling and analytical/testing methods - the development of analytical methods or other testing methods (e.g., biological, ecological, GIS), or used for validation purposes; and
   * development, modification, and use of mathematical and probabilistic models and other software applications (information technology products).

The QAPP shall include sufficient descriptions of planned activities in order to meet the requirements given in Clauses 7.5 and 7.7. In addition, Annex C (Planning) may be particularly helpful in defining project planning needs.

 B3.4.1 Monitoring/Direct Measurements

The following QAPP elements cover aspects of measurement and testing equipment (M&TE) systems design and implementation in the project. These elements reflect traditional sampling and analysis activities for the collection of new data or information.

The elements shall include:
   * sampling collection design, experimental design, and sampling tasks;
   * sampling procedures and requirements;
   * sample handling, custody procedures, and documentation;
   * sample collection documentation;
   * analytical methods requirements and task description;
   * quality control requirements;
   * instrument/equipment testing, inspection, and maintenance requirements, QA/QC supplies and consumables; and
   * data management requirements.

Refer to EPA 2013d for specifics on satisfying requirements in these areas. For assistance in sampling process design, the organization should refer to Guidance for Choosing a Sampling Design for Environmental Data Collection, EPA QA/G-5S (EPA 2002b).

 B3.4.2 Collection of Data from Other Sources

For those circumstances in which environmental data being acquired from sources other than direct measurements by EPA or its partners (for example, published literature, other federal agencies, the Internet) the QAPP shall describe how the requirements of this Annex will be met. Similarly, when data from EPA or its partners are intended for a use other than for which it was collected originally, the QAPP shall document the acceptance criteria for the use of these data. The QAPP shall identify and describe the data sources and types of data needed that are to be obtained from other sources, and shall define acceptance criteria for the use of such data in the project.

The acceptance criteria for evaluating existing data shall be sufficiently detailed to allow for both quantitative and qualitative criteria to be developed.

Quantitative criteria will allow for the statistical consideration of data, while qualitative data or information will allow for contextual consideration of data and information. If data from other sources are to be integrated into new data or information, the acceptance criteria shall address how the integration will be achieved and verified for appropriateness. If the quality of the data cannot be determined, the QAPP shall document this limitation and the outputs or products of the project include disclaimers or discussion reflecting this limitation.

Key elements of the process of reviewing environmental data for potential use may include:
      * the need to satisfy the major requirements of the Information Quality Act (2002) (e.g., objectivity, utility, and integrity);
      * a description of the data collection methodology employed;
      * the principal indicators of environmental quality (e.g., precision, bias, representativeness, comparability, completeness, and sensitivity); and
      * the general usability factors (e.g., relevance, temporal compatibility, spatial inclusivity, credibility, aggregation level, compatibility, reproducibility, and degree of augmentation).

The following QAPP elements cover aspects of the design and implementation for the collection of data in the project.

Where applicable, the elements shall be addressed in the QAPP and include:
   * proposed data source originator and publication information;
   * data format and accessibility;
   * establishment of acceptance criteria;
   * sample data collection methodology;
   * quality program and quality assurance procedures used by data originator;
   * documentation of sample quality assurance procedures; and
   * data management requirements.

Refer to EPA 2013d and to Annex C for assistance in constructing qualitative and quantitative acceptance criteria for the investigation of data obtained from undocumented sources.

 B3.4.3 Sampling and Analytical/Testing Methods Development and Validation

Projects for the purpose of developing or validating sampling and analytical testing methods typically will involve the collection, production, and use of environmental data as part of the design, testing, and verification/validation of the method. In some situations, the requirements of Clause B3.4.1 may also apply to methods development and validation and they should be applied as applicable. The QAPP shall describe how the requirements of Clause 7.5 and this Annex will be met.

Refer to EPA 2013d for specifics on satisfying requirements in these areas.

 B3.4.4 Development and Use of Mathematical and Electronic Models (Information Technology Products)

The development and use of mathematical and electronic models (i.e., information technology products) shall be described in a QAPP. The QAPP shall meet the requirements of Clauses 7.5 and 7.6 of this Standard. Reference should be made to EPA 2013d for guidance on addressing QAPP needs for model development, modification, and application.

The QAPP shall discuss requirements for evaluation of soundness of science underlying a model, quality and quantity of available data, degree of correspondence with observed conditions, and the appropriateness of a model for a given application.

 B3.4.4.1 Quality Assurance Project Plan Requirements for Model Development/Modification

The QAPP shall include, but not be limited to, consideration, development, and documentation of the specifications for model development identified below. While some of these aspects may not be available at the start of a model development project, the QAPP should be amended as soon as the information becomes available.

Problem Specification and Identification of Model Purpose and Scope

The QAPP should define:
   * the objectives (i.e., what questions the model needs to answer);
   * the most important functions that the software application must perform;
   * the model scope (e.g., spatial, temporal and process detail); and
   * how the quantitative or qualitative model performance criteria will be determined.

Model Development or Selection Process

The QAPP should identify and define:
   * the theory that forms the basis for the model, the mathematical algorithms, and approaches used in executing the model computations;
   * the existing software to be used in development of a new model, how it will be used and why it was selected;
   * the computer hardware and operating system requirements for the software application;
   * any assumptions and limitations and effect on model applicability; and
   * the procedures for controlling, documenting, and archiving all significant changes to the software and hardware.
   
Data Requirements

The QAPP should define clearly:
   * the data needs/inputs/sources; 
   * any boundary or initial condition specifications; 
   * data reporting requirements and format for model inputs and outputs; and
   * the data quality and quantity objectives for model input and data used for model development.
      
Evaluation of the Model

It is necessary for the QAPP to define:
   * how the science underlying each component of the model will be evaluated; 
   * the model testing strategies including design and code verification, individual module tests, integration tests, systems testing, acceptance testing, and beta testing, as applicable.  
   * the procedure for each test shall be provided and the process of confirming the test results included (EPA 2009);  
   * the use of uncertainty and sensitivity analysis; and
   * the requirements for project documentation (e.g., design document, source code, and user guide).
      
 B3.4.4.2 Quality Assurance Project Plan Requirements for Model Application

The QAPP shall include, but not be limited to, consideration and development and documentation of the detailed specifications for model application identified below: 

Problem Specification, Model Purpose, and Model Selection

The QAPP should define:
*       the basis for the selection of model from other available models; and
*       the research or regulatory objectives and the purpose of model application and analytical work.

Model Application

The QAPP needs to define:
   * the scenarios to be simulated; and
   * how the modeling analysis and its results will be documented.
   * the model scope (e.g., spatial, temporal and process detail); and
   * how the quantitative or qualitative model performance criteria will be determined.

Data Requirements

The QAPP should define clearly the quality and quantity of available data and procedures for excluding data.

Evaluation of the Model

It is necessary for the QAPP to define:
*       the quality objectives to be used in the model evaluation;
*       the requirements for qualitative and/or quantitative model corroboration;
   * the use of sensitivity and uncertainty analysis; 
   * any requirements for model post-auditing;
*       the requirements for peer review of the model application; 
*       how the model evaluation will be documented;
*       the model parameters and method of estimation; and
*       the model calibration methods.

 B3.5 ASSESSMENTS

The QAPP shall describe how the requirements of Clause 7.7 of this Standard, including the roles and responsibilities of project personnel, will be addressed in a project involving the collection of new data or inclusion of data from other sources. The QAPP elements that address the processes for assessing the effectiveness of the implementation of the project, and associated QA and QC activities such as assessments and response/corrective actions must also be addressed. 

Examples of assessments include:
   * technical systems assessments;
   * performance audits of measurement and analytical systems;
   * surveillance operations;
   * audits of data quality;
   * qualitative comparisons to acceptance criteria;
   * quantitative comparisons to acceptance criteria; 
   * assessments to other criteria;
   * interim assessments of data quality
   * evaluation of unconventional measurements; and
   * evaluation of unconventional monitoring projects. 

It should be noted that when using existing data, only the last six QA/QC activities listed apply. Refer to EPA 2013d for guidance on satisfying requirements in these areas.
 B3.6 REVIEW, EVALUATION OF USABILITY, AND REPORTING REQUIREMENTS

 B3.6.1 Implement and Document Review, Verification and Validation, and Evaluation of Usability

The QAPP shall describe how the requirements of Clause 7.9, including the roles and responsibilities of project personnel, will be addressed in the project.

The QAPP elements that address the processes for the review, assessment, and verification of usability include:
   * data verification and validation targets and methods;
   * quantitative and qualitative evaluations of usability;
   * potential limitations on interpretation;
   * reconciliation with project requirements; and
   * reports to management.

If the quality of the data cannot be determined, the QAPP shall note that any limitations impacting the work products shall include appropriate disclaimers on the use of the products.

For specific advice in data verification and validation, refer to Guidance on Environmental Data Verification and Validation, EPA QA/G-8 (EPA 2002c); for data usability, refer to Data Quality Assessment: Statistical Methods for Practitioners EPA QA/G-9S (EPA 2006d), and also EPA 2013d.

 B3.6.2 Implement and Document Model Review

The critical review of a model or its application shall be conducted by individuals who are independent of the work and who collectively have technical expertise equivalent to those who performed the work. Internal and external peer review may help to ensure that the model is technically adequate, competently performed, properly documented, and satisfies established quality requirements through the review of assumptions, calculations, extrapolations, alternate interpretations, methodology, acceptance criteria, and/or conclusions from a model or its application (modified from EPA 2009). The process for reviewing a model shall be documented in the QAPP.
                                                        EPA Classification No.:
DRAFT FINAL
(2106-S-02.0 Annex C)
                                                             CIO Approval Date:

                                                           CIO Transmittal No.:

                                                                   Review Date:
09/12/12 
                                       
                             ANNEX C (INFORMATIVE)
                 GUIDELINES ON PLANNING FOR PROJECTS INVOLVING
                              EXTERNAL AGREEMENTS
Annex B presented the minimum specifications to be addressed in a Quality Assurance Project Plan (QAPP) developed by non-EPA organizations as part of an external agreement. Annex C gives guidelines for systematic planning in addressing projects involving the collection, analysis, and reporting for environmental data, including the development of conceptual models of the data collection activities.  Systematic planning involves defining and characterizing the project and desired outcomes, and designing quality assurance (QA) and quality control (QC) requirements and procedures that will ensure the data collected, generated, or used during the project will support expected and needed project outcomes. 

The elements of systematic planning can be grouped into three process areas:
   * project planning (Clause C1);
   * design of data collection activities (Clause C2); and
   * design and use of conceptual models when appropriate (Clause C3).

The elements in both planning and development are iterative in nature as information from one area may change or alter the results in other areas. For example, a change in the conceptual model that describes the scientific mechanism driving the project can change the design of data collection and even the scope of investigation. The combination and synthesis of all three areas lead to a data collection activity capable of withstanding robust scrutiny.

As an added aid to planning, Clause C4 discusses the relationship between systematic planning and the QAPP, including the Data Quality Objectives (DQO) process and an example application of the DQO process to the Uniform Federal Policy QAPP Workbook (EPA 2005).

C1. PROJECT PLANNING

Project planning should be coordinated among the participants and include, as applicable, the following steps:
   * a definition of the project;
   * the scope of the investigation;
   * the needs or demands of the project; and
   * the reporting requirements established in the external agreement for the project or investigation.

C1.1 Definition of the Project

The definition step creates an overall framework upon which the project will be developed and should consider:
   * the definition of project/task scope and objectives and the desired action or result from the work; 
   * an identification of program requirements and regulations that will be supported by the project;
   * the development of project goals, task/activity lists, time frames and logistical needs;
   * the development of a conceptual model that describes the physical, chemical, scientific, engineering principles or mechanism relating to the objectives of the project;
   * the need to plan and conduct pilot or reconnaissance studies that could provide sufficient data to assist in the development of the conceptual model; and
   * the identification of the different environmental risk assessment pathways that will be evaluated by the project, if appropriate.
C1.2 Scope of the Investigation

The scope step defines the boundaries (i.e., temporal, physical, and logistical) of the project and may include:
   * an identification of special applicable regulatory requirements and key spatial and temporal constraints such as budget, timing, authorization, permits, and access;
   * the recognition of organizations (e.g., sampling groups or analytical laboratories) that need to participate in the project and their role in planning, implementation, and assessment activities;
   * an outline of the composition of the project management team (i.e., the management, technical, and administrative support team including any contractors and subcontractors); 
   * a listing of conditions under which suspension of work will be necessary;
   * an identification of necessary personnel, their needed skills, and required types of equipment; 
   * the results of literature searches and background information collection about the project including industries, products and wastes generated by the industries, and previous analytical results; and
   * a preliminary determination of assessment tools needed such as technical reviews, audits, statistical or mathematical analyses, or engineering capability studies.

C1.3 Needs or Demands of the Project

In order to effectively support successful completion of the project, this step gives attention to:
   * an identification of data gaps that should be filled by the new project;  
   * a consideration of accompanying metadata needed for development of the conceptual model;
   * a listing of procedures to handle unavoidable deviations from the Agency-approved QAPP;
   * an identification of standard operating procedures (SOPs) for developing and implementing corrective actions;
   * the identification of possible methods/procedures for characterization and disposal of contaminated sample material that may be accumulated during the project; and
   * an outline of the documentation needed to adequately describe the quality of the results.

C1.4 Reporting Requirements Established for the Project or Investigation

Successful projects demand good communication both during the progress of the project and in the reporting of results. 

The reporting step gives attention to:
   * the establishment of communication and authority among the project team participants and  identification of the roles and responsibilities of each member of the project team;
   * an identification of project reports that will need to be prepared and the frequency of submission to management;
   * a working schedule for data reporting;
   * a determination of data reporting format; and
   * an identification of methods/procedures for storing, retrieving, analyzing, and reporting the data produced (based on the intended use of the data).

C2. DESIGN OF ENVIRONMENTAL DATA AND INFORMATION COLLECTION OR ACQUISITION

The design process should include, but not necessarily be limited to, consideration and development of detailed plans for the collection or acquisition of data or information, the analysis or synthesis of data and information, and specific data or information reporting requirements.

C2.1 Collection or Acquisition of Data or Information

Significant attention should be devoted to designing and documenting all aspects of how the environmental data and information are collected. These considerations should include:
   * documentation of the degree to which the data or information collected will be representative of the conditions established by the goals of the project;
   * documentation of the sampling strategy, rationale, and necessary procedures for collecting the data or information required to achieve the desired action or result;
   * documentation of the sampling design and rationale together with the linkage to the conceptual  model established for this project;
   * the determination of the quantity of data needed to achieve the project goals or support a decision with the desired level of confidence;
   * an integration of logistical needs, costs and schedule into the preferred sampling design;
   * documentation on the sample locations, sample types and number of subsamples per matrix;
   * an assessment of the necessity for on-site field and laboratory assessments to be performed prior to and during project implementation; and
   * the selection of field sampling or testing methodology (except where specified by statute or regulation), including specific sampling or field analytical instrumentation requirements and other analytical testing requirements.

Practical advice on how to address these considerations may be found in current guidance from the Quality Staff, including EPA 2002b and from standard works on environmental statistics (e.g., Gilbert 1987, Manly 2001, Ginevan & Splitstone 2003).

C2.2 Analysis or Synthesis of Data or Information

After collection, data should be analyzed and information synthesized before definitive conclusions can be drawn. In planning for the analysis or synthesis of data, attention should be given to:
   * the selection of analytical methods, test procedures (except where specified by statute or regulation), and their associated quality performance expectations and identification of measureable data quality or QA/QC performance criteria against which results will be compared;
   * an identification of measurement data quality criteria or QA/QC requirements necessary to establish the quality of the data collected or produced;
   * the selection of analytical facility (or laboratory) together with measures for verification that the laboratory not only is certified to an appropriate and applicable standard, but is competent and has demonstrated capability in performing the analyses needed;
   * establishment of measurement quality objectives for specific data quality indicators such as precision, bias, representativeness, completeness, comparability, and sensitivity;
   * identification of field and laboratory QA/QC requirements for sample handling, packaging, shipping, custody, and health and safety procedures;
   * the integrity of data management procedures in the field and laboratory (e.g., data retrieval, security, QA and QC, storage, and retention);
   * the documentation of how information will be summarized and synthesized;
   * the construction of acceptance criteria for data or information obtained from external sources;
   * an outline of how metadata (or ancillary information) will be used;
   * documentation of proposed information privacy and integrity procedures; and
   * the integrity of information management procedures in both the field and information collection centers.
C2.3 Specific Data or Information Reporting Requirements

When an external agreement requires release of data summaries or information reports, care should be given to the documentation needed to ensure the integrity, clarity, transparency, and utility of the data.  

Attention should be focused on:
   * the determination of the levels of data or information verification and validation needed to meet project goals;
   * documentation of procedures used for data or information validation and assessment;
   * a summary of potential techniques for assessing limitations on data or caveats on information use;
   * the data reporting deliverables and electronic data formats to be used; and
   * the type of reports expected together with their frequency of submission, for example,
         o field sampling/information gathering report,
         o summaries of analytical data results,
         o data validation reports,
         o interim information syntheses, and 
         o draft and final project reports.

C3. DESIGN OF CONCEPTUAL MODEL DEVELOPMENT, EVALUATION, AND APPLICATION

A "model" is something that can be used to make a prediction and, in the first instance, based on the conceptual model that describes the physical, chemical, scientific, engineering principles, or mechanism relating to the objectives of the project (see Clause C.1, Project Planning). As a conceptual model is only an approximation of a complex environmental phenomenon, attention has to be given to the documentation of conceptual model development so that the context may be understood fully.

The documentation of a general conceptual model should include, as applicable:
   * the requirements for qualitative and/or quantitative conceptual model corroboration;
   * a consideration of the requirements for application of multiple conceptual models;
   * the use of conceptual model parameters characterized from direct measurements;
   * the integration of cost or schedule constraints into design;
   * the documentation of data reporting requirements and required formatting;
   * a listing of the requirements for sensitivity and uncertainty analysis; and
   * an outline of the requirements of documentation to ensure transparency of conceptual model development.

C4. RELATIONSHIP OF SYSTEMATIC PLANNING TO THE QUALITY ASSURANCE PROJECT PLAN (QAPP)
The Quality Standard requires the use of systematic planning for any activities involving the collection, production, and uses of environmental data (Clause 7.7 of this Standard) and also the documentation of type, quantity, and quality of environmental data for their intended use. Systematic planning is a process based on the widely-accepted scientific method and uses a common sense approach to ensure that the level of documentation and rigor of effort in planning is commensurate with the intended use of the information and the available resources for data collection and models shall be documented in a QAPP or equivalent document.
The elements of systematic planning are outlined in Table C-1.
                  Table C-1. Elements of Systematic Planning
                                   Elements
Organization: Identification and involvement of the project manager, sponsoring organization and responsible official, project personnel, stakeholders, scientific experts, customers, suppliers, and Quality Manager.
Schedule: Identification of project schedule, resources (including budget), milestones, regulatory or contractual requirements, and circumstances that could adversely affect the progress of the project.
Project Goal: Description of the project goal, objectives, study questions, and conceptual model that describes the environmental phenomenon, contamination pathways, and impact on human health.
Quality Assurance: Specification of needed quality assurance and quality control activities to ensure performance criteria (for new data) or acceptance criteria (for existing data) are satisfied.  
Data Needs: Identification of the type of data needed, how the data will be used to support the project's objectives, and identification of potential limitations in data interpretation.
Criteria: Determination of the quantity of data needed and specification of performance criteria for measuring quality for new data, or acceptance criteria for the use of existing data.
Data Collection: Description of how and where the data will be obtained (including existing data or data intended for secondary use) and identification of any constraints on data collection.
Analysis: Description of how data will be analyzed, evaluated, and assessed against their intended use with respect to the established performance or acceptance criteria.

C4.1 Types of Systematic Planning
Various government agencies and scientific disciplines have established and adopted different variations to systematic planning tailored to their specific application areas. For example, the Observational Method is a variation on systematic planning that is used by many engineering professions. However, it is the Data Quality Objectives (DQO) Process (EPA 2006e) that is the most commonly used application of systematic planning in the general environmental community and is EPA's recommended systematic planning process when data are to be used for decision making (e.g., compliance with an environmental standard) or for estimation (e.g., ascertain the mean concentration level of a contaminant).
C4.2 The DQO Process
The DQO Process can be used to establish performance or acceptance criteria, which serve as the basis for designing a plan for collecting data of sufficient quality and quantity to support the goals of a study. The DQO Process may be applied to all programs involving the collection and use of environmental data and applies to programs with objectives that cover decision making, estimation, modeling in support of research studies, monitoring programs, regulation development, and compliance support activities. The DQO Process is grounded in the scientific method of hypothesis, experiment, conclusion, and has been broken into seven steps to facilitate its use. The outputs of the DQO Process are used to develop a QAPP, and, after data collection, for use in data quality assessment or statistical analysis of the resulting data or information. The steps of the DQO Process are shown in Figure C-1 and discussed in more detail in EPA 2006e.
The QAPP describes in comprehensive detail the necessary QA activities that must be implemented to ensure the results of the work performed satisfy the specified performance or acceptance criteria established for the project. The details of the QAPP may be assembled into convenient groups of elements to facilitate successful implantation. An example is the Uniform Federal Policy for Quality Assurance Project Plans (UFP-QAPP) Workbook, EPA 2005, which, although is intended for use at Federal facility hazardous waste sites primarily, draws on multiple Agency documents on QA. The elements of the UFP-QAPP are shown in Table C-2.
The planning outputs from the seven steps of the DQO Process fit into the requirements for implementation of a QAPP. The relationships among systematic planning, the DQO Process, and, for example, the elements of a UFP-QAPP are shown below in Table C-3. Several EPA organizations have accepted the UFP-QAPP as an alternative quality document to the EPA QAPP described in Annex B. The use of the UFP-QAPP or any other alternative QAPP document is subject to the approval of the EPA organization responsible for the work.
                                                                                                                                                                                                                
                          Step 1. State the Problem.
               Define the problem that necessitates the study, 
     identify the planning team, examine the budget, and set the schedule.

           

                    Step 2. Identify the Goal of the Study.
     State how environmental data will be used in meeting objectives and 
solving the problem, identify the study questions, and define alternative outcomes.


                     Step 3. Identify Information Inputs.
        Identify data and information needed to answer study questions.


                  Step 4. Define the Boundaries of the Study.
        Specify the target population and characteristics of interest,
        define spatial and temporal limits, and the scale of inference.


                    Step 5. Develop the Analytic Approach.
       Define the parameter of interest, specify the type of inference, 
         and develop the logic for drawing conclusions from findings.


              Step 6. Specify Performance or Acceptance Criteria.

                                       

                                       
           Select the resource-effective sampling and analysis plan
                      that meets the performance criteria
                                  Estimation
Develop performance criteria for new data being collected or acceptance criteria for existing data being considered for use.
                                Decision Making
Specify probability limits for false rejection and false acceptance decision errors.

                                       
                                       
                 Step 7. Develop the Plan for Obtaining Data.
           Select the resource-effective sampling and analysis plan
                     that meets the performance criteria.





                          Figure C-1. The DQO Process

Table C-2. The Elements of a Uniform Federal Policy Quality Assurance Project Plan (UFP-QAPP)

                     A. Project Management and Objectives
A1   Title and Approval Page                                               
A2	Document Format and Table of Contents                                                           
A3	Distribution List and Sign-off Sheet  
A4	Project Organization
A5 	Project Planning/Problem Definition                                                                                                           
A6   Project Quality Objectives and
       Measurement Performance Criteria
A7   Secondary Data Evaluation
A8   Project Overview and Schedule

                        B. Measurement/Data Acquisition
B1	Sampling Tasks        
B2	Analytical Tasks                                                                
B3	Sample Collection Documentation                                     
 
B4	Quality Control (QC) Samples                                                             
B5    Data Management Tasks                                                        
                            C. Assessment/Oversight
C1	Assessments and Response Actions                            
C3   Final Project Report
C2   QA Management Reports                

                                D. Data Review
D1	Overview   
D2	Data Review Steps (Verification/Validation)                   
D3   Streamline Data Review


Table C-3. The Relationship between Systematic Planning, the DQO Process, and a UFP-QAPP
    Systematic Planning
          DQO Process
         Elements of a UFP-QAPP
Organization 
Step 1: Define the problem
A1 - A5
Schedule
Step 1: Define the problem
A4, A8, C3
Project Goal
Step 2: Identify goals
Step 3: identify information
            needed for the study
A4, A6
Quality Assurance
Step 3: Identify information 
            needed for the study
B2  -  B5
Data Needs
Step 4: Define boundaries
Step 5: Develop the analytic 
            approach
A6, B1
Criteria
Step 6: Specify performance 
            or acceptance criteria
A7, B1, C1, D1 - D2
Data Collection
Step 7: Develop the plan for 
            obtaining data
A7 - A8, B1
Analysis
Step 7: Develop the plan for
            obtaining data
B1, A7, C1 - C2, D3

