Developing FACDQ Recommendations around Data Quality Objectives

November 20, 2006

Introduction

At its November 1, 2006 meeting, the Policy Work Group agreed to step
back from efforts to reach consensus on Measurement Quality Objectives
(MQOs) and to focus first on reaching consensus on broader data quality
objectives (DQOs) for various detection and quantitation uses.  

The Policy Work Group tasked a subgroup (Jim Pletl, Mary Smith, Brian
Englert, Tom Mugan, Nan Thomey, Michael Murray, and John Phillips) with
developing a product to stimulate discussion at the committee’s
December meeting.  The subgroup developed two sets of questions related
to these issues:  fundamental questions related to data quality
objectives and specific questions related to the FACDQ’s
recommendations.  

On November 17, the Policy Work Group agreed that the questions posed in
this document are significant and need to be addressed.  The Policy Work
Group agreed to ask all members of the FACDQ to prepare and send their
responses to the questions to the facilitation team by Thursday,
November 30.  The facilitators will compile the responses so they can be
arrayed for discussion at the December 6-8 FACDQ meeting.  Where members
have already submitted responses, they are included in italics.

Definitions

Data Quality Objectives 

DQOs are qualitative and quantitative statements that clarify the
purpose of a study, define the most appropriate type of information to
collect, determine the most appropriate conditions from which to collect
that information, and specify tolerable levels of potential decision
errors.  The DQO Process is a series of steps applied to decision-making
(e.g., compliance/non-compliance with a standard) and estimation (e.g.,
ascertaining the mean concentration level of a contaminant).  This
definition is taken from “Guidance on Systematic Planning Using the
Data Quality Objectives Process,” EPA QA/G-4, Office of Environmental
Information EPA/2401/B-06/001 Feb. 2006 (available at
http://www.epa.gov/QUALITY/qs-docs/g4-final.pdf).

Measurement Quality Objectives 

Measurement Quality Objectives are defined as qualitative and
quantitative statements of the overall level of uncertainty that a
decision maker is willing to accept in results or decisions derived from
measurements.  

Together, MQOs and DQOs provide the statistical framework for planning
and managing measurement plans consistent with the data user’s needs.
(Glossary of Terms, FACDQ 07/05/06)

Fundamental Questions

A.  Should FACDQ make recommendations regarding how to take Data Quality
Objectives into consideration for decision-making?

Yes/No and Why ?

Yes, the FACDQ "Uses in the Clean Water Act program" portion of our
charter specifically ties to setting DQOs. 

Maybe, because it relates to the Uses process.

B.  What Data Quality Objectives should be considered by the FACDQ for
recommendations?

What are they? (e.g., specific MQO values or ranges, uncertainty,
representativeness, matrix effects, etc.)

1.   Several of the DQO Process steps as defined in the EPA QA/G4
guidance document.

Introduction, section 0.9 page 11, "Categories of intended uses for
environmental data" 

Step One, section 1.2 page 17, "How do you identify the type of intended
use for the study data?"

Step Two, section 2.2 page 22, "Decision and Estimation Problems"

Step Three, section 3.2 pages 27-28, "Availability of appropriate
sampling and analysis methods."

Step Five, pages 39-43, specifically page 41, "How are measurement
detection limits important to selecting an action level?"

Step Six, pages 45-70, specifically page 46, "False Positives, False
Negatives, Uncertainty" also Figs. 7, 8 & 9 and "Types of Intervals"

C.  What are the decisions around which FACDQ should provide advice or
recommendations on goals for DQOs?

List decisions.

Generally speaking EPA guidance document QA/G4 does a fine job
discussing the DQO process, but the FACDQ could provide advice and/or
recommendations specifically regarding Step Six i.e. knowing or
determining False Negative error rate, False Positive error rate, and
Uncertainty (Precision, Bias and Data Comparability)  

D.  Should there be goals for analytical methods?  

Yes/No

Yes, but it may be Use dependent.

I think of methods as having data quality characteristics associated
with them, including an implied assumption that they are used
competently with properly functioning equipment.  Data quality
objectives are established based on the use of the data in
decision-making.  The characteristics of the method are then evaluated
to select a method that has the appropriate characteristics.

E.  If so, what are those goals?

List goals.

False Positive error rate

False Negative error rate

Method Precision at QL

Method Bias at QL

Data Comparability between labs 

Policy Work Group Specific Questions  

What DQOs are applicable to the different identified uses?

I would restate this question, "What MQOs are needed to allow for the
DQOs for different identified uses to be met?"  (MQOs - False Positive
error rate, False Negative error rate, Method Precision at QL, Method
Bias at QL, and Data Comparability between labs).

I think there may be different DQOs within a single use depending on the
relationship between the WQBEL and the concentration of the analyte in
the sample.

What are the goals for method performance?

Must answer the prior question first.

How do we account for uncertainty in methods (e.g. sampling, testing)?

We don't need to "account for uncertainty" that should be part of the
DQO process.  We just need to know how to measure uncertainty.

I don’t think there is agreement on how uncertainty is to be measured
and reported.

Does bias have to be taken into consideration in the decision process?

Yes, it is a component of Uncertainty

Can representative data be defined?  What about a robust reasonable
potential process?

This is part of the DQO process.

Should error rates be considered for different uses?

Do you mean FP and FN error rates?  See response to item 1.

Change question to: Should different error rates be considered for
different uses?  Comment: Probably.

What is an acceptable level of decision error, taking into account MQOs
and uncertainty? 

Part of the DQO process.

Is it the same in all instances?

If data do not meet specified accuracy, how should they be used (should
data be ignored or should the uncertainty be factored in)?

Must meet specified accuracy or must factor it in as part of the DQO
process.

This should be stated in DQOs and depends on the relationship between
the WQBEL and the concentration of the analyte in the sample.

How do we account for MQOs in reference matrices and in real world
matrices?

Some comparison is needed between the reference matrix performance used
for daily method QC and the real world matrix to determine the
variability or uncertainty added to the method due to sample matrix
effects.

If matrix effects create a huge percent range in testing results, then
do we recommend matrix specific spikes?

Sample matrix must be evaluated at some frequency, as determined by
DQOs.

If MQOs are set, how do you verify that those MQOs are achieved?

On-going QC including: Blanks, FNQS, LCS/LCSD, MS/MSD, also other method
QC performance controls such as calibrations, internal standards and
surrogates. 

If MQOs are set, what should be done with those numbers?

Compare them to the methods and various DQO needs.

If we agree to set MQOs, should they be goals or specific requirements?

MQOs are "Objectives."  This means they are the targets which need to be
achieved for a successful outcome of the project.  If you miss the
target you will not bag the trophy.  MQOs may not be specific
requirements, but if they are not achieved the data may need to be
qualified and it may impact both data usability and completeness.

14.  What MQOs, if any, at this time, do you want the FACDQ to recommend
to EPA?   

	List responses to the following.

False Positives

Recommended Value(s)

Ramifications

Discussions

False Negatives

Recommended Value(s)

Ramifications

Discussions

Accuracy

Recommended Value(s)

Ramifications

Discussions

Precision

Recommended Value(s)

Ramifications

Discussions

11/20/2006

Draft for Discussion

Document # FACDQ6-07

 PAGE   

 PAGE   1 

Federal Advisory Committee on Detection and Quantitation Approaches and

Uses In Clean Water Act Programs

Policy Work Group 

Draft Recommendations Around DQOs

