SUMMARY
REPORT
PEER
REVIEW
OF
THE
PLAN
FOR
THE
ASSESSMENT
OF
DETECTION
AND
QUANTITATION
LIMITS
UNDER
SECTION
304(
h)
OF
THE
CLEAN
WATER
ACT
Contract
No.
68­
C­
98­
189
Versar
Work
Assignment
No.
3­
42
Prepared
for:

U.
S.
Environmental
Protection
Agency
Office
of
Water
Office
of
Science
and
Technology
Health
and
Ecological
Criteria
Division
401
M
Street,
S.
W.
Washington,
D.
C.
20460
Prepared
by:

Versar,
Inc.
6850
Versar
Center
Springfield,
Virginia
22151
February
2002
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
TABLE
OF
CONTENTS
I.
INTRODUCTION
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
1
II.
BACKGROUND
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
2
III.
CHARGE
TO
THE
PEER
REVIEWERS
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
3
IV.
GENERAL
COMMENTS
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
4
V.
RESPONSE
TO
CHARGE
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
9
VI.
SPECIFIC
COMMENTS
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
16
VII.
MISCELLANEOUS
COMMENTS
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
18
APPENDIX
A:
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
and
Associated
References
APPENDIX
B:
Charge
to
Peer
Reviewers
APPENDIX
C:
Reviewer
A
Curriculum
Vitae
APPENDIX
D:
Reviewer
C
Curriculum
Vitae
APPENDIX
E:
Reviewer
E
Curriculum
Vitae
APPENDIX
F:
Reviewer
F
Curriculum
Vitae
APPENDIX
G:
Reviewer
A
Comments
APPENDIX
H:
Reviewer
C
Comments
APPENDIX
I:
Reviewer
E
Comments
APPENDIX
J:
Reviewer
F
Comments
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
1
of
18
I.
INTRODUCTION
The
purpose
of
this
peer
review
was
to
evaluate
the
scientific
credibility
of
the
document
entitled
"
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act."
The
Plan
and
associated
references
are
provided
in
Appendix
A.
The
peer
review
was
performed
to
obtain
an
independent
evaluation
by
expert
authorities
on
analytical
chemistry
and
statistics
of
chemical
measurement
to
determine
whether
the
Agency's
plan
for
assessment
of
detection
and
quantitation
limit
methodology
is
sound.

Peer
review
is
a
process
for
enhancing
a
scientific
or
technical
work
document
so
that
the
decision
or
position
taken
by
the
Agency,
based
on
the
technical
document,
has
a
sound,
credible
basis.
The
goal
of
the
Agency's
Peer
Review
Policy
is
to
ensure
that
scientific
and
technical
work
products
receive
appropriate
levels
of
critical
scrutiny
from
scientific
and
technical
experts
as
part
of
the
overall
decision
making
process.
Generally,
this
technical
review
precedes
the
customary,
more
broadly
based
public
review
of
the
total
decision.

The
"
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
under
Section
304(
h)
of
the
Clean
Water
Act"
was
reviewed
by
a
panel
of
four
peer
reviewers.
The
charge
to
the
peer
reviewers
is
provided
in
Appendix
B.
These
panelists
were
selected
because
of
their
expertise
in
the
field
of
statistics
and
analytical
chemistry
and
absence
of
conflict
of
interest.
The
peer
review
panel
did
not
include
any
experts
that
directly
or
indirectly
contributed
to
the
development
of
EPA's
method
detection
limit
(
MDL)
or
minimum
level
(
ML).
Based
on
the
requirements
of
the
Work
Plan,
the
identity
of
the
peer
reviewers
has
been
kept
confidential.
Curriculum
vitae
for
each
of
the
four
panelists,
with
identifying
information
(
e.
g.,
peer
reviewer's
name,
address)
removed
are
provided
in
Appendices
C
through
F.
The
four
reviewers
are
referred
to
as
Reviewer
A,
Reviewer
C,
Reviewer
E,
and
Reviewer
F
in
this
report.
Peer
Review
comments
received
from
each
of
the
four
panelists
are
provided
in
Appendices
G
through
J.

The
comments
and
recommendations
from
the
four
reviewers
have
been
combined
and
organized
as
follows:


General
comments;


Response
to
charge;


Specific
comments
by
page
number
referenced
by
commentor;


Miscellaneous
comments.
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
2
of
18
II
BACKGROUND
On
June
8,
1999,
EPA
promulgated
Method
1631B:
Mercury
in
Water
by
Oxidation,
Purge
and
Trap,
and
Cold
Vapor
Atomic
Fluorescence
Spectrometry
(
64
FR
30417).
The
method
was
developed
specifically
to
measure
concentrations
of
mercury
at
ambient
water
quality
criteria
levels,
and
includes
a
method
detection
limit
(
MDL)
of
0.2
ng/
L
(
ppt)
­
approximately
400­
times
lower
than
previously
approved
methods.

Shortly
after
promulgation,
EPA
was
notified
of
a
legal
challenge
to
the
method.
The
basis
of
the
challenge
included
several
specific
aspects
of
Method
1631
itself,
as
well
as
the
procedures
used
to
establish
the
specific
MDL
and
ML
published
in
the
method.
EPA
has
applied
those
procedures
to
most
of
the
environmental
measurement
methods
published
by
the
Office
of
Water,
and
as
noted
earlier,
the
MDL
procedure
has
been
widely
adopted
across
EPA.
On
October
19,
2000,
EPA
entered
into
a
settlement
agreement
with
the
Alliance
of
Automobile
Manufacturers,
Inc.,
the
Chemical
Manufacturers
Association,
the
Utility
Water
Act
Group
and
the
American
Forest
and
Paper
Association.
The
settlement
agreement
requires
EPA
to
reassess
EPA's
method
detection
limit
(
MDL;
40
CFR
136,
Appendix
B)
and
minimum
level
of
quantitation
(
ML)
procedures.
The
U.
S.
EPA
conducted
this
peer
review
of
the
"
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
under
Section
304(
h)
of
the
Clean
Water
Act"
as
part
of
the
settlement
agreement.
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
3
of
18
III.
CHARGE
TO
THE
PEER
REVIEWERS
The
charge
for
the
peer
reviewers
was
provided
by
the
EPA
WAM
as
an
attachment
to
the
Work
Assignment
and
included
an
introduction
and
background
information
related
to
the
document,
along
with
the
charge
questions.
The
charge
to
the
peer
reviewers
is
provided
in
Appendix
B.
The
reviewers
were
asked
to
respond
to
the
following
charge
questions:

(
1)
Evaluate
the
conceptual
soundness
of
the
assessment
approach
presented
in
the
study
plan.
Is
the
list
of
issues
included
in
Item
3
of
the
study
plan
sufficient?
Are
there
other
significant
issues
that
EPA
should
add
or
delete?
How
should
EPA
prioritize
the
issues?

(
2)
Based
on
your
understanding
of
the
issues,
are
there
any
types
of
criteria
that
you
suggest
EPA
include
to
evaluate
detection
and
quantitation
procedures?
If
yes,
please
identify
the
criterion
and
provide
a
rationale
for
its
inclusion.

(
3)
What
existing
detection
and
quantitation
procedures
should
EPA
evaluate
or
focus
on?
Procedures
from
the
open
scientific
literature?
Voluntary
consensus
standards
procedures?
Procedures
from
other
organizations?

(
4)
Are
the
data
sets
listed
in
the
plan
appropriate
for
use
in
the
detection
and
quantitation
reassessment?
Are
there
other
data
sets
that
you
recommend
EPA
evaluate?

(
5)
Is
it
appropriate
to
include
specific
selection
criteria
in
the
plan
or
is
it
more
appropriate
to
develop
selection
criteria
based
on
the
analysis
of
issues
that
will
be
conducted
as
part
of
the
plan?
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
4
of
18
IV.
GENERAL
COMMENTS
Reviewer
A
The
entire
issue
of
lower
limits
of
analytical
methods
is
one
that
has
been
fraught
with
intellectual
confusion,
even
when
it
has
not
been
politically
and
economically
controversial.
The
term
"
detection
limit"
has
been
used
for
a
number
of
dissimilar
concepts,
and
this
has
led
to
further
confusion.
For
the
purpose
of
this
review,
I
will
use
a
version
of
Currie's
terminology.

The
Critical
Level
is
an
observed
value
of
the
response
(
such
as
peak
area)
or
equivalently,
of
the
estimated
concentration
which
would
occur
by
chance
in
a
sample
in
which
the
analyte
was
not
present
only
a
specified
fraction
of
the
time,
say
1%.
As
with
any
such
limit
or
level,
it
can
depend
on
the
entire
analytical
method,
but
is
a
well
defined
construct.
This
is
the
limit
that
tells
the
analyst
when
detection
has
occurred.
Such
detection
is
almost
always
quantitative,
so
long
as
there
is
a
measured
value
and
a
standard
error.

The
Minimum
Detectable
Quantity
is
the
smallest
true
concentration
of
the
analyte
such
that
with
specified
probability,
such
as
99%,
the
estimated
concentration
after
measurement
will
exceed
the
critical
level.
This
level
has
no
interpretation
once
measurement
has
occurred.
It
is
useful
for
planning,
but
not
for
interpretation
of
existing
measurements.

For
example,
if
the
standard
deviation
at
zero
or
a
low
level
of
an
analyte
is
2ppt
exactly,
then
the
critical
level
at
99%
confidence
is,
using
the
normal
percentage
point,
(
2.326)(
2)
=
4.652.
If
the
standard
deviation
is
estimated,
say
by
a
sample
of
7
replicates,
then
one
would
use
the
t
distribution
with
6
degrees
of
freedom
so
that
the
critical
level
would
be
(
3.143)(
2)
=
6.286.
Using
the
latter
value
for
illustration,
the
practical
use
of
the
critical
level
of
about
6ppt
is
that
any
measured
concentration
exceeding
that
demonstrates
that
the
analyte
is
present
in
non­
zero
quantities.
If
we
have
a
measured
value
of
8ppt,
for
example,
then
the
standard
deviation
(
assuming
that
it
is
constant
over
the
range)
is
2ppt,
so
a
95%
confidence
interval
for
the
true
concentration
is
(
using
the
t
distribution
again)
is
8ppt
±
(
2.447)(
2)
ppt
or
approximately
(
3ppt,
13ppt).
This
is
a
quantitative
assessment.
We
know
the
true
concentration
to
within
5ppt,
which
is
pretty
accurate.
The
coefficient
of
variation
(
2/
8
=
25%)
is
not
a
relevant
measure
as
to
whether
we
have
a
quantitative
measurement
of
the
concentration.

If
we
use
99%/
1%
also
for
the
minimum
detectable
quantity,
and
if
we
assume
that
within
the
range
up
to
this
level
the
standard
deviation
of
measurement
is
approximately
constant,
then
the
minimum
detectable
quantity
is
twice
the
critical
level.
In
the
case
illustrated
above,
this
would
be
twice
6.286,
or
about
11ppt.
This
value
of
11ppt
might
be
a
good
spike
concentration
for
quality
control,
because
it
should
almost
always
generate
a
detection
signal.
However,
it
provides
no
information
useful
in
interpreting
a
measured
value
of,
say,
8ppt.

Glaser
et
al.
(
1981),
along
with
many
others,
have
seriously
confused
the
issue.
Consider
the
oft
quoted
definition
of
the
MDL:
" 
the
minimum
concentration
of
a
substance
that
can
be
identified,
measured,
and
reported
with
99%
confidence
that
the
analyte
concentration
is
greater
than
zero "
The
first
half
of
this
definition
refers
to
an
actual
concentration,
like
the
minimum
detectable
quantity,
but
the
second
half
treats
it
as
a
measured
value
like
the
critical
level.
Clearly,
we
understand
this
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
5
of
18
better
than
we
did
in
1981.
What
the
MDL
estimates
is
the
critical
level,
which
is
the
threshold
of
detection,
and
it
therefore
should
not
be
described
as
an
actual
concentration.

EPA
should
use
the
review
that
they
are
engaged
in
to
clarify
these
issues,
and
to
provide
lower
limits
that
are
scientifically
and
statistically
valid.
In
the
first
place,
this
means
distinguishing
carefully
between
the
critical
level
and
the
minimum
detectable
quantity,
and
specifying
when
each
one
is
to
be
used.

Another
issue
that
deserves
a
hard
look
and
a
scientific
analysis
is
that
of
the
so­
called
limit
of
quantitation.
For
an
analytical
measurement
to
be
quantitative,
there
are
two
requirements.
First,
there
needs
to
be
an
estimated
quantity
produced
by
a
known
analytical
procedure.
Second,
there
needs
to
be
an
estimate
of
the
precision
of
this
measurement.
Modern
metrological
techniques
(
Rocke
and
Lorenzato
1995;
Zorn,
Gibbons,
and
Sonzogni
1997)
can
provide
the
precision
estimates
at
all
measured
values
of
an
analyte.
Thus,
any
measured
value
by
a
calibrated
system
is
quantitative,
and
there
is
no
such
thing
as
a
limit
of
quantitation
if
it
is
defined
by
standard
deviation,
coefficient
of
variation
and
the
like.
The
only
scientifically
valid
definition
is
akin
to
the
original
ML
of
1984.
If
the
instrument
cannot
produce
a
number,
then
we
are
below
the
limit
of
quantitation.
This
will
usually
be
because
the
peak
cannot
be
identified
that
should
be
integrated
to
produce
the
estimated
concentration.

The
procedures
provided
by
EPA
and
the
questions
to
be
examined
will
be
sufficient
to
deal
with
these
and
other
issues,
so
long
as
the
attention
is
focused
on
scientifically
and
statistically
valid
methods.
This
review
provides
an
opportunity
for
EPA
to
clarify
these
vexed
issues,
and
provide
unambiguous,
practical,
interpretable,
and
valid
methods
for
lower
limits
of
analytical
methods.

The
plan
itself
seems
comprehensive
and
well
thought
out.
All
important
issues
will
be
considered
in
the
process.
Of
course,
the
most
important
proof
of
the
soundness
of
the
plan
for
assessment
will
be
the
product
of
the
assessment
to
follow.
At
this
stage,
I
believe
that
EPA
has
provided
a
procedure
that
can
address
all
of
the
facets
necessary
to
produce
a
scientifically
and
statistically
valid
re­
examination
of
the
issue
of
lower
limits
for
analytical
measurements.

Reviewer
C
I
think
the
plan
is
very
thorough,
with
the
various
events
called
for
touching
on
many
important
issues,
and
I
can't
find
anything
that
seems
to
have
been
overlooked.
The
issues
considered
deal
with
the
legal
aspects
of
the
situation,
and
also
with
doing
good
science
and
searching
for
a
reliable
MDL
determination
procedure.

Reviewer
E
The
only
document
that
was
sent
that
I
have
any
comments
on
is
the
12
page
"
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act,
dated
12­
10­
2001.
The
other
materials
are
fine.

The
"
environmental
chemistry"
community
has
been
using
the
Fed.
Reg.
MDL
method
for
about
10
years
to
demonstrate
detection
limits.
I
feel
this
is
an
acceptable
approach.
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
6
of
18
Each
EPA
method
should
include
a
statement
about
the
range
of
MDLs
that
have
been
demonstrated
during
the
validation
of
the
method.
Each
EPA
method
should
state
that
any
lab
using
this
method
must
demonstrate
their
own
MDL,
which
may
be
higher
or
lower
than
the
range
mentioned
in
the
method.

The
labs
can
inform
their
"
clients"
about
their
lab
MDLs
and
the
"
environmental
chemistry
market"
will
drive
these
MDLs.
If
a
regulation
requires
low
detection
limit
then
labs
will
be
motivation
to
demonstrate
the
needed
MDL.

Another
decision
EPA
needed
to
make
is
to
define
the
factor
that
is
used
to
convert
the
MDL
to
a
Reporting
Limit,
or
what
every
it
will
be
called.
For
example
if
the
RL
or
ML
is
defined
as
3.18
times
the
MDL
then
everyone
could
understand
and
use
it
even
though
they
may
not
agree.

Reviewer
F
This
reviewer
has
read
and
reread
these
documents.
The
second
paragraph
of
the
Charge
to
Peer
Review
Panel
requests
an
opinion
regarding
the
validity
of
the
Plan
and
the
likelihood
that
it
will
result
in
an
"
objective
and
useful
assessment
of
detection
(
limit)
and
quantitation
(
limit)
procedures".
None
of
the
five
Questions
to
Peer
Reviewers
(
included
in
the
Charge
to
Peer
Reviewers)
address
the
purpose
indicated
in
the
Plan's
introduction
as
the
reason
the
plan
was
developed,
namely,
responding
to
Clause
6
of
the
settlement
agreement.
In
addition,
only
the
Plan
document
identifies
Method
1631B;
the
other
documents
seem
to
address
EPA
methods
in
general.
Consequently,
I
will
try
to
address
all
the
review
considerations
starting
with
the
Plan.

The
Plan
is
too
vague
and
lacking
in
technical
detail
to
generate
an
assessment
of
its
scientific
merit.
The
approach
described
in
the
Plan
does
not
contain
sufficient
detail
to
determine
whether
or
not
it
is
valid.
Neither
does
it
contain
enough
information
with
respect
to
how
the
assessment
will
be
done
to
determine
the
likelihood
that
it
will
be
objective
and/
or
useful.
It
is
also
not
clear
whether
the
Plan
is
intended
to
address
only
Method
1631B
or
EPA
methods
in
general.
If
it
is
EPA
methods
in
general,
the
magnitude
of
the
job
is
much,
much
greater
and
it
seems
unlikely
that
a
valid
technical
effort
could
be
conducted
in
the
time
frame
identified.

The
Plan
identifies
4
key
items
associated
with
satisfying
the
requirements
of
Clause
6
of
the
settlement
agreement.
These
are:
Reassessment
of
EPA's
existing
MDL
and
ML
procedures;
External
review
of
the
reassessment;
External
review
of
any
alternate
procedures
EPA
has
under
consideration;
and
Proposal
of
revised/
alternate
procedures,
if
warranted.
This
is
to
be
accomplished
by
the
end
of
February
2003.
To
this
end,
EPA
has
established
a
series
of
tasks
and
milestones
leading
up
to
the
due
date.
Although
not
specifically
stated,
it
is
presumed
that
a
document
will
be
generated
which
addresses
the
four
items
in
Clause
6.
Also,
because
Method
1631B
is
cited
in
the
Introduction
along
with
Clause
6,
it
is
assumed
that
the
Plan
document
is
intended
to
address
only
this
method.

EPA
has
organized
this
effort
into
16
tasks/
milestones
that
are
identified
in
a
chart
on
page
12.
The
first
two
tasks
have
been
completed,
the
resulting
output
being
the
documents
under
review.
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
7
of
18
The
organization
of
the
remaining
tasks,
however,
is
somewhat
confusing.
First,
there
seem
to
be
six
first­
draft
reports,
excluding
the
Federal
Register
Notice,
plus
two
second­
draft
reports.
I
suspect
these
are
sections
in
the
final
report
document
but
this
is
not
clear.
It
appears
that
the
drafts
are
assembled
into
a
single
report
as
the
interim,
and
only
milestone,
of
Task
9.
External
Peer
Review
is
then
preformed
over
a
three­
month
period
shown
as
Task
10,
starting
in
May
of
`
02.
Final
selection
of
the
"
Recommend
Option"
takes
place
as
a
milestone
(
task
9)
after
External
Peer
Review
is
complete.

However,
even
though
the
report
has
been
sent
out
for
external
peer
review,
development
tasks
(
Tasks
3,
4
and
7),
evaluation
tasks
(
Tasks
5
and
8),
and
the
reassessment
task
(
Task
6),
continue
to
run
on
through
Feb
`
03.
Consequently,
it
appears
that
the
material
developed
in
these
tasks
after
April
'
02
will
not
be
subject
to
external
peer
review.
This
appears
to
be
inconsistent
with
the
requirements
of
Clause
6.
Further,
why
should
these
tasks
continue
past
the
time
the
Recommended
Option
has
been
selected?
If
the
tasks
continue
does
that
mean
some
other
Option
could
replace
the
one
identified
in
task
11?

In
addition,
some
of
the
tasks
identified
in
the
Plan
do
not
appear
to
be
clearly
related
to
dealing
with
the
four
points
of
Clause
6.
The
first
point
of
Clause
6
is
addressed
by
Task
6,
the
second
point
by
Tasks
9
and
10,
the
third
point
by
Tasks
8,
9
and
10,
and
the
fourth
point
by
Task
7.
Tasks
1
and
2
were
used
to
create
the
documents
reviewed
her.
That
leaves
Tasks
12
through
16,
which
appear
to
be
EPA
administrative
issues
that
must
fit
into
the
time
line
prior
to
the
completion
date.
If
this
is
the
case,
it
is
difficult
to
understand
how
the
development,
evaluation
and
reassessment
tasks
identified
above
could
continue
while
EPA
is
writing
a
Federal
Register
Notice
(
Task
12),
the
Agency
is
reviewing
(
Task
13)
and
OMB
is
reviewing
(
Task
14).

At
the
end
of
the
Introduction
to
the
Plan,
there
is
a
statement
that
the
"
document
contains
the
detailed
plan
for
the
reassessment
called
for
in
Clause
6
 ".
In
fact
the
Plan
contains
very
little
detail.
The
only
thing
that
comes
close
to
detail
is
the
listing
of
"
issues"
that
are
part
of
Task
3.
There
is
no
description
of
the
processes
and/
or
criteria
that
will
be
used
in
the
reassessment.
EPA
should
identify
the
specific
criteria
that
will
be
used
to
select
the
most
appropriate
mechanism(
s)
for
determining
detection
limit(
s)
and/
or
quantitation
limit(
s).
If
this
is
not
done
up
front,
it
may
appear
that
EPA
first
selected
mechanisms,
then
created
justification
to
support
them.
This
is
what
Clause
6
seems
to
be
trying
to
get
away
from.

Presumably
the
people
involved
have
the
skills
and
knowledge,
and
EPA
will
allow
them
the
time
and
provide
the
resources,
needed
to
do
the
job.
Unfortunately
there
is
no
way
to
assess
this
based
on
the
Plan.
EPA
identifies
a
"
Core
Workgroup
consisting
of
measurement
analysts
and
statisticians
within
the
Office
of
Science
and
Technology"
but
does
not
provide
information
on
their
background
or
experience.
The
success
of
this
project
will
depend
on
the
people
involved,
so
this
is
critical
information.

The
Plan
indicates
that
Episode
6000
data
will
be
reviewed.
Are
these
the
only
data
that
will
be
reviewed?
Are
these
data
associated
with
Method
1631B?
Are
there
other
data
that
could
be
used
in
addition?
If
so,
will
they
be
used
or
excluded?
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
8
of
18
In
summary,
it
is
very
difficult
to
see
how
this
Plan
will
result
in
EPA
being
where
they
want
to
be
when
they
want
to
be
there.
They
may
succeed,
but
working
to
this
Plan
will
not
be
the
reason
for
their
success.
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
9
of
18
V.
RESPONSE
TO
CHARGE
QUESTIONS
Reviewer
A
The
assessment
approach
is
conceptually
sound.
The
list
of
issues
in
Item
3
is
sufficient.
I
have
some
comments
below
[
The
comments
that
Reviewer
A
is
referring
to
are
provided
in
the
General
Comments
section
of
this
report]
on
what
I
think
to
be
the
most
significant,
high­
priority
issues.

Reviewer
C
Events
3
through
8
give
chief
steps
in
the
assessment
approach
which
is
proposed.
These
seem
to
thoroughly
cover
many
relevant
issues.
Key
to
the
assessment
plan
is
Event
3,
since
so
many
important
issues
are
identified.
Giving
adequate
attention
to
these
issues
up
front
should
help
to
insure
that
the
key
issues
are
understood
so
that
careful
choices
can
be
made
before
the
evaluation
and
selection
begins.
I
can't
think
of
anything
to
add
to
the
list
for
Event
3.
At
the
same
time,
I'd
be
hesitant
to
delete
anything
at
this
point,
although
I
think
that
it
will
be
the
case
that
some
issues
will
be
found
to
be
not
nearly
as
complex
as
other
issues,
and
that
these
simpler
issues
can
be
addressed
somewhat
quickly.

With
regard
to
prioritizing
the
issues
listed
for
Event
3,
items
a,
b,
f,
and
o
seem
to
be
important
and
may
also
require
more
time
than
some
of
the
other
items.
However
I
suspect
that
at
least
some
parts
of
the
literature
review
are
well
underway
(
or
complete).
Still,
it
will
be
very
important
to
take
the
results
of
the
literature
review
and
carefully
organize
and
summarize
the
material
(
as
opposed
to
just
providing
a
lengthy
listing
of
issues
and
proposed
methods),
and
this
may
prove
to
be
a
rather
difficult
job.
Item
f
deals
with
the
experimental
design
of
the
studies
that
will
be
performed
to
test
various
methods.
This
seems
to
be
a
very
important
part
of
the
overall
plan,
since
the
design
should
take
into
account
many
important
issues
that
are
identified
in
completing
the
many
other
items
of
Event
3.
Overall,
it
seems
that
a
large­
scale
and
carefully
performed
experimental
comparison
of
a
good
number
of
well
selected
methods
will
be
how
one
finds
out
which
methods
are
the
most
accurate.
Since
time
and
resource
constraints
always
tend
to
limit
the
size
of
studies
that
are
to
be
performed,
it
is
important
to
use
a
good
design
in
order
to
produce
the
most
effective
results.
Items
b
and
o
touch
upon
legal
and
regulatory
concerns,
and
thus
may
be
somewhat
complex.

Important
items
which
might
not
be
so
time
consuming
include
items
e
and
i.
Item
e
deals
with
the
development
of
detection/
quantitation
procedures
based
on
statistical
tolerance
and
prediction.
Item
i,
which
calls
for
consideration
of
false
positives
and
false
negatives,
shouldn't
take
much
time,
but
it
is
of
the
upmost
importance
since
the
definition
of
MDL
utilizes
the
probability
of
a
false
positive.
False
negatives
are
also
important,
since
they
are
cases
where
contaminants
are
1.
Evaluate
the
conceptual
soundness
of
the
assessment
approach
presented
in
the
study
plan.
Is
the
list
of
issues
included
in
Item
3
of
the
study
plan
sufficient?
Are
there
other
significant
issues
that
EPA
should
add
or
delete?
How
should
EPA
prioritize
the
issues?
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
10
of
18
present,
but
a
nondetect
is
recorded.
Since
false
results
are
such
a
key
issue,
it
will
be
important
that
they
are
carefully
considered
when
the
experimental
design
is
developed.
For
example,
if
the
false
positive
rate
is
to
be
controlled
to
be
no
greater
than
0.01,
then
a
large
number
of
trials
under
the
null
hypothesis
condition
of
no
analyte
present
must
be
performed
to
accurately
assess
whether
or
not
the
false
positive
rate
is
adequately
controlled.
Also,
one
needs
to
carefully
consider
the
various
ways
in
which
hypothesis
testing
can
be
used
in
the
assessment.
If
the
null
hypothesis
is
taken
to
be
p
<=
0.01,
where
p
is
the
probability
of
a
false
positive
resulting
from
a
particular
MDL
method,
then
one
has
to
keep
in
mind
that
a
failure
to
reject
does
not
provide
statistically
significant
evidence
that
p
<=
0.01,
since
a
failure
to
reject
could
be
due
to
low
power,
resulting
from
too
few
experimental
trials.
To
truly
protect
against
a
method
having
a
false
positive
rate
exceeding
0.01,
one
may
wish
to
make
p
>
0.01
the
null
hypothesis
and
p
<=
0.01
the
alternate
hypothesis.
It
can
get
to
be
a
bit
confusing,
since
there
are
two
different
null
hypothesis
situations
(
at
two
different
levels):
the
null
situation
of
no
analyte
present,
for
which
we
are
concerned
about
the
false
positive
rate
of
an
MDL
method,
and
the
null
hypothesis
that
when
no
analyte
is
present,
the
false
positive
rate
is
greater
than
0.01,
in
which
case
a
type
I
error
would
be
a
claim
that
the
rate
is
less
than
or
equal
to
0.01
when
in
fact
it
is
not.

While
I
do
not
think
that
items
c,
d,
and
m
are
by
themselves
unimportant,
I
do
think
that
it's
important
to
view
them
as
being
linked,
and
to
keep
in
mind
that
the
main
concern
is
to
determine
which
methods
for
determining
the
MDL
have
the
greatest
accuracy,
and
not
which
methods
and
models
are
best
for
a
particular
stage
in
the
overall
determination
of
the
MDL.
This
is
particularly
true
if
the
model
upon
which
a
method
is
based
is
only
a
loose
approximation
of
reality,
in
which
case
the
interpretation
of
parameters
may
be
viewed
as
being
rather
fuzzy.
In
the
end,
it's
whether
or
not
a
particular
method
is
reasonably
accurate
­­­
viewing
the
performance
of
the
overall
method,
and
not
examining
too
closely
each
of
it's
components.
Rather
than
spend
so
much
time
on
model
selection
and
assessment,
and
parameter
estimation,
it
may
be
better
to
put
more
resources
into
the
testing
stage,
and
thoroughly
test
a
large
number
of
methods
which
have
been
proposed
by
others.
However,
I
do
think
that
some
thought
can
go
into
parameter
estimation.
For
example,
if
one
is
uncertain
about
how
to
identify
and
handle
outliers,
one
could
replace
the
commonly
used
sample
standard
deviation
(
it's
used
in
Method
1631B)
with
a
more
robust
measure
of
scale.
But
again,
it's
important
to
keep
in
mind
the
overall
goal
of
finding
an
accurate
way
to
determine
the
MDL.
It's
not
so
important
to
know
whether
or
not
a
robust
measure
of
scale
is
a
better
estimator
in
the
traditional
sense
of
estimating
a
true
distribution
parameter
accurately.
Rather,
what
is
important
is
finding
out
whether
or
not
replacing
the
sample
standard
deviation
with
a
robust
measure
improves
the
performance
of
the
overall
determination
of
the
MDL.
Fortunately,
in
some
cases,
using
alternative
estimators,
in
addition
to
the
one
originally
proposed
for
use
with
a
particular
method,
won't
require
that
additional
observations
be
made,
but
will
only
require
that
alternate
determinations
of
the
MDL
be
made
using
the
same
set
of
data.

Reviewer
E
I
think
the
Plan
is
sufficient
and
would
prioritize
the
need
for
MDLs
that
are
performance
driven
by
the
labs.
EPA
only
provides
some
guidance
with
ranges
of
MDLs
in
a
method.

Reviewer
F
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
11
of
18
Conceptual
soundness
cannot
be
evaluated
due
to
insufficient
detail.
No
description
is
provided
on
what
parameters
and/
or
criteria
will
be
used
to
assess
methods
for
identifying
detection
limit
or
quantitation
limit.
If
one
technique/
method
is
to
be
better
than
another,
what
makes
it
better?

Reviewer
A
Here
are
some
of
what
I
think
to
be
the
most
important
criteria
for
lower
limits
of
analytical
procedures:

a)
Definitions
should
be
consistent,
scientifically
meaningful,
and
statistically
valid.
For
some
specific
comments
on
this
issue,
see
below.
b)
Procedures
that
implement
the
definitions
should
be
clear
and
well
defined.
They
should
also
implement
the
definition.
c)
The
data
used
to
establish
specific
limits
should
be
realistic.
For
example,
if
matrix
effects
are
presumed
to
be
important,
then
matrix
spikes
rather
than
reagent
water
spikes
should
be
used.
d)
There
should
be
no
artificial
limits.
Each
defined
and
implemented
limit
should
be
based
on
a
scientific
and
statistical
analysis
of
the
true
limits
of
knowledge,
not
on
preferences
of
involved
parties
for
higher
or
lower
levels.
If
we
can
measure
something
and
provide
an
estimated
standard
deviation,
then
we
should
be
able
to
use
the
data
in
science
and
in
regulation.

Reviewer
C
I
think
that
the
main
focus
should
be
the
probability
of
a
false
positive
result
in
the
null
case
of
no
analyte
present.
This
probability
should
be
as
close
as
possible
to
the
allowable
limit
(
say
0.01)
without
going
over
the
limit.
While
exceeding
the
limit
is
particularly
bad,
since
false
detections
could
result
in
lawsuits
and
also
contribute
to
a
lack
of
confidence
in
water
quality
studies,
being
too
far
under
the
limit,
which
results
from
MDL
determinations
that
are
larger
than
they
need
to
be,
hurts
the
power
of
the
method
to
detect
the
analyte
when
it
is
present.
Alternatively,
and
perhaps
nearly
equivalently,
one
can
view
the
situation
in
terms
of
both
false
positives
and
false
negatives,
with
a
goal
being
to
find
the
method
with
the
smallest
false
negative
rate
that
also
satisfies
the
false
positive
criterion.
The
trouble
with
this
approach
is
that
the
false
negative
rate
will
almost
surely
depend
on
how
much
analyte
is
present,
and
one
method
may
be
better
than
another
at
low
levels
while
the
opposite
is
true
at
higher
levels.
(
Putting
it
in
statistical
terms,
there
may
not
be
a
uniformly
most
powerful
test.)

It
should
be
kept
in
mind
that
the
methods
will
be
used
in
a
variety
of
settings,
and
so
the
null
situation
of
no
analyte
present
should
be
tested
with
a
variety
of
water
sources.
If
one
observes
differing
false
positive
rates
for
the
variety
of
water
sources
(
all
having
no
analyte
present),
then
rather
than
average
all
of
the
observed
false
positive
rates,
I
think
it
may
be
prudent
to
focus
on
2.
Based
on
your
understanding
of
the
issues,
are
there
any
types
of
criteria
that
you
suggest
EPA
include
to
evaluate
detection
and
quantitation
procedures?
If
yes,
please
identify
the
criterion
and
provide
a
rationale
for
its
inclusion.
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
12
of
18
the
highest
value
and
require
that
it
satisfy
the
false
positive
criterion,
since
one
will
want
to
believe
that
the
false
positive
rate
limit
is
adhered
to
no
matter
what
the
circumstances
(
within
reason)
may
be.

In
addition
to
focusing
on
the
false
positive
rate,
which
is
tied
to
the
definition
of
the
MDL,
I
also
think
that
the
MDL
values
from
the
various
methods
investigated
should
be
compared
to
an
estimate
of
the
99th
percentile
of
the
null
distribution,
by
which
I
mean
the
distribution
of
determinations
obtained
in
the
case
of
no
analyte
present.
It
seems
to
me
that
the
99th
percentile
of
this
distribution
could
be
viewed
as
a
value
for
the
MDL.
Using
any
value
less
than
this
99th
percentile
as
the
MDL
would
mean
that
if
no
analyte
were
present,
the
probability
of
obtaining
a
measured
concentration
greater
than
the
MDL
would
exceed
0.01,
and
thus
violate
Glaser's
definition
(
which
is
also
the
definition
given
in
Appendix
B
to
Part
136).
To
estimate
the
99th
percentile
of
the
null
distribution,
a
simple
nonparametric
quantile
estimator
could
be
used
(
with
the
choice
of
estimator
being
dependent
on
the
shape
of
the
distribution
and
the
number
of
observations
(
where
here
each
observation
is
a
determination
of
the
concentration
of
analyte
from
a
sample
in
which
no
analyte
is
present)).
It
should
be
kept
in
mind
that
in
order
to
get
a
decent
estimate
of
the
99th
percentile
of
a
distribution,
it
may
take
several
hundred
observations.
So
this
method
of
determining
the
MDL
may
be
impractical
for
ordinary
use.
But
perhaps
it
can
be
used
as
a
spot
check
in
at
least
a
small
number
of
cases.

I
think
claims
made
in
the
literature
about
various
methods
should
not
be
used
to
judge
them,
since
previous
investigations
of
accuracy
were
not
done
the
same
way
for
all
of
the
methods.
Instead,
the
determination
of
the
most
reliable
methods
should
be
based
on
the
results
of
a
large
well­
designed
and
properly
executed
experiment
(
or
series
of
experiments).

Finally,
with
regard
to
criteria,
some
weight
has
to
be
put
on
simplicity
and
ease
of
use.
The
method(
s)
selected
will
be
used
by
a
large
number
of
people,
and
some
will
be
more
skilled
than
others.

Reviewer
E
I
do
not
have
any
criteria
in
mind.

Reviewer
F
EPA
has
not
provided
a
single
criterion
for
performing
this
evaluation.
Without
some
idea
of
what
they
are
planning
to
do,
it
is
not
possible
to
suggest
alternatives.

Reviewer
A
I
believe
that
the
EPA
should
take
a
fresh
look
at
all
of
the
suggested
definitions
and
procedures
It
is
correct
to
evaluate
suggestions
from
the
scientific
literature,
from
standards
organizations,
3.
What
existing
detection
and
quantitation
procedures
should
EPA
evaluate
or
focus
on?
Procedures
from
the
open
scientific
literature?
Voluntary
consensus
standards
procedures?
Procedures
from
other
organizations?
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
13
of
18
and
from
other
organizations.
However,
it
may
be
that
none
of
these
procedures
meets
the
criteria
of
2),
and
meeting
those
criteria
is
far
more
important
than
precedent.

Reviewer
C
Of
course,
methods
currently
used
by
the
EPA,
such
as
Method
1631B,
should
be
included.
In
addition,
I'm
in
favor
of
including
several
promising
methods
that
are
to
be
identified
from
the
open
scientific
literature,
as
well
as
perhaps
a
small
number
of
alternate
procedures
that
are
used
by
other
organizations,
provided
that
some
justification
can
be
found
for
them
other
than
that
they
are
sometimes
used.
(
I
fear
that
sometimes
a
method
can
become
popular
for
no
real
good
reason.
One
organization
may
select
a
method
for
use
somewhat
haphazardly,
and
then
another
organization
can
pick
up
the
method
just
because
others
have
used
it.
Since
methods
are
sometimes
selected
based
in
part
on
political
considerations,
or
concerns
about
practicality
that
may
not
be
pertinent
in
all
cases,
I
tend
not
to
give
a
lot
of
weight
to
the
popularity
of
methods,
and
I'm
much
more
impressed
by
studies,
performed
by
people
other
than
the
originators
of
the
method,
that
demonstrate
that
a
method
is
reasonably
accurate.)

Methods
from
the
literature
are
nice
to
consider,
because
as
time
passes,
refinements
seem
to
be
made
upon
previously
proposed
methods,
and
one
can
be
hopeful
that
the
class
of
methods
that
get
the
most
attention
of
this
sort
are
the
ones
that
perform
best,
and
thus
hope
that
recent
papers
giving
the
latest
adjustments
to
an
established
general
scheme
will
include
some
that
are
(
near)
state­
of­
the­
art.
Of
course,
one
has
to
be
wary
of
authors
who
are
too
eager
to
publish
and
develop
a
small
adjustment
in
hope
of
getting
a
quick
and
easy
publication.
Also,
since
many
published
methods
seem
to
be
highly
derivative
of
previously
published
ones,
there
might
not
be
enough
variety
found
in
the
literature
and
so
one
may
want
to
consider
other
sources.

With
regard
to
identifying
alternate
methods
from
the
literature
review,
I
think
that
the
1998
memorandum
from
Raphael
Kuznetsovski
of
SAIC
should
be
of
great
help.
There
are
27
papers
and
books
described
in
the
method
detection
limits
and
quantification
section.
The
summaries
provided
for
each
of
the
27
works
can
be
used
to
narrow
the
focus
to
perhaps
a
dozen
or
fewer
key
entries
worthy
of
careful
consideration.
In
general,
I
recommend
examining
recent
articles,
slightly
older
articles
from
the
chemistry
literature
by
authors
who
are
well
established
in
the
field,
and
also
articles
from
the
statistical
literature
in
order
to
perhaps
gain
a
slightly
different
perspective.

I'll
now
make
some
brief
comments
about
some
of
the
works
described
in
the
SAIC
memo.
(
Since
complete
citations
are
provided
in
that
memo,
I
won't
duplicate
such
information
here.)
Several
of
the
papers
pertain
to
the
Hubaux
and
Vos
approach:
the
original
1970
paper,
the
1978
paper
by
Bayley,
Cox,
and
Springer,
and
the
1997
paper
by
Coleman,
Auses,
and
Grams,
with
this
last
paper
being
by
far
the
most
recent
of
the
three,
and
perhaps
worthy
of
a
close
look.
Gibbons
has
written
a
lot
of
pertinent
articles,
and
I
think
several
of
his
works
should
be
studied,
particularly
those
which
compare
various
methods,
like
his
1994
book.
Similarly,
Currie
has
written
a
lot,
but
some
of
his
work
is
rather
dated,
and
some
of
it
deals
with
related
issues
and
not
particular
methods.
But
his
1997
paper
may
be
of
interest,
since
it
touches
on
assumptions
and
approximations.
Davis'
1994
article,
the
1991
paper
of
Lambert,
Peterson,
and
Terpenning,
and
the
1997
paper
of
Spiegelman
and
Tarlow
are
examples
of
works
from
the
statistical
literature
that
could
be
of
interest,
and
the
1987
paper
of
Clayton,
Hines,
and
Elkins
and
the
1988
paper
of
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
14
of
18
Garner
and
Robertson
are
papers
from
the
chemical
literature
that
could
be
considered,
with
the
later
paper
of
particular
interest
since
it
touches
on
the
issue
of
assumptions.

Finally,
I
think
that
it
will
be
important
to
consider
variety
with
regard
to
the
complexity
of
the
methods
to
be
chosen.
It
might
be
good
to
include
some
simple
methods
in
addition
to
those
with
are
rather
involved.
Not
only
are
the
more
complex
methods
more
difficult
for
some
to
accurately
use,
but
it
may
be
the
case
that
some
of
the
simpler
methods
could
outperform
some
of
the
complex
ones,
due
to
poor
assumptions
underlying
some
of
the
methods.

Reviewer
E
I
think
EPA
should
focus
on
the
MDL
approach
and
the
detection
limit
should
be
performance
driven.
Each
lab
should
develop
their
own
MDLs
and
the
EPA
method
should
provide
some
examples
and
some
expectations
for
each
analytes
and
matrix
but
let
the
labs
demonstrate
their
own
MDLs.
The
"
market"
will
then
drive
the
MDLs.
If
there
is
a
need
for
more
sensitive
MDLs
then
the
successful
labs
will
improve
their
MDLs.

Reviewer
F
I
believe
the
charge
to
EPA
is
that
they
look
at
all
existing
procedures,
see
if
they
can
come
up
with
any
new
ones
and
then,
with
some
predefined
rational,
pick
the
best
one.
This
sounds
like
a
justifiable
charge.
If
the
intent
of
this
question
is
to
determine
if
I
have
a
favorite
that
I
want
to
be
sure
is
on
the
list,
I
do
not.

Reviewer
A
I
am
familiar
with
some
of
the
data
sets
proposed,
but
not
all
of
them.
Data
sets
for
a
procedure
like
this
one
should
have
several
to
many
replicates
at
a
number
of
concentrations
from
zero
or
near
zero,
to
near
the
critical
level
(
see
below)
to
high
concentrations.

Reviewer
C
Based
on
my
rather
limited
knowledge
about
the
data
sets
referred
to,
they
do
seem
appropriate
for
use.
At
this
time,
I
don't
have
any
other
data
sets
to
recommend.
However,
it
may
be
good
to
produce
some
new
data
for
this
important
study,
since
existing
data
may
not
be
ideal.

Reviewer
E
I
am
aware
of
MDL
data
sets
for
the
MDLs
in
method
1638
that
could
be
evaluated.

Reviewer
F
There
is
only
one
data
set
identified
in
the
Plan,
Episode
6000
data.
If
these
are
mercury
data
derived
from
Method
1631B,
they
are
appropriate
for
that
method.
However,
because
I
haven't
4.
Are
the
data
sets
listed
in
the
plan
appropriate
for
use
in
the
detection
and
quantitation
reassessment?
Are
there
other
data
sets
that
you
recommend
EPA
evaluate?
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
15
of
18
seen
them,
I
don't
know.
However,
I
would
be
surprised
to
hear
that
Episode
6000
data
are
applicable
to
all
EPA
methods.
If
they
are,
how
was
this
determined?
I
know
EPA
has
other
data
associated
detection
studies
because
I've
seen
EPA
statisticians
present
them
at
various
technical
meetings.
Also,
based
on
the
references
identified
in
the
memo
to
Chuck
White
from
Raphael
Kuznetsovski,
it
would
appear
that
there
are
a
number
of
other
data
sets
outside
EPA.
Perhaps
the
authors
would
be
willing
to
share
them.
What
I'm
not
sure
of
is
whether
or
not
the
Plan
is
intended
to
address
other
EPA
methods.

Reviewer
A
It
is
better
to
develop
selection
criteria
during
the
development
of
the
project.
However,
possible
criteria
should
be
included
in
the
plan.

Reviewer
C
I
believe
that
it
is
better
to
firm
up
the
selection
criteria
as
the
study
progresses,
thinking
that
people
will
develop
a
better
understanding
of
the
issues
as
they
put
more
thought
into
them.
Plus,
instead
of
specifying
the
criteria
up
front,
it
may
be
better
to
wait
and
see
what
happens
and
then
examine
the
pros
and
cons
of
the
various
methods
when
making
specific
comparisons.

Reviewer
E
I
think
the
selection
criteria
should
be
included
so
each
lab
demonstrates
their
MDLs
by
analysis
of
replicate
samples
of
actual
field
samples
,
either
spiked
or
unspiked
depending
on
the
concentration
in
the
matrix.
EPA
would
provide
the
guidance
in
terms
of
number
of
replicates.

Reviewer
F
It
is
not
just
"
appropriate"
to
include
specific
selection
criteria,
it
is
essential
if
an
objective
assessment
is
to
be
conducted.
Otherwise
there
is
the
specter
that
EPA
has
simply
selected
criteria
that
support
the
conclusion(
s)
that
they
wish
to
make.
This
is
likely
to
be
obvious
to
the
plaintiffs
in
the
suit
against
EPA.
5.
Is
it
appropriate
to
include
specific
selection
criteria
in
the
plan
or
is
it
more
appropriate
to
develop
selection
criteria
based
on
the
analysis
of
issues
that
will
be
conducted
as
part
of
the
plan?
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
16
of
18
VI.
SPECIFIC
COMMENTS
Reviewer
E
Page
4.
Section
ii.
Method
Development
and
Promulgation
I
strongly
believe
that
detection
limits
should
not
be
cast
in
stone
when
the
method
is
promulgated.
The
DL's
often
improve
radically
as
labs
become
more
proficient
and
improvements
are
made
to
techniques.
I
have
experienced
clients
and
regulators
who
flag
data
as
below
the
DL
because
it
is
below
the
Method­
specified
DL
but
well
above
our
achieved
DL!

Reviewer
E
Page
5
Section
vii.
Descriptive
versus
Prescriptive
Uses
of
Lower
Limits
to
Measurement
I
am
very
much
in
agreement
with
this
statement
which
I
assume
means
that
the
detection
limit
can
change
depending
on
the
measurement
method
and
the
proficiency
of
the
Lab.

Reviewer
E
Page
6
Section
j.
Measurement
Quality
Over
the
Life
of
a
Method
I
agree
with
this
statement
that
the
measurement
quality
will
usually
improve
over
the
life
of
the
method.
This
is
a
very
important
issue
and
therefore
the
DL
for
a
method
should
be
allowed
to
change
as
the
equipment
and
analyst
change.

Reviewer
E
Page
7
Section
m.
Outliers
The
term
"
outliers"
is
confusing
as
used
in
the
statement.
My
understanding
of
outliers
are
data
points
that
can
be
separated
from
the
other
data
points
with
a
statistical
test
called
an
"
outlier
test".
There
should
be
an
acceptable
procedure
to
determine
when
data
is
not
appropriate
to
be
included
in
an
MDL.

Reviewer
E
Page
8
Section
t.
Cost
to
Implement
Limit
Procedures
A
procedure
for
developing
"
theoretical
MDL's"
for
solids,
such
as
sediment
and
tissue,
based
on
liquid
MDL's
should
be
explored.
For
example
a
true
MDL
for
mercury
in
fish
tissue
can
not
be
experimentally
determined
because
all
fish
tissue
has
relatively
high
levels
of
mercury
compare
to
the
sensitivity
of
the
method.
However,
using
water
derived
MDL
it
is
possible
to
calculate
the
MDL
for
fish.
This
situation
is
true
for
many
metals
that
are
naturally
abundant
in
sediment,
soil
and
tissue.

Reviewer
F
Page
10
There
are
a
few
typos,
judgment
is
misspelled,
and
Task
16
on
page
10
should
identify
December
2002,
not
2001.
It's
not
the
nits
that
need
to
be
addressed,
it's
organization
and
the
lack
of
detail.
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
17
of
18
Reviewer
C
Page
12,
Schedule
In
the
schedule
given
on
page
12
some
events
seem
out
of
sequence.
From
pages
8
and
9
it
seems
as
though
Events
5
and
6,
which
deal
with
evaluation
and
reassessment,
should
follow
Event
4,
which
is
criteria
development,
and
this
seems
sensible
to
me.
Yet
on
page
12,
it
appears
as
though
Event
4
lags
behind
Events
5
and
6.
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
Page
18
of
18
VII.
MISCELLANEOUS
COMMENTS
Reviewer
C
This
doesn't
pertain
explicitly
to
the
12
page
plan.
But
in
reviewing
supplementary
material
that
was
sent
to
me
along
with
the
plan,
I
noticed
a
few
typos
in
Appendix
B
to
Part
136.
The
most
serious
mistake
is
in
part
(
b)
of
Step
7.
If
the
ratio
of
variances
is
less
than
3.05,
then
one
would
compute
the
pooled
standard
deviation.
So
the
first
>
3.05
should
be
<
3.05.
(
One
can
check
this
in
the
original
description
of
the
method
given
in
the
1981
paper
by
Glaser,
Foest,
McKee,
Quane,
and
Budde.)
Less
serious
are
two
minor
typos:
(
i)
below
the
sum
at
the
bottom
of
page
2
it
should
just
be
i
=
1,
not
i
=
1S;
(
ii)
on
the
3rd
line
of
page
3,
S
should
be
replaced
by
a
sigma.

I'll
also
comment
that
I'm
a
bit
suspicious
of
Method
136B.
The
justification
given
for
the
approach
in
the
paper
by
Glaser
et
al
makes
use
of
assumptions
and
approximations.
Since
the
final
formula
given
for
the
MDL
doesn't
seem
quite
right,
I
wonder
if
perhaps
too
many
assumptions
were
made
along
the
way.
Also,
distribution
skewness
could
harm
the
accuracy
of
the
method.
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
APPENDIX
A
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
APPENDIX
B
Charge
to
Peer
Reviewers
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
APPENDIX
C
Reviewer
A
Curriculum
Vitae
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
APPENDIX
D
Reviewer
C
Curriculum
Vitae
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
APPENDIX
E
Reviewer
E
Curriculum
Vitae
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
APPENDIX
F
Reviewer
F
Curriculum
Vitae
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
APPENDIX
G
Reviewer
A
Comments
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
APPENDIX
H
Reviewer
C
Comments
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
APPENDIX
I
Reviewer
E
Comments
Peer
Review
of
the
Plan
for
the
Assessment
of
Detection
and
Quantitation
Limits
Under
Section
304(
h)
of
the
Clean
Water
Act
February
2002
APPENDIX
J
Reviewer
F
Comments
