Science
Policy
Council
HANDBOOK
United
States
Office
of
Science
Policy
EPA
100­
B­
98­
001
Environmental
Protection
Office
of
Research
and
Development
January
1998
Agency
Washington,
DC
20460
www.
epa.
gov
THIS
PAGE
LEFT
INTENTIONALLY
BLANK
EPA
100­
B­
98­
001
January
1998
U.
S.
Environmental
Protection
Agency
PEER
REVIEW
HANDBOOK
Prepared
for
the
U.
S.
Environmental
Protection
Agency
by
members
of
the
Peer
Review
Advisory
Group,
a
group
of
EPA's
Science
Policy
Council
Principal
Authors
Kerry
L.
Dearfield,
Ph.
D.
A.
Robert
Flaak
Office
of
Science
Policy
Science
Advisory
Board
Office
of
Research
and
Development
Office
of
the
Administrator
Major
Contributors
Jean
C.
Schumann
Nancy
W.
Wentworth
Office
of
Solid
Waste
Office
of
Research
and
and
Emergency
Response
Development
Roland
B.
Hemmett,
Ph.
D.
Arnold
M.
Kuzmack,
Ph.
D.
Region
2
Office
of
Water
Science
Policy
Council
U.
S.
Environmental
Protection
Agency
Washington,
DC
20460
Peer
Review
Handbook
Page
ii
DISCLAIMER
This
document
has
been
reviewed
in
accordance
with
U.
S.
Environmental
Protection
Agency
policy
and
approved
for
publication
and
distribution
to
the
Agency.
Mention
of
trade
names
or
commercial
products
does
not
constitute
endorsement
or
recommendation
for
use.
Peer
Review
Handbook
Page
iii
TABLE
OF
CONTENTS
FOREWORD
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
ix
SUMMARY
OF
THE
PEER
REVIEW
PROCESS
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
1
Figure
1
­
Flowchart
for
Planning
a
Peer
Review
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
2
Figure
2
­
Flowchart
for
Conducting
a
Peer
Review
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
3
Figure
3
­
Flowchart
for
Completing
a
Peer
Review
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
4
Managers
Planning
Checklist
for
Peer
Review
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
5
PEER
REVIEW
GUIDANCE
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
7
1.
THE
NEED
FOR
PEER
REVIEW
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
9
1.1
Overview
Statement
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
9
1.2
Understanding
Peer
Review
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
9
1.2.1
Why
use
Peer
Review?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
9
1.2.2
What
is
Peer
Involvement?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
10
1.2.3
What
is
Peer
Review?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
10
1.2.4
What
is
Peer
Input?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
11
1.2.5
How
is
Peer
Review
Different
from
Peer
Input?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
11
1.2.6
Can
Someone
Who
Provided
Peer
Input
Become
an
Independent
Peer
Reviewer
for
the
Same
Work
Product
Later
in
the
Process?
Page
12
1.2.7
How
is
Peer
Review
Different
from
Public
Comment?
.
.
.
.
.
.
.
.
Page
12
1.2.8
How
is
Peer
Review
Different
from
Stakeholder
Involvement?
.
.
Page
12
1.2.9
What
Role
does
Peer
Review
have
in
the
Regulatory
Development
Process?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
13
1.2.10
What
Role
does
Peer
Review
have
in
Regulatory
Negotiations?
.
Page
14
1.3
Annual
Agency
Reporting
Requirements
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
14
1.3.1
What
are
the
Annual
Reporting
Requirements?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
14
1.3.2
What
Listings
are
Required
for
the
Annual
Reporting?
.
.
.
.
.
.
.
.
Page
14
1.3.3
Is
There
a
Coding
System
for
Work
Products
on
the
Annual
Reporting
Lists?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
16
1.3.4
When
will
the
Handbook
Itself
be
Revised?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
17
1.4
The
Roles
of
People
and
Organizations
in
Peer
Review
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
17
1.4.1
Who
is
Ultimately
Accountable
for
Peer
Review?
.
.
.
.
.
.
.
.
.
.
.
.
Page
17
1.4.2
Who
are
the
Agency
Staff
involved
in
Peer
Review?
.
.
.
.
.
.
.
.
.
.
Page
17
Peer
Review
Handbook
Page
iv
1.4.3
Who
are
the
Decision­
Makers
&
What
are
Their
Responsibilities?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
17
1.4.4
Who
are
the
Peer
Review
Leaders
&
What
are
Their
Responsibilities?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
18
1.4.5
Who
are
the
Peer
Review
Coordinators
&
What
are
Their
Responsibilities?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
19
1.4.6
Who
are
the
Peer
Reviewers?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
21
1.4.7
What
are
the
Responsibilities
of
Peer
Reviewers?
.
.
.
.
.
.
.
.
.
.
.
.
Page
21
1.4.8
What
is
an
Independent
Peer
Reviewer?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
21
1.4.9
When
does
an
Agency
Internal
Peer
Reviewer
Qualify
as
Independent?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
21
1.4.10
What
is
a
Peer
Review
Panel?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
22
1.4.11
What
is
a
Subject
Matter
Expert?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
22
1.4.12
What
is
the
Role
of
the
Science
Policy
Council
(
SPC)?
.
.
.
.
.
.
.
Page
22
1.4.13
What
is
the
Role
of
the
Peer
Review
Advisory
Group
(
PRAG)?
.
Page
22
1.4.14
What
is
the
Role
of
the
Office
of
Research
and
Development
(
ORD)?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
23
2.
PLANNING
A
PEER
REVIEW
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
25
2.1
Overview
Statement
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
25
2.2
Determining
Which
Work
Products
to
Peer
Review
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
25
2.2.1
What
are
Scientific
and
Technical
Work
Products?
.
.
.
.
.
.
.
.
.
.
.
Page
25
2.2.2
What
Scientific
and
Technical
Work
Products
Need
Peer
Review?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
25
2.2.3
How
Does
One
Determine
Whether
a
Scientific
and/
or
Technical
Work
Product
is
"
Major"?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
26
2.2.4
What
Economic
Work
Products
Need
Peer
Review?
.
.
.
.
.
.
.
.
.
.
Page
27
2.2.5
How
Should
Peer
Review
be
Handled
for
Products
Developed
under
an
Interagency
Agreement?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
28
2.2.6
Should
Products
from
Grants,
Contracts
and
Cooperative
Agreements
Receive
Peer
Review?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
28
2.2.7
Should
Site
Specific
Decisions
be
Subject
to
Peer
Review?
.
.
.
.
.
Page
28
2.2.8
Should
NEPA
Products
(
e.
g.,
EISs)
be
Subject
to
Peer
Review?
.
Page
29
2.2.9
Should
Environmental
Regulatory
Models
be
Peer
Reviewed?
.
.
Page
29
2.2.10
Is
Peer
Review
Needed
for
Other
Organization's
Work
Products
that
Have
been
Submitted
to
EPA
for
Use
in
Decision
Making?
.
Page
29
2.2.11
Can
Work
Products
That
are
Not
Determined
to
be
Major
Still
be
Peer
Reviewed?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
30
Peer
Review
Handbook
Page
v
2.3
Determining
Which
Work
Products
Do
Not
Receive
Peer
Review
.
.
.
.
.
Page
30
2.3.1
Are
There
Circumstances
When
a
Major
Work
Product
is
Not
Peer
Reviewed?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
30
2.3.2
What
Products
Normally
Do
Not
Need
Peer
Review?
.
.
.
.
.
.
.
.
.
Page
31
2.4
Choosing
a
Peer
Review
Mechanism
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
32
2.4.1
How
Do
You
Determine
the
Appropriate
Peer
Review
Mechanism?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
32
2.4.2
What
are
Examples
of
Internal
Peer
Review?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
33
2.4.3
What
are
Examples
of
External
Peer
Review?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
34
2.4.4
What
is
the
Role
of
Peer
Review
by
a
Refereed
Scientific
Journal?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
35
2.4.5
Is
Peer
Review
Necessary
when
Journal
Articles
are
Used
in
an
Agency
Work
Product?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
35
2.4.6
When
and
How
Often
Should
Peer
Review
Occur?
.
.
.
.
.
.
.
.
.
.
.
Page
36
2.4.7
What
Factors
are
Considered
in
Setting
the
Time
Frame
for
Peer
Review?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
36
2.4.8
Which
Office/
Region
or
Other
Agency
is
Responsible
for
Conducting
the
Peer
Review?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
37
2.5
Creating
the
Peer
Review
Record
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
37
2.5.1
What
is
the
Peer
Review
Record?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
37
2.5.2
How
Can
the
Peer
Review
Record
Improve
the
Peer
Review
Process?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
37
2.5.3
What
Should
Be
in
the
Peer
Review
Record?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
37
2.5.4
What
Should
I
Do
with
a
Peer
Review
Record
That
Pertains
to
a
Rulemaking
Action?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
38
2.5.5
When
Should
the
Peer
Review
Record
Building
Process
Begin?
.
Page
38
2.5.6
What
are
the
Differences
in
Record
Keeping
for
a
Review
by
an
Individual
Compared
to
a
Panel?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
39
2.5.7
Where
Should
the
Peer
Review
Record
be
Kept
and
For
How
Long?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
39
2.6
Budget
Planning
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
40
2.6.1
What
Budgetary
Factors
Should
I
Consider
in
a
Peer
Review?
.
.
Page
40
2.6.2
What
Input
is
Needed
for
the
Annual
Budget
Formulation
and
Budget
Execution
Process?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
40
2.7
Legal
Considerations
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
40
2.7.1
Are
There
Legal
Ramifications
From
the
Peer
Review
Policy?
.
.
Page
40
2.7.2
Is
Legal
Advice
Needed?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
41
2.7.3
Is
Peer
Review
Subject
to
the
Federal
Advisory
Committee
Act
(
FACA)?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
41
Peer
Review
Handbook
Page
vi
3.
CONDUCTING
A
PEER
REVIEW
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
43
3.1
Overview
Statement
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
43
3.2
Charge
to
the
Peer
Reviewers
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
43
3.2.1
What
is
a
Charge?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
43
3.2.2
What
are
the
Essential
Elements
of
a
Charge?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
44
3.2.3
Where
Can
I
Get
an
Example
of
a
Charge?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
44
3.3
Time
Line
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
44
3.3.1
What
are
the
Factors
in
Scheduling
a
Peer
Review?
.
.
.
.
.
.
.
.
.
.
.
Page
44
3.4
Selection
of
Peer
Reviewers
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
45
3.4.1
What
are
Considerations
for
Selecting
Peer
Reviewers?
.
.
.
.
.
.
.
Page
45
3.4.2
Where
Do
I
Find
Peer
Reviewers?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
45
3.4.3
Are
External
or
Internal
Peer
Reviewers
Preferred?
.
.
.
.
.
.
.
.
.
.
.
Page
46
3.4.4
What
is
Important
in
the
Mix
of
a
Peer
Review
Panel?
.
.
.
.
.
.
.
.
Page
46
3.4.5
What
is
a
Conflict
of
Interest?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
47
3.4.6
What
Techniques
Help
Ensure
Disclosure
and
Appropriate
Resolution
of
Conflicts
of
Interest?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
47
3.4.7
Can
Parties
External
to
EPA
Pay
for
Their
Own
Peer
Reviews?
.
Page
48
3.4.8
Are
There
Constraints
to
Selecting
Peer
Reviewers?
.
.
.
.
.
.
.
.
.
.
Page
49
3.5
Materials
for
Peer
Reviewers
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
49
3.5.1
What
Instructions
Do
You
Give
Peer
Reviewers?
.
.
.
.
.
.
.
.
.
.
.
.
Page
49
3.5.2
What
Materials
Should
be
Sent
to
Peer
Reviewers?
.
.
.
.
.
.
.
.
.
.
.
Page
50
3.5.3
How
Closely
can
EPA
Interact
with
Peer
Reviewers
During
the
Review?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
51
3.6
Peer
Review
Services
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
52
3.6.1
What
are
Gratuitous
Services
for
Peer
Review?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
52
3.6.2
Can
I
Use
a
Contract
to
Obtain
Peer
Review
Services?
.
.
.
.
.
.
.
.
Page
52
3.6.3
How
Do
I
Write
a
Statement
of
Work
for
Contracts?
.
.
.
.
.
.
.
.
.
.
Page
53
3.6.4
What
are
Advisory
and
Assistance
Services
(
AAS)
or
Sensitive
Activities?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
53
3.6.5
What
are
Some
Management
Controls
for
Contracts?
.
.
.
.
.
.
.
.
.
Page
54
3.6.6
Can
the
Agency
Identify
and/
or
Select
Peer
Reviewers
Through
a
Contract?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
56
3.6.7
Can
I
Use
Simplified
Acquisition
Procedures
to
Obtain
Peer
Reviewers?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
57
3.6.8
How
is
Travel
Handled
with
Contracts
or
Purchase
Orders?
.
.
.
.
Page
58
3.6.9
How
is
Travel
Handled
with
Special
Government
Employees?
.
.
Page
59
Peer
Review
Handbook
Page
vii
4.
COMPLETING
A
PEER
REVIEW
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
61
4.1
Overview
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
61
4.2
Final
Work
Product
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
61
4.2.1
How
Do
I
Incorporate
Peer
Review
Comments
into
the
Final
Work
Product?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
61
4.2.2
What
Actions
are
Potentially
Forthcoming
from
Peer
Review?
.
.
Page
62
4.2.3
Can
the
Identity
of
Peer
Reviewers
be
Kept
Anonymous?
.
.
.
.
.
.
Page
62
4.3
Completing
the
Peer
Review
Record
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
63
4.3.1
How
Do
I
Complete
the
Peer
Review
Record?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
63
4.3.2
Where
Should
the
Peer
Review
Records
be
Kept,
and
for
How
Long?
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
63
SUBJECT
INDEX
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
65
COMMONLY
USED
ACRONYMS
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
67
APPENDIX
A
­
Peer
Review
Policy
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
A­
1
Peer
Review
Program
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
A­
2
Peer
Review
and
Peer
Involvement
at
the
U.
S.
Environmental
Protection
Agency
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
A­
5
APPENDIX
B
­
Examples
of
Charges
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
B­
1
CHARGE
EXAMPLE
1
­
Charge
to
Reviewers
for
the
WTI
Draft
Final
Risk
Assessment
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
B­
2
CHARGE
EXAMPLE
2
­
IRIS
Pilot
Program
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
B­
11
CHARGE
EXAMPLE
3
­
Science
Advisory
Board
Review
of
the
Agency's
National
Risk
Management
Research
Laboratory's
(
NRMRL)
Program
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
B­
14
CHARGE
EXAMPLE
4
­
Science
Advisory
Board
(
SAB)
Review
of
the
Technical
Aspects
of
the
Multi­
Agency
Radiation
Survey
and
Site
Investigation
Manual
(
MARSSIM)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
B­
15
CHARGE
EXAMPLE
5
­
Science
Advisory
Board
(
SAB)
Review
of
the
Statistical
Performance
of
the
Agency's
Protozoan
Oocyst
Monitoring
Methods
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
B­
16
CHARGE
EXAMPLE
6
­
Science
Advisory
Board
(
SAB)
Review
of
the
Environment
Monitoring
and
Assessment
Program
(
EMAP)
Research
Strategy
and
Research
Plan
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
B­
17
Peer
Review
Handbook
Page
viii
APPENDIX
C
­
Guidance
on
Requesting
a
Review
by
the
Science
Advisory
Board
(
SAB)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
C­
1
APPENDIX
D
­
Example
Statements
of
Work
for
Contracts
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
D­
1
STATEMENT
OF
WORK
­
EXAMPLE
1
­
Statement
of
Work:
Technical
Review
Contractor
for
Panel
Review
of
Assistance
Agreement
or
Fellowship
Applications
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
D­
2
STATEMENT
OF
WORK
­
EXAMPLE
2
­
Peer
Review
of
Prioritization
Tool
Report
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
D­
5
STATEMENT
OF
WORK
­
EXAMPLE
3
­
External
Peer
Review
of
Protozoa
Method
Development
Criteria
Document
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
D­
12
APPENDIX
E
­
References
Concerning
Peer
Review
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Page
E­
1
Peer
Review
Handbook
Page
ix
The
Policy
requires
peer
review
of
the
basis
of
the
decision
(
i.
e.,
the
underlying
major
scientific
and/
or
technical
work
products),
not
the
decision
itself.
FOREWORD
EPA's
Science
Policy
Council
(
SPC)
has
organized
this
Peer
Review
Handbook
as
guidance
to
EPA
staff
and
managers
on
the
organization
and
conduct
of
peer
review
pursuant
to
the
Administrator's
June
7,
1994
Peer
Review
Policy
statement.
The
Handbook
is
based
in
part
on
the
central
themes
set
forth
in
the
Policy
statement
(
see
Appendix
A
for
the
full
policy):

Major
scientifically
and
technically
based
work
products
related
to
Agency
decisions
normally
should
be
peer
reviewed.
Agency
managers
within
Headquarters,
Regions,
laboratories,
and
field
components
determine
and
are
accountable
for
the
decision
whether
to
employ
peer
review
in
particular
instances
and,
if
so,
its
character,
scope,
and
timing.
These
decisions
are
made
in
conformance
with
program
goals
and
priorities,
resource
constraints,
and
statutory
or
court­
ordered
deadlines.
For
those
work
products
that
are
intended
to
support
the
most
important
decisions
or
that
have
special
importance
in
their
own
right,
external
peer
review
is
the
procedure
of
choice.
Peer
review
is
not
restricted
to
the
penultimate
version
of
work
products;
in
fact,
peer
review
at
the
planning
stage
can
often
be
extremely
beneficial.

In
addition,
the
Handbook
augments
these
themes
by
stating
and
explaining
widelyaccepted
principles
and
practices
that
have
long
guided
peer
review
in
the
universities,
in
private
research
organizations,
and
at
the
EPA
and
other
government
agencies.

The
goal
of
the
Peer
Review
Policy
and
this
Handbook
is
to
enhance
the
quality
and
credibility
of
Agency
decisions
by
ensuring
that
the
scientific
and
technical
work
products
underlying
these
decisions
receive
appropriate
levels
of
peer
review
by
independent
scientific
and
technical
experts.
To
serve
this
goal,
the
Handbook
provides
information
and
outlines
procedures
in
several
different
areas:


basic
principles
and
definitions,
including
distinctions
between
peer
review
and
peer
input,
public
comment,
and
stakeholder
involvement;


preparing
for
peer
review,
including
identifying
work
products,
identifying
appropriate
peer
review
mechanisms,
and
identifying
qualified
experts;
and,


conducting
and
completing
peer
reviews,
including
materials
required
for
peer
review,
creating
a
peer
review
record,
and
utilizing
peer
review
comments.
Peer
Review
Handbook
Page
x
This
Handbook
has
three
parts.
The
first
contains
flow
charts
that
outline
the
key
steps
in
conducting
a
peer
review,
along
with
a
managers'
checklist
for
planning
peer
reviews.
The
second
part
contains
peer
review
guidance
detailing
the
procedures
outlined
in
the
flow
charts
in
a
question
and
answer
format.
The
third
part
contains
Appendices
including
the
1994
Peer
Review
Policy
and
examples
to
help
perform
quality
peer
reviews.
Some
procedures
outlined
in
the
1994
Policy
have
been
completed
while
others
are
continuing
to
change
in
line
with
Agency
experience.
These
changes
have
been
incorporated
into
the
current
Handbook.

Dorothy
E.
Patton,
Ph.
D.
Executive
Director,
Science
Policy
Council
Science
Policy
Council
Fred
Hansen,
OA,
Chair
Lynn
Goldman,
OPPTS,
Vice­
Chair
Joseph
Alexander,
ORD
Donald
Barnes,
OA
Tudor
Davies,
OW
Jeanne
Fox,
Region
2
Sylvia
Lowrance,
OECA
Ramona
Trovato,
OA
Michael
Shapiro,
OSWER
Lawrence
Weinstock,
OAR
Robert
Wolcott,
OPPE
Science
Policy
Council
Steering
Committee
Dorothy
Patton,
ORD,
Chair
Donald
Barnes,
OA
Patricia
Cirone,
Region
10
William
Farland,
ORD
Michael
Feldman,
OCFO
Penelope
Fenner­
Crisp,
OPPTS
Michael
Firestone,
OPPTS
Jerri­
Anne
Garl,
Region
5
Roland
Hemmett,
Region
2
Carl
Mazza,
OAR
James
Nelson,
OGC
Jennifer
Orme­
Zavaleta,
ORD
Peter
Preuss,
ORD
Larry
Reed,
OSWER
Joseph
Reinert,
OPPE
Rosemarie
Russo,
Region
4
Vanessa
Vu,
OPPTS
Mary
Ellen
Weber,
OPPTS
Jeanette
Wiltse,
OW
William
Wood,
ORD
Science
Policy
Council
Staff
Edward
Bender
Kerry
Dearfield
James
Rowe
Mary
McCarthy­
O'Reilly
Dorothy
Patton,
Executive
Director,
Science
Policy
Council
Peer
Review
Handbook
Page
1
U.
S.
Environmental
Protection
Agency
SUMMARY
OF
THE
PEER
REVIEW
PROCESS
This
section
of
the
Peer
Review
Handbook
contains
flowcharts
and
descriptions
of
the
major
steps
in
conducting
a
peer
review.
Cross
references
to
the
appropriate
section
on
Peer
Review
Guidance
are
shown
in
parenthesis
and
bolded.

The
Managers
Planning
Checklist
for
Peer
Review
(
on
page
5)
is
designed
to
give
Managers/
Decision­
Makers
a
simple
tool
to
help
plan
for
a
successful
peer
review.
It
asks
questions
that
should
be
considered
by
a
manager
or
Decision­
Maker
during
the
peer
review
process
to
insure
that
necessary
actions
are
taking
place.
This
checklist
is
also
intended
to
be
used
by
staff
(
especially
Peer
Review
Leaders
and
Coordinators)
to
inform
managers
and
Decision­
Makers
on
some
of
the
key
steps
and
considerations
that
are
necessary
in
carrying
out
a
successful
peer
review.
Prepare
Work
Product
Work
Product
is
Not
Subject
to
Peer
Review
Policy
Work
Product
is
Not
Placed
on
any
List
Is
it
a
Scientific
and
Technical
Work
Product?

Is
the
Work
Product
Major?
Would
the
Work
Product
Still
Benefit
from
Peer
Review?

Work
Product
is
Not
a
Candidate
for
Peer
Review
Place
Work
Product
on
List
C
Work
Product
is
a
Candidate
for
Peer
Review
­
Identify
Basis
for
Charge
­
Identify
Key
Staff
­
Create
Peer
Review
Record
­
Ensure
Availability
of
Funds
­
Determine
Overall
Time
Frame
Place
Work
Product
on
List
B
NO
YES
YES
NO
NO
YES
Continue
with
Conducting
a
Peer
Review
Does
Work
Product
Consist
only
of
Science
Previously
Peer
Reviewed?
YES
NO
Peer
Review
Handbook
Page
2
Figure
1
­
Flowchart
for
Planning
a
Peer
Review
1.
Determination
of
"
major
scientific
and
technical
work
product":
­­
It
is
a
scientific,
engineering,
economic,
or
statistical
document
(
§
2.2.1)
­­
Determine
if
the
work
product
is
major
(
§
2.2.3)
­­
Major
products
meet
certain
criteria
(
§
2.2.3)

2.
Work
product
is
a
candidate
for
peer
review
:
­­
Major
work
products
are
subject
to
peer
review
(
§
2.2.2)
­­
Some
non­
major
work
products
have
to
be
evaluated
to
determine
if
peer
review
would
still
benefit
the
product
(
§
2.2.11)

3.
Work
product
is
not
a
candidate
for
peer
review
:
­­
Most
non­
major
work
products
are
typically
not
candidates
for
peer
review
(
§
2.3.2)
­­
Major
work
product
consists
only
of
science
previously
peer
reviewed
and
adequate
under
the
Agency's
Policy
(
§
2.3.1)
­­
Place
work
product
on
List
C
(
§
1.3.2c))

4.
If
a
work
product
is
subject
to
peer
review:
­­
Identify
basis
for
charge
(
§
3.2.1)
­­
Identify
key
staff
(
§
1.4)
­­
Create
a
peer
review
record
(
§
2.5)
­­
Ensure
source
of
funding
for
the
peer
review
(
§
2.6)
­­
Estimate
completion
date
for
peer
review
(
§
3.3.1)
­­
Place
the
work
product
on
List
B
(
§
1.3.2b))
Work
Product
(
Ready
for
Peer
Review)

Develop
the
Charge
Select
the
Peer
Review
Mechanism
Select
Peer
Reviewers
Set
the
Timelines,
including
Deadlines
Send
Materials
to
Peer
Reviewers
&
Conduct
Review
Complete
Peer
Review
and
obtain
Comments
of
Peer
Reviewers
Add
to
Peer
Review
Record
Continue
with
Completing
a
Peer
Review
Peer
Review
Handbook
Page
3
Figure
2
­
Flowchart
for
Conducting
a
Peer
Review
1.
Develop
the
charge
(
§
3.2.2):
­­
Determine
which
key
issues
to
address
­­
Include
in
peer
review
record
(
§
2.5.3)

2.
Select
a
peer
review
mechanism
(
§
2.4)
­­
Internal
(
§
2.4.2)
­­
External
(
§
2.4.3)
­­
Mail
review
(
§
2.4.3)
­­
Face
to
face
meeting
(
§
2.4.3)
­­
One
time
or
multiple
meetings
(
§
2.4.6)
­­
Include
logistical
information
in
peer
review
record
(
§
2.5.3)

3.
Determine
the
specific
time
line
(
§
3.3):
­­
When
will
the
review
be
started
­­
What
are
the
intermediate
check
points
­­
What
is
the
deadline
for
completion
4.
Select
peer
reviewers
(
§
3.4):
­­
Determine
sources
of
peer
reviewers
(
§
3.4.2)
­­
Determine
expertise
required
(
§
3.4.4)
­­
Consider
balance/
address
(
§
3.4.4)
­­
Consider
conflicts
of
interest
(
§
3.4.5
&
3.4.6)
­­
Include
documentation
in
peer
review
record
(
§
2.5.3)

5.
Materials
for
the
peer
review
(
§
3.5):
­­
Obtain
materials
from
Program
for
review
­­
Prepare
instructions
for
peer
reviews
(
§
3.5.1)
­­
Forward
materials
to
peer
reviewers
(
§
3.5.2)
­­
Include
copy
of
materials
in
peer
review
record
(
§
2.5.3)

6.
Conduct
the
peer
review
­­
Obtain
written
comments
from
reviewers
­­
Include
in
peer
review
record
(
§
2.5.3)

Note:
Some
of
these
steps
may
occur
concurrently.
Receive
Comments
from
Peer
Reviewers
Are
Peer
Reviewers
Comments
used
in
Work
Product?

Document
Why
Not
Used
and
Place
in
Peer
Review
Record
Document
How
Used
and
Place
in
Peer
Review
Record
Incorporate
Comments
into
Work
Product
Finalize
the
Work
Product
Update
the
Peer
Review
Record
Move
the
Work
Product
from
List
B
to
List
A
Done
NO
YES
Peer
Review
Handbook
Page
4
Figure
3
­
Flowchart
for
Completing
a
Peer
Review
1.
Evaluate
comments
from
peer
reviewers
(
§
4.2.1)
­­
Responding
and
reacting
to
comments
­­
Obtaining
clarification,
if
needed
2.
Comments
that
are
considered,
but
not
used
(
§
4.3.1)
­­
Determine
why
not
used
and
document
­­
Include
comments
in
peer
review
record
(
§
2.5.3)

3.
Comments
that
are
useful
(
§
4.3.1)
­­
Revise
the
work
product
by
incorporating
comments
­­
Send
revised
work
product
back
to
peer
reviewers,
if
necessary
­­
Include
comments
in
peer
review
record
(
§
2.5.3)

4.
Finalize
work
product
(
§
4.3.1)
­­
Include
in
peer
review
record
(
§
2.5.3)
­­
Move
work
product
from
List
B
to
List
A
Peer
Review
Handbook
Page
5
Managers
Planning
Checklist
for
Peer
Review
1)
Title
of
Work
Product:
___________________________________________________

2)
What
Decision/
Rule/
Regulation/
Action
Does
this
Work
Product
Support:
________
_______________________________________________________________________

3)
Determination
of
Major
Scientific
and
Technical
Work
Products

Is
the
work
product
scientific
or
technical
__
yes
__
no?


Is
the
work
product
__
major
or
__
non­
major?

4)
Determining
What
Peer
Review
is
Needed

If
major,
peer
review
is
needed

If
not
major,
is
peer
review
still
needed

When
does
the
review
need
to
be
done?


How
much
time
will
be
needed
to
conduct/
complete
the
review?


Are
there
court
ordered
deadlines
or
other
constraints?


Has
senior
management
(
AA/
RA/
others)
been
informed
of
progress/
problems?


What
would
constitute
success
for
this
review?

5)
Determining
the
Resources
for
Peer
Review

What
is
the
priority
of
this
project
relative
to
other
projects
in
the
same
office?


What
resources
are
needed
to
conduct
the
review?


What
are
the
impacts
of
the
review
on
personnel?


Who
will
lead
the
peer
review?


Who
will
conduct
the
peer
review?


Who
will
maintain
the
peer
review
record?


Where
will
the
peer
review
record
be
kept?


What
mechanism
will
be
used
for
the
peer
review?


Has
the
charge
been
developed?


Has
internal
and
external
coordination
been
initiated/
completed?


Have
arrangements
for
interim/
final
sign­
offs
(
e.
g.,
for
the
charge,
the
panel,
on
any
changes
to
the
final
work
product)
been
made?


How
will
results
of
the
review
be
presented
and
addressed
in
the
final
work
product
(
e.
g.,
in
a
preamble,
in
an
accompanying
appendix
­­
as
well
as
changes
in
the
work
product
itself)?


Has
the
work
product
been
entered
onto
List
B
or
C,
as
appropriate?

6)
Comments:
_____________________________________________________________
Peer
Review
Handbook
Page
6
THIS
PAGE
LEFT
INTENTIONALLY
BLANK
Peer
Review
Handbook
Page
7
U.
S.
Environmental
Protection
Agency
PEER
REVIEW
GUIDANCE
Peer
Review
Handbook
Page
8
THIS
PAGE
LEFT
INTENTIONALLY
BLANK
Peer
Review
Handbook
Page
9
Peer
review
is
not
free;
however,
not
doing
peer
review
can
be
costly.
1.
THE
NEED
FOR
PEER
REVIEW
1.1
Overview
Statement
Peer
review
at
the
U.
S.
Environmental
Protection
Agency
(
EPA)
takes
many
different
forms
depending
on
the
nature
of
the
work
product,
relevant
statutory
requirements,
and
officespecific
policies
and
practices.
In
January
1993,
responding
to
recommendations
in
the
report
Safeguarding
the
Future:
Credible
Science,
Credible
Decisions,
former
Administrator
William
Reilly
issued
an
Agency­
wide
policy
for
peer
review.
Administrator
Carol
Browner
reaffirmed
the
central
role
of
peer
review
in
the
Agency
on
June
7,
1994
and
instituted
an
Agency­
wide
implementation
program
(
see
Appendix
A).
Following
Agency­
wide
implementation,
office
and
region­
specific
standard
operating
procedures
(
SOPs)
were
written
and
have
been
used
since
1994.
Based
on
the
experiences
of
those
managing
peer
review
throughout
the
Agency
since
then,
as
reported
to
the
Science
Policy
Council
(
SPC)
and
its
Peer
Review
Advisory
Group
(
PRAG),
we
found
that
a
single
Agency­
wide
document
on
peer
review
was
needed.
Therefore,
this
Peer
Review
Handbook
is
created
as
a
single,
centralized
form
of
implementation
guidance
for
Agency
staff
and
managers.
While
the
Handbook
supersedes
the
original
SOPs,
Agency
offices
and
regions
can
still
prepare
brief,
tailored
guidance
that
meets
their
individual
needs
to
supplement
the
information
in
this
Handbook.

1.2
Understanding
Peer
Review
1.2.1
Why
use
Peer
Review?

Peer
review
is
intended
to
uncover
any
technical
problems
or
unresolved
issues
in
a
preliminary
(
or
draft)
work
product
through
the
use
of
independent
experts.
This
information
is
then
used
to
revise
that
draft
product
so
that
the
final
work
product
will
reflect
sound
technical
information
and
analyses.
Peer
review
is
a
process
for
enhancing
a
scientific
or
technical
work
product
so
that
the
decision
or
position
taken
by
the
Agency,
based
on
that
product,
has
a
sound,
credible
basis.
To
be
most
effective,
peer
review
of
a
major
scientific
and/
or
technical
work
product
needs
to
be
incorporated
into
the
up­
front
planning
of
any
action
based
on
the
work
product
­­
this
includes
obtaining
the
proper
resource
commitments
(
people
and
money)
and
establishing
realistic
schedules.

Peer
review
of
major
scientific
and
technical
work
products
should
not
be
looked
upon
as
another
"
hurdle"
in
the
Agency
decision
making
processes.
While
peer
review
requires
that
time
Peer
Review
Handbook
Page
10
and
resources
be
planned
into
the
decision
making
process,
the
benefits
justifies
the
added
cost.
Peer
review
enhances
the
credibility
and
acceptance
of
the
decision
based
on
the
work
product.
By
ensuring
a
sound
basis
for
decisions,
greater
cost
savings
are
realized
since
decisions
will
not
be
challenged
as
often
and
extra
effort
will
not
be
required
to
go
back
and
redo
the
work
product.
So
while
peer
review
is
not
free,
the
cost
of
not
doing
peer
review
is
usually
much
more
expensive.
Furthermore,
not
conducting
a
peer
review
can
potentially
place
the
Agency
in
the
position
of
attempting
to
defend
a
scientifically
invalid
position
­­
which
can
be
very
costly
in
terms
of
both
resources,
and
more
importantly,
credibility.

1.2.2
What
is
Peer
Involvement?

As
defined
in
the
Peer
Review
Policy,
peer
involvement
is
the
process
whereby
Agency
staff
involve
subject­
matter
experts
from
outside
their
program
in
one
or
more
aspects
of
the
development
of
work
products.
Peer
involvement,
therefore,
constitutes
active
outreach
to
and
participation
by
the
broad
scientific,
engineering,
and
economics
communities
beyond
the
Agency
(
external)
as
well
as
within
the
Agency
(
internal).
Typically,
peer
involvement
takes
two
general
forms:
peer
input
(
ongoing
discussions
during
the
development
of
the
work
product)
and
peer
review
(
an
evaluation
of
a
workplan,
preliminary
draft
or
the
like,
or
most
often,
the
critical,
final
objective
expert
evaluation
of
the
work
product).

1.2.3
What
is
Peer
Review?

Peer
review
is
a
documented
critical
review
of
a
specific
Agency
major
scientific
and/
or
technical
work
product.
The
peer
review
is
conducted
by
qualified
individuals
(
or
organizations)
who
are
independent
of
those
who
performed
the
work,
but
who
are
collectively
equivalent
in
technical
expertise
(
i.
e.,
peers)
to
those
who
performed
the
original
work.
The
peer
review
is
conducted
to
ensure
that
activities
are
technically
adequate,
competently
performed,
properly
documented,
and
satisfy
established
quality
requirements.
The
peer
review
is
an
in­
depth
assessment
of
the
assumptions,
calculations,
extrapolations,
alternate
interpretations,
methodology,
acceptance
criteria,
and
conclusions
pertaining
to
the
specific
major
scientific
and/
or
technical
work
product
and
of
the
documentation
that
supports
them.
Peer
review
may
provide
an
evaluation
of
a
subject
where
quantitative
methods
of
analysis
or
measures
of
success
are
unavailable
or
undefined;
such
as
research
and
development.
Peer
review
is
usually
characterized
by
a
one­
time
interaction
or
a
limited
number
of
interactions
by
independent
peer
reviewers.
Peer
review
can
occur
during
the
early
stages
of
the
project
or
methods
selection,
or
as
typically
used,
as
part
of
the
culmination
of
the
work
product,
ensuring
that
the
final
product
is
technically
sound.
Peer
Review
Handbook
Page
11
Peer
Input
is
not
a
substitute
for
Peer
Review
1.2.4
What
is
Peer
Input?

Many
Agency
work
products
are
developed
with
the
input
of
various
scientific
and
technical
experts
inside
and
outside
the
Agency.
Like
the
contribution
made
by
peer
reviewers,
peer
input
is
valuable
and
enhances
the
scientific
or
technical
basis
of
the
products.
Peer
input,
sometimes
referred
to
as
peer
consultation,
generally
connotes
an
interaction
during
the
development
of
an
evolving
Agency
work
product,
providing
an
open
exchange
of
data,
insights,
and
ideas.
Peer
input
may
be
characterized
by
a
continued
and
iterative
interaction
with
scientific
experts
during
work
product
development.
A
common
example
of
peer
input
is
the
input
received
from
workgroup
members
during
development
of
a
product.
Many
Agency
products
are
developed
through
the
efforts
of
a
workgroup,
which
may
include
external
experts,
such
as
State
and
Tribal
representatives.
These
workgroup
members
have
an
active,
ongoing
participation
in
developing
the
work
product.
Another
example
of
obtaining
peer
input
is
of
an
Agency
office
sending
a
draft
work
product
to
a
list
of
stakeholder
representatives
for
general
comments
(
stakeholder
representatives
often
include
experts
who
could
be
considered
"
peers").

1.2.5
How
is
Peer
Review
Different
from
Peer
Input?

The
key
distinctions
between
peer
input
as
described
above
and
formal
peer
review
are
the
independence
of
the
peer
reviewers
and
their
level
of
involvement.
The
goal
of
peer
review
is
to
obtain
an
independent,
third­
party
review
of
the
product
from
experts
who
haven't
substantially
contributed
to
its
development.
When
experts
have
a
material
stake
in
the
outcome
of
the
peer
review
(
such
as
a
regulated
party)
or
have
participated
substantially
in
the
development
of
the
product
(
such
as
a
workgroup
member),
those
experts'
reviews
may
not
qualify
as
unbiased,
independent
peer
review
and
may
be
better
characterized
as
peer
input.

It
is
clear
that
peer
input
provides
valuable
contributions
to
the
development
of
the
work
product.
However,
peer
input
does
not
substitute
for
peer
review.
Once
a
work
product
is
considered
major,
it
is
a
candidate
for
peer
review
and
entered
on
List
B
(
Candidate
Products
for
Future
Peer
Review
­­
see
section
1.3.2b))
­­
even
though
the
work
product
may
already
have
a
substantial
amount
of
peer
input.
In
other
words,
one
cannot
argue
that
a
peer
review
is
not
necessary
if
a
major
work
product
has
received
"
enough"
peer
input.
If
the
work
product
is
not
considered
major
and
has
had
peer
input,
it
is
entered
on
List
C
(
Products
for
Which
a
Decision
has
been
made
not
to
Peer
Review
­­
see
section
1.3.2c))
with
comments
about
the
extent
of
the
peer
input.
Peer
Review
Handbook
Page
12
Public
comment
does
not
substitute
for
peer
review.

Stakeholder
involvement
is
not
a
peer
review
mechanism.
1.2.6
Can
Someone
Who
Provided
Peer
Input
Become
an
Independent
Peer
Reviewer
for
the
Same
Work
Product
Later
in
the
Process?

Generally,
the
answer
is
no
as
that
expert
is
no
longer
independent,
but
rather
a
contributor
to
the
work
product.
There
may
be
special
circumstances
where
the
expertise
is
so
narrow
that
another
peer
reviewer
isn't
available.
The
Peer
Review
Leader
(
see
section
1.4.4)
will
normally
be
responsible
for
making
this
determination.

1.2.7
How
is
Peer
Review
Different
from
Public
Comment?

Peer
review
and
public
comment
are
mutually
exclusive.
Public
comment
solicited
from
the
general
public
through
the
Federal
Register
or
by
other
means
is
often
required
by
the
Administrative
Procedures
Act,
relevant
statutes
or
both.
Public
comment
can
also
be
solicited
for
policy
purposes.
The
Agency
takes
public
comment
on
some
strictly
scientific
products
and
almost
all
regulatory
decisions.
Public
commenters
usually
include
a
broad
array
of
people
with
an
interest
in
the
technical
analysis
or
the
regulatory
decision;
some
are
scientific
experts
(
which
may
provide
some
peer
input),
some
are
experts
in
other
areas,
and
some
are
interested
non­
experts.
The
critical
distinction
is
that
public
comment
doesn't
necessarily
draw
the
kind
of
independent,
expert
information
and
in­
depth
analyses
expected
from
the
peer
review
process.
Public
comment
is
open
to
all
issues,
whereas
the
peer
review
process
is
limited
to
consideration
of
technical
issues.
While
it
may
be
an
important
component
of
the
review
process,
public
comment
does
not
substitute
for
peer
review.

1.2.8
How
is
Peer
Review
Different
from
Stakeholder
Involvement?

Stakeholder
involvement
occurs
when
the
Agency
works
with
external
interest
groups
that
have
some
stake
in
or
concerns
over
the
outcome
of
the
technical
work
product
or
regulatory
position.
This
is
an
interactive
process,
working
with
other
agencies,
industry
groups,
regulated­
community
experts,
environmental
groups,
other
interest
groups
that
represent
a
broad
spectrum
of
the
regulated
community,
etc.,
and
usually
strives
for
a
consensus
approach.
The
goal
of
peer
review,
on
the
other
hand,
is
to
obtain
an
independent,
third­
party
review.
Stakeholder
involvement
is
not
a
peer
review
mechanism
­­
as
with
peer
input,
once
a
decision
is
made
that
peer
review
is
needed,
stakeholder
involvement
does
not
substitute
for
peer
review
even
though
it
adds
value
to
the
work
product.
Peer
Review
Handbook
Page
13
1.2.9
What
Role
does
Peer
Review
have
in
the
Regulatory
Development
Process?

The
peer
review
of
scientific
and
technical
work
products
that
support
rulemaking
actions
is
an
important,
fundamental
step
in
the
policy
setting
process
and
which
affirms
the
credibility
of
the
Agency.
Because
new
rules,
and
the
work
products
supporting
them,
must
often
withstand
intense
scrutiny
by
the
general
public
and
the
stakeholders
involved
in
the
action,
the
peer
review
process
selected
for
such
work
products
needs
to
be
well
planned
and
documented.
The
rule
or
regulation
itself
is
not
subject
to
the
Peer
Review
Policy.
However,
if
the
rule
or
regulation
is
supported
by
a
major
scientific
and/
or
technical
work
product,
that
work
product
should
be
peer
reviewed
prior
to
its
use
in
the
rule
(
see
section
2.2
for
determination
of
major
work
products).
The
decision
to
peer
review
or
not
peer
review
any
scientific
and/
or
technical
work
product
will
be
documented
through
the
Agency's
annual
peer
review
reporting
process
(
see
section
1.3).
Remember,
public
comment
and
stakeholder
involvement
do
not
constitute
peer
review.

Tier
1
and
Tier
2
rulemakings,
are
by
definition
important,
major
Agency
rulemakings
within
the
Agency.
Therefore,
work
products
supporting
Tier
1
and
Tier
2
rules
in
particular
(
including
rules
that
are
determined
to
be
"
significant"
by
OMB
under
Executive
Order
12866
because
they
have
an
economic
impact
of
$
100
million
or
more)
should
be
closely
scrutinized
to
determine
whether
they
meet
the
criteria
for
major
(
see
section
2.2.3).
Work
products
supporting
Tier
3
rulemakings
may
also
be
considered
major
and
thus
candidates
for
peer
review.
External
peer
review
is
the
procedure
of
first
choice
for
a
work
product
that
is
intended
to
support
a
Tier
1
or
Tier
2
rulemaking.
Although
acceptable
in
certain
circumstances,
any
decision
to
use
an
internal
peer
review
mechanism
for
such
work
products
would
be
the
exception
rather
than
the
rule.
For
work
products
supporting
a
Tier
3
rule,
internal
or
external
peer
review
may
be
appropriate
depending
on
the
nature
of
the
product
and
other
factors
(
see
section
2.4.1).
For
Tier
1
and
Tier
2
rulemakings,
the
Final
Agency
Review/
closure
memo
needs
to
indicate
that
the
Peer
Review
Policy
was
followed.
For
Tier
3
rulemakings,
the
action
memo
needs
to
indicate
that
the
Peer
Review
Policy
was
followed.

Analytic
blueprints
are
required
for
Tier
1
and
Tier
2
rulemakings,
and
are
encouraged
for
Tier
3
rulemakings;
some
individual
EPA
offices
require
it
for
Tier
3.
For
peer
review
purposes,
the
analytic
blueprint
is
the
process
whereby
the
project
manager
identifies
the
supporting
scientific
and
technical
work
products
and
identifies
needed
peer
review.
It
also
shows
the
schedule
of
the
peer
review
in
the
context
of
the
schedule
for
the
overall
rulemaking.
In
general,
peer
review
should
be
completed
as
early
in
the
process
as
practicable.
Where
possible,
peer
review
of
work
products
should
be
completed
prior
to
issuance
of
the
draft
regulation.
In
some
cases,
support
work
products
for
final
regulations
may
require
an
additional
peer
review
if
those
scientific
and
technical
work
products
change
significantly
after
the
public
comment
period.
Peer
Review
Handbook
Page
14
1.2.10
What
Role
does
Peer
Review
have
in
Regulatory
Negotiations?

Regulatory
negotiations
are
not
candidates
for
peer
review;
however,
to
ensure
final
decisions
are
based
on
sound
and
credible
science,
the
major
scientific
and
technical
work
products
that
support
the
negotiation
need
peer
review
before
the
negotiation
takes
place.

1.3
Annual
Agency
Reporting
Requirements
1.3.1
What
are
the
Annual
Reporting
Requirements?

The
Peer
Review
Coordinator
(
for
each
AA/
RA;
see
section
1.4.5)
will
organize
an
annual
review
of
all
peer
review
activities
and
submit
this
information
to
the
Office
of
Research
and
Development
(
ORD).
ORD
will
staff
this
function
at
the
direction
of
the
Deputy
Administrator.
In
the
Deputy
Administrator's
annual
call
for
submissions,
guidance
on
format
and
submission
of
this
information
will
be
provided.
ORD
will
review
the
submissions
for
completeness,
i.
e.
all
information
is
provided
and
products
are
accounted
for
each
year.
ORD
will
then
provide
a
review
of
the
completeness
of
the
information
in
the
submissions
through
consultation
with
the
appropriate
persons
in
each
organization
(
see
section
1.4).
ORD
will
then
consolidate
the
information
and
findings
for
the
SPC
and
the
Deputy
Administrator.
Any
conflicts
arising
from
the
review
will
be
resolved
by
the
Deputy
Administrator.
The
due
date
for
the
annual
reporting
will
be
announced
each
year
in
the
annual
call
letter;
however,
for
planning
purposes,
it
is
normally
due
in
the
early
summer.

1.3.2
What
Listings
are
Required
for
the
Annual
Reporting?

Three
listings
of
products
are
required
for
the
annual
reporting:

a)
List
A
­
Products
Peer
Reviewed
Since
1991
1)
List
A
is
a
cumulative
list
of
peer
reviewed
products
from
1991
to
the
present.

2)
For
each
new
work
product
entered
onto
List
A,
a
short
report
summarizing
the
peer
review
must
also
be
provided
and
attached
to
List
A.
This
summary
report
is
signed
by
the
appropriate
Decision­
Maker
(
see
section
1.4.3).
Signatures
by
Decision­
Makers
should
be
affixed
as
originals
in
the
peer
review
record
and
copies
submitted
with
List
A.
However,
for
reports
submitted
by
electronic
means,
the
submitted
report
should
identify
the
name
of
the
signer
and
the
date
signed.
Peer
Review
Handbook
Page
15
3)
The
format
and
content
of
the
summary
report
will
be
provided
in
the
annual
call
letter
(
see
section
1.3.1).

b)
List
B
­
Candidate
Products
for
Future
Peer
Review
1)
List
B
is
a
listing
of
products
that
are
expected
to
be
peer
reviewed
in
the
near
future.

2)
This
List
contains
major
scientific
and
technical
work
products
and
any
non­
major
scientific
and
technical
work
products
for
which
peer
review
has
been
deemed
necessary
or
appropriate.

3)
Work
products
placed
on
List
B
remain
on
List
B
until
they
are
either
peer
reviewed
(
after
which
they
are
moved
to
List
A)
or
a
decision
is
made
not
to
peer
review
that
work
product
(
at
which
point
it
is
moved
to
List
C).

4)
The
peer
review
summary
report
that
eventually
accompanies
each
List
A
peer
review
product
should
be
initiated
at
the
time
a
work
product
is
decided
to
be
peer
reviewed
and
filled
in
as
the
peer
review
proceeds.
This
is
easier
than
waiting
until
the
peer
review
of
the
work
product
is
completed
while
the
details
are
still
fresh.

c)
List
C
­
Products
for
Which
a
Decision
has
Been
Made
Not
to
Peer
Review
1)
List
C
is
a
cumulative
listing
of
all
scientific
and
technical
work
products
that
do
not
receive
peer
review
(
see
section
2.3).

2)
List
C
includes:
(
a)
any
major
scientific
and
technical
work
products
for
which
a
decision
was
made
not
to
peer
review;
(
b)
those
work
products
which
were
originally
placed
on
List
B,
but
for
which
it
was
decided
that
peer
review
was
not
necessary
(
e.
g.,
the
product
was
not
used
in
decision
making;
the
project
was
canceled);
and
(
c)
all
non­
major
scientific
and
technical
work
products.

3)
List
C
also
includes
several
categories
of
work
products
that
would
not
normally
receive
peer
review.
It
is
not
necessary
to
list
these
work
products
individually
on
List
C,
however,
the
total
number
prepared
by
the
organization
must
be
included
on
List
C.
The
organization
may
need
to
identify
the
individual
products
if
requested
(
e.
g.,
due
to
litigation,
FOIA,
Peer
Review
Handbook
Page
16
etc.).
Such
categories
can
include,
but
may
not
be
limited
to:
chemical
action
reports,
RCRA
permits,
scientific
analyses
for
Premanufacturing
Notices
(
PMNs)
that
are
conducted
on
a
routine
basis
and
that
do
not
deviate
from
established
practice,
and
NPDES
permits.

4)
List
C
also
includes
any
scientific
papers
(
articles)
that
were
published
in
a
credible
peer
reviewed
journal
(
i.
e.
for
purposes
of
listing
journal
articles,
only
published
articles,
rather
than
projected
articles,
is
intended).
It
is
not
necessary
to
list
these
papers
individually
on
List
C,
however,
the
total
number
prepared
by
the
organization
and
peer
reviewed
by
journals
must
be
included
on
List
C.
The
organization
may
need
to
identify
the
individual
papers
and
where
published,
if
requested
(
e.
g.,
due
to
litigation,
FOIA,
etc.).
(
See
sections
2.4.4
and
2.4.5).

5)
Each
work
product
on
List
C
needs
a
brief
description
in
the
"
comments"
column
of
the
reason(
s)
it
is
not
being
peer
reviewed.
In
addition,
each
Decision­
Maker
must
provide
signed
documentation
that
identifies
the
work
products
that
they
decided
not
to
peer
review
(
other
than
work
products
that
are
included
in
categories
in
3)
and
4),
above).
ORD
will
provide
a
standardized
format
for
this
documentation.
Signatures
by
Decision­
Makers
should
be
affixed
as
originals
in
the
peer
review
record
and
copies
submitted
with
List
C.
However,
for
reports
submitted
by
electronic
means,
the
submitted
document
should
identify
the
name
of
the
signer
and
the
date
signed.

1.3.3
Is
There
a
Coding
System
for
Work
Products
on
the
Annual
Reporting
Lists?

The
work
products
that
are
placed
on
these
lists
should
be
assigned
a
discrete
code
number.
A
coding
system
will
uniquely
identify
each
work
product
when
it
is
first
listed
on
any
of
these
lists.
This
will
assist
in
tracking
each
work
product
from
year
to
year
and
when
it
moves
from
one
list
to
another.
The
code
will
be:
YEAR
(
2
digits)
­
OFFICE/
REGION
(
usual
initials)
­
RUNNING
NUMBER
(
4
digits
and
in
sequential
numbering
for
that
year
the
work
product
was
first
placed
on
any
list);
for
example,
97­
OAR­
0001,
98­
OW­
0010,
99­
R05­
0012,
00­
OPPTS­
0054,
or
01­
R10­
0002.

The
code
number
needs
to
be
assigned
at
the
time
the
annual
report
is
developed
(
although
the
code
number
can
be
assigned
anytime
during
the
year
that
the
peer
review
is
Peer
Review
Handbook
Page
17
planned).
The
code
number
is
assigned
by
the
Peer
Review
Coordinator
to
provide
a
single,
consistent
numbering
sequence.

1.3.4
When
will
the
Handbook
Itself
be
Revised?

During
each
annual
reporting
cycle,
suggestions
for
revisions
to
the
Handbook
should
be
submitted.
A
decision
will
be
made
by
the
Science
Policy
Council
(
SPC)
on
whether
to
revise
the
Handbook,
in
part
or
total,
or
not
based
on
the
suggestions.
The
SPC
will
then
direct
the
Peer
Review
Advisory
Group
(
PRAG)
to
draft
the
revision(
s)
for
Agency
comment
and
SPC
approval.

1.4
The
Roles
of
People
and
Organizations
in
Peer
Review
1.4.1
Who
is
Ultimately
Accountable
for
Peer
Review?

Under
the
June
7,
1994
Peer
Review
Policy,
the
Administrator
has
designated
the
Assistant
Administrators
and
Regional
Administrators
(
AAs
and
RAs)
to
be
accountable
for
implementing
the
Policy
in
their
respective
organizations.
The
Deputy
Administrator
is
ultimately
responsible
for
peer
review
across
the
Agency
and
is
the
final
arbitrator
of
conflicts
and
concerns
about
peer
review.

1.4.2
Who
are
the
Agency
Staff
involved
in
Peer
Review?

The
principal
Agency
staff
involved
are
Decision­
Makers
(
and
their
line
managers),
Peer
Review
Leaders
and
Peer
Review
Coordinators.
In
addition,
staff
in
ORD
have
been
designated
by
the
Deputy
Administrator
for
ensuring
the
Agency's
Peer
Review
Policy
requirements
are
met.

1.4.3
Who
are
the
Decision­
Makers
&
What
are
Their
Responsibilities?

The
AA/
RA
is
the
ultimate
Decision­
Maker
for
their
organization
and
is
accountable
for
the
decisions
regarding
the
identification
of
major
scientific
and
technical
work
products
and
the
mechanism(
s)
of
peer
review
utilized
for
each
of
the
products.
The
AA/
RA
may
designate
the
Office
Directors
and
Division
Directors
(
or
other
appropriate
level
line­
managers)
as
the
frontline
Decision­
Makers.

Generally,
the
Decision­
Makers
[
usually
line­
managers]
decide
whether
a
work
product
is
major
and
needs
peer
review
or
not,
and
what
peer
review
mechanism
to
use.
Furthermore,
the
Decision­
Makers
commit
the
resources
needed
to
ensure
a
proper
peer
review.
Decision­
Makers
are
responsible
for
ensuring
that
the
peer
reviews
are
properly
performed
and
documented.
Peer
Review
Handbook
Page
18
Specific
responsibilities
of
the
Decision­
Maker(
s)
are
the
following:

a)
Determine
which
work
products
in
their
organization
require
peer
review
b)
Designate
(
in
conjunction
with
the
Project
Manager)
a
Peer
Review
Leader
to
organize
the
peer
review
c)
Provide
advice,
guidance,
and
support
to
the
Peer
Review
Leader
in
the
preparation,
conduct,
and
completion
of
the
peer
review
d)
Ensure
that
sufficient
funds
are
designated
in
the
office's
budget
request
to
conduct
the
peer
review;
also
ensure
that
adequate
resources
and/
or
extramural
management
support
are
available
for
the
peer
review
e)
Establish
a
realistic
peer
review
schedule
f)
Designate
the
stage(
s)
of
product
development
where
peer
review
is
appropriate
g)
Ensure
that
the
results
of
peer
review
are
carried
forward
in
the
work
product
h)
By
signature,
document
the
decisions
made
that
are
reported
in
the
annual
reporting
to
the
SPC
and
Deputy
Administrator;
this
includes
signing
the
peer
review
summary
report
for
each
completed
peer
review
i)
Certify
any
decision
NOT
to
peer
review
a
product
by
signature
on
a
List
C
submission
1.4.4
Who
are
the
Peer
Review
Leaders
&
What
are
Their
Responsibilities?

The
Peer
Review
Leader
organizes
and
oversees
the
peer
review
for
a
specific
individual
work
product.
This
person(
s)
can
be
the
Decision­
Maker(
s),
but
will
usually
be
someone
who
is
authorized
by
the
Decision­
Maker
to
organize,
conduct,
and
complete
the
peer
review.
This
individual
may
also
be
the
Project
Manager
for
the
work
product.
The
Peer
Review
Leader
will
obtain
the
assistance
and
support
of
the
Peer
Review
Coordinator
(
see
below)
as
well
as
any
others
within
the
Agency
to
help
perform
the
peer
review.
The
Peer
Review
Leader
will
be
chosen
on
a
case
by
case
basis
depending
on
the
work
product
needing
peer
review.
Peer
Review
Handbook
Page
19
Specific
responsibilities
of
the
Peer
Review
Leader
are
these:

a)
Keep
the
Decision­
Maker
informed
of
the
status
of
a
given
project;
provide
Peer
Review
Coordinator
with
data
for
the
annual
report
b)
Organize,
conduct,
and
complete
the
peer
review
following
Agency
procedures
c)
Establish
and
maintain
the
peer
review
record
for
the
specific
individual
peer
review
currently
being
performed
(
see
section
2.5);
this
includes
initiating
the
peer
review
summary
report
for
the
Decision­
Maker
to
sign
when
the
peer
review
is
completed
d)
Select
the
peer
reviewers
in
consultation
with
others
involved
with
the
peer
review
(
e.
g.,
Decision­
Maker
or
responsible
line
manager)

e)
Advise
peer
reviewers
of
their
responsibilities
f)
Brief
management
and
obtain
management
approval
on
the
approach
to
responding
to
peer
reviewer
comments
g)
Provide
information
to
Decision­
Maker
on
the
charge,
profile
of
peer
reviewers,
the
peer
review
comments,
and
how
those
comments
are
to
be
used
h)
Notify
the
Peer
Review
Coordinator
that
the
peer
review
is
completed
for
the
annual
report
i)
Archive
the
peer
review
record
in
a
manner
consistent
with
their
organization's
archiving
procedures
1.4.5
Who
are
the
Peer
Review
Coordinators
&
What
are
Their
Responsibilities?

The
Peer
Review
Coordinator
is
designated
by
the
AA/
RA
to
coordinate
and
monitor
peer
review
activities
in
their
respective
organization
or
organizational
unit.
This
person
must
be
of
sufficient
stature
and
judgment
to
have
the
access
to
and
confidence
of
all
levels
of
office
or
regional
management
when
needed.
The
Peer
Review
Coordinator
is
the
main
contact
for
their
organization;
they
can
also
direct
interested
parties
to
other
persons/
contacts
in
the
office
on
specific
work
products
(
e.
g.,
Peer
Review
Leader).
Peer
Review
Handbook
Page
20
Specific
responsibilities
of
the
Peer
Review
Coordinator
are
these:

a)
General
oversight
responsibility
for
the
Office's
or
Region's
peer
review
process
b)
Report
peer
review
activities
to
the
AA/
RA
c)
Help
mediate
difficult
issues
between
their
organization
and
others;
if
they
can't
resolve
issue,
then
bring
the
issue
to
the
attention
of
the
appropriate
level
Decision­
Makers
in
each
organization
for
resolution.

d)
Function
as
the
liaison
with
ORD
and
the
Science
Policy
Council
(
SPC):

1)
Represent
office/
region
before
the
SPC
2)
Advise
ORD
of
any
changes
in
the
list
of
work
products
and
peer
review
mechanisms
during
the
annual
reporting,
and
when
necessary,
at
other
times
3)
Participate
in
Agency
peer
review
training,
workshops,
etc.,
as
requested
and
disseminate
this
information
to
the
organization;
coordinate
and/
or
present
training
within
their
organization
e)
Submit
information
on
organization's
peer
review
candidates
for
each
year
as
requested
(
this
is
the
annual
reporting,
see
section
1.3)

1)
Generate
and
update
Lists
A,
B,
and
C
2)
Assure
the
proper
approval
signature
on
the
completed
submission
with
the
accompanying
explanation
for
any
departures
from
the
Policy
f)
Establish
procedures
to
assure
that
the
required
work
product
peer
review
documentation
(
i.
e.,
peer
review
record)
is
filed
and
maintained
in
an
appropriate
manner
(
see
section
2.5)

g)
Provide
advice,
guidance,
and
support
to
the
various
Peer
Review
Leaders
for
the
performance
of
the
peer
reviews
h)
Distribute
Agency­
wide
peer
review
guidance
and
materials
to
appropriate
office/
region
personnel,
as
requested
Peer
Review
Handbook
Page
21
The
quality
of
the
peer
review
is
dependent
on
the
competence
and
independence
of
the
reviewers.
1.4.6
Who
are
the
Peer
Reviewers?

Peer
reviewers
are
individuals
who
have
technical
expertise
in
the
subject
matter
of
the
work
product
undergoing
peer
review.
Peer
reviewers
can
come
from
EPA,
another
Federal
agency,
or
from
outside
of
the
Federal
government.

1.4.7
What
are
the
Responsibilities
of
Peer
Reviewers?

Peer
reviewers
should
maintain
the
confidentiality
of
the
product,
perform
the
review
in
a
timely
manner,
and
be
unbiased
and
objective.
Peer
reviewers
need
to
be
willing
participants
in
the
peer
review
process
­­
they
should
agree
to
read
all
materials,
attend
all
sessions,
and
protect
confidential
information
that
arises.

1.4.8
What
is
an
Independent
Peer
Reviewer?

An
independent
peer
reviewer
is
an
expert
who
wasn't
associated
with
the
generation
of
the
specific
work
product
either
directly
by
substantial
contribution
to
its
development
or
indirectly
by
significant
consultation
during
the
development
of
the
specific
product.
The
independent
peer
reviewer,
thus,
is
expected
to
be
objective.

Independence
is
freedom
from
institutional,
ideological,
or
technical
bias
regarding
the
issues
under
review
and
is
necessary
for
objective,
fair,
and
responsible
evaluation
of
the
work
product.
If
a
selected
reviewer
has
a
particular
scientific
or
technical
perspective,
it
may
be
desirable
to
balance
the
review
with
peer
reviewers
of
other
perspectives.
Ideally,
peer
reviewers
should
be
free
of
real
or
perceived
conflicts­
of­
interest
or
there
should
be
a
balancing
of
interests
among
peer
reviewers.
If
there
are
potential
conflicts
of
interest
(
real
or
perceived),
they
should
be
fully
identified
to
ensure
a
credible
peer
review.
(
See
section
3.4
for
further
information).

1.4.9
When
does
an
Agency
Internal
Peer
Reviewer
Qualify
as
Independent?

An
Agency
independent
peer
reviewer
is
one
who
comes
from
a
different
organizational
unit
than
the
one
where
the
review
question
or
document
originates.
A
different
organizational
unit
usually
denotes,
at
minimum,
a
different
office
(
i.
e.,
above
a
division
level)
within
the
organization.
In
particular,
a
reviewer
shouldn't
come
from
within
the
chain
of
command,
either
upward
or
downward.
Peer
Review
Handbook
Page
22
1.4.10
What
is
a
Peer
Review
Panel?

A
peer
review
panel
can
range
from
a
few
individuals
to
ten
or
more,
depending
on
the
issue
being
investigated,
the
time
available
and
any
limitations
on
resources.
Individuals
who
serve
as
peer
reviewers
must
have
appropriate
scientific
and
technical
expertise
that
covers
the
broad
spectrum
of
expertise
required
to
treat
the
issues/
questions
presented
in
the
charge.

1.4.11
What
is
a
Subject
Matter
Expert?

A
subject
matter
expert
is
one
who
has
specific
scientific
and
technical
expertise
in
the
matter
under
review.
The
importance
of
scientific
and
technical
expertise
in
the
subject
matter
is
obvious,
however,
knowledge
or
just
"
knowing"
about
the
subject
area
isn't
equivalent
to
expertise
in
the
subject
matter.
For
Agency
decisions,
a
multi­
disciplinary
group
of
experts
corresponding
to
the
disciplines
that
contribute
to
complex
Agency
decisions
is
often
necessary
for
a
full
and
complete
peer
review.
For
example,
a
risk
assessment
that
relies
on
both
animal
and
human
data
usually
requires
experts
in
both
areas
for
a
complete
review.
For
economic
analyses,
experts
from
the
corresponding
economic
disciplines
are
necessary.

1.4.12
What
is
the
Role
of
the
Science
Policy
Council
(
SPC)?

According
to
the
1994
Peer
Review
Policy
statement:
"
The
Science
Policy
Council
is
responsible
for
overseeing
Agency­
wide
implementation.
Its
responsibilities
include
promoting
consistent
interpretation,
assessing
Agency­
wide
progress,
and
developing
recommendations
for
revisions
of
the
policy
as
necessary."
The
SPC
meets
its
responsibilities
through
coordination
with
the
Peer
Review
Coordinators,
the
Peer
Review
Advisory
Group
(
PRAG)
and
the
Office
of
Research
and
Development
(
ORD).

The
SPC,
PRAG
and
ORD
are
not
responsible
for
identifying
specific
products
for
peer
review
or
determining
the
level
of
review
or
mechanisim
for
that
review;
those
functions
are
the
responsibility
of
management
within
each
Office
or
Region.

1.4.13
What
is
the
Role
of
the
Peer
Review
Advisory
Group
(
PRAG)?

The
Science
Policy
Council
has
created
the
Peer
Review
Advisory
Group
(
PRAG)
to
assist
in
the
implementation
of
the
Agency's
Peer
Review
Policy.
The
primary
role
of
the
PRAG
is
to
provide
interpretation
of
the
policy
and
to
assist
the
SPC
and
Agency
Offices
and
Regions
in
the
annual
update
of
the
Peer
Review
Handbook.
Peer
Review
Handbook
Page
23
1.4.14
What
is
the
Role
of
the
Office
of
Research
and
Development
(
ORD)?

The
Deputy
Administrator
has
designated
the
Office
of
Research
and
Development
(
ORD)
to
assist
the
Program
Offices
and
Regions
in
the
collection
and
review
of
information
that
is
contained
in
the
annual
submission
of
Lists
A,
B
and
C
(
for
detailed
information,
see
sections
1.3.1,
1.3.2
and
1.3.3).
Peer
Review
Handbook
Page
24
THIS
PAGE
LEFT
INTENTIONALLY
BLANK
Peer
Review
Handbook
Page
25
When
in
doubt
about
whether
to
peer
review
a
work
product
or
not,
always
decide
to
make
it
a
candidate
for
peer
review.
2.
PLANNING
A
PEER
REVIEW
2.1
Overview
Statement
Planning
a
peer
review
is
a
critical
first
step
to
ensure
a
successful
peer
review
of
a
work
product.
The
initial
step
is
to
determine
whether
your
work
product
requires
peer
review.
Once
you
have
determined
that
a
peer
review
will
be
conducted,
the
Decision­
Makers
and
Peer
Review
Leaders
need
to
plan
an
appropriate
review.
This
includes
identification
of
resources
(
budget
and
personnel),
the
schedule
for
the
completion
of
the
peer
review,
the
mechanism
for
peer
review,
the
choice
of
peer
reviewers,
and
the
development
of
the
peer
review
record.

2.2
Determining
Which
Work
Products
to
Peer
Review
2.2.1
What
are
Scientific
and
Technical
Work
Products?

The
first
step
in
determining
which
work
products
require
peer
review,
is
to
identify
products
that
are
scientific
and/
or
technical
in
nature.
Scientific
and
technical
work
products
are
documents
or
positions
that
are
used
to
support
a
research
agenda,
regulatory
program,
policy
position
or
other
Agency
position
or
action.
Categories
include
the
following:
risk
assessments,
technical
studies
and
guidance,
analytical
methods,
scientific
database
designs,
technical
models,
technical
protocols,
statistical
survey/
studies,
technical
background
materials,
technical
guidance,
and
research
plans
and
strategies.

Products
that
wouldn't
be
considered
scientific
and
technical
work
products
can
include
those:
that
address
procedural
matters
(
e.
g.,
planning,
reporting,
coordination,
notification);
that
are
primarily
policy
statements
(
e.
g.,
relocation
policy);
that
are
conference
proceedings
(
unless
the
proceedings
are
used
as
the
scientific
basis
for
an
Agency
action
or
decision);
and
that
are
decision
documents
(
e.
g.,
Record
of
Decision
(
ROD)
­­
the
decision
document
itself
is
not
subject
to
the
Peer
Review
Policy,
but
the
underlying
scientific
and/
or
technical
support
work
product
is
a
candidate
for
peer
review).

2.2.2
What
Scientific
and
Technical
Work
Products
Need
Peer
Review?

The
principle
underlying
the
Peer
Review
Policy
is
that
all
major
scientific
and
technical
work
products
used
in
decision
making
will
be
peer
reviewed.
The
process
for
identifying
which
of
these
products
is
Peer
Review
Handbook
Page
26
"
major"
(
and
thus
a
candidate
for
peer
review)
and
then
determining
the
mechanism
of
review
will
take
into
account
various
criteria
and
the
circumstances
surrounding
the
use
of
that
work
product.
To
maintain
flexibility,
the
Decision­
Maker(
s)
for
peer
review
should
consider
the
full
field
of
possible
work
products
that
could
benefit
from
peer
review
and
the
full
spectrum
of
peer
review
mechanisms
for
each
product.
Once
a
decision
is
made
to
perform
peer
review,
the
product
is
listed
in
the
annual
submission
of
Candidate
Products
for
Future
Peer
Review
(
List
B
­
see
section
1.3.2b)).

2.2.3
How
Does
One
Determine
Whether
a
Scientific
and/
or
Technical
Work
Product
is
"
Major"?

Determinations
of
a
scientific
and/
or
technical
work
product
as
"
Major"
will
largely
be
case­
by­
case.
The
continuum
of
work
products
covers
the
range
from
the
obviously
major,
which
clearly
need
peer
review,
to
those
products
which
are
not
major
and
clearly
don't
need
peer
review.
The
rest
of
the
work
products
fall
in­
between
those
two
distinctions.
This
"
middleground
probably
represents
the
majority
of
work
products,
each
of
which
needs
to
be
evaluated
closely
and
be
compared
to
certain
criteria
(
see
below).
The
Decision­
Maker
needs
to
make
a
judgment
as
to
whether
a
work
product
meets
the
criteria
for
major
or
not.
There
is
no
easy
single
yes/
no
test
of
major
covering
the
whole
continuum
of
work
products.
A
rule
of
thumb
to
remember
­­
if
there
is
any
doubt
about
whether
a
work
product
needs
peer
review,
then
go
ahead
and
consider
it
a
candidate
for
peer
review
(
and
place
it
on
List
B
­
Candidate
Products
for
Future
Peer
Review).

Scientific
and
technical
work
products
that
are
used
to
support
a
regulatory
program
or
policy
position
and
that
meet
one
or
more
of
the
following
criteria
are
candidates
for
peer
review:

a)
Establishes
a
significant
precedent,
model,
or
methodology
b)
Addresses
significant
controversial
issues
c)
Focuses
on
significant
emerging
issues
d)
Has
significant
cross­
Agency/
inter­
agency
implications
e)
Involves
a
significant
investment
of
Agency
resources
f)
Considers
an
innovative
approach
for
a
previously
defined
problem/
process/
methodology
Peer
Review
Handbook
Page
27
g)
Satisfies
a
statutory
or
other
legal
mandate
for
peer
review
Usually,
a
major
scientific
and/
or
technical
work
product
supports
a
regulatory
decision
or
policy/
guidance
of
major
impact.
Major
impact
can
mean
that
it
will
have
applicability
to
a
broad
spectrum
of
regulated
entities
and
other
stakeholders,
or
that
it
will
have
narrower
applicability,
but
with
significant
consequences
on
a
smaller
geographic
or
practical
scale.
The
scientific
and/
or
technical
work
that
underlies
many
of
the
Agency's
major
rulemakings
and
policy
and
guidance
documents
of
general
applicability
would
be
designated
"
major"
under
this
scope
of
impact
criterion
because
of
their
far­
reaching
or
significant
impacts.

The
novelty
or
controversy
associated
with
the
work
product
helps
determine
whether
it
is
major
or
not.
A
major
work
product
may
be
novel
or
innovative,
precedential,
controversial,
or
emerging
("
cutting
edge").
An
application
of
an
existing,
adequately
peer
reviewed
methodology
or
model
to
a
situation
that
departs
significantly
from
the
situation
it
was
originally
designed
to
address
is
a
candidate
for
peer
review.
Similarly,
a
modification
of
an
existing,
adequately
peer
reviewed
methodology
or
model
that
departs
significantly
from
its
original
approach
is
a
candidate
for
peer
review.
Determination
of
"
significant
departure"
as
used
in
this
section
is
the
responsibility
of
the
Decision­
Maker.

In
summary,
a
major
scientific
or
technical
work
product
has
a
major
impact,
involves
precedential,
novel,
and/
or
controversial
issues,
or
the
Agency
has
a
legal
and/
or
statutory
obligation
to
conduct
a
peer
review.

2.2.4
What
Economic
Work
Products
Need
Peer
Review?

The
following
three
types
of
major
economic
work
products
require
peer
review:

a)
internal
Agency
guidance
for
conducting
economic
analysis
b)
new
economic
methodologies
or
unique
or
novel
applications
of
existing
economic
methodologies,
particularly
those
that
are
pathbreaking
c)
broad­
scale
economic
assessments
of
regulatory
programs,
such
as
the
Congressionally­
mandated
study
of
the
costs
and
benefits
of
the
Clean
Air
Act
For
all
these
types
of
economic
work
products,
we
will
pursue
an
independent
and
expert
external
peer
review,
such
as
by
the
recently
reconstituted
Environmental
Economics
Advisory
Committee
(
a
subcommittee
of
the
SAB),
or
other
appropriate
experts.
The
straight­
forward
application
of
accepted,
previously
peer­
reviewed
economic
methods
or
analyses
in
regulatory
Peer
Review
Handbook
Page
28
impact
analyses
supporting
rulemakings
or
policy
development,
however,
will
typically
not
be
subject
to
formal
peer
review.
Of
course,
if
the
particular
facts
and
circumstances
of
any
piece
of
economic
analysis
warrant
peer
review
beyond
that
described
above,
the
Agency
will
accommodate
those
on
a
case­
by­
case
basis.

2.2.5
How
Should
Peer
Review
be
Handled
for
Products
Developed
under
an
Interagency
Agreement?

When
funds
are
passed
to
another
agency,
it
will
likely
be
placed
in
a
contract,
cooperative
agreement,
or
grant.
In
many
instances,
the
receiving
agency's
guidance
for
peer
review
will
not
be
the
same
as
ours.
(
See
sections
2.2.6,
2.2.8,
2.2.10
and
3.6
for
further
details).

2.2.6
Should
Products
from
Grants,
Contracts
and
Cooperative
Agreements
Receive
Peer
Review?

If
there
is
a
scientific
and/
or
technical
work
product
resulting
from
a
grant,
contract,
or
cooperative
agreement
and
it
is
considered
major
and
will
likely
be
used
in
Agency
decisionmaking
the
work
product
needs
peer
review.
Since
it
would
probably
result
in
a
perceived,
if
not
real,
conflict
of
interest,
a
group
that
is
generating
the
work
product
usually
cannot
conduct
or
perform
the
peer
review
of
its
own
work
product.
Exceptions
may
be
made
in
certain
instances
for
organizations
that
have
adequate
and
well
established
recognized
procedures
for
peer
review,
such
as
the
National
Academy
of
Sciences.
In
practice,
the
Agency
may
need
to
peer
review
the
product
on
its
own,
or
arrange
with
an
independent
third
group
(
e.
g.,
via
another
extramural
vehicle)
to
arrange
for
the
peer
review.
The
Agency
should
not
use
the
major
scientific
and
technical
work
products
from
grants,
contracts,
or
cooperative
agreements
to
support
decision
making
unless
the
work
products
are
peer
reviewed
for
both
scientific
and
technical
rigor
and
applicability
to
the
specific
use
to
be
made
of
the
product.

2.2.7
Should
Site
Specific
Decisions
be
Subject
to
Peer
Review?

The
site
specific
decision
itself
is
not
subject
to
peer
review
and
doesn't
need
peer
review
based
solely
on
the
Peer
Review
Policy.
However,
if
a
site
specific
decision
is
supported
by
a
major
scientific
and/
or
technical
work
product,
that
work
product
needs
peer
review.
While
the
same
considerations
for
major
apply
here,
several
of
the
criteria
above
(
see
section
2.2.3;
specifically
criteria
b,
c,
d,
and
g)
are
considered
more
useful
for
regional
consideration
than
other
criteria.
So
generally
speaking,
a
close
examination
of
how
the
underlying
major
scientific
and/
or
technical
work
product
is
adapted
to
the
site
specific
circumstances
is
required.
Peer
Review
Handbook
Page
29
2.2.8
Should
NEPA
Products
(
e.
g.,
EISs)
be
Subject
to
Peer
Review?

Not
everything
requires
peer
review,
and
in
the
case
of
an
Environmental
Impact
Statement
(
EIS)
prepared
under
the
requirements
of
the
National
Environmental
Policy
Act
(
NEPA),
the
document
already
has
received
extensive
review
through
the
"
scoping"
and
interagency
review
processes
that
are
part
of
NEPA.

The
rule
of
thumb
is
that
if
the
underlying
scientific
and/
or
technical
work
product
is
major,
then
the
work
product
needs
peer
review.
In
general,
the
Agency's
role
in
the
NEPA
document
would
suggest
what
sort
of
review
the
document
gets.
If
EPA
is
developing
the
document
as
part
of
an
EPA
action/
decision
(
EPA
is
the
Lead
agency
under
NEPA),
and
it
meets
the
definition
of
"
major,"
then
it
needs
independent
peer
review.
If
it
is
not
a
major
work
product
(
little
impact,
non­
controversial,
etc.),
then
peer
input/
continuing
involvement
might
well
be
appropriate.

On
the
other
hand,
if
EPA
is
reviewing
an
EIS
from
another
Agency
(
EPA
is
not
the
Lead
agency
under
NEPA),
it
is
likely
that
we
are
reviewing
for
conflicts
with
EPA
policy
and
general
environmental
concerns.
However,
EPA
must
ask
if
the
underlying
major
scientific
and/
or
technical
work
product
that
supports
the
EIS
has
been
peer
reviewed.
If
not,
this
would
raise
concern
about
the
full
credibility
and
soundness
of
the
EIS
based
on
the
science
and
technical
support.
EPA
should
work
with
the
other
organization/
agency
to
ensure
that
the
major
scientific
and/
or
technical
work
product
receives
adequate
peer
review.

2.2.9
Should
Environmental
Regulatory
Models
be
Peer
Reviewed?

Generally,
yes.
Specific
guidelines
for
the
peer
review
of
environmental
regulatory
models
have
been
published
by
the
Agency.
These
can
be
found
on
the
EPA
web
site
under
the
Science
Policy
Council
home
page
(
http://
www.
epa.
gov/
ORD/
spc).

2.2.10
Is
Peer
Review
Needed
for
Other
Organization's
Work
Products
that
Have
been
Submitted
to
EPA
for
Use
in
Decision
Making?

Yes.
Any
scientific
and/
or
technical
work
product
that
is
used
in
Agency
decision
making
and
is
considered
major
needs
peer
review
regardless
if
the
work
product
is
produced
by
the
Agency
or
another
organization.
It
is
hoped
that
the
other
organization
outside
the
Agency
has
had
the
work
product
independently
peer
reviewed
and
the
peer
review
meets
the
intent
of
the
Agency's
Peer
Review
Policy
and
EPA's
proposed
use
of
the
product.
Agency
staff
should
examine
closely
the
particulars
of
the
peer
review
to
ensure
independence
and
a
conscious
effort
to
incorporate
the
peer
reviewers'
comments
into
the
final
work
product.
If
there
are
perceived,
Peer
Review
Handbook
Page
30
or
real,
conflicts
of
interest,
this
may
preclude
the
use
of
that
peer
review
and,
in
those
instances,
another
peer
review
would
be
needed.
See
section
2.4
for
further
details
on
peer
review
mechanisms.

2.2.11
Can
Work
Products
That
are
Not
Determined
to
be
Major
Still
be
Peer
Reviewed?

Yes,
they
could
be.
Scientific
and
technical
work
products
that
do
not
come
under
the
"
major"
distinction
discussed
above
may
nonetheless
be
candidates
for
peer
review.
For
example,
a
project
manager
may
decide
to
use
peer
review
because
of
particular
program
needs
and
goals.
Peer
review
may
also
be
warranted
because
it
adds
substantial
value
to
the
work
product.

2.3
Determining
Which
Work
Products
Do
Not
Receive
Peer
Review
2.3.1
Are
There
Circumstances
When
a
Major
Work
Product
is
Not
Peer
Reviewed?

There
may
be
circumstances
where
a
work
product
is
considered
major,
but
a
decision
for
no
peer
review
can
then
be
justified.
For
example:

a)
Additional
peer
review
is
not
required
with
work
that
has
been
previously
reviewed
by
recognized
experts
or
an
expert
body.
For
example,
a
cancer
risk
assessment
methodology
or
an
exposure
modeling
technique
that
was
the
subject
of
earlier
peer
review
would
not
require
additional
peer
review,
even
if
the
product
supported
a
significant
Agency
decision.

b)
Additional
peer
review
is
not
required
if
an
application
of
an
adequately
peer
reviewed
work
product
does
not
depart
significantly
from
its
scientific
or
technical
approach
(
see
section
2.2.3).

c)
Additional
peer
review
is
not
required
when
the
scientific
and/
or
technical
methodologies
or
information
being
used
are
commonly
accepted
in
the
field
of
expertise
(
this
would
need
the
appropriate
documentation
to
support
the
commonly
held
view).

d)
Most
often,
a
major
work
product
would
not
receive
peer
review
when
the
regulatory
activity
or
action
which
the
work
product
supports
is
terminated
or
canceled
­­
no
further
action,
including
peer
review,
is
necessary.
Peer
Review
Handbook
Page
31
e)
In
a
few
instances,
statutory
and
court
ordered
deadlines
and
other
time
constraints
may
limit
or
preclude
peer
review
of
products
that
would
otherwise
be
considered
major.
However,
it
is
up
to
the
Decision­
Maker(
s)
to
make
every
attempt
possible
to
assure
that
peer
review
of
major
work
products
occurs
taking
into
account
these
deadlines.

f)
Very
rarely,
resource
limitations
may
also
restrict
peer
review.
Programs
or
Regions
will
evaluate
these
circumstances
on
a
case
by
case
basis;
decisions
will
be
based
on
consultations
involving
line
management,
the
Project
Manager,
the
Peer
Review
Leader,
and
the
Peer
Review
Coordinator.

If
peer
review
of
a
major
scientific
and/
or
technical
work
product
is
not
conducted,
a
written
justification
must
be
placed
in
the
"
comments"
section
of
List
C
(
Products
for
Which
a
Decision
has
Been
Made
Not
to
Peer
Review).
The
justification
is
signed­
off
by
the
appropriate
Decision­
Maker
(
see
section
1.3.2c)).

2.3.2
What
Products
Normally
Do
Not
Need
Peer
Review?

Products
that
are
not
major
scientific
and
technical
work
products
normally
do
not
require
peer
review
under
the
intent
of
the
Peer
Review
Policy.
Most
of
these
scientific
and
technical
work
products
are
then
placed
on
List
C
with
a
written
comment
of
why
it
was
determined
to
be
not
major
(
see
section
1.3.2c)).
This
justification
is
signed­
off
by
the
appropriate
Decision­
Maker
to
assure
that
all
scientific
and
technical
work
products
received
consideration
for
peer
review.

Some
scientific
and
technical
work
products
are
not
considered
major
and
generally
do
not
need
to
be
placed
on
List
C
(
Products
for
Which
a
Decision
has
Been
Made
Not
to
Peer
Review).
These
types
of
work
products
typically
include:
derivative
products
(
i.
e.,
a
product
that
only
summarizes
an
already
peer
reviewed
product
or
products),
compendiums
of
existing
models,
methods
and/
or
technologies;
or
preliminary
or
incidental
analyses
prepared
separately
from
the
work
product
ultimately
used
to
support
an
Agency
action
or
decision
(
e.
g.,
during
the
course
of
developing
a
rule,
managers
may
direct
staff
to
prepare
various
"
what
if"
analyses;
those
that
aren't
used
in
the
work
product
do
not
need
to
be
listed).
Peer
Review
Handbook
Page
32
The
mechanism
of
the
peer
review
matches
the
importance
and
complexity
of
the
major
work
product.
2.4
Choosing
a
Peer
Review
Mechanism
2.4.1
How
Do
You
Determine
the
Appropriate
Peer
Review
Mechanism?

During
the
planning
of
a
peer
review,
the
Decision­
Maker
and
the
Peer
Review
Leader
may
consider
several
mechanisms
for
the
peer
review
of
major
scientific
and
technical
work
products.
These
options
range
from
consultations
with
EPA
colleagues
not
involved
in
developing
the
product
to
a
large
and
formal
panel
of
outside
subject
matter
experts.
The
peer
review
effort
might
be
a
focused
one­
time
evaluation,
or
could
encompass
several
examinations
over
the
course
of
a
project.
In
principle,
peer
review
provides
the
greatest
credibility
for
major
work
products
when
it
involves
wellqualified
external
reviewers,
is
intensive
in
its
examination,
and
operates
through
a
more
or
less
formal
process.
As
a
practical
matter,
however,
time
and
resource
considerations
in
many
cases
impose
limitations
on
what
can
be
reasonably
achieved.
Arranging
for
the
most
appropriate
and
feasible
peer
review
will
involve
good
judgment
and
a
willingness
to
consider
substance,
time,
and
resource
tradeoffs.
Developing
a
peer
review
plan
that
provides
for
appropriate
depth,
timing,
and
content
is
an
important
matter
for
early
consideration
by
the
Decision­
Maker
and
Peer
Review
Leader.
Note
that
use
of
peer
input,
public
or
stakeholder
involvement
does
not
qualify
as
peer
review.

The
approach
best
suited
to
a
specific
work
product
will
depend
on
the
nature
of
the
topic
and
the
intended
final
product.
Generally,
the
more
novel
or
complex
the
science
or
technology,
the
greater
the
cost
implications
of
the
impending
decision,
and
the
more
controversial
the
issue,
then
the
stronger
the
indication
for
a
more
extensive
and
involved
peer
review
and
for
external
peer
review
in
particular.
Certain
work
products
will
clearly
lend
themselves
to
extensive
external
peer
review;
generally
these
will
be
products
with
large
impacts
(
e.
g.,
those
that
support
Tier
1
and
Tier
2
rulemakings).
Other
major
work
products
may
not
need
a
large
scale
external
peer
review
and
may
utilize
a
less
involved,
less
resource
intensive
review.
The
peer
review
of
some
products
may
be
better
served
with
some
form
of
internal
peer
review
or
a
combination
of
internal
and
external
peer
review.

The
choice
of
peer
review
mechanism
will
depend
upon
the
experience
and
assessment
of
the
Decision­
Maker(
s)
dealing
with
peer
review
issues.
It
is
important
to
make
this
choice
at
the
time
that
the
work
is
planned
(
for
products
supporting
rule
makings,
at
the
analytic
blueprint
stage)
so
that
peer
review
costs
and
time
can
be
budgeted
into
the
work
plan.
Essentially,
the
level
of
peer
review
matches
the
impact
and
complexity
of
the
major
work
product.
For
Peer
Review
Handbook
Page
33
example,
a
rule
under
development
carries
considerable
weight
and
deserves
careful
handling
and
attention;
therefore,
the
supporting
work
product
deserves
similar
care
and
attention
for
its
peer
review.
Both
internal
and
external
peer
review
mechanisms
are
available,
have
been
used
in
the
past,
and
have
served
to
address
the
needs
and
challenges
of
a
particular
peer
review
situation.
Nevertheless,
no
single
peer
review
mechanism
is
likely
to
work
best
in
all
situations.
Some
useful
rules­
of­
thumb
include:

a)
Major
work
products
intended
to
support
the
most
important
decisions,
or
that
have
special
importance
in
their
own
right,
ordinarily
should
be
the
subject
of
external
peer
review.
Generally,
the
more
complex,
novel
and/
or
controversial
the
product,
the
more
the
Decision­
Maker
should
consider
implementing
a
large­
scale
peer
review
involving
external
experts.

b)
Major
work
products
that
are
less
complex,
novel,
or
controversial
may
not
need
such
a
large­
scale
and
external
peer
review.
These
products
might
be
subject
to
one
of
the
less
extensive,
less
resource­
intensive
review
processes.

c)
Group
discussion
with
peer
reviewers
can
be
very
helpful
at
some
point
in
the
peer
review
process.
On
the
other
hand,
simply
soliciting
individual
comments
is
easier,
faster,
and
less
expensive.
Individual
review
is
probably
more
appropriate
for
peer
review
at
the
early
stages
of
a
product's
development
or
for
products
with
less
impact
and
complexity.

d)
Strict
time
constraints,
such
as
a
court­
ordered
deadline,
can
make
a
less
involved
or
formal
peer
review
mechanism
imperative.
But
Decision­
Makers
and
Peer
Review
Leaders
must
make
maximum
efforts
to
assure
that
such
a
process
is
perceived
as
systematic
and
objective.

2.4.2
What
are
Examples
of
Internal
Peer
Review?

a)
Independent
experts
from
within
the
Agency
(
e.
g.,
ORD
experts
on
non­
cancer
effects
of
lead
reviews
a
draft
article
on
benchmark
dose)

b)
An
ad
hoc
panel
of
independent
experts
from
within
the
Agency
(
e.
g.,
an
independent
internal
workgroup
convened
to
examine
the
case
for
the
classification
of
a
chemical
as
a
carcinogen)

c)
Technical
merit
review
by
scientists
in
an
Agency
laboratory
(
e.
g.,
an
initial
review
of
the
risk
assessment
for
a
regional
incinerator
by
Agency
scientists)
Peer
Review
Handbook
Page
34
2.4.3
What
are
Examples
of
External
Peer
Review?

a)
Independent
experts
from
outside
the
Agency
(
e.
g.,
a
letter
review
by
outside
scientists)

b)
An
ad
hoc
panel
of
independent
experts
outside
the
Agency
(
e.
g.,
a
group
is
convened
to
develop
a
consensus
on
the
carcinogenicity
of
a
particular
industrial
chemical)

c)
Agency­
sponsored
peer
review
workshops
(
e.
g.,
a
review
of
potential
indicators
of
ecosystem
damage)

d)
Review
by
an
established
Federal
advisory
committee
such
as
the
Science
Advisory
Board
(
SAB),
FIFRA
Scientific
Advisory
Panel
(
SAP),
ORD's
Board
of
Scientific
Counselors,
or
the
Clean
Air
Scientific
Advisory
Committee
(
e.
g.,
a
review
of
a
criteria
document
for
a
particular
chemical
risk)

e)
Agency­
based
federal
advisory
committee
(
other
than
those
established
and
discussed
in
d
above)

f)
Agency
appointed
special
board
or
commission
(
e.
g.,
a
review
of
the
risk
assessment
methodology
prepared
by
the
Clean
Air
Act
Commission
on
Risk
Assessment)
Note:
The
Office
of
General
Counsel
should
be
consulted
regarding
EPA's
authority
to
establish
and
finance
the
activities
of
a
commission
or
board.

g)
Interagency
committee
(
e.
g.,
a
review
of
prospective
research
plans
by
the
Committee
on
the
Environment
and
Natural
Resources
coordinated
by
the
White
House)

h)
A
committee
convened
by
another
federal
agency
or
government
organization
(
e.
g.,
a
review
of
the
Dioxin
Reassessment
by
the
HHS
Committee
to
Coordinate
Environmentally
Related
Programs)

i)
Review
by
non­
governmental
groups
(
e.
g.,
a
Society
of
Risk
Analysis
review
of
cancer
guidelines)

j)
Review
by
the
National
Academy
of
Sciences
(
e.
g.,
a
review
of
the
state
of
current
knowledge
about
children's
health
risks
from
pesticide
exposures)
Peer
Review
Handbook
Page
35
2.4.4
What
is
the
Role
of
Peer
Review
by
a
Refereed
Scientific
Journal?

Peer
review
performed
under
the
auspices
of
a
peer
reviewed,
refereed,
published
journal
contributes
to
the
scientific
and
technical
credibility
of
the
reviewed
product.
Peer
review
of
an
EPA
produced
scientific
and
technical
article
by
a
recognized
refereed
journal
is
a
satisfactory
form
of
peer
review
for
the
purposes
of
publication
in
that
journal.
Peer
review
by
that
journal
is
also
considered
adequate
for
reviewing
the
scientific
credibility
and
validity
of
the
findings
(
or
data)
in
that
article.
Products
so
reviewed
are
included
on
List
C
(
Products
for
Which
a
Decision
has
Been
Made
Not
to
Peer
Review)
(
see
section
1.3.2c)).
No
other
action
is
needed
under
the
Peer
Review
Policy
unless
the
conditions
outlined
in
section
2.4.5
apply.

Prior
to
submitting
an
article
to
a
journal
for
peer
review,
EPA
employees
are
encouraged
to
have
the
article
internally
peer
reviewed
(
see
section
1.4.9);
such
internal
peer
review
is
already
common
procedure
in
certain
parts
of
EPA.
Articles
may
also
need
examination
in
accordance
with
any
organizational
clearance
procedures,
especially
when
the
author
is
presenting
him
or
herself
as
an
EPA
employee.
For
EPA
employees,
conflict
of
interest
regulations
will
also
apply.

2.4.5
Is
Peer
Review
Necessary
when
Journal
Articles
are
Used
in
an
Agency
Work
Product?

Peer
review
for
publication
in
a
journal
is
usually
performed
for
specific
reasons
for
that
journal.
The
use
of
articles
that
have
been
peer
reviewed
by
a
credible
journal
strengthens
the
scientific
and
technical
credibility
of
any
work
product
in
which
the
article
appears,
but
does
not
eliminate
the
need
to
have
the
work
product
itself
peer
reviewed.
For
instance,
journal
peer
review
may
not
cover
issues
and
concerns
that
the
Agency
would
want
peer
reviewed
to
support
an
Agency
action.
Under
these
circumstances,
the
major
scientific
and/
or
technical
work
product
in
which
the
article
appears
becomes
a
candidate
for
peer
review.
A
journal
article
authored
by
EPA
employees
would
be
used
in
the
same
manner
as
an
article
published
by
anyone
else
in
a
credible,
well
recognized
journal.

If
an
Agency
work
product
is
based
solely
on
a
single
article
that
has
received
peer
review
by
a
credible
journal
(
e.
g.,
where
a
model
is
suggested
for
a
singular
use
that
fits
a
specific
Agency
need),
journal
peer
review
of
the
article
is
not
a
substitute
for
peer
review
of
the
Agency
work
product.
Generally,
when
a
major
work
product
is
supported
by
more
than
one
journal
article,
that
work
product
becomes
a
candidate
for
peer
review.
These
decisions
need
to
be
documented
in
the
peer
review
record.
Peer
Review
Handbook
Page
36
One
important
factor
to
remember
with
regard
to
the
use
of
articles
that
have
received
journal
peer
review
deals
with
the
availability
of
documentation
from
that
peer
review.
Ideally,
EPA
needs
to
maintain
a
clear,
easily
accessible
record
of
the
peer
review
to
assure
the
credibility
and
validity
of
the
peer
review
(
see
section
2.5
for
details
on
peer
review
record).
Much
of
the
information
required
for
the
Agency
peer
review
record
may
not
be
readily
accessible
from
a
journal
publisher
or
editorial
board
(
e.
g.,
names
of
the
peer
reviewers,
the
charge
to
them,
and
their
specific
comments).
However,
in
most
cases
this
documentation
from
the
journal
peer
review
is
not
needed
to
complete
the
Agency's
peer
review
record
for
a
given
action
­­
in
extreme
cases,
such
as
where
litigation
is
involved,
such
documentation
may
be
needed.

2.4.6
When
and
How
Often
Should
Peer
Review
Occur?

The
Decision­
Maker
and
Peer
Review
Leader
have
significant
discretion
in
deciding
the
timing
and
the
frequency
of
peer
review.
Options
abound,
each
with
merits
depending
on
the
context
and
specified
peer
review
objectives.

A
single
peer
review
event,
beginning
when
the
final
draft
work
product
becomes
available,
is
the
approach
usually
taken
in
many
situations.
On
the
other
hand,
a
peer
review
far
earlier
in
a
project
cycle
could
be
a
superior
approach
in
some
circumstances.
For
example,
early
review
might
be
beneficial
at
the
stage
of
research
design
or
data
collection
planning
where
the
product
involves
extensive
primary
data
collection.
Or,
there
may
be
substantial
incremental
benefit
to
conducting
several
peer
reviews
along
the
way,
particularly
where
a
project
involves
complex
tasks,
has
decision
branching
points,
or
could
be
expected
to
produce
controversial
findings.
(
See
also
section
3.4
on
the
selection
of
peer
reviewers).

2.4.7
What
Factors
are
Considered
in
Setting
the
Time
Frame
for
Peer
Review?

Several
factors
impact
how
quickly
a
peer
review
may
be
needed.
These
include
deadlines
for
completion
of
a
project,
research
program,
or
rulemaking,
funding
availability,
availability
of
quality
peer
reviewers,
and
statutory
and/
or
court­
ordered
deadlines.

Peer
review
sometimes
leads
to
new
information
and
analyses.
Reviewers
may
make
recommendations
for
new
research
that
would
alter
the
work
product
and
thus
modify
the
scientific/
technical
basis
for
the
action
or
rule
it
supports.
For
this
reason,
a
completed
peer
review
is
desirable
before
issuing
any
proposal
for
public
comment.
If
that
is
not
logistically
possible
because
of
court
or
statutory
deadlines,
or
other
appropriate
reasons,
the
Decision­
Maker
should
make
every
effort
to
complete
the
peer
review
before
the
close
of
the
comment
period.
Because
peer
review
comments
on
such
work
products
could
be
of
sufficient
magnitude
to
Peer
Review
Handbook
Page
37
warrant
a
revision
to
the
proposed
action
or
rule,
Decision­
Makers
should
exercise
diligence
in
completing
the
peer
review
prior
to
the
proposal
stage
whenever
possible.

2.4.8
Which
Office/
Region
or
Other
Agency
is
Responsible
for
Conducting
the
Peer
Review?

The
organization
of
the
Decision­
Maker
is
normally
responsible
for
conducting
the
peer
review.
Responsibility
for
conducting
a
peer
review
can
be
negotiable
when
more
than
one
Agency
office
or
region
or
other
agencies
are
involved.
Usually,
the
degree
of
involvement
by
any
of
the
organizations
and
agencies
and
their
ability
to
fund
peer
review
will
often
determine
who
has
the
lead
for
the
peer
review.

2.5
Creating
the
Peer
Review
Record
2.5.1
What
is
the
Peer
Review
Record?

It
is
the
formal
record
(
file)
of
decision
on
the
conduct
of
the
peer
review,
the
type
of
peer
review
performed,
and
a
summary
of
the
outcome
of
that
peer
review.
It
includes
sufficient
documentation
for
an
uninvolved
person
to
understand
what
actually
happened
and
why.
The
Peer
Review
Leader
(
with
the
program
manager
if
there
is
one)
creates
a
separate,
clearly
marked
peer
review
file
section
within
the
overall
file
for
development
of
the
work.
Once
the
peer
review
is
completed,
it
is
the
responsibility
of
the
Peer
Review
Leader
to
ensure
that
the
peer
review
record
is
filed
and
maintained
in
accordance
with
the
organization's
procedures.

2.5.2
How
Can
the
Peer
Review
Record
Improve
the
Peer
Review
Process?

A
good
peer
review
record
allows
future
reference
to
what
happened
during
the
peer
review,
and
helps
Decision­
Makers
make
appropriate
use
of
peer
reviewer
input.
In
addition,
a
good
record
helps
ensure
that
EPA's
Peer
Review
Policy
is
followed.
The
Peer
Review
Leader
is
responsible
for
ensuring
that
the
peer
review
record
for
individual
work
products
is
collected
and
maintained
until
completion
of
the
peer
review
effort.

2.5.3
What
Should
Be
in
the
Peer
Review
Record?

The
peer
review
record
should
include
all
materials
considered
by
the
individual
peer
reviewers
of
the
peer
review
panel,
as
well
as
their
written
comments
and
other
input.
Such
materials
include,
at
a
minimum
(
see
also
section
4.3.1):

a)
The
draft
work
product
submitted
for
peer
review
Peer
Review
Handbook
Page
38
b)
Materials
and
information
(
including
the
charge)
given
to
the
peer
reviewers
c)
Written
comments,
information,
and
materials
received
from
the
peer
reviewers
d)
Information
about
the
peer
reviewers
(
such
as
reviewers'
names,
affiliations,
and
identified
potential
conflicts
and
their
resolution)

e)
Logistical
information
about
conduct
of
the
peer
review
(
such
as
times
and
locations
of
meetings)

f)
A
memorandum
or
other
written
record
responding
to
the
peer
review
comments
specifying
acceptance
or,
where
thought
appropriate,
rebuttal
and
non­
acceptance
g)
The
final
work
product
h)
Peer
review
summary
report
(
see
sections
1.3.1
and
1.3.2)

When
deciding
if
particular
materials
should
be
included
in
the
record,
the
Peer
Review
Leader
should
consider
whether
the
materials
would
help
reconstruct
the
peer
review
process
and
results
at
a
later
time.
If
the
materials
may
be
helpful,
they
should
be
part
of
the
peer
review
record.

In
addition
to
hard
copies
of
materials,
Peer
Review
Leaders
need
to
maintain
electronic
copies
of
the
materials
(
e.
g.,
charge)
that
are
necessary
for
the
annual
reports
compiled
by
the
Peer
Review
Coordinators.
Peer
Review
Leaders
should
discuss
what
are
those
materials
and
requirements
for
the
annual
report
with
their
coordinators.

2.5.4
What
Should
I
Do
with
a
Peer
Review
Record
That
Pertains
to
a
Rulemaking
Action?

The
Peer
Review
Leader
should
coordinate
with
their
program's
docket
office
to
see
that
proper
docketing
requirements
are
satisfied
for
a
peer
review
of
a
work
product
supporting
a
new
rule.
The
Peer
Review
Leader
is
also
responsible
for
notifying
the
workgroup
chair
as
well
as
the
Peer
Review
Coordinator
(
for
the
annual
report)
that
a
peer
review
is
completed.

2.5.5
When
Should
the
Peer
Review
Record
Building
Process
Begin?

An
early
start
at
developing
and
maintaining
a
peer
review
record
will
help
ensure
the
record
is
complete
and
helpful.
Ideally,
the
record
begins
when
the
decision
to
peer
review
a
Peer
Review
Handbook
Page
39
work
product
is
made
(
and
includes
the
written
decision).
The
Peer
Review
Leader
needs
to
construct
the
peer
review
record
from
this
point
on
­­
this
will
avoid
potentially
time­
consuming
reconstruction
at
a
later
point.
Note
that
the
peer
review
record
is
not
complete
until
it
contains
a
copy
of
the
final
work
product
which
addresses
the
peer
review
comments.

2.5.6
What
are
the
Differences
in
Record
Keeping
for
a
Review
by
an
Individual
Compared
to
a
Panel?

Strictly
speaking,
a
true
peer
review
requires
more
than
a
single
reviewer.
A
review
conducted
by
one
individual
will
rarely
provide
the
depth
of
commentary
required
to
improve
the
work
product.
In
addition,
you
will
not
receive
the
range
of
views
and
richness
necessary
to
ensure
improvement.

In
the
case
of
a
review
panel,
there
will
often
be
conflicting
comments
that
must
be
resolved.
This
resolution
should
be
in
the
record.

2.5.7
Where
Should
the
Peer
Review
Record
be
Kept
and
For
How
Long?

During
the
active
conduct
of
the
peer
review,
the
Peer
Review
Leaders
maintain
the
peer
review
record
with
themselves
until
the
peer
review
is
totally
completed.
Minimally,
the
file
should
be
maintained
until
one
year
after
the
completed
peer
review
is
reported
in
the
next
annual
reporting.
After
that,
the
peer
review
record
should
be
maintained
for
a
"
reasonable
period
of
time."
Establishment
and
maintenance
of
the
archive
where
the
peer
review
records
ultimately
reside
are
an
organization's
responsibility
(
i.
e.,
not
that
of
an
individual
program
manager
or
Peer
Review
Leader).
Generally,
to
allow
flexibility,
individual
offices
and
regions
will
decide
the
appropriate
level
of
organizational
responsibility
and
how
they
will
meet
any
"
routinely
available"
requirements.
The
peer
review
record
may
be
kept
with
other
records
relating
to
the
overall
project,
as
long
as
it
is
easily
and
separately
identifiable.

There
are
also
specific
requirements
regarding
the
use
of
dockets
for
record­
keeping;
however,
these
are
not
covered
in
this
Peer
Review
Handbook.
The
documents
contained
in
the
peer
review
record
should
be
maintained
in
accordance
with
the
Agency's
record
keeping
retention
schedule
for
such
records.
One
long­
term
archiving
mechanism
may
be
the
formal
archiving
at
the
Federal
Records
Center
in
Suitland,
MD.
Peer
Review
Handbook
Page
40
Peer
review
is
part
of
the
normal
cost
of
doing
business.
2.6
Budget
Planning
2.6.1
What
Budgetary
Factors
Should
I
Consider
in
a
Peer
Review?

Resources
needed
to
implement
the
Peer
Review
Policy
need
to
be
requested
through
the
usual
Agency
budgetary
processes.
The
budget
formulation
process
within
the
Executive
Branch
is
followed,
after
appropriation
bills
are
passed
by
Congress,
by
budget
execution.
These
two
processes
provide
opportunities
to
secure
resources
for
activities
carried
out
by
Headquarters
and
Regional
offices,
including
peer
review.
The
major
work
products
for
which
decisions
for
peer
review
have
been
made
(
List
B
candidates)
need
to
have
adequate
funding
for
peer
review
in
budget
requests
for
the
coming
fiscal
year.
Similarly,
adequate
funding
needs
to
appear
in
the
actual
approved
operating
budget
to
ensure
their
conduct.
For
purposes
of
budget
planning,
the
costs
of
peer
review
would
include
the
FTE
cost
of
staff,
the
contract
or
other
costs
associated
with
the
use
of
outside
peer
reviewers,
and
the
administrative
costs
of
conducting
a
review
(
copying,
travel
expenses,
etc).

2.6.2
What
Input
is
Needed
for
the
Annual
Budget
Formulation
and
Budget
Execution
Process?

Senior
Management
in
Office
and
Regions
(
including
Decision­
Makers
and
budget
officers)
need
to
be
sure
that
budget
requests
include
anticipated
resources
for
peer
review.
Peer
review
needs
to
be
considered
as
a
normal
part
of
doing
business.
Peer
review
resource
considerations
should
also
be
addressed
in
the
analytic
blueprint
for
Agency
rule­
making
actions.

2.7
Legal
Considerations
2.7.1
Are
There
Legal
Ramifications
From
the
Peer
Review
Policy?

The
Peer
Review
Policy
does
not
establish
or
affect
legal
rights
or
obligations.
Rather,
it
confirms
the
importance
of
peer
review
where
appropriate,
outlines
relevant
principles,
and
identifies
factors
Agency
staff
should
consider
in
implementing
the
policy.
Except
where
provided
otherwise
by
law,
peer
review
is
not
a
formal
part
of
or
substitute
for
notice
and
comment
rulemaking
or
adjudicative
procedures.
EPA's
decision
to
conduct
peer
review
in
any
particular
case
is
wholly
within
the
Agency's
discretion.
Similarly,
nothing
in
the
Policy
creates
a
legal
requirement
that
EPA
respond
to
peer
reviewers.
However,
to
the
extent
that
EPA
Peer
Review
Handbook
Page
41
decisions
rely
on
scientific
and
technical
work
products
that
have
been
subjected
to
peer
review,
the
remarks
of
peer
reviewers
should
be
included
in
the
record
for
that
decision.

2.7.2
Is
Legal
Advice
Needed?

AA/
RA
staff
and
management
should
work
regularly
with
individual
OGC/
Regional
Counsel
(
RC)
staff
assigned
to
Agency
activities.
Peer
Review
Leaders
should
initially
consult
with
their
customary
OGC/
RC
advisors
for
legal
advice
or
referral.
Headquarters
attorneys
have
specialties
in
specific
areas
and
can
be
consulted
as
needed
(
e.
g.,
FACA
considerations
(
see
below);
contractual
responsibilities;
ethics
and
potential
conflicts
of
interest).

2.7.3
Is
Peer
Review
Subject
to
the
Federal
Advisory
Committee
Act
(
FACA)?

It
depends
on
how
the
peer
review
is
carried
out.
The
Federal
Advisory
Committee
Act,
5
U.
S.
C.
App.
2,
imposes
certain
open
meeting,
balanced
membership,
and
chartering
requirements
(
with
OMB
and
GSA
approval)
before
the
Agency
"
establishes
or
utilizes"
an
"
advisory
committee"
for
advice
or
recommendations.

Peer
review
carried
out
by
formal
and
established
(
chartered)
Federal
advisory
committees
(
such
as
the
Science
Advisory
Board
or
the
FIFRA
Science
Advisory
Panel)
is
always
subject
to
FACA
requirements.
However,
not
all
peer
review
is
carried
out
by
established
Federal
advisory
committees.
For
example,
if
EPA
conducts
a
peer
review
meeting
for
the
purpose
of
obtaining
advice
from
the
individual
attendees
and
not
for
the
purpose
of
obtaining
a
peer
review
product
from
the
group
as
a
whole
(
consensus),
the
meeting
would
not
be
subject
to
FACA.

Committees
established
by
an
outside
organization
(
such
as
by
a
Federal
contractor)
to
provide
the
outside
party
with
advice
and
recommendations
are
probably
exempt
from
FACA.
You
should
be
aware,
however,
that
the
outside
party's
committee
could
be
subject
to
FACA
if
EPA
manages
or
controls
the
committee
(
e.
g.,
EPA
selects
the
members
of
the
panel,
runs
the
meeting,
etc.).
Also
remember,
a
contractor
cannot
arrange
to
have
peer
review
performed
on
a
work
product
the
contractor
itself
prepared.

EPA
officials
conducting
peer
review
may
wish
to
mitigate
the
potential
for
a
FACA
challenge
by
seeking
balanced
participation
at
peer
review
meetings
and
allowing
interested
members
of
the
public
to
attend.
Questions
concerning
the
applicability
of
FACA
to
peer
review
meetings
should
be
addressed
to
the
FACA
experts
in
the
Cross­
Cutting
Issues
Division
of
OGC
(
Mail
Code
2322
at
Headquarters).
Peer
Review
Handbook
Page
42
THIS
PAGE
LEFT
INTENTIONALLY
BLANK
Peer
Review
Handbook
Page
43
The
time
spent
preparing
a
good
charge
is
well
spent,
and
is
crucial
for
an
effective
peer
review.
3.
CONDUCTING
A
PEER
REVIEW
3.1
Overview
Statement
The
success
and
usefulness
of
any
peer
review
depends
on
the
quality
of
the
draft
work
product
submitted
for
peer
review,
the
care
given
to
the
statement
of
the
issues
or
"
charge,"
the
match
between
the
peer
review
draft
product
and
the
form
of
peer
review,
the
match
between
the
peer
review
draft
product
and
the
scientific/
technical
expertise
of
the
reviewers,
and
Agency
use
of
peer
review
comments
in
the
final
product.
It
is
not
simply
enough
to
conduct
a
peer
review;
each
of
the
foregoing
elements
requires
serious
attention.

3.2
Charge
to
the
Peer
Reviewers
3.2.1
What
is
a
Charge?

As
part
of
each
peer
review,
the
Peer
Review
Leader
must
formulate
a
clear,
focused
charge
that
identifies
recognized
issues
and
invites
comments
or
assistance.
This
request
signals
the
Agency's
awareness
of
potential
issues
and
its
receptivity
to
expert
recommendations.
The
charge
to
peer
reviewers
usually
makes
two
general
requests.
First,
it
focuses
the
review
by
presenting
specific
questions
and
concerns
that
the
Agency
expects
the
reviewers
to
address.
Secondly,
it
invites
general
comments
on
the
entire
work
product.
The
specific
and
general
comments
should
focus
mostly
on
the
scientific
and
technical
merits
of
the
work
product
and,
where
germane,
whether
the
scientific/
technical
studies
have
been
applied
in
a
sound
manner.
Remember,
the
peer
review
is
not
for
the
decision
or
action
itself,
but
for
the
underlying
scientific
and/
or
technical
work
product.
Focused
questions
greatly
simplify
the
task
of
collating,
analyzing
and
synthesizing
peer
review
comments
on
a
topical
basis.
The
questions
should
be
specific
enough
to
get
helpful
comments,
but
not
so
specific
(
unless
very
specific
points
are
needed
to
be
addressed)
that
they
preclude
creative
responses.
Moreover,
the
written
responses
to
these
questions
by
peer
reviewers
help
the
Agency
create
a
peer
review
record.
As
a
general
rule,
the
time
drafting
a
good
charge
letter
is
well­
spent
and
is
necessary
for
an
effective
peer
review.
Peer
Review
Handbook
Page
44
3.2.2
What
are
the
Essential
Elements
of
a
Charge?

a)
Brief
overview
or
introduction
(
describe
what
the
work
product
is,
how
it
was
developed,
how
it
will
be
used),

b)
As
needed,
a
brief
description
or
listing
of
any
background
materials
provided
to
the
peer
reviewers,
and
c)
The
issues
or
questions
to
be
addressed
by
the
peer
reviewer(
s).

In
addition,
the
following
should
also
be
included
as
important
administrative
components
of
a
charge:
a)
the
due
date
of
reviewer
comments
and
format
of
reviewer
responses,
and
b)
the
point
of
contact
in
case
peer
reviewers
have
questions.

3.2.3
Where
Can
I
Get
an
Example
of
a
Charge?

Appendix
B
contains
examples
of
successful
charges
that
cover
a
variety
of
issues.
These
include:
WTI
Risk
Assessment,
Region
10
review
of
the
Eastern
Columbia
Plateau
sole
source
aquifer,
letter
reviews
for
IRIS
(
Integrated
Risk
Information
System)
entries,
and
several
examples
of
charges
for
SAB
reviews.
Appendix
C
provides
guidance
for
obtaining
Science
Advisory
Board
(
SAB)
services.

3.3
Time
Line
3.3.1
What
are
the
Factors
in
Scheduling
a
Peer
Review?

The
peer
review
schedule
is
a
critical
feature
of
the
process.
The
schedule
must
take
into
account
the
availability
of
a
quality
draft
work
product,
availability
of
appropriate
experts,
time
available
for
using
peer
review
comments,
deadlines
for
the
final
work
product,
and
logistical
aspects
of
the
peer
review
(
e.
g.,
contracting
procedures).

The
schedule
for
peer
review
should
take
into
account
the
overall
rulemaking
(
or
other
decision
making)
schedule.
For
rules,
in
particular
those
in
Tier
1
and
Tier
2,
the
scheduling
of
the
peer
review
should
be
included
in
the
development
of
the
analytic
blueprint.
Peer
review
sometimes
leads
to
new
information
and
analyses,
or
recommendations
for
new
research
that
would
alter
the
work
product
and
thus
modify
the
scientific/
technical
basis
for
the
action.
For
this
reason,
it
is
usually
advisable
to
complete
the
peer
review
before
taking
public
comment,
or
at
least
before
the
close
of
the
public
comment
period.
Peer
Review
Handbook
Page
45
3.4
Selection
of
Peer
Reviewers
3.4.1
What
are
Considerations
for
Selecting
Peer
Reviewers?

Selection
of
independent
peer
reviewers
is
not
a
trivial
task,
and
it
is
crucial
to
an
effective
peer
review.
It
is
important
that
peer
reviewers
be
selected
for
independence
and
scientific/
technical
expertise.
However,
the
very
need
to
have
experienced
individuals
on
a
peer
review,
along
with
the
desire
to
have
appropriate
technical
balance
and
representation,
can
mean
that
the
selection
of
potential
peer
reviewers
often
comes
from
those
who
are
considered
as
having
a
potential
bias.
To
reduce
the
concern
that
a
potential
panel
may
have
unnecessary
bias,
it
may
be
useful
to
obtain
an
informal
review
of
the
expertise
and
balance
of
potential
peer
reviewers
from
others
in
your
organization,
from
OGC
or
even
from
outside
groups.
Sometimes
selecting
individuals
who
have
served
in
a
variety
of
organizations
rather
than
a
single
one
for
an
extended
period,
provides
expertise
with
diverse
perspective.
The
emphasis
on
independence
and
expertise
applies
equally
to
government
experts
and
experts
from
the
larger
scientific
community.

Some
peer
reviews
can
be
conducted
with
two
or
three
reviewers;
others
involve
panels
of
peer
reviewers.
In
either
case,
each
peer
reviewer
should
have
recognized
technical
expertise
that
bears
on
the
subject
matter
under
discussion.
In
instances
where
there
is
more
than
one
peer
reviewer,
the
peer
reviewers
of
a
work
product
should
represent
a
balanced
range
of
technically
legitimate
points
of
view.
In
addition,
cultural
diversity
and
"
address"
(
e.
g.,
industrial,
academic,
or
environmental
community)
are
other
factors
that
can
play
a
role
in
selecting
peer
reviewers.

3.4.2
Where
Do
I
Find
Peer
Reviewers?

Recommendations
for
potential
peer
reviewers
can
be
identified
from
a
number
of
organizations.
These
include
external
groups
such
as
the
affected
party(
ies),
special
interest
groups,
public
interest
groups,
environmental
groups,
professional
societies,
trade
or
business
associations,
state
organizations
or
agencies,
Native
American
Tribes,
colleges
and
universities,
the
National
Research
Council,
and
other
Federal
agencies
with
an
involvement
in
or
familiarity
with
the
issue.
Agency
associated
groups
include
the
staff
of
the
Science
Advisory
Board
(
SAB)
or
the
Scientific
Advisory
Panel
(
SAP),
and
relevant
scientific
and
technical
experts
from
Program
or
Regional
offices.

In
certain
circumstances,
existing
peer
review
organizations
such
as
the
SAB
or
SAP
may
be
used
to
conduct
a
peer
review.
These
groups
establish
their
own
criteria
for
accepting
work
and
coordination
must
be
made
directly
with
them
(
see
Appendix
C
for
SAB
procedures).
Both
SAB
and
SAP
conduct
formal,
public,
external
peer
reviews.
Peer
Review
Handbook
Page
46
Occasionally,
a
member
of
the
scientific
community
will
offer
his/
her
services
for
peer
review
during
an
ongoing
peer
review.
These
offers
may
be
at
no
cost
or
based
on
an
expectation
that
reimbursement
will
be
made.
Disposition
of
these
unsolicited
offers
will
be
handled
on
a
case
by
case
basis
by
the
Peer
Review
Leader,
and
as
necessary,
in
consultation
with
the
Peer
Review
Coordinator,
the
Office
of
General
Counsel
(
OGC),
and
appropriate
Decision­
Makers.

If
you
use
a
contract
mechanism
to
conduct
a
peer
review,
the
contractor
may
have
its
own
pool
of
scientific
and
technical
experts
for
peer
review.
With
contractors,
EPA
can
provide
information
on
potential
sources
of
peer
reviewers
for
conducting
a
peer
review
if
such
a
listing
were
prepared
in
alphabetical
order,
and
contained
an
Agency
disclaimer
that
EPA
doesn't
recommend
any
particular
individual
or
firm
on
the
list.
Furthermore,
when
utilizing
a
contract
mechanism
to
conduct
peer
review,
EPA
is
not
permitted
to
direct
the
prime
contractor
to
a
specific
sub­
contractor
or
peer
reviewer
nor
is
EPA
permitted
to
directly
interact
with
the
peer
reviewer
(
sub­
contractor).
All
interaction
must
be
with
the
prime
contractor's
designated
representative
and
not
the
contractor's
staff.

3.4.3
Are
External
or
Internal
Peer
Reviewers
Preferred?

External
peer
reviewers
are
generally
preferred,
particularly
for
most
final
work
products.
For
some
work
products,
like
those
reviewed
at
interim
steps,
either
external
or
internal
peer
review
may
be
appropriate.
Selection
of
internal
peer
reviewers
should
be
based
upon
technical
expertise,
available
time
and
"
address"
­­
that
is,
they
should
not
come
from
the
immediate
office
or
group
producing
the
product
or
have
any
other
connection
with
the
product
or
document
being
peer
reviewed.
External
peer
reviewers
should
be
selected
based
upon
technical
expertise
as
well,
however,
care
must
be
taken
not
to
use
individuals
who
have
been
involved
in
the
development
of
the
work
product.
(
See
also
sections
1.4.6
to
1.4.9).

3.4.4
What
is
Important
in
the
Mix
of
a
Peer
Review
Panel?

A
peer
review
panel
or
group
can
number
from
just
a
few
individuals
to
ten
or
more,
depending
on
the
issue,
the
time
and
resources
available,
and
the
broad
spectrum
of
expertise
required
to
treat
the
range
of
issues/
questions
in
the
charge.
Naturally,
experts
whose
understanding
of
the
specific
technical
area(
s)
being
evaluated
are
critical;
nevertheless,
it
is
also
important
to
include
a
broad
enough
spectrum
of
other
related
experts
to
completely
evaluate
the
relevant
impacts
on
other
less
obvious
concerns
(
i.
e.,
to
comment
not
only
if
the
job
is
being
done
right,
but
also
whether
the
right
job
is
being
done).
For
example,
for
health
related
peer
reviews,
experts
in
such
fields
as
ecology
and
economics
may
provide
very
useful
insights.
Although
persons
who
are
familiar
and
have
a
substantial
reputation
in
the
field
are
often
called
Peer
Review
Handbook
Page
47
upon
repeatedly
to
be
reviewers,
it
is
important
to
keep
a
balance
with
new
people
who
bring
fresh
perspectives
to
the
review
of
a
work
product.

There
is
usually
a
continuum
of
views
on
any
issue.
To
the
extent
possible
or
practicable,
selected
experts
should
have
technically
legitimate
points
of
views
that
fall
to
either
side
of
the
central
position
along
the
continuum,
but
not
too
far
to
either
extreme.
This
will
help
maintain
a
balanced
viewpoint,
while
allowing
all
views
to
be
expressed
and
discussed.
A
balanced
panel
will
allow
consensus
building
(
if
consensus
is
the
object
of
a
particular
peer
review;
if
not,
it
allows
a
spectrum
of
reviews
for
the
Agency
to
evaluate).
As
a
general
rule,
experts
who
have
made
public
pronouncements
on
an
issue
(
e.
g.,
those
who
have
clearly
"
taken
sides")
may
have
difficulty
in
reaching
consensus
positions
and
should
be
avoided.

3.4.5
What
is
a
Conflict
of
Interest?

Conflict
of
interest
is
a
situation
in
which
a
person
is
unable
or
potentially
unable
to
render
impartial
assistance
or
advice
to
the
Agency,
or
the
person's
objectivity
in
performing
the
work
is
or
might
be
otherwise
impaired,
or
a
person
has
an
unfair
competitive
advantage.
Generally,
a
conflict
of
interest
or
a
perceived
conflict
of
interest
arises
when
the
person
is
affected
by
his/
her
private
interests,
or
when
he/
she
or
his/
her
associates
would
derive
benefit
from
incorporation
of
their
point
of
view
in
an
Agency
product.
Whenever
there
are
questions
about
conflicts
of
interest,
you
should
contact
the
appropriate
official
in
OGC
for
clarification.

3.4.6
What
Techniques
Help
Ensure
Disclosure
and
Appropriate
Resolution
of
Conflicts
of
Interest?

Before
finalizing
the
selection
of
reviewers,
the
Peer
Review
Leader
should
ask
potential
reviewers
if
they
have
any
real
or
perceived
conflicts
of
interest.
One
way
of
identifying
conflicts
is
to
ask
potential
reviewers
about
current
and
prior
work,
and
prior
clients
that
might
create
conflicts
or
the
appearance
thereof.
This
information
becomes
part
of
the
peer
review
record.

Care
must
be
taken
to
reduce
the
possibility
for
real
or
apparent
conflicts
of
interest
between
the
reviewers
and
the
work
product
under
review.
Various
tools
are
available
to
identify
and
limit
conflicts
of
interest
(
e.
g.,
attention
to
the
employment,
financial,
and
professional
affiliations
of
the
participants;
filing
Confidential
Financial
Disclosure
Forms
(
SF­
450)
in
the
case
of
members
of
Federal
Advisory
Committees;
exploring
directly
the
issue
with
each
of
the
participants
before
the
review
process
takes
place;
and
disclosing
publicly
at
the
beginning
of
meetings
any
previous
involvement
with
the
issue).
Established
peer
review
groups
such
as
the
Science
Advisory
Board
provide
useful
models
for
addressing
balance
and
conflict­
of­
interest
Peer
Review
Handbook
Page
48
issues.
Assistance
in
determining
legal
conflicts
of
interest
and
in
providing
an
appropriate
response
can
be
obtained
through
the
Office
of
the
General
Counsel.
In
addition,
additional
advice
can
be
obtained
from
the
Designated
Agency
Ethics
Officials
(
DAEO).

Of
course,
conflicts
do
not
necessarily
arise
merely
because
a
peer
reviewer
knows
something
about
the
subject
matter.
In
fact,
experts
with
a
stake
in
the
outcome
­­
and
therefore
a
potential
conflict
­­
may
be
some
of
the
most
knowledgeable
and
up­
to­
date
experts
because
they
have
concrete
reasons
to
maintain
their
expertise.
Such
experts
could
be
used
provided
the
potential
conflicts
of
interest
are
disclosed
and
the
peer
review
panel
or
group
being
used
as
a
whole
is
balanced.
In
some
cases,
however,
the
conflict
may
be
so
direct
and
substantial
as
to
rule
out
a
particular
expert,
for
instance,
a
potential
peer
reviewer
who
may
have
a
client
or
employer
with
a
direct
financial
stake
in
the
matter
under
review
(
e.
g.,
a
manufacturer
of
a
chemical
under
review).
However,
review
of
a
general
methodology
that
applies
to
numerous
chemicals
would
not
necessarily
raise
such
a
concern.
(
Note:
language
that
reflects
these
concerns
should
be
made
part
of
contracts/
statements
of
work
(
SOW)
or
purchase
orders
(
PO)
­­
see
section
3.6
and
Appendix
D).

A
Peer
Review
Leader
may
want
to
adopt
measures
that
will
prevent
creation
of
conflicts
as
the
peer
review
is
underway.
Such
measures
might
include
clauses
in
a
contract
or
purchase
order
that
requires
reviewers
to
receive
advance
approval
on
future
work,
or
places
limits
on
such
work,
while
they
are
performing
the
current
peer
review.
The
intent
behind
such
measures
is
to
avoid
having
the
peer
reviewer
gain
an
unfair
future
advantage
as
a
result
of
their
participation
on
a
peer
review
process.
Note
that
at
some
level
these
types
of
measures
will
discourage
experts
from
serving
as
peer
reviewers.
(
See
section
3.6.5
for
further
information
dealing
with
contracts
and
suggestions
for
appropriate
management
controls).

3.4.7
Can
Parties
External
to
EPA
Pay
for
Their
Own
Peer
Reviews?

There
may
be
instances
where
parties
external
to
EPA
will
want
to
conduct
and/
or
pay
for
a
peer
review
on
a
particular
work
product
(
presumably
their
own
work
product
or
one
they
are
closely
interested
in,
or
they
wouldn't
be
interested
in
expending
resources).
This
may
look
benign
at
first
blush,
but
is
a
very
complex
and
sensitive
situation
that
can
raise
significant
concerns
for
perceived
and/
or
actual
conflicts
of
interest
for
interested
parties
"
paying"
for
a
peer
review
of
their
own
work
product.
While
the
Agency
cannot
prevent
external
parties
from
conducting
and
paying
for
a
peer
review,
it
is
desireable
that
any
such
peer
review
meets
the
intent
of
the
Agency's
Peer
Review
Policy
and
adheres
to
the
principles
and
guidance
in
this
Handbook.
If
the
external
party
submits
their
work
product
and
accompanying
peer
review,
the
materials
will
be
treated
by
the
Agency
as
anything
else
submitted
for
the
Agency's
evaluation.
Peer
Review
Handbook
Page
49
We
will
evaluate
the
work
product
and
the
peer
review
for
scientific
credibility
and
validity
before
making
any
decisions
based
on
the
materials.

3.4.8
Are
There
Constraints
to
Selecting
Peer
Reviewers?

Sometimes
the
need
for
a
peer
review
is
accelerated
due
to
a
court­
ordered
deadline
or
other
time­
sensitive
requirements.
In
such
cases,
it
is
difficult,
if
not
impossible
to
obtain
external
peer
reviewers
in
time
to
conduct
a
full
external
peer
review.
It
may
even
be
impossible
to
conduct
a
small
scale
internal
peer
review
using
just
a
few
individuals.
Mechanisms
for
identifying
and
using
a
small
number
of
peer
reviewers
should
be
developed
so
that
quick,
effective
peer
review
can
be
included
for
even
the
most
rapidly
moving
products.

Another
possible
constraint
involves
confidential
business
information
(
CBI).
To
evaluate
certain
Agency­
generated
studies
properly,
some
peer
reviewers
may
need
access
to
CBI.
However,
unless
the
reviewers
are
Federal
employees
with
CBI
clearance,
the
Agency
doesn't
have
the
independent
authority
to
disclose
CBI
to
them.
Therefore,
whenever
contemplating
the
use
of
outside
peer
reviewers,
Agency
staff
should
determine
whether
the
reviewers
will
need
access
to
CBI.
If
they
don't
have
CBI
clearance,
the
Office
of
the
General
Counsel
should
be
consulted
on
whether
it
is
practical
to
obtain
the
consent
of
CBI
submitters
to
disclose
the
information
to
peer
reviewers.

Offices
need
to
be
aware
of
the
requirements
of
the
Federal
Advisory
Committee
Act
(
FACA)
when
establishing
peer
review
mechanisms
(
see
section
2.7.3).
Federal
advisory
committees
that
are
subject
to
chartering
by
the
General
Services
Administration
must
hold
meetings
that
are
open
to
the
public,
and
have
balanced
membership
requirements.
The
Office
of
the
General
Counsel
should
be
consulted
regarding
the
applicability
of
FACA
to
peer
review
panels.

3.5
Materials
for
Peer
Reviewers
3.5.1
What
Instructions
Do
You
Give
Peer
Reviewers?

The
Peer
Review
Leader
is
responsible
for
ensuring
that
peer
reviewers
understand
and
comply
with
these
responsibilities
(
see
section
3.6
if
a
contract
is
involved):

a)
Advise
the
Agency
of
any
real
or
perceived
conflicts­
of­
interest
b)
Provide
written
comments
in
specified
format
by
the
specified
deadline
that
are
responsive
to
the
charge
Peer
Review
Handbook
Page
50
c)
Comply
with
the
request
for
not
disclosing
draft
work
products
to
the
public
3.5.2
What
Materials
Should
be
Sent
to
Peer
Reviewers?

For
a
peer
review
to
be
successful,
peer
reviewers
should
receive
several
documents
at
the
beginning
of
the
process.
Typically,
the
most
important
among
these
documents
are
the
charge
letter
and
the
current
work
product.
The
charge
letter
describes
what
the
peer
reviewers
are
being
asked
to
do,
and
should
serve
to
focus
and
structure
the
review.
The
work
product
is,
of
course,
the
material
being
subject
to
peer
review.

Remember,
no
documents
should
be
provided
directly
to
a
potential
peer
reviewer
when
that
reviewer
is
going
to
be
working
under
a
contract
or
purchase
order.
In
the
case
of
a
contract,
the
Agency
provides
the
work
product
with
associated
background
material
to
be
peer
reviewed
to
the
prime
contractor
who
in
turn
distributes
these
documents
to
the
peer
reviewers.
In
the
case
of
a
purchase
order,
the
"
charge
or
statement
of
work"
must
be
part
of
the
PO
(
purchase
order)
and
the
provision
of
any
documents
needs
to
be
coordinated
with
the
contracting
officer
handling
the
order.

a)
Essential
documentation
for
each
peer
reviewer
includes:

1)
A
current
copy
of
the
work
product
to
be
peer
reviewed
with
associated
background
material.
The
work
product
needs
to
be
of
the
best
possible
scientific/
technical
quality
to
ensure
an
adequate
and
useful
peer
review.

2)
A
clear
charge
or
statement
of
work
seeking
informed
comment
on
identified
issues
to
properly
focus
the
efforts
of
the
peer
reviewers
and
ensure
that
their
individual
efforts
can
be
compared
or
contrasted.

3)
Some
information
concerning
the
process
that
you
use
for
the
peer
review,
including
the
due
date
of
reviewer
comments,
the
format
of
those
responses,
and
a
point
of
contact
in
case
the
peer
reviewer
has
questions.
Responses
should
be
written
and
submitted
to
the
Peer
Review
Leader
by
an
agreed
upon
deadline.
In
certain
rare
cases,
oral
commentary
may
be
sufficient.
However,
in
such
cases,
a
follow­
up
written
response
for
the
record
is
required.

4)
In
some
cases,
Agency
materials
being
peer
reviewed
will
be
available
to
the
public,
even
if
they
are
marked
as
drafts.
For
example,
all
materials
reviewed
by
the
SAB
are
available.
Agency
managers
may
also
decide
Peer
Review
Handbook
Page
51
that
a
broad
accessibility
has
benefits
for
the
Agency.
In
other
cases,
confidentiality
needs
to
be
maintained.
In
these
cases,
each
peer
reviewer
must
be
informed
of
the
need
for
confidentiality
with
regard
to
the
release
of
Agency
products
that
are
stamped
as
"
DRAFT"
or
"
DRAFT
­
Do
Not
Cite,
Quote,
or
Release."
Premature
release
of
draft
Agency
products,
views,
or
positions
is
inappropriate
and
can
be
damaging
to
the
credibility
of
the
Agency
or
the
peer
reviewer.
While
not
necessarily
having
legal
consequences,
such
language
will
be
included
in
the
charge
to
the
peer
reviewers.
Other
mechanisms
to
use
in
discouraging
premature
release
include
a
disclaimer
that
appears
in
a
separate
section
at
the
front
of
the
document
and
creating
the
document
with
watermarks
clearly
delineating
DRAFT
status
(
or
a
header
or
footer
that
states
DRAFT
status)
on
every
page.
In
addition,
in
any
solicitation
for
peer
reviewers,
the
necessity
for
confidentiality
and
the
non­
release
of
materials
shall
be
emphasized.

b)
Useful,
but
not
critical
materials
that
may
be
sent
to
peer
reviewers
include:

1)
The
name,
address,
and
phone
and
fax
numbers,
and/
or
Internet
address
of
each
peer
reviewer
working
on
the
specific
review
2)
A
bibliography
and/
or
any
particularly
relevant
scientific
articles
from
the
literature
3)
A
work
product
that
has
line
numbering
added
in
the
margin
for
ease
in
providing
and
referencing
comments
c)
Peer
Reviewers
should
be
given
what
is
needed
to
complete
their
task
­­
they
should
not
be
overburdened
with
excess
material.

3.5.3
How
Closely
can
EPA
Interact
with
Peer
Reviewers
During
the
Review?

The
Peer
Review
Leader
normally
has
administrative
contacts
with
the
reviewers
during
the
development
and
conduct
of
the
peer
review.
In
some
cases
(
e.
g.,
SAB
peer
review),
peer
reviewers
may
also
receive
a
briefing
on
the
product
to
be
peer
reviewed.
Otherwise,
the
Peer
Review
Leader
and
other
EPA
staff
should
not
contact
the
reviewers
during
the
course
of
the
review.
Such
contact
can
lead
to
perceived
conflicts
or
inappropriate
direction
that
could
compromise
the
independence
of
the
review.
Peer
Review
Handbook
Page
52
If
peer
review
is
conducted
under
a
contract
mechanism,
EPA
must
limit
direct
contact
to
the
prime
contractor's
designated
representative
and
not
have
general
contact
and
direction
to
the
contractor's
staff
or
peer
reviewers
(
sub­
contractors).
Note,
when
a
peer
review
is
conducted
under
a
contract,
there
are
constraints
where
EPA
staff
are
prohibited
from
contacting
peer
reviewers
to
avoid
personal
services
arrangements.
Personal
services
contracts
exist
when
the
nature
of
the
relationship
between
the
contractor
and
the
EPA
can
be
characterized
as
an
employer
­
employee
relationship.
Any
communications
with
peer
reviewers
must
be
only
from
the
Peer
Review
Leader
or
contracting
officer.

3.6
Peer
Review
Services
A
range
of
peer
review
services
are
available
to
the
Agency
including
internal,
external
(
gratuitous
services,
contracts,
purchase
order),
and
Special
Government
Employee
(
SGE)
mechanisms.
The
mechanism
selected
is
generally
based
on
the
nature
of
the
scientific
or
technical
work
product.

3.6.1
What
are
Gratuitous
Services
for
Peer
Review?

The
provision
of
peer
review
products
or
services
to
EPA
without
compensation
are
provided
as
so­
called
"
gratuitous"
services.
If
a
person
or
organization
wishes
to
perform
peer
review
services
for
EPA
without
compensation,
the
program
must
ask
them
to
sign
an
agreement
whereby
the
person
or
organization
agrees
to
provide
the
prescribed
peer
review
services
as
gratuitous
services,
with
no
expectation
of
receiving
compensation
for
these
services
from
EPA.
An
agreement
must
be
executed
because
the
Antideficiency
Act
(
31
U.
S.
C.
§
1342)
prohibits
the
Agency's
acceptance
of
"
voluntary"
services.
"
Voluntary"
services
are
provided
to
EPA
without
an
agreement
in
advance
that
such
services
are
provided
at
no
cost
to
EPA.
Note
that
persons
cannot
waive
compensation
(
i.
e.,
agree
to
provide
gratuitous
services)
for
which
there
is
a
statutory
right
to
payment,
unless
a
law
permits
the
waiver.

3.6.2
Can
I
Use
a
Contract
to
Obtain
Peer
Review
Services?

The
Agency
may
contract
for
peer
review
services.
The
contract
may
be
written
solely
for
peer
reviews
or
be
included
as
one
of
several
specifically
described
interrelated
tasks
in
a
contract
that
requires
the
contractor
to
provide
more
than
just
peer
review
services.

For
assistance
in
preparing
the
necessary
pre­
award
documents,
program
officials
should
consult
The
Cookbook:
How
to
Get
Contracts
Awarded
in
EPA
and
Chapter
2
of
the
Contracts
Management
Manual
(
CMM).
The
Cookbook
and
Chapter
2
are
available
on
EPA's
Intranet
Peer
Review
Handbook
Page
53
(
http://
www.
intranet.
epa.
gov/
oamintra)
and
on
the
Agency
LAN
Services
(
Administration/
Management/
OAM
Procurement
Policy
Information).

3.6.3
How
Do
I
Write
a
Statement
of
Work
for
Contracts?

The
statement
of
work
(
SOW)
must
clearly
specify
that
the
contractor
is
responsible
for
preparing
peer
review
evaluations
and
set
forth
guidelines
for
the
peer
review
of
scientific
or
technical
documents.
The
contractor
may
perform
the
peer
review
with
in­
house
staff,
subcontractors
or
consultants.
Any
guidelines
for
performing
peer
reviews
to
ensure
soundness
and
defensibility
must
be
developed
by
the
program
office
and
made
part
of
the
contract.
The
contractor
would
then
ensure
that
the
peer
reviews
adhered
to
the
guidelines.

The
SOW
cannot
simply
define
the
role
of
the
prime
contractor
as
arranging
for
the
services
of
others
to
perform
peer
reviews
and
logistics
for
meetings.
Unless
the
prime
contractor
is
clearly
tasked
with
responsibility
for
performing
peer
reviews,
individual
peer
reviewers'
fees
and
associated
travel
expenses
are
not
payable
under
the
contract.

The
EPA
may
pay
for
the
reviewer's
comments
or
evaluation,
and
also
for
attendance
at
a
meeting
with
the
Agency
and
other
reviewers
to
discuss
the
results
of
the
peer
review.
If
the
SOW
calls
for
the
preparation
of
comments
or
an
evaluation,
and
specifies
a
meeting
with
the
Agency
and
other
peer
reviewers
to
discuss
the
results
of
the
peer
review,
payment
is
appropriate.
The
peer
reviewer's
attendance
at
the
meeting
would
then
be
part
of
contract
performance.

Example
statements
of
work
are
found
in
Appendix
D.

3.6.4
What
are
Advisory
and
Assistance
Services
(
AAS)
or
Sensitive
Activities?

Contracts
that
provide
services
that
support
or
improve
Agency
decision­
making
or
policy
development
are
subject
to
special
management
controls.
These
services
include
services
acquired
from
non­
Governmental
sources
by
contract
to
support
or
improve
Agency
policy
development,
decision­
making,
management,
and
administration,
or
research
and
development
activities.
See
Federal
Acquisition
Regulation
(
FAR)
37.201
for
a
more
specific
definition
of
AAS.
Such
services
may
take
the
form
of
information,
advice,
opinions,
alternatives,
conclusions,
recommendations,
training,
and
direct
assistance.
For
additional
information
on
advisory
and
assistance
services
and
sensitive
activities,
program
officials
should
review
Chapter
2
of
the
CMM.

New
contracts
for
these
services
require
management
approvals
prior
to
issuance
of
the
solicitation.
For
the
thresholds
that
have
been
established
for
approval
of
these
justifications,
see
Peer
Review
Handbook
Page
54
Figure
5,
Item
B
Management
Approvals
set
forth
after
Chapter
2
of
the
CMM
(
for
current
approval
levels).

3.6.5
What
are
Some
Management
Controls
for
Contracts?

Contracting
for
peer
review
services
is
permitted.
However,
because
of
the
potential
for
improper
use
of
these
contracts,
special
management
controls
are
required.

a)
Inherently
Governmental
Functions
(
IGFs)
­­
OFPP
Policy
Letter
92­
1,
dated
September
23,
1992,
describes
(
1)
functions
that
are
inherently
governmental
and
must
be
performed
only
by
Government
employees
and
(
2)
functions
that
may
be
contracted,
but
so
closely
support
Government
employees
in
their
performance
of
IGFs
that
the
contract
terms
and
performance
require
close
scrutiny
by
Federal
officials.
Federal
Acquisition
Regulation
(
FAR)
coverage
of
inherently
governmental
functions
is
at
FAR
Subpart
7.5.

Peer
reviews
represent
only
a
contractor's
recommendations,
advice
or
analysis
of
a
document.
Agency
officials
must
make
the
official
Agency
decision
regarding
acceptability
and/
or
quality
of
the
document.
To
ensure
that
Agency
officials
are
not
improperly
influenced
by
recommendations
in
the
peer
review,
management
controls
must
be
included
in
the
contract.
One
possible
control
would
be
to
require
the
peer
reviewers
to
submit
with
their
evaluations
or
comments
a
description
of
the
procedures
used
to
arrive
at
their
recommendations;
a
summary
of
their
findings;
a
list
of
sources
relied
upon;
and
make
clear
and
substantiate
the
methods
and
considerations
upon
which
their
recommendations
are
based.
To
the
extent
possible,
the
contract
should
set
forth
any
guidelines
or
criteria
for
performance
of
the
peer
review.
Agency
officials
should
document
their
evaluations
of
the
quality
and
validity
of
the
peer
review.

b)
Conflict
of
Interest
(
COI)
­­
Another
important
factor
is
that
the
objectivity
of
the
peer
review
should
not
be
improperly
influenced
or
undermined
by
the
contractor
performing
the
review.
To
identify
and
avoid
or
mitigate
actual
or
potential
COI,
the
contract
should
include
controls.
Such
controls
might
require
the
contractor
to
report
on
prior
and
current
work,
and
prior
clients
that
might
create
COI.
Other
controls
might
include
Agency
review
and
placing
limits
or
advance
approval
on
future
work.
There
should
also
be
procedures
implemented
to
assure
that
the
contractor
does
not
gain
an
unfair
advantage
in
future
requirements
as
a
result
of
their
performance
of
peer
reviews.
Program
officials
should
consult
the
Contracting
Officer
(
CO)
for
special
contract
clauses.
Peer
Review
Handbook
Page
55
FAR
coverage
of
conflicts
of
interest
is
at
FAR
Subpart
9.5.
The
EPA
Acquisition
Regulations
(
EPAAR)
at
48
CFR
Subpart
1509.5
generally
mandates
conflicts
of
interest
solicitation
provisions
and
contract
clauses,
but
makes
them
optional
for
procurements
accomplished
through
the
use
of
simplified
acquisition
procedures
as
set
forth
in
FAR
Part
13.
Also
see
48
CFR
1509.507­
1(
b)(
3)
and
48
CFR
1552.209­
70,
­
71
&
­
72
as
additional
resources.

Contract
for
peer
review
services:
An
EPA
contracting
officer
will
include
conflicts
of
interest
solicitation
provisions
and
contract
clauses
as
a
matter
of
course
without
involvement
by
the
EPA
project
officer,
if
the
peer
review
services
are
not
obtained
pursuant
to
the
simplified
acquisition
procedures
in
FAR
Part
13.
If
the
peer
review
services
are
subcontracted
pursuant
to
a
prime
contract,
then
the
prime
contractor
is
ordinarily
required
to
include
a
conflicts
of
interest
clause
substantially
similar
to
the
conflicts
of
interest
clause
in
the
primary
contract
in
its
subcontract
to
the
peer
reviewer.

Although
the
EPA
contracting
officer
and/
or
prime
contractor
has
the
primary
responsibility
to
include
the
required
conflicts
of
interest
provisions/
clauses,
the
EPA
project
officer
may
nevertheless
wish
to:

1)
Highlight
the
conflict
of
interest
requirements
in
the
Scope
of
Work
for
the
procurement
of
the
peer
review
services
2)
Develop
a
specific
conflict
of
interest
clause
regarding
the
peer
review
at
issue
as
a
substitute
or
in
addition
to
the
standard
conflicts
of
interest
clause
3)
Review
the
solicitation/
contract
to
make
sure
that
the
required
conflicts
of
interest
clause
has
been
included
Simplified
acquisitions
of
peer
review
services:
Although
conflict
of
interests
requirements
are
optional
for
simplified
acquisitions,
they
are
nevertheless
a
good
idea.
Accordingly,
an
EPA
project
officer
obtaining
peer
review
services
with
simplified
acquisition
procedures
should
request
the
purchasing
agent/
contracting
officer
to
include
a
conflict
of
interest
solicitation
provision
and
contract
clause
in
the
purchase
order.
Peer
Review
Handbook
Page
56
c)
Confidential
Business
Information
(
CBI)/
Privacy
Act
Protected
Information
and
Other
Sensitive
Information
­­
When
peer
reviewers
are
not
employees
of
the
United
States
Government,
it
is
unlikely
that
the
EPA
will
have
authority
to
give
reviewers
access
to
confidential
business
information
in
the
absence
of
consent
for
such
disclosure
by
the
CBI
submitter.
Therefore,
all
documents
provided
to
non­
Federal
reviewers
must
be
screened
for
information
claimed
as
CBI.
Even
where
business
information
has
not
been
explicitly
claimed
as
CBI,
if
it
is
of
a
kind
where
the
submitter
might
be
expected
to
object
to
its
release,
prior
to
release
the
submitter
must
be
asked
whether
it
wishes
to
assert
a
claim,
unless
the
submitter
has
previously
been
informed
that
failure
to
assert
a
CBI
claim
may
result
in
disclosure
without
notice.
Language
is
included
in
the
contract
to
clearly
identify
any
required
procedures
or
processes
prior
to
release
of
any
protected
information,
including
any
requirements
for
confidentiality
agreements,
as
well
as
limits
on
use
and
disclosure
of
the
data
by
contractor
personnel.

d)
Personal
services
­­
Under
contracts,
the
EPA
may
not
engage
the
peer
reviewers
in
any
improper
personal
services
relationships,
i.
e.,
an
arrangement
under
which
contractor
personnel
are
subject
to
relatively
continuous
supervision
and
direct
control
by
an
Agency
official
or
employee.
These
relationships
are
characterized
as
one
where
the
contractor
employee
interacts
with
the
Agency
in
a
manner
similar
to
that
of
a
Federal
employee.

To
avoid
these
improper
relationships,
program
officials
should
write
welldefined
SOWs.
The
SOWs
should
set
forth
the
requirements
in
detail
for
work
to
be
performed
independently,
including
the
manner
in
which
it
will
be
evaluated.
The
SOW
must
set
forth
what
work
is
to
be
performed
not
how
the
work
is
to
be
performed.
Technical
direction
may
be
used
to
clarify
ambiguous
technical
requirements
to
ensure
efficient
and
effective
contractor
performance,
and
is
not
considered
supervision
or
assignment
of
tasks.
For
additional
information,
program
officials
should
consult
EPA
Order
1901.1A,
Use
of
Contractor
Services
to
Avoid
Improper
Contracting
Relationships
and
FAR
Subpart
37.1
3.6.6
Can
the
Agency
Identify
and/
or
Select
Peer
Reviewers
Through
a
Contract?

Program
officials
cannot
interfere
in
a
contractor's
authority
and
responsibility
to
perform
work
by
"
selecting"
who
will
perform
the
peer
review
(
doing
so
may
invoke
FACA
­­
see
section
2.7.3).
The
Federal
Acquisition
Regulation
(
FAR)
governs
the
Contracting
Officer's
(
CO's)
and
program
officials'
relationship
with
the
contractor.
Peer
Review
Handbook
Page
57
However,
the
CO
does
have
the
ability
to
review
and
consent
to
subcontractors
and
consultants.
The
contract
can
also
specify
which
individuals
are
key
personnel
and
include
peer
reviewers.
The
contract
will
set
forth
the
procedures
governing
the
use
and
replacement
of
key
personnel.
Further,
the
contract
can
require
workplans
for
approval
by
the
Agency,
wherein
the
contractor
will
propose
the
peer
reviewers
it
is
considering
for
selection.

Program
officials
should
identify
the
qualifications
required
to
perform
the
review
work
and
the
criteria
for
technical
acceptability.
The
EPA
may
identify
a
pool
of
qualified
subcontractors
and
consultants
to
the
prime
contractor
(
listed
in
alphabetical
order),
but
cannot
direct
the
use
of
any
particular
subcontractor
or
consultant.

3.6.7
Can
I
Use
Simplified
Acquisition
Procedures
to
Obtain
Peer
Reviewers?

In
some
instances,
peer
reviewers
can
be
obtained
via
simplified
acquisition
procedures.
The
acquisition
of
supplies
or
non­
personal
services
from
the
open
market
and
on
a
sole
source
basis
when
the
aggregate
amount
involved
in
any
one
transaction
does
not
exceed
$
100,000
constitutes
a
simplified
acquisition
(
FAR
Part
13).
The
same
considerations
in
the
section
3.6.5
discussion
on
IGFs,
COI,
access
to
CBI,
and
personal
services
apply
to
simplified
acquisitions.
Normally,
the
Government
issues
a
purchase
order
directly
to
the
individual
peer
reviewer,
instead
of
to
a
prime
contractor
who
may
subcontract
for
performance
of
the
peer
review.

a)
Approvals
­­
All
small
purchases
for
peer
reviews
are
considered
AAS.
See
Chapter
2
of
the
Contract
Management
Manual
(
CMM)
for
the
appropriate
approval
levels
for
AAS.

b)
Competition
­­
The
Federal
Acquisition
Regulation
(
FAR)
requires
competition
for
purchases
in
excess
of
$
2,500.
Purchases
for
more
than
$
2,500
and
not
more
than
$
100,000
are
to
be
made
only
from
small
businesses
unless
the
Contracting
Officer
is
not
able
to
obtain
two
or
more
offers
from
small
businesses
that
are
competitive
in
terms
of
market
price,
quality,
and
delivery.
Only
one
source
need
be
solicited
if
the
Contracting
Officer
determines
that
only
one
source
is
reasonably
available.
Contracting
Officers
are
encouraged
to
use
best
value.

c)
Procurement
Requests
­­
Program
Officers
should
include
the
following
in
all
PRs
for
the
purchase
of
peer
reviews:

1)
A
fixed­
price
amount
at
or
below
the
simplified
acquisition
threshold
Peer
Review
Handbook
Page
58
2)
A
detailed
description
of
the
requested
services,
inclusive
of:

(
a)
Total
quantity
per
line
item
(
b)
Estimated
unit
price
per
line
item
(
c)
Total
cost
per
line
item
(
d)
Specific
deliverables
for
each
line
item
(
e)
Total
cost
of
the
purchase
request
3)
Reference
FAR
Subpart
3.6
and
Environmental
Protection
Agency
Acquisition
Regulation
(
EPAAR)
Subpart
1503.601
regarding
sources
from
Government
employees
or
organizations
owned
and
controlled
by
them.

4)
Provide
sources
from
small
businesses,
if
available.

5)
If
the
request
is
a
sole
source
purchase,
justification
must
be
provided
in
accordance
with
the
EPAAR
Subpart
1513.170­
1.

3.6.8
How
is
Travel
Handled
with
Contracts
or
Purchase
Orders?

Funds
obligated
on
a
contract
or
purchase
order
are
available
to
pay
for
the
costs
of
producing
the
peer
review
including
the
travel
costs
and
fee
of
the
peer
reviewer.

The
EPA
acquires
peer
reviews
through
simplified
acquisitions
issued
directly
to
peer
reviewers
or
through
contracts
with
companies,
which
provide
the
peer
review
services.
By
issuing
a
purchase
order
or
awarding
a
contract
for
peer
review
services,
the
EPA
may
pay
not
only
for
the
peer
review
services/
comments,
but
also
for
attendance
at
a
meeting
with
the
Agency
and
other
reviewers
to
discuss
comments.
The
scope
of
work
of
the
contract
must
require
the
contractor
or
individual
peer
reviewer
as
appropriate
to
provide
peer
review
services
and
indicate
whether
the
contractor
or
peer
reviewer
will
be
required
to
discuss
a
specific
peer
review
work
product
with
the
Agency
and/
or
with
other
peer
reviewers.
Attendance
at
a
meeting
to
discuss
a
peer
review
work
product
would
then
be
part
of
the
contract's
performance.
Thus,
the
contract
may
serve
as
the
mechanism
to
pay
for
peer
review
services
and
associated
travel
expenses
to
provide
comments
to
the
EPA.
Peer
Review
Handbook
Page
59
3.6.9
How
is
Travel
Handled
with
Special
Government
Employees?

The
term
Special
Government
Employee
(
SGE)
is
defined
in
18
U.
S.
C.
202(
a)
as
an
officer
or
employee
of
an
agency
who
performs
temporary
duties,
with
or
without
compensation,
for
not
more
than
130
days
in
a
period
of
365
days,
either
on
a
full­
time
or
intermittent
basis.

Travel
and
per
diem
expenses
of
experts
hired
as
SGEs
for
peer
review
may
only
be
paid
through
the
issuance
of
invitational
travel
orders
(
5
U.
S.
C.
§
5703).
These
invitational
travel
and
per
diem
expenses
should
be
charged
to
an
appropriate
EPA
travel
account.

Members
of
the
SAB,
SAP,
and
other
FACA
advisory
committees
are
often
brought
on
board
as
SGEs.
It
is
not
appropriate
to
reimburse
travel
or
per
diem
expenses
of
advisory
committee
members
or
other
SGEs
through
a
contract.
Peer
Review
Handbook
Page
60
THIS
PAGE
LEFT
INTENTIONALLY
BLANK
Peer
Review
Handbook
Page
61
The
peer
review
is
not
completed
until
the
peer
review
comments
are
incorporated
into
the
final
work
product.
4.
COMPLETING
A
PEER
REVIEW
4.1
Overview
Performance
of
the
formal
peer
review
is
not
the
final
stage
in
the
development
of
the
work
product.
Rather,
it
is
an
important
stage
in
developing
the
work
product,
with
the
final
work
product
representing
the
true
end
of
the
peer
review.
As
a
result,
the
peer
review
process
closes
with
three
major
activities:
evaluating
comments
and
recommendations,
utilizing
peer
review
comments
for
completing
the
final
work
product,
and
organizing
and
maintaining
a
record
of
the
peer
review.

Careful
attention
to
all
of
these
elements,
singly
and
together,
assures
a
credible
peer
review
process.
Conversely,
inattention
can
nullify
the
peer
review
attempt.
A
well­
planned
peer
review
applied
to
a
reasonable
quality
starting
work
product,
followed
by
responsible,
visible
utilization
of
peer
review
suggestions
in
the
final
product
assures
a
credible
product
for
use
in
Agency
decision­
making.

The
peer
review
is
not
completed
until
the
peer
review
comments
are
incorporated
into
the
final
work
product,
or
reasons
are
stated
why
such
comments
are
not
to
be
incorporated.
Once
the
peer
review
is
completed,
the
Peer
Review
Coordinator
moves
the
peer
reviewed
scientific
and/
or
technical
work
product
from
List
B
(
or
List
C
in
some
instances)
to
List
A
in
their
office's
annual
report.

4.2
Final
Work
Product
4.2.1
How
Do
I
Incorporate
Peer
Review
Comments
into
the
Final
Work
Product?

The
Peer
Review
Leader
must
carefully
evaluate
and
analyze
all
peer
review
comments
and
recommendations.
As
discussed
earlier,
a
carefully
crafted
charge
to
the
peer
reviewers
simplifies
organizing
and
analyzing
comments.
Also,
any
other
issues
that
are
raised
need
to
be
identified
and
evaluated.

The
validity
and
objectivity
of
the
comments
need
to
be
evaluated.
Analyses
may
include
consultation
with
other
experts/
staff
within
the
Office
and/
or
Agency.
Adequate
documentation
is
needed
to
show
that
comments
are
accepted
or
rejected
­­
the
documentation
can
be
brief,
but
Peer
Review
Handbook
Page
62
must
address
the
legitimate,
valid
comments.
It
is
the
responsibility
of
the
Peer
Review
Leader
to
obtain
management
approval
of
the
approach
to
addressing
the
peer
review
comments.

Comments
that
have
significant
impact
on
time,
budgetary,
and/
or
resource
requirements
are
particularly
important
and
need
to
be
evaluated
in
consultation
with
management.
These
comments
may
lead
to
allocation
of
additional
resources
and
a
revised
schedule
for
the
completion
of
the
work
product.

4.2.2
What
Actions
are
Potentially
Forthcoming
from
Peer
Review?

Peer
review
comments
and
recommendations
may
entail
significant
impacts
on
the
planned
project
schedule,
budget,
or
other
resource
requirements.
Management
decisions
related
to
revisions
in
one
or
more
of
these
areas
may
be
appropriate.

The
substantive
issues
or
concerns
expressed
by
peer
reviewers
may
suggest
that
wider
scientific
and
technical
consultation
is
needed
to
ensure
the
adequacy
of
the
work
product.

The
peer
review
comments
and
recommendations
on
a
final
product
may
provide
a
basis
for
bringing
the
associated
project
to
closure.

4.2.3
Can
the
Identity
of
Peer
Reviewers
be
Kept
Anonymous?

If
the
matter
has
gone
to
litigation,
the
litigating
parties
can
discover
the
names
of
anyone
who
contributed
to
a
Federal
product,
including
peer
reviewers.
Therefore,
it
is
not
possible
to
totally
shield
peer
reviewers.
In
addition,
it
may
be
difficult
to
shield
the
names
of
the
peer
reviewers
when
the
Agency
is
responding
to
a
Freedom
of
Information
request.
However,
in
the
ordinary
course
of
events,
you
can
often
offer
comments
received
without
attributing
the
comments
to
a
specific
reviewer.

If
a
peer
reviewer
requests
anonymity
at
the
outset
of
the
peer
review,
the
Peer
Review
Leader
needs
to
inform
the
peer
reviewer
of
the
above
possible
eventualities.
The
Agency
will
in
the
ordinary
course
of
events
attempt
to
maintain
the
confidentiality
of
the
peer
reviewers
and
their
comments
for
public
consumption,
but
it
is
recognized
in
many
instances,
for
example
open
public
meetings
and
the
above
circumstances,
this
can't
be
assured.
Remember,
the
Agency
is
committed
to
working
"
as
if
in
a
fishbowl"
and
most
of
its
activities
are
transparent
to
the
public
(
except
where
confidential
business
information
is
concerned).
It
is
recognized
that
this
may
be
a
deterrent
to
possible
peer
reviewers,
but
this
is
a
reality
that
has
to
be
understood.
Peer
Review
Handbook
Page
63
4.3
Completing
the
Peer
Review
Record
4.3.1
How
Do
I
Complete
the
Peer
Review
Record?

Once
the
Peer
Review
Leader
has
completed
the
peer
review,
the
peer
review
record
is
brought
up
to
date
and
then
archived
according
to
that
organization's
procedure
(
see
section
2.5.7).
The
peer
review
record
must
be
indexed
and
maintained
in
an
organization's
archive
(
repository).
The
location
of
the
peer
review
record
needs
to
be
readily
identifiable
so
interested
parties
can
locate
and
obtain
materials
easily
and
quickly.
The
peer
review
record
should
be
placed
in
any
associated
established
public
docket,
if
required,
in
addition
to
the
organizational
archive.
As
a
courtesy,
a
copy
of
the
revised
work
product
may
be
sent
to
the
peer
reviewers
for
information.

The
Peer
Review
Leader
will
collect
the
following
materials
for
the
peer
review
record
and
submit
for
archiving;
including
at
least
(
see
also
section
2.5.3):

a)
The
draft
work
product
submitted
for
peer
review
b)
Materials
and
information
(
including
the
charge)
given
to
the
peer
reviewers
c)
Written
comments,
information,
and
materials
received
from
the
peer
reviewers
d)
Information
about
the
peer
reviewers
(
such
as
reviewers'
names,
affiliations,
and
identified
potential
conflicts
and
their
resolution)

e)
Logistical
information
about
conduct
of
the
peer
review
(
such
as
times
and
locations
of
meetings)

f)
A
memorandum
or
other
written
record
responding
to
the
peer
review
comments
specifying
acceptance
or,
where
thought
appropriate,
rebuttal
and
non­
acceptance
g)
The
final
work
product
h)
Peer
review
summary
report
(
see
sections
1.3.1
and
1.3.2)

4.3.2
Where
Should
the
Peer
Review
Records
be
Kept,
and
for
How
Long?

During
the
active
conduct
of
the
peer
review,
the
Peer
Review
Leaders
maintain
the
peer
review
record
with
themselves
until
the
peer
review
is
totally
completed.
Minimally,
the
file
Peer
Review
Handbook
Page
64
should
be
maintained
until
one
year
after
the
completed
peer
review
is
reported
in
the
next
annual
reporting.
After
that,
the
peer
review
record
should
be
maintained
for
a
"
reasonable
period
of
time."
Establishment
and
maintenance
of
the
archive
where
the
peer
review
records
ultimately
reside
are
an
organization's
responsibility
(
i.
e.,
not
that
of
an
individual
program
manager
or
Peer
Review
Leader).
Generally,
to
allow
flexibility,
individual
offices
and
regions
will
decide
the
appropriate
level
of
organizational
responsibility
and
how
they
will
meet
the
"
routinely
available"
requirement.
The
peer
review
record
may
be
kept
with
other
records
relating
to
the
overall
project,
as
long
as
it
is
easily
and
separately
identifiable.
The
peer
review
record
should
be
maintained
in
accordance
with
the
Agency's
record
keeping
schedule
for
such
records.
One
long
term
archiving
mechanism
may
be
the
formal
archiving
at
the
Federal
Records
Center
in
Suitland,
MD.
(
Note:
This
is
the
same
question
as
section
2.5.7,
but
applies
in
this
chapter
as
well).
Peer
Review
Handbook
Page
65
SUBJECT
INDEX
This
is
an
alphabetical
listing
of
subjects
from
the
Handbook
and
the
pertinent
page
numbers
where
they
are
found.

­
A
­
Administrative
Procedures
Act
(
12)
Designated
Agency
Ethics
Official
(
48)
Administrator
(
9,
17)
Division
Directors
(
17)
Advisory
and
Assistance
Services
(
53)
Docket
(
38,
39,
63)
Analytic
blueprints
(
13,
32,
40)
­
E
­
Annual
call
letter
(
15)
Economic
work
products
(
27)
Annual
reporting
requirements
(
14,
18)
Electronic
records
(
14,
16,
38)
Anonymity
of
peer
reviewers
(
62)
Environmental
Impact
Statement
(
EIS)
(
29)
Archiving
(
39,
63)
Environmental
regulatory
models
(
29)
Assistant
Administrators
(
17)
External
peer
review
(
33,
34,
46)
­
B
­
­
F
­
Balancing
of
peer
reviewers
(
21)
Federal
Acquisition
Regulation
(
FAR)
(
53)
Board
of
Scientific
Counselors
(
BOSC)
(
34)
Federal
Advisory
Committee
Act
(
FACA)
Budget
(
18,
32,
40)
(
41,
49)
­
C
­
Final
work
product
(
61)
Categories
of
work
products
(
15)
­
G
­
Charge
(
19,
37,
43)
Grants
(
28)
Chemical
action
reports
(
16)
Gratuitous
services
(
52)
Clean
Air
Scientific
Advisory
Committee
­
I
­
(
34)
Independent
peer
reviewer
(
10,
12,
21,
45)
Coding
system
(
16)
Inherently
governmental
functions
(
IGFs)
Compensation
(
52)
(
54)
Competition
(
57)
Interacting
with
peer
reviewers
(
51)
Completing
a
peer
review
(
4,
61)
Interagency
agreement
(
28)
Conducting
a
peer
review
(
3,
43)
Internal
peer
review
(
33)
Conference
proceedings
(
25)
Internal
peer
reviewer
(
21)
Confidential
business
information
(
CBI)
(
49,
­
J
­
56)
Journal
article
(
16,
35)
Confidential
Financial
Disclosure
(
47)
­
L
­
Confidentiality
(
51)
Legal
considerations
(
40)
Conflicts
of
interest
(
21,
28,
47,
54)
Line­
managers
(
17)
Contracting
officer
(
50,
52,
54,
56)
List
A
(
14)
Contracts
(
28,
55)
List
B
(
15,
26)
Contracts
Management
Manual
(
CMM)
(
52,
List
C
(
15)
53)
Litigation
(
15,
62)
Controversy
(
27,
32)
­
M
­
Cooperative
agreements
(
28)
Major
impact
(
27,
32)
Court
ordered
deadlines
(
31,
36,
49)
Major
scientific
and
technical
work
product
­
D
­
(
5,
9,
15,
25,
26,
30)
Decision­
Maker
(
14,
16,
17,
26,
32)
Managers
planning
checklist
for
peer
review
Deputy
Administrator
(
14,
17,
18)
(
5)
Peer
Review
Handbook
Page
66
Materials
for
peer
reviewers
(
49)
Scientific
and
technical
work
product
(
25)
Mechanisms
for
peer
review
(
32)
Selection
of
peer
reviewers
(
45)
­
N
­
Sensitive
activities
(
53)
National
Academy
of
Sciences
(
34)
Signatures
(
14,
16)
NEPA
Products
(
29)
Simplified
acquisitions
of
peer
review
Non­
major
scientific
and
technical
work
services
(
55,
57)
products
(
15,
30,
31)
Site
specific
decisions
(
28)
Novelty
(
27,
32)
Special
Government
Employee
(
SGE)
(
59)
NPDES
permits
(
16)
Stakeholder
involvement
(
12)
­
O
­
Statement
of
work
for
contracts
(
53)
Office
Directors
(
17)
Subject
matter
expert
(
22)
Office
of
Research
and
Development
(
ORD)
­
T
­
(
14,
16,
20,
23)
Tier
1
and
Tier
2
(
13,
44)
Open
meeting
(
41)
Tier
3
(
13)
­
P
­
Time
frame
for
peer
review
(
36,
44)
Peer
consultation
(
11)
Travel
(
58)
Peer
input
(
11)
­
V
­
Peer
involvement
(
10)
Voluntary
services
(
52)
Peer
review
(
9,
10)
Peer
Review
Advisory
Group
(
PRAG)
(
17)
Peer
review
comments
(
19,
38,
61)
Peer
Review
Coordinator
(
14,
17,
19,
22)
Peer
Review
Leader
(
18,
18,
32,
39,
48,
49,
61,
63)
Peer
review
mechanism
(
5,
32)
Peer
review
panel
(
22,
45,
46)
Peer
review
record
(
5,
20,
36,
37,
63)
Peer
review
schedule
(
18)
Peer
review
services
(
52)
Peer
review
summary
report
(
14,
15,
18,
38)
Peer
reviewers
(
21,
21,
38,
49,
56)
Personal
services
(
56)
Planning
a
peer
review
(
2,
25)
Premanufacturing
notices
(
PMNs)
(
16)
Procurement
requests
(
57)
Project
manager
(
13,
18,
30)
Public
comment
(
12)
­
R
­
RCRA
permits
(
16)
Record
of
Decision
(
ROD)
(
25)
Regional
Administrators
(
17)
Regulatory
negotiations
(
14)
Rulemaking
(
13,
27,
38)
­
S
­
Science
Advisory
Board
(
SAB)
(
34,
45)
Science
Policy
Council
(
SPC)
(
14,
17,
20,
22)
Scientific
Advisory
Panel
(
SAP)
(
34,
45)
Peer
Review
Handbook
Page
67
COMMONLY
USED
ACRONYMS
AA
Assistant
Administrator
AAS
Advisory
and
Assistance
Services
BOSC
Board
of
Scientific
Counselors
CBI
Confidential
Business
Information
CFR
Code
of
Federal
Regulations
CMM
Contracts
Management
Manual
CO
Contract
Officer
COI
Conflict
of
Interest
EIS
Environmental
Impact
Statement
EPA
Environmental
Protection
Agency
EPAAR
EPA
Acquisition
Regulations
FACA
Federal
Advisory
Committee
Act
FAR
Federal
Acquisition
Regulations
FOIA
Freedom
of
Information
Act
FTE
Full
Time
Equivalent
GSA
General
Services
Administration
HHS
Health
and
Human
Services
IGF
Inherently
Governmental
Function
IRIS
Integrated
Risk
Information
System
LAN
Local
Area
Network
NEPA
National
Environmental
Policy
Act
NPDES
National
Pollutant
Discharge
Elimination
System
OAM
Office
of
Acquisition
Management
OGC
Office
of
General
Counsel
OMB
Office
of
Management
and
Budget
ORD
Office
of
Research
and
Development
PMNs
Premanufacture
Notice
PRAG
Peer
Review
Advisory
Group
RA
Regional
Administrator
RC
Regional
Counsel
RCRA
Resource
Conservation
and
Recovery
Act
RIAs
Regulatory
Impact
Analyses
ROD
Record
of
Decision
SAB
Science
Advisory
Board
SAP
Scientific
Advisory
Panel
SGE
Special
Government
Employee
SOPs
Standard
Operating
Procedures
SOW
Statement
of
Work
SPC
Science
Policy
Council
WTI
Waste
Technologies
Industries
Peer
Review
Handbook
Page
68
THIS
PAGE
LEFT
INTENTIONALLY
BLANK
Peer
Review
Handbook
Page
A­
1
APPENDIX
A
U.
S.
Environmental
Protection
Agency
Peer
Review
Policy
June
7,
1994
Peer
Review
Handbook
Page
A­
2
June
7,
1994
MEMORANDUM
SUBJECT:
Peer
Review
Program
TO:
Assistant
Administrators
General
Counsel
Inspector
General
Associate
Administrators
Regional
Administrators
Staff
Office
Directors
Today,
I
am
reaffirming
the
central
role
of
peer
review
in
our
efforts
to
ensure
that
EPA
policy
decisions
rest
on
sound,
credible
science
and
data
(
see
attached
policy
statement).
Toward
that
end,
as
its
first
major
task,
EPA's
Science
Policy
Council
(
SPC)
is
instituting
a
program
to
expand
and
improve
peer
review
in
all
EPA
offices.
This
memorandum
gives
an
overview
of
current
practices
and
outlines
the
new
program.

Peer
Review
Practices
and
Policy
Peer
review
at
EPA
takes
several
different
forms,
ranging
from
informal
consultations
with
Agency
colleagues
who
were
not
involved
in
developing
the
product
to
the
formal,
public
processes
of
the
Science
Advisory
Board
(
SAB)
and
the
FIFRA
Scientific
Advisory
Panel
(
SAP).
In
any
form,
peer
review
assists
the
Agency's
work
by
bringing
independent
expert
experience
and
judgment
to
bear
on
issues
before
the
Agency
to
the
benefit
of
the
final
product.

EPA's
Peer
Review
Policy,
which
responds
in
part
to
recommendations
in
the
"
Credible
Science,
Credible
Decisions"
report,
outlines
general
principles
for
peer
review
at
EPA.
Different
EPA
offices
have
undertaken
various
implementing
activities,
including
an
Agencywide
information
and
planning
workshop,
internal
guideline
development,
and
numerous
specific
peer
reviews.
Even
with
these
activities,
however,
I
am
concerned
that
EPA
does
not
yet
have
a
comprehensive
Agency­
wide
program
for
implementing
its
Peer
Review
Policy.
I
therefore
welcome
the
SPC
initiative
toward
effective,
efficient
implementation
of
the
policy
in
all
the
program
areas
to
which
it
applies.
Peer
Review
Handbook
Page
A­
3
Expanding
and
Improving
Peer
Review
The
Science
Policy
Council
and
its
Steering
Committee
have
outlined
a
dual­
track
implementation
program
of
planning
and
assistance
for
all
Agency
offices.
The
first
track
has
three
major
milestones.

First,
during
the
next
few
weeks,
Steering
Committee
members
will
consult
with
senior
management
in
each
office
to
exchange
information
on
current
peer
review
activities,
assistance
needed,
possible
obstacles
to
implementation,
and
implementation
planning.

Second,
using
information
and
materials
developed
during
the
first
stage,
peer
review
task
groups
in
each
office
will
develop
standard
operating
procedures
(
SOPs)
for
use
in
each
office,
based
in
part
on
generic
guidance
to
be
issued
by
the
SPC
and
in
part
on
peer
review
needs
and
capabilities
specific
to
each
office.
The
resulting
SOPs
will
delineate
as
appropriate
the
scope
of
application
of
peer
review
with
respect
to
various
types
of
scientific
and
technical
work
products
such
as
reports
of
original
research,
risk
assessments,
and
analytical
methods
of
economic
analysis.
OARM
and
OGC
staff
will
assist
each
office
as
needed
on
legal,
budget
and
administrative
matters.
Each
AA
and
RA
will
submit
draft
SOPs
for
Steering
Committee
review
by
July
15.

Third,
the
SPC
review
group
will
work
with
each
office
to
complete
each
plan
by
September
15.

In
parallel
with
the
above,
consistent
with
the
Peer
Review
Policy,
the
Science
Policy
Council
will
work
with
each
AA
and
RA
to
identify
"
major
scientific
and
technical
work
products"
as
peer
review
candidates
for
the
coming
year.
This
process
will
consider
existing
and
new
plans
for
internal
reviews
and
for
SAB,
FIFRA
SAP,
and
other
external
reviews.
The
two­
fold
objective
is
to
plan
reviews
for
technical
products
covered
by
the
Peer
Review
Policy
and
to
gain
experience
with
options
and
obstacles.
We
will
use
this
experience
to
review
and
revise
the
SOPs
as
needed.
Also,
to
establish
a
baseline
for
comparison,
each
AA
and
RA
will
identify
the
"
major
technical
products"
completed
within
his/
her
program
during
the
past
12
months.

The
Science
Policy
Council
has
sent
additional
information
to
each
office
offering
guidance
on
the
procedures
that
you
are
asked
to
develop
and
the
schedule
for
these
activities.
Please
note,
however,
that
because
the
policy
is
effective
immediately,
current
peer
review
planning
should
continue
on
present
schedules
in
parallel
with
developing
the
formal
SOPs.
Peer
Review
Handbook
Page
A­
4
To
begin
this
process,
I
have
asked
each
Assistant
Administrator
and
Regional
Administrator
to
designate
a
Peer
Review
Coordinator
to
work
with
the
Steering
Committee
on
implementation
activities
specific
to
each
office.
I
am
very
pleased
that
the
Science
Policy
Council
is
taking
this
important
step.
A
comprehensive
peer
review
program
is
essential
to
maintaining
and
improving
the
quality
of
the
analyses
that
underlie
Agency
actions.
I
look
forward
to
working
with
you
and
your
staff
on
this
important
activity.

/
s/

Carol
M.
Browner
Attachment
cc:
Science
Policy
Council
Science
Policy
Council
Steering
Committee
Peer
Review
Handbook
Page
A­
5
PEER
REVIEW
AND
PEER
INVOLVEMENT
AT
THE
U.
S.
ENVIRONMENTAL
PROTECTION
AGENCY
This
document
establishes
the
policy
of
the
United
States
Environmental
Protection
Agency
(
EPA)
for
peer
review
of
scientifically
and
technically
based
work
products
that
are
intended
to
support
Agency
decisions.
Peer
review
is
presented
in
the
context
of
the
broader
concept,
peer
involvement.

BACKGROUND
The
report
"
Safeguarding
the
Future:
Credible
Science,
Credible
Decisions"
focused
on
1
the
state
of
science
at
EPA.
The
panel
of
experts
who
prepared
the
report
emphasized
the
importance
of
peer
review,
especially
external
peer
review,
and
the
need
for
broader
and
more
systematic
use
of
it
at
EPA
to
evaluate
scientific
and
technical
work
products.
Their
specific
recommendation
regarding
peer
review
reads
as
follows:

"
Quality
assurance
and
peer
review
should
be
applied
to
the
planning
and
results
of
all
scientific
and
technical
efforts
to
obtain
data
used
for
guidance
and
decisions
at
EPA,
including
such
efforts
in
the
program
and
regional
offices.
Such
a
requirement
is
essential
if
EPA
is
to
be
perceived
as
a
credible,
unbiased
source
of
environmental
and
health
information,
both
in
the
United
States
and
throughout
the
world."

In
response
to
this
recommendation,
then­
Administrator
Reilly
directed
staff
to
develop
an
EPAwide
policy
statement,
which
he
issued
in
January,
1993.
The
paragraphs
below
preserve
the
core
of
that
earlier
statement
while
updating
it
to
specify
the
role
of
the
Science
Policy
Council
in
guiding
further
implementation
of
the
policy.
Effective
use
of
peer
review
is
indispensable
for
fulfilling
the
EPA
mission
and
therefore
deserves
high­
priority
attention
from
program
managers
and
scientists
within
all
pertinent
Headquarters
and
Regional
Offices.

EPA/
600/
9­
91/
050,
March
1992.
1
Peer
Review
Handbook
Page
A­
6
PEER
INVOLVEMENT
AND
PEER
REVIEW
EPA
strives
to
ensure
that
the
scientific
and
technical
underpinnings
of
its
decisions
meet
two
important
criteria:
they
should
be
based
upon
the
best
current
knowledge
from
science,
engineering,
and
other
domains
of
technical
expertise;
and
they
should
be
judged
credible
by
those
who
deal
with
the
Agency.
EPA
staff
therefore
frequently
rely
upon
peer
involvement
­­
that
is,
they
augment
their
capabilities
by
inviting
relevant
subject­
matter
experts
from
outside
the
program
to
become
involved
in
one
or
more
aspects
of
the
development
of
the
work
products
that
support
policies
and
actions.

One
particularly
important
type
of
peer
involvement
occurs
when
scientifically
and
technically
based
work
products
undergo
peer
review
­­
that
is,
when
they
are
evaluated
by
relevant
experts
from
outside
the
program
who
are
peers
of
the
program
staff,
consultants,
and/
or
contractor
personnel
who
prepared
the
product.
Properly
applied,
peer
review
not
only
enriches
the
quality
of
work
products
but
also
adds
a
degree
of
credibility
that
cannot
be
achieved
in
any
other
way.
Further,
peer
review
early
in
the
development
of
work
products
in
some
cases
may
conserve
future
resources
by
steering
the
development
along
the
most
efficacious
course.

Peer
review
generally
takes
one
of
two
forms.
The
review
team
may
consist
primarily
of
relevant
experts
from
within
EPA,
albeit
individuals
who
have
no
other
involvement
with
respect
to
the
work
product
that
is
to
be
evaluated
(
internal
peer
review).
Or
the
review
team
may
consist
primarily
of
independent
experts
from
outside
EPA
(
external
peer
review).

POLICY
STATEMENT
Major
scientifically
and
technically
based
work
products
related
to
Agency
decisions
normally
should
be
peer­
reviewed.
Agency
managers
within
Headquarters,
Regions,
laboratories,
and
field
components
determine
and
are
accountable
for
the
decision
whether
to
employ
peer
review
in
particular
instances
and,
if
so,
its
character,
scope,
and
timing.
These
decisions
are
made
in
conformance
with
program
goals
and
priorities,
resource
constraints,
and
statutory
or
court­
ordered
deadlines.
For
those
work
products
that
are
intended
to
support
the
most
important
decisions
or
that
have
special
importance
in
their
own
right,
external
peer
review
is
the
procedure
of
choice.
Peer
review
is
not
restricted
to
the
penultimate
version
of
work
products;
in
fact,
peer
review
at
the
planning
stage
can
often
be
extremely
beneficial.

SCOPE
Agency
managers
routinely
make
regulatory
and
other
decisions
that
necessarily
involve
many
different
considerations.
This
policy
applies
to
major
work
products
that
are
primarily
scientific
and
technical
in
nature
and
may
contribute
to
the
basis
for
policy
or
regulatory
decisions.
By
contrast,
this
policy
does
not
apply
to
non­
major
or
nontechnical
matters
that
Peer
Review
Handbook
Page
A­
7
Agency
managers
consider
as
they
make
decisions.
Similarly,
this
policy
does
not
apply
to
these
ultimate
decisions.

This
policy
applies
where
appropriate,
as
determined
by
the
National
and
Regional
Program
Managers,
to
major
scientifically
and
technically
based
work
products
initiated
subsequent
to
the
date
of
issuance.
Peer
review
should
be
employed
to
the
extent
reasonable
to
relevant
work
products
that
currently
are
under
development.
This
policy
does
not
apply
to
the
bases
for
past
decisions,
unless
and
until
the
relevant
scientific
and
technical
issues
are
considered
anew
in
the
Agency's
decision­
making
processes.

Except
where
it
is
required
by
law,
formal
peer
review
(
as
distinguished
from
the
Agency's
normal
internal
review
procedures)
should
be
conducted
in
a
manner
that
will
not
cause
EPA
to
miss
or
need
extension
of
a
statutory
or
court­
ordered
deadline.
Agency
managers
still
may
undertake
peer
review
if
it
can
be
conducted
concurrently
with
necessary
rulemaking
steps.

LEGAL
EFFECT
This
policy
statement
does
not
establish
or
affect
legal
rights
or
obligations.
Rather,
it
confirms
the
importance
of
peer
review
where
appropriate,
outlines
relevant
principles,
and
identifies
factors
Agency
staff
should
consider
in
implementing
the
policy.
On
a
continuing
basis,
Agency
management
is
expected
to
evaluate
the
policy
as
well
as
the
results
of
its
application
throughout
the
Agency
and
undertake
revisions
as
necessary.
Therefore,
the
policy
does
not
stand
alone;
nor
does
it
establish
a
binding
norm
that
is
finally
determinative
of
the
issues
addressed.
Minor
variations
in
its
application
from
one
instance
to
another
are
appropriate
and
expected;
they
thus
are
not
a
legitimate
basis
for
delaying
or
complicating
action
on
otherwise
satisfactory
scientific,
technical,
and
regulatory
products.

Except
where
provided
otherwise
by
law,
peer
review
is
not
a
formal
part
of
or
substitute
for
notice
and
comment
rulemaking
or
adjudicative
procedures.
EPA's
decision
whether
to
conduct
peer
review
in
any
particular
case
is
wholly
within
the
Agency's
discretion.
Similarly,
nothing
in
this
policy
creates
a
legal
requirement
that
EPA
respond
to
peer
reviewers.
However,
to
the
extent
that
EPA
decisions
rely
on
scientific
and
technical
work
products
that
have
been
subjected
to
peer
review,
the
remarks
of
peer
reviewers
should
be
included
in
the
record
for
that
decision.

IMPLEMENTATION
The
Science
Policy
Council
is
responsible
for
overseeing
Agency­
wide
implementation.
Its
responsibilities
include
promoting
consistent
interpretation,
assessing
Agency­
wide
progress,
and
developing
recommendations
for
revisions
of
the
policy
as
necessary.
Peer
Review
Handbook
Page
A­
8
The
Science
Policy
Council
will
oversee
a
peer­
review
work
group,
which
will
include
representatives
from
program
units
throughout
EPA
to
effect
a
consistent,
workable
implementation
of
the
policy.
The
work
group
will
assist
the
programs
in
(
1)
formulating
and,
as
necessary,
revising
standard
operating
procedures
(
SOPs)
for
peer
review
consistent
with
this
policy;
(
2)
identifying
work
products
that
are
subject
to
review;
and
(
3)
for
each
major
work
product,
selecting
an
appropriate
level
and
timing
of
peer
review.

In
assisting
the
programs,
the
work
group
will
take
into
account
statutory
and
court
deadlines,
resource
implications,
and
availability
of
disinterested
peer
reviewers.
The
group
will
work
closely
with
Headquarters
offices
and
the
Regional
Offices
toward
ensuring
effective,
efficient
uses
of
peer
review
in
supporting
their
mission
objectives.
However,
the
Assistant
Administrators
and
Regional
Administrators
remain
ultimately
responsible
for
developing
SOPs,
identifying
work
products
subject
to
peer
review,
determining
the
type
and
timing
of
such
review,
documenting
the
process
and
outcome
of
each
peer
review,
and
otherwise
implementing
the
policy
within
their
organizational
units.

Because
peer
review
can
be
time­
consuming
and
expensive,
Agency
managers
within
Headquarters,
Regions,
laboratories,
and
field
components
are
expected
to
plan
carefully
with
respect
to
its
use
­­
taking
account
of
program
priorities,
resource
considerations,
and
any
other
relevant
constraints
as
well
as
the
policy
goal
of
achieving
high­
quality,
credible
underpinnings
for
decisions.
External
peer
reviewers
should
be
chosen
carefully
to
ensure
an
independent
and
objective
evaluation.
The
affiliations
of
peer
reviewers
should
be
identified
on
the
public
record,
so
as
to
avoid
undercutting
the
credibility
of
the
peer­
review
process
by
conflicts
of
interest.

The
policy
is
effective
immediately.
The
peer­
review
work
group
mentioned
above
will
identify
the
focal
point
to
whom
comments
and
questions
should
be
addressed
and,
from
time
to
time,
will
provide
further
information
about
implementation
activities.

/
s/
APPROVED:
DATE:
JUN
7
1994
CAROL
M.
BROWNER,
ADMINISTRATOR
Peer
Review
Handbook
Page
B­
1
APPENDIX
B
Examples
of
Charges
Please
note
­­
certain
questions
that
are
posed
in
charges
can
be
responded
to
with
a
yes
or
no
answer.
Clearly,
this
is
not
the
type
of
response
we
generally
want,
therefore,
it
is
important
to
phrase
charge
questions
carefully
to
ensure
that
you
receive
a
fully
satisfactory
and
thoughtful
response.
Where
a
yes
or
no
answer
might
be
expected,
be
sure
to
ask
for
a
full
explanation
supporting
the
yes
or
no
answer.
Peer
Review
Handbook
Page
B­
2
CHARGE
EXAMPLE
1
­
CHARGE
TO
REVIEWERS
FOR
THE
WTI
DRAFT
FINAL
RISK
ASSESSMENT
The
draft
final
WTI
risk
assessment
is
divided
into
several
volumes
covering
the
scientific
disciplines
of
toxicology,
environmental
fate
and
transport,
combustion
engineering,
atmospheric
modeling,
exposure
assessment,
ecological
risk
assessment,
and
accident
analysis.
As
a
reviewer
of
the
WTI
draft
final
risk
assessment,
you
should
use
your
best
technical
knowledge
and
professional
judgment
to
comment
on
the
technical
accuracy,
completeness
and
scientific
soundness
of
the
assessment.
Each
reviewer
is
asked
to
focus
on
several
specific
issues
in
his
or
her
area
of
expertise
with
comments
on
other
areas
invited
but
optional.
Your
comments
will
be
considered
in
finalizing
the
risk
assessment.

For
the
peer
review
workshop
reviewers
will
be
organized
into
5
work
groups:
Combustion
Engineering,
Air
Dispersion
and
Deposition
Modeling
and
Accident
Analysis,
Toxicology,
Exposure
Assessment,
and
Ecological
Risk
Assessment.
All
reviewers
should
be
familiar
with
the
Executive
Summary
(
Volume
I)
and
the
Facility
Background
(
Volume
II)
sections
of
the
draft
risk
assessment.
In
addition,
each
work
group
should
focus
on
specific
Volumes
as
specified
below:

Workgroup
Risk
Assessment
Volumes
Combustion
Engineering
Volume
III
­
Facility
Emissions
Air
Dispersion
and
Volume
IV
­
Atmospheric
Volume
VII
­
Accident
Deposition
Modeling
and
Dispersion
and
Deposition
Analysis
Accident
Analysis
Modeling
Toxicology
Volume
V
­
Human
Health
Volume
VII
­
Accident
Risk
Assessment
Analysis
Exposure
Assessment
Volume
V
­
Human
Health
Volume
VII
­
Accident
Risk
Assessment
Analysis
Ecological
Risk
Assessment
Volume
VI
­
Screening
Ecological
Risk
Assessment
Peer
Review
Handbook
Page
B­
3
While
reviewing
these
sections
of
the
document,
please
address
the
following
general
issues.

1)
Comment
on
the
organization
of
the
risk
assessment
document.
Does
the
layout
follow
a
logical
format?
Is
the
presentation
of
information
in
the
document
clear,
concise
and
easy
to
follow?

2)
Does
the
executive
summary
accurately
reflect
the
data
and
methodologies
used
and
the
conclusions
derived
in
the
risk
assessment?

3)
Were
the
major
recommendations
of
the
1993
peer
review
workshop
for
the
risk
assessment
plan
addressed?

4)
As
with
any
risk
assessment,
there
are
always
additional
data
and
method
development
efforts
that
could
be
undertaken
to
reduce
the
level
of
uncertainty.
However,
are
there
any
major
data
or
methodological
gaps
that
would
preclude
the
use
of
this
risk
assessment
for
decision
making?
If
so,
how
should
they
be
addressed?

5)
What
long­
term
research
would
you
recommend
that
could
improve
risk
assessments
of
this
type
in
the
future?

In
addition,
the
following
workgroup
specific
issues
should
be
addressed.

Emissions
Characterization
Emissions
characterization
includes
identification
of
substances
of
concern
and
the
development
of
emission
rates
for
these
contaminants.
Emission
rates
were
developed
through
a
combination
of
site
specific
stack
test
data
and
models.
Please
comment
on
the
following
issues
with
respect
to
this
aspect
of
the
draft
risk
assessment.

1)
To
characterize
the
nature
of
the
emissions,
waste
stream
profiles
were
developed
and
entered
into
a
database.
Several
refinements
and
adjustments
(
e.
g.,
the
Subtraction
Correction
Factor
for
chlorinated
compounds)
were
applied
to
the
profiles
before
substances
of
concern
were
identified.
Please
comment
on
whether
or
not
these
adjustments
are
appropriate.
What
is
the
anticipated
effect
on
the
risk
assessment?

2)
Comment
on
the
selected
chemicals
of
concern.
Have
important
chemicals
been
missed
due
to
the
selection
technique?
Peer
Review
Handbook
Page
B­
4
3)
Comment
on
the
approaches
used
to
estimate
stack
emission
rates
(
e.
g.,
use
of
the
95%
UCL
of
the
arithmetic
mean
or
the
maximum
detected
value,
whichever
is
smaller,
for
high
end
emission
rates).
Are
the
approaches
appropriate?
Are
their
effects
on
the
risk
assessment
adequately
characterized?
Comment
on
the
adjustment
made
to
PCDD/
PCDF
emission
rates
to
account
for
brominated
dioxin­
like
compounds.
Also,
comment
on
the
approach
to
characterizing
emission
rates
from
fugitive
sources
(
e.
g.,
use
of
the
TANKS
2
model
for
the
Carbon
Adsorption
Bed).

4)
Comment
on
the
identified
sources
of
fugitive
emissions.
Was
the
approach
used
to
select
these
sources
appropriate?
Have
important
sources
been
missed?
Have
emissions
from
process
upsets
been
given
appropriate
consideration?

5)
There
have
been
a
number
of
controlled
burns
at
the
WTI
facility.
Please
comment
on
the
adequacy
of
these
data
in
estimating
potential
exposure.
Please
comment
on
the
assumptions
made
from
the
tests
in
regard
to
composition
of
wastes
received
at
WTI
and
emissions
when
the
plant
operates
in
the
future.

6)
Comment
on
the
use
of
emission
factors
from
coal
burning
to
estimate
the
emission
rate
of
fly
ash
from
WTI.
Are
the
factors
used
to
adjust
the
coal
emission
rate
appropriate?
Are
the
uncertainties
introduced
from
this
approach
adequately
characterized?

7)
Overall,
is
the
identification
of
the
key
assumptions
used
in
characterizing
the
nature
and
magnitude
of
emissions
thorough?
Are
the
magnitude
and
direction
of
effect
of
these
assumptions
on
the
overall
risk
assessment
accurately
characterized?
Is
the
uncertainty
and
variability
inherent
in
this
analysis
adequately
discussed?
Does
the
sensitivity
analysis
cover
the
major
parameters
expected
to
have
an
effect
on
the
risk
assessment?

Dispersion
and
Deposition
Modeling
To
develop
this
risk
assessment,
computer
models
have
been
used
with
site
specific
data
on
emission
rates
and
meteorological
conditions
to
simulate
the
air
concentrations
and
deposition
rates
for
contaminants
potentially
emitted
from
the
WTI
facility.
The
models
used
include
the
Industrial
Source
Complex
­
Complex
Terrain
Deposition
(
ISC­
COMPDEP),
the
CALPUFF,
and
the
INPUFF
models.
In
your
review,
please
address
the
following
issues.

1)
Since
the
1993
peer
review
of
the
risk
assessment
plan,
a
number
of
efforts
have
been
completed
to
reduce
the
uncertainty
associated
with
the
air
dispersion
and
deposition
modeling.
These
efforts
include
the
collection
of
site­
specific
data
for
Peer
Review
Handbook
Page
B­
5
emission
rates
and
meteorological
conditions.
Also,
a
wind
tunnel
study
was
conducted
to
evaluate
the
effects
of
the
complex
terrain
surrounding
the
WTI
facility.
Does
the
risk
assessment
document
adequately
summarize
these
activities?
Is
the
link
between
these
data
collection
efforts,
the
air
dispersion
models,
and
the
risk
assessment
clearly
established?

2)
The
results
of
12
sets
of
sensitivity
tests
indicate
that
geophysical
variables
(
e.
g.,
terrain)
are
more
likely
to
affect
dispersion
and
deposition
than
emission
variables
(
e.
g.,
stack
temperature).
Were
these
sensitivity
analyses
adequate?
Comment
on
the
conclusions
reached.
To
further
examine
the
effect
of
geophysical
variables,
wind
tunnel
testing
was
conducted
to
model
the
terrain
induced
flow
effects
expected
near
WTI.
It
was
concluded
that
changes
in
peak
concentrations
attributed
to
these
effects
are
relatively
minor
and
that
the
ISC­
COMPDEP
model
is
sufficiently
conservative.
Comment
on
this
conclusion.
Have
these
analyses
helped
to
characterize
and/
or
reduce
the
uncertainty
in
the
air
dispersion
modeling
associated
with
the
complex
terrain
surrounding
WTI.

3)
The
ISC­
COMPDEP
model
does
not
allow
for
non­
steady
state
conditions
such
as
calm
winds
and
strong
temperature
inversions.
Therefore,
CALPUFF
was
used
to
estimate
air
dispersion
and
deposition
under
these
conditions.
However,
CALPUFF
gave
similar
peak,
24
hour,
and
annual
average
concentrations
as
ISCCOMPDEP
Comment
on
the
adequacy
of
this
analysis.
Comment
on
the
conclusions
reached.
Has
this
analysis
helped
to
characterize
and/
or
reduce
the
uncertainty
in
the
air
dispersion
modeling
associated
with
non­
steady
state
meteorological
conditions?

4)
Atmospheric
dispersion
modeling
was
used
to
estimate
air
concentrations
of
hazardous
chemicals
for
the
accident
analysis.
The
SLAB
model
was
used
for
vapor
releases
from
spills
and
the
mixing
of
incompatible
wastes.
ISCCOMPDEP
was
used
for
releases
associated
with
fires.
Comment
on
the
selection
of
the
models
and
inputs.
Are
they
appropriate
selections?

5)
Overall,
have
adequate
sensitivity
tests
been
conducted
to
demonstrate
the
magnitude
of
variation
in
concentrations
and
deposition
estimates
with
model
inputs?
Please
explain
fully.

Human
Health
Risks
Human
Health
Risk
Assessment
includes
hazard
identification,
dose­
response
evaluation,
exposure
assessment,
and
risk
characterization.
To
develop
the
risk
assessment,
potentially
exposed
populations
have
been
identified
and
the
magnitude,
frequency,
and
duration
of
their
Peer
Review
Handbook
Page
B­
6
exposure
quantified.
This
information
was
then
integrated
with
the
hazard
identification
and
dose
response
evaluation
for
the
risk
characterization.
For
this
risk
assessment,
both
carcinogenic
and
non­
carcinogenic
health
effects
have
been
evaluated.
In
your
review,
please
comment
on
the
following
issues.

Exposure
1)
EPA's
Exposure
Assessment
Guidelines
identify
certain
exposure
descriptors
that
should
be
used
to
characterize
exposure
estimates.
The
Guidelines
define
high
end
exposure
estimates
as
those
representing
individuals
above
the
90th
percentile
on
the
exposure
distribution
but
not
higher
than
the
individual
in
the
population
who
has
the
highest
exposure.
Bounding
exposure
estimates
are
those
that
are
higher
than
the
exposure
incurred
by
the
person
in
the
population
with
the
highest
exposure.
Central
tendency
exposure
estimates
are
defined
as
the
best
representation
of
the
center
of
the
exposure
distribution
(
e.
g.,
arithmetic
mean
for
normal
distributions).
Comment
on
whether
or
not
the
WTI
exposure
assessment
properly
characterizes
each
of
the
exposure
estimates
in
terms
of
these
descriptors.

2)
The
factors
that
go
into
estimating
a
central
tendency
or
high
end
exposure,
once
the
population
has
been
defined,
include
the
environmental
media
concentration,
the
intake
rate,
and
the
duration
and/
or
frequency
of
exposure.
Comment
on
whether
or
not
the
WTI
exposure
assessment
does
an
adequate
job
of
describing
the
logical
procedure
of
combining
these
factors
to
develop
central
tendency,
high
end,
and/
or
bounding
estimates
of
exposure
for
each
of
the
exposed
subpopulations.

3)
An
important
factor
in
an
exposure
assessment
is
identifying
all
of
the
important
exposure
sources.
Please
comment
on
the
adequacy
of
the
WTI
assessment
in
identifying
the
important
sources
and
pathways
of
exposure.

4)
Have
the
key
assumptions
for
estimation
of
chemical
concentration
and
for
estimation
of
exposure
been
identified?
Are
the
magnitude
and
direction
of
effect
correct
for
the
assumptions
that
have
been
identified?

5)
Supposedly,
conservative
assumptions
have
been
applied
in
this
assessment
to
account
for
uncertainty.
Are
the
conservative
assumptions
appropriately
factored
into
the
ultimate
characterization
of
what
descriptor
best
applies
to
each
exposure
estimate?
Please
comment
on
whether
the
uncertainties
were
confronted
in
an
adequate
manner.
If
they
were
not,
please
state
what
should
be
done
differently.
Peer
Review
Handbook
Page
B­
7
Hazard
Identification/
Dose
Response
and
Risk
Characterization
1)
To
select
surrogate
compounds
for
quantitative
risk
assessment,
a
two
step
process
was
used
in
which
chemicals
were
ranked
on
the
basis
of
emission
rate,
toxicity
(
both
cancer
and
non­
cancer),
and
bioaccumulation
potential.
Please
comment
on
this
selection
process.
Are
the
ranking
factors
appropriate?
Could
important
compounds
have
been
omitted
from
the
analysis
based
on
the
ranking
procedure?

2)
For
the
majority
of
the
chemicals
of
concern,
traditional
approaches
to
dose
response
evaluation
were
employed
(
e.
g.,
use
of
a
slope
factor
for
cancer
and
use
of
a
RfD/
RfC
for
non­
cancer).
However
for
certain
chemicals
or
groups
of
compounds
a
different
methodology
was
used.
Specifically,
dioxins,
furans,
PAHs,
lead,
mercury,
nickel,
chromium,
acid
gases,
and
particulate
matter
were
given
special
consideration.
Please
comment
on
the
methodology
used
for
these
compounds.
Was
it
appropriate?
Have
the
uncertainties
associated
with
the
methodology
been
adequately
characterized?
Comment
on
the
assumptions
used
due
to
a
lack
of
chemical
specific
data.

3)
Please
comment
on
the
selection
of
the
overall
population
and
the
various
subpopulations
at
risk.
Were
site
specific
data,
such
as
the
informal
home
gardening
survey,
properly
utilized
to
identify
these
subpopulations?

4)
It
is
stated
in
the
risk
assessment
that
average
risk
estimates
are
based
on
average
emission
rates,
average
air
dispersion/
deposition
within
a
subarea,
and
typical
exposure
factors.
Further,
maximum
risks
are
based
on
average
emission
rates,
typical
exposure
factors,
and
the
maximum
air
concentration
within
a
subarea.
Please
comment
on
this
use
of
the
terms
average
and
maximum
risks.
Are
these
descriptive
terms
appropriate
given
the
parameters
used
to
derive
each?
Please
explain
fully.

5)
Comment
on
whether
or
not
the
non­
cancer
risks
of
chemicals
of
concern
have
been
adequately
addressed
by
the
risk
assessment?
For
example,
has
an
adequate
discussion
of
endocrine
disrupters
been
provided
which
either
characterizes
their
risks
or
clearly
explains
why
their
risks
cannot
be
characterized?
Further,
have
non­
cancer
chronic
toxicities
of
dioxins
and
furans
been
adequately
addressed
in
the
risk
assessment?

6)
Please
comment
on
whether
or
not
the
uncertainties
associated
with
the
additivity
and/
or
synergy
of
risks
from
pollutants
emitted
together
from
the
WTI
facility
are
adequately
discussed
in
the
risk
assessment.
Peer
Review
Handbook
Page
B­
8
7)
Have
the
key
assumptions
for
estimation
of
dose
and
risk
been
identified?
Are
the
magnitude
and
direction
of
effect
correct
for
the
assumptions
that
have
been
identified?
Please
comment
on
whether
the
uncertainties
were
confronted
in
an
adequate
manner.
If
they
were
not,
please
state
what
should
be
done
differently.

8)
Please
comment
on
the
overall
adequacy
of
the
risk
characterization.
Does
the
risk
characterization
include
a
statement
of
confidence
in
the
risk
assessment
including
a
discussion
of
the
major
uncertainties.
Are
the
hazard
identification,
dose­
response
assessment,
and
exposure
assessment
clearly
presented?
Have
sufficient
risk
descriptors
which
include
important
subgroups
been
presented
and
discussed?

Screening
Ecological
Risk
Assessment
As
with
the
human
health
risk
assessment,
the
ecological
risk
assessment
pulls
together
elements
of
exposure
analysis
and
dose­
response
evaluations
to
develop
a
risk
characterization.
For
the
Screening
level
Ecological
Risk
Assessment
(
SERA),
Ecological
Chemicals
of
Concern
(
ECOC)
and
indicator
species
have
been
identified
to
provide
conservative
estimates
of
risk.
Please
address
the
following
issues
in
your
review.

1)
Are
there
any
components
of
the
SERA
which
you
feel
undermine
the
scientific
validity
of
the
assessment?
If
so,
what
are
they
and
can
you
provide
suggestions
to
strengthen
the
identified
components?

2)
Is
the
organization
of
the
document
clear
and
does
it
present
the
material
in
a
clear
and
concise
manner
consistent
with
the
Framework
for
Ecological
Risk
Assessment
(
EPA,
1992)?
Please
explain
fully.

3)
Uncertainties
are
discussed
in
numerous
sections
of
the
SERA
and
compose
Section
VIII
of
the
SERA.
In
each
case,
do
these
discussions
cover
all
relevant
and
important
aspects
of
the
uncertainties
which
you
think
should
be
addressed
in
the
SERA?

4)
In
your
opinion,
what
is
the
weakest
and
what
is
the
strongest
aspect
of
the
SERA?
Can
you
make
any
suggestions
on
how
the
weakest
parts
can
be
strengthened
by
the
Agency?

5)
In
Section
II,
are
the
stressors,
ecological
effects,
and
both
the
assessment
and
measurement
endpoints
adequately
characterized?
Are
the
five
emission
scenarios
adequate
to
characterize
the
exposures
for
the
WTI
facility?
Are
there
other
emission
scenarios
which
you
think
should
be
included
in
the
SERA?
Peer
Review
Handbook
Page
B­
9
6)
In
Section
III,
is
the
site
characterization
adequate
to
support
the
SERA?
Why
or
why
not?

7)
In
Section
IV,
is
the
tiered
process
used
to
identify
the
ecological
chemicals
of
concern
(
ECOC)
from
the
initial
list
of
potential
chemicals
considered
scientifically
defensible?
Does
application
of
this
tiered
approach
support
the
statement
made
in
the
SERA
"
by
focusing
on
the
potential
risk
from
the
selected
ECOCs,
the
SERA
provides
a
thorough
screening­
level
evaluation
for
the
WTI
facility?"

8)
In
Sections
V
and
VI,
are
the
exposure
and
ecological
effects
adequately
characterized?
Are
the
most
appropriate
estimation
techniques
available
used?
Are
the
assumptions
clearly
stated?
Please
explain
fully.

9)
In
Section
VIII,
are
there
any
major
elements
missing
from
the
risk
characterization
which
you
think
need
to
be
included
or
which
would
strengthen
the
risk
characterization?
Does
the
risk
characterization
support
the
summary
and
conclusions
presented
in
Section
IX?

10)
In
Section
IX,
given
the
assumptions
made
and
the
processes
used
to
select
and
evaluate
chemicals,
receptors,
and
exposure
pathways,
do
you
think
the
SERA
adequately
met
its
objective
of
not
inadvertently
underestimating
risk?

Accident
Analysis
The
Accident
Analysis
for
the
WTI
incinerator
involves
evaluating
the
probability
of
an
emergency
incident
occurring
which
results
in
the
release
of
hazardous
waste.
The
consequences
of
this
release
are
also
evaluated
using
exposure
and
human
health
effects
information.
Unlike
the
human
health
risk
assessment
which
has
a
primary
goal
of
quantifying
risks,
the
accident
analysis
typically
provides
information
that
can
be
used
to
reduce
the
likelihood,
extent
and
impact
of
possible
accidents.
Please
comment
on
the
following
issues
in
your
review
of
this
aspect
of
the
risk
assessment.

1)
The
WTI
accident
assessment
selected
five
scenarios
for
quantitative
evaluation
that
were
considered
to
be
of
primary
concern.
The
scenarios
are
an
on­
site
spill,
an
on­
site
fire,
an
on­
site
mixing
of
incompatible
waste,
an
off­
site
spill,
and
an
off­
site
spill
and
fire.
Please
comment
on
the
selection
of
these
scenarios.
Were
any
significant
scenarios
missed?

2)
Specific
chemicals
were
selected
to
evaluate
each
scenario.
Please
comment
on
the
selections.
Would
other
chemicals
have
been
more
appropriate?
Peer
Review
Handbook
Page
B­
10
3)
Chemical
specific
release
rates
are
calculated
for
each
scenario.
Please
comment
on
the
procedures
used
to
estimate
the
release
rates.
Was
an
appropriate
approach
used?

4)
Atmospheric
dispersion
modeling
was
used
to
estimate
air
concentrations
of
hazardous
chemicals.
Specifically,
the
SLAB
model
was
used
for
vapor
releases
from
spills
and
the
mixing
of
incompatible
wastes.
ISC­
COMPDEP
was
used
for
releases
associated
with
fires.
Comment
on
the
selection
of
the
models
and
inputs
Are
they
appropriate
selections?
Should
other
models
or
inputs
been
used?

5)
Please
comment
on
the
assessment's
conclusions
on
the
severity
of
consequences
and
probability
of
occurrence.
Has
the
report
correctly
categorized
the
severity
of
the
consequences
of
the
different
accident
scenarios?
Has
the
assessment
adequately
justified
the
reported
probability
of
occurrence
of
each
of
the
accident
events?

6)
Key
assumptions
were
made
in
the
identification
of
accident
scenarios
and
the
description
of
the
conservative
and
typical
events.
Included
were
a
description
of
the
magnitude
of
the
effect
of
the
assumptions
and
direction
of
the
effect.
Please
comment
on
the
assumptions.
Are
they
justified?
Are
the
descriptions
of
the
magnitude
and
directions
of
the
effects
correct?
Has
the
accident
assessment
adequately
confronted
the
uncertainties
involved
in
doing
this
type
of
analysis?
If
not,
what
else
should
be
done?

7)
Comment
on
the
appropriateness
of
using
IDLH
values
for
characterizing
the
severity
of
consequences
in
the
accident
analysis.
Comment
on
the
appropriateness
of
using
10
X
LOC
for
chemicals
for
which
IDLH
values
have
not
been
established.

8)
In
the
accident
analysis,
IDLH
(
or
10
X
LOC)
values
were
used
to
determine
the
downwind
distances
over
which
adverse
human
health
effects
might
occur.
To
evaluate
the
uncertainty
introduced
by
using
the
IDLH,
a
sensitivity
analysis
was
conducted
where
these
distances
were
recalculated
using
the
LOC
(
a
more
stringent
health
criteria).
Other
sources
of
uncertainty
that
are
identified
in
the
accident
analysis
include
concentration
averaging
times,
chemical
concentrations,
emission
rates,
and
meteorological
conditions.
For
most
of
these
parameters
it
is
stated
that
conservative
assumptions
were
used
to
avoid
underestimating
risks.
Have
the
uncertainties
inherent
in
the
accident
analysis
been
adequately
characterized?
For
those
parameters
where
sensitivity
analyses
were
not
conducted,
is
the
conclusion
that
conservative
assumptions
have
avoided
underestimation
valid?
Peer
Review
Handbook
Page
B­
11
CHARGE
EXAMPLE
2
­
IRIS
Pilot
Program
Instructions
to
Peer
Reviewers
for
Reviewing
IRIS
Summaries
and
Supporting
Documentation
The
U.
S.
EPA
is
conducting
a
peer
review
of
the
scientific
basis
supporting
the
health
hazard
and
dose
response
assessments
for
the
subject
chemical
that
will
appear
on
the
Agency's
online
database,
the
Integrated
Risk
Information
System
(
IRIS).
Materials
to
be
reviewed
include
the
summary
information
that
will
appear
on
IRIS
(
the
inhalation
reference
concentration
[
RfC],
oral
reference
dose
[
RfD],
and
cancer
assessment)
and
the
supporting
document,
the
Toxicological
Review,
which
will
also
be
made
available
to
the
public.

A
listing
of
Agency
Guidelines
and
Methodologies
that
were
used
in
the
development
of
these
hazard
and
dose­
response
assessments
included
the
following:
The
Risk
Assessment
Guidelines
(
1986),
the
(
new)
Proposed
Guidelines
for
Carcinogen
Risk
Assessment
(
1996),
Guidelines
for
Developmental
Toxicity
Risk
Assessment,
(
proposed)
Interim
Policy
for
Particle
Size
and
Limit
Concentration
Issues
in
Inhalation
Toxicity,
(
proposed)
Guidelines
for
Neurotoxicity
Risk
Assessment,
Methods
for
Derivation
of
Inhalation
Reference
Concentrations
and
Application
of
Inhalation
Dosimetry,
Recommendations
for
and
Documentation
of
Biological
Values
for
Use
in
Risk
Assessment
and
Use
of
the
Benchmark
Dose
Approach
in
Health
Risk
Assessment.
Copies
of
these
documents
(
and/
or
their
relevant
sections)
will
be
made
to
the
reviewer
upon
request.

Peer
review
is
meant
to
ensure
that
science
is
used
credibly
and
appropriately
in
derivation
of
these
dose­
response
assessments.
You
have
been
chosen
as
an
expert
on
the
chemical
under
consideration,
on
a
scientific
discipline
related
to
at
least
one
of
the
assessments,
or
in
the
field
of
risk
assessment.
At
least
three
peer
reviewers
per
chemical
are
being
chosen
to
review
the
scientific
basis
of
these
draft
dose­
response
assessments
before
they
are
forwarded
on
to
the
EPA's
Consensus
Process
for
final
approval
and
adoption
by
the
EPA.
These
hazard
and
dose­
response
assessments
will
then
appear
on
IRIS
and
become
available
as
Agency
consensus
health
effect
information.

The
primary
function
of
the
peer
reviewer
should
be
to
judge
whether
the
choice,
use,
and
interpretation
of
data
employed
in
the
derivation
of
the
assessments
is
appropriate
and
scientifically
sound.
This
review
is
not
of
the
recommended
Agency
risk
assessment
guidelines
or
methodologies
used
to
derive
cancer
or
RfD/
C
assessments
as
these
have
been
reviewed
by
external
scientific
peers,
the
public,
and
EPA
Science
Advisory
Boards.
The
reviewer's
comments
on
the
application
of
these
guidelines/
methodologies
within
the
individual
assessments
is,
however,
welcomed
and
encouraged.
For
example,
the
reviewer
may
ascertain
whether
or
not
there
is
data
sufficient
to
support
use
of
other
than
default
assumptions
for
areas
such
as
sensitive
subpopulations
or
linear
cancer
extrapolation.
The
reviewer
may
also
have
Peer
Review
Handbook
Page
B­
12
opinions
on
other
areas
of
uncertainty
such
as
subchronic
to
chronic
duration
(
when
only
a
subchronic
study
is
available)
or
an
incomplete
data
base
but
should
focus
on
the
specific
area
of
uncertainty
rather
than
on
the
magnitude
of
the
overall
estimate.

Below
are
two
groups
of
questions
regarding
this
review.
The
first
is
a
set
of
general
questions
that
are
meant
to
guide
you
through
your
review.
It
is
not
imperative
that
you
specifically
answer
each
question
of
this
group.
The
second
group
of
questions,
however,
are
specific
for
the
chemical
assessments
and
deal
with
areas
of
scientific
controversy
or
uncertainty
in
which
the
Agency
may
have
to
make
a
scientific
judgment.
Your
input
to
this
set
of
questions
is
considered
vital
to
the
review
process.

Questions
for
IRIS
Peer
Reviewers
­
General
1)
Are
you
aware
of
any
other
data/
studies
that
are
relevant
(
i.
e.,
useful
for
the
hazard
identification
or
dose­
response
assessment)
for
the
assessment
of
the
adverse
health
effects,
both
cancer
and
noncancer,
of
this
chemical?
Please
explain
fully.

2)
For
the
RfD
and
RfC,
has
the
most
appropriate
critical
effect
been
chosen
(
i.
e.,
that
adverse
effect
appearing
first
in
a
dose­
response
continuum)?
For
the
cancer
assessment,
are
the
tumors
observed
biologically
significant?
relevant
to
human
health?
Points
relevant
to
this
determination
include
whether
or
not
the
choice
follows
from
the
dose­
response
assessment,
whether
the
effect
is
considered
adverse,
and
if
the
effect
(
including
tumors
observed
in
the
cancer
assessment)
and
the
species
in
which
it
is
observed
is
a
valid
model
for
humans.

3)
Have
the
noncancer
and
cancer
assessments
been
based
on
the
most
appropriate
studies?
These
studies
should
present
the
critical
effect/
cancer
(
tumors
or
appropriate
precursor)
in
the
clearest
dose­
response
relationship.
If
not,
what
other
study
(
or
studies)
should
be
chosen
and
why?

4)
Studies
included
in
the
RfD
and
RfC
under
the
heading
"
Supporting/
Additional
studies"
are
meant
to
lend
scientific
justification
for
the
designation
of
critical
effect
by
including
any
relevant
pathogenesis
in
humans,
any
applicable
mechanistic
information,
any
evidence
corroborative
of
the
critical
effect,
or
to
establish
the
comprehensiveness
of
the
data
base
with
respect
to
various
endpoints
(
such
as
reproductive/
developmental
toxicity
studies).
Should
other
studies
be
included
under
the
"
Supporting/
Additional"
category?
Should
some
studies
be
removed?
Peer
Review
Handbook
Page
B­
13
5)
For
the
noncancer
assessments,
are
there
other
data
that
should
be
considered
in
developing
the
uncertainty
factors
or
the
modifying
factor?
Do
you
consider
that
the
data
support
use
of
different
(
default)
values
than
those
proposed?

6)
Do
the
Confidence
statements
and
weight­
of­
evidence
statements
present
a
clear
rationale
and
accurately
reflect
the
utility
of
the
studies
chosen,
the
relevancy
of
the
effects
(
cancer
and
noncancer)
to
humans,
and
the
comprehensiveness
of
the
data
base?
Do
these
statements
make
sufficiently
apparent
all
the
underlying
assumptions
and
limitations
of
these
assessments?
If
not,
what
needs
to
be
added?

Questions
for
IRIS
Peer
Reviewers
­
Chemical
Specific
[
example:
cumene]

1)
Based
on
the
information
noted
in
the
Principal
study
currently
designated
(
Cushman
et
al.,
1995)
is
the
discounting
of
the
renal
effects
in
males
justified?
Is
sufficient
rationale
given
to
let
stand
the
organ
weight
changes
in
female
rats
as
a
critical
effect?

2)
Is
the
information
in
the
Toxicological
Review
sufficient
to
consider
cumene
as
having
a
low
potential
for
causing
reproductive
effects?
Please
explain
fully.

RECOMMENDATIONS
Based
on
your
reading
and
analysis
of
the
information
provided,
please
identify
your
overall
recommendation
for
the
IRIS
materials
you
have
reviewed
as
1)
Acceptable
as
is
2)
Acceptable
with
minor
revision
(
as
indicated)
3)
Acceptable
with
major
revision
(
as
outlined)
4)
Not
acceptable
Peer
Review
Handbook
Page
B­
14
CHARGE
EXAMPLE
3
­
Science
Advisory
Board
Review
of
the
Agency's
National
Risk
Management
Research
Laboratory's
(
NRMRL)
Program
The
Office
of
Research
and
Development
(
ORD)
requests
that
the
Science
Advisory
Board
review
the
Agency's
National
Risk
Management
Research
Laboratory's
(
NRMRL)
program.

In
the
"
Strategic
Plan
for
the
Office
of
Research
and
Development"
(
EPA,
1996a),
ORD
described
the
relationship
of
risk
assessment
to
the
risk
management
process,
and
emphasized
the
need
for
scientific
and
engineering
research
to
enable
sound
risk
management
decisions
and
actions.
Within
the
framework
of
that
strategic
plan,
NRMRL's
mission
is
to
conduct
research
to
reduce
uncertainties
and
costs
associated
with
making
and
implementing
environmental
risk
management
decisions.
NRMRL
has
therefore
developed
a
research
agenda
to
reduce
risk
uncertainty
that
also
focuses
on
those
important,
relevant
issues
where
it
can
make
a
difference.

The
charge
to
the
SAB
is
to:

1)
Examine
and
critique
the
research
programmatic
directions
such
as
whether
NRMRL
is
pursuing
the
most
appropriate
research
problem
areas;

2)
Comment
on
strategic
directions,
e.
g.
use
of
its
core
technical
competencies,
transition
from
primarily
extramural
to
an
intramural
R&
D
organization,
leveraging
with
other
agencies
and
organization;

3)
Review
and
comment
on
the
effectiveness
of
NRMRL's
approach
to
science
management,
e.
g.
measures
of
success
and
science
quality,
soundness
of
peer
review
process;

4)
Examine
and
critique
the
relationship
of
NRMRL's
risk
management
research
and
its
intended
role
in
the
risk
assessment/
risk
management
paradigm;
and
5)
Review
and
comment
on
the
strategic
balance
for
the
next
decade
among
pollution
prevention,
technology
development,
remediation,
and
risk
management
assessment
activities.
Peer
Review
Handbook
Page
B­
15
CHARGE
EXAMPLE
4
­
Science
Advisory
Board
(
SAB)
Review
of
the
Technical
Aspects
of
the
Multi­
Agency
Radiation
Survey
and
Site
Investigation
Manual
(
MARSSIM)

The
EPA
Science
Advisory
Board
(
SAB)
is
asked
to
review
the
technical
aspects
of
the
Multi­
Agency
Radiation
Survey
and
Site
Investigation
Manual
(
MARSSIM).
The
review
document
was
developed
collaboratively
by
four
Federal
agencies,
departments
and
commissions
having
authority
for
control
of
radioactive
materials:
Department
of
Defense,
Department
of
Energy,
Environmental
Protection
Agency,
and
Nuclear
Regulatory
Commission.
MARSSIM
addresses
the
need
for
a
nationally
consistent
approach
to
conducting
radiation
surveys
of
potentially
radioactively
contaminated
sites
that
are
being
considered
for
release
to
the
public.
A
condition
of
release
is
a
demonstration
that
residual
radioactivity
levels
do
not
exceed
a
specified
risk
or
dose
level,
also
known
as
a
release
criterion.
MARSSIM
provides
guidance
to
users
performing
and
assessing
the
results
of
such
a
demonstration
for
surface
soils
and
building
surfaces.

The
SAB
is
asked
by
the
Agency's
Office
of
Radiation
and
Indoor
Air
(
ORIA)
to
respond
to
the
following
charge
in
its
review:

1)
Is
the
overall
approach
to
the
planning,
data
acquisition,
data
assessment,
and
data
interpretation
as
described
in
the
MARSSIM
technically
acceptable?
Please
explain
fully.

2)
Are
the
methods
and
assumptions
for
demonstrating
compliance
with
a
dose­
or
risk­
based
regulation
technically
acceptable?
Please
explain
fully.

3)
Are
the
hypotheses
and
statistical
tests
and
their
method
of
application
appropriate?
Please
explain
fully.
Peer
Review
Handbook
Page
B­
16
CHARGE
EXAMPLE
5
­
Science
Advisory
Board
(
SAB)
Review
of
the
Statistical
Performance
of
the
Agency's
Protozoan
Oocyst
Monitoring
Methods
The
Science
Advisory
Board
(
SAB)
is
asked
to
review
a
report
describing
the
statistical
performance
of
the
Agency's
protozoan
oocyst
monitoring
methods.
Agency
staff
recognized
that
the
protozoan
analysis
methodology
that
had
been
formally
adopted
for
the
Information
Collection
Rule
(
ICR)
was
crude
and
had
very
low
and
highly
variable
recoveries
of
added
oocysts.
The
statistical
methodology
was
considered
by
Agency
staff
to
be
necessary
to
determine
whether
the
Agency
can
take
advantage
of
the
large
monitoring
program
agreed
to
under
the
negotiated
rulemaking
process.

The
Agency
charge
asks
that
the
SAB
evaluate
the
report
and
address
the
following
concerns:

1)
Evaluate
the
factual
and
conceptual
soundness
of
the
approach
and
methods
used,
and
the
soundness
of
the
results
and
conclusions
of
the
report.

2)
Evaluate
the
viability
of
the
assumptions
and
conditions
tested
in
the
report.

3)
Evaluate
the
suitability
of
the
report
as
a
basis
for
making
a
decision
on
the
use
of
protozoan
monitoring
data
for
a
national
impact
assessment.

4)
Evaluate
whether
the
degree
of
accuracy
and
precision
of
the
protozoan
method
is
acceptable
for
an
impact
analysis.
Peer
Review
Handbook
Page
B­
17
CHARGE
EXAMPLE
6
­
Science
Advisory
Board
(
SAB)
Review
of
the
Environment
Monitoring
and
Assessment
Program
(
EMAP)
Research
Strategy
and
Research
Plan
The
Science
Advisory
Board
(
SAB)
is
asked
to
review
the
Environment
Monitoring
and
Assessment
Program
(
EMAP)
Research
Strategy
and
Research
Plan.
The
review
is
requested
by
the
Office
of
Research
and
Development
(
ORD)
with
the
following
specific
charge
issues:

1)
Previous
peer
reviews
recommended
that
EMAP
develop
a
close
working
relation
with
EPA
Program
Offices
and
other
federal
monitoring
efforts.
Does
the
EMAP
strategy
support
the
[
Office
of
Science
and
Technology
Policy's
Committee
on
Environment
and
Natural
Resources]
CENR
National
Monitoring
Framework
and
EPA
Program
Offices?

2)
Previous
peer
reviews
recommended
that
EMAP
initiate
a
focused
research
program
on
indicator
development.
Does
the
intramural
EMAP
program
on
ecological
indicator
development,
coordinated
with
the
[
ORD
Science
To
Achieve
Results]
STAR
solicitations,
respond
to
this
research
need?

3)
Previous
peer
reviews
recommended
that
the
EMAP
design
be
modified
to
include
a
set
of
nonrandomly
selected
sentinel
sites
with
intensive
data
collection.
Does
the
development
of
Index
Sites
as
outdoor
laboratories
in
the
national
parks
([
National
Park
Service]
NPS
and
[
US
Geological
Survey]
USGS)
and
selected
estuaries
([
National
Oceanic
and
Atmospheric
Administration]
NOAA)
add
this
dimension
to
the
EMAP?

4)
Previous
peer
reviews
recommended
that
EMAP
combine
effects­
oriented
and
stressor­
oriented
monitoring
approaches.
Do
the
focused
geographic
demonstration
pilot
studies
(
initially
in
the
Mid­
Atlantic
region)
combine
these
elements?
Peer
Review
Handbook
Page
B­
18
THIS
PAGE
LEFT
INTENTIONALLY
BLANK
Peer
Review
Handbook
Page
C­
1
APPENDIX
C
GUIDANCE
ON
REQUESTING
A
REVIEW
BY
THE
SCIENCE
ADVISORY
BOARD
(
SAB)
Peer
Review
Handbook
Page
C­
2
Guidance
on
Requesting
a
Review
By
the
Science
Advisory
Board
(
SAB)

Summary
The
Science
Advisory
Board
(
SAB)
annually
solicits
proposals
for
review
projects
every
spring
for
the
following
fiscal
year.
This
appendix
provides
guidance
to
Programs
and
Regions
to
help
them
submit
requests
for
SAB
reviews.
Requests
should
be
submitted
to
the
Science
Advisory
Board
in
both
hard
copy
and
electronic
versions
usually
by
mid­
June
for
the
following
fiscal
year.
The
requests
may
be
part
of
the
annual
submissions
that
respond
to
the
peer
review
activities
of
the
Agency,
or
they
may
be
submitted
directly
to
the
Board.
Although
providing
requests
at
one
defined
time
in
the
spring
helps
with
SAB
planning,
we
recognize
that
projects
also
come
to
light
during
other
parts
of
the
year.
Please
contact
the
SAB
staff
for
details
on
making
submissions
during
the
remainder
of
the
year
(
see
end
of
this
document
for
contacts).

Background
A
key
priority
for
Administrator
Browner
is
to
base
Agency
actions
on
sound
scientific
data,
analyses,
and
interpretations.
She
issued
the
Agency's
Peer
Review
Policy
to
increase
the
quality
of
the
technical
foundations
upon
which
EPA's
regulatory
structures
are
built.
The
SAB
is
a
key
scientific
peer
review
mechanism
available
to
Programs
and
Regions
in
implementing
the
Peer
Review
Policy.
However,
because
the
Board
has
finite
resources
it
cannot
conduct
all
reviews.
This
document
is
designed
to
help
Programs
and
Regions
determine
which
projects
to
submit
to
the
SAB.
Note
particularly
that
the
SAB
focuses
on
the
technical
underpinnings
of
Agency
positions;
i.
e.,
risk
assessment
issues,
in
contrast
to
risk
management
issues.

The
topics
that
are
best
suited
for
the
Board's
agenda
are
those
that
satisfy
several
of
the
following
criteria:

1)
Integrate
science
into
Agency
actions
in
new
ways.
2)
Influence
long­
term
technological
developments.
3)
Impact
overall
environmental
protection.
4)
Address
novel
scientific
problems
or
principles.
5)
Address
problems
that
transcend
federal­
agency
or
other
organizational
boundaries.
6)
Strengthen
the
Agency's
basic
capabilities.
7)
Serve
Congressional
or
other
leadership
interests.
8)
Deal
with
controversial
issues.

In
suggesting
issues
for
SAB
involvement,
Programs
and
Regions
should
note
the
breadth
of
SAB
activities:
Peer
Review
Handbook
Page
C­
3
Historically,
most
of
the
outputs
of
the
Board
are
in
the
form
of
"
full"
reports.
They
present
the
findings
of
peer
reviews
of
Agency
document(
s)
and
contain
considerable
detail
about
the
findings
and
recommendations
of
the
Board.
They
also
address
the
specific
questions
posed
by
the
Charge
to
the
Board.
"
Letter"
reports
fulfill
the
same
function
as
reports,
but
are
simply
shorter
in
length.
Due
to
the
need
to
be
more
responsive
with
advice,
the
Board
has
recently
begun
to
produce
more
short
letter
reports
than
full
reports,
as
they
can
be
produced
and
finalized
in
less
time.

The
SAB
has
also
introduced
the
"
Consultation"
as
a
means
of
conferring­­
in
public
session­­
with
the
Agency
on
a
technical
matter
before
the
Agency
has
begun
substantive
work
on
that
issue.
The
goal
is
to
leaven
EPA's
thinking
on
an
issue
by
brainstorming
a
variety
of
approaches
to
the
problem
very
early
in
the
development
process.
There
is
no
attempt
or
intent
to
express
an
SAB
consensus
or
to
generate
an
SAB
report.
The
Board,
via
a
brief
letter
simply
notifies
the
Administrator
that
a
Consultation
has
taken
place.

More
recently,
the
Board
introduced
a
new
vehicle
for
communicating
with
its
clients
­­
the
"
Advisory"
­­
which
provides,
via
a
formal
SAB
consensus
report,
critical
input
on
technical
issues
that
arise
during
the
Agency's
issue
development
process.
The
Advisory
generally
involves
a
review
of
a
multi­
year
Agency
project.
The
intent
is
to
provide
some
mid­
course
assessment
to
see
if
the
Agency
is
heading
in
a
scientifically
credible
direction.
In
order
to
maintain
an
objective,
arms­
length
relation
with
the
Agency
and
its
projects,
the
SAB
review
of
the
final
product
at
some
point
in
the
future
will
include
experts
who
did
not
participate
in
producing
the
Advisory.

The
Agenda
Setting
Process
Each
Assistant
Administrator
and
Regional
Administrator
is
normally
asked
to
submit
a
list
of
candidate
topics
for
SAB
action/
review.
A
"
project
sheet"
(
see
attached
example)
is
used
to
define
each
topic
that
is
nominated
for
SAB
review.
The
project
sheet
is
prepared
by
the
requesting
office
and
contains
the
following
information:

1)
Project
title/
subject
(
Descriptive
short
title
of
project)

2)
Requesting
Organization/
Office
(
Primary
office
requesting
review,
AA/
RA
level)

3)
Requesting
Official
(
Name
and
position
of
senior
official
requesting
review,
usually
office
or
division
level
­
this
is
the
person
who
may
receive
a
summary
briefing
from
the
Chair
following
the
review)
Peer
Review
Handbook
Page
C­
4
4)
Program
Contact
(
Name/
phone
number/
mail
code
­
this
is
the
principal
contact
for
SAB
Staff
to
interact
with
during
development
of
the
SAB
review)

5)
Background
(
brief
history
of
the
project
and
why
it
is
important)

6)
Tentative
Charge
(
what
the
SAB
is
being
asked
to
comment
on
­
usually
a
set
of
questions)

7)
Tentative
Schedule
and
Committee
(
when
the
review
is
expected
to
be
conducted,
e.
g.,
Winter
1998;
and
which
SAB
committee
is
appropriate
for
the
review
­­
final
choice
as
to
the
review
committee
is
at
the
discretion
of
the
Board)

8)
Budget
Estimate
(
Rough
estimates
of
Agency
funding
for
the
subject
over
the
past
5
years
(
if
applicable)
and
for
the
next
2
years
(
if
applicable).
The
SAB
Executive
Committee
has
asked
for
this
information
to
help
it
better
appreciate
the
level
of
Agency
involvement
in
and
commitment
to
the
issue)

9)
Preparer
(
name,
phone,
office
of
preparer
of
Project
Sheet
and
the
date
prepared)

The
proposed
topics
will
be
examined
and
discussed
in
a
number
of
forums:

1)
The
individual
SAB
Committees
­
Throughout
the
late
Spring
and
Summer,
the
SAB
Committees
will
be
examining
options
for
the
following
fiscal
year,
including
all
suggestions
made
by
the
Agency.

2)
The
Science
Policy
Council­
Steering
Committee
(
SPC­
SC)
­
The
SPC­
SC
usually
meets
in
early
summer
to
examine
the
proposals
for
each
fiscal
year.
The
goal
is
to
provide
cross­
office
critique/
integration
of
the
proposals.
The
SPC­
SC
will
be
used
as
a
forum
for
continuing
discussion
throughout
the
process
and
throughout
the
year
as
new
topics
emerge.

3)
The
Deputy's
perspective
­
In
the
summer,
the
Deputy
Administrator
will
review
the
requests
and
provide
insights
on
priorities.

4)
The
SAB
Executive
Committee
­
During
its
summer
meeting,
the
SAB's
EC
will
examine
the
nominated
topics,
adding
its
own
perspective
on
an
appropriate
agenda,
using
its
selection
criteria.

5)
The
Administrator
­
In
September,
the
list
of
proposed
topics
will
be
delivered
to
Administrator
Browner
for
information
and
added
insights.
Peer
Review
Handbook
Page
C­
5
The
completed
project
sheets
should
be
submitted
electronically
to
the
SAB
Deputy
Staff
Director
(
fowle.
jack@
epamail.
epa.
gov)
and
in
signed
hard
copy
(
mail
code
1400).

SAB
Staff
Contacts
Dr.
Donald
Barnes
(
202­
260­
4126)
­
Staff
Director;
Designated
Federal
Officer
(
DFO)
for
the
Executive
Committee:

Dr.
Jack
Fowle
(
202­
260­
8325)
­
Deputy
Staff
Director.

Mr.
Robert
Flaak
(
202­
260­
5133)
­
Team
Leader,
Committee
Operations
Staff;
DFO
for
the
Clean
Air
Scientific
Advisory
Committee
(
CASAC);
DFO
for
the
Research
Strategies
Advisory
Committee
(
RSAC).

Ms.
Kathleen
Conway
(
202­
260­
2558)
­
DFO
for
the
Environmental
Engineering
Committee
(
EEC).

Dr.
Jack
Kooyoomjian
(
202­
260­
2560)
­
DFO
for
the
Radiation
Advisory
Committee
(
RAC);
DFO
for
the
Advisory
Council
on
Clean
Air
Compliance
Analysis
(
Council).

Mr.
Tom
Miller
(
202­
260­
5886)
­
DFO
for
the
Drinking
Water
Committee
(
DWC);
DFO
for
the
Environmental
Economics
Advisory
Committee
(
EEAC).

Mr.
Sam
Rondberg
(
202­
260­
2559)
and
LCDR
Roslyn
Edson
(
PHS)
­
DFOs
for
the
Environmental
Health
Committee
(
EHC);
DFOs
for
the
Integrated
Human
Exposure
Committee
(
IHEC)

Ms.
Stephanie
Sanzone
(
202­
260­
6557)
­
DFO
for
the
Ecological
Processes
and
Effects
Committee
(
EPEC).

All
SAB
staff
can
be
contacted
via
Agency
Email.
Peer
Review
Handbook
Page
C­
6
Science
Advisory
Board
Proposed
Project
Project
title/
subject:
Proposed
Amendments
to
the
Risk
Assessment
Guidelines
for
Carcinogens
Requesting
Organization/
Office:
Office
of
Research
and
Development
(
ORD)

Requesting
Official:
Name,
Title,
Office/
Organization
Program
Contact:
Name,
Title,
Office/
Organization,
202­
260­
xxxx
Background:
EPA's
Health
Risk
Assessment
Guidelines
provide
generic
science
and
science
policy
guidance
on
risk
assessment
issues
for
use
in
all
Agency
offices.
EPA
has
currently
issued
or
proposed
nine
guidelines
(
or
amendments)
in
this
series,
all
of
which
have
been
submitted
to
the
Science
Advisory
Board
for
review.

The
current
guidelines
for
carcinogen
risk
assessment
were
reviewed
by
the
SAB
and
issued
as
final
guidance
in
1986.
In
1988,
the
Forum
initiated
a
public
process
for
considering
amendments
to
these
guidelines.
A
Risk
Assessment
Technical
Panel
considered
submissions
from
the
public
as
well
as
information
developed
by
experts
at
two
public
workshops
in
revising
these
guidelines.

Tentative
Charge:
Review
the
amended
and
expanded
guidance,
with
special
emphasis
on
(
a)
weight­
of­
evidence
issues,
(
b)
a
new
classification
system,
(
c)
dose
response
modeling,
and
(
d)
the
use
of
pharmacokinetic
and
metabolic
data.
A
more
detailed
charge
will
be
negotiated
with
SAB
at
a
later
date.

Tentative
Schedule
and
Committee:
Winter,
1998,
Environmental
Health
Committee
Budget:
FY
1995
­
$
xxx
and
yy
FTE
FY
1996
­
$
xxx
and
yy
FTE
FY
1997
­
$
xxx
and
yy
FTE
FY
1998
­
$
xxx
and
yy
FTE
FY
1999
­
FY2000
­
estimated
costs
of
$
xxx
and
yy
FTE
each
year
Preparer:
Name,
Title,
Office/
Organization,
202­
260­
xxxx
Date:
June
1,
1997
Peer
Review
Handbook
Page
D­
1
APPENDIX
D
Example
Statements
of
Work
for
Contracts
Peer
Review
Handbook
Page
D­
2
STATEMENT
OF
WORK
­
EXAMPLE
1
­
Statement
of
Work:
Technical
Review
Contractor
for
Panel
Review
of
Assistance
Agreement
or
Fellowship
Applications
1)
Purpose
The
purpose
of
this
contract
is
to
purchase
peer
review
services
of
a
contractor
with
expertise
in
Exploratory
Research;
Environmental
Chemistry.
The
services
are
for
peer
reviewing
applications
received
in
response
to
the
Office
of
Research
and
Development's
(
ORD's)
1997
Science
to
Achieve
Results
program.
These
reviews
shall
be
completed
and
the
evaluation
sheets
shall
be
prepared
prior
to
the
reviewer's
participation
in
a
3­
day
panel
discussion
to
be
held
in
Washington,
D.
C.
on
May
5
­
7,
1997.

2)
Statement
of
Work
ORD's
National
Center
for
Environmental
Research
and
Quality
Assurance
(
NCERQA)
is
responsible
for
overseeing
the
recently
expanded
research
grants
and
fellowships
programs.
Each
year
NCERQA
(
alone
or
in
conjunction
with
other
organizations)
solicits
applications
in
each
of
these
programs.
The
applications
to
be
reviewed
under
this
contract
were
submitted
in
response
to
the
solicitation
for
the
1997
Science
to
Achieve
Results
program.
As
part
of
the
selection
process,
NCERQA
must
conduct
a
peer
review
that
is
designed
to
evaluate
the
scientific
quality
of
each
application;
this
is
accomplished
through
the
ad
hoc
use
of
technical
experts.

The
peer
review
services
required
by
this
contract
necessitate
the
independent
review
of
a
maximum
of
10
applications
and
the
preparation
of
a
typed
evaluation
summary
and
an
overall
rating
for
each
of
these
applications.
Each
evaluation
summary
shall
support
and
be
consistent
with
the
overall
rating
that
is
assigned;
it
also
shall
be
completed
prior
to
the
contractor's
participation
in
the
panel
discussions.
After
the
panel
discussions
for
the
applications
assigned
to
the
contractor,
the
contractor
shall
submit
all
completed
evaluation
summaries
to
the
designated
Science
Review
Administrator
(
SRA).

The
contractor
also
shall
serve
as
the
panel's
rapporteur
for
approximately
6
­
8
of
the
applications
assigned.
As
rapporteur,
the
contractor
shall
be
responsible
for
preparing
a
typed
evaluation
summary
(
on­
site
typing
support
will
be
provided
by
NCERQA)
that
reflects
the
panel's
discussion
of
the
respective
application
as
well
as
the
panel's
overall
rating
(
the
criteria
for
the
panel's
overall
rating
are
the
same
as
those
for
each
peer
reviewer's
overall
rating).
As
rapporteur,
the
contractor
shall
submit
these
panel
evaluation
summaries
to
the
designated
SRA
prior
to
leaving
the
panel
meeting.
Peer
Review
Handbook
Page
D­
3
Before
the
contractor
shall
be
allowed
to
participate
in
the
review
process,
the
contractor
shall
have
disclosed
any
actual
or
potential
conflicts
of
interest
and
shall
have
signed
and
submitted
to
NCERQA
a
Conflict
of
Interest/
Confidentiality
Form.
The
contractor
is
directed
to
assure
that
none
of
the
conflicts
disclosed
are
so
direct
and
substantial
as
to
rule
out
a
particular
reviewer.
Upon
receipt
of
an
approved
Purchase
Order
(
PO),
NCERQA
will
send
the
following
items
to
the
contractor:

a)
A
copy
of
the
Purchase
Order
or
the
Purchase
Order
number
b)
The
applications
assigned
to
the
contractor
c)
For
grants,
a
set
of
abstracts
for
all
the
applications
being
reviewed
by
the
panel
d)
For
persons
reviewing
grant
applications,
a
copy
of
the
pertinent
section(
s)
of
the
solicitation
package
to
provide
background
information;
for
persons
reviewing
fellowship
applications,
information
on
how
to
access
the
solicitation
package
on
the
Internet
e)
A
sample
evaluation
form
to
help
the
contractor
prepare
an
acceptable
evaluation
form
for
each
assigned
application
f)
A
blank
evaluation
form
for
each
assigned
application
and
the
criteria
for
completing
the
form
and
determining
the
overall
rating
g)
A
blank
and
sample
invoice
as
well
as
instructions
for
completing
and
submitting
the
invoice
to
EPA
h)
Information
on
the
points
of
contact
for
additional
information
(
e.
g.,
NCERQA's
SRA)
i)
Logistics
information
on
the
location
and
time
of
the
panel
discussions
NCERQA
will
transmit
the
above
items
under
a
cover
letter.
In
this
cover
letter,
NCERQA
will
provide
additional
details
about
each
item,
including
(
as
needed)
more
specific
instructions
for
the
set
of
applications
assigned
to
the
reviewer.

Each
contractor
shall
be
responsible
for
making
his/
her
own
travel
reservations
for
hotel
and
transportation.

3)
Reviewer
Tasks
a)
Review
the
assigned
applications
using
the
guidance
provided
with
NCERQA's
evaluation
form.

b)
Submit
completed
evaluation
forms
to
the
SRA
designated
in
the
cover
letter
immediately
following
the
panel
discussions
for
the
applications
assigned
to
the
contractor.
THE
COMPLETED
FORMS
MUST
BE
TYPED,
AND
THE
EVALUATION
SUMMARY
FOR
EACH
APPLICATION
MUST
SUPPORT
AND
BE
CONSISTENT
WITH
THE
OVERALL
RATING
THAT
IS
ASSIGNED
BY
THE
CONTRACTOR.
IN
SITUATIONS
Peer
Review
Handbook
Page
D­
4
WHERE
THESE
CONDITIONS
ARE
NOT
MET,
THE
SRA
WILL
ASK
THE
CONTRACTOR
TO
REDO
THE
FORM.

c)
For
those
applications
for
which
the
contractor
is
serving
as
the
panel's
rapporteur,
submit
a
completed
panel
evaluation
summary
to
the
SRA
designated
in
the
cover
letter
prior
to
leaving
the
panel
meeting.
THE
PANEL'S
EVALUATION
SUMMARY
MUST
BE
TYPED
(
ON­
SITE
TYPING
SUPPORT
WILL
BE
PROVIDED
BY
NCERQA)
AND
BE
CONSISTENT
WITH
THE
PANEL'S
OVERALL
RATING.
IN
SITUATIONS
WHERE
THESE
CONDITIONS
ARE
NOT
MET,
THE
SRA
WILL
ASK
THE
CONTRACTOR
TO
REDO
THE
FORM.

d)
Make
own
airline
and
hotel
accommodations
for
participation
in
the
panel
review
meeting.
Round­
trip
air
fare
must
be
a
commercial
REFUNDABLE
ticket.
Peer
Review
Handbook
Page
D­
5
STATEMENT
OF
WORK
­
EXAMPLE
2
­
Peer
Review
of
Prioritization
Tool
Report
Work
Assignment
No.:

Title:
Peer
Review
of
Prioritization
Tool
Report
Work
Assignment
Manager
(
WAM):

Name:
John
Q.
Government
Employee
Address:
Office
of
Solid
Waste
Phone
No.:
(
202)
260­
XXXX
Background:

The
Waste
Minimization
Branch
(
WMB)
in
the
Office
of
Solid
Waste
(
OSW)
is
in
the
process
of
implementing
the
Waste
Minimization
National
Plan,
announced
by
the
Agency
on
November
18,
1994.
The
Plan
reaffirms
the
Agency's
commitment
to
promote
source
reduction
over
waste
management,
in
keeping
with
the
policy
stated
in
the
1984
amendments
to
the
Resource,
Conservation,
and
Recovery
Act
(
RCRA)
and
in
the
1990
Pollution
Prevention
Act
(
PPA).
The
Plan
outlines
major
goals,
objectives,
and
action
items
to
achieve
national
reductions
in
the
generation
of
hazardous
wastes.

One
of
the
objectives
of
the
Plan
is
to:
"
develop
a
framework
for
setting
national
priorities;
develop
and
distribute
a
flexible
screening
tool
for
identifying
priorities
at
individual
facilities;
[
and]
identify
constituents
of
concern."
This
objective
is
a
key
building
block
in
implementing
subsequent
objectives
of
the
Plan.

In
September
1995,
WMB
formed
the
Waste
Minimization
Prioritization
Team,
which
includes
representatives
from
EPA
regions
and
states,
to
implement
this
objective.
The
Team
has
worked
to
assess
stakeholder
needs
for
prioritization
tools
and
to
evaluate
prioritization
tools
that
are
currently
available.
The
Team
plans
to
summarize
this
work,
along
with
its
recommendations,
in
a
report
(
referred
to
herein
as
the
Prioritization
Tool
report)
that
would
be
available
in
draft
form
in
July
1996.

WMB
and
the
Team
wish
to
obtain
independent
peer
review
of
the
Prioritization
Tool
report
prior
to
briefing
EPA
management.
The
report
is
being
prepared
with
the
support
of
ICF,
Inc.;
therefore,
for
the
peer
review
to
be
considered
independent,
it
must
be
performed
by
another
contractor.
Peer
Review
Handbook
Page
D­
6
Purpose
and
Scope
of
Work:

The
purpose
of
this
work
assignment
is
to
provide
support
to
WMB
and
the
Team
in
finalizing
the
Prioritization
Tool
report
by
conducting
an
independent
peer
review
of
the
report.

Work
Statement:

Task
1
­
Management
work
plan
and
budget
Within
15
days
of
CO
approval
of
this
work
assignment,
the
contractor
shall
deliver
a
management
work
plan
including
a
proposed
level
of
effort,
schedule,
and
budget
for
all
tasks.

Task
2
­
Provide
independent
peer
review
of
Prioritization
Tool
report
The
contractor
shall
provide
support
to
WMB
and
the
Team
in
preparing
the
Prioritization
Tool
report
by
performing
an
independent
peer
review
of
the
report.
The
contractor
shall
establish
a
panel
of
peer
reviewers
including
three
senior­
level
persons
who
collectively
have
extensive
expertise
in
particular
areas
to
be
identified
by
the
WAM
upon
approval
of
the
work
assignment.

Within
three
weeks
of
work
assignment
approval,
receipt
from
the
WAM
of
the
necessary
qualifications
of
peer
reviewers
(
in
a
TD),
and
receipt
from
the
WAM
of
the
peer
review
"
charge"
(
in
a
TD),
whichever
comes
latest,
the
contractor
shall
identify
the
three
peer
reviewers
and
prepare
a
memo
that
lists
the
names
of
the
peer
reviewers
and
their
affiliations
and
includes
the
peer
reviewers'
bio's.
Within
five
weeks
of
WAM
approval
of
the
of
the
peer
reviewers
(
via
a
TD)
and
receipt
of
the
draft
Prioritization
Tool
report
from
the
WAM
(
via
a
TD),
whichever
comes
later,
the
contractor
shall
conduct
the
peer
review,
assemble
the
peer
review
comments
and
recommendations
in
a
peer
review
report
organized
by
charge
question,
prepare
an
introduction
to
the
peer
review
report
with
a
clear
and
concise
overview
of
the
comments,
and
attach
to
the
peer
review
report
any
marginal
comments
the
peer
reviewers
had
on
the
Prioritization
Tool
report.

It
is
not
necessary
that
the
peer
reviewers
jointly
reach
consensus
on
their
findings
and
recommendations,
since
there
may
be
limited
overlap
in
the
peer
reviewers'
areas
of
expertise
and
in
the
charge
questions
that
they
focus
on.
The
contractor
shall
assume,
for
the
purpose
of
estimating
costs,
that
the
draft
Prioritization
Tool
report
is
roughly
100
pages
in
length
with
200
pages
of
appendices,
and
that
each
peer
reviewer
will
spend
40
hours
in
reviewing
the
report
and
writing
comments.
EPA
plans
to
provide
the
report
to
the
contractor
in
mid­
July.
Peer
Review
Handbook
Page
D­
7
Deliverables
and
Schedule:

Task
Deliverable
Schedule
1
Work
plan
and
budget
Within
15
days
of
CO
approval
of
work
assignment
2
Memo
identifying
peer
reviewers
Within
3
weeks
of
work
assignment
approval,
receipt
of
peer
reviewer
qualifications
from
WAM,
and
receipt
of
charge
from
WAM,
whichever
comes
latest
3
Peer
review
report
Within
five
weeks
of
WAM
approval
of
peer
reviewers
and
receipt
of
draft
Prioritization
Tool
report
from
WAM,
whichever
comes
later
Other
Requirements:

CONTRACTOR
COMMUNICATIONS
Upon
approval
of
the
Work
Plan,
the
contractor
shall
maintain
at
least
weekly
communications
with
the
Work
Assignment
Manager
regarding
the
status
of
work
on
the
Work
Assignment.

CONFLICT
OF
INTEREST
The
contractor
must
adhere
to
the
following
requirements:

1)
Upon
receipt
of
a
Work
Assignment,
QRT,
or
similar
tasking
document,
and
prior
to
commencement
of
any
work,
notify
both
the
CO
and
PO
of
any
actual
or
potential
organizational
or
personal
conflicts
of
interest.

2)
Provide
a
written
certification,
within
20
days
of
receipt
of
a
Work
Assignment,
QRT,
or
similar
tasking
document,
that:

a)
Either
all
actual
or
potential
organizational
conflicts
of
interest
have
been
reported
to
the
CO
or
that
no
actual
or
potential
organizational
conflicts
of
interest
exist.
The
contractor
is
directed
to
assure
that
none
of
the
conflicts
disclosed
are
so
direct
and
substantial
as
to
rule
out
a
particular
reviewer.

b)
All
personnel
who
perform
work
under
this
Work
Assignment
or
relating
to
this
Work
Assignment
have
been
informed
of
their
obligation
to
report
personal
and
organizational
conflicts
of
interest
to
the
CO.
Peer
Review
Handbook
Page
D­
8
c)
The
Contractor
recognizes
its
continuing
obligation
to
identify
and
report
any
actual
or
potential
conflicts
of
interest
arising
during
performance
of
this
Work
Assignment.

3)
If
an
actual
or
potential
organizational
conflict
of
interest
is
identified
during
performance
under
this
Work
Assignment,
the
Contractor
shall
immediately
make
a
full
disclosure
in
writing
to
the
CO.
The
disclosure
shall
include
a
description
of
action
which
the
Contractor
has
taken
or
proposes
to
take,
after
consultation
with
the
CO,
to
avoid,
mitigate,
or
neutralize
the
actual
or
potential
conflict
of
interest.

EXPENDITURE
OF
FUNDS/
HOURS
In
addition
to
the
requirements
of
the
contract,
the
contractor
shall
notify
both
the
Project
Officer
and
the
Work
Assignment
Manager
when
75%
of
funds
or
hours
for
this
Work
Assignment
have
been
expended.

INFORMATION
COLLECTION
Any
other
provision
of
this
Work
Assignment
notwithstanding,
the
contractor
shall
not
proceed
with
any
information
collection
where
the
same
or
similar
information
will
be
collected
from
ten
or
more
public
respondents
until
written
approval
is
received
from
the
Contracting
Officer.
This
approval
will
cite
an
approval
number
from
the
Office
of
Management
and
Budget
as
required
by
the
Paperwork
Reduction
Act
(
PRA).

Only
Federal
agencies
and
their
employees
are
exempt
from
the
PRA
definition
of
"
public
respondent."
State
agencies
and
their
employees
are
classified
as
"
public
respondents."

Soliciting
similar
information
applies
to
any
collection
method,
i.
e.,
written,
oral,
electronic,
etc.,
and
utilizing
any
approach,
i.
e.,
surveys,
phone
calls,
focus
groups,
TQM,
etc.
The
PRA
applies
equally
to
"
willing
participants"
and
participation
that
is
mandated
by
law.

Any
question
of
applicability
of
the
PRA
shall
be
resolved
by
submitting
a
complete
description
of
the
circumstances
in
a
written
request
to
the
Contracting
Officer.
No
collection
shall
be
undertaken
until
the
Contracting
Officer
provides
written
notice
to
the
Contractor
as
to
the
applicability
of
the
PRA.
If
the
PRA
is
determined
to
be
applicable,
the
Contractor
shall
not
initiate
any
collection
until
the
requisite
approval
is
received.

The
General
Services
Administration
(
GSA),
under
FIRMR
Bulletin
B­
2
administers
the
Interagency
Reports
Management
Program
as
derived
from
44
U.
S.
C.
Chapters
29
and
31.
All
work
performed
under
this
Work
Assignment
involving
federal
interagency
reporting
must
be
done
in
full
compliance
with
these
GSA
procedures.
Peer
Review
Handbook
Page
D­
9
CONFIDENTIAL
BUSINESS
INFORMATION
If
this
Work
Assignment
requires
use
of
RCRA
Confidential
Business
Information
(
CBI),
the
contract
must
specifically
authorize
the
contractor
to
have
access
to
RCRA
CBI
and
the
contractor
shall
abide
by
all
RCRA
CBI
requirements
and
stipulations
found
in
the
RCRA
CBI
Security
Manual
and
in
the
contract.
The
contractor
shall
identify
in
the
Work
Plan
budget
all
estimated
costs
for
dealing
with
CBI
requirements.
All
CBI
must
be
returned
to
EPA
as
soon
as
it
is
no
longer
needed
under
this
Work
Assignment
or
before
the
expiration
of
the
Work
Assignment,
whichever
occurs
first.

PRINTING
AND
DUPLICATION
The
contractor
is
prohibited
from
performing
any
printing
under
the
Government
Printing
and
Binding
Regulations.
Duplication
is
allowed
to
the
extent
it
does
not
exceed
5,000
impressions
of
a
single­
page
document
or
25,000
impressions
of
a
multiple­
page
stand­
alone
document,
is
limited
to
one
color
(
black)
copies,
and
does
not
exceed
the
maximum
image
size
of
10
3/
4
by
14
1/
4
inches.
For
all
duplication
jobs
in
excess
of
5,000
impressions,
the
EPA
WAM
will
determine
in
advance
if
the
work
can
be
performed
more
cost
effectively
and
under
the
job
or
time
constraints
at
the
EPA
Print
Shop.
If
the
total
number
of
photocopies
for
this
Work
Assignment
exceeds
5,000
impressions,
the
contractor
shall
identify
in
their
Work
Plan
the
photocopying
costs
by
task
and
deliverable.

WORK
ASSIGNMENT/
WORK
PLAN
BUDGETS
The
contractor
shall
not
exceed
either
the
dollar
or
PL
hour
budget
contained
in
the
approved
Work
Plan.
In
addition,
on
Quick
Response
Tasks
(
QRTs)
the
contractor
shall
not
exceed
the
PL
hour
budget
of
the
QRT.

TECHNICAL
DIRECTION
The
Designated
Work
Assignment
Manager
(
WAM)
on
this
Work
Assignment
is
authorized
to
provide
technical
direction
to
the
extent
allowed
under
EPAAR
(
1552.237­
71)
(
APR
1984)
(
DEVIATION).
Other
than
the
Designated
WAM,
only
the
Project
Officer
and
the
Contracting
Officer
are
authorized
to
provide
technical
direction.

Technical
direction
includes:

(
1)
Direction
to
the
contractor
which
assists
the
contractor
in
accomplishing
the
Statement
of
Work
(
2)
Comments
on
and
approval/
acceptance
of
reports
or
other
deliverables
Peer
Review
Handbook
Page
D­
10
Technical
direction
must
be
within
the
contract
and
the
Work
Assignment
statement
of
work.
The
Project
Officer
and
the
WAM
do
not
have
the
authority
to
issue
technical
direction
which
(
1)
institutes
additional
work
outside
the
scope
of
either
the
contract
or
this
Work
Assignment;
(
2)
constitutes
a
change
as
defined
in
the
"
Changes"
clause;
(
3)
causes
an
increase
or
decrease
in
the
estimated
cost
of
the
contract
or
Work
Assignment;
(
4)
alters
the
period
of
performance
or
deliverable
due
dates;
or
(
5)
changes
any
of
the
other
express
terms
or
conditions
of
the
contract
or
Work
Assignment.

Technical
direction
will
be
issued
in
writing
or
confirmed
in
writing
within
five
(
5)
calendar
days
after
verbal
issuance.
The
technical
direction
memorandum
will
be
provided
to
the
contractor
and
copies
will
be
forwarded
to
the
Contracting
Officer
and
the
Project
Officer.
If
the
contractor
has
not
received
written
confirmation
within
five
(
5)
calendar
days
of
a
oral
issuance,
the
contractor
must
so
notify
the
Project
Officer.

INHERENTLY
GOVERNMENTAL
FUNCTIONS
The
contractor
shall
not
perform
any
inherently
governmental
functions
(
IGF)
under
this
Work
Assignment.
If
during
the
course
of
developing
the
plan
of
work,
through
receipt
of
technical
direction,
or
in
carrying
out
the
assignment
any
portion
of
the
effort
is
considered
to
possibly
be
an
inherently
governmental
function,
the
contractor
must
immediately
notify
the
Project
Officer
and
the
Contracting
Officer.

OCCUPATIONAL
HEALTH
AND
SAFETY
Facility
site
visits
conducted
under
a
Work
Assignment
that
include
on­
site
inspections
or
sampling
must
be
conducted
in
full
compliance
with
the
Department
of
Labor,
Occupational
Safety,
and
Health
Administration
rules
under
29
CFR
Part
1910
and
EPA
Order
1440
(
Occupational
Health
and
Safety
Manual).

TRAVEL
COSTS
The
contractor
shall
follow
the
requirements
of
Subpart
31.2
of
the
FAR
and
the
Federal
regulations
in
incurring
allowable
travel
costs
under
this
Work
Assignment,
and
correspondingly
must
at
all
times
seek
and
obtain
Government
rates
whenever
available
and
observe
current
subsistence
ceilings.

QUICK
RESPONSE
TASKS
Each
Quick
Response
Task
(
QRT)
shall
be
confirmed
in
writing
and
approved
by
the
Project
Officer.
The
contractor
shall
respond
by
letter
to
the
PO
with
copies
to
the
WAM
and
the
Peer
Review
Handbook
Page
D­
11
CO
within
two
working
days,
giving
a
brief
description
of
the
plan
of
work,
including
best
estimate
of
hours
(
by
P­
level)
and
a
break­
out
of
costs
to
accomplish
the
task.

No
task
shall
exceed
a
duration
of
30
calendar
days
from
start
date
to
completion
date.
The
level
of
effort
for
each
task
shall
be
limited
to
a
maximum
of
250
labor
hours.

Quick
Response
Task
Requests
do
not
change
the
dollar
or
professional
labor
hour
budgets
of
a
Work
Assignment.
Peer
Review
Handbook
Page
D­
12
STATEMENT
OF
WORK
­
EXAMPLE
3
­
External
Peer
Review
of
Protozoa
Method
Development
Criteria
Document
Period
of
Performance:
Work
Plan
Approval
to
August
1,
1997
Work
Assignment
Manager:
Sally
Q.
Government
Employee
Office
of
Water
U.
S.
Environmental
Protection
Agency
LOE:
196
hours
SOW:
2.4
BACKGROUND
INFORMATION:

The
United
States
Environmental
Protection
Agency
(
EPA),
Office
of
Water
is
charged
with
protecting
public
health
and
the
environment
from
adverse
exposure
to
chemicals
and
microbials
in
water
media,
such
as
ambient
and
drinking
waters,
wastewater/
sewage
sludge
and
sediments.
In
support
of
this
mission
OW's
Office
of
Science
and
Technology
(
OST)
develops
health
standards,
health
criteria,
health
advisories,
and
technical
guidance
documents
for
water
and
water­
related
media.
Under
this
work
assignment,
documents
prepared
by
OST
are
to
undergo
peer
review.

Peer
review
is
an
important
component
of
the
scientific
process.
It
provides
a
focused,
objective
evaluation
of
a
research
proposal,
publication,
risk
assessment,
health
advisory,
guidance
or
other
document
submitted
for
review.
The
criticism,
suggestions
and
new
ideas
provided
by
the
peer
reviewers
stimulate
creative
thought,
strengthens
the
reviewed
document
and
confer
credibility
on
the
product.
Comprehensive,
objective
peer
reviews
leads
to
good
science
and
product
acceptance
within
the
scientific
community.

Under
this
work
assignment,
the
contractor
will
receive
one
document
(
Protozoa
Method
Development
Criteria
Document)
for
peer
review
which
is
related
to
human
health
and
ecological
effects.

STATEMENT
OF
WORK:

Task
1.
The
contractor
shall
develop
a
work
plan
to
address
all
tasks
in
this
work
assignment.
The
work
plan
shall
describe
the
steps
that
will
be
taken
by
the
contractor
to
provide
for
peer
review,
including
selection
of
peer
reviewer
candidates
with
appropriate
expertise,
determining
absence
of
conflict
of
interest,
document
and
reference
distribution,
establishing
schedules,
preparing
the
peer
review
report,
and
submittal
of
the
peer
review
package.
Curriculum
vitae
for
all
Peer
Review
Handbook
Page
D­
13
persons
assigned
to
complete
this
work
assignment
shall
be
provided.
All
P
levels,
hours
and
total
costs
for
each
task
will
be
provided
and
costs
greater
than
$
100.00
shall
be
itemized
in
detail.

Task
2.
The
contractor
shall
select
a
group
of
peer
reviewers
and
determine
their
availability
for
the
task
and
absence
of
conflict
of
interest,
and
establish
a
schedule
for
the
peer
review.
The
contractor
is
directed
to
assure
that
none
of
the
conflicts
disclosed
are
so
direct
and
substantial
as
to
rule
out
a
particular
reviewer.
Three
peer
reviewers
shall
participate
in
the
review.
No
single
peer
reviewer
may
charge
more
than
40
hours
to
this
task.
It
is
fully
acceptable
for
peer
reviewers
to
commit
to
less
than
40
hours.
The
peer
review
will
be
conducted
for
the
Protozoa
Method
Development
Criteria
Document.
Reviewers
selected
by
and
working
for
the
contractor
shall
be
approved
by
the
EPA
Project
Officer
in
writing
prior
to
their
beginning
work.
Minimally,
all
peer
reviewers
shall
be
accomplished
in
protozoan
methods
for
sample
recovery
and
analysis
from
water.
Approval
submissions
shall
include
the
reviewers'
names
and
curriculum
vitae.

Task
3.
The
contractor
shall
arrange
for
the
selected
peer
reviewers
to
review
the
EPA
document.
Prepare
the
charge
to
the
peer
reviewers
based
on
technical
direction
received
from
the
EPA
WAM.
Provide
the
peer
reviewers
with
copies
of
the
candidate
report
and
all
relevant
references
and
instruct
the
selected
peer
reviewers
to
undertake
the
review.
The
WAM
will
provide
the
contractor
with
the
final
version
of
the
document
to
be
reviewed.

Task
4.
The
contractor
shall
monitor
peer
reviewers'
progress
to
assure
timely
completion.
The
contractor
shall
collate
peer
review
comments,
and
organize
the
comments
in
the
peer
review
"
for
comments"
document.
Provide
the
peer
review
document
and
all
materials
submitted
by
the
peer
reviewers
to
the
EPA
WAM.

SCHEDULE
AND
DELIVERABLES:

Task
1.
(
Work
Plan)
15
days
after
receipt
of
work
assignment
Task
2.
1
week
after
work
plan
approval
Task
3.
1
week
after
selection
of
peer
reviewers
Task
4.
1
week
after
receiving
comments
from
the
peer
reviewers
TRAVEL:
No
travel
is
anticipated
under
this
work
assignment.
Any
travel
directly
chargeable
to
this
work
assignment
must
be
submitted
and
approved
by
the
project
officer.
Peer
Review
Handbook
Page
D­
14
THIS
PAGE
LEFT
INTENTIONALLY
BLANK
Peer
Review
Handbook
Page
E­
1
APPENDIX
E
References
Concerning
Peer
Review
American
Chemical
Society
and
the
Conservation
Foundation
(
1985)
Issues
in
Peer
Review
of
the
Scientific
Basis
for
Regulatory
Decisions,
Washington,
DC,
November
1985.

Browner,
C.
(
1994)
Peer
Review
Program,
Washington,
DC,
Memorandum
issued
June
7,
1994.
(
NOTE:
Attached
as
Appendix
A
to
this
Handbook)

Chubin,
D.
(
1994)
Grants
peer
review
in
theory
and
practice,
Evaluation
Review
18:
20­
30.

Chubin,
D.
and
E.
Hackett
(
1990)
Peerless
Science:
Peer
Review
and
Science
Policy,
Albany,
NY:
State
University
of
New
York
Press.

Jasanoff,
S.
(
1990)
The
Fifth
Branch:
Science
Advisors
as
Policymakers,
Cambridge,
MA:
Harvard
University
Press.

Kostoff,
R.
(
1996)
Peer
Review
in
Selected
Federal
Agencies,
presented
at
AAAS
Annual
Meeting,
Baltimore,
MD,
February
9,
1996.

National
Environmental
Policy
Institute
(
1996)
Enhancing
the
Integrity
and
Transparency
of
Science
in
the
Regulatory
Process,
Washington,
DC:
National
Environmental
Policy
Institute,
Fall
1996.

National
Research
Council
(
1995)
Interim
Report
of
the
Committee
on
Research
and
Peer
Review
in
EPA,
Washington,
DC:
National
Academy
Press,
March
1995.

Reilly,
W.
(
1993)
Peer­
review
Policy,
Washington,
DC,
Memorandum
issued
January
19,
1993.

Science
Advisory
Board
(
1992)
Safeguarding
the
Future:
Credible
Science,
Credible
Decisions,
Washington,
DC,
SAB
Report
issued
March
1992.

Spitzer,
H.
(
1995)
Peer
Review
Practices
in
the
Federal
Government,
Bethesda,
MD:
Environmental
Network,
report
prepared
for
the
American
Industrial
Health
Council,
April
26,
1995)
Peer
Review
Handbook
Page
E­
2
U.
S.
General
Accounting
Office
(
1994)
Peer
Review:
EPA
Needs
Implementation
Procedures
and
Additional
Controls,
GAO/
RCED­
94­
89,
Washington,
DC:
U.
S.
Government
Printing
Office,
February
1994.

U.
S.
General
Accounting
Office
(
1996)
Peer
Review:
EPA's
Implementation
Remains
Uneven,
GAO/
RCED­
96­
236,
Washington,
DC:
U.
S.
Government
Printing
Office,
September
1996.
Peer
Review
Handbook
Notes
and
Comments
Peer
Review
Handbook
Notes
and
Comments
Peer
Review
Handbook
Notes
and
Comments
Peer
Review
Handbook
Notes
and
Comments
