United
States
Office
of
Science
Policy
Environmental
Protection
Office
of
Research
and
Development
December
2000
Agency
Washington,
DC
20460
www.
epa.
gov
EPA
100­
B­
00­
001
Science
Policy
Council
HANDBOOK
2nd
Edition
EPA
100­
B­
00­
001
December
2000
U.
S.
Environmental
Protection
Agency
PEER
REVIEW
HANDBOOK
2nd
Edition
Prepared
for
the
U.
S.
Environmental
Protection
Agency
by
members
of
the
Peer
Review
Advisory
Group,
a
group
of
EPA s
Science
Policy
Council
Principal
Authors
Kerry
L.
Dearfield,
Ph.
D.
A.
Robert
Flaak
Office
of
Science
Policy
EPA
Science
Advisory
Board
Office
of
Research
and
Development
Office
of
the
Administrator
Major
Contributors
Jean
C.
Schumann
Office
of
Solid
Waste
and
Emergency
Response
Roland
B.
Hemmett,
Ph.
D.
Region
2
Brett
Snyder
Nancy
W.
Wentworth
Office
of
Environmental
Information
Arnold
M.
Kuzmack,
Ph.
D.
Office
of
Water
Office
of
Policy,
Economics,
and
Innovation
Science
Policy
Council
U.
S.
Environmental
Protection
Agency
Washington,
DC
20460
Page
ii
Peer
Review
Handbook
DISCLAIMER
This
document
has
been
reviewed
in
accordance
with
U.
S.
Environmental
Protection
Agency
policy
and
approved
for
publication
and
distribution
to
the
Agency.
Mention
of
trade
names
or
commercial
products
does
not
constitute
endorsement
or
recommendation
for
use.
Peer
Review
Handbook
Page
iii
TABLE
OF
CONTENTS
FOREWORD..........................................................
Page
xiii
ACKNOWLEDGMENTS
................................................
Page
xvii
SUMMARY
OF
THE
PEER
REVIEW
PROCESS
..............................
Page
1
Figure
1
­
Flowchart
for
Planning
a
Peer
Review
..........................
Page
2
Figure
2
­
Flowchart
for
Conducting
a
Peer
Review
........................
Page
3
Figure
3
­
Flowchart
for
Completing
a
Peer
Review........................
Page
4
Manager s
Planning
Checklist
for
Peer
Review
...........................
Page
5
PEER
REVIEW
GUIDANCE
...............................................
Page
7
1.
THE
NEED
FOR
PEER
REVIEW.........................................
Page
9
1.1
Overview
Statement...........................................
Page
9
1.2
Understanding
Peer
Review.....................................
Page
9
1.2.1
Why
use
Peer
Review?
..................................
Page
9
1.2.2
What
is
Peer
Involvement?
..............................
Page
10
1.2.3
What
is
Peer
Review?
..................................
Page
10
1.2.4
What
is
Peer
Input?
....................................
Page
10
1.2.5
How
is
Peer
Review
Different
from
Peer
Input?
..............
Page
11
1.2.6
Can
Someone
Who
Provided
Peer
Input
Become
an
Independent
Peer
Reviewer
for
the
Same
Work
Product
Later
in
the
Process?
........................................
Page
11
1.2.7
Can
the
Same
Peer
Reviewer
be
Used
More
than
Once
if
a
Product
will
Be
Peer
Reviewed
More
than
Once,
and
can
the
Same
Peer
Reviewer
be
Used
Again
and
Again
for
Different
Products?
............................................
Page
12
1.2.8
How
is
Peer
Review
Different
from
Public
Comment?
.........
Page
12
1.2.9
How
is
Peer
Review
Different
from
Stakeholder
Involvement?
..
Page
12
1.2.10
What
Role
does
Peer
Review
have
in
the
Regulatory
Development
Process?..................................
Page
13
1.2.11
What
Role
does
Peer
Review
have
in
Regulatory
Negotiations?
.
Page
14
1.3
Annual
Agency
Reporting
Requirements
.........................
Page
14
1.3.1
What
are
the
Annual
Reporting
Requirements?
..............
Page
14
1.3.2
What
Listings
are
Required
for
the
Annual
Reporting?
........
Page
14
1.3.3
When
will
the
Handbook
Itself
be
Revised?
.................
Page
17
Page
iv
Peer
Review
Handbook
1.4
The
Roles
of
People
and
Organizations
in
Peer
Review
..............
Page
17
1.4.1
Who
is
Ultimately
Accountable
for
Peer
Review?
............
Page
17
1.4.2
Who
are
the
Agency
Staff
involved
in
Peer
Review?
..........
Page
17
1.4.3
Who
are
the
Decision
Makers
&
What
are
Their
Responsibilities?
......................................
Page
17
1.4.4
Who
are
the
Peer
Review
Leaders
&
What
are
Their
Responsibilities?
......................................
Page
19
1.4.5
Who
are
the
Peer
Review
Coordinators
&
What
are
Their
Responsibilities?
......................................
Page
20
1.4.6
Who
are
the
Peer
Reviewers?
............................
Page
21
1.4.7
What
are
the
Responsibilities
of
Peer
Reviewers?
............
Page
21
1.4.8
What
is
an
Independent
Peer
Reviewer?
....................
Page
21
1.4.9
When
does
an
Agency
Internal
Peer
Reviewer
Qualify
as
Independent?
.........................................
Page
22
1.4.10
What
is
a
Peer
Review
Panel?
............................
Page
22
1.4.11
What
is
a
Subject
Matter
Expert?
.........................
Page
22
1.4.12
What
is
the
Role
of
the
Science
Policy
Council
(
SPC)?
........
Page
23
1.4.13
What
is
the
Role
of
the
Peer
Review
Advisory
Group
(
PRAG)?
.
Page
23
1.4.14
What
is
the
Role
of
the
Office
of
Research
and
Development
(
ORD)?
..................................
Page
23
2.
PLANNING
A
PEER
REVIEW..........................................
Page
25
2.1
Overview
Statement..........................................
Page
25
2.2
Determining
Which
Work
Products
to
Peer
Review
.................
Page
25
2.2.1
What
are
Scientific
and
Technical
Work
Products?
...........
Page
25
2.2.2
What
Scientific
and
Technical
Work
Products
Need
Peer
Review?
.............................................
Page
26
2.2.3
How
Does
One
Determine
Whether
a
Scientific
and/
or
Technical
Work
Product
is
 
Major ?
......................
Page
26
2.2.4
What
Economic
Work
Products
Need
Peer
Review?
..........
Page
27
2.2.5
Should
Economic
Work
Products
Prepared
in
Support
of
Regulations
that
are
Classified
as
"
Major"
or
"
Economically
Significant"
be
Peer
Reviewed?...........................
Page
28
2.2.6
What
Other
Economic
Work
Products
Might
Benefit
from
Peer
Review?
.........................................
Page
29
2.2.7
What
Other
Social
Science
Work
Products
Need
Peer
Review?
..
Page
30
2.2.8
How
Should
Peer
Review
be
Handled
for
Products
Developed
under
an
Interagency
Agreement
(
IAG)?....................
Page
31
2.2.9
Should
Products
from
Contracts,
Grants,
and
Cooperative
Agreements
Receive
Peer
Review?
........................
Page
31
Peer
Review
Handbook
Page
v
2.2.10
How
Does
Peer
Review
Apply
to
Products
Generated
Through
EPA
Grants
or
Cooperative
Agreements?
...................
Page
31
2.2.11
Can
the
Recipient
of
a
Grant
or
Cooperative
Agreement
Use
Agreement
Funds
to
Pay
Peer
Reviewers
of
their
Work
Products?
Page
34
2.2.12
Do
Products
Generated
under
EPA
Grants
or
Cooperative
Agreements
Need
to
be
Reported
in
the
Peer
Review
Product
Tracking
(
PRPT)
Database?
.............................
Page
34
2.2.13
Should
Site
Specific
Decisions
be
Subject
to
Peer
Review?
.....
Page
35
2.2.14
Should
NEPA
Products
(
e.
g.,
EISs)
be
Subject
to
Peer
Review?
.
Page
35
2.2.15
Should
Environmental
Regulatory
Models
be
Peer
Reviewed?
..
Page
36
2.2.16
Is
Peer
Review
Needed
for
Other
Organization s
Work
Products
that
Have
been
Submitted
to
EPA
for
Use
in
Decision
Making?
.
Page
36
2.2.17
Can
Work
Products
That
are
Not
Determined
to
be
Major
Still
be
Peer
Reviewed?.....................................
Page
37
2.3
Determining
Which
Work
Products
Do
Not
Receive
Peer
Review
......
Page
37
2.3.1
Are
There
Circumstances
When
a
Major
Work
Product
is
Not
Peer
Reviewed?
.......................................
Page
37
2.3.2
What
Products
Normally
Do
Not
Need
Peer
Review?
.........
Page
38
2.3.3
Do
Voluntary
Consensus
Standards
Require
Peer
Review?
.....
Page
39
2.4
Choosing
a
Peer
Review
Mechanism.............................
Page
39
2.4.1
How
Do
You
Determine
the
Appropriate
Peer
Review
Mechanism?..........................................
Page
39
2.4.2
What
are
Examples
of
Internal
Peer
Review?
................
Page
41
2.4.3
What
are
Examples
of
External
Peer
Review?
...............
Page
41
2.4.4
What
is
the
Role
of
Peer
Review
by
a
Refereed
Scientific
Journal?
.............................................
Page
42
2.4.5
Do
Agency
Work
Products
Become
Candidates
for
Peer
Review
when
Peer
Reviewed
Journal
Articles
are
Used
in
Support
of
that
Work
Product?
...........................
Page
43
2.4.6
When
and
How
Often
Should
Peer
Review
Occur?
...........
Page
43
2.4.7
What
Factors
are
Considered
in
Setting
the
Time
Frame
for
Peer
Review?
.........................................
Page
44
2.4.8
Which
Office/
Region
or
Other
Agency
is
Responsible
for
Conducting
the
Peer
Review?
............................
Page
45
2.5
Creating
the
Peer
Review
Record
...............................
Page
45
2.5.1
What
is
the
Peer
Review
Record?
.........................
Page
45
2.5.2
How
Can
the
Peer
Review
Record
Improve
the
Peer
Review
Process?
......................................
Page
45
2.5.3
What
Should
Be
in
the
Peer
Review
Record?
................
Page
45
2.5.4
What
Should
I
Do
with
a
Peer
Review
Record
That
Pertains
to
a
Rulemaking
Action?
...................................
Page
46
Page
vi
Peer
Review
Handbook
2.5.5
When
Should
the
Peer
Review
Record
Building
Process
Begin?
.
Page
46
2.5.6
What
are
the
Differences
in
Record
Keeping
for
a
Review
by
an
Individual
Compared
to
a
Panel?..........................
Page
47
2.5.7
Where
Should
the
Peer
Review
Record
be
Kept
and
For
How
Long?...........................................
Page
47
2.5.8
Are
Internal
Peer
Review
Comments
Included
in
the
Peer
Review
Record?.......................................
Page
47
2.6
Budget
Planning.............................................
Page
48
2.6.1
What
Budgetary
Factors
Should
I
Consider
in
a
Peer
Review?
...
Page
48
2.6.2
What
Input
is
Needed
for
the
Annual
Budget
Formulation
and
Budget
Execution
Process?
..............................
Page
48
2.7
Legal
Considerations
.........................................
Page
49
2.7.1
Are
There
Legal
Ramifications
From
the
Peer
Review
Policy?
..
Page
49
2.7.2
Is
Legal
Advice
Needed?................................
Page
49
2.8
Federal
Advisory
Committee
Act
(
FACA)
Considerations
............
Page
49
2.8.1
When
Do
FACA
Requirements
Apply
to
EPA­
Run
Peer
Reviews?
........................................
Page
50
2.8.2
When
Are
EPA­
Run
Peer
Reviews
Not
Subject
to
FACA?
.....
Page
50
2.8.3
How
Do
I
Ensure
that
a
Contractor­
Run
Peer
Review
Does
Not
Become
Subject
to
FACA?...........................
Page
51
3.
CONDUCTING
A
PEER
REVIEW.......................................
Page
53
3.1
Overview
Statement..........................................
Page
53
3.2
Charge
to
the
Peer
Reviewers
..................................
Page
53
3.2.1
What
is
a
Charge?
.....................................
Page
53
3.2.2
What
are
the
Essential
Elements
of
a
Charge?
...............
Page
53
3.2.3
Where
Can
I
Get
an
Example
of
a
Charge?..................
Page
54
3.2.4
Can
a
Stakeholder
Provide
Input
to
the
Charge
to
the
Peer
Reviewers?...........................................
Page
54
3.2.5
Who
Writes
the
Charge
When
I
Hire
a
Contractor
to
Conduct
the
Peer
Review?
......................................
Page
54
3.2.6
Is
it
Okay
For
Me
to
Ask
a
Contractor
to
Develop
the
Charge
to
the
Peer
Reviewers?..................................
Page
55
3.3
Time
Line..................................................
Page
55
3.3.1
What
are
the
Factors
in
Scheduling
a
Peer
Review?
...........
Page
55
3.4
Selection
of
Peer
Reviewers
...................................
Page
55
3.4.1
What
are
Considerations
for
Selecting
Peer
Reviewers?
........
Page
55
3.4.2
Where
Do
I
Find
Peer
Reviewers?.........................
Page
56
3.4.3
Are
External
or
Internal
Peer
Reviewers
Preferred?
...........
Page
57
3.4.4
What
is
Important
in
the
Mix
of
a
Peer
Review
Panel?
.........
Page
57
3.4.5
What
is
a
Conflict
of
Interest?
............................
Page
58
Peer
Review
Handbook
Page
vii
3.4.6
What
Techniques
Help
Ensure
Disclosure
and
Appropriate
Resolution
of
Conflicts
of
Interest?
........................
Page
59
3.4.7
Can
Parties
External
to
EPA
Pay
for
Their
Own
Peer
Reviews?
.
Page
61
3.4.8
Are
There
Constraints
to
Selecting
Peer
Reviewers?
..........
Page
61
3.4.9
If
State
Employees
are
Used
as
Peer
Reviewers,
Can
EPA
Pay
Them
for
this
Service?
..............................
Page
62
3.5
Materials
for
Peer
Reviewers...................................
Page
62
3.5.1
What
Instructions
Do
You
Give
Peer
Reviewers?
.............
Page
62
3.5.2
What
Materials
Should
be
Sent
to
Peer
Reviewers?
...........
Page
63
3.5.3
How
Closely
can
EPA
Interact
with
Peer
Reviewers
During
the
Review?
..........................................
Page
64
3.6
Peer
Review
Services.........................................
Page
65
3.6.1
What
are
Gratuitous
Services
for
Peer
Review?
..............
Page
65
3.6.2
Can
I
Use
a
Contract
to
Obtain
Peer
Review
Services?
........
Page
66
3.6.3
How
Do
I
Write
a
Statement
of
Work
for
Contracts?
..........
Page
66
3.6.4
What
are
Advisory
and
Assistance
Services
(
AAS)
or
Sensitive
Activities?
...........................................
Page
67
3.6.5
What
are
Some
Management
Controls
for
Contracts?
.........
Page
67
3.6.6
Can
I
Identify
and/
or
Select
Peer
Reviewers
When
Using
a
Contract?...........................................
Page
70
3.6.7
Can
I
Use
Simplified
Acquisition
Procedures
to
Obtain
Peer
Reviewers?...........................................
Page
70
3.6.8
How
is
Travel
Handled
with
Contracts
or
Purchase
Orders?
....
Page
72
3.6.9
How
is
Travel
Handled
with
Special
Government
Employees?
..
Page
72
4.
COMPLETING
A
PEER
REVIEW
.......................................
Page
73
4.1
Overview
..................................................
Page
73
4.2
Final
Work
Product
..........................................
Page
73
4.2.1
How
Do
I
Incorporate
Peer
Review
Comments
into
the
Final
Work
Product?
...................................
Page
73
4.2.2
What
Actions
are
Potentially
Forthcoming
from
Peer
Review?
..
Page
74
4.2.3
What
Should
the
Final
Work
Product
Say
About
the
Peer
Review
Process?
......................................
Page
74
4.2.4
Can
the
Identity
of
Peer
Reviewers
be
Kept
Anonymous?
......
Page
75
4.3
Completing
the
Peer
Review
Record.............................
Page
75
4.3.1
How
Do
I
Complete
the
Peer
Review
Record?
...............
Page
75
4.3.2
Where
Should
the
Peer
Review
Records
be
Kept,
and
for
How
Long?...........................................
Page
76
4.3.3
Is
Information
Regarding
a
Peer
Review
Subject
to
Release
under
FOIA?
.........................................
Page
77
Page
viii
Peer
Review
Handbook
SUBJECT
INDEX
.......................................................
Page
79
COMMONLY
USED
ACRONYMS.........................................
Page
81
APPENDIX
A
­
U.
S.
ENVIRONMENTAL
PROTECTION
AGENCY
PEER
REVIEW
POLICY................................................
Page
A­
1
APPENDIX
B
­
SOUND
SCIENCE
AND
PEER
REVIEW
IN
RULEMAKING
.....
Page
B­
1
APPENDIX
C
­
EXAMPLES
OF
CHARGES
................................
Page
C­
1
CHARGE
EXAMPLE
1
­
Framework
for
Developing
a
Living
Resources
Research
and
Monitoring
Plan
for
the
Peconic
Estuary
..............
Page
C­
2
CHARGE
EXAMPLE
2
­
EPA
Science
Advisory
Board
(
SAB)
Review
of
the
Environment
Monitoring
and
Assessment
Program
(
EMAP)
Research
Strategy
and
Research
Plan
...........................
Page
C­
5
CHARGE
EXAMPLE
3
­
EPA
Science
Advisory
Board
Review
of
the
Agency's
National
Risk
Management
Research
Laboratory s
(
NRMRL)
Program
.........................................
Page
C­
6
CHARGE
EXAMPLE
4
­
EPA
Science
Advisory
Board
(
SAB)
Review
of
the
Technical
Aspects
of
the
Multi­
Agency
Radiation
Survey
and
Site
Investigation
Manual
(
MARSSIM)
.............................
Page
C­
7
CHARGE
EXAMPLE
5
­
Economics
­
Benefits
Transfer
from
Adults
to
Children
................................................
Page
C­
8
CHARGE
EXAMPLE
6
­
Economics
­
Study
of
Municipal
Government
Costs
and
Financial
Impacts
from
Environmental
Regulations
........
Page
C­
9
CHARGE
EXAMPLE
7
­
Economics
­
Valuation
of
Fatal
Cancer
Risks
.....
Page
C­
11
CHARGE
EXAMPLE
8
­
Hudson
River
PCBs
Site
Reassessment
RI/
FS
Modeling
Approach
Charge:
Peer
Review
1
.....................
Page
C­
15
CHARGE
EXAMPLE
9
­
IRIS
Pilot
Program
­
Instructions
to
Peer
Reviewers
for
Reviewing
IRIS
Summaries
and
Supporting
Documentation............................................
Page
C­
18
CHARGE
EXAMPLE
10
­
Charge
to
Reviewers
for
the
WTI
Draft
Final
Risk
Assessment
..........................................
Page
C­
21
APPENDIX
D
­
GUIDANCE
ON
REQUESTING
A
REVIEW
BY
THE
US
EPA
SCIENCE
ADVISORY
BOARD
(
SAB)
..........................
Page
D­
1
APPENDIX
E
­
EXAMPLE
STATEMENTS
OF
WORK
FOR
CONTRACTS
......
Page
E­
1
STATEMENT
OF
WORK
­
EXAMPLE
1
­
Statement
of
Work:
Technical
Review
Contractor
for
Panel
Review
of
Assistance
Agreement
or
Fellowship
Applications
.....................................
Page
E­
2
Peer
Review
Handbook
Page
ix
STATEMENT
OF
WORK
­
EXAMPLE
2
­
Peer
Review
of
Prioritization
Tool
Report
...............................................
Page
E­
5
STATEMENT
OF
WORK
­
EXAMPLE
3
­
External
Peer
Review
of
Protozoa
Method
Development
Criteria
Document
...............
Page
E­
12
APPENDIX
F
­
USEFUL
FORMS
.........................................
Page
F­
1
PEER
REVIEW
CONFLICT
OF
INTEREST
INQUIRY
..................
Page
F­
2
PEER
REVIEW
CHECKLIST
FOR
DETERMINING
WHETHER
A
WORK
PRODUCT
NEEDS
PEER
REVIEW
........
Page
F­
3
PEER
REVIEW
CHECKLIST
FOR
CONDUCTING
A
PEER
REVIEW
.....
Page
F­
7
VOLUNTEER
SERVICE
PROGRAM
PARTICIPATION
AGREEMENT
...
Page
F­
14
APPENDIX
G
.........................................................
Page
G­
1
REFERENCES
CONCERNING
PEER
REVIEW
.......................
Page
G­
1
Peer
Review
Handbook
Page
xi
FOREWORD
(
2nd
Edition)

It
has
been
three
years
since
EPA's
Science
Policy
Council
(
SPC)
issued
the
1st
Edition
of
the
Peer
Review
Handbook
(
January
1998).
The
original
Handbook
has
been
used
extensively
by
personnel
throughout
the
Agency
for
the
peer
review
of
Agency
work
products.
At
the
time
it
was
issued,
we
planned
a
two
to
three­
year
period
during
which
Agency
staff
could
use
the
Handbook
and
provide
feedback
on
its
utility,
and
at
the
same
time,
identify
questions
or
issues
not
originally
addressed.

To
date,
we
have
received
numerous
comments
and
suggestions
from
regional
and
headquarters
personnel
concerning
the
Handbook.
These
comments
have
been
uniformly
positive
and
encouraging.
The
comments
indicate
that
the
Handbook
contributed
greatly
to
the
Agency
goal
of
sound
science
and
substantially
improving
EPA's
peer
review
process.

We
have
collected
the
comments
and
additional
questions,
discussed
them
with
the
SPC's
Peer
Review
Advisory
Group
and
Agency
Peer
Review
Coordinators,
and
developed
revised
and
new
answers
to
the
questions
received.
In
addition,
we
also
incorporated
many
revisions
in
response
to
recommendations
from
the
EPA
Science
Advisory
Board
(
SAB;
from
"
An
SAB
Report:
Review
of
the
Peer
Review
Program
of
the
Environmental
Protection
Agency"
a
review
by
the
Research
Strategies
Advisory
Committee
(
RSAC)
of
the
SAB;
report
#
EPA­
SAB­
RSAC­
00­
002
dated
November
1999),
the
EPA
Office
of
the
Inspector
General
(
OIG;
from
"
EPA's
Selection
of
Peer
Reviewers"
;
report
#
1999­
P­
217
dated
September
29,
1999),
and
the
National
Research
Council
(
NRC;
from
"
Strengthening
Science
at
the
U.
S.
Environmental
Protection
Agency:
Research
Management
and
Peer
Review
Practices";
report
issued
2000).
After
revising
the
Handbook
,
the
revisions
were
circulated
extensively
throughout
the
Agency
for
final
comment
before
the
SPC
approved
the
2nd
Edition
of
the
Peer
Review
Handbook
at
their
October
17,
2000
meeting.

While
we
strived
to
make
the
Peer
Review
Handbook
a
definitive
source
of
information
about
peer
review
and
its
processes,
we
need
to
sound
one
precautionary
note.
During
the
revision
process,
we
heard
many
requests
to
put
as
much
detail
as
possible
into
the
Handbook
to
make
it
a
one
stop
source
for
all
things
related
to
peer
review.
We
have
attempted
to
do
this
as
much
as
possible;
however,
much
of
the
detail
in
several
instances
delves
into
areas
for
which
there
is
specialized
expertise
elsewhere
in
the
Agency,
particularly
contracting
and
legal
issues.
If
we
provide
all
the
detail
requested
in
many
of
these
areas,
the
Handbook
would
turn
into
a
contract
manual
or
legal
book,
something
we
did
not
want
to
do.
Since
many
of
these
processes
(
like
contracting)
are
updated
frequently,
we
did
not
want
to
outdate
the
Handbook
unnecessarily.
Also,
we
felt
if
the
Handbook
became
much
bigger
than
it
is
now,
its
utility
and
Page
xii
Peer
Review
Handbook
ease
of
use
would
decrease.
We
have
provided
pointers
and
contacts
where
necessary
in
the
Handbook
to
help
Agency
people
deal
with
these
specific
issues
in
relation
to
peer
review.

We
are
proud
to
bring
the
2nd
Edition
to
you
for
your
use
in
peer
review
at
EPA.
The
Handbook
represents
a
large
Agency
collective
effort
to
improve
its
science
to
the
highest
levels
possible.
Peer
review
done
correctly
significantly
adds
to
our
goal
of
sound
and
credible
science
that
underlays
all
Agency
decisions
and
actions.

Kerry
Dearfield
and
Robert
Flaak
Co­
Chairs,
Peer
Review
Advisory
Group
(
PRAG)
Science
Policy
Council
Science
Policy
Council
Norine
Noonan,
ORD,
Vice­
Chair
Robert
Brenner,
OAR
Steven
Galson,
OPPTS
Albert
McGartland,
OPEI
Michael
Ryan,
OCFO
Elaine
Stanley,
OEI
W.
Michael
McCabe,
OA,
Chair
Donald
Barnes,
OA
Tudor
Davies,
OW
Sylvia
Lowrance,
OECA
William
Muszynski,
Region
2
Michael
Shapiro,
OSWER
Ramona
Trovato,
OCHP
Science
Policy
Council
Steering
Committee
Donald
Barnes,
OA
Reginald
Cheatham,
OEI
William
Farland,
ORD
Penelope
Fenner­
Crisp,
OPPTS
Jerri­
Anne
Garl,
Region
5
Roland
Hemmett,
Region
2
Carl
Mazza,
OAR
Jennifer
Orme­
Zavaleta,
ORD
Larry
Reed,
OSWER
Rosemarie
Russo,
Region
4
Mary
Ellen
Weber,
OPPTS
William
Wood,
ORD
Michael
Brody,
OCFO
Patricia
Cirone,
Region
10
Michael
Feldman,
OCFO
Michael
Firestone,
OCHP
Peter
Grevatt,
OSWER
Kate
Mahaffey,
OPPTS
James
Nelson,
OGC
Peter
Preuss,
ORD
Joseph
Reinert,
OPEI
Vanessa
Vu,
ORD
Jeanette
Wiltse,
OW
Science
Policy
Council
Staff
Edward
Bender
Kerry
Dearfield
James
Rowe
Peer
Review
Handbook
Page
xiii
FOREWORD
(
From
1st
Edition)

EPA s
Science
Policy
Council
(
SPC)
has
organized
this
Peer
Review
Handbook
as
guidance
to
EPA
staff
and
managers
on
the
organization
and
conduct
of
peer
review
pursuant
to
the
Administrator s
June
7,
1994
Peer
Review
Policy
statement.
The
Handbook
is
based
in
part
on
the
central
themes
set
forth
in
the
Policy
statement
(
see
Appendix
A
for
the
full
policy):

Major
scientifically
and
technically
based
work
products
related
to
Agency
decisions
normally
should
be
peer
reviewed.
Agency
managers
within
Headquarters,
Regions,
laboratories,
and
field
components
determine
and
are
accountable
for
the
decision
whether
to
employ
peer
review
in
particular
instances
and,
if
so,
its
character,
scope,
and
timing.
These
decisions
are
made
in
conformance
with
program
goals
and
priorities,
resource
constraints,
and
statutory
or
court­
ordered
deadlines.
For
those
work
products
that
are
intended
to
support
the
most
important
decisions
or
that
have
special
importance
in
their
own
right,
external
peer
review
is
the
procedure
of
choice.
Peer
review
is
not
restricted
to
the
penultimate
version
of
work
products;
in
fact,
peer
review
at
the
planning
stage
can
often
be
extremely
beneficial.

In
addition,
the
Handbook
augments
these
themes
by
stating
and
explaining
widely­
accepted
principles
and
practices
that
have
long
guided
peer
review
in
the
universities,
in
private
research
organizations,
and
at
the
EPA
and
other
government
agencies.

The
goal
of
the
Peer
Review
Policy
and
this
Handbook
is
to
enhance
the
quality
and
credibility
of
Agency
decisions
by
ensuring
that
the
scientific
and
technical
work
products
underlying
these
decisions
receive
appropriate
levels
of
peer
review
by
independent
scientific
and
technical
experts.
To
serve
this
goal,
the
Handbook
provides
information
and
outlines
procedures
in
several
different
areas:
The
Policy
requires
peer
review
of
the
basis
of
the
decision
(
i.
e.,
the
underlying
major
scientific
and/
or
technical
work
products),
not
the
decision
itself.

P
basic
principles
and
definitions,
including
distinctions
between
peer
review
and
peer
input,
public
comment,
and
stakeholder
involvement;

P
preparing
for
peer
review,
including
identifying
work
products,
identifying
appropriate
peer
review
mechanisms,
and
identifying
qualified
experts;
and,

P
conducting
and
completing
peer
reviews,
including
materials
required
for
peer
review,
creating
a
peer
review
record,
and
utilizing
peer
review
comments.
Page
xiv
Peer
Review
Handbook
This
Handbook
has
three
parts.
The
first
contains
flow
charts
that
outline
the
key
steps
in
conducting
a
peer
review,
along
with
a
managers 
checklist
for
planning
peer
reviews.
The
second
part
contains
peer
review
guidance
detailing
the
procedures
outlined
in
the
flow
charts
in
a
question
and
answer
format.
The
third
part
contains
Appendices
including
the
1994
Peer
Review
Policy
and
examples
to
help
perform
quality
peer
reviews.
Some
procedures
outlined
in
the
1994
Policy
have
been
completed
while
others
are
continuing
to
change
in
line
with
Agency
experience.
These
changes
have
been
incorporated
into
the
current
Handbook.

Dorothy
E.
Patton,
Ph.
D.
Executive
Director,
Science
Policy
Council
Peer
Review
Handbook
Page
xv
ACKNOWLEDGMENTS
Compiling
a
comprehensive
handbook
that
incorporates
the
most
relevant,
as
well
as
some
of
the
obscure
policies
and
guidance,
regarding
peer
review
and
its
application
throughout
EPA
has
been
a
herculean
task
involving
not
just
the
members
of
the
Peer
Review
Advisory
Group,
but
many
others
as
well.

We
want
to
acknowledge
the
contributions
of
the
headquarters
and
regional
Peer
Review
Coordinators
and
their
respective
office
and
division
coordinators,
the
Science
Policy
Council
(
SPC)
and
its
Steering
Committee,
the
Office
of
Environmental
Information s
(
OEI s)
Quality
Assurance
Staff,
Office
of
Science
Policy
(
OSP)
Program
Support
Staff,
and
the
many
other
individuals
who
provided
comments
on
the
revisions
to
the
current
edition.

In
particular,
we
want
to
acknowledge
the
following
individuals
who
provided
critical
expertise
and
timely
revisions
(
often
several
times)
to
various
sections
of
the
Peer
Review
Handbook:
Hale
Hawbecker
(
Office
of
General
Counsel),
Theresa
Trainor
(
Office
of
Water),
and
Bruce
Bakaysa
(
Office
of
Acquisition
Management).

Finally,
we
want
to
acknowledge
the
tireless
efforts
and
mentorship
of
the
recently
retired
Executive
Director
of
the
Science
Policy
Council,
Dr.
Dorothy
Patton.
She
is
one
of
the
pillars
of
the
Agency s
efforts
to
improve
the
quality
of
science
and
peer
review
and
we
are
indebted
to
her.
Peer
Review
Handbook
Page
1
U.
S.
Environmental
Protection
Agency
SUMMARY
OF
THE
PEER
REVIEW
PROCESS
This
Section
of
the
Peer
Review
Handbook
contains
flowcharts
and
descriptions
of
the
major
steps
in
conducting
a
peer
review.
Cross
references
to
the
appropriate
section
on
Peer
Review
Guidance
are
shown
in
parenthesis
and
bolded.

The
Managers
Planning
Checklist
for
Peer
Review
(
on
page
5)
is
designed
to
give
Managers/
Decision
Makers
a
simple
tool
to
help
plan
for
a
successful
peer
review.
It
asks
questions
that
should
be
considered
by
a
manager
or
Decision
Maker
during
the
peer
review
process
to
insure
that
necessary
actions
are
taking
place.
This
checklist
is
also
intended
to
be
used
by
staff
(
especially
Peer
Review
Leaders
and
Coordinators)
to
inform
managers
and
Decision
Makers
on
some
of
the
key
steps
and
considerations
that
are
necessary
in
carrying
out
a
successful
peer
review.
Expanded
Check
Lists
which
can
be
adapted
for
your
use
are
also
included
in
Appendix
F
­
Useful
Forms
(
courtesy
of
Region
V).
Page
2
Peer
Review
Handbook
Figure
1
­
Flowchart
for
Planning
a
Peer
Review
1.
Determination
of
 
major
scientific
and
technical
work
product :
­­
It
is
a
scientific,
engineering,
economic,
or
statistical
document
(
§
2.2.1)
­­
Determine
if
the
work
pro
duct
is
major
(
§
2.2.3,
2.2.4,
2.2.7)
­­
Major
p
roducts
m
eet
certain
criter
ia
(
§
2.2.3)

2.
Work
product
is
a
candidate
for
peer
review
:
­­
Major
work
products
are
subject
to
peer
review
(
§
2.2.2)
­­
Some
non­
major
work
products
have
to
be
evaluated
to
determine
if
peer
review
is
still
warranted
(
§
2.2.17)

3.
Work
p
roduct
is
not
a
candidate
for
peer
review
:
­­
Most
non­
major
work
products
are
typically
not
candidates
for
peer
review
(
§
2.3.2)
­­
Major
work
p
roduct
consists
only
of
science
previously
peer
reviewed
and
adequate
under
the
Agency s
Policy
(
§
2.3.1)
­­
Place
wor
k
produc
t
on
List
C
(
§
1.3.2c))

4.
If
a
work
pro
duct
is
subje
ct
to
peer
re
view:
­­
Identify
basis
for
charge
(
§
3.2.1)
­­
Identify
key
staff
(
§
1.4)
­­
Create
a
peer
review
record
(
§
2.5)
­­
Ensure
source
of
funding
for
the
peer
review
(
§
2.6)
­­
Determine
overall
time
frame
for
peer
review
(
§
3.3.1)
­­
Place
the
work
product
on
List
B
(
§
1.3.2b))
Peer
Review
Handbook
Page
3
Figure
2
­
Flowchart
for
Conducting
a
Peer
Review
1.
Develop
the
charge
(
§
3.2):
­­
Determine
which
key
issues
to
add
ress
­­
Include
in
peer
review
record
(
§
2.5.3)

2.
Select
a
peer
review
me
chanism
(
§
2.4)
­­
Internal
(
§
2.4.2)
­­
External
(
§
2.4.3)
­­
Mail
(
e.
g.,
Letter)
review
(
§
2.4.3)
­­
Face
to
face
meeting
(
§
2.4.3)
­­
One
time
or
multiple
meetings
(
§
2.4.6)
­­
Include
logistical
information
in
peer
review
record
(
§
2.5.3)

3.
Determine
the
specific
time
line
(
§
3.3):
­­
When
will
the
review
be
started
­­
What
are
the
intermed
iate
check
p
oints
­­
What
is
the
deadline
for
completion
4.
Select
peer
reviewers
(
§
3.4):
­­
Determine
sources
of
peer
reviewers
(
§
3.4.2)
­­
Determine
expertise
required
(
§
3.4.4)
­­
Consider
balance/
add
ress
(
§
3.4.4)
­­
Consider
c
onflicts
of
interest
(
§
3.4.5
&
3.4.6)
­­
Include
documentation
in
peer
review
record
(
§
2.5.3)

5.
Materials
for
the
peer
review
(
§
3.5):
­­
Obtain
materials
from
Program
for
review
­­
Prepare
instructions
for
peer
reviews
(
§
3.5.1)
­­
Forward
materials
to
peer
reviewers
(
§
3.5.2)
­­
Include
copy
of
materials
in
peer
review
record
(
§
2.5.3)

6.
Conduct
the
peer
review
­­
Obtain
written
comments
from
reviewers
­­
Include
in
peer
review
record
(
§
2.5.3)

Note:
Some
of
these
steps
may
occur
concurrently.
Page
4
Peer
Review
Handbook
Figure
3
­
Flowchart
for
Completing
a
Peer
Review
1.
Evaluate
comments
from
peer
reviewers
(
§
4.2.1)
­­
Respond
and
react
to
comme
nts
­­
Obtain
clarification,
if
needed
2.
Brief
your
Decision
Maker;
obtain
written
management
approval
of
response
to
comments
(
§
4.2.1)

3.
Comments
that
are
considered,
but
not
used
(
§
4.3.1)
­­
Determine
why
not
used
and
document
­­
Include
comments
in
peer
review
record
(
§
2.5.3)

4.
Comments
that
are
useful
(
§
4.3.1)
­­
Revise
the
wo
rk
produ
ct
by
incorpora
ting
comm
ents
­­
Send
revised
work
product
back
to
peer
reviewers,
if
necessary
­­
Include
comments
in
peer
review
record
(
§
2.5.3)

5.
Finalize
wo
rk
produ
ct
(
§
4.3.1)
­­
Include
in
peer
review
record
(
§
2.5.3)
­­
Move
w
ork
prod
uct
from
List
B
to
List
A
Peer
Review
Handbook
Page
5
Manager s
Planning
Checklist
for
Peer
Review
1)
Title
of
Work
Product:
___________________________________________________

2)
What
Decision/
Rule/
Regulation/
Action
Does
this
Work
Product
Support:
________
______________________________________________________________________________

3)
Determination
of
Major
Scientific
and
Technical
Work
Products
G
Is
the
work
product
scientific
or
technical
__
yes
__
no?
G
Is
the
work
product
__
major
or
__
non­
major?

4)
Determining
What
Peer
Review
is
Needed
G
If
major,
peer
review
is
needed?
G
If
not
major,
is
peer
review
still
needed?
G
What
peer
review
mechanism
is
needed
(
internal
and/
or
external)?
G
When
does
the
review
need
to
be
done?
G
How
much
time
will
be
needed
to
conduct/
complete
the
review?
G
Are
there
court
ordered
deadlines
or
other
constraints?
G
Has
senior
management
(
AA
/
RA/
others)
been
informed
of
progress/
problems?
G
What
would
constitute
success
for
this
review?

5)
Determining
the
Resources
for
Peer
Review
G
What
is
the
priority
of
this
project
relative
to
other
projects
in
the
same
office?
G
What
resources
are
needed
to
conduct
the
review?
G
What
are
the
impacts
of
the
review
on
personnel?
G
Who
will
lead
the
peer
review?
G
Who
will
conduct
the
peer
review?
G
Who
will
maintain
the
peer
review
record?
G
Where
will
the
peer
review
record
be
kept?
G
What
mechanism
will
be
used
for
the
peer
review?
G
Has
the
charge
been
developed?
G
Has
internal
and
external
coordination
been
initiated/
completed?
G
Have
arrangements
for
interim/
final
sign­
offs
(
e.
g.,
for
the
charge,
the
panel,
on
any
changes
to
the
final
work
product)
been
made?
G
How
will
results
of
the
review
be
presented
and
addressed
in
the
final
work
product
(
e.
g.,
in
a
preamble,
in
an
accompanying
appendix
­­
as
well
as
changes
in
the
work
product
itself)?
G
Has
the
work
product
been
entered
onto
List
B
or
C,
as
appropriate?

6)
Comments:
_____________________________________________________________

See
Appendix
F
(
Useful
Forms)
for
expanded
checklists
Peer
Review
Handbook
Page
7
U.
S.
Environmental
Protection
Agency
PEER
REVIEW
GUIDANCE
Peer
Review
Handbook
Page
9
1.
THE
NEED
FOR
PEER
REVIEW
1.1
Overview
Statement
Peer
review
at
the
U.
S.
Environmental
Protection
Agency
(
EPA)
takes
many
different
forms
depending
on
the
nature
of
the
work
product,
relevant
statutory
requirements,
and
office­
specific
policies
and
practices.
In
January
1993,
responding
to
recommendations
in
the
report
Safeguarding
the
Future:
Credible
Science,
Credible
Decisions
,
Administrator
William
Reilly
issued
an
Agency­
wide
policy
for
peer
review.
Administrator
Carol
Browner
reaffirmed
the
central
role
of
peer
review
in
the
Agency
on
June
7,
1994
and
instituted
an
Agency­
wide
implementation
program
(
see
Appendix
A
­
USEPA
Peer
Review
Policy).
Following
Agency­
wide
implementation,
office
and
region­
specific
standard
operating
procedures
(
SOPs)
were
written
and
used
from
1994
to
1998.
In
1998,
this
Peer
Review
Handbook
was
created
as
a
single,
centralized
form
of
implementation
guidance
for
Agency
staff
and
managers.
The
Peer
Review
Handbook
was
revised
throughout
calendar
year
2000
and
was
reissued
as
Edition
2
(
the
current
version)
in
December
2000.
Agency
offices
and
regions,
however,
can
still
prepare
brief,
tailored
guidance
that
meets
their
individual
needs
to
supplement
the
information
in
this
Handbook.

1.2
Understanding
Peer
Review
1.2.1
Why
use
Peer
Review?

Peer
review
is
intended
to
uncover
any
technical
problems
or
unresolved
issues
in
a
preliminary
(
or
draft)
work
product
through
the
use
of
independent
experts.
This
information
is
then
used
to
revise
that
draft
product
so
that
the
final
work
product
will
reflect
sound
technical
information
and
analyses.
Peer
review
is
a
process
for
enhancing
a
scientific
or
technical
Peer
review
is
not
free;
work
product
so
that
the
decision
or
position
taken
by
the
however,
not
doing
peer
Agency,
based
on
that
product,
has
a
sound,
credible
basis.
To
review
can
be
costly.
be
most
effective,
peer
review
of
a
major
scientific
and/
or
technical
work
product
needs
to
be
incorporated
into
the
up­
front
planning
of
any
action
based
on
the
work
product
­
this
includes
obtaining
the
proper
resource
commitments
(
people
and
money)
and
establishing
realistic
schedules.

Peer
review
of
major
scientific
and
technical
work
products
should
not
be
looked
upon
as
another
 
hurdle 
in
the
Agency
decision
making
processes.
While
peer
review
requires
that
time
and
resources
be
planned
into
the
decision
making
process,
the
benefits
justify
the
added
cost.
Peer
review
enhances
the
credibility
and
acceptance
of
the
decision
based
on
the
work
product.
Page
10
Peer
Review
Handbook
By
ensuring
a
sound
basis
for
decisions,
greater
cost
savings
are
realized
since
decisions
will
not
be
challenged
as
often
and
extra
effort
will
not
be
required
to
go
back
and
redo
the
work
product.
So
while
peer
review
is
not
free,
the
cost
of
not
doing
peer
review
is
usually
much
more
expensive.
Furthermore,
not
conducting
a
peer
review
can
potentially
place
the
Agency
in
the
position
of
attempting
to
defend
a
scientifically
invalid
position
­­
which
can
be
very
costly
in
terms
of
both
resources,
and
more
importantly,
credibility.

1.2.2
What
is
Peer
Involvement?

As
defined
in
the
Peer
Review
Policy,
peer
involvement
is
the
process
whereby
Agency
staff
involve
subject­
matter
experts
from
outside
their
program
in
one
or
more
aspects
of
the
development
of
work
products.
Peer
involvement,
therefore,
constitutes
active
outreach
to
and
participation
by
the
broad
scientific,
engineering,
and
economics
and
social
science
communities
beyond
the
Agency
(
external)
as
well
as
within
the
Agency
(
internal).
Typically,
peer
involvement
takes
two
general
forms:
peer
input
(
ongoing
discussions
during
the
development
of
the
work
product)
and
peer
review
(
an
evaluation
of
a
workplan,
preliminary
draft
or
the
like,
or
most
often,
the
critical,
final
objective
expert
evaluation
of
the
work
product).

1.2.3
What
is
Peer
Review?

Peer
review
is
a
documented
critical
review
of
a
specific
Agency
major
scientific
and/
or
technical
work
product.
The
peer
review
is
conducted
by
qualified
individuals
(
or
organizations)
who
are
independent
of
those
who
performed
the
work,
but
who
are
collectively
equivalent
in
technical
expertise
(
i.
e.,
peers)
to
those
who
performed
the
original
work.
The
peer
review
is
conducted
to
ensure
that
activities
are
technically
adequate,
competently
performed,
properly
documented,
and
satisfy
established
quality
requirements.
The
peer
review
is
an
in­
depth
assessment
of
the
assumptions,
calculations,
extrapolations,
alternate
interpretations,
methodology,
acceptance
criteria,
and
conclusions
pertaining
to
the
specific
major
scientific
and/
or
technical
work
product
and
of
the
documentation
that
supports
them.
Peer
review
may
provide
an
evaluation
of
a
subject
where
quantitative
methods
of
analysis
or
measures
of
success
are
unavailable
or
undefined;
such
as
research
and
development.
Peer
review
is
usually
characterized
by
a
one­
time
interaction
or
a
limited
number
of
interactions
by
independent
peer
reviewers.
Peer
review
can
occur
during
the
early
stages
of
the
project
or
methods
selection,
or
as
typically
used,
as
part
of
the
culmination
of
the
work
product,
ensuring
that
the
final
product
is
technically
sound.

1.2.4
What
is
Peer
Input?

Many
Agency
work
products
are
developed
with
the
input
of
various
scientific
and
technical
experts
inside
and
outside
the
Agency.
Like
the
contribution
made
by
peer
reviewers,
peer
input
is
valuable
and
enhances
the
scientific
or
technical
basis
of
the
products.
Peer
input,
Peer
Review
Handbook
Page
11
sometimes
referred
to
as
peer
consultation,
generally
connotes
an
interaction
during
the
development
of
an
evolving
Agency
work
product,
providing
an
open
exchange
of
data,
insights,
and
ideas.
Peer
input
may
be
characterized
by
a
continued
and
iterative
interaction
with
scientific
experts
during
work
product
development.
A
common
example
of
peer
input
is
the
input
received
from
workgroup
members
during
development
of
a
product.
Many
Agency
products
are
developed
through
the
efforts
of
a
workgroup,
which
may
include
external
experts,
such
as
State
and
Tribal
representatives.
These
workgroup
members
have
an
active,
ongoing
participation
in
developing
the
work
product.
Another
example
of
obtaining
peer
input
is
of
an
Agency
office
sending
a
draft
work
product
to
a
list
of
stakeholder
representatives
for
general
comments
(
stakeholder
representatives
often
include
experts
who
could
be
considered
 
peers ).

1.2.5
How
is
Peer
Review
Different
from
Peer
Input?

The
key
distinctions
between
peer
input
as
described
above
and
formal
peer
review
are
the
independence
of
the
peer
reviewers
and
their
level
of
involvement.
The
goal
of
peer
review
is
to
obtain
an
independent,
third­
party
review
of
the
product
from
experts
who
haven t
substantially
contributed
to
its
development.
When
experts
have
a
material
stake
in
the
outcome
of
the
peer
review
(
such
as
a
regulated
party)
or
have
participated
substantially
in
the
development
of
the
product
(
such
as
a
workgroup
member),
those
experts 
reviews
may
not
qualify
as
unbiased,
independent
peer
review
and
may
be
better
characterized
as
peer
input.

It
is
clear
that
peer
input
provides
valuable
contributions
to
the
development
of
the
work
product.
Peer
Input
is
not
a
However,
peer
input
does
not
substitute
for
peer
review.
Once
substitute
for
Peer
Review
a
work
product
is
considered
major,
it
is
a
candidate
for
peer
review
and
entered
on
List
B
(
Candidate
Products
for
Future
Peer
Review
­­
see
Section
1.3.2b))
­­
even
though
the
work
product
may
already
have
a
substantial
amount
of
peer
input.
In
other
words,
one
cannot
argue
that
a
peer
review
is
not
necessary
if
a
major
work
product
has
received
 
enough 
peer
input.
If
the
work
product
is
not
considered
major
and
has
had
peer
input,
it
is
entered
on
List
C
(
Products
for
Which
a
Decision
has
been
made
not
to
Peer
Review
­­
see
Section
1.3.2c))
with
comments
about
the
extent
of
the
peer
input.

1.2.6
Can
Someone
Who
Provided
Peer
Input
Become
an
Independent
Peer
Reviewer
for
the
Same
Work
Product
Later
in
the
Process?

Generally,
the
answer
is
no
as
that
expert
is
no
longer
independent,
but
rather
a
contributor
to
the
work
product.
There
may
be
special
circumstances
where
the
expertise
is
so
narrow
that
another
peer
reviewer
isn t
available.
The
Peer
Review
Leader
(
see
Section
1.4.4)
will
normally
be
responsible
for
making
this
determination
and
documenting
the
decision
in
the
peer
review
record.
Page
12
Peer
Review
Handbook
1.2.7
Can
the
Same
Peer
Reviewer
be
Used
More
than
Once
if
a
Product
will
Be
Peer
Reviewed
More
than
Once,
and
can
the
Same
Peer
Reviewer
be
Used
Again
and
Again
for
Different
Products?

There
is
no
prohibition
on
using
the
same
peer
reviewer
more
than
once
on
the
same
product
or
for
multiple
products
of
the
same
office.
However,
it
is
a
good
idea
to
use
some
different
people
each
time
where
possible
to
provide
a
broader
perspective.
When
using
a
contractor
to
provide
peer
review
services,
you
should
recognize
that
contractors
may
have
a
 
pool 
of
reviewers
that
they
use
regularly.
If
the
same
peer
reviewers
are
used
repeatedly,
they
may
lose
their
independence
(
or
the
appearance
of
independence)
from
the
work
product(
s).
If
a
peer
reviewer
is
asked
to
participate
in
multiple
reviews
of
the
same
product
it
should
be
noted
in
the
peer
review
record.

1.2.8
How
is
Peer
Review
Different
from
Public
Comment?

Peer
review
and
public
comment
are
mutually
exclusive.
Public
comment
solicited
from
the
general
public
through
the
Federal
Register
or
by
other
means
is
often
required
by
the
Administrative
Procedures
Act,
relevant
statutes
or
both.
Public
comment
can
also
be
solicited
for
policy
purposes.
The
Agency
takes
public
comment
on
some
strictly
scientific
products
and
almost
all
regulatory
decisions.
Public
commenters
usually
include
a
broad
array
of
people
with
an
interest
in
the
technical
analysis
or
the
regulatory
decision;
some
are
scientific
experts
(
which
may
provide
some
peer
input),
some
are
experts
in
other
Public
comment
does
not
substitute
areas,
and
some
are
interested
non­
experts.
The
for
peer
review.
critical
distinction
is
that
public
comment
doesn t
necessarily
draw
the
kind
of
independent,
expert
information
and
in­
depth
analyses
expected
from
the
peer
review
process.
Public
comment
is
open
to
all
issues,
whereas
the
peer
review
process
is
limited
to
consideration
of
technical
issues.
While
it
may
be
an
important
component
of
the
review
process,
public
comment
does
not
substitute
for
peer
review.

1.2.9
How
is
Peer
Review
Different
from
Stakeholder
Involvement?

Stakeholder
involvement
occurs
when
the
Agency
works
with
external
interest
groups
that
have
some
stake
in
or
concerns
over
the
outcome
of
the
technical
work
product
or
regulatory
position.
This
is
an
interactive
process,
working
with
other
agencies,
industry
groups,
regulated­
community
experts,
environmental
groups,
other
interest
groups
that
Stakeholder
involvement
is
not
represent
a
broad
spectrum
of
the
regulated
community,
a
peer
review
mechanism.
etc.,
and
usually
strives
for
a
consensus
approach.
The
goal
of
peer
review,
on
the
other
hand,
is
to
obtain
an
Peer
Review
Handbook
Page
13
independent,
third­
party
review
for
ensuring
scientific
integrity
and
technical
credibility
of
the
work
product
that
supports
a
policy
or
decision.
Stakeholder
involvement
is
not
a
peer
review
mechanism
and
it
does
not
substitute
for
peer
review
even
though
it
adds
value
to
the
work
product.

1.2.10
What
Role
does
Peer
Review
have
in
the
Regulatory
Development
Process?

The
peer
review
of
scientific
and
technical
work
products
that
support
rulemaking
actions
is
an
important,
fundamental
step
in
the
policy
setting
process
and
which
affirms
the
credibility
of
the
Agency.
Because
new
rules,
and
the
work
products
supporting
them,
must
often
withstand
intense
scrutiny
by
the
general
public
and
the
stakeholders
involved
in
the
action,
the
peer
review
process
selected
for
such
work
products
needs
to
be
well
planned
and
documented.
The
rule
or
regulation
itself
is
not
subject
to
the
Peer
Review
Policy.
However,
if
the
rule
or
regulation
is
supported
by
a
major
scientific
and/
or
technical
work
product,
that
work
product
should
be
peer
reviewed
prior
to
its
use
in
the
rule
(
see
Section
2.2
for
determination
of
major
work
products).
The
decision
to
peer
review
or
not
peer
review
any
scientific
and/
or
technical
work
product
will
be
documented
through
the
Agency s
annual
peer
review
reporting
process
(
see
Section
1.3).
Remember,
public
comment
and
stakeholder
involvement
do
not
constitute
peer
review.

Tier
1
and
Tier
2
rulemakings,
are
by
definition
important,
major
Agency
rulemakings
within
the
Agency.
Therefore,
work
products
supporting
Tier
1
and
Tier
2
rules
in
particular
(
including
rules
that
are
determined
to
be
 
significant 
by
OMB
under
Executive
Order
12866
because
they
have
an
economic
impact
of
$
100
million
or
more)
should
be
closely
scrutinized
to
determine
whether
they
meet
the
criteria
for
major
(
see
Section
2.2.3).
Work
products
supporting
Tier
3
rulemakings
may
also
be
considered
major
and
thus
candidates
for
peer
review.
External
peer
review
is
the
procedure
of
first
choice
for
a
work
product
that
is
intended
to
support
a
Tier
1
or
Tier
2
rulemaking.
Although
acceptable
in
certain
circumstances,
any
decision
to
use
an
internal
peer
review
mechanism
for
such
work
products
would
be
the
exception
rather
than
the
rule.
For
work
products
supporting
a
Tier
3
rule,
internal
or
external
peer
review
may
be
appropriate
depending
on
the
nature
of
the
product
and
other
factors
(
see
Section
2.4.1).
For
Tier
1
and
Tier
2
rulemakings,
the
Final
Agency
Review/
closure
memo
needs
to
indicate
that
the
Peer
Review
Policy
was
followed.
For
Tier
3
rulemakings,
the
action
memo
needs
to
indicate
that
the
Peer
Review
Policy
was
followed.

Analytic
blueprints
are
required
for
Tier
1
and
Tier
2
rulemakings,
and
are
encouraged
for
Tier
3
rulemakings;
some
individual
EPA
offices
require
it
for
Tier
3.
For
peer
review
purposes,
development
of
the
analytic
blueprint
is
the
process
whereby
the
project
manager
identifies
the
supporting
scientific
and
technical
work
products
and
identifies
needed
peer
review.
In
the
directive
from
the
Deputy
Administrator
(
memo
from
Acting
Deputy
Administrator
Robertson,
June
15,
1999:
Sound
Science
and
Peer
Review
in
Rulemaking;
see
Page
14
Peer
Review
Handbook
Appendix
B
­
Sound
Science
and
Peer
Review
in
Rulemaking),
planning
for
peer
review
must
be
specifically
addressed
in
each
new
analytic
blueprint.
You
are
required
to
integrate
peer
review
into
the
rulemaking
process
and
to
include
specific
language
addressing
peer
review
in
the
action
memorandum.
The
analytic
blueprint
shows
the
schedule
of
the
peer
review
in
the
context
of
the
schedule
for
the
overall
rulemaking.
In
general,
peer
review
should
be
completed
as
early
in
the
process
as
practicable.
Where
possible,
peer
review
of
work
products
should
be
completed
prior
to
issuance
of
the
proposed
regulation.
In
some
cases,
support
work
products
for
final
regulations
may
require
an
additional
peer
review
if
those
scientific
and
technical
work
products
change
significantly
after
the
public
comment
period.

1.2.11
What
Role
does
Peer
Review
have
in
Regulatory
Negotiations?

Regulatory
negotiations
are
not
candidates
for
peer
review;
however,
to
ensure
final
decisions
are
based
on
sound
and
credible
science,
the
major
scientific
and
technical
work
products
that
support
the
negotiation
need
peer
review
before
the
negotiation
takes
place.

1.3
Annual
Agency
Reporting
Requirements
1.3.1
What
are
the
Annual
Reporting
Requirements?

The
Peer
Review
Coordinator
(
for
each
AA/
RA;
see
Section
1.4.5)
will
organize
an
annual
review
of
all
peer
review
activities
and
submit
this
information
to
the
Office
of
Research
and
Development
(
ORD).
ORD
will
staff
this
function
at
the
direction
of
the
Deputy
Administrator.
In
the
Deputy
Administrator s
annual
call
for
submissions,
guidance
on
format
and
submission
of
this
information
will
be
provided.
ORD
will
review
the
submissions
for
completeness,
i.
e.,
all
information
is
provided
and
products
are
accounted
for
each
year.
ORD
will
then
provide
a
review
of
the
completeness
of
the
information
in
the
submissions
through
consultation
with
the
appropriate
persons
in
each
organization
(
see
Section
1.4).
ORD
will
then
consolidate
the
information
and
findings
for
the
SPC
and
the
Deputy
Administrator.
Any
conflicts
arising
from
the
review
will
be
resolved
by
the
Deputy
Administrator.
The
due
date
for
the
annual
reporting
will
be
announced
each
year
in
the
annual
call
letter;
however,
for
planning
purposes,
it
is
normally
due
in
the
early
summer.
The
Office
of
Environmental
Information
(
OEI)
assists
ORD
in
these
annual
reporting
functions.

1.3.2
What
Listings
are
Required
for
the
Annual
Reporting?

The
Peer
Review
Product
Tracking
(
PRPT)
database
is
a
Lotus
Notes
shared­
database
developed
to
track
and
report
peer
review
activities
across
the
Agency.
The
database
is
the
single
repository
for
product­
specific
peer
review
reporting
and
tracking
and
uses
a
common
reporting
form
for
all
reporting.
Work
products
for
which
peer
review
is
completed
are
reported
as
List
A
(
Peer
Review
Completed);
candidate
work
products
for
future
peer
review
are
reported
Peer
Review
Handbook
Page
15
as
List
B
(
Peer
Review
Needed);
and
work
products
that
do
not
meet
peer
review
criteria
are
reported
as
List
C
(
Peer
Review
Not
Needed).
List
D
(
Scientific
Articles/
Reports
Peer
Reviewed
Outside
of
EPA)
was
recently
added
largely
to
track
the
articles
that
EPA
staff
produce.

Four
listings
of
products
are
required
for
the
annual
reporting:

a)
List
A
­
Products
Peer
Reviewed
Since
1991
1)
List
A
is
a
cumulative
list
of
peer
reviewed
products
from
1991
to
the
present.

2)
Each
work
product
entered
onto
List
A
must
include
information
summarizing
the
peer
review.
The
database
will
specify
the
needed
information.
The
entry
must
be
electronically
signed
(
verified)
by
the
Peer
Review
Leader,
the
Peer
Review
Coordinator,
and
the
Decision
Maker
involved
(
see
Section
1.4.3
to
help
determine
who
the
Decision
Makers
are).

b)
List
B
­
Candidate
Products
for
Future
Peer
Review
1)
List
B
is
a
list
of
products
that
are
expected
to
be
peer
reviewed
in
the
near
future
.

2)
This
List
contains
major
scientific
and
technical
work
products
and
any
non­
major
scientific
and
technical
work
products
for
which
peer
review
has
been
deemed
necessary
or
appropriate.
All
entries
on
List
B
must
be
electronically
signed
(
verified)
by
the
Peer
Review
Leader,
the
Peer
Review
Coordinator,
and
the
Decision
Maker
involved.

3)
Work
products
placed
on
List
B
remain
on
List
B
until
they
are
either
peer
reviewed
(
after
which
they
are
moved
to
List
A)
or
a
decision
is
made
not
to
peer
review
that
work
product
(
at
which
point
it
is
moved
to
List
C
including
the
rationale
for
not
conducting
peer
review).
If
the
product
is
terminated,
it
is
moved
to
List
C.

c)
List
C
­
Products
for
Which
a
Decision
has
Been
Made
Not
to
Peer
Review
1)
List
C
is
a
cumulative
list
of
all
scientific
and
technical
work
products
that
do
not
receive
peer
review
(
see
Section
2.3).
Page
16
Peer
Review
Handbook
2)
List
C
includes:
(
a)
any
major
scientific
and
technical
work
products
for
which
a
decision
was
made
not
to
peer
review;
(
b)
those
work
products
which
were
originally
placed
on
List
B,
but
for
which
it
was
decided
that
peer
review
was
not
necessary
(
e.
g.,
the
product
was
not
used
in
decision
making;
the
project
was
canceled);
and
(
c)
all
non­
major
scientific
and
technical
work
products.

3)
List
C
also
includes
several
categories
of
work
products
that
would
not
normally
receive
peer
review.
It
is
not
necessary
to
list
these
work
products
individually
on
List
C,
however,
the
total
number
prepared
by
the
organization
must
be
included
on
List
C.
The
organization
may
need
to
identify
the
individual
products
if
requested
(
e.
g.,
due
to
litigation,
FOIA,
etc.).
Such
categories
can
include,
but
may
not
be
limited
to:
chemical
action
reports,
RCRA
permits,
scientific
analyses
for
Premanufacturing
Notices
(
PMNs)
that
are
conducted
on
a
routine
basis
and
that
do
not
deviate
from
established
practice,
and
NPDES
permits.

4)
Each
work
product
on
List
C
needs
a
brief
description
of
the
reason(
s)
it
is
not
being
peer
reviewed.
All
entries
on
List
C
must
be
electronically
signed
(
verified)
by
the
Peer
Review
Leader,
the
Peer
Review
Coordinator,
and
the
Decision
Maker
involved.

d)
List
D
­
Scientific
Articles
and
Reports
That
Are
Peer
Reviewed
by
Organizations
Outside
of
EPA
1)
List
D
includes
EPA
authored
scientific
papers
(
articles)
that
are
peer
reviewed
by
a
credible
refereed
scientific
journal
(
See
Sections
2.4.4
and
2.4.5).

2)
Listing
such
documents
in
this
manner
shows
that
these
items
have
already
been
peer
reviewed,
but
differentiates
them
from
Agency
work
products.
In
addition,
it
gives
EPA
an
opportunity
to
highlight
the
extensive
work
it
produces
in
the
scientific
literature.

3)
It
is
not
necessary
to
list
these
papers
individually
on
List
D,
although
some
organizations
may
choose
to
do
so.
However,
at
a
minimum,
the
total
number
prepared
by
the
organization
and
peer
reviewed
by
journals
must
be
included
on
List
D.
The
organization
may
ultimately
need
to
identify
the
individual
papers
and
where
published,
if
requested
(
e.
g.,
due
to
litigation,
FOIA,
etc.).
Peer
Review
Handbook
Page
17
4)
Once
any
of
these
articles
or
reports
are
used
in
an
Agency
work
product,
that
work
product
becomes
a
candidate
for
peer
review
and
must
be
listed
on
List
B
(
or
List
C
if
a
decision
is
made
not
to
peer
review
the
work
product).

5)
Entries
on
List
D
do
not
require
an
electronic
signature.

1.3.3
When
will
the
Handbook
Itself
be
Revised?

During
each
annual
reporting
cycle,
suggestions
for
revisions
to
the
Handbook
should
be
submitted.
Periodically,
a
decision
will
be
made
by
the
Science
Policy
Council
(
SPC)
on
whether
to
revise
the
Handbook,
in
part
or
total,
or
not
based
on
the
suggestions.
The
SPC
will
then
direct
the
Peer
Review
Advisory
Group
(
PRAG)
to
draft
the
revision(
s)
for
Agency
comment
and
SPC
approval.

1.4
The
Roles
of
People
and
Organizations
in
Peer
Review
1.4.1
Who
is
Ultimately
Accountable
for
Peer
Review?

Under
the
June
7,
1994
Peer
Review
Policy
(
Appendix
A
­
USEPA
Peer
Review
Policy),
the
Administrator
has
designated
the
Assistant
Administrators
and
Regional
Administrators
(
AAs
and
RAs)
to
be
accountable
for
implementing
the
Policy
in
their
respective
organizations.
The
Deputy
Administrator
is
ultimately
responsible
for
peer
review
across
the
Agency
and
is
the
final
arbitrator
of
conflicts
and
concerns
about
peer
review.

1.4.2
Who
are
the
Agency
Staff
involved
in
Peer
Review?

The
principal
Agency
staff
involved
are
Decision
Makers
(
and
their
line
managers),
Peer
Review
Leaders
and
Peer
Review
Coordinators.
In
addition,
there
are
many
other
critical
staff
in
each
office
and
region
who
have
responsibility
for
peer
review
activities
(
e.
g.,
office
and
division
peer
review
coordinators,
technical
information
managers,
and,
of
course,
any
Agency
staff
that
serve
as
internal
peer
reviewers).
Finally,
ORD
has
oversight,
as
designated
by
the
Deputy
Administrator,
for
ensuring
the
Agency s
Peer
Review
Policy
requirements
are
met.

1.4.3
Who
are
the
Decision
Makers
&
What
are
Their
Responsibilities?

The
AA/
RA
is
the
ultimate
Decision
Maker
for
their
organization
and
is
accountable
for
the
decisions
regarding
the
identification
of
major
scientific
and
technical
work
products
and
the
mechanism(
s)
of
peer
review
utilized
for
each
of
the
products.
The
AA/
RA
may
designate
Office
Directors
and/
or
Division
Directors
as
the
front­
line
Decision
Makers.
Page
18
Peer
Review
Handbook
Generally,
the
Decision
Makers
decide
whether
a
work
product
is
major
and
needs
peer
review
or
not,
and
what
peer
review
mechanism
to
use.
Furthermore,
the
Decision
Makers
commit
the
resources
needed
to
ensure
a
proper
peer
review.
Decision
Makers
are
responsible
for
ensuring
that
the
peer
reviews
are
properly
performed
and
documented.

In
order
to
ensure
greater
independence
of
peer
reviews,
it
is
necessary
to
strictly
separate
management
of
work
products
from
the
actual
peer
review
of
those
work
products.
Therefore,
the
Decision
Maker
and
the
Peer
Review
Leader
(
see
Section
1.4.4)
for
a
work
product
should
never
be
the
same
person.

The
Decision
Maker
needs
appropriate
training
on
how
to
manage
the
peer
review
process.
The
Peer
Review
Coordinator
for
the
office
can
advise
the
Decision
Maker
on
how
to
obtain
the
appropriate
training;
the
training
approach
varies
among
the
different
EPA
offices.

Specific
responsibilities
of
the
Decision
Maker(
s)
are
the
following:

a)
Determine
which
work
products
in
their
organization
require
peer
review
b)
Designate
(
in
conjunction
with
the
Project
Manager)
a
Peer
Review
Leader
to
organize
the
peer
review
c)
Provide
advice,
guidance,
and
support
to
the
Peer
Review
Leader
in
the
preparation,
conduct,
and
completion
of
the
peer
review
d)
Ensure
that
sufficient
funds
are
designated
in
the
office s
budget
request
to
conduct
the
peer
review;
also
ensure
that
adequate
resources
and/
or
extramural
management
support
are
available
for
the
peer
review
e)
Establish
a
realistic
peer
review
schedule
f)
Designate
the
stage(
s)
of
product
development
where
peer
review
is
appropriate
g)
Ensure
all
relevant
issues
and
comments
raised
by
the
peer
reviewer(
s)
are
adequately
addressed
and
documented
for
the
record,
and
where
appropriate,
incorporated
into
the
work
product
that
is
used
as
basis
for
decision
making
h)
By
signature,
document
the
decisions
made
that
are
reported
in
the
annual
reporting
to
the
SPC
and
Deputy
Administrator
i)
Certify
any
decision
NOT
to
peer
review
a
product
by
signature
on
a
List
C
(
Products
for
Which
a
Decision
has
been
made
not
to
Peer
Review)
submission
Peer
Review
Handbook
Page
19
1.4.4
Who
are
the
Peer
Review
Leaders
&
What
are
Their
Responsibilities?

The
Peer
Review
Leader
is
assigned
by
the
Decision
Maker
to
organize,
conduct
and
complete
the
peer
review
for
a
specific
individual
work
product.
The
Peer
Review
Leader
will
obtain
the
assistance
and
support
of
the
Peer
Review
Coordinator
(
see
below)
as
well
as
any
others
within
the
Agency
to
help
perform
the
peer
review.
The
Peer
Review
Leader
will
be
chosen
on
a
case
by
case
basis
depending
on
the
work
product
needing
peer
review.
The
Peer
Review
Leader
cannot
be
the
Decision
Maker.
The
Peer
Review
Leader
could
be
the
Project
Manager
for
the
work
product.

The
Peer
Review
Leader
must
have
appropriate
training
on
how
to
conduct
a
peer
review
before
conducting
the
peer
review.
The
Peer
Review
Coordinator
for
the
office
can
advise
the
Peer
Review
Leader
on
how
to
obtain
the
appropriate
training;
the
training
approach
varies
among
the
different
EPA
offices
however,
uniform
training
modules
are
available.

Specific
responsibilities
of
the
Peer
Review
Leader
are
these:

a)
Keep
the
Decision
Maker
informed
of
the
status
of
a
given
project;
provide
Peer
Review
Coordinator
with
data
for
the
annual
report
b)
Organize,
conduct,
and
complete
the
peer
review
following
Agency
procedures
c)
Establish
and
maintain
the
peer
review
record
for
the
specific
individual
peer
review
currently
being
performed
(
see
Section
2.5);
this
includes
providing
the
peer
review
summary
information
in
the
Peer
Review
Product
Tracking
(
PRPT)
Database
for
the
Decision
Maker
to
sign
when
the
peer
review
is
completed
d)
Select
the
peer
reviewers
in
consultation
with
others
involved
with
the
peer
review
(
e.
g.,
Decision
Maker)
and
ensure
that
conflict
of
interest
issues
are
addressed
and
documented
in
the
peer
review
record
e)
Advise
peer
reviewers
of
their
responsibilities
f)
Provide
information
to
the
Decision
Maker
(
including
all
appropriate
managers
in
the
Peer
Review
Leader s
chain
of
command)
on
the
charge,
profile
of
peer
reviewers,
the
peer
review
comments,
and
a
proposal
on
how
to
address
the
comments.
Obtain
Decision
Maker
approval
on
the
approach
to
responding
to
peer
reviewer
comments.
Clearly
identify
for
the
Decision
Maker
any
peer
review
comments
that
will
not
be
addressed
in
the
agreed
upon
approach.
Page
20
Peer
Review
Handbook
g)
Notify
the
Peer
Review
Coordinator
that
the
peer
review
is
completed
for
the
annual
report
h)
Archive
the
peer
review
record
in
a
manner
consistent
with
their
organization s
archiving
procedures
When
a
contractor
is
used
to
conduct
a
peer
review,
some
of
the
above
responsibilities
are
assumed
by
the
contractor
(
see
Section
3.6).

1.4.5
Who
are
the
Peer
Review
Coordinators
&
What
are
Their
Responsibilities?

The
Peer
Review
Coordinator
is
designated
by
the
AA/
RA
to
coordinate
and
monitor
peer
review
activities
in
their
respective
organization
or
organizational
unit.
This
person
must
be
of
sufficient
stature
and
judgment
to
have
the
access
to
and
confidence
of
all
levels
of
office
or
regional
management
when
needed.
The
Peer
Review
Coordinator
is
the
main
contact
for
their
organization;
they
can
also
direct
interested
parties
to
other
persons/
contacts
in
the
office
on
specific
work
products
(
e.
g.,
Peer
Review
Leader).

Specific
responsibilities
of
the
Peer
Review
Coordinator
are
these:

a)
General
oversight
responsibility
for
the
Office s
or
Region s
peer
review
process
b)
Report
peer
review
activities
to
the
AA/
RA
c)
Help
mediate
difficult
issues
between
their
organization
and
others;
if
they
can t
resolve
issue,
then
bring
the
issue
to
the
attention
of
the
appropriate
level
Decision
Makers
in
each
organization
for
resolution.

d)
Function
as
the
liaison
with
ORD
and
the
Science
Policy
Council
(
SPC):

1)
Represent
office/
region
before
the
SPC
2)
Advise
ORD
of
any
changes
in
the
list
of
work
products
and
peer
review
mechanisms
during
the
annual
reporting,
and
when
necessary,
at
other
times
3)
Participate
in
Agency
peer
review
training,
workshops,
etc.,
as
requested
and
disseminate
this
information
to
the
organization;
coordinate
and/
or
present
training
within
their
organization
Peer
Review
Handbook
Page
21
e)
Submit
information
on
organization s
peer
review
candidates
for
each
year
as
requested
(
this
is
the
annual
reporting,
see
Section
1.3)

1)
Generate
and
update
Lists
A,
B,
C,
and
D
(
See
Section
1.3.2)

2)
Assure
the
proper
approval
signature
on
the
completed
submission
with
the
accompanying
explanation
for
any
departures
from
the
Policy
f)
Establish
procedures
to
assure
that
the
required
work
product
peer
review
documentation
(
i.
e.,
peer
review
record)
is
filed
and
maintained
in
an
appropriate
manner
(
see
Section
2.5)

g)
Provide
advice,
guidance,
and
support
to
the
various
Peer
Review
Leaders
for
the
performance
of
the
peer
reviews
h)
Distribute
Agency­
wide
peer
review
guidance
and
materials
to
appropriate
office/
region
personnel,
as
requested
1.4.6
Who
are
the
Peer
Reviewers?

Peer
reviewers
are
individuals
who
have
technical
expertise
in
the
subject
matter
of
the
work
product
undergoing
peer
review.
Peer
reviewers
can
come
from
EPA,
another
Federal
agency,
or
from
outside
of
the
Federal
government.

1.4.7
What
are
the
Responsibilities
of
Peer
Reviewers?

Peer
reviewers
need
to
be
willing
participants
in
the
peer
review
process
­­
they
should
agree
to
read
all
materials,
participate
fully,
and
protect
confidential
information
that
arises.
Peer
reviewers
should
maintain
the
confidentiality
of
the
product,
perform
the
review
in
a
timely
manner,
and
be
unbiased
and
objective.

1.4.8
What
is
an
Independent
Peer
Reviewer?

An
independent
peer
reviewer
is
an
expert
who
wasn t
associated
with
the
generation
of
the
specific
work
product
either
directly
by
substantial
contribution
to
its
development
or
indirectly
by
significant
consultation
during
the
development
of
the
specific
product.
The
independent
peer
reviewer,
thus,
is
expected
to
be
objective
(
See
Sections
1.2.6
&
1.2.7
for
further
information)
Page
22
Peer
Review
Handbook
Independence
is
freedom
from
institutional,
ideological,
or
technical
bias
regarding
the
issues
The
quality
of
the
peer
review
is
under
review
and
is
necessary
for
objective,
fair,
and
dependent
on
the
competence
and
responsible
evaluation
of
the
work
product.
If
a
independence
of
the
reviewers.
selected
reviewer
has
a
particular
scientific
or
technical
perspective,
it
may
be
desirable
to
balance
the
review
with
peer
reviewers
with
other
perspectives.
Ideally,
peer
reviewers
should
be
free
of
real
or
perceived
conflicts­
of­
interest
or
there
should
be
a
balancing
of
interests
among
peer
reviewers.
If
there
are
potential
conflicts
of
interest
(
real
or
perceived),
they
should
be
fully
identified
to
ensure
a
credible
peer
review.
(
See
Sections
3.4.5
&
3.4.6
for
further
information).

1.4.9
When
does
an
Agency
Internal
Peer
Reviewer
Qualify
as
Independent?

An
Agency
independent
peer
reviewer
is
one
who
comes
from
a
different
organizational
unit
than
the
one
where
the
review
question
or
document
originates.
A
different
organizational
unit
usually
denotes,
at
minimum,
a
different
office
(
i.
e.,
above
division
level
in
programs;
above
branch
level
in
regions)
within
the
organization.
In
particular,
a
reviewer
shouldn t
come
from
within
the
chain
of
command,
either
upward
or
downward.

1.4.10
What
is
a
Peer
Review
Panel?

A
peer
review
panel
can
range
from
a
few
individuals
to
ten
or
more,
depending
on
the
issue
being
investigated,
the
time
available
and
any
limitations
on
resources.
Individuals
who
serve
as
peer
reviewers
must
have
appropriate
scientific
and
technical
expertise
such
that
the
review
panel
covers
the
broad
spectrum
of
expertise
required
to
treat
the
issues/
questions
presented
in
the
charge.

1.4.11
What
is
a
Subject
Matter
Expert?

A
subject
matter
expert
is
one
who
has
specific
scientific
and
technical
expertise
in
the
matter
under
review.
The
importance
of
scientific
and
technical
expertise
in
the
subject
matter
is
obvious,
however,
knowledge
or
just
 
knowing 
about
the
subject
area
isn t
equivalent
to
expertise
in
the
subject
matter.
For
Agency
decisions,
a
multi­
disciplinary
group
of
experts
corresponding
to
the
disciplines
that
contribute
to
complex
Agency
decisions
is
often
necessary
for
a
full
and
complete
peer
review.
For
example,
a
risk
assessment
that
relies
on
both
animal
and
human
data
usually
requires
experts
in
both
areas
for
a
complete
review.
For
economic
analyses,
experts
from
the
corresponding
economic
disciplines
are
necessary.
Peer
Review
Handbook
Page
23
1.4.12
What
is
the
Role
of
the
Science
Policy
Council
(
SPC)?

According
to
the
1994
Peer
Review
Policy
statement:
 
The
Science
Policy
Council
is
responsible
for
overseeing
Agency­
wide
implementation.
Its
responsibilities
include
promoting
consistent
interpretation,
assessing
Agency­
wide
progress,
and
developing
recommendations
for
revisions
of
the
policy
as
necessary. 
The
SPC
meets
its
responsibilities
through
coordination
with
the
Peer
Review
Coordinators,
the
Peer
Review
Advisory
Group
(
PRAG)
and
the
Office
of
Research
and
Development
(
ORD).

The
SPC,
PRAG
and
ORD
are
not
responsible
for
identifying
specific
products
for
peer
review
or
determining
the
level
of
review
or
mechanism
for
that
review;
those
functions
are
the
responsibility
of
management
within
each
Office
or
Region.

1.4.13
What
is
the
Role
of
the
Peer
Review
Advisory
Group
(
PRAG)?

The
Science
Policy
Council
has
created
the
Peer
Review
Advisory
Group
(
PRAG)
to
assist
in
the
implementation
of
the
Agency's
Peer
Review
Policy.
The
primary
role
of
the
PRAG
is
to
provide
interpretation
of
the
policy
and
to
assist
the
SPC
and
Agency
Offices
and
Regions
in
the
annual
update
of
the
Peer
Review
Handbook.

1.4.14
What
is
the
Role
of
the
Office
of
Research
and
Development
(
ORD)?

The
Deputy
Administrator
has
designated
the
Office
of
Research
and
Development
(
ORD)
to
provide
oversight
to
the
Program
Offices
and
Regions
in
the
collection
and
review
of
information
that
is
contained
in
the
annual
submission
of
Lists
A,
B,
C,
and
D
(
for
detailed
information,
see
Section
1.3).
The
Office
of
Environmental
Information
(
OEI)
assists
ORD
in
its
oversight
role.
Peer
Review
Handbook
Page
25
2.
PLANNING
A
PEER
REVIEW
2.1
Overview
Statement
Planning
a
peer
review
is
a
critical
first
step
to
ensure
a
successful
peer
review
of
a
work
product.
The
initial
step
is
to
determine
whether
your
work
product
requires
peer
review.
Once
you
have
determined
that
a
peer
review
will
be
conducted,
the
Decision
Makers
and
Peer
Review
Leaders
need
to
plan
an
appropriate
review.
This
includes
identification
of
resources
(
budget
and
personnel),
the
schedule
for
the
completion
of
the
peer
review,
the
mechanism
for
peer
review,
the
choice
of
peer
reviewers,
and
the
development
of
the
peer
review
record.

2.2
Determining
Which
Work
Products
to
Peer
Review
2.2.1
What
are
Scientific
and
Technical
Work
Products?

The
first
step
in
determining
which
work
products
require
peer
review,
is
to
identify
products
that
are
scientific
and/
or
technical
in
nature.
Scientific
and
technical
work
products
are
used
to
support
a
research
agenda,
regulatory
program,
policy
position
or
other
Agency
position
or
action.
Scientific
and
technical
work
products
include
economic
and
social
science
work
products.
Categories
of
work
products
include,
for
example:
risk
assessments,
technical
studies
and
guidance,
analytical
methods,
scientific
database
designs,
technical
models,
technical
protocols,
statistical
survey/
studies,
technical
background
materials,
technical
guidance
(
except
for
guidance
providing
policy
judgments),
research
plans,
and
research
strategies.

Products
that
wouldn t
be
considered
scientific
and
technical
work
products
can
include
those:
that
address
procedural
matters
(
e.
g.,
planning,
reporting,
coordination,
notification);
that
are
primarily
policy
statements
(
e.
g.,
relocation
policy);
that
are
conference
proceedings
(
unless
the
proceedings
are
used
as
the
scientific
basis
for
an
Agency
action
or
decision);
and
that
are
decision
documents
(
e.
g.,
Record
of
Decision
(
ROD)
­­
the
decision
document
itself
is
not
subject
to
the
Peer
Review
Policy,
but
the
underlying
scientific
and/
or
technical
support
work
product
is
a
candidate
for
peer
review).
In
addition,
the
following
Agency
documents
are
not
considered
scientific
and/
or
technical
work
products
under
the
Peer
Review
Policy:
strategic
plans,
analytic
blueprints,
and
goals
documents.

Making
final
determinations
concerning
which
work
products
are
scientific
and
technical
is
a
responsibility
of
the
Decision
Maker
(
See
Section
1.4.3)
Page
26
Peer
Review
Handbook
2.2.2
What
Scientific
and
Technical
Work
Products
Need
Peer
Review?

The
principle
underlying
the
Peer
Review
Policy
is
that
all
major
scientific
and
technical
work
products
used
in
decision
making
will
be
peer
reviewed.
The
process
for
identifying
When
in
doubt
about
whether
to
peerwhich
of
these
products
is
 
major 
(
and
thus
review
a
work
product
or
not,
alwaysa
candidate
for
peer
review)
and
then
determining
the
mechanism
of
review
will
decide
to
make
it
a
candidate
for
peer
take
into
account
various
criteria
and
the
review.

circumstances
surrounding
the
use
of
that
work
product.
To
maintain
flexibility,
the
Decision
Maker(
s)
for
peer
review
should
consider
the
full
field
of
possible
work
products
to
identify
those
additional
products
that
might
still
warrant
peer
review
as
well
as
the
full
spectrum
of
peer
review
mechanisms
for
each
product.
Once
a
decision
is
made
to
perform
peer
review,
the
product
is
listed
in
the
annual
submission
of
Candidate
Products
for
Future
Peer
Review
(
List
B
­
see
Section
1.3.2b)).
If
a
decision
is
made
not
to
peer
review
a
scientific
and
technical
work
product,
the
product
is
listed
in
List
C
­
Products
for
Which
a
Decision
has
Been
Made
Not
to
Peer
Review
(
see
Section
1.3.2c)).

2.2.3
How
Does
One
Determine
Whether
a
Scientific
and/
or
Technical
Work
Product
is
 
Major ?

Determinations
of
a
scientific
and/
or
technical
work
product
as
 
Major 
will
largely
be
case­
by­
case.
The
continuum
of
work
products
covers
the
range
from
the
obviously
major,
which
clearly
need
peer
review,
to
those
products
which
are
not
major
and
clearly
don t
need
peer
review.
The
rest
of
the
work
products
fall
in­
between
those
two
distinctions.
This
 
middle­
ground 
probably
represents
the
majority
of
work
products,
each
of
which
needs
to
be
evaluated
closely
and
be
compared
to
certain
criteria
(
see
below).
The
Decision
Maker
needs
to
make
a
judgment
as
to
whether
a
work
product
meets
the
criteria
for
major
or
not.
There
is
no
easy
single
yes/
no
test
of
major
covering
the
whole
continuum
of
work
products.
A
rule
of
thumb
to
remember
­­
if
there
is
any
doubt
about
whether
a
work
product
needs
peer
review,
then
go
ahead
and
consider
it
a
candidate
for
peer
review
(
and
place
it
on
List
B
­
Candidate
Products
for
Future
Peer
Review).

Scientific
and
technical
work
products
that
are
used
to
support
a
regulatory
program
or
policy
position
and
that
meet
one
or
more
of
the
following
criteria
are
candidates
for
peer
review:

a)
Establishes
a
significant
precedent,
model,
or
methodology
b)
Addresses
significant
controversial
issues
Peer
Review
Handbook
Page
27
c)
Focuses
on
significant
emerging
issues
d)
Has
significant
cross­
Agency/
inter­
agency
implications
e)
Involves
a
significant
investment
of
Agency
resources
f)
Considers
an
innovative
approach
for
a
previously
defined
problem/
process/
methodology
g)
Satisfies
a
statutory
or
other
legal
mandate
for
peer
review
Usually,
a
major
scientific
and/
or
technical
work
product
supports
a
regulatory
decision
or
policy/
guidance
of
major
impact.
Major
impact
can
mean
that
it
will
have
applicability
to
a
broad
spectrum
of
regulated
entities
and
other
stakeholders,
or
that
it
will
have
narrower
applicability,
but
with
significant
consequences
on
a
smaller
geographic
or
practical
scale.
The
scientific
and/
or
technical
work
that
underlies
many
of
the
Agency s
major
rulemakings
and
policy
and
guidance
documents
of
general
applicability
would
be
designated
 
major 
under
this
scope
of
impact
criterion
because
of
their
far­
reaching
or
significant
impacts.

The
novelty
or
controversy
associated
with
the
work
product
helps
determine
whether
it
is
major
or
not.
A
major
work
product
may
be
novel
or
innovative,
precedential,
controversial,
or
emerging
( 
cutting
edge ).
An
application
of
an
existing,
adequately
peer
reviewed
methodology
or
model
to
a
situation
that
departs
significantly
from
the
situation
it
was
originally
designed
to
address
is
a
candidate
for
peer
review.
Similarly,
a
modification
of
an
existing,
adequately
peer
reviewed
methodology
or
model
that
departs
significantly
from
its
original
approach
is
a
candidate
for
peer
review.
Determination
of
 
significant
departure 
as
used
in
this
Section
is
the
responsibility
of
the
Decision
Maker.

In
summary,
a
major
scientific
or
technical
work
product
has
a
major
impact,
involves
precedential,
novel,
and/
or
controversial
issues,
or
the
Agency
has
a
legal
and/
or
statutory
obligation
to
conduct
a
peer
review.

2.2.4
What
Economic
Work
Products
Need
Peer
Review?

Economic
analyses
and
reports
are
considered
scientific
and
technical
work
products,
and
as
such
they
are
candidates
for
peer
review.
The
following
economic
work
products
will
normally
be
classified
as
major
and
will
require
peer
review:

a)
internal
Agency
guidance
for
conducting
economic
and
financial
analysis
Page
28
Peer
Review
Handbook
b)
new
economic
and
financial
methodologies
that
will
serve
as
a
principal
method
or
protocol
used
to
conduct
economic
analyses
within
a
program
c)
unique
or
novel
applications
of
existing
economic
and
financial
methodologies,
particularly
those
that
are
recognized
to
be
outside
of
mainstream
economic
practices
d)
broad­
scale
economic
assessments
of
regulatory
programs,
such
as
those
required
by
Congressional­
mandates
(
e.
g.,
the
Clean
Air
Act
reports
to
Congress
on
benefits
and
costs)

e)
new
stated
preference
(
e.
g.,
contingent
valuation)
and
revealed
preference
surveys
(
e.
g.,
recreational
travel
cost
surveys)
developed
to
assist
in
the
economic
analysis
of
a
regulation
or
program
f)
new
national
surveys
of
costs
and
expenditures
for
environmental
protection
(
e.
g.,
financial
needs
surveys,
pollution
abatement
expenditures
surveys)

g)
economic
research
plans
developed
to
assess
and
advance
the
state­
of­
science
in
economic
theory,
methodologies
or
modeling
(
in
particular,
the
technical
feasibility
of
the
plan s
components)

h)
new
meta­
analyses
that
re­
analyze
existing
published
literature
and
supporting
data
on
the
measurement
of
economic
benefits,
costs
and
impacts
Generally,
because
of
the
nature
of
these
types
of
economic
work
products,
you
should
conduct
an
external
peer
review.
External
peer
reviews
can
be
provided
by
the
Science
Advisory
Board s
Environmental
Economics
Advisory
Committee,
or
other
appropriate
outside
organizations
or
individuals
that
have
expertise
in
the
technical
economic
issues
raised
in
the
economic
work
product.

2.2.5
Should
Economic
Work
Products
Prepared
in
Support
of
Regulations
that
are
Classified
as
"
Major"
or
"
Economically
Significant"
be
Peer
Reviewed?

Normally
they
would
not
require
peer
review,
if
the
economic
work
product
applies
accepted,
previously
peer
reviewed
methods
in
a
straightforward
manner.
Economic
studies
prepared
to
support
 
major 
or
 
economically
significant 
regulations
( 
major 
as
defined
below
in
this
section)
typically
do
not
utilize
innovative
or
untried
economic
methods.
It
is
unnecessary
to
conduct
peer
reviews
of
straightforward
applications
or
transfers
of
accepted,
previously
peer
reviewed
economic
methods
or
analyses.
Economic
assessments
prepared
to
support
the
regulatory
development
process
routinely
make
use
of
previously
published
peer
Peer
Review
Handbook
Page
29
reviewed
literature
and
adopt
tools
that
allow
for
the
transfer
or
adaptation
of
these
techniques
and
information.
The
procedures
used
to
transfer
or
adapt
this
work
will
generally
be
established
by
separate
economic
guidance
documents
which
have
been
peer
reviewed.
Therefore,
economic
documents
that
are
developed
using
these
procedures
will
not
normally
require
additional
peer
review,
even
those
prepared
in
support
of
 
major 
and
 
economically
significant 
rules.

Even
where
peer
review
is
not
needed,
additional
peer
input
can
be
beneficial
in
the
development
of
economic
work
products
for
 
major 
and
 
economically
significant 
rules.
At
present,
some
peer
input
of
these
analyses
is
already
likely
to
be
included
as
part
of
the
regulatory
development
process,
including
input
received
from
other
EPA
offices
represented
on
the
workgroup
for
the
rule,
the
Agency s
Regulatory
Steering
Committee,
and
from
the
public
as
part
of
the
public
comment
process
for
the
rule.
But
there
may
be
added
benefit
to
employing
additional
peer
input
procedures,
such
as
actively
soliciting
input
from
economists
in
other
Agency
offices
on
the
quality
and
completeness
of
the
economic
analysis.

 
Economically
significant 
rules
under
Executive
Order
12866
are
defined
as
rules
that
may
have
an
annual
effect
on
the
economy
of
$
100
million
or
more
or
adversely
affect
in
a
material
way
the
economy,
a
sector
of
the
economy,
productivity,
competition,
jobs,
the
environment,
public
health
or
safety,
or
State,
local,
or
tribal
governments
or
communities.
The
term
 
major, 
as
used
in
this
Q&
A,
does
not
mean
the
same
thing
as
the
definition
of
 
major 
for
purposes
of
deciding
whether
a
work
product
requires
peer
review.
Here,
we
are
using
the
term
as
defined
in
the
Congressional
Review
Act,
which
defines
a
 
major
rule 
as
one
that
has
resulted
in
or
is
likely
to
result
in:
an
annual
effect
on
the
economy
of
$
100
million
or
more;
a
major
increase
in
costs
or
prices
for
consumers,
individual
industries,
Federal,
State,
or
local
government
agencies,
or
geographic
regions;
or
significant
adverse
effects
on
competition,
employment,
investment,
productivity,
innovation,
or
on
the
ability
of
U.
S.­
based
enterprises
to
compete
with
foreign­
based
enterprises
in
domestic
and
export
markets.

2.2.6
What
Other
Economic
Work
Products
Might
Benefit
from
Peer
Review?

There
may
be
other
economic
work
products
not
covered
in
the
preceding
sections
for
which
peer
review
might
be
useful.
Examples
of
such
work
products
are
presented
below;
however,
we
do
not
intend
to
establish
a
presumption
of
peer
review
for
these
work
products.

a)
analyses
measuring
the
economic
impacts
and
effectiveness
of
adopting
market­
based
or
economic
incentives
as
regulatory
management
instruments
b)
analyses
of
economic
policies
established
under
other
government
organizations
(
e.
g.,
economic
models
used
to
study
transportation,
economic
development,
and
international
trade
policies)
Page
30
Peer
Review
Handbook
Most
of
these
types
of
economic
work
products
do
not
exhibit
the
degree
of
complexity,
or
establish
an
innovative
or
untried
approach,
that
would
warrant
a
peer
review.
However,
other
factors,
such
as
the
potential
significance
of
the
analysis
for
cross­
Agency
or
inter­
agency
practices,
or
the
significance
of
the
issues
addressed,
may
make
peer
review
desirable.

2.2.7
What
Other
Social
Science
Work
Products
Need
Peer
Review?

Typically,
a
social
science
work
product
is
one
that
includes
empirical,
logic­
based
approaches
to
answer
technical
questions
about
human
motivation,
human
behavior,
social
interactions,
and
social
processes,
which
are
relevant
to
the
environmental
issues
being
addressed.
The
term
 
behavior 
includes
overt
actions;
underlying
psychological
processes
such
as
cognition,
emotion,
temperament
and
motivation;
and
biobehavioral
interactions.
The
term
 
social 
includes
sociocultural,
socioeconomic,
and
sociodemographic
status;
biosocial
interactions;
and
the
various
levels
of
social
context
from
small
groups
to
complex
cultural
systems.
Examples
of
social
science
work
products
include
analyses
and/
or
evaluations
related
to
such
topics
as
pollution
prevention,
risk
communication,
environmental
information,
environmental
justice,
quality
of
life,
decision­
making,
and
public
participation.

The
following
social
science
work
products
will
normally
be
classified
as
major
and
will
require
peer
review:

a)
internal
Agency
guidance
for
conducting
social
impact
assessments
and
other
community
cultural
assessments
related
to
different
environmental
protection
approaches
such
as
community­
based
watershed
protection
(
heretofore
referred
to
as
social
assessments).

b)
new
social
science
methodologies
that
will
serve
as
a
principal
method
or
protocol
used
to
conduct
social
assessments.

c)
unique
or
novel
applications
of
existing
social
science
methods
such
as
surveys,
focus
groups,
interviews,
network
analyses,
comparative
analyses,
and
content
analyses.

d)
new
national
surveys
of
values,
perceptions
and
preferences
related
to
environmental
protection.

e)
innovative
research
or
analyses
that
address
the
human
dimensions
of
environmental
protection
or
environmental
change
in
terms
of
social
trends,
future
predictions
and/
or
behavioral
generalizations.
Peer
Review
Handbook
Page
31
f)
social
science
research
plans
developed
to
assess
and
advance
the
state­
of­
science
in
social
science
theory,
methodologies
or
modeling
(
in
particular,
the
technical
feasibility
of
the
plan s
components)

2.2.8
How
Should
Peer
Review
be
Handled
for
Products
Developed
under
an
Interagency
Agreement
(
IAG)?

Under
an
interagency
agreement
(
IAG),
EPA
provides
funds
to
another
agency
for
that
agency
to
use
for
a
specific
purpose.
The
receiving
agency's
guidance
for
peer
review
will
most
likely
be
different
from
the
EPA
peer
review
policy.
Therefore,
if
EPA
plans
to
use
any
work
products
from
that
agreement,
EPA
must
decide
whether
that
document
needs
review
under
the
EPA
peer
review
policy
(
see
Section
2.2.16).

2.2.9
Should
Products
from
Contracts,
Grants,
and
Cooperative
Agreements
Receive
Peer
Review?

If
there
is
a
scientific
and/
or
technical
work
product
resulting
from
a
grant,
contract,
or
cooperative
agreement
and
it
is
considered
major
and
will
likely
be
used
in
Agency
decision­
making,
the
work
product
needs
peer
review.
Since
it
would
probably
result
in
a
perceived,
if
not
real,
conflict
of
interest,
a
group
that
is
generating
the
work
product
usually
cannot
conduct
or
perform
the
peer
review
of
its
own
work
product.
Exceptions
may
be
made
in
certain
instances
for
organizations
that
have
adequate
and
well
established
recognized
procedures
for
peer
review,
such
as
the
National
Academy
of
Sciences
(
NAS).
In
practice,
the
Agency
may
need
to
peer
review
the
product
on
its
own,
or
arrange
with
an
independent
third
group
(
e.
g.,
via
another
extramural
vehicle)
to
conduct
the
peer
review.
The
Agency
should
not
use
the
major
scientific
and
technical
work
products
from
contracts,
grants,
or
cooperative
agreements
to
support
decision
making
unless
the
work
products
are
peer
reviewed
for
both
scientific
and
technical
rigor
and
applicability
to
the
specific
use
to
be
made
of
the
product.

Be
aware
that
contracts
are
very
different
from
grants
and
cooperative
agreements.
Please
note
that
there
are
important
legal
restrictions
(
discussed
in
2.2.10
below)
on
the
direct
use
of
work
products
developed
under
grants
and
cooperative
agreements
in
the
Agency s
decision­
making
process.

2.2.10
How
Does
Peer
Review
Apply
to
Products
Generated
Through
EPA
Grants
or
Cooperative
Agreements?

Major
scientific
and
technical
work
products
that
are
generated
through
EPA
grants
or
cooperative
agreements
are
candidates
for
peer
review,
but
special
considerations
apply.
Page
32
Peer
Review
Handbook
First,
you
need
to
understand
some
background
information
on
the
proper
use
of
assistance
agreements
(
grants
and
cooperative
agreements)
versus
contracts.
Under
the
Federal
Grant
and
Cooperative
Agreement
Act,
grants
or
cooperative
agreements
may
only
be
used
where
the
principal
purpose
of
the
agreement
is
to
accomplish
a
public
purpose
that
is
authorized
by
statute.
EPA
may
derive
some
incidental
use
or
benefit
from
the
award
as
long
as
the
principle
purpose
of
the
project
is
public
support.
However,
if
the
principal
purpose
of
the
agreement
is
to
obtain
a
product
or
service
for
the
direct
benefit
or
use
of
EPA,
a
contract
must
be
used
rather
than
a
grant
or
cooperative
agreement.

EPA
Order
5700.1,
 
Policy
for
Distinguishing
Between
Assistance
and
Acquisition, 
includes
the
following
examples
of
projects
that
canno
t
be
performed
using
grants
or
cooperative
agreements:

a)
Research
and
studies
which
gather
specific
information
desired
by
EPA
for
its
own
use.

b)
Research
which
provides
technical
or
analytical
advice
for
EPA s
direct
benefit
or
use,
such
as
information
used
to
set
guidelines.

c)
Projects
that
produce
specific
information
that
will
be
directly
incorporated
by
EPA
into
technical,
policy,
or
regulatory
decisions.

Note
that
under
the
Order,
EPA
may
legally
provide
financial
assistance
for
research
that
is
intended
to
stimulate
or
support
development
of
scientific
knowledge
that
is
not
primarily
for
EPA s
direct
use
or
benefit.
The
resulting
work
products
would
be
widely
disseminated
either
through
publication
in
scientific
journals
or
through
other
means
as
opposed
to
a
report
tailored
to
EPA s
specific
needs
and
requirements.
EPA
can
consider
these
work
products
just
as
it
can
review
other
published
scientific
works
when
formulating
its
programs
and
policies.
Further,
EPA
retains
a
royalty
free,
nonexclusive
and
irrevocable
right
to
use
the
work
products
for
Federal
purposes,
even
if
the
recipient
has
copyrighted
the
material.
[
40
CFR
30.36(
a).]

Consult
the
Order
and
OGC
for
more
information
about
making
the
proper
choice
between
an
assistance
agreement
and
a
contract.

Provided
that
EPA s
use
of
a
grant
or
cooperative
agreement
recipient s
work
product
is
incidental
to
the
agreement s
principal
purpose,
EPA
may
still
determine
that
the
recipient s
work
product
is
a
 
major
scientific
or
technical
work
product 
under
EPA s
peer
review
policy
because:
1)
it
will
be
used
to
support
an
EPA
program
or
policy
position
(
assuming
this
use
is
incidental
to
the
principal
purpose
of
the
agreement),
and
2)
it
meets
one
or
more
of
the
seven
criteria
outlined
in
Section
2.2.3.
In
this
situation,
the
work
product
requires
peer
review.
Peer
Review
Handbook
Page
33
The
following
are
options
for
peer
reviewing
the
product:

a)
EPA
can
have
the
product
peer
reviewed
with
the
participation
of
the
assistance
agreement
recipient/
author
.
In
this
case,
EPA
could
arrange
for
an
independent
peer
review
of
the
product
itself,
or
may
contract
with
a
third
party
to
conduct
the
review.
EPA
would
also
enter
into
a
contract
with
the
author
(
formerly
the
recipient),
which
would
task
the
author
to
prepare
a
response
to
the
peer
reviewers 
comments,
and
to
revise
or
prepare
an
addendum
to
the
product
in
response
to
peer
reviewer
comments
as
determined
appropriate
by
EPA.

A
caveat
to
this
approach
is
that
it
may
be
difficult
to
get
the
recipient/
author
to
agree
to
allow
EPA
to
determine
how
to
revise
the
product
in
response
to
the
peer
review
comments.

b)
EPA
can
have
the
product
peer
reviewed
without
the
participation
of
the
recipient/
author
.
EPA
could
arrange
for
the
peer
review
itself,
or
could
contract
with
a
third
party
to
conduct
the
review.
In
this
case,
however,
the
work
product
would
not
be
revised
to
incorporate
the
peer
review
comments.
Instead,
EPA
would
receive
the
comments
and
prepare
a
statement
that
documents
its
own
response
to
the
comments.
The
EPA
Decision
Maker(
s)
who
is
using
the
work
product
to
support
an
EPA
program
or
policy
decision
must
be
provided
information
on
both
the
conclusions
of
the
recipient s
work
product
and
EPA s
own
conclusions
from
the
peer
review.

c)
Recipients
can
get
their
products
peer
reviewed
on
their
own.
Recipients
may
determine
on
their
own
that
peer
review
would
benefit
the
credibility
of
their
product.
Provided
EPA
agrees
that
a
peer
review
would
further
the
public
purpose
of
the
assistance
agreement,
EPA
may
include
funds
for
the
peer
review
in
the
agreement.
(
See
Section
2.2.11
for
additional
information.)
Alternatively,
the
recipient
may
make
arrangements
for,
and
fund,
an
independent
peer
review
of
their
product.
In
either
case,
EPA
would
need
to
evaluate
whether
the
peer
review
process
undertaken
by
the
recipient
was
acceptable
for
the
purposes
for
which
EPA
was
planning
to
use
the
work
product.
EPA
may
accept
the
peer
review
if
it
determines
that
it
fulfilled
the
requirements
of
EPA s
Peer
Review
Handbook
and
that
EPA
could
defend
the
peer
review
as
if
it
were
conducted
by
EPA
itself.

Under
options
a)
or
b),
issues
may
arise
over
obtaining
access
for
peer
reviewers
to
the
raw
data
used
by
the
recipient
to
generate
the
work
product.
Under
40
CFR
30.36(
c)(
2),
EPA
has
a
right
to
obtain
raw
data
produced
by
a
non­
profit
organization
or
university
under
an
assistance
agreement,
even
where
the
agreement
doesn t
specifically
provide
for
this
access.
Page
34
Peer
Review
Handbook
Nevertheless,
it
may
be
prudent
to
include
a
specific
term
in
the
assistance
agreement
clarifying
this
point
to
avoid
misunderstandings.
EPA
must
pay
for
obtaining
access
to
the
data
if
its
transmittal
imposes
additional
costs
on
the
recipient.
Assuming
our
use
of
the
data
is
incidental
to
the
principal
purpose
of
the
agreement,
we
also
have
a
specific
right
to
authorize
peer
reviewers
to
use
the
data
for
Federal
purposes
under
40
CFR
30.36(
c)(
2).

The
only
time
EPA
cannot
obtain
access
to
the
raw
data
is
where
EPA
specifically
bargains
away
this
right
in
the
assistance
agreement.

Again,
consult
OGC
for
help
in
drafting
appropriate
language
for
your
assistance
agreement.

2.2.11
Can
the
Recipient
of
a
Grant
or
Cooperative
Agreement
Use
Agreement
Funds
to
Pay
Peer
Reviewers
of
their
Work
Products?

As
noted
in
Section
2.2.10
above,
provided
EPA
agrees
that
a
peer
review
would
further
the
public
purpose
of
the
assistance
agreement,
EPA
may
include
funds
for
the
peer
review
in
the
agreement.
A
payment
to
peer
reviewers
in
exchange
for
their
review
of
a
scientific/
technical
work
product
is
allowable
as
a
fee
for
professional
services
under
assistance
agreements.
(
To
accurately
characterize
this
cost,
however,
it
is
important
that
the
payment
be
referred
to
as
a
fee,
rather
than
an
honorarium.)
See
OMB
Circular
A­
21,
Section
J,
item
32,
Professional
Services
Cost
(
Educational
Institutions),
OMB
Circular
A­
122,
Attachment
B,
Item
39,
Professional
Services
Costs
(
Non­
Profit
Organizations),
and
OMB
Circular
A­
87,
Attachment
B,
Item
33,
Professional
Services
Costs
(
State,
Local
and
Indian
Tribal
Governments).
(
See
http://
www.
whitehouse.
gov/
OMB/
circulars/
for
further
details.)

2.2.12
Do
Products
Generated
under
EPA
Grants
or
Cooperative
Agreements
Need
to
be
Reported
in
the
Peer
Review
Product
Tracking
(
PRPT)
Database?

If
a
grant
or
cooperative
agreement
product
is
determined
to
be
a
major
scientific
and/
or
technical
work
product
and
is
used
in
Agency
decision
making
(
assuming
this
use
is
incidental
to
the
principal
purpose
of
the
agreement),
it
must
generally
be
considered
a
candidate
for
peer
review.
Therefore:

a)
if
it
is
determined
to
be
a
major
work
product
and
will
be
peer
reviewed
by
EPA,
the
work
product
is
entered
onto
List
B
(
Candidate
Products
for
Future
Peer
Review)
as
a
 
future
peer
review 
product,
then
moved
to
List
A
(
Products
Peer
Reviewed
Since
1991)
when
the
peer
review
is
completed.

b)
if
it
is
determined
to
be
a
major
work
product,
but
will
not
be
peer
reviewed
(
see
the
possible
reasons
for
not
peer
reviewing
a
major
product
in
Section
2.3.1),
it
is
Peer
Review
Handbook
Page
35
entered
onto
List
C
(
Products
for
Which
a
Decision
has
been
made
not
to
Peer
Review),
with
a
justification
for
not
peer
reviewing
it.

c)
if
it
is
determined
to
not
be
a
major
work
product
but
EPA
decides
that
it
still
warrants
peer
review
(
see
Section
2.2.17),
it
is
entered
onto
List
B
(
Candidate
Products
for
Future
Peer
Review)
as
a
 
future
peer
review 
product,
then
moved
to
List
A
(
Products
Peer
Reviewed
Since
1991)
when
the
peer
review
is
completed.

d)
if
EPA
decides
to
use
the
work
product
(
due
to
its
importance
in
EPA
environmental
decision
making),
but
considers
it
not
major
and
not
a
candidate
for
peer
review,
it
is
entered
onto
List
C
(
Products
for
Which
a
Decision
has
been
made
not
to
Peer
Review)
with
the
reasons
for
it
being
non­
major.

e)
if
the
grant
or
cooperative
agreement
product
will
not
be
used
in
Agency
decision
making,
it
is
not
generally
considered
a
candidate
for
peer
review.
Such
products
do
not
have
to
be
listed.

2.2.13
Should
Site
Specific
Decisions
be
Subject
to
Peer
Review?

The
site
specific
decision
itself
is
not
subject
to
peer
review
and
doesn t
need
peer
review
based
solely
on
the
Peer
Review
Policy.
However,
if
a
site
specific
decision
is
supported
by
a
major
scientific
and/
or
technical
work
product,
that
work
product
needs
peer
review.
While
the
same
considerations
for
major
apply
here,
several
of
the
criteria
above
(
see
Section
2.2.3;
specifically
criteria
b,
c,
d,
and
g)
are
considered
more
useful
for
regional
consideration
than
other
criteria.
So
generally
speaking,
a
close
examination
of
how
the
underlying
major
scientific
and/
or
technical
work
product
is
adapted
to
the
site
specific
circumstances
is
required.

2.2.14
Should
NEPA
Products
(
e.
g.,
EISs)
be
Subject
to
Peer
Review?

Not
everything
requires
peer
review,
and
in
the
case
of
an
Environmental
Impact
Statement
(
EIS)
prepared
under
the
requirements
of
the
National
Environmental
Policy
Act
(
NEPA),
the
document
already
has
received
extensive
review
(
although
not
necessarily
peer
review)
through
the
 
scoping 
and
interagency
review
processes
that
are
part
of
NEPA.

The
rule
of
thumb
is
that
if
the
underlying
scientific
and/
or
technical
work
product
is
major,
then
the
work
product
needs
peer
review.
In
general,
the
Agency s
role
in
the
NEPA
document
would
suggest
what
sort
of
review
the
document
gets.
If
EPA
is
developing
the
document
as
part
of
an
EPA
action/
decision
(
EPA
is
the
Lead
agency
under
NEPA),
and
it
meets
the
definition
of
"
major,"
then
it
needs
independent
peer
review.
If
it
is
not
a
major
work
Page
36
Peer
Review
Handbook
product
(
little
impact,
non­
controversial,
etc.),
then
peer
input/
continuing
involvement
might
well
be
appropriate.

On
the
other
hand,
if
EPA
is
reviewing
an
EIS
from
another
Agency
(
EPA
is
not
the
Lead
agency
under
NEPA),
it
is
likely
that
we
are
reviewing
for
conflicts
with
EPA
policy
and
general
environmental
concerns.
However,
EPA
must
ask
if
the
underlying
major
scientific
and/
or
technical
work
product
that
supports
the
EIS
has
been
peer
reviewed.
If
not,
this
would
raise
concern
about
the
full
credibility
and
soundness
of
the
EIS
based
on
the
science
and
technical
support.
EPA
should
work
with
the
other
organization/
agency
to
ensure
that
the
major
scientific
and/
or
technical
work
product
receives
peer
review
adequate
for
EPA
purposes.

2.2.15
Should
Environmental
Regulatory
Models
be
Peer
Reviewed?

Generally,
yes.
Specific
guidelines
for
the
peer
review
of
environmental
regulatory
models
have
been
published
by
the
Agency.
These
can
be
found
on
the
EPA
web
site
under
the
Science
Policy
Council
home
page
(
http://
www.
epa.
gov/
ORD/
spc).

2.2.16
Is
Peer
Review
Needed
for
Other
Organization s
Work
Products
that
Have
been
Submitted
to
EPA
for
Use
in
Decision
Making?

Any
scientific
and/
or
technical
work
product
that
is
used
in
Agency
decision
making
and
is
considered
major
becomes
a
candidate
for
peer
review
regardless
of
whether
the
work
product
is
produced
by
the
Agency
or
another
organization.
Therefore,
all
major
work
products
important
to
EPA
environmental
decision
making
that
are
independently
generated
by
other
organizations
(
e.
g.,
other
Federal
agencies,
interagency
groups,
State
and
Tribal
bodies,
environmental
groups,
industry,
educational
institutions,
international
bodies)
need
to
be
considered
as
candidates
for
peer
review
just
as
major
work
products
generated
by
EPA
are
considered
(
these
would
then
be
included
on
List
B
(
Candidate
Products
for
Future
Peer
Review)
if
EPA
is
conducting
or
arranging
for
the
peer
review).

If
possible,
when
EPA
knows
that
a
work
product
is
being
generated
by
another
organization
and
is
of
interest
to
EPA
for
future
use,
the
appropriate
EPA
office(
s)
should
work
with
that
organization,
and
others,
as
appropriate
(
e.
g.,
the
states),
to
promote
the
use
of
peer
review.
For
example,
the
Office
of
International
Activities
(
OIA)
as
well
as
the
impacted
program
or
regional
office(
s)
should
be
included
when
international
products
are
being
considered
for
EPA
use.

It
is
hoped
that
if
the
other
organization
has
the
work
product
independently
peer
reviewed,
the
peer
review
will
meet
the
intent
of
the
Agency s
Peer
Review
Policy
and
EPA s
proposed
use
of
the
product
(
i.
e.,
the
peer
review
is
basically
equivalent
to
what
EPA
would
do).
Agency
staff
from
the
appropriate
office(
s)
should
examine
closely
the
particulars
of
the
peer
Peer
Review
Handbook
Page
37
review
to
ensure
independence
and
a
conscious
effort
to
incorporate
the
peer
reviewers 
comments
into
the
final
work
product.
If
there
are
perceived,
or
real,
conflicts
of
interest,
this
may
preclude
the
use
of
that
peer
review
and,
in
those
instances,
another
peer
review
would
be
needed.
See
Section
3.4.7
for
considerations
for
when
an
outside
party
conducts
and/
or
funds
peer
review
of
their
own
work
product
and
submits
it
to
the
Agency.

If
the
outside
organization
does
not
have
the
major
work
product
peer
reviewed
and
EPA
decides
it
needs
peer
review,
the
appropriate
EPA
office(
s)
will
have
to
ensure
peer
review
of
that
work
product
occurs
prior
to
the
Agency s
use
of
the
work
product
in
decision
making.
Peer
review
can
be
accomplished
by
asking
the
outside
organization
to
do
so,
or
if
they
refuse,
EPA
may
conduct
or
arrange
for
the
peer
review.
If
EPA
is
conducting
or
arranging
the
peer
review,
the
product
should
be
entered
on
to
List
B
(
Candidate
Products
for
Future
Peer
Review).

2.2.17
Can
Work
Products
That
are
Not
Determined
to
be
Major
Still
be
Peer
Reviewed?

Yes,
they
could
be.
Scientific
and
technical
work
products
that
do
not
come
under
the
 
major 
distinction
discussed
above
may
nonetheless
be
candidates
for
peer
review.
For
example,
a
project
manager
may
decide
to
use
peer
review
because
of
particular
program
needs
and
goals.
Peer
review
may
also
be
warranted
because
it
adds
substantial
value
to
the
work
product.
In
these
cases,
the
product
should
be
entered
onto
List
B
(
Candidate
Products
for
Future
Peer
Review)
as
a
candidate
for
future
peer
review.

2.3
Determining
Which
Work
Products
Do
Not
Receive
Peer
Review
2.3.1
Are
There
Circumstances
When
a
Major
Work
Product
is
Not
Peer
Reviewed?

There
may
be
circumstances
where
a
work
product
is
considered
major,
but
a
decision
for
no
peer
review
can
then
be
justified.
For
example:

a)
Additional
peer
review
is
not
required
with
work
that
has
been
previously
reviewed
by
recognized
experts
or
an
expert
body.
For
example,
a
cancer
risk
assessment
methodology
or
an
exposure
modeling
technique
that
was
the
subject
of
earlier
peer
review
would
not
require
additional
peer
review,
even
if
the
product
supported
a
significant
Agency
decision.

b)
Additional
peer
review
is
not
required
if
an
application
of
an
adequately
peer
reviewed
work
product
does
not
depart
significantly
from
its
scientific
or
technical
approach.
Page
38
Peer
Review
Handbook
c)
Additional
peer
review
is
not
required
when
the
scientific
and/
or
technical
methodologies
or
information
being
used
are
commonly
accepted
in
the
field
of
expertise
(
e.
g.,
Control
Techniques
Guidelines
(
CTGs)
or
other
such
compilations).
This
would
need
the
appropriate
documentation
to
support
the
commonly
held
view.

d)
Most
often,
a
major
work
product
would
not
receive
peer
review
when
the
regulatory
activity
or
action
which
the
work
product
supports
is
terminated
or
canceled
­­
no
further
action,
including
peer
review,
is
necessary.

e)
In
a
few
instances,
statutory
or
court
ordered
deadlines
or
other
time
constraints
may
limit
or
preclude
peer
review
of
product
that
would
otherwise
be
considered
major.
However,
it
is
up
to
the
Decision
Maker(
s)
to
make
every
attempt
possible
to
assure
that
peer
review
of
major
products
occurs
taking
into
account
these
deadlines.
With
proper
up­
front
planning
for
peer
reviews
of
major
products
and
the
ability
to
tailor
the
method
of
review
to
the
product
and
circumstances,
it
should
rarely
be
the
case
that
major
products
don t
receive
some
type
of
peer
review
due
to
time
constraints.

f)
Very
rarely,
resource
limitations
may
also
restrict
peer
review.
Programs
or
Regions
will
evaluate
these
circumstances
on
a
case
by
case
basis;
decisions
will
be
based
on
consultations
involving
line
management,
the
Project
Manager,
the
Peer
Review
Leader,
and
the
Peer
Review
Coordinator.

If
peer
review
of
a
major
scientific
and/
or
technical
work
product
is
not
conducted,
a
written
justification
must
be
placed
in
the
 
Justification
for
Non­
Peer
Review
Product 
Section
of
the
data
base
entry
for
that
product
on
List
C
(
Products
for
Which
a
Decision
has
been
made
not
to
Peer
Review).
The
justification
is
signed­
off
by
the
appropriate
Decision
Maker
(
see
Section
1.3.2c)).

2.3.2
What
Products
Normally
Do
Not
Need
Peer
Review?

Products
that
are
not
major
scientific
and
technical
work
products
normally
do
not
require
peer
review
under
the
intent
of
the
Peer
Review
Policy.
Most
of
these
scientific
and
technical
work
products
are
then
placed
on
List
C
(
Products
for
Which
a
Decision
has
Been
Made
Not
to
Peer
Review)
with
a
written
explanation
of
why
it
was
determined
to
be
not
major
(
see
Section
1.3.2c)).
This
justification
is
signed­
off
by
the
appropriate
Decision
Maker
to
assure
that
all
scientific
and
technical
work
products
received
consideration
for
peer
review.

Some
scientific
and
technical
work
products
are
not
considered
major
and
generally
do
not
need
to
be
placed
on
List
C
(
Products
for
Which
a
Decision
has
Been
Made
Not
to
Peer
Peer
Review
Handbook
Page
39
Review).
These
types
of
work
products
typically
include:
derivative
products
(
i.
e.,
a
product
that
only
summarizes
an
already
peer
reviewed
product
or
products;
fact
sheets),
compendiums
of
existing
models,
methods
and/
or
technologies;
minor
intermediate
products
(
e.
g.,
a
technical
memorandum
from
a
contractor)
describing
methods
or
results
incorporated
in
a
larger
product
which
will
be
peer
reviewed;
or
preliminary
or
incidental
analyses
prepared
separately
from
the
work
product
ultimately
used
to
support
an
Agency
action
or
decision
(
e.
g.,
during
the
course
of
developing
a
rule,
managers
may
direct
staff
to
prepare
various
 
what
if 
analyses;
those
that
aren t
used
in
the
work
product
do
not
need
to
be
listed).

2.3.3
Do
Voluntary
Consensus
Standards
Require
Peer
Review?

Generally,
no.
The
National
Technology
Transfer
and
Advancement
Act
of
1995
(
NTTAA)
directs
EPA
to
use
available
voluntary
consensus
standards
in
its
regulatory
activities
unless
to
do
so
would
be
inconsistent
with
applicable
law
or
otherwise
impractical.
For
purposes
of
the
NTTAA,
voluntary
consensus
standards
are
defined
as
technical
standards
(
e.
g.,
materials
specifications,
test
methods,
sampling
procedures,
and
business
practices)
that
are
developed
or
adopted
by
voluntary
consensus
bodies
(
such
as
ISO,
ASTM).
The
general
purpose
of
the
NTTAA
is
to
reduce
private
and
governmental
costs
by
avoiding
having
the
government
 
reinvent
the
wheel. 
Voluntary
consensus
standards
would
normally
not
require
peer
review
because
the
underlying
process
used
by
issuing
organizations
to
develop
and
approve
these
standards
is
generally
considered
adequate
for
purposes
of
the
Agency s
peer
review
policy.
EPA
reserves
the
right
to
conduct
a
peer
review
if
it
determines
that
the
standard
is
not
a
voluntary
consensus
standard
under
the
NTTAA.

2.4
Choosing
a
Peer
Review
Mechanism
2.4.1
How
Do
You
Determine
the
Appropriate
Peer
Review
Mechanism?

During
the
planning
of
a
peer
review,
the
Decision
Maker
and
the
Peer
Review
Leader
may
consider
several
mechanisms
for
the
peer
review
of
major
scientific
and
technical
work
products.
These
options
range
from
consultations
with
EPA
colleagues
not
involved
in
developing
the
product
to
a
large
and
formal
panel
of
outside
subject
matter
experts.
The
peer
review
effort
might
be
a
focused
one­
time
evaluation,
or
could
The
mechanism
of
the
peer
encompass
several
examinations
over
the
course
of
a
review
matches
the
project.
In
principle,
peer
review
provides
the
greatest
importance
and
complexity
of
credibility
for
major
work
products
when
it
involves
well­
the
major
work
product.
qualified
external
reviewers,
is
intensive
in
its
examination,
and
operates
through
a
more
or
less
formal
process.
As
a
practical
matter,
however,
time
and
resource
considerations
in
many
cases
impose
limitations
on
what
can
be
reasonably
achieved.
Arranging
for
the
most
appropriate
and
feasible
peer
review
will
involve
good
judgment
and
a
willingness
to
consider
substance,
time,
and
resource
tradeoffs.
Developing
a
peer
review
plan
that
provides
for
appropriate
depth,
timing,
Page
40
Peer
Review
Handbook
and
content
is
an
important
matter
for
early
consideration
by
the
Decision
Maker
and
Peer
Review
Leader.
Note
that
use
of
peer
input,
public
or
stakeholder
involvement
does
not
qualify
as
peer
review.

The
approach
best
suited
to
a
specific
work
product
will
depend
on
the
nature
of
the
topic
and
the
intended
final
product.
Generally,
the
more
novel
or
complex
the
science
or
technology,
the
greater
the
cost
implications
of
the
impending
decision,
and
the
more
controversial
the
issue,
then
the
stronger
the
indication
for
a
more
extensive
and
involved
peer
review
and
for
external
peer
review
in
particular.
Certain
work
products
will
clearly
lend
themselves
to
extensive
external
peer
review;
generally
these
will
be
products
with
large
impacts
(
e.
g.,
those
that
support
Tier
1
and
Tier
2
rulemakings).
Other
major
work
products
may
not
need
a
large
scale
external
peer
review
and
may
utilize
a
less
involved,
less
resource
intensive
review.
The
peer
review
of
some
products
may
be
better
served
with
some
form
of
internal
peer
review
or
a
combination
of
internal
and
external
peer
review.

It
is
important
to
make
the
choice
of
peer
review
mechanism
at
the
time
that
the
work
is
planned
(
for
products
supporting
rule
makings,
at
the
analytic
blueprint
stage)
so
that
peer
review
costs
and
time
can
be
budgeted
into
the
work
plan.
Essentially,
the
level
of
peer
review
matches
the
impact
and
complexity
of
the
major
work
product.
For
example,
a
rule
under
development
carries
considerable
weight
and
deserves
careful
handling
and
attention;
therefore,
the
supporting
work
product
deserves
similar
care
and
attention
for
its
peer
review.
Both
internal
and
external
peer
review
mechanisms
are
available,
have
been
used
in
the
past,
and
have
served
to
address
the
needs
and
challenges
of
a
particular
peer
review
situation.
Nevertheless,
no
single
peer
review
mechanism
is
likely
to
work
best
in
all
situations.
Some
useful
guidance
includes:

a)
Major
work
products
intended
to
support
the
most
important
decisions,
or
that
have
special
importance
in
their
own
right,
ordinarily
should
be
the
subject
of
external
peer
review.
Generally,
the
more
complex,
novel
and/
or
controversial
the
product,
or
the
higher
impact
it
has,
the
more
the
Decision
Maker
should
consider
implementing
a
large­
scale
peer
review
involving
external
experts.

b)
Major
work
products
that
are
less
complex,
novel,
or
controversial,
or
have
a
lower
impact
may
not
need
such
a
large­
scale
and
external
peer
review.
These
products
might
be
subject
to
one
of
the
less
extensive,
less
resource­
intensive
review
processes.

c)
Group
discussion
with
peer
reviewers
can
be
very
helpful
at
some
point
in
the
peer
review
process.
On
the
other
hand,
simply
soliciting
individual
comments
is
easier,
faster,
and
less
expensive.
Individual
review
is
probably
more
appropriate
Peer
Review
Handbook
Page
41
for
peer
review
at
the
early
stages
of
a
product s
development
or
for
products
with
less
impact
and
complexity.

d)
Strict
time
constraints,
such
as
a
court­
ordered
deadline,
can
make
a
less
involved
or
formal
peer
review
mechanism
imperative.
But
Decision
Makers
and
Peer
Review
Leaders
must
make
maximum
efforts
to
assure
that
such
a
process
is
perceived
as
systematic
and
objective.

2.4.2
What
are
Examples
of
Internal
Peer
Review?

a)
Independent
experts
from
within
the
Agency
(
e.
g.,
ORD
experts
on
non­
cancer
effects
of
lead
reviews
a
draft
article
on
benchmark
dose)

b)
An
ad
hoc
panel
of
independent
experts
from
within
the
Agency
(
e.
g.,
an
independent
internal
workgroup
convened
to
examine
the
case
for
the
classification
of
a
chemical
as
a
carcinogen)

c)
Technical
merit
review
by
scientists
in
an
Agency
laboratory
(
e.
g.,
an
initial
review
of
the
risk
assessment
for
a
regional
incinerator
by
Agency
scientists)

2.4.3
What
are
Examples
of
External
Peer
Review?

a)
Independent
experts
from
outside
the
Agency
(
e.
g.,
a
letter
review
by
outside
scientists)

b)
An
ad
hoc
panel
of
independent
experts
outside
the
Agency
(
e.
g.,
a
group
is
convened
to
develop
a
consensus
on
the
carcinogenicity
of
a
particular
industrial
chemical)

c)
Agency­
sponsored
peer
review
workshops
(
e.
g.,
a
review
of
potential
indicators
of
ecosystem
damage)

d)
Review
by
an
established
Federal
advisory
committee
such
as
the
Science
Advisory
Board
(
SAB),
FIFRA
Scientific
Advisory
Panel
(
SAP),
ORD s
Board
of
Scientific
Counselors,
or
the
Clean
Air
Scientific
Advisory
Committee
(
e.
g.,
a
review
of
a
criteria
document
for
a
particular
chemical
risk)

e)
Agency­
based
federal
advisory
committee
(
other
than
those
established
and
discussed
in
d
above)
Page
42
Peer
Review
Handbook
f)
Agency
appointed
special
board
or
commission
(
e.
g.,
a
review
of
the
risk
assessment
methodology
prepared
by
the
Clean
Air
Act
Commission
on
Risk
Assessment)
Note:
The
Office
of
General
Counsel
should
be
consulted
regarding
EPA's
authority
to
establish
and
finance
the
activities
of
a
commission
or
board.

g)
Interagency
committee
(
e.
g.,
a
review
of
prospective
research
plans
by
the
Committee
on
the
Environment
and
Natural
Resources
coordinated
by
the
White
House)

h)
A
committee
convened
by
another
federal
agency
or
government
organization
(
e.
g.,
a
review
of
the
Dioxin
Reassessment
by
the
Health
and
Human
Services
(
HHS)
Committee
to
Coordinate
Environmentally
Related
Programs)

i)
Review
by
non­
governmental
groups
(
e.
g.,
a
Society
of
Risk
Analysis
review
of
cancer
guidelines)

j)
Review
by
the
National
Academy
of
Sciences
(
e.
g.,
a
review
of
the
state
of
current
knowledge
about
children s
health
risks
from
pesticide
exposures)

2.4.4
What
is
the
Role
of
Peer
Review
by
a
Refereed
Scientific
Journal?

Peer
review
of
journal
articles
(
written
by
EPA
or
non­
EPA
authors)
performed
by
a
credible,
refereed
scientific
journal
contributes
to
the
scientific
and
technical
credibility
of
the
reviewed
product.
EPA
considers
peer
review
by
such
journals
as
adequate
for
reviewing
the
scientific
credibility
and
validity
of
the
findings
(
or
data)
in
that
article,
and
is
therefore
a
satisfactory
form
of
peer
review.
However,
in
some
cases,
peer
review
of
an
Agency
work
product
that
uses
these
articles
may
be
required
(
see
Section
2.4.5).

EPA
authored
journal
articles,
whether
used
in
an
Agency
work
product
or
not,
are
tracked
using
List
D
(
Scientific
Articles
and
Reports
That
Are
Peer
Reviewed
by
Organizations
Outside
of
EPA
­­
see
Section
1.3.2d)
for
details).
This
shows
that
these
items
have
been
peer
reviewed,
and
differentiates
them
from
other
Agency
work
products.
In
addition,
it
gives
EPA
an
opportunity
to
highlight
the
extensive
work
it
produces
in
the
scientific
literature.

It
is
not
necessary
to
list
these
papers
individually
on
List
D
(
Scientific
Articles
and
Reports
That
Are
Peer
Reviewed
by
Organizations
Outside
of
EPA),
although
some
organizations
may
choose
to
do
so.
However,
at
a
minimum,
the
total
number
prepared
by
the
organization
and
peer
reviewed
by
journals
must
be
included
on
List
D.
The
organization
may
ultimately
need
to
identify
the
individual
papers
and
where
published,
if
requested
(
e.
g.,
due
to
litigation,
FOIA,
etc.).
Peer
Review
Handbook
Page
43
Prior
to
submitting
an
article
to
a
journal
for
peer
review,
EPA
employees
are
encouraged
to
have
the
article
internally
peer
reviewed
(
see
Section
1.4.9);
such
internal
peer
review
is
already
common
procedure
in
certain
parts
of
EPA.
Articles
may
also
need
examination
in
accordance
with
any
organizational
clearance
procedures,
especially
when
the
author
is
presenting
him
or
herself
as
an
EPA
employee.
For
EPA
employees,
conflict
of
interest
regulations
will
also
apply.

2.4.5
Do
Agency
Work
Products
Become
Candidates
for
Peer
Review
when
Peer
Reviewed
Journal
Articles
are
Used
in
Support
of
that
Work
Product?

In
most
instances,
major
Agency
work
products
are
candidates
for
peer
review
even
when
supported
by
peer
reviewed
journal
article(
s).
Although
the
use
of
articles
that
have
been
peer
reviewed
by
a
credible
journal
strengthens
the
scientific
and
technical
credibility
of
any
work
product
in
which
the
article(
s)
appears
or
is
referenced,
it
does
not
automatically
eliminate
the
need
to
have
the
work
product
itself
peer
reviewed.
In
most
cases,
journal
peer
review
may
not
cover
issues
and
concerns
that
the
Agency
would
want
peer
reviewed
to
support
an
Agency
action.
Under
these
circumstances,
the
major
scientific
and/
or
technical
work
product
in
which
the
article(
s)
appears
or
is
referenced
becomes
a
candidate
for
peer
review.
A
journal
article
authored
by
EPA
employees
would
be
used
in
the
same
manner
as
an
article
published
by
anyone
else
in
a
credible,
well
recognized
journal.

If
an
Agency
work
product
is
based
solely
on
a
single
article
that
has
received
peer
review
by
a
credible
journal
(
e.
g.,
where
a
model
is
suggested
for
a
singular
use
that
fits
a
specific
Agency
need),
peer
review
of
the
Agency
work
product
may
or
may
not
be
necessary
depending
on
how
closely
you
apply
the
findings
from
the
article.
If
an
Agency
work
product
is
based
on
two
or
more
articles
that
have
received
peer
review
by
a
credible
journal(
s),
the
Agency
work
product
generally
becomes
a
candidate
for
peer
review.
Decisions
to
make
(
or
not
make)
a
work
product
a
candidate
for
peer
review
needs
to
be
documented
in
the
peer
review
record.

One
important
factor
to
remember
with
regard
to
the
use
of
articles
that
have
received
journal
peer
review
deals
with
the
availability
of
documentation
from
that
peer
review.
Ideally,
EPA
needs
to
maintain
a
clear,
easily
accessible
record
of
the
peer
review
to
assure
the
credibility
and
validity
of
the
peer
review
(
see
Section
2.5
for
details
on
the
peer
review
record).
However,
the
documentation
from
a
journal
peer
review
would
not
normally
be
available
to
the
Agency,
so
such
documentation
is
not
expected
in
the
peer
review
record.

2.4.6
When
and
How
Often
Should
Peer
Review
Occur?

The
Decision
Maker
and
Peer
Review
Leader
have
significant
discretion
in
deciding
on
the
timing
and
the
frequency
of
peer
review.
Options
abound,
each
with
merits
depending
on
the
context
and
specified
peer
review
objectives.
In
many
situations,
a
single
peer
review
event,
Page
44
Peer
Review
Handbook
beginning
when
the
final
draft
work
product
becomes
available,
is
the
approach
taken.
However,
it
is
increasingly
apparent
that
peer
review
performed
earlier
in
the
work
product
development
stages
is
a
superior
approach
for
some
work
products.
There
may
be
substantial
incremental
benefit
to
conducting
more
than
one
peer
review
during
the
whole
process
of
work
product
development,
particularly
where
it
involves
complex
tasks,
has
decision
branching
points,
or
could
be
expected
to
produce
controversial
findings.
In
addition,
early
review
would
be
beneficial
at
the
stage
of
research
design
or
data
collection
planning
where
the
product
involves
extensive
primary
data
collection.
The
Decision
Maker
and
Peer
Review
Leader
need
to
determine
when
the
peer
review(
s)
should
occur,
considering
the
type
of
work
product
under
development
and
at
what
point
in
its
development
process
a
peer
review
would
be
most
beneficial.

Other
types
of
work
products
that
would
benefit
from
early,
up­
front
peer
review
in
their
development
are
scientific
and
technical
planning
products.
Examples
of
such
products
are
research
proposals,
plans,
and
strategies.
Also,
while
not
products
per
se,
ongoing
research
programs
can
be
peer
reviewed.

Remember
though,
that
while
more
than
one
peer
review
can
be
beneficial,
the
distinction
between
peer
input
and
peer
review
needs
to
be
kept
in
mind.
Experts
providing
input
during
the
development
or
planning
stages
of
the
work
product
generally
do
not
become
peer
reviewers
of
that
product
(
see
Sections
1.2.2
to
1.2.7
for
full
discussion
on
this
distinction).

2.4.7
What
Factors
are
Considered
in
Setting
the
Time
Frame
for
Peer
Review?

Several
factors
impact
how
quickly
a
peer
review
may
be
needed.
These
include
deadlines
for
completion
of
a
project,
research
program,
or
rulemaking,
funding
availability,
availability
of
quality
peer
reviewers,
and
statutory
and/
or
court­
ordered
deadlines.

Peer
review
sometimes
leads
to
new
information
and
analyses.
Reviewers
may
make
recommendations
for
new
research
that
would
alter
the
work
product
and
thus
modify
the
scientific/
technical
basis
for
the
action
or
rule
it
supports.
For
this
reason,
a
completed
peer
review
is
desirable
before
issuing
any
proposal
for
public
comment.
If
that
is
not
logistically
possible
because
of
court
or
statutory
deadlines,
or
other
appropriate
reasons,
the
Decision
Maker
should
make
every
effort
to
complete
the
peer
review
before
the
close
of
the
comment
period.
Because
peer
review
comments
on
such
work
products
could
be
of
sufficient
magnitude
to
warrant
a
revision
to
the
proposed
action
or
rule,
Decision
Makers
should
exercise
diligence
in
completing
the
peer
review
prior
to
the
proposal
stage
whenever
possible.
Peer
Review
Handbook
Page
45
2.4.8
Which
Office/
Region
or
Other
Agency
is
Responsible
for
Conducting
the
Peer
Review?

The
organization
of
the
Decision
Maker
is
normally
responsible
for
conducting
the
peer
review.
Responsibility
for
conducting
a
peer
review
can
be
negotiable
when
more
than
one
Agency
office
or
region
or
other
agencies
are
involved.
Usually,
the
degree
of
involvement
by
any
of
the
organizations
and
agencies
and
their
ability
to
fund
peer
review
will
often
determine
who
has
the
lead
for
the
peer
review.

2.5
Creating
the
Peer
Review
Record
2.5.1
What
is
the
Peer
Review
Record?

It
is
the
formal
record
(
file)
of
decision
on
the
conduct
of
the
peer
review,
the
type
of
peer
review
performed,
and
an
explanation
of
how
the
peer
review
comments
were
addressed.
It
includes
sufficient
documentation
for
an
uninvolved
person
to
understand
what
actually
happened
and
why.
The
Peer
Review
Leader
(
with
the
program
manager
if
there
is
one)
creates
a
separate,
clearly
marked
peer
review
file
Section
within
the
overall
file
for
development
of
the
work.
Once
the
peer
review
is
completed,
it
is
the
responsibility
of
the
Peer
Review
Leader
to
ensure
that
the
peer
review
record
is
filed
and
maintained
in
accordance
with
the
organization s
procedures.

The
Peer
Review
Record
is
separate
from
the
entry
in
the
Peer
Review
Product
Tracking
(
PRPT)
Database.
While
some
information
from
the
peer
review
record
appears
in
the
database,
the
peer
review
record
is
the
official
record
of
the
peer
review.

2.5.2
How
Can
the
Peer
Review
Record
Improve
the
Peer
Review
Process?

A
good
peer
review
record
allows
future
reference
to
what
happened
during
the
peer
review,
and
helps
Decision
Makers
make
appropriate
use
of
peer
reviewer
input.
In
addition,
a
good
record
helps
ensure
that
EPA s
Peer
Review
Policy
is
followed.
The
Peer
Review
Leader
is
responsible
for
ensuring
that
the
peer
review
record
for
individual
work
products
is
collected
and
maintained
until
completion
of
the
peer
review
effort.

2.5.3
What
Should
Be
in
the
Peer
Review
Record?

The
peer
review
record
should
include
all
materials
considered
by
the
individual
peer
reviewers
of
the
peer
review
panel,
as
well
as
their
written
comments
and
other
input.
Such
materials
include,
at
a
minimum
(
see
also
Section
4.3.1):

a)
The
draft
work
product
submitted
for
peer
review
Page
46
Peer
Review
Handbook
b)
Materials
and
information
(
including
the
charge
)
given
to
the
peer
reviewers
c)
Written
comments,
information,
and
materials
received
from
the
peer
reviewers
d)
Information
about
the
peer
reviewers
(
such
as
reviewers 
names,
affiliations,
and
a
statement
concerning
potential
conflicts
and
their
resolution)

e)
Logistical
information
about
conduct
of
the
peer
review
(
such
as
times
and
locations
of
meetings)

f)
A
memorandum,
or
other
written
record,
approved
by
the
Decision
Maker,
responding
to
the
peer
review
comments
specifying
acceptance
or,
where
thought
appropriate,
rebuttal
and
non­
acceptance
g)
The
final
work
product
When
deciding
if
particular
materials
should
be
included
in
the
record,
the
Peer
Review
Leader
should
consider
whether
the
materials
would
help
reconstruct
the
peer
review
process
and
results
at
a
later
time.
If
the
materials
may
be
helpful,
they
should
be
part
of
the
peer
review
record.

In
addition
to
hard
copies
of
materials,
Peer
Review
Leaders
need
to
maintain
electronic
copies
of
the
materials
(
e.
g.,
charge)
that
are
necessary
for
the
annual
reports
to
the
Peer
Review
Tracking
Database.
Peer
Review
Leaders
consult
with
their
Peer
Review
Coordinators
to
identify
those
materials.

2.5.4
What
Should
I
Do
with
a
Peer
Review
Record
That
Pertains
to
a
Rulemaking
Action?

The
Peer
Review
Leader
should
coordinate
with
their
program s
docket
office
to
see
that
proper
docketing
requirements
are
satisfied
for
a
peer
review
of
a
work
product
supporting
a
new
rule.
The
Peer
Review
Leader
is
also
responsible
for
notifying
the
workgroup
chair
as
well
as
the
Peer
Review
Coordinator
(
for
the
annual
report)
that
a
peer
review
is
completed.

2.5.5
When
Should
the
Peer
Review
Record
Building
Process
Begin?

An
early
start
at
developing
and
maintaining
a
peer
review
record
will
help
ensure
the
record
is
complete
and
helpful.
Ideally,
the
record
begins
when
the
decision
to
peer
review
a
work
product
is
made.
The
Peer
Review
Leader
needs
to
construct
the
peer
review
record
from
this
point
on
­­
this
will
avoid
potentially
time­
consuming
reconstruction
at
a
later
point.
Note
Peer
Review
Handbook
Page
47
that
the
peer
review
record
is
not
complete
until
it
contains
a
copy
of
the
final
work
product
which
addresses
the
peer
review
comments.

2.5.6
What
are
the
Differences
in
Record
Keeping
for
a
Review
by
an
Individual
Compared
to
a
Panel?

Strictly
speaking,
a
true
peer
review
requires
more
than
a
single
reviewer.
A
review
conducted
by
one
individual
will
rarely
provide
the
depth
of
commentary
required
to
improve
the
work
product.
In
addition,
you
will
not
receive
the
range
of
views
and
richness
necessary
to
ensure
improvement.

In
the
case
of
a
review
panel,
there
will
often
be
conflicting
comments
that
must
be
resolved.
This
resolution
should
be
in
the
record.

2.5.7
Where
Should
the
Peer
Review
Record
be
Kept
and
For
How
Long?

During
the
active
conduct
of
the
peer
review,
the
Peer
Review
Leaders
maintain
the
peer
review
record
with
themselves
until
the
peer
review
is
totally
completed.
Minimally,
the
file
should
be
maintained
until
one
year
after
the
completed
peer
review
is
reported
in
the
next
annual
reporting.
After
that,
the
peer
review
record
should
be
maintained
for
a
 
reasonable
period
of
time. 
Establishment
and
maintenance
of
the
archive
where
the
peer
review
records
ultimately
reside
are
an
organization s
responsibility
(
i.
e.,
not
that
of
an
individual
program
manager
or
Peer
Review
Leader).
Generally,
to
allow
flexibility,
individual
offices
and
regions
will
decide
the
appropriate
level
of
organizational
responsibility
and
how
they
will
meet
any
 
routinely
available 
requirements.
The
peer
review
record
may
be
kept
with
other
records
relating
to
the
overall
project,
as
long
as
it
is
easily
and
separately
identifiable.

There
are
also
specific
requirements
regarding
the
use
of
dockets
for
record­
keeping;
however,
these
are
not
covered
in
this
Peer
Review
Handbook.
The
documents
contained
in
the
peer
review
record
should
be
maintained
in
accordance
with
the
Agency s
record
keeping
retention
schedule
for
such
records
(
for
details,
see
EPA s
National
Records
Management
Program;
http://
www.
epa.
gov/
records/
).
One
long­
term
archiving
mechanism
may
be
the
formal
archiving
at
the
Federal
Records
Center
in
Suitland,
MD.

2.5.8
Are
Internal
Peer
Review
Comments
Included
in
the
Peer
Review
Record?

An
internal
EPA
peer
review
must
be
a
formal
process
to
be
considered
a
legitimate
peer
review.
This
process
adheres
to
the
guidance
found
in
this
Handbook
for
planning,
conducting,
and
completing
a
peer
review.
When
you
follow
this
formal
process
to
obtain
peer
review
from
EPA
peers
(
see
Section
1.4.9),
then
the
whole
record
of
that
internal
peer
review
is
included
in
the
peer
review
record.
This
includes
all
the
materials
detailed
in
Section
2.5.3
(
also
see
Section
Page
48
Peer
Review
Handbook
4.3.1).
Conducting
a
formal
internal
peer
review
is
not
the
same
thing
as
informal
input
from
your
EPA
colleagues
(
i.
e.,
 
colleagues
down
the
hall ),
nor
peer
input
from
Agency
personnel
helping
to
develop
the
work
product,
nor
organizational
review
and
clearance
processes.
Such
inputs
from
these
informal
processes
should
not
be
placed
in
the
peer
review
record.
The
peer
review
record
should
contain
only
the
information
obtained
when
you
conduct
a
formal
internal
peer
review.

In
some
cases,
an
internal
EPA
peer
review
may
be
followed
by
a
separate
external
peer
review.
In
this
case,
the
formal
record
of
the
internal
peer
review
should
be
included
in
the
peer
review
record,
however,
the
external
peer
review
will
generally
stand
as
the
peer
review
of
greater
significance
since
it
is
viewed
more
independent
in
nature,
has
broader
fields
of
available
expertise
which
can
be
brought
to
bear
on
the
issues
and
often
includes
greater
depth
for
specific
disciplines.
Peer
Review
Leaders
should
ensure
that
any
applicable
EPA
record
keeping
regulations
are
followed
(
for
details,
see
EPA s
National
Records
Management
Program;
http://
www.
epa.
gov/
records/).

2.6
Budget
Planning
2.6.1
What
Budgetary
Factors
Should
I
Consider
in
a
Peer
Review?

Resources
needed
to
implement
the
Peer
Review
Policy
need
to
be
requested
through
the
usual
Agency
budgetary
processes.
The
budget
formulation
process
within
the
Executive
Branch
is
followed,
after
appropriation
bills
are
passed
by
Congress,
by
budget
execution.
These
two
processes
provide
opportunities
to
secure
resources
for
activities
Peer
review
is
part
of
the
carried
out
by
Headquarters
and
Regional
offices,
normal
cost
of
doing
including
peer
review.
The
major
work
products
for
which
business.

decisions
for
peer
review
have
been
made
(
List
B
(
Candidate
Products
for
Future
Peer
Review)
candidates)
need
to
have
adequate
funding
for
peer
review
in
budget
requests
for
the
coming
fiscal
year.
Similarly,
adequate
funding
needs
to
appear
in
the
actual
approved
operating
budget
to
ensure
their
conduct.
For
purposes
of
budget
planning,
the
costs
of
peer
review
would
include
the
FTE
cost
of
staff,
the
contract
or
other
costs
associated
with
the
use
of
outside
peer
reviewers,
and
the
administrative
costs
of
conducting
a
review
(
copying,
travel
expenses,
etc).

2.6.2
What
Input
is
Needed
for
the
Annual
Budget
Formulation
and
Budget
Execution
Process?

Senior
Management
in
Office
and
Regions
(
including
Decision
Makers
and
budget
officers)
need
to
be
sure
that
budget
requests
include
anticipated
resources
for
peer
review.
Peer
review
needs
to
be
considered
as
a
normal
part
of
doing
business.
Peer
review
resource
Peer
Review
Handbook
Page
49
considerations
should
also
be
addressed
in
the
analytic
blueprint
for
Agency
rule­
making
actions.

2.7
Legal
Considerations
2.7.1
Are
There
Legal
Ramifications
From
the
Peer
Review
Policy?

The
Peer
Review
Policy
does
not
establish
or
affect
legal
rights
or
obligations.
Rather,
it
confirms
the
importance
of
peer
review
where
appropriate,
outlines
relevant
principles,
and
identifies
factors
Agency
staff
should
consider
in
implementing
the
policy.
Except
where
provided
otherwise
by
law,
peer
review
is
not
a
formal
part
of
or
substitute
for
notice
and
comment
rulemaking
or
adjudicative
procedures.
EPA s
decision
to
conduct
peer
review
in
any
particular
case
is
wholly
within
the
Agency s
discretion.
Similarly,
nothing
in
the
Policy
creates
a
legal
requirement
that
EPA
respond
to
peer
reviewers.
However,
to
the
extent
that
EPA
decisions
rely
on
scientific
and
technical
work
products
that
have
been
subjected
to
peer
review,
the
remarks
of
peer
reviewers
should
be
included
in
the
record
for
that
decision.

2.7.2
Is
Legal
Advice
Needed?

AA/
RA
staff
and
management
should
work
regularly
with
individual
OGC/
Regional
Counsel
(
RC)
staff
assigned
to
Agency
activities.
Peer
Review
Leaders
should
initially
consult
with
their
customary
OGC/
RC
advisors
for
legal
advice
or
referral.
Headquarters
attorneys
have
specialties
in
specific
areas
and
can
be
consulted
as
needed
(
e.
g.,
FACA
considerations
(
see
below);
contractual
responsibilities;
ethics
and
potential
conflicts
of
interest).

2.8
Federal
Advisory
Committee
Act
(
FACA)
Considerations
The
Federal
Advisory
Committee
Act,
5
U.
S.
C.
App.
2,
imposes
certain
open
meeting
(
public
announcement
in
the
Federal
Register),
balanced
membership,
and
chartering
requirements
(
with
the
approval
of
the
General
Services
Administration
(
GSA))
before
the
Agency
establishes,
controls
or
manages
an
 
advisory
committee 
for
advice
or
recommendations.
Peer
review
carried
out
by
formal
and
established
(
chartered)
Federal
advisory
committees,
such
as
the
Science
Advisory
Board
(
SAB)
or
the
FIFRA
Scientific
Advisory
Panel
(
SAP),
is
always
subject
to
FACA
requirements.
However,
FACA
does
not
apply
to
every
EPA
and
contractor­
run
peer
reviews.

In
this
section
you
will
find
information
on
the
applicability
of
FACA
to
EPA­
or
contractor­
run
peer
reviews.
Page
50
Peer
Review
Handbook
2.8.1
When
Do
FACA
Requirements
Apply
to
EPA­
Run
Peer
Reviews?

In
most
cases,
Federal
Advisory
Committee
Act
(
FACA)
requirements
apply
to
EPA­
run
peer
reviews
that
are
conducted
by
formal
and
established
(
chartered)
Federal
advisory
committees,
such
as
the
Science
Advisory
Board
(
SAB)
or
the
FIFRA
Scientific
Advisory
Panel
(
SAP).
These
groups
have
the
following
characteristics:

a)
the
group
is
established,
controlled,
or
managed
by
EPA;

b)
the
group
has
a
fixed
membership,
established
purpose,
and
a
set
agenda;
and
c)
the
group
strives
to
produce
collegial,
rather
than,
individual
advice
to
EPA
EPA
run
peer
reviews
that
are
not
originally
intended
to
be
subject
to
FACA,
but
which
exhibit
the
above
characteristics,
may
unintentionally
become
subject
to
FACA.
Questions
concerning
the
applicability
of
FACA
to
peer
review
meetings
should
be
addressed
to
the
FACA
experts
in
the
Cross­
Cutting
Issues
Law
Office
of
OGC
(
Mail
Code
2322A
at
Headquarters),
or
the
appropriate
Office
of
Regional
Counsel.

2.8.2
When
Are
EPA­
Run
Peer
Reviews
Not
Subject
to
FACA?

If
EPA
conducts
a
peer
review
with
the
purpose
of
obtaining
advice
from
the
individual
peer
reviewers
and
not
for
the
purpose
of
obtaining
a
peer
review
product
from
the
group
(
as
a
collective
or
consensus
body),
the
peer
review
would,
in
most
cases,
not
be
subject
to
FACA.
Peer
review
participants
provide
only
their
own
views
or
recommendations
and
do
not
vote
nor
do
they
provide
collective
or
consensus
recommendations
to
EPA.
When
referring
to
the
recommendations
of
the
individual
reviewers,
EPA
should
not
characterize
these
recommendations
using
terms
such
as
 
collective 
or
 
consensus. 
As
a
general
matter,
workshops
and
 
letter
reviews 
that
seek
individual
views
or
comments
are
usually
not
subject
to
the
requirements
of
FACA.

In
addition
to
ensuring
that
peer
reviewers
only
provide
comments
as
individuals,
EPA
officials
may
wish
to
lessen
the
potential
for
a
challenge
under
the
Federal
Advisory
Committee
Act
(
FACA)
by
seeking
balanced
participation
at
peer
review
meetings,
and
allowing
interested
members
of
the
public
to
attend,
and
ensuring
that
they
have
access
to
appropriate
materials.

Non­
FACA
meetings
may
be
announced
in
the
Federal
Register
(
providing
that
it
is
clear
in
the
notice
that
such
meetings
are
not
subject
to
FACA)
as
it
provides
the
public
with
useful
information
and
a
point
of
contact
concerning
the
peer
review.
In
addition,
non­
FACA
(
as
well
as
FACA)
meetings
should
also
be
advertised
via
other
avenues
(
e.
g.,
the
Web,
local
newspapers,
and
mailing
lists).
Peer
Review
Handbook
Page
51
2.8.3
How
Do
I
Ensure
that
a
Contractor­
Run
Peer
Review
Does
Not
Become
Subject
to
FACA?

Committees
(
or
other
peer
review
groups)
established,
controlled
or
managed
by
an
outside
organization
(
such
as
by
an
EPA
contractor)
to
provide
that
outside
organization
with
advice
and
recommendations
(
that
will
be
submitted
eventually
to
EPA
as
a
contractor
report)
are
not
subject
to
FACA.
Although
FACA
should
not
apply
to
contractor­
run
peer
reviews,
there
are
things
that
you
(
i.
e.,
EPA)
can
inadvertently
do
that
may
invoke
FACA.

The
following
are
considerations
that
you
should
be
aware
of
when
a
contractor
runs
a
peer
review
(
e.
g.,
letter
review,
panel,
workshop,
etc.)
for
EPA:

a)
the
outside
party s
peer
review
could
be
subject
to
FACA
if
EPA
establishes,
manages
or
controls
the
peer
review
group
(
e.
g.,
EPA
selects
the
members
of
the
panel,
runs
the
meeting,
etc.).

b)
EPA
should
not
provide
contractors
with
a
draft
agenda
or
suggested
format
for
peer
review
meetings.
EPA
contractors
should
manage
and
control
the
process,
including
running
any
meetings.

c)
At
the
request
of
the
EPA
contractor,
EPA
can
provide
a
briefing
to
the
peer
reviewers
(
e.
g.,
in
a
conference
call
with
the
contractor
on
the
line)
on
the
history
or
background
of
the
development
of
the
peer
review
document.
EPA
only
should
provide
technical
or
background
information
and
not
use
the
call
to
take
over
the
contractor s
peer
review
group.
Not
only
should
the
contractor
be
 
on
the
line 
but
it
should
be
very
clear
to
all
participants
that
the
contractor
is
in
charge
of
the
call.
The
contractor,
not
EPA,
should
invite
people
to
participate,
make
all
administrative
arrangements,
conduct
the
meeting,
and
control
the
agenda.

d)
EPA
employees
may
attend
the
peer
review
panel
meetings
or
workshops.
However,
they
may
not
take
over
the
control
of
the
meeting.
The
contractor
should
call
on
them
to
speak
when
appropriate,
but
they
should
limit
their
participation
to
providing
technical
and/
or
background
information,
and
not
attempt
to,
or
appear
to,
take
over
the
contractor s
meeting.

e)
Since
FACA
does
not
apply
when
a
contractor
establishes,
controls,
or
manages
a
peer
review,
the
contractor
need
not
avoid
terms
such
as
 
collective 
or
 
consensus 
when
reporting
agreement
among
its
peer
reviewers
(
subcontractors).
Page
52
Peer
Review
Handbook
f)
EPA
may
provide
comments
to
the
contractor
on
the
contractor s
peer
review
only
to
the
extent
that
the
Agency
is
verifying
that
the
contractor
has
completed
satisfactorily
the
report
in
accordance
with
the
work
assignment.
EPA
should
not
attempt
to
make
changes
in
the
contractor s
conclusions;
this
would
compromise
the
independence
of
the
peer
review
conducted
by
the
contractor.
Peer
Review
Handbook
Page
53
3.
CONDUCTING
A
PEER
REVIEW
3.1
Overview
Statement
The
success
and
usefulness
of
any
peer
review
depends
on
the
quality
of
the
draft
work
product
submitted
for
peer
review,
the
care
given
to
the
statement
of
the
issues
or
"
charge,"
the
match
between
the
peer
review
draft
product
and
the
form
of
peer
review,
the
match
between
the
peer
review
draft
product
and
the
scientific/
technical
expertise
of
the
reviewers,
and
Agency
use
of
peer
review
comments
in
the
final
product.
It
is
not
simply
enough
to
conduct
a
peer
review;
each
of
the
foregoing
elements
requires
serious
attention.

3.2
Charge
to
the
Peer
Reviewers
3.2.1
What
is
a
Charge?

As
part
of
each
peer
review,
the
Peer
Review
Leader
must
formulate
a
clear,
focused
charge
that
identifies
recognized
issues
and
invites
comments
or
assistance.
This
request
signals
the
Agency's
awareness
of
potential
issues
and
its
receptivity
to
expert
recommendations.
The
charge
to
peer
reviewers
usually
makes
two
general
requests.
First,
it
focuses
the
review
by
presenting
specific
questions
and
concerns
that
the
Agency
expects
the
reviewers
to
address.
Secondly,
it
invites
general
comments
on
the
entire
work
The
time
spent
preparing
a
good
product.
The
specific
and
general
comments
should
charge
is
well
spent,
and
is
crucial
focus
mostly
on
the
scientific
and
technical
merits
of
for
an
effective
peer
review.
the
work
product
and,
where
germane,
whether
the
scientific/
technical
studies
have
been
applied
in
a
sound
manner.
Remember,
the
peer
review
is
not
for
the
decision
or
action
itself,
but
for
the
underlying
scientific
and/
or
technical
work
product.
Focused
questions
greatly
simplify
the
task
of
collating,
analyzing
and
synthesizing
peer
review
comments
on
a
topical
basis.
The
questions
should
be
specific
enough
to
get
helpful
comments,
but
not
so
specific
(
unless
very
specific
points
are
needed
to
be
addressed)
that
they
preclude
creative
responses.
Moreover,
the
written
responses
to
these
questions
by
peer
reviewers
help
the
Agency
create
a
peer
review
record.
As
a
general
rule,
the
time
drafting
a
good
charge
letter
is
well­
spent
and
is
necessary
for
an
effective
peer
review.

3.2.2
What
are
the
Essential
Elements
of
a
Charge?

a)
Brief
overview
or
introduction
(
describe
what
the
work
product
is,
how
it
was
developed,
how
it
will
be
used)
Page
54
Peer
Review
Handbook
b)
As
needed,
a
brief
description
or
listing
of
any
background
materials
provided
to
the
peer
reviewers
c)
The
issues
or
questions
to
be
addressed
by
the
peer
reviewer(
s)

1)
The
due
date
of
reviewers 
comments
2)
The
format
of
reviewer
responses
3)
The
point
of
contact
in
case
peer
reviewers
have
questions
3.2.3
Where
Can
I
Get
an
Example
of
a
Charge?

Appendix
C
(
Examples
of
Charges)
contains
examples
of
successful
charges
that
cover
a
variety
of
issues.
Appendix
D
(
Guidance
on
Requesting
a
Review
by
the
Science
Advisory
Board)
provides
guidance
for
obtaining
Science
Advisory
Board
(
SAB)
services.

3.2.4
Can
a
Stakeholder
Provide
Input
to
the
Charge
to
the
Peer
Reviewers?

Yes.
EPA
may
decide
to
obtain
stakeholder
input
on
the
charge
to
the
peer
reviewers,
but
EPA
must
make
the
final
determination
on
what
elements
to
include
in
the
charge
to
ensure
that
it
meets
the
needs
of
the
EPA
peer
review.

a)
If
you
obtain
stakeholder
input,
you
must
include
any
and
all
interested
parties
to
the
extent
feasible
based
upon
statutory,
regulatory,
budgetary
and/
or
time
constraints.
Do
not
limit
input
to
one
stakeholder
only
(
e.
g.,
a
responsible
party
or
environmental
group).

b)
If
EPA
has
hired
a
contractor
to
perform
the
peer
review,
it
should
still
be
EPA
personnel
who
obtain
stakeholder
input
and
provide
the
list
of
charge
questions
to
the
contractor.

c)
If
you
form
a
committee
of
stakeholders
to
help
develop
the
charge,
be
aware
that
your
committee
may
become
subject
to
the
requirements
of
FACA
(
see
Section
2.8).

3.2.5
Who
Writes
the
Charge
When
I
Hire
a
Contractor
to
Conduct
the
Peer
Review?

In
general,
if
EPA
hires
a
contractor
to
perform
the
peer
review,
EPA
must
allow
the
contractor
independence
in
conducting
the
review.
However,
with
regard
to
the
charge,
EPA
can
Peer
Review
Handbook
Page
55
and
should
provide
the
contractor
with
the
substantive
list
of
questions
that
EPA
wants
included
in
the
charge
letter
to
ensure
that
the
peer
review
meets
EPA s
needs.
The
list
of
charge
questions
can
be
incorporated
into
the
Statement
of
Work
if
the
EPA
project
manager
has
the
list
ready
at
that
time.
Based
on
this
list,
the
contractor
would
then
prepare
and
submit
the
actual
charge
letter
to
the
peer
reviewers.
Prior
to
submitting
the
charge
letter
to
the
peer
reviewers,
the
contractor
should
give
EPA
an
opportunity
to
review
the
charge
letter
to
ensure
that
it
meets
EPA s
needs.
EPA
cannot
submit
the
charge
directly
to
the
peer
reviewers.

3.2.6
Is
it
Okay
For
Me
to
Ask
a
Contractor
to
Develop
the
Charge
to
the
Peer
Reviewers?

No.
EPA
will
provide
the
charge
questions
to
the
contractor
as
discussed
above.
The
contractor,
though,
may
provide
assistance
and
advice
in
the
development
of
the
charge.

3.3
Time
Line
3.3.1
What
are
the
Factors
in
Scheduling
a
Peer
Review?

The
peer
review
schedule
is
a
critical
feature
of
the
process.
The
schedule
must
take
into
account
the
availability
of
a
quality
draft
work
product,
availability
of
appropriate
experts,
time
available
for
using
peer
review
comments,
deadlines
for
the
final
work
product,
and
logistical
aspects
of
the
peer
review
(
e.
g.,
contracting
procedures).

The
schedule
for
peer
review
should
take
into
account
the
overall
rulemaking
(
or
other
decision
making)
schedule.
For
rules,
in
particular
those
in
Tier
1
and
Tier
2,
the
scheduling
of
the
peer
review
should
be
included
in
the
development
of
the
analytic
blueprint.
Peer
review
sometimes
leads
to
new
information
and
analyses,
or
recommendations
for
new
research
that
would
alter
the
work
product
and
thus
modify
the
scientific/
technical
basis
for
the
action.
For
this
reason,
it
is
usually
advisable
to
complete
the
peer
review
before
taking
public
comment,
or
at
least
before
the
close
of
the
public
comment
period.

3.4
Selection
of
Peer
Reviewers
3.4.1
What
are
Considerations
for
Selecting
Peer
Reviewers?

Selection
of
independent
peer
reviewers
is
not
a
trivial
task,
and
it
is
crucial
to
an
effective
peer
review.
It
is
important
that
peer
reviewers
be
selected
for
independence
and
scientific/
technical
expertise.
Therefore,
EPA
should
always
make
every
effort
to
use
peer
reviewers
who
do
not
have
any
real
or
perceived
bias
or
conflict
of
interest
and
who
are
completely
independent.
However,
the
very
need
to
have
experienced
individuals
on
a
peer
review,
along
with
the
desire
to
have
appropriate
technical
balance
and
representation,
can
mean
Page
56
Peer
Review
Handbook
that
the
selection
of
potential
peer
reviewers
often
comes
from
those
who
are
considered
as
having
a
potential
bias.
To
reduce
the
concern
that
a
potential
panel
may
have
unnecessary
bias,
it
may
be
useful
to
obtain
an
informal
review
of
the
expertise
and
balance
of
potential
peer
reviewers
from
others
in
your
organization,
from
OGC
or
even
from
outside
groups.
Sometimes
selecting
individuals
who
have
served
in
a
variety
of
organizations
rather
than
a
single
one
for
an
extended
period,
provides
expertise
with
diverse
perspective.
The
emphasis
on
independence
and
expertise
applies
equally
to
government
experts
and
experts
from
the
larger
scientific
community.

Some
peer
reviews
can
be
conducted
with
two
or
three
reviewers;
others
involve
panels
of
peer
reviewers.
In
either
case,
each
peer
reviewer
should
have
recognized
technical
expertise
that
bears
on
the
subject
matter
under
discussion.
The
peer
reviewers
of
a
work
product
should
represent
a
balanced
range
of
technically
legitimate
points
of
view.
In
addition,
cultural
diversity
and
"
address"
(
e.
g.,
industrial,
academic,
or
environmental
community)
are
other
factors
that
can
play
a
role
in
selecting
peer
reviewers.

3.4.2
Where
Do
I
Find
Peer
Reviewers?

Recommendations
for
potential
peer
reviewers
can
be
identified
from
a
number
of
organizations.
These
include
external
groups
such
as
the
affected
party(
ies),
special
interest
groups,
public
interest
groups,
environmental
groups,
professional
societies,
trade
or
business
associations,
state
organizations
or
agencies,
Native
American
Tribes,
colleges
and
universities,
the
National
Research
Council,
and
other
Federal
agencies
with
an
involvement
in
or
familiarity
with
the
issue.
Agency
associated
groups
include
the
staff
of
the
Science
Advisory
Board
(
SAB)
or
the
Scientific
Advisory
Panel
(
SAP),
and
relevant
scientific
and
technical
experts
from
Program
or
Regional
offices.

In
certain
circumstances,
existing
peer
review
organizations
such
as
the
SAB
or
SAP
may
be
used
to
conduct
a
peer
review.
These
groups
establish
their
own
criteria
for
accepting
work
and
coordination
must
be
made
directly
with
them
(
see
Appendix
D
­
Guidance
on
Requesting
a
Review
by
the
Science
Advisory
Board)
for
SAB
procedures.
Both
SAB
and
SAP
conduct
formal,
public,
external
peer
reviews.

Occasionally,
a
member
of
the
scientific
community
will
offer
his/
her
services
for
peer
review
during
an
ongoing
peer
review.
These
offers
may
be
at
no
cost
or
based
on
an
expectation
that
reimbursement
will
be
made.
Disposition
of
these
unsolicited
offers
will
be
handled
on
a
case
by
case
basis
by
the
Peer
Review
Leader,
and
as
necessary,
in
consultation
with
the
Peer
Review
Coordinator,
the
Office
of
General
Counsel
(
OGC),
and
appropriate
Decision
Makers.
Peer
Review
Handbook
Page
57
If
you
use
a
contract
mechanism
to
conduct
a
peer
review,
the
contractor
will
have
its
own
pool
of
scientific
and
technical
experts
for
peer
review.
With
contractors,
EPA
can
provide
information
on
potential
sources
of
peer
reviewers
for
conducting
a
peer
review
if
such
a
listing
is
prepared
in
alphabetical
order.
In
no
way
should
EPA
insist
that
the
contractor
select
from
EPA s
list,
or
require
that
the
contractor
receive
EPA
approval
before
selecting
any
peer
reviewer
(
sometimes
known
as
a
 
subcontractor ).
However,
EPA
should
review
the
list
of
reviewers
for
conformance
to
work
assignment
specifications
and
adherence
to
conflict
of
interest
concerns.
Furthermore,
when
utilizing
a
contract
mechanism
to
conduct
peer
review,
EPA
is
not
permitted
to
direct
the
prime
contractor
to
a
specific
subcontractor
(
or
peer
reviewer)
nor
is
EPA
permitted
to
direct
the
peer
reviewer
(
subcontractor).
All
interactions
with
the
peer
reviewers
must
be
coordinated
through
the
prime
contractor.
(
See
Section
3.6
for
further
information)

Keep
in
mind
that
for
contracting
purposes,
contractors
are
required
to
obtain
Contracting
Officer
(
CO)
approval
of
subcontractors
and
the
CO
generally
seeks
the
input
of
the
work
assignment
manager
(
WAM
­­
the
WAM
may
be
the
Peer
Review
Leader
in
many
cases)
before
approving
the
use
of
subcontractors.
In
this
case,
as
noted
above,
the
Peer
Review
Leader
should
review
the
list
of
potential
reviewers
for
conformance
to
the
work
assignment
specifications
and
adherence
to
conflict
of
interest
concerns.

3.4.3
Are
External
or
Internal
Peer
Reviewers
Preferred?

External
peer
reviewers
are
generally
preferred,
particularly
for
most
final
work
products.
For
some
work
products,
like
those
reviewed
at
interim
steps,
either
external
or
internal
peer
review
may
be
appropriate.
Selection
of
internal
peer
reviewers
should
be
based
upon
technical
expertise,
available
time
and
"
address"
­­
that
is,
they
should
not
come
from
the
immediate
office
or
group
producing
the
product
or
have
any
other
connection
with
the
product
or
document
being
peer
reviewed.
External
peer
reviewers
should
be
selected
based
upon
technical
expertise
as
well,
however,
care
must
be
taken
not
to
use
individuals
who
have
been
involved
in
the
development
of
the
work
product.
(
See
Section
1.2.6;
see
also
Sections
1.4.6
to
1.4.9).

3.4.4
What
is
Important
in
the
Mix
of
a
Peer
Review
Panel?

A
peer
review
panel
or
group
can
number
from
just
a
few
individuals
to
ten
or
more,
depending
on
the
issue,
the
time
and
resources
available,
and
the
broad
spectrum
of
expertise
required
to
treat
the
range
of
issues/
questions
in
the
charge.
Objective
technical
expertise
and
lack
of
a
conflict
of
interest
are
critical
in
selecting
peer
reviewers.
Naturally,
experts
whose
understanding
of
the
specific
technical
area(
s)
being
evaluated
are
necessary;
nevertheless,
it
is
also
important
to
include
a
broad
enough
spectrum
of
other
related
experts
to
completely
evaluate
the
relevant
impacts
on
other
less
obvious
concerns
(
i.
e.,
to
comment
not
only
if
the
job
is
being
done
right,
but
also
whether
the
right
job
is
being
done).
For
example,
for
health
related
Page
58
Peer
Review
Handbook
peer
reviews,
experts
in
such
fields
as
ecology
and
economics
may
provide
very
useful
insights.
Although
persons
who
are
familiar
and
have
a
substantial
reputation
in
the
field
are
often
called
upon
repeatedly
to
be
reviewers,
it
is
important
to
keep
a
balance
with
new
people
who
bring
fresh
perspectives
to
the
review
of
a
work
product.

There
is
usually
a
continuum
of
views
on
any
issue.
To
the
extent
possible
or
practicable,
selected
experts
should
have
technically
legitimate
points
of
views
that
fall
along
the
continuum.
A
review
panel
should
include
experts
that
are
considered
 
mainstream 
(
nearer
the
center
of
the
continuum)
as
well
as
those
further
along
the
continuum
(
while
generally
avoiding
the
extremes).
This
will
help
maintain
a
balanced
review
panel,
while
allowing
all
views
to
be
expressed
and
discussed.
A
balanced
panel
will
allow
consensus
building
(
if
consensus
is
the
object
of
a
particular
peer
review;
if
not,
it
allows
a
spectrum
of
(
re)
views
for
the
Agency
to
evaluate).
As
a
general
rule,
experts
who
have
made
public
pronouncements
on
an
issue
(
e.
g.,
those
who
have
clearly
"
taken
sides")
may
have
difficulty
in
being
objective
and
should
be
avoided.

3.4.5
What
is
a
Conflict
of
Interest?

The
matter
of
obtaining
a
fair
and
credible
peer
review,
as
well
as
maintaining
the
credibility
of
the
Agency
and
the
Agency s
scientific
products,
is
of
paramount
importance.
Peer
review
leaders
are
strongly
encouraged
to
obtain
peer
reviewers
who
do
not
have
a
legal
or
perceived
conflict
of
interest
(
i.
e.,
creates
the
appearance
that
the
peer
reviewer
lacks
impartiality
or
objectivity).
In
reality,
we
recognize
that
a
totally
independent
peer
reviewer
is
rare
and
very
difficult
to
identify
(
see
Section
3.4.6
for
steps
to
take
to
ensure
a
credible
peer
review).

Conflict
of
interest
is
a
situation
in
which,
because
of
other
activities
or
relationships
with
other
persons,
an
individual
is
unable
or
potentially
unable
to
render
impartial
assistance
or
advice
to
the
Agency,
or
the
person s
objectivity
in
performing
the
work
is
or
might
be
otherwise
impaired,
or
a
person
has
an
unfair
competitive
advantage.
Generally,
a
conflict
of
interest
of
interest
arises
when
the
person
is
affected
by
his/
her
private
interests,
when
he/
she
or
his/
her
associates
would
derive
benefit
from
incorporation
of
their
point
of
view
in
an
Agency
product,
or
when
their
professional
standing
and
status
or
the
significance
of
their
principal
area
of
work
might
be
affected
by
the
outcome
of
the
peer
review.
Clearly,
peer
reviewers
should
not
be
placed
in
the
position
of
reviewing
their
own
research
and
analyses
that
form
the
basis
of
the
work
product
under
review
as
this
might
impair
their
objectivity.
Whenever
there
are
questions
about
conflicts
of
interest,
you
should
contact
the
appropriate
official
in
OGC
for
clarification.

Be
aware
that
COI
can
be
used
in
a
generic
fashion
in
speaking
of
common,
everyday
experience
that
also
involves
elements
of
ethics.
In
most
cases,
that
is
how
it
is
treated
in
the
Peer
Review
Handbook.
However,
COI
is
a
very
specific
issue
when
used
in
a
governmental
Peer
Review
Handbook
Page
59
contracting
sense.
In
the
contracting
sense,
the
Federal
Acquisition
Regulations
(
FAR),
the
Environmental
Protection
Agency
Acquisition
Regulations
(
EPAAR),
and
other
internal
Agency
documents
define
COI
and
describe
the
Agency s
COI
policies
and
procedures.
The
Handbook
provides
guidance
in
some
of
these
aspects
related
to
peer
review
(
see
Section
3.6
and
Appendix
E
­
Example
Statements
of
Work
for
Contracts),
however,
specific
questions
should
be
addressed
to
the
Office
of
Acquisition
Management.

3.4.6
What
Techniques
Help
Ensure
Disclosure
and
Appropriate
Resolution
of
Conflicts
of
Interest?

Before
finalizing
the
selection
of
reviewers,
the
Peer
Review
Leader
should
ascertain
whether
each
potential
peer
reviewer s
involvement
in
certain
activities
could
pose
a
conflict
of
interest
(
COI)
or
create
the
appearance
that
the
peer
reviewer
lacks
impartiality.
One
way
of
identifying
conflicts
is
to
ask
potential
reviewers
about
current
and
prior
work,
and
prior
clients
that
might
create
conflicts
or
the
appearance
of
a
lack
of
impartiality
in
carrying
out
peer
review
activities.
This
information
obtained
by
the
Peer
Review
Leader
becomes
part
of
the
peer
review
record.
When
the
peer
review
process
is
being
conducted
by
a
contractor,
the
requirement
for
addressing
peer
reviewers 
possible
conflicts
of
interest
should
be
highlighted
in
the
Statement
of
Work
of
the
work
ordering
instrument
(
e.
g.,
Work
Assignment,
Delivery
Order,
Task
Order,
etc.)
and
is
a
matter
that
is
bound
by
contractual
clauses
with
the
Contracting
Officer
as
the
final
Decision
Maker
in
contracting
matters.

Care
must
be
taken
to
reduce
actual
or
potential
organizational
or
personal
conflicts
of
interest
between
the
reviewers
and
the
work
product
under
review.
Remember,
each
potential
conflict
situation
is
unique
and
must
be
treated
on
a
case­
by­
case
basis.
The
following
are
considerations
that
should
be
addressed
in
evaluating
COI:

a)
attention
to
the
employment,
financial,
and
professional
affiliations
of
the
participants;

b)
exploring
directly
the
issue
with
each
of
the
participants
before
the
review
process
takes
place;

c)
disclosing
publicly
at
the
beginning
of
meetings
any
previous
involvement
with
the
issue;

d)
in
the
cases
of
regular
government
employees
and
Special
Government
Employees
(
SGEs),
filing
annual
(
or
updated,
as
appropriate)
Confidential
Financial
Disclosure
Forms
(
OGE
Form
450),
discussing
any
potential
conflicts
of
interest
with
their
designated
ethics
official
(
DEO),
and
advising
the
Peer
Review
Leader
of
any
relevant
concerns;
Page
60
Peer
Review
Handbook
e)
in
the
case
of
non­
Federal
peer
reviewers,
provide
them
a
copy
of
the
peer
review
COI
inquiry
(
see
Appendix
F
­
Useful
Forms).
This
form
is
sent
to
each
prospective
peer
reviewer
by
the
Peer
Review
Leader
(
or
contractor,
in
the
case
of
contractor­
run
reviews)
to
advise
them
of
the
need
to
address
COI
issues
prior
to
the
actual
review
taking
place.
A
follow­
up
contact
with
the
Peer
Review
Leader
(
or
contractor,
in
the
case
of
contractor­
run
reviews)
is
then
made
to
discuss
any
relevant
issues.
The
Peer
Review
Leader
then
documents
this
effort
in
the
peer
review
record;
this
includes
a
summary
provided
by
the
contractor
documenting
their
inquiries
and
efforts.

The
Peer
Review
Leader
needs
to
ensure
that
the
peer
review
COI
inquiry
(
see
Appendix
F
­
Useful
Forms)
took
place
and
this
appears
in
the
peer
review
record.
You
should
ensure
that
any
documentation
in
the
peer
review
record
does
not
contain
information
subject
to
privacy
laws
(
for
example,
see
The
Privacy
Act
of
1974
,
5
U.
S.
C.
§
552a
­
Records
maintained
on
individuals;
a
copy
can
be
found
at:
http://
www.
usdoj.
gov/
04foia/
privstat.
htm).
Peer
Review
Leaders
should
ensure
that
any
applicable
EPA
record
keeping
regulations
are
followed
(
for
details,
see
EPA s
National
Records
Management
Program;
http://
www.
epa.
gov/
records/).
For
information
on
Freedom
of
Information
Act
(
FOIA)
requests,
please
see
EPA s
Ethics
Advisory
88­
12
(
http://
intranet.
epa.
gov/
ogcrmo01/
ethics/
88­
12.
htm).
If
you
have
any
questions,
be
sure
to
ask
your
appropriate
Office
of
General
Counsel
and/
or
office
contract
official(
s).

Established
peer
review
groups
such
as
the
Science
Advisory
Board
provide
useful
models
for
addressing
balance
and
COI
issues.
Assistance
in
identifying
conflicts
of
interest
and
in
providing
an
appropriate
response
can
be
obtained
from
the
Office
of
the
General
Counsel.
Additional
advice
can
be
obtained
from
one
of
the
Agency s
designated
ethics
officials
(
DEOs).
Assistance
in
evaluating
conflicts
of
interest
in
a
contractual
situation
require
the
involvement
of
the
Contracting
Officer
and
the
resources
available
within
the
Office
of
Acquisition
Management.

Of
course,
conflicts
do
not
necessarily
arise
merely
because
a
peer
reviewer
knows
something
about
the
subject
matter.
In
fact,
experts
with
a
stake
in
the
outcome
­­
and
therefore
a
potential
conflict
­­
may
be
some
of
the
most
knowledgeable
and
up­
to­
date
experts
because
they
have
concrete
reasons
to
maintain
their
expertise.
Such
experts
could
be
used
provided
the
potential
conflicts
of
interest
are
disclosed
and
the
peer
review
panel
or
group
being
used
as
a
whole
is
balanced.
In
some
cases,
however,
the
conflict
may
be
so
direct
and
substantial
as
to
rule
out
a
particular
expert,
for
instance,
a
potential
peer
reviewer
who
may
have
a
client
or
employer
with
a
direct
financial
stake
in
the
particular
specific
party
matter
under
review
(
e.
g.,
i)
a
Federal
grant
or
contract
to
the
potential
peer
reviewer
or
his/
her
employer
that
relates
to
the
matter
under
review;
ii)
the
potential
peer
reviewer s
or
their
company s
work
on
a
specific
chemical
under
review).
However,
review
of
a
general
methodology
that
applies
to
numerous
Peer
Review
Handbook
Page
61
chemicals
would
not
necessarily
raise
such
a
concern.
(
Note
:
COI
language
should
be
made
part
of
contracts/
statements
of
work
(
SOW)
or
purchase
orders
(
PO)
­­
see
Section
3.6.)

A
Peer
Review
Leader
may
also
want
to
adopt
measures
that
will
prevent
creation
of
conflicts
as
the
peer
review
is
underway.
Any
measures
contemplated
that
involve
a
contractual
action
must
be
coordinated
with
the
cognizant
Contracting
Officer.
Some
measures
might
include
clauses
in
a
contract
or
purchase
order
that
require
reviewers
to
receive
advance
approval
on
future
work,
or
place
limits
on
such
work,
while
they
are
performing
the
current
peer
review.
Note
that
at
some
level
these
types
of
measures
will
discourage
experts
from
serving
as
peer
reviewers.
(
See
Section
3.6.5
for
further
information
dealing
with
contracts
and
suggestions
for
appropriate
management
controls.)

3.4.7
Can
Parties
External
to
EPA
Pay
for
Their
Own
Peer
Reviews?

There
may
be
instances
where
parties
external
to
EPA
will
want
to
conduct
and/
or
pay
for
a
peer
review
on
a
particular
work
product
(
presumably
their
own
work
product
or
one
they
are
closely
interested
in,
or
they
wouldn t
be
interested
in
expending
resources).
This
may
look
benign
at
first
blush,
but
is
a
very
complex
and
sensitive
situation
that
can
raise
significant
concerns
for
perceived
and/
or
actual
conflicts
of
interest
for
interested
parties
 
paying 
for
a
peer
review
of
their
own
work
product.
While
the
Agency
cannot
prevent
external
parties
from
conducting
and
paying
for
a
peer
review,
it
is
desirable
that
any
such
peer
review
meets
the
intent
of
the
Agency s
Peer
Review
Policy
and
adheres
to
the
principles
and
guidance
in
this
Handbook.
If
the
external
party
submits
their
work
product
and
accompanying
peer
review,
the
materials
will
be
treated
by
the
Agency
as
anything
else
submitted
for
the
Agency s
evaluation.

We
will
evaluate
the
work
product
and
the
peer
review
for
scientific
credibility
and
validity
before
making
any
decisions
based
on
the
materials.

[
This
issue
is
under
active
Agency
consideration
at
this
time;
therefore,
users
of
this
Handbook
should
be
aware
that
the
language
in
Section
3.4.7
will
likely
change
in
the
future.]

3.4.8
Are
There
Constraints
to
Selecting
Peer
Reviewers?

Sometimes
the
need
for
a
peer
review
is
accelerated
due
to
a
court­
ordered
deadline
or
other
time­
sensitive
requirements.
In
such
cases,
it
is
difficult,
if
not
impossible
to
obtain
external
peer
reviewers
in
time
to
conduct
a
full
external
peer
review.
It
may
even
be
impossible
to
conduct
a
small
scale
internal
peer
review
using
just
a
few
individuals.
Mechanisms
for
identifying
and
using
a
small
number
of
peer
reviewers
should
be
developed
so
that
quick,
effective
peer
review
can
be
included
for
even
the
most
rapidly
moving
products.
Page
62
Peer
Review
Handbook
Another
possible
constraint
involves
confidential
business
information
(
CBI).
To
evaluate
certain
Agency­
generated
studies
properly,
some
peer
reviewers
may
need
access
to
CBI.
However,
unless
the
reviewers
are
Federal
employees
with
CBI
clearance,
the
Agency
doesn t
have
the
independent
authority
to
disclose
CBI
to
them.
Therefore,
whenever
contemplating
the
use
of
outside
peer
reviewers,
Agency
staff
should
determine
whether
the
reviewers
will
need
access
to
CBI.
If
they
don t
have
CBI
clearance,
the
Office
of
the
General
Counsel
should
be
consulted
on
whether
it
is
practical
to
obtain
the
consent
of
CBI
submitters
to
disclose
the
information
to
peer
reviewers.

Offices
need
to
be
aware
of
the
requirements
of
the
Federal
Advisory
Committee
Act
(
FACA)
when
establishing
peer
review
mechanisms
(
see
Section
2.8).
Federal
advisory
committees
that
are
subject
to
chartering
by
the
General
Services
Administration
must
hold
meetings
that
are
open
to
the
public,
and
have
balanced
membership
requirements.
The
Office
of
the
General
Counsel
should
be
consulted
regarding
the
applicability
of
FACA
to
peer
review
panels.

3.4.9
If
State
Employees
are
Used
as
Peer
Reviewers,
Can
EPA
Pay
Them
for
this
Service?

Maybe.
First,
the
Peer
Review
Leader
needs
to
determine:
a)
if
the
State
agency
has
a
policy
on
whether
their
employees
can
perform
this
type
of
work
for
EPA,
and
b)
if
the
person
will
be
acting
as
an
individual
or
as
an
employee
of
the
State.
If
the
State
allows
the
person
to
be
a
peer
reviewer
for
EPA
and
the
person
is
acting
as
an
individual,
we
can
pay
them
for
their
peer
review
services.
If
the
State
allows
the
person
to
be
a
peer
reviewer
and
the
person
will
be
acting
as
a
State
employee,
we
can
only
pay
them
if
State
policy
allows
us
to.
The
most
efficient
vehicle
for
paying
the
State
employee
will
likely
be
a
contract
or
purchase
order
(
see
Sections
3.6.6
&
3.6.7),
but
you
should
consult
with
OGC
and
the
procurement
office
for
advice.
In
most
instances,
EPA
can
also
pay
travel
expenses
(
consult
with
your
administrative
staff
for
details).

If
we
don t
pay
a
State
person
for
their
peer
review
services,
they
must
sign
an
agreement
stating
that
they
don t
expect
payment
(
See
Section
3.6.1
­
gratuitous
services).

3.5
Materials
for
Peer
Reviewers
3.5.1
What
Instructions
Do
You
Give
Peer
Reviewers?

The
Peer
Review
Leader
is
responsible
for
ensuring
that
peer
reviewers
understand
their
responsibilities
(
see
Section
3.6
if
a
contract
is
involved):
Peer
Review
Handbook
Page
63
a)
Advise
the
Agency
of
actual
or
potential
organizational
or
personal
conflicts
of
interest
or
other
matters
that
would
create
the
appearance
of
a
lack
of
impartiality
(
e.
g.,
see
Section
3.4.6)

b)
Provide
written
comments
in
specified
format
by
the
specified
deadline
that
are
responsive
to
the
charge
c)
Comply
with
any
requests
for
not
disclosing
draft
work
products
to
the
public
3.5.2
What
Materials
Should
be
Sent
to
Peer
Reviewers?

For
a
peer
review
to
be
successful,
peer
reviewers
should
receive
several
documents
at
the
beginning
of
the
process.
Typically,
the
most
important
among
these
documents
are
the
charge
letter
and
the
current
work
product.
The
charge
letter
describes
what
the
peer
reviewers
are
being
asked
to
do,
and
should
serve
to
focus
and
structure
the
review.
The
work
product
is,
of
course,
the
material
being
subject
to
peer
review.

Remember,
no
documents
should
be
provided
directly
to
a
potential
peer
reviewer
when
that
reviewer
is
going
to
be
working
under
a
contract
or
purchase
order.
In
the
case
of
a
contract,
the
Agency
provides
the
work
product
with
associated
background
material
to
be
peer
reviewed
to
the
prime
contractor
who
in
turn
distributes
these
documents
to
the
peer
reviewers.
In
the
case
of
a
purchase
order,
the
 
charge
or
statement
of
work 
must
be
part
of
the
PO
(
purchase
order)
and
the
provision
of
any
documents
needs
to
be
coordinated
with
the
purchasing
agent
handling
the
order.

a)
Essential
documentation
for
each
peer
reviewer
includes:

1)
A
current
copy
of
the
work
product
to
be
peer
reviewed
with
associated
background
material.
The
work
product
needs
to
be
of
the
best
possible
scientific/
technical
quality
to
ensure
an
adequate
and
useful
peer
review.

2)
A
clear
charge
or
statement
of
work
seeking
informed
comment
on
identified
issues
to
properly
focus
the
efforts
of
the
peer
reviewers
and
ensure
that
their
individual
efforts
can
be
compared
or
contrasted.

3)
Some
information
concerning
the
process
that
you
use
for
the
peer
review,
including
the
due
date
of
reviewer
comments,
the
format
of
those
responses,
and
a
point
of
contact
in
case
the
peer
reviewer
has
questions.
Responses
should
be
written
and
submitted
to
the
Peer
Review
Leader
by
an
agreed
upon
deadline.
In
certain
rare
cases,
oral
commentary
may
be
Page
64
Peer
Review
Handbook
sufficient.
However,
in
such
cases,
a
follow­
up
written
response
for
the
record
is
required.

4)
In
some
cases,
Agency
materials
being
peer
reviewed
will
be
available
to
the
public,
even
if
they
are
marked
as
drafts.
For
example,
all
materials
reviewed
by
the
SAB
are
available.
Agency
managers
may
also
decide
that
a
broad
accessibility
has
benefits
for
the
Agency.
In
other
cases,
confidentiality
needs
to
be
maintained.
In
these
cases,
each
peer
reviewer
must
be
informed
of
the
need
for
confidentiality
with
regard
to
the
release
of
Agency
products
that
are
stamped
as
"
DRAFT"
or
"
DRAFT
­
Do
Not
Cite,
Quote,
or
Release."
Premature
release
of
draft
Agency
products,
views,
or
positions
is
inappropriate
and
can
be
damaging
to
the
credibility
of
the
Agency
or
the
peer
reviewer.
While
not
necessarily
having
legal
consequences,
such
language
will
be
included
in
the
charge
to
the
peer
reviewers.
Other
mechanisms
to
use
in
discouraging
premature
release
include
a
disclaimer
that
appears
in
a
separate
section
at
the
front
of
the
document
and
creating
the
document
with
watermarks
clearly
delineating
DRAFT
status
(
or
a
header
or
footer
that
states
DRAFT
status)
on
every
page.
In
addition,
in
any
solicitation
for
peer
reviewers,
the
necessity
for
confidentiality
and
the
non­
release
of
materials
shall
be
emphasized.

b)
Useful,
but
not
critical
materials
that
may
be
sent
to
peer
reviewers
include:

1)
The
name,
address,
and
phone
and
fax
numbers,
and/
or
Internet
address
of
each
peer
reviewer
working
on
the
specific
review
2)
A
bibliography
and/
or
any
particularly
relevant
scientific
articles
from
the
literature
3)
A
work
product
that
has
line
numbering
added
in
the
margin
for
ease
in
providing
and
referencing
comments
c)
Peer
Reviewers
should
be
given
what
is
needed
to
complete
their
task
­­
they
should
not
be
overburdened
with
excess
material.

3.5.3
How
Closely
can
EPA
Interact
with
Peer
Reviewers
During
the
Review?

a)
When
EPA
Conducts
the
Peer
Review
­
The
Peer
Review
Leader
normally
has
administrative
contacts
with
the
reviewers
during
the
development
and
conduct
of
the
peer
review.
In
some
cases
(
e.
g.,
SAB
peer
review),
peer
reviewers
may
also
receive
a
briefing
on
the
product
to
be
peer
reviewed.
Otherwise,
the
Peer
Peer
Review
Handbook
Page
65
Review
Leader
and
other
EPA
staff
should
not
contact
the
reviewers
during
the
course
of
the
review.
Such
contact
can
lead
to
perceived
conflicts
or
inappropriate
direction
that
could
compromise
the
independence
of
the
review.

b)
When
the
Contractor
Conducts
the
Peer
Review
­
If
peer
review
is
conducted
under
a
contract
mechanism,
EPA
must
limit
direct
contact
to
the
prime
contractor s
designated
representative
and
not
have
general
contact
and
direction
to
the
contractor s
staff
or
peer
reviewers
(
sub­
contractors).
Note,
when
a
peer
review
is
conducted
under
a
contract,
there
are
constraints
where
EPA
staff
are
prohibited
from
contacting
peer
reviewers
to
avoid
personal
services
arrangements.
Personal
services
contracts
exist
when
the
nature
of
the
relationship
between
the
contractor
and
the
EPA
can
be
characterized
as
an
employer
­
employee
relationship.
Any
communications
with
peer
reviewers
must
be
coordinated
through
the
prime
contractor.

3.6
Peer
Review
Services
A
range
of
peer
review
services
are
available
to
the
Agency
including
internal,
external
(
gratuitous
services,
contracts,
purchase
order),
and
Special
Government
Employee
(
SGE)
mechanisms.
The
mechanism
selected
is
generally
based
on
the
nature
of
the
scientific
or
technical
work
product.

3.6.1
What
are
Gratuitous
Services
for
Peer
Review?

The
provision
of
peer
review
products
or
services
to
EPA
without
compensation
are
provided
as
so­
called
 
gratuitous 
services.
If
a
person
wishes
to
perform
peer
review
services
for
EPA
without
compensation,
the
EPA
must
ask
them
to
sign
an
agreement
whereby
the
person
agrees
to
provide
the
prescribed
peer
review
services
as
gratuitous
services,
with
no
expectation
of
receiving
compensation
for
these
services
from
EPA
(
see
Appendix
F
for
EPA
Form
3100­
14
which
is
used
in
situations
where
a
gratuitous
services
contract
is
not
used).
An
agreement
(
such
as
Form
3100­
14)
must
be
executed
because
the
Antideficiency
Act
(
31
U.
S.
C.
§
1342)
prohibits
the
Agency s
acceptance
of
 
voluntary 
services
( 
voluntary
services 
are
services
provided
to
EPA
without
an
agreement
in
advance
that
such
services
are
provided
at
no
cost
to
EPA).
Note
that
persons
cannot
waive
compensation
(
i.
e.,
agree
to
provide
gratuitous
services)
for
which
there
is
a
statutory
right
to
payment,
unless
a
law
permits
the
waiver.
(
For
situations
concerning
State
employees,
see
Section
3.4.10.
Laws
that
permit
services
without
compensation
are
5
U.
S.
C.
§
3109
for
experts
or
consultants
and
5
U.
S.
C.
§
3111
for
student
volunteers.)
Page
66
Peer
Review
Handbook
3.6.2
Can
I
Use
a
Contract
to
Obtain
Peer
Review
Services?

The
Agency
may
contract
for
peer
review
services.
The
contract
may
be
written
solely
for
peer
reviews
or
be
included
as
one
of
several
specifically
described
interrelated
tasks
in
a
contract
that
requires
the
contractor
to
provide
more
than
just
peer
review
services.

For
assistance
in
preparing
the
necessary
pre­
award
documents,
program
officials
should
consult
Chapter
2
of
the
Contracts
Management
Manual
(
CMM).
Chapter
2
is
available
on
EPA s
Intranet
(
http://
epawww.
epa.
gov/
oamintra/
policy/
cmm.
pdf
)
and
on
the
Agency
LAN
Services
(
Administration/
Management/
OAM
Procurement
Policy
Information).

3.6.3
How
Do
I
Write
a
Statement
of
Work
for
Contracts?

The
Statement
of
Work
(
SOW)
must
clearly
specify
that
the
contractor
is
responsible
for
preparing
peer
review
evaluations
and
set
forth
guidelines
for
the
peer
review
of
scientific
or
technical
documents.
The
contractor
may
perform
the
peer
review
with
in­
house
staff,
subcontractors
or
consultants.
Any
guidelines
for
performing
peer
reviews
to
ensure
soundness
and
defensibility
must
be
developed
by
the
program
office
and
made
part
of
the
contract.
The
contractor
would
then
ensure
that
the
peer
reviews
adhere
to
the
guidelines.

The
SOW
should
include
the
list
of
questions
that
EPA
wants
the
contractor
to
include
in
the
charge
to
the
peer
reviewers.
While
the
contractor
will
be
the
one
that
will
prepare
and
send
the
formal
charge
to
the
peer
reviewers,
EPA
should
provide
the
list
of
questions
to
the
contractor
(
see
Section
3.2
for
general
discussion
on
charge
to
peer
reviewers).

Within
the
SOW,
the
contractor
should
be
directed
to
inquire
whether
prospective
peer
reviewers
have
any
actual
or
potential
organizational
or
personal
conflicts
of
interest
or
other
matters
that
would
create
the
appearance
of
a
lack
of
impartiality,
including
whether
they
have
had
or
presently
have
a
financial
relationship
with
EPA.
Further,
the
SOW
needs
to
specify
that
contractors
are
required
to
provide
EPA
with
a
summary
of
the
procedures
and
efforts
made
to
identify,
disclose,
and
ensure
that
no
independence
or
conflict
of
interest
concerns
arise
during
the
performance
of
the
contract.

The
SOW
cannot
simply
define
the
role
of
the
prime
contractor
as
arranging
for
the
services
of
others
to
perform
peer
reviews
and
logistics
for
meetings.
Unless
the
prime
contractor
is
clearly
tasked
with
responsibility
for
performing
peer
reviews,
individual
peer
reviewers'
fees
and
associated
travel
expenses
are
not
payable
under
the
contract.

The
EPA
may
pay
for
the
reviewer's
comments
or
evaluation,
and
also
for
attendance
at
a
meeting
with
the
Agency
and
other
reviewers
to
discuss
the
results
of
the
peer
review.
If
the
SOW
calls
for
the
preparation
of
comments
or
an
evaluation,
and
specifies
a
meeting
with
the
Peer
Review
Handbook
Page
67
Agency
and
other
peer
reviewers
to
discuss
the
results
of
the
peer
review,
payment
is
appropriate.
The
peer
reviewer's
attendance
at
the
meeting
would
then
be
part
of
contract
performance.

Example
statements
of
work
are
found
in
Appendix
E
(
Example
Statements
of
Work
for
Contracts).

3.6.4
What
are
Advisory
and
Assistance
Services
(
AAS)
or
Sensitive
Activities?

Contracts
that
provide
services
that
support
or
improve
Agency
decision­
making
or
policy
development
are
subject
to
special
management
controls.
These
services
include
services
acquired
from
non­
Governmental
sources
by
contract
to
support
or
improve
Agency
policy
development,
decision­
making,
management,
and
administration,
or
research
and
development
activities.
See
Federal
Acquisition
Regulation
(
FAR)
37.201
for
a
more
specific
definition
of
AAS.
Such
services
may
take
the
form
of
information,
advice,
opinions,
alternatives,
conclusions,
recommendations,
training,
and
direct
assistance.
For
additional
information
on
advisory
and
assistance
services
and
sensitive
activities,
program
officials
should
review
Chapter
2
of
the
CMM.

New
contracts
for
these
services
require
management
approvals
prior
to
issuance
of
the
solicitation.
For
the
thresholds
that
have
been
established
for
approval
of
these
justifications,
see
Figure
5,
Item
B
Management
Approvals
set
forth
after
Chapter
2
of
the
CMM
(
for
current
approval
levels).

3.6.5
What
are
Some
Management
Controls
for
Contracts?

Contracting
for
peer
review
services
is
permitted.
However,
because
of
the
potential
for
improper
use
of
these
contracts,
special
management
controls
are
required.

a)
Inherently
Governmental
Functions
(
IGFs)
 
OFPP
Policy
Letter
92­
1,
dated
September
23,
1992,
describes
(
1)
functions
that
are
inherently
governmental
and
must
be
performed
only
by
Government
employees
and
(
2)
functions
that
may
be
contracted,
but
so
closely
support
Government
employees
in
their
performance
of
IGFs
that
the
contract
terms
and
performance
require
close
scrutiny
by
Federal
officials.
Federal
Acquisition
Regulation
(
FAR)
coverage
of
inherently
governmental
functions
is
at
FAR
Subpart
7.5.

Peer
reviews
represent
only
a
contractor's
recommendations,
advice
or
analysis
of
a
document.
Agency
officials
must
make
the
official
Agency
decision
regarding
acceptability
and/
or
quality
of
the
document.
To
ensure
that
Agency
officials
are
not
improperly
influenced
by
recommendations
in
the
peer
review,
management
Page
68
Peer
Review
Handbook
controls
must
be
included
in
the
contract.
One
possible
control
would
be
to
require
the
peer
reviewers
to
submit
with
their
evaluations
or
comments
a
description
of
the
procedures
used
to
arrive
at
their
recommendations;
a
summary
of
their
findings;
a
list
of
sources
relied
upon;
and
make
clear
and
substantiate
the
methods
and
considerations
upon
which
their
recommendations
are
based.
To
the
extent
possible,
the
contract
should
set
forth
any
guidelines
or
criteria
for
performance
of
the
peer
review.
Agency
officials
should
document
their
evaluations
of
the
quality
and
validity
of
the
peer
review.

b)
Conflict
of
Interest
(
COI)
­­
Another
important
factor
is
that
the
objectivity
of
the
peer
review
should
not
be
improperly
influenced
or
undermined
by
the
contractor
performing
the
review.
To
identify
and
avoid
or
mitigate
actual
or
potential
COI,
the
contract
should
include
controls.
Such
controls
might
require
the
contractor
to
report
on
prior
and
current
work,
and
prior
clients
that
might
create
COI.
Other
controls
might
include
Agency
review
and
placing
limits
or
advance
approval
on
future
work.
There
should
also
be
procedures
implemented
to
assure
that
the
contractor
does
not
gain
an
unfair
advantage
in
future
requirements
as
a
result
of
their
performance
of
peer
reviews.
Program
officials
should
consult
the
Contracting
Officer
(
CO)
for
special
contract
clauses.

FAR
coverage
of
conflicts
of
interest
is
at
FAR
Subpart
9.5.
The
EPA
Acquisition
Regulations
(
EPAAR)
at
48
CFR
Subpart
1509.5
generally
mandates
conflicts
of
interest
solicitation
provisions
and
contract
clauses,
but
makes
them
optional
for
procurements
accomplished
through
the
use
of
simplified
acquisition
procedures
as
set
forth
in
FAR
Part
13.
Also
see
48
CFR
1509.507­
1(
b)(
3)
and
48
CFR
1552.209­
70,
­
71
&
­
72
as
additional
resources.

Contract
for
peer
review
services:
An
EPA
contracting
officer
will
include
conflicts
of
interest
solicitation
provisions
and
contract
clauses
as
a
matter
of
course
without
involvement
by
the
EPA
project
officer,
if
the
peer
review
services
are
not
obtained
pursuant
to
the
simplified
acquisition
procedures
in
FAR
Part
13.
If
the
peer
review
services
are
subcontracted
pursuant
to
a
prime
contract,
then
the
prime
contractor
is
ordinarily
required
to
include
a
conflicts
of
interest
clause
substantially
similar
to
the
conflicts
of
interest
clause
in
the
primary
contract
in
its
subcontract
to
the
peer
reviewer.

Although
the
EPA
contracting
officer
and/
or
prime
contractor
has
the
primary
responsibility
to
include
the
required
conflicts
of
interest
provisions/
clauses,
the
EPA
project
officer
may
nevertheless
wish
to:
Peer
Review
Handbook
Page
69
1)
Highlight
the
conflict
of
interest
requirements
in
the
Statement
of
Work
(
SOW)
for
the
procurement
of
the
peer
review
services
2)
Develop
a
specific
conflict
of
interest
clause
regarding
the
peer
review
at
issue
as
a
substitute
or
in
addition
to
the
standard
conflicts
of
interest
clause
3)
Review
the
solicitation/
contract
to
make
sure
that
the
required
conflicts
of
interest
clause
has
been
included
4)
Ensure
that
the
SOW
includes
a
requirement
for
the
contractor
to
provide
EPA
a
summary
of
the
work
performed
to
identify
and
resolve
conflict
of
interest/
independence
concerns
with
peer
review
candidates
Simplified
acquisitions
of
peer
review
services:
Although
conflict
of
interests
requirements
are
optional
for
simplified
acquisitions,
they
are
nevertheless
a
good
idea.
Accordingly,
an
EPA
project
officer
obtaining
peer
review
services
with
simplified
acquisition
procedures
(
see
Section
3.6.7)
should
request
the
purchasing
agent/
contracting
officer
to
include
a
conflict
of
interest
solicitation
provision
and
contract
clause
in
the
purchase
order.

c)
Confidential
Business
Information
(
CBI)/
Privacy
Act
Protected
Information
and
Other
Sensitive
Information
­­
When
peer
reviewers
are
not
employees
of
the
United
States
Government,
it
is
unlikely
that
the
EPA
will
have
authority
to
give
reviewers
access
to
confidential
business
information
in
the
absence
of
consent
for
such
disclosure
by
the
CBI
submitter.
Therefore,
all
documents
provided
to
non­
Federal
reviewers
must
be
screened
for
information
claimed
as
CBI.
Even
where
business
information
has
not
been
explicitly
claimed
as
CBI,
if
it
is
of
a
kind
where
the
submitter
might
be
expected
to
object
to
its
release,
prior
to
release
the
submitter
must
be
asked
whether
it
wishes
to
assert
a
claim,
unless
the
submitter
has
previously
been
informed
that
failure
to
assert
a
CBI
claim
may
result
in
disclosure
without
notice.
Language
is
included
in
the
contract
to
clearly
identify
any
required
procedures
or
processes
prior
to
release
of
any
protected
information,
including
any
requirements
for
confidentiality
agreements,
as
well
as
limits
on
use
and
disclosure
of
the
data
by
contractor
personnel.

d)
Personal
services
­­
Under
contracts,
the
EPA
may
not
engage
the
peer
reviewers
in
any
improper
personal
services
relationships,
i.
e.,
an
arrangement
under
which
contractor
personnel
are
subject
to
relatively
continuous
supervision
and
direct
control
by
an
Agency
official
or
employee.
These
relationships
are
characterized
Page
70
Peer
Review
Handbook
as
one
where
the
contractor
employee
interacts
with
the
Agency
in
a
manner
similar
to
that
of
a
Federal
employee.

To
avoid
these
improper
relationships,
program
officials
should
write
well­
defined
SOWs.
The
SOWs
should
set
forth
the
requirements
in
detail
for
work
to
be
performed
independently,
including
the
manner
in
which
it
will
be
evaluated.
The
SOW
must
set
forth
what
work
is
to
be
performed
not
how
the
work
is
to
be
performed.
Technical
direction
may
be
used
to
clarify
ambiguous
technical
requirements
to
ensure
efficient
and
effective
contractor
performance,
and
is
not
considered
supervision
or
assignment
of
tasks.
For
additional
information,
program
officials
should
consult
EPA
Order
1901.1A,
Use
of
Contractor
Services
to
Avoid
Improper
Contracting
Relationships
and
FAR
Subpart
37.1
3.6.6
Can
I
Identify
and/
or
Select
Peer
Reviewers
When
Using
a
Contract?

No.
Program
officials
cannot
interfere
in
a
contractor's
authority
and
responsibility
to
perform
work
by
"
selecting"
who
will
perform
the
peer
review
(
doing
so
may
invoke
FACA
­­
see
Section
2.8).
The
Federal
Acquisition
Regulation
(
FAR)
governs
the
Contracting
Officer s
(
CO's)
and
program
officials'
relationship
with
the
contractor.
Keep
in
mind
that
for
contracting
purposes,
contractors
are
required
to
obtain
Contracting
Officer
(
CO)
approval
of
subcontractors
and
the
CO
generally
seeks
the
input
of
the
work
assignment
manager
(
WAM
­­
the
WAM
may
be
the
Peer
Review
Leader
in
many
cases)
before
approving
the
use
of
subcontractors.

EPA
can
establish
criteria
for
the
sort
of
individuals
that
might
participate
on
a
peer
review
panel.
However,
the
Agency
must
not
be
involved
in
the
selection
of
individual
peer
reviewers,
and
should
avoid
commenting
on
the
contractor s
selection
of
peer
reviewers
other
than
to
determine
whether
the
panel,
once
selected,
meets
the
criteria
established.
EPA
may
identify
a
pool
of
qualified
subcontractors
and
consultants
to
the
prime
contractor
(
listed
in
alphabetical
order),
but
cannot
direct
the
use
of
any
particular
subcontractor
or
consultant.

3.6.7
Can
I
Use
Simplified
Acquisition
Procedures
to
Obtain
Peer
Reviewers?

In
some
instances,
peer
reviewers
can
be
obtained
via
simplified
acquisition
procedures.
The
acquisition
of
supplies
or
non­
personal
services
from
the
open
market
and
on
a
sole
source
basis
when
the
aggregate
amount
involved
in
any
one
transaction
does
not
exceed
$
100,000
constitutes
a
simplified
acquisition
(
FAR
Part
13).
The
same
considerations
in
the
Section
3.6.5
discussion
on
IGFs,
COI,
access
to
CBI,
and
personal
services
apply
to
simplified
acquisitions.
Normally,
the
Government
issues
a
purchase
order
directly
to
the
individual
peer
reviewer,
instead
of
to
a
prime
contractor
who
may
subcontract
for
performance
of
the
peer
review.
Peer
Review
Handbook
Page
71
a)
Approvals
­­
All
small
purchases
for
peer
reviews
are
considered
Advisory
and
Assistance
Services
(
AAS).
See
Chapter
2
of
the
Contract
Management
Manual
(
CMM)
for
the
appropriate
approval
levels
for
AAS.

b)
Competition
­­
The
Federal
Acquisition
Regulation
(
FAR)
requires
competition
for
purchases
in
excess
of
$
2,500.
Purchases
for
more
than
$
2,500
and
not
more
than
$
100,000
are
to
be
made
only
from
small
businesses
unless
the
Contracting
Officer
is
not
able
to
obtain
two
or
more
offers
from
small
businesses
that
are
competitive
in
terms
of
market
price,
quality,
and
delivery.
Only
one
source
need
be
solicited
if
the
Contracting
Officer
determines
that
only
one
source
is
reasonably
available.
Contracting
Officers
are
encouraged
to
use
best
value.

c)
Procurement
Requests
­­
Program
Officers
should
include
the
following
in
all
PRs
for
the
purchase
of
peer
reviews:

1)
A
fixed­
price
amount
at
or
below
the
simplified
acquisition
threshold
2)
A
detailed
description
of
the
requested
services,
inclusive
of:

(
a)
Total
quantity
per
line
item
(
b)
Estimated
unit
price
per
line
item
(
c)
Total
cost
per
line
item
(
d)
Specific
deliverables
for
each
line
item
(
e)
Total
cost
of
the
purchase
request
3)
Reference
FAR
Subpart
3.6
and
Environmental
Protection
Agency
Acquisition
Regulation
(
EPAAR)
Subpart
1503.601
regarding
sources
from
Government
employees
or
organizations
owned
and
controlled
by
them.

4)
Provide
sources
from
small
businesses,
if
available.

5)
If
the
request
is
a
sole
source
purchase,
justification
must
be
provided
in
accordance
with
the
EPAAR
Subpart
1513.170­
1.
Page
72
Peer
Review
Handbook
3.6.8
How
is
Travel
Handled
with
Contracts
or
Purchase
Orders?

Funds
obligated
on
a
contract
or
purchase
order
are
available
to
pay
for
the
costs
of
producing
the
peer
review
including
the
travel
costs
and
fee
of
the
peer
reviewer.

The
EPA
acquires
peer
reviews
through
simplified
acquisitions
issued
directly
to
peer
reviewers
or
through
contracts
with
companies,
which
provide
the
peer
review
services.
By
issuing
a
purchase
order
or
awarding
a
contract
for
peer
review
services,
the
EPA
may
pay
not
only
for
the
peer
review
services/
comments,
but
also
for
participation
in
a
meeting
with
the
Agency
and
other
reviewers
to
discuss
comments.
The
scope
of
work
of
the
contract
must
require
the
contractor
or
individual
peer
reviewer
as
appropriate
to
provide
peer
review
services
and
indicate
whether
the
contractor
or
peer
reviewer
will
be
required
to
discuss
a
specific
peer
review
work
product
with
the
Agency
and/
or
with
other
peer
reviewers.
Participation
in
a
meeting
to
discuss
a
peer
review
work
product
would
then
be
part
of
the
contract's
performance.
Thus,
the
contract
may
serve
as
the
mechanism
to
pay
for
peer
review
services
and
associated
travel
expenses
to
provide
comments
to
the
EPA.

3.6.9
How
is
Travel
Handled
with
Special
Government
Employees?

The
term
Special
Government
Employee
(
SGE)
is
defined
in
18
U.
S.
C.
202(
a)
as
an
officer
or
employee
of
an
agency
who
performs
temporary
duties,
with
or
without
compensation,
for
not
more
than
130
days
in
a
period
of
365
days,
either
on
a
full­
time
or
intermittent
basis.

Travel
and
per
diem
expenses
of
experts
hired
as
SGEs
for
peer
review
may
only
be
paid
through
the
issuance
of
invitational
travel
orders
(
5
U.
S.
C.
§
5703).
These
invitational
travel
and
per
diem
expenses
should
be
charged
to
an
appropriate
EPA
travel
account.
The
Federal
Travel
Regulations
govern
the
invited
travelers
reimbursement.

Members
of
the
SAB,
SAP,
and
other
FACA
advisory
committees
are
often
brought
on
board
as
SGEs.
It
is
not
appropriate
to
reimburse
travel
or
per
diem
expenses
of
advisory
committee
members
(
SGEs)
through
a
contract.
Peer
Review
Handbook
Page
73
4.
COMPLETING
A
PEER
REVIEW
4.1
Overview
Performance
of
the
formal
peer
review
is
not
the
final
stage
in
the
development
of
the
work
product.
Rather,
it
is
an
important
stage
in
developing
the
work
product,
with
the
final
work
product
representing
the
true
end
of
the
peer
review.
As
a
result,
the
peer
review
process
closes
with
three
major
activities:
evaluating
comments
and
recommendations,
utilizing
peer
review
comments
for
completing
the
final
work
product,
and
organizing
and
maintaining
a
record
of
the
peer
review.

Careful
attention
to
all
of
these
elements,
singly
and
together,
assures
a
credible
peer
review
process.
Conversely,
inattention
can
nullify
the
peer
review
attempt.
A
well­
planned
peer
review
applied
to
a
reasonable
quality
starting
work
product,
followed
by
responsible,
visible
utilization
of
peer
review
suggestions
in
the
final
product
assures
a
credible
product
for
use
in
Agency
decision­
making.
The
peer
review
is
not
complete
untilThe
peer
review
is
not
complete
until
the
the
peer
review
comments
are
peer
review
comments
are
incorporated
into
the
incorporated
into
the
final
work
final
work
product,
or
reasons
are
stated
why
product.
such
comments
are
not
to
be
incorporated.
However,
for
the
purposes
of
the
annual
report
to
the
Agency s
Peer
Review
Product
Tracking
Database
only,
the
work
product
can
be
moved
from
List
B
(
Candidate
Products
for
Future
Peer
Review)
(
or
List
C
­
Products
for
Which
a
Decision
has
been
made
not
to
Peer
Review)
in
some
instances)
to
List
A
(
Products
Peer
Reviewed
Since
1991)
when
the
Decision
Maker
decides
on
how
the
peer
review
comments
will
be
addressed
and
this
decision
is
documented
in
the
peer
review
record.

The
peer
review
record
is
completed
only
when
it
contains
a
copy
of
the
final
work
product
(
when
there
is
one)
that
addresses
the
peer
review
comments.

4.2
Final
Work
Product
4.2.1
How
Do
I
Incorporate
Peer
Review
Comments
into
the
Final
Work
Product?

The
Peer
Review
Leader
must
carefully
evaluate
and
analyze
all
peer
review
comments
and
recommendations.
As
discussed
earlier,
a
carefully
crafted
charge
to
the
peer
reviewers
simplifies
organizing
and
analyzing
comments.
Also,
any
other
issues
that
are
raised
need
to
be
identified
and
evaluated.
Page
74
Peer
Review
Handbook
The
validity
and
objectivity
of
the
comments
need
to
be
evaluated.
Analyses
may
include
consultation
with
other
experts/
staff
within
the
Office
and/
or
Agency.
Adequate
documentation
is
needed
to
show
that
comments
are
accepted
or
rejected
 
the
documentation
can
be
brief,
but
must
address
the
legitimate,
valid
comments.
The
peer
review
record
must
contain
a
document
describing
the
Agency s
response
to
the
peer
review
comments.

The
Peer
Review
Leader
should
brief
the
Decision
Maker
(
including
all
appropriate
managers
in
the
Peer
Review
Leader s
chain
of
command)
on
the
charge,
profile
of
peer
reviewers,
the
peer
review
comments,
and
provide
a
proposal
on
how
to
address
the
peer
review
comments.
It
is
the
responsibility
of
the
Peer
Review
Leader
to
obtain
Decision
Maker
approval
of
the
approach
to
addressing
the
peer
review
comments.
The
Peer
Review
Leader
should
clearly
identify
for
the
Decision
Maker
any
major
peer
review
comments
that
will
not
be
accepted
and
why.

Comments
that
have
significant
impact
on
time,
budgetary,
and/
or
resource
requirements
are
particularly
important
and
need
to
be
evaluated
in
consultation
with
management.
These
comments
may
lead
to
allocation
of
additional
resources
and
a
revised
schedule
for
the
completion
of
the
work
product.

4.2.2
What
Actions
are
Potentially
Forthcoming
from
Peer
Review?

Peer
review
comments
and
recommendations
may
entail
significant
impacts
on
the
planned
project
schedule,
budget,
or
other
resource
requirements.
Management
decisions
related
to
revisions
in
one
or
more
of
these
areas
may
be
appropriate.

The
substantive
issues
or
concerns
expressed
by
peer
reviewers
may
suggest
that
wider
scientific
and
technical
consultation
is
needed
to
ensure
the
adequacy
of
the
work
product.

The
peer
review
comments
and
recommendations
on
a
final
product
may
provide
a
basis
for
bringing
the
associated
project
to
closure.

4.2.3
What
Should
the
Final
Work
Product
Say
About
the
Peer
Review
Process?

If
the
product
has
been
peer
reviewed,
you
should
describe
the
peer
review
in
the
document.
Frequently,
this
will
be
part
of
a
description
of
the
process
of
developing
the
product.
It
can
be
brief
and
does
not
need
to
describe
the
process
or
discuss
the
peer­
review
comments
in
great
detail.
The
description
can
be
included
in
an
introduction,
preamble,
or
appendix.

When
there
were
significant
peer­
review
comments,
and
particularly
if
they
are
not
being
accepted,
the
document
will
generally
discuss
the
issue
and
describe
the
reasons
for
the
Agency s
choices
in
the
appropriate
sections
of
the
document.
The
level
of
detail
that
is
needed
Peer
Review
Handbook
Page
75
is
a
matter
of
judgment
and
will
depend
on
the
importance
and
degree
of
controversy
of
the
issue.

If
a
major
scientific
or
technical
product
has
not
been
peer­
reviewed,
this
fact
should
be
noted
in
the
document,
perhaps
in
an
introduction
or
description
of
its
scope.
This
section
should
briefly
indicate
the
reasons
that
peer
review
was
not
conducted.

Derivative
products
of
major
scientific
or
technical
products
(
such
as
fact
sheets,
press
releases,
and
brochures)
do
not
need
to
discuss
whether
the
underlying
products
were
peer
reviewed.

4.2.4
Can
the
Identity
of
Peer
Reviewers
be
Kept
Anonymous?

Yes,
up
to
a
point.
In
the
ordinary
course
of
events,
you
can
often
discuss
comments
received
without
attributing
the
comments
to
a
specific
reviewer.
However,
if
the
matter
has
gone
to
litigation,
the
litigating
parties
can
discover
the
names
of
anyone
who
contributed
to
a
Federal
product,
including
peer
reviewers.
Therefore,
it
is
not
possible
to
totally
shield
peer
reviewers.
In
addition,
it
may
be
difficult
to
shield
the
names
of
the
peer
reviewers
when
the
Agency
is
responding
to
a
Freedom
of
Information
Act
(
FOIA)
request.

If
a
peer
reviewer
requests
anonymity
at
the
outset
of
the
peer
review,
the
Peer
Review
Leader
needs
to
inform
the
peer
reviewer
of
the
above
possible
eventualities.
The
Agency
will
in
the
ordinary
course
of
events
attempt
to
maintain
the
confidentiality
of
the
peer
reviewers
and
their
comments
from
public
disclosure,
but
it
is
recognized
in
many
instances,
for
example
open
public
meetings
and
the
above
circumstances,
this
can t
be
assured.
Remember,
the
Agency
is
committed
to
working
 
as
if
in
a
fishbowl 
and
most
of
its
activities
are
transparent
to
the
public
(
except
where
confidential
business
information
is
concerned).
It
is
recognized
that
this
may
be
a
deterrent
to
possible
peer
reviewers,
but
this
is
a
reality
that
has
to
be
understood.

4.3
Completing
the
Peer
Review
Record
4.3.1
How
Do
I
Complete
the
Peer
Review
Record?

Once
the
Peer
Review
Leader
has
completed
the
peer
review
and
the
final
work
product
(
where
one
is
prepared),
the
peer
review
record
is
brought
up
to
date
and
then
archived
according
to
that
organization s
procedure
(
see
Section
4.3.2).
The
peer
review
record
must
be
indexed
and
maintained
in
an
organization s
archive
(
repository).
The
location
of
the
peer
review
record
needs
to
be
readily
identifiable
so
interested
parties
can
locate
and
obtain
materials
easily
and
quickly.
The
peer
review
record
should
be
placed
in
any
associated
established
public
docket,
if
required,
in
addition
to
the
organizational
archive.
As
a
courtesy,
a
copy
of
the
revised
work
product
may
be
sent
to
the
peer
reviewers
for
information.
Page
76
Peer
Review
Handbook
The
Peer
Review
Leader
will
collect
the
following
materials
for
the
peer
review
record
and
submit
for
archiving;
including
at
least
(
see
also
Section
2.5.3):

a)
The
draft
work
product
submitted
for
peer
review
b)
Materials
and
information
(
including
the
charge)
given
to
the
peer
reviewers
c)
Written
comments,
information,
and
materials
received
from
the
peer
reviewers
d)
Information
about
the
peer
reviewers
(
such
as
reviewers 
names,
affiliations,
and
a
summary
of
efforts
to
identify
potential
conflicts
and
their
resolution)

e)
Logistical
information
about
conduct
of
the
peer
review
(
such
as
times
and
locations
of
meetings)

f)
A
memorandum,
or
other
written
record,
approved
by
the
Decision
Maker,
responding
to
the
peer
review
comments
specifying
acceptance
or,
where
thought
appropriate,
rebuttal
and
non­
acceptance
g)
The
final
work
product
4.3.2
Where
Should
the
Peer
Review
Records
be
Kept,
and
for
How
Long?

During
the
active
conduct
of
the
peer
review,
the
Peer
Review
Leaders
maintain
the
peer
review
record
with
themselves
until
the
peer
review
is
totally
completed.
Minimally,
the
file
should
be
maintained
until
one
year
after
the
completed
peer
review
is
reported
in
the
next
annual
reporting.
After
that,
the
peer
review
record
should
be
maintained
for
a
 
reasonable
period
of
time. 
Establishment
and
maintenance
of
the
archive
where
the
peer
review
records
ultimately
reside
are
an
organization s
responsibility
(
i.
e.,
not
that
of
an
individual
program
manager
or
Peer
Review
Leader).
Generally,
to
allow
flexibility,
individual
offices
and
regions
will
decide
the
appropriate
level
of
organizational
responsibility
and
how
they
will
meet
the
 
routinely
available 
requirement.
The
peer
review
record
may
be
kept
with
other
records
relating
to
the
overall
project,
as
long
as
it
is
easily
and
separately
identifiable.
The
peer
review
record
should
be
maintained
in
accordance
with
the
Agency s
record
keeping
schedule
for
such
records.
One
long
term
archiving
mechanism
may
be
the
formal
archiving
at
the
Federal
Records
Center
in
Suitland,
MD.
(
Note:
This
is
the
same
question
as
Section
2.5.7,
but
applies
in
this
chapter
as
well).
Peer
Review
Handbook
Page
77
4.3.3
Is
Information
Regarding
a
Peer
Review
Subject
to
Release
under
FOIA?

Yes,
it
is
subject
to
release
if
EPA
receives
a
Freedom
of
Information
Act
(
FOIA)
request,
unless
the
peer
review
information
meets
the
criteria
for
an
exemption
under
the
FOIA.
(
http://
www.
epic.
org/
open_
gov/
foia/
us_
foia_
act.
html).
Peer
Review
Handbook
Page
79
SUBJECT
INDEX
This
is
an
alphabetical
listing
of
subjects
from
the
Handbook
and
the
pertinent
page
numbers
where
they
are
found.

­
A
­
Administrative
Procedures
Act
(
12)
Administrator
(
9,
17)
Advisory
and
Assistance
Services
(
66)
Analytic
blueprints
(
13,
25,
40,
49)
Annual
reporting
requirements
(
14,
18)
Anonymity
of
peer
reviewers
(
75)
Archiving
(
47,
75)
Assistant
Administrators
(
17)
­
B
­
Balancing
of
peer
reviewers
(
22)
Board
of
Scientific
Counselors
(
BOSC)
(
41)
Budget
(
18,
40,
48)
­
C
­
Candidate
for
peer
review
(
26)
Categories
of
work
products
(
16,
25)
Charge
(
46,
53,
76)
Chemical
action
reports
(
16)
Clean
Air
Scientific
Advisory
Committee
(
41)
Compensation
(
65)
Competition
(
70)
Completing
a
peer
review
(
4,
73)
Conducting
a
peer
review
(
3,
53)
Conference
proceedings
(
25)
Confidential
business
information
(
CBI)
(
61,
69)
Confidential
Financial
Disclosure
(
59)
Confidentiality
(
63)
Conflicts
of
interest
(
22,
31,
58,
67)
Contracting
officer
(
57,
63,
68,
69)
Contracts
(
31,
64,
65,
68)
Contracts
Management
Manual
(
CMM)
(
65,
67)
Controversy
(
27,
40)
Cooperative
agreements
(
31)
Court
ordered
deadlines
(
44,
61)
­
D
­
Decision­
Maker
(
17,
26,
39)
Deputy
Administrator
(
14,
17,
18)
Designated
Ethics
Official
(
DEO)
(
59)
Division
Directors
(
17)
Docket
(
46,
47,
75)
­
E
­
Economic
work
products
(
27)
Electronic
records
(
46)
Environmental
Impact
Statement
(
EIS)
(
35)
Environmental
regulatory
models
(
36)
Executive
Order
12866
(
13,
29)
External
peer
review
(
40,
41,
57)
­
F
­
Federal
Acquisition
Regulation
(
FAR)
(
67)
Federal
Advisory
Committee
Act
(
FACA)
(
49,
62)
Federal
Register
(
12,
50)
Final
work
product
(
73)
Freedom
of
Information
Act
(
FOIA)
(
16,
42,
60,
77)
Frequency
of
peer
review
(
43)
­
G
­
Grants
(
31)
Gratuitous
services
(
65)
­
I
­
Independent
peer
reviewer
(
10,
11,
21,
55)
Inherently
governmental
functions
(
IG
Fs)
(
67)
Interacting
with
peer
reviewers
(
64)
Interagency
agreement
(
31)
Internal
peer
review
(
41)
Internal
peer
reviewer
(
22)
­
J
­
Journal
article
(
16,
42)
­
L
­
Legal
considerations
(
49)
Letter
reviews
(
41,
50)
List
A
(
15)
List
B
(
15)
List
C
(
15,
16)
List
D
(
16,
42)
Litigation
(
16,
75)
­
M
­
Major
impact
(
27,
40)
Major
scientific
and
technical
work
product
(
5,
9,
15,
26,
37)
Page
80
Peer
Review
Handbook
Managers
planning
checklist
for
peer
review
(
5)
Materials
for
peer
reviewers
(
62)
Mechanisms
for
peer
review
(
39)
­
N
­
National
Academy
of
Sciences
(
31,
42)
NEPA
Products
(
35)
Non­
major
scientific
and
technical
work
products
(
16,
37,
38)
Novelty
(
27,
40)
NPDES
permits
(
16)
­
O
­
Office
Directors
(
17)
Office
of
Research
and
Development
(
ORD)
(
14,
20,
23)
Open
meeting
(
49)
Other
organization s
work
products
(
36)
­
P
­
Peer
consultation
(
11)
Peer
input
(
10)
Peer
involvement
(
10)
Peer
review
(
9,
10)
Peer
Review
Advisory
Group
(
PRAG)
(
17,
23)
Peer
review
comments
(
46,
47,
73,
76)
Peer
Review
Coordinator
(
14,
17,
20,
23)
Peer
Review
Leader
(
17­
19,
39,
47,
61,
62,
73,
75)
Peer
review
materials
(
62)
Peer
review
mechanism
(
5,
39)
Peer
review
panel
(
22,
56,
57)
Peer
Review
Product
Tracking
(
PRPT)
database
(
14,
19,
34,
45)
Peer
review
record
(
5,
21,
45,
75)
Peer
review
schedule
(
18)
Peer
review
services
(
65)
Peer
reviewers
(
21,
46,
55,
56,
61,
70,
76)
Personal
services
(
69)
Planning
a
peer
review
(
2,
25)
Premanufacturing
notices
(
PMNs)
(
16)
Privacy
Act
(
60)
Procurement
requests
(
71)
Project
manager
(
18,
19,
37)
Public
comment
(
12)
­
R
­
RCRA
permits
(
16)
Record
of
Decision
(
ROD)
(
25)
Records
management
(
60)
Regional
Administrators
(
17)
Regulatory
development
process
(
13)
Regulatory
negotiations
(
14)
Rulemaking
(
13,
27,
46)
­
S
­
Science
Advisory
Board
(
SAB)
(
28,
41,
50,
56)
Science
Policy
Council
(
SPC)
(
14,
17,
20,
23)
Scientific
Advisory
Panel
(
SAP)
(
41,
50,
Scientific
and
technical
work
product
(
41,
50,
56)
Selection
of
peer
reviewers
(
25)
Sensitive
activities
(
55)
Simplified
acquisitions
of
peer
review
services
(
66)
Site
specific
decisions
(
69,
70)
Social
Science
work
product
(
35)
Special
Government
Employee
(
SGE)
(
30)
Stakeholder
involvement
(
59,
72)
State
employees
(
12,
13,
40,
54)
Statement
of
work
for
contracts
(
62)
Strategic
plans
(
59,
66)
Subcontractor
(
25)
Subject
matter
expert
(
51,
57)
­
T
­
Tier
1
and
Tier
2
(
22)
Tier
3
(
13,
55)
Time
frame
for
peer
review
(
13)
Training
(
44,
55)
Travel
(
18,
19)
­
V
­
Voluntary
consensus
standards
(
71)
Voluntary
services
(
39)
­
W
­
Work
Assignment
Manager
(
WAM)
(
57)
Peer
Review
Handbook
Page
81
COMMONLY
USED
ACRONYMS
AA
Assistant
Administrator
AAS
Advisory
and
Assistance
Services
ASTM
American
Society
of
Testing
and
Materials
BOSC
Board
of
Scientific
Counselors
CBI
Confidential
Business
Information
CFR
Code
of
Federal
Regulations
CMM
Contracts
Management
Manual
CO
Contract(
ing)
Officer
COI
Conflict
of
Interest
DEO
Designated
Ethics
Official
EIS
Environmental
Impact
Statement
EPA
Environmental
Protection
Agency
EPAAR
EPA
Acquisition
Regulations
FACA
Federal
Advisory
Committee
Act
FAR
Federal
Acquisition
Regulations
FIFRA
Federal
Insecticide,
Fungicide
and
Rodenticide
Act
FOIA
Freedom
of
Information
Act
FTE
Full
Time
Equivalent
GSA
General
Services
Administration
HHS
Health
and
Human
Services
IAG
Interagency
Agreement
IGF
Inherently
Governmental
Function
IRIS
Integrated
Risk
Information
System
ISO
International
Organization
for
Standardization
LAN
Local
Area
Network
NAS
National
Academy
of
Sciences
NEPA
National
Environmental
Policy
Act
NPDES
National
Pollutant
Discharge
Elimination
System
NTTAA
National
Technology
Transfer
and
Advancement
Act
of
1995
OAM
Office
of
Acquisition
Management
OEI
Office
of
Environmental
Information
OGC
Office
of
General
Counsel
OMB
Office
of
Management
and
Budget
ORD
Office
of
Research
and
Development
PMNs
Premanufacture
Notice
PO
Purchase
Order
PRAG
Peer
Review
Advisory
Group
PRPT
Peer
Review
Product
Tracking
Database
RA
Regional
Administrator
RC
Regional
Counsel
RCRA
Resource
Conservation
and
Recovery
Act
RIAs
Regulatory
Impact
Analyses
Page
82
Peer
Review
Handbook
ROD
Record
of
Decision
SAB
Science
Advisory
Board
SAP
Scientific
Advisory
Panel
SGE
Special
Government
Employee
SOPs
Standard
Operating
Procedures
SOW
Statement
of
Work
SPC
Science
Policy
Council
USC
United
States
Code
WTI
Waste
Technologies
Industries
WAM
Work
Assignment
Manager
Peer
Review
Handbook
Page
A­
1
APPENDIX
A
­
U.
S.
ENVIRONMENTAL
PROTECTION
AGENCY
PEER
REVIEW
POLICY
June
7,
1994
Page
A­
2
Peer
Review
Handbook
June
7,
1994
MEMORANDUM
SUBJECT:
Peer
Review
Program
TO:
Assistant
Administrators
General
Counsel
Inspector
General
Associate
Administrators
Regional
Administrators
Staff
Office
Directors
Today,
I
am
reaffirming
the
central
role
of
peer
review
in
our
efforts
to
ensure
that
EPA
policy
decisions
rest
on
sound,
credible
science
and
data
(
see
attached
policy
statement).
Toward
that
end,
as
its
first
major
task,
EPA s
Science
Policy
Council
(
SPC)
is
instituting
a
program
to
expand
and
improve
peer
review
in
all
EPA
offices.
This
memorandum
gives
an
overview
of
current
practices
and
outlines
the
new
program.

Peer
Review
Practices
and
Policy
Peer
review
at
EPA
takes
several
different
forms,
ranging
from
informal
consultations
with
Agency
colleagues
who
were
not
involved
in
developing
the
product
to
the
formal,
public
processes
of
the
Science
Advisory
Board
(
SAB)
and
the
FIFRA
Scientific
Advisory
Panel
(
SAP).
In
any
form,
peer
review
assists
the
Agency s
work
by
bringing
independent
expert
experience
and
judgment
to
bear
on
issues
before
the
Agency
to
the
benefit
of
the
final
product.

EPA s
Peer
Review
Policy,
which
responds
in
part
to
recommendations
in
the
 
Credible
Science,
Credible
Decisions 
report,
outlines
general
principles
for
peer
review
at
EPA.
Different
EPA
offices
have
undertaken
various
implementing
activities,
including
an
Agency­
wide
information
and
planning
workshop,
internal
guideline
development,
and
numerous
specific
peer
reviews.
Even
with
these
activities,
however,
I
am
concerned
that
EPA
does
not
yet
have
a
comprehensive
Agency­
wide
program
for
implementing
its
Peer
Review
Policy.
I
therefore
welcome
the
SPC
initiative
toward
effective,
efficient
implementation
of
the
policy
in
all
the
program
areas
to
which
it
applies.
Peer
Review
Handbook
Page
A­
3
Expanding
and
Improving
Peer
Review
The
Science
Policy
Council
and
its
Steering
Committee
have
outlined
a
dual­
track
implementation
program
of
planning
and
assistance
for
all
Agency
offices.
The
first
track
has
three
major
milestones.

First,
during
the
next
few
weeks,
Steering
Committee
members
will
consult
with
senior
management
in
each
office
to
exchange
information
on
current
peer
review
activities,
assistance
needed,
possible
obstacles
to
implementation,
and
implementation
planning.

Second,
using
information
and
materials
developed
during
the
first
stage,
peer
review
task
groups
in
each
office
will
develop
standard
operating
procedures
(
SOPs)
for
use
in
each
office,
based
in
part
on
generic
guidance
to
be
issued
by
the
SPC
and
in
part
on
peer
review
needs
and
capabilities
specific
to
each
office.
The
resulting
SOPs
will
delineate
as
appropriate
the
scope
of
application
of
peer
review
with
respect
to
various
types
of
scientific
and
technical
work
products
such
as
reports
of
original
research,
risk
assessments,
and
analytical
methods
of
economic
analysis.
OARM
and
OGC
staff
will
assist
each
office
as
needed
on
legal,
budget
and
administrative
matters.
Each
AA
and
RA
will
submit
draft
SOPs
for
Steering
Committee
review
by
July
15.

Third,
the
SPC
review
group
will
work
with
each
office
to
complete
each
plan
by
September
15.

In
parallel
with
the
above,
consistent
with
the
Peer
Review
Policy,
the
Science
Policy
Council
will
work
with
each
AA
and
RA
to
identify
 
major
scientific
and
technical
work
products 
as
peer
review
candidates
for
the
coming
year.
This
process
will
consider
existing
and
new
plans
for
internal
reviews
and
for
Science
Advisory
Board
(
SAB),
FIFRA
Scientific
Advisory
Panel
(
SAP),
and
other
external
reviews.
The
two­
fold
objective
is
to
plan
reviews
for
technical
products
covered
by
the
Peer
Review
Policy
and
to
gain
experience
with
options
and
obstacles.
We
will
use
this
experience
to
review
and
revise
the
SOPs
as
needed.
Also,
to
establish
a
baseline
for
comparison,
each
AA
and
RA
will
identify
the
 
major
technical
products 
completed
within
his/
her
program
during
the
past
12
months.

The
Science
Policy
Council
has
sent
additional
information
to
each
office
offering
guidance
on
the
procedures
that
you
are
asked
to
develop
and
the
schedule
for
these
activities.
Please
note,
however,
that
because
the
policy
is
effective
immediately,
current
peer
review
planning
should
continue
on
present
schedules
in
parallel
with
developing
the
formal
SOPs.
Page
A­
4
Peer
Review
Handbook
To
begin
this
process,
I
have
asked
each
Assistant
Administrator
and
Regional
Administrator
to
designate
a
Peer
Review
Coordinator
to
work
with
the
Steering
Committee
on
implementation
activities
specific
to
each
office.
I
am
very
pleased
that
the
Science
Policy
Council
is
taking
this
important
step.
A
comprehensive
peer
review
program
is
essential
to
maintaining
and
improving
the
quality
of
the
analyses
that
underlie
Agency
actions.
I
look
forward
to
working
with
you
and
your
staff
on
this
important
activity.

/
s/

Carol
M.
Browner
Attachment
cc:
Science
Policy
Council
Science
Policy
Council
Steering
Committee
Peer
Review
Handbook
Page
A­
5
PEER
REVIEW
AND
PEER
INVOLVEMENT
AT
THE
U.
S.
ENVIRONMENTAL
PROTECTION
AGENCY
This
document
establishes
the
policy
of
the
United
States
Environmental
Protection
Agency
(
EPA)
for
peer
review
of
scientifically
and
technically
based
work
products
that
are
intended
to
support
Agency
decisions.
Peer
review
is
presented
in
the
context
of
the
broader
concept,
peer
involvement.

BACKGROUND
The
report
 
Safeguarding
the
Future:
Credible
Science,
Credible
Decisions 
1
focused
on
the
state
of
science
at
EPA.
The
panel
of
experts
who
prepared
the
report
emphasized
the
importance
of
peer
review,
especially
external
peer
review,
and
the
need
for
broader
and
more
systematic
use
of
it
at
EPA
to
evaluate
scientific
and
technical
work
products.
Their
specific
recommendation
regarding
peer
review
reads
as
follows:

 
Quality
assurance
and
peer
review
should
be
applied
to
the
planning
and
results
of
all
scientific
and
technical
efforts
to
obtain
data
used
for
guidance
and
decisions
at
EPA,
including
such
efforts
in
the
program
and
regional
offices.
Such
a
requirement
is
essential
if
EPA
is
to
be
perceived
as
a
credible,
unbiased
source
of
environmental
and
health
information,
both
in
the
United
States
and
throughout
the
world. 

In
response
to
this
recommendation,
then­
Administrator
Reilly
directed
staff
to
develop
an
EPA­
wide
policy
statement,
which
he
issued
in
January,
1993.
The
paragraphs
below
preserve
the
core
of
that
earlier
statement
while
updating
it
to
specify
the
role
of
the
Science
Policy
Council
in
guiding
further
implementation
of
the
policy.
Effective
use
of
peer
review
is
indispensable
for
fulfilling
the
EPA
mission
and
therefore
deserves
high­
priority
attention
from
program
managers
and
scientists
within
all
pertinent
Headquarters
and
Regional
Offices.

1
EPA/
600/
9­
91/
050,
March
1992.
Page
A­
6
Peer
Review
Handbook
PEER
INVOLVEMENT
AND
PEER
REVIEW
EPA
strives
to
ensure
that
the
scientific
and
technical
underpinnings
of
its
decisions
meet
two
important
criteria:
they
should
be
based
upon
the
best
current
knowledge
from
science,
engineering,
and
other
domains
of
technical
expertise;
and
they
should
be
judged
credible
by
those
who
deal
with
the
Agency.
EPA
staff
therefore
frequently
rely
upon
peer
involvement
­­
that
is,
they
augment
their
capabilities
by
inviting
relevant
subject­
matter
experts
from
outside
the
program
to
become
involved
in
one
or
more
aspects
of
the
development
of
the
work
products
that
support
policies
and
actions.

One
particularly
important
type
of
peer
involvement
occurs
when
scientifically
and
technically
based
work
products
undergo
peer
review
­­
that
is,
when
they
are
evaluated
by
relevant
experts
from
outside
the
program
who
are
peers
of
the
program
staff,
consultants,
and/
or
contractor
personnel
who
prepared
the
product.
Properly
applied,
peer
review
not
only
enriches
the
quality
of
work
products
but
also
adds
a
degree
of
credibility
that
cannot
be
achieved
in
any
other
way.
Further,
peer
review
early
in
the
development
of
work
products
in
some
cases
may
conserve
future
resources
by
steering
the
development
along
the
most
efficacious
course.

Peer
review
generally
takes
one
of
two
forms.
The
review
team
may
consist
primarily
of
relevant
experts
from
within
EPA,
albeit
individuals
who
have
no
other
involvement
with
respect
to
the
work
product
that
is
to
be
evaluated
(
internal
peer
review).
Or
the
review
team
may
consist
primarily
of
independent
experts
from
outside
EPA
(
external
peer
review
).

POLICY
STATEMENT
Major
scientifically
and
technically
based
work
products
related
to
Agency
decisions
normally
should
be
peer­
reviewed.
Agency
managers
within
Headquarters,
Regions,
laboratories,
and
field
components
determine
and
are
accountable
for
the
decision
whether
to
employ
peer
review
in
particular
instances
and,
if
so,
its
character,
scope,
and
timing.
These
decisions
are
made
in
conformance
with
program
goals
and
priorities,
resource
constraints,
and
statutory
or
court­
ordered
deadlines.
For
those
work
products
that
are
intended
to
support
the
most
important
decisions
or
that
have
special
importance
in
their
own
right,
external
peer
review
is
the
procedure
of
choice.
Peer
review
is
not
restricted
to
the
penultimate
version
of
work
products;
in
fact,
peer
review
at
the
planning
stage
can
often
be
extremely
beneficial.

SCOPE
Agency
managers
routinely
make
regulatory
and
other
decisions
that
necessarily
involve
many
different
considerations.
This
policy
applies
to
major
work
products
that
are
primarily
scientific
and
technical
in
nature
and
may
contribute
to
the
basis
for
policy
or
regulatory
decisions.
By
contrast,
this
policy
does
not
apply
to
non­
major
or
nontechnical
matters
that
Agency
managers
consider
as
they
make
decisions.
Similarly,
this
policy
does
not
apply
to
these
ultimate
decisions.
Peer
Review
Handbook
Page
A­
7
This
policy
applies
where
appropriate,
as
determined
by
the
National
and
Regional
Program
Managers,
to
major
scientifically
and
technically
based
work
products
initiated
subsequent
to
the
date
of
issuance.
Peer
review
should
be
employed
to
the
extent
reasonable
to
relevant
work
products
that
currently
are
under
development.
This
policy
does
not
apply
to
the
bases
for
past
decisions,
unless
and
until
the
relevant
scientific
and
technical
issues
are
considered
anew
in
the
Agency s
decision­
making
processes.

Except
where
it
is
required
by
law,
formal
peer
review
(
as
distinguished
from
the
Agency s
normal
internal
review
procedures)
should
be
conducted
in
a
manner
that
will
not
cause
EPA
to
miss
or
need
extension
of
a
statutory
or
court­
ordered
deadline.
Agency
managers
still
may
undertake
peer
review
if
it
can
be
conducted
concurrently
with
necessary
rulemaking
steps.

LEGAL
EFFECT
This
policy
statement
does
not
establish
or
affect
legal
rights
or
obligations.
Rather,
it
confirms
the
importance
of
peer
review
where
appropriate,
outlines
relevant
principles,
and
identifies
factors
Agency
staff
should
consider
in
implementing
the
policy.
On
a
continuing
basis,
Agency
management
is
expected
to
evaluate
the
policy
as
well
as
the
results
of
its
application
throughout
the
Agency
and
undertake
revisions
as
necessary.
Therefore,
the
policy
does
not
stand
alone;
nor
does
it
establish
a
binding
norm
that
is
finally
determinative
of
the
issues
addressed.
Minor
variations
in
its
application
from
one
instance
to
another
are
appropriate
and
expected;
they
thus
are
not
a
legitimate
basis
for
delaying
or
complicating
action
on
otherwise
satisfactory
scientific,
technical,
and
regulatory
products.

Except
where
provided
otherwise
by
law,
peer
review
is
not
a
formal
part
of
or
substitute
for
notice
and
comment
rulemaking
or
adjudicative
procedures.
EPA s
decision
whether
to
conduct
peer
review
in
any
particular
case
is
wholly
within
the
Agency s
discretion.
Similarly,
nothing
in
this
policy
creates
a
legal
requirement
that
EPA
respond
to
peer
reviewers.
However,
to
the
extent
that
EPA
decisions
rely
on
scientific
and
technical
work
products
that
have
been
subjected
to
peer
review,
the
remarks
of
peer
reviewers
should
be
included
in
the
record
for
that
decision.

IMPLEMENTATION
The
Science
Policy
Council
is
responsible
for
overseeing
Agency­
wide
implementation.
Its
responsibilities
include
promoting
consistent
interpretation,
assessing
Agency­
wide
progress,
and
developing
recommendations
for
revisions
of
the
policy
as
necessary.

The
Science
Policy
Council
will
oversee
a
peer­
review
work
group,
which
will
include
representatives
from
program
units
throughout
EPA
to
effect
a
consistent,
workable
implementation
of
the
policy.
The
work
group
will
assist
the
programs
in
(
1)
formulating
and,
as
necessary,
revising
standard
operating
procedures
(
SOPs)
for
peer
review
consistent
with
this
Page
A­
8
Peer
Review
Handbook
policy;
(
2)
identifying
work
products
that
are
subject
to
review;
and
(
3)
for
each
major
work
product,
selecting
an
appropriate
level
and
timing
of
peer
review.

In
assisting
the
programs,
the
work
group
will
take
into
account
statutory
and
court
deadlines,
resource
implications,
and
availability
of
disinterested
peer
reviewers.
The
group
will
work
closely
with
Headquarters
offices
and
the
Regional
Offices
toward
ensuring
effective,
efficient
uses
of
peer
review
in
supporting
their
mission
objectives.
However,
the
Assistant
Administrators
and
Regional
Administrators
remain
ultimately
responsible
for
developing
SOPs,
identifying
work
products
subject
to
peer
review,
determining
the
type
and
timing
of
such
review,
documenting
the
process
and
outcome
of
each
peer
review,
and
otherwise
implementing
the
policy
within
their
organizational
units.

Because
peer
review
can
be
time­
consuming
and
expensive,
Agency
managers
within
Headquarters,
Regions,
laboratories,
and
field
components
are
expected
to
plan
carefully
with
respect
to
its
use
­­
taking
account
of
program
priorities,
resource
considerations,
and
any
other
relevant
constraints
as
well
as
the
policy
goal
of
achieving
high­
quality,
credible
underpinnings
for
decisions.
External
peer
reviewers
should
be
chosen
carefully
to
ensure
an
independent
and
objective
evaluation.
The
affiliations
of
peer
reviewers
should
be
identified
on
the
public
record,
so
as
to
avoid
undercutting
the
credibility
of
the
peer­
review
process
by
conflicts
of
interest.

The
policy
is
effective
immediately.
The
peer­
review
work
group
mentioned
above
will
identify
the
focal
point
to
whom
comments
and
questions
should
be
addressed
and,
from
time
to
time,
will
provide
further
information
about
implementation
activities.

/
s/
DATE:
JUN
7
1994
CAROL
M.
BROWNER
,
ADMINISTRATOR
APPROVED:
Peer
Review
Handbook
Page
B­
1
APPENDIX
B
­
SOUND
SCIENCE
AND
PEER
REVIEW
IN
RULEMAKING
Using
EPA s
regulatory
agenda
as
the
cornerstone
of
the
Agency s
mission,
the
Deputy
Administrator s
memo
implements
an
additional
requirement
as
part
of
the
ongoing
effort
to
use
peer
review
to
enhance
the
quality
and
credibility
of
Agency
decisions.
Page
B­
2
Peer
Review
Handbook
June
15,
1999
MEMORANDUM
SUBJECT:
Sound
Science
and
Peer
Review
in
Rulemaking
TO:
The
Regulatory
Policy
Council
Over
a
year
ago,
the
Agency
enhanced
its
peer
review
policy
and
procedures
with
the
finalization
and
release
of
the
Peer
Review
Handbook.
The
Handbook
was
developed
to
serve
as
a
guide
for
Agency
scientists
and
managers
on
the
organization
and
conduct
of
peer
review.
With
our
regulatory
agenda
as
the
cornerstone
of
the
Agency s
mission,
I
would
now
like
to
implement
an
additional
requirement
as
part
of
the
ongoing
effort
to
use
peer
review
to
enhance
the
quality
and
credibility
of
Agency
decisions.

Beginning
July
12,
1999,
the
action
memorandum
accompanying
each
rule
submitted
for
signature
must
explicitly
address
the
use
of
peer
review
in
that
rulemaking.
The
Administrator
and
I
want
to
ensure
that
the
Agency
takes
complete
advantage
of
the
benefits
that
thorough
peer
reviews
can
provide
in
our
deliberations
on
the
many
challenging
decisions
that
must
be
made.
A
key
step
in
producing
credible
regulatory
decisions
supported
by
well­
documented
scientific
data
is
ensuring
that
our
work
is
based
on
broadly
accepted
scientific
studies.
Peer
review
is
an
essential
tool
for
accomplishing
this
goal.

In
addition,
I
ask
that,
beginning
today,
planning
for
peer
review
be
specifically
addressed
in
each
new
Analytic
Blueprint.
The
general
guidance
on
integrating
peer
reviews
into
the
rulemaking
process
is
included
as
Attachment
A.
Detailed
guidance
on
action
memoranda,
updated
to
address
peer
review,
is
provided
in
Attachment
B.
Guidance
regarding
which
scientific
or
technical
studies
must
be
peer
reviewed
is
contained
in
the
fact
sheet
on
 
Peer
Review
in
the
Rulemaking
Process 
in
Attachment
C
and
in
the
Peer
Review
Handbook
available
at
http://
www.
epa.
gov/
ordntrnt/
ORD/
spc/
sopmenu.
htm.
I
am
also
taking
this
opportunity
to
bring
to
your
attention
a
memorandum
from
the
Office
of
General
Counsel
that
discusses
the
implications
of
a
recent
decision
in
a
challenge
to
a
peer
review
brought
against
the
Agency
under
the
Federal
Advisory
Committee
Act
(
Attachment
D)
[
note
added:
this
attachment
is
not
attached
to
the
Peer
Review
Handbook].

I
appreciate
your
support
in
this
effort.
For
additional
information
about
peer
review
in
the
rulemaking
process,
please
contact
your
Steering
Committee
representative,
your
peer
review
coordinator
(
Attachment
E)
[
note
added:
this
attachment
is
not
attached
to
the
Peer
Review
Handbook],
or
Phil
Schwartz
in
the
Office
of
Policy
at
xxx­
xxxx.

[
signed
6/
15/
99,
AX
9807793]
Peter
D.
Robertson
Acting
Deputy
Administrator
Peer
Review
Handbook
Page
B­
3
ATTACHMENT
A
Regulatory
Management
Guidance
PEER
REVIEW
REQUIREMENTS
Office
of
Policy
FOR
EPA
RULEMAKING
June
1999
Reference:
RMD
5
BACKGROUND
On
February
25,
1998,
the
Agency
reaffirmed
its
commitment
to
quality
decision­
making
by
issuing
the
comprehensive
Peer
Review
Handbook.
The
Peer
Review
Handbook
is
available
at:
http://
www.
epa.
gov/
ordntrnt/
ORD/
spc/
sopmenu.
htm,
which
amplified
and
clarified
key
Agency
peer
review
policy
issues.
The
Handbook
states
that,
in
general,
all
major
scientific
and
technical
work
products
related
to
Agency
decisions
should
normally
be
peer
reviewed.
Peer
review
is
a
documented,
critical
review
of
a
work
product,
performed
by
experts
who
are
independent
of
those
who
developed
the
product.
Work
products
that
are
developed
to
support
regulations,
such
as
risk
assessments,
should
be
evaluated
to
determine
whether
they
meet
the
criteria
for
"
major
scientific
and/
or
technical
work
products"
and
thus
require
peer
review
(
See
Attachment
C).

Regulations
themselves
are
not
peer
reviewed.
The
straight­
forward
application
of
accepted,
previously
peer­
reviewed
economic
methods
or
analyses
in
Regulatory
Impact
Analyses
(
RIA's),
are
not
typically
subject
to
formal
peer
review.
If,
however,
the
particular
facts
and
circumstances
of
any
piece
of
economic
analysis
in
an
RIA
warrant
peer
review,
the
Agency
will
accommodate
those
needs
on
a
case­
by­
case
basis.

PLANNING
FOR
PEER
REVIEW
Remember
that:

 
Analytical
Blueprints
should
discuss
the
plan
for
peer
review
of
major
scientific
and
technical
products.
This
should
include
a
schedule
for
the
review
and
should
identify
the
resources
that
will
be
needed.

 
When
negotiating
rule
deadlines
with
courts
or
with
litigants,
be
sure
that
your
proposals
allow
adequate
time
for
any
needed
peer
reviews.

PEER
REVIEW
REQUIREMENTS
Program
offices
have
the
primary
responsibility
for
complying
with
the
Agency
peer
review
requirements.
Senior
managers
must
confirm
in
their
Action
Memoranda
that
their
offices
have
complied
with
the
peer
review
requirements;
detailed
explanations
are
not
necessary.
If
no
major
scientific
and
technical
product
as
defined
by
the
Peer
Review
Handbook
was
used
to
support
the
action,
include
the
following
statement
in
the
action
memorandum:
Page
B­
4
Peer
Review
Handbook
Peer
Review
There
were
no
major
scientific
or
technical
products
supporting
this
action
as
defined
by
the
Agency's
Peer
Review
Handbook.
We
did
not,
therefore,
submit
any
support
documents
for
peer
review.

If
a
major
scientific
or
technical
product
was
used
to
support
the
action,
include
the
following
statement
in
the
Action
Memorandum:

Peer
Review
[
Insert
Name
of
AAship]
has
complied
with
the
Agency's
Peer
Review
requirements
with
respect
to
the
underlying
major
scientific
and
technical
products
supporting
this
action.

You
may
add
any
details
you
think
important,
but
you
may
not
modify
this
compliance
statement.
If
you
utilized
peer­
reviewed
products,
but
you
could
not
fully
comply
with
the
peer
review
requirements,
explain
in
the
Action
Memorandum
why
the
deviation
was
necessary.

FINAL
AGENCY
REVIEW
(
FAR)

For
rules
that
go
through
Final
Agency
Review,
OP
will
document
compliance
in
the
FAR
memorandum.
This
documentation
will
be
based
on
the
peer
review
information
included
in
the
draft
Action
Memorandum.

These
requirements
will
be
effective
on
July
12,
1999.
After
that
date,
OP's
Office
of
Regulatory
Management
and
Information
(
ORMI)
will
only
forward
for
signature
those
actions
that
include
one
of
the
following:

 
a
statement
that
no
major
scientific
or
technical
documents
were
utilized
to
support
the
rulemaking;

 
a
statement
of
compliance
with
the
peer
review
requirements
for
major
scientific
and
technical
documents;

 
an
explanation
for
deviation
from
the
peer
review
requirements.

Questions?
Contact
your
Regulatory
Steering
Committee
representative
or
Phil
Schwartz
in
the
Regulatory
Management
Division
at
(
202)
260­
5493.
Peer
Review
Handbook
Page
B­
5
ATTACHMENT
B
Regulatory
Management
Guidance
Office
of
Policy
FRAMEWORK
FOR
ACTION
MEMORANDA
Revised
June
1999
Reference
RMD­
6
OVERVIEW
The
overview
provides
background
information
which
briefly
describes
the
rule,
characterizes
the
environmental
issue(
s)
or
public
health
problem(
s)
being
addressed,
and
summarizes
the
history
of
the
regulatory
action.
It
should
explain
why
EPA
is
taking
this
action,
and
where
appropriate,
cover
the
following
points:

 
Define
the
court
or
statutory
deadline;

 
Identify
whether
the
action
amends
the
CFR,
and
if
so,
explain
what
kind
of
amendment
(
technical,
procedural,
etc.);

 
Identify
other
regulatory
actions
underway
that
will
affect
this
particular
program
or
sector;

 
Describe
the
specific
environmental
issue(
s)
or
public
health
problem(
s)
being
addressed,
and
the
goal
intended
by
taking
this
action;

 
Describe
what
the
action
does,
and
specifically,
how
the
regulated
community
is
affected
(
performance
standards,
specific
requirements);
and
 
Describe
how
flexible
the
implementation
of
the
action
will
be
for
states
and
regulated
entities.

REGULATORY
IMPACTS
Summarize
the
costs
and
benefits
of
the
regulatory
action
(
include
costs
to
State
and
local
governments,
and
tribal
communities)
and
the
results
of
the
economic
and
risk
analyses.
Discuss
the
economic
impacts
on
all
stakeholders
including,
where
appropriate,
the
results
of
the
regulatory
flexibility
analysis
and
the
specific
effects
of
the
action
on
small
entities.
Explain
the
roles
of
cost­
benefit
and
cost­
effectiveness
analyses
in
shaping
the
regulatory
approach.

Describe
the
reporting
and
Record
keeping
burden
and
what
the
Agency
has
done
to
reduce
it.
Indicate
possible
impacts
on
other
Federal
agencies.
Page
B­
6
Peer
Review
Handbook
ADMINISTRATOR'S
PRIORITIES
If
appropriate,
discuss
how
the
regulatory
action
relates
to
the
Administrator's
priorities.
Explain
how
the
regulatory
action
relates
to
a
specific
priority
only
where
the
relationship
is
significant,
clear
and
distinct.

STAKEHOLDER
INVOLVEMENT
Briefly
discuss
the
role
of
both
government
entities
and
private
sector
stakeholders
in
the
development
of
the
action.
Summarize
the
concerns
they
have
raised
and
what
the
Agency
has
done
to
address
them.

INTERNAL
REVIEW
Identify
whether
the
regulatory
action
was
developed
under
Tier
1,2,
or
3.
Describe
any
outstanding
issues
from
workgroup
closure
or
other
internal
review.
Identify
program
offices
or
regions
with
outstanding
issues
and
indicate
why
they
cannot
be
resolved
or
accommodated.
Also,
provide
the
basis
for
any
decision
made
to
not
address
an
identified
cross­
media
impact.
Where
there
is
not
a
formal
workgroup,
identify
those
offices
that
have
reviewed
the
package.

PEER
REVIEW
If
you
did
not
utilize
major
scientific
and
technical
products
as
defined
by
the
Peer
Review
Handbook
to
support
the
action,
include
the
following
statement:

There
were
no
major
scientific
or
technical
products
supporting
this
action
as
defined
by
the
Agency's
Peer
Review
Handbook.
We
did
not,
therefore,
submit
any
support
documents
for
peer
review.

If
a
major
scientific
or
technical
product
has
been
used
to
support
the
action,
include
the
following
statement:

[
Insert
Name
of
AAship]
has
followed
the
Agency's
Peer
Review
requirements
with
respect
to
the
underlying,
major
scientific
and
technical
products
supporting
this
action.

You
may
add
any
details
you
think
important,
but
you
may
not
modify
this
compliance
statement.
If
you
utilized
peer­
reviewed
products,
but
you
could
not
fully
comply
with
the
peer
review
requirements,
explain
why
the
deviation
was
necessary.
Peer
Review
Handbook
Page
B­
7
PLAIN
LANGUAGE
Either
in
the
action
memorandum
or
in
a
separate
note
briefly
describe
which
elements
of
plain
language
have
been
addressed.

OMB
TRANSACTION
Highlight
significant
issues
resulting
from
OMB's
review
under
Executive
Order
12866.
Explain
any
changes
made
to
the
regulatory
action
as
a
result
of
that
review.
If
OMB
review
was
waived,
please
indicate
this.

ANTICIPATED
PUBLIC
REACTION
Describe
the
type
of
public
response
anticipated
and
identify
both
the
involved
stakeholders
and
the
nature
of
their
expected
response.
Characterize
the
likely
reaction
to
the
action
by
all
interested
parties
including
industry;
environmental
groups;
Congress;
state,
local,
and
tribal
governments;
and
OMB.
Explain
why
the
Agency
should
take
the
action
despite
any
controversial
response
anticipated
from
the
public.

If
the
regulatory
action
will
not
be
issued
for
public
comment
(
i.
e.,
direct
final
rule,
administrative
stay,
etc.),
explain
the
basis
for
the
Agency's
decision
not
to
solicit
comment.

RECOMMENDATION
Identify
the
action
the
administrator
is
expected
to
take.

NOTE:
An
Action
Memo
should
generally
not
exceed
5
pages.
Page
B­
8
Peer
Review
Handbook
ATTACHMENT
C
Fact
Sheet
Peer
Review
in
the
Rulemaking
Process
What
Role
does
Peer
Review
have
in
the
Regulatory
Development
Process?
Peer
review
of
scientific
and
technical
work
products
affirms
the
scientific
credibility
of
the
Agency's
rulemakings.
Work
products
need
to
be
well
planned
and
documented
because
new
rules,
and
the
work
products
supporting
them,
may
have
substantial
cost
impacts
and
must
often
withstand
intense
public
scrutiny.

What
Work
Products
Need
Peer
Review?
While
rules
themselves
are
not
subject
to
peer
review,
any
major
scientific
and/
or
technical
work
products
that
support
rules
should
be
peer
reviewed.
(
See
Section
2.2
of
the
Peer
Review
Handbook
http://
www.
epa.
gov/
ordntrnt/
ORD/
spc/
sopmenu.
htm
to
determine
which
work
products
are
considered
major.)
Designations
of
a
scientific
and/
or
technical
work
product
as
"
major"
will
largely
be
case­
by­
case.
Those
that
are
used
to
support
a
regulatory
program
or
policy
position
and
that
meet
one
or
more
of
the
following
criteria
are
candidates
for
peer
review:

a)
Establishes
a
significant
precedent,
model,
or
methodology
b)
Addresses
significant
controversial
issues
c)
Focuses
on
significant
emerging
issues
d)
Has
significant
cross­
Agency/
inter­
agency
implications
e)
Involves
a
significant
investment
of
Agency
resources
f)
Considers
an
innovative
approach
for
a
previously
defined
problem/
process/
methodology
The
straight­
forward
application
of
accepted,
previously
peer­
reviewed
economic
methods
or
analyses
in
Regulatory
Impact
Analyses
(
RIA's),
are
not
typically
subject
to
formal
peer
review.
If,
however,
the
particular
facts
and
circumstances
of
any
piece
of
economic
analysis
in
an
RIA
warrant
peer
review,
the
Agency
will
accommodate
those
needs
on
a
case­
by­
case
basis.
The
decision
whether
to
peer
review
any
work
product
will
be
documented
through
the
Agency's
annual
peer
review
reporting
process
(
see
Section
1.3
of
the
Handbook).

When
should
determinations
about
the
need
for
peer
review
be
made?
The
need
for,
and
scheduling
of,
peer
reviews
should
be
addressed
in
the
Analytic
Blueprint
that
is
required
for
all
Tier
1
and
2
regulatory
actions.
Work
products
supporting
Tier
3
rules
may
also
be
considered
major
and
Analytic
Blueprints
are
encouraged
for
Tier
3
rules.
In
any
event,
the
need
for,
and
scheduling
of,
peer
reviews
should
be
addressed
early
in
the
development
of
Tier
3
actions.
Peer
Review
Handbook
Page
C­
1
APPENDIX
C
­
EXAMPLES
OF
CHARGES
Please
note
­­
certain
questions
that
are
posed
in
charges
can
be
responded
to
with
a
yes
or
no
answer.
Clearly,
this
is
not
the
type
of
response
we
generally
want,
therefore,
it
is
important
to
phrase
charge
questions
carefully
to
ensure
that
you
receive
a
fully
satisfactory
and
thoughtful
response.
Where
a
yes
or
no
answer
might
be
expected,
be
sure
to
ask
for
a
full
explanation
supporting
the
yes
or
no
answer.

Charges
can
run
the
gamut
from
rather
simplistic
to
highly
complex.
The
examples
shown
here
cover
a
variety
of
types.
Examples
1
through
6
have
less
complex
questions
and
are
looking
for
the
overall
quality
of
the
reports.
Examples
7
through
11
have
numerous
technical
questions
that
need
to
be
addressed
and
are
therefore
more
complex
in
their
nature.
Page
C­
2
Peer
Review
Handbook
CHARGE
EXAMPLE
1
­
Framework
for
Developing
a
Living
Resources
Research
and
Monitoring
Plan
for
the
Peconic
Estuary
Background
The
Peconic
Estuary
is
located
on
the
eastern
end
of
Long
Island,
New
York.
Under
the
Federal
Clean
Water
Act,
the
Peconic
Estuary
was
named
an
"
Estuary
of
National
Significance"
in
1992.
Because
of
its
high
concentration
of
rare,
threatened
and
endangered
species
and
habitats,
The
Nature
Conservancy
named
the
Peconic
Ecosystem
as
one
of
the
"
Last
Great
Places
in
the
Western
Hemisphere."

The
Peconic
Estuary
Program
(
PEP)
released
its
draft
Comprehensive
Conservation
and
Management
Plan
(
CCMP)
in
September
2,
1999.
The
draft
Plan
addresses
the
environmental
management
issues
facing
the
estuary
and
its
watershed,
including
brown
tide
(
a
nuisance
algal
bloom),
nutrients,
habitats
and
living
resources,
pathogens,
and
toxics,
as
well
post­
CCMP
management
structure,
public
education
and
outreach,
and
financing.
A
final
Plan
is
now
under
preparation.
The
preparation
of
the
Framework
document
is
called
for
in
the
draft
CCMP.

With
population
increasing
in
the
watershed,
the
Peconic
Estuary
is
being
threatened
by
over­
development
and
misuse
of
its
resources.
To
fully
realize
the
impacts
of
people
and
their
activities
on
this
system,
there
must
be
a
better
understanding
of
how
the
Peconic
Estuary
functions
ecologically.
This
knowledge
can
only
be
achieved
through
comprehensive
research
and
monitoring
of
the
entire
ecosystem.

The
long­
term
goal
of
this
Framework
for
Developing
a
Living
Resources
Research
and
Monitoring
Plan
is
to
develop
a
strategy
of
coordinated
research
and
monitoring
to
fill
significant
information
gaps
and
assist
in
the
planning,
conservation,
and
management
of
the
Peconic
Estuary.
A
key
component
of
this
strategy
is
to
develop
an
applied,
multi­
scale,
integrated
approach
to
gain
a
better
understanding
of
the
estuary.
To
achieve,
this
goal,
the
Framework
for
Developing
a
Living
Resources
Research
and
Monitoring
Plan
sets
broad
priorities,
provides
the
context
for
specific
studies,
guidelines
for
a
detailed
living
resources
research
and
monitoring
plan,
and
stimulus
for
funding
agencies
and
organizations
and
researchers.
The
short­
term
goal
of
this
Framework
is
to
provide
a
basis
for
securing
funding
for
the
priority
initiatives
described
in
the
Framework
document.

In
order
of
importance,
the
objectives
of
this
Framework
document
are
to:

1)
understand
threats
so
as
to
improve
resource
protection
through
management
and
conservation;

2)
examine
the
biology
and
ecology
of
particular
organisms
identified
in
the
CCMP
as
important
either
due
to
their
commercial
or
recreational
value
or
their
role
in
the
food
web
and
ecosystem;
and,
Peer
Review
Handbook
Page
C­
3
3)
undertake
system­
wide
studies
of
the
Peconic
Estuary
to
understand
the
significance
of
key
habitats
and
ecosystem
productivity.

Recommendations
included
in
this
framework
are
based
on
information
gaps
identified
in
the
Characterization
Report
of
the
Living
Resources
of
the
Peconic
Estuary
(
Bortman
and
Niedowski
1998),
the
Habitat
Module
of
the
draft
CCMP,
PEP
Natural
Resource
Subcommittee
meetings,
and
recommendations
made
at
a
PEP­
sponsored
Living
Resources
Research
and
Monitoring
workshop
held
in
1998.
Copies
of
the
draft
CCMP
and
Characterization
Report
are
included
with
this
package.

Tasks
The
peer
review
charge
is
to
assess
the
adequacy
of
this
document
to
provide
a
framework
for
integrated,
system­
wide
ecological
research
and
monitoring
to
understand
the
dynamic,
multi­
scale
ecological
patterns
and
processes
that
sustain
biota
and
their
supporting
natural
systems
in
the
Peconic
Estuary.

Specific
Questions:

Peer
Reviewers
shall
answer/
comment
on
the
following:

1)
Is
the
organization
of
the
document
appropriate
and
does
it
present
the
material
in
a
clear
and
concise
manner?
Please
explain
fully.

2)
In
your
opinion,
what
are
the
weakest
and
the
strongest
aspects
of
the
framework?
Please
make
suggestions
on
how
the
weakest
parts
can
be
strengthened.

3)
Are
there
any
elements
missing
from
the
framework
which
you
think
need
to
be
included
or
which
would
strengthen
the
document?
Please
explain
fully.

4)
Are
you
aware
of
any
other
significant
data/
studies
that
are
relevant
and
should
be
included
or
referenced
in
this
document?
Please
explain
fully.

5)
Is
the
stated
goal
realistic?
Are
the
stated
objectives
adequately
met?
Please
explain
fully.

6)
Examine
and
critique
the
interrelationship
among
the
specified
assessment,
research,
and
monitoring
priorities.

7)
Does
this
framework
present
a
holistic
approach
to
preserve,
protect
and
restore
the
estuary?
Does
the
approach
cover
all
the
relevant
and
important
areas
that
warrant
investigation?
If
not,
please
state
what
should
be
done
differently.
Page
C­
4
Peer
Review
Handbook
8)
Are
the
initial
set
of
assessment,
research,
and
monitoring
priorities
adequate
to
provide
the
information
needed
to
successfully
protect
and
manage
this
system?
Please
comment
on
the
selection
of
priorities.
Were
any
significant
priorities
missed?
What
other
priorities
should
be
specified
and
why?
Peer
Review
Handbook
Page
C­
5
CHARGE
EXAMPLE
2
­
EPA
Science
Advisory
Board
(
SAB)
Review
of
the
Environment
Monitoring
and
Assessment
Program
(
EMAP)
Research
Strategy
and
Research
Plan
The
Science
Advisory
Board
(
SAB)
is
asked
to
review
the
Environment
Monitoring
and
Assessment
Program
(
EMAP)
Research
Strategy
and
Research
Plan.
The
review
is
requested
by
the
Office
of
Research
and
Development
(
ORD)
with
the
following
specific
charge
issues:

1)
Previous
peer
reviews
recommended
that
EMAP
develop
a
close
working
relation
with
EPA
Program
Offices
and
other
federal
monitoring
efforts.
Does
the
EMAP
strategy
support
the
[
Office
of
Science
and
Technology
Policy's
Committee
on
Environment
and
Natural
Resources]
CENR
National
Monitoring
Framework
and
EPA
Program
Offices?

2)
Previous
peer
reviews
recommended
that
EMAP
initiate
a
focused
research
program
on
indicator
development.
Does
the
intramural
EMAP
program
on
ecological
indicator
development,
coordinated
with
the
[
ORD
Science
To
Achieve
Results]
STAR
solicitations,
respond
to
this
research
need?

3)
Previous
peer
reviews
recommended
that
the
EMAP
design
be
modified
to
include
a
set
of
nonrandomly
selected
sentinel
sites
with
intensive
data
collection.
Does
the
development
of
Index
Sites
as
outdoor
laboratories
in
the
national
parks
([
National
Park
Service]
NPS
and
[
US
Geological
Survey]
USGS)
and
selected
estuaries
([
National
Oceanic
and
Atmospheric
Administration]
NOAA)
add
this
dimension
to
the
EMAP?

4)
Previous
peer
reviews
recommended
that
EMAP
combine
effects­
oriented
and
stressor­
oriented
monitoring
approaches.
Do
the
focused
geographic
demonstration
pilot
studies
(
initially
in
the
Mid­
Atlantic
region)
combine
these
elements?
Page
C­
6
Peer
Review
Handbook
CHARGE
EXAMPLE
3
­
EPA
Science
Advisory
Board
Review
of
the
Agency's
National
Risk
Management
Research
Laboratory s
(
NRMRL)
Program
The
Office
of
Research
and
Development
(
ORD)
requests
that
the
Science
Advisory
Board
review
the
Agency's
National
Risk
Management
Research
Laboratory s
(
NRMRL)
program.

In
the
 
Strategic
Plan
for
the
Office
of
Research
and
Development 
(
EPA,
1996a),
ORD
described
the
relationship
of
risk
assessment
to
the
risk
management
process,
and
emphasized
the
need
for
scientific
and
engineering
research
to
enable
sound
risk
management
decisions
and
actions.
Within
the
framework
of
that
strategic
plan,
NRMRL s
mission
is
to
conduct
research
to
reduce
uncertainties
and
costs
associated
with
making
and
implementing
environmental
risk
management
decisions.
NRMRL
has
therefore
developed
a
research
agenda
to
reduce
risk
uncertainty
that
also
focuses
on
those
important,
relevant
issues
where
it
can
make
a
difference.

The
charge
to
the
SAB
is
to:

1)
Examine
and
critique
the
research
programmatic
directions
such
as
whether
NRMRL
is
pursuing
the
most
appropriate
research
problem
areas;

2)
Comment
on
strategic
directions,
e.
g.,
use
of
its
core
technical
competencies,
transition
from
primarily
extramural
to
an
intramural
R&
D
organization,
leveraging
with
other
agencies
and
organization;

3)
Review
and
comment
on
the
effectiveness
of
NRMRL s
approach
to
science
management,
e.
g.,
measures
of
success
and
science
quality,
soundness
of
peer
review
process;

4)
Examine
and
critique
the
relationship
of
NRMRL s
risk
management
research
and
its
intended
role
in
the
risk
assessment/
risk
management
paradigm;
and
5)
Review
and
comment
on
the
strategic
balance
for
the
next
decade
among
pollution
prevention,
technology
development,
remediation,
and
risk
management
assessment
activities.
Peer
Review
Handbook
Page
C­
7
CHARGE
EXAMPLE
4
­
EPA
Science
Advisory
Board
(
SAB)
Review
of
the
Technical
Aspects
of
the
Multi­
Agency
Radiation
Survey
and
Site
Investigation
Manual
(
MARSSIM)

The
EPA
Science
Advisory
Board
(
SAB)
is
asked
to
review
the
technical
aspects
of
the
Multi­
Agency
Radiation
Survey
and
Site
Investigation
Manual
(
MARSSIM).
The
review
document
was
developed
collaboratively
by
four
Federal
agencies,
departments
and
commissions
having
authority
for
control
of
radioactive
materials:
Department
of
Defense,
Department
of
Energy,
Environmental
Protection
Agency,
and
Nuclear
Regulatory
Commission.
MARSSIM
addresses
the
need
for
a
nationally
consistent
approach
to
conducting
radiation
surveys
of
potentially
radioactively
contaminated
sites
that
are
being
considered
for
release
to
the
public.
A
condition
of
release
is
a
demonstration
that
residual
radioactivity
levels
do
not
exceed
a
specified
risk
or
dose
level,
also
known
as
a
release
criterion.
MARSSIM
provides
guidance
to
users
performing
and
assessing
the
results
of
such
a
demonstration
for
surface
soils
and
building
surfaces.

The
SAB
is
asked
by
the
Agency s
Office
of
Radiation
and
Indoor
Air
(
ORIA)
to
respond
to
the
following
charge
in
its
review:

1)
Is
the
overall
approach
to
the
planning,
data
acquisition,
data
assessment,
and
data
interpretation
as
described
in
the
MARSSIM
technically
acceptable?
Please
explain
fully.

2)
Are
the
methods
and
assumptions
for
demonstrating
compliance
with
a
dose­
or
risk­
based
regulation
technically
acceptable?
Please
explain
fully.

3)
Are
the
hypotheses
and
statistical
tests
and
their
method
of
application
appropriate?
Please
explain
fully.
Page
C­
8
Peer
Review
Handbook
CHARGE
EXAMPLE
5
­
Economics
­
Benefits
Transfer
from
Adults
to
Children
Background
EPA
established
the
Office
of
Children s
Health
Protection
(
OCHP)
in
1997
to
support
its
efforts
to
increase
the
protection
of
children s
health
throughout
its
programs.
One
of
the
many
difficult
issues
the
Office
is
addressing
is
the
appropriate
treatment
of
children s
health
effects
in
the
economic
analyses
performed
by
the
Agency.
Policy
analysis
efforts
at
the
Agency
often
rely
on
the
benefits
transfer
technique,
and
very
few
of
the
Agency s
benefit
transfers
have
explicitly
addressed
children s
health
issues.
In
addition,
no
accepted
systematic
process
for
conducting
benefits
transfer
currently
exists.
To
assist
the
Agency
in
its
efforts,
this
paper
discusses
the
benefits
transfer
technique
as
it
applies
to
estimating
values
for
children s
health.
The
first
section
provides
some
general
background
on
the
technique,
and
its
application
to
estimate
health­
related
values.
The
second
section
raises
important
general
issues
to
consider
when
conducting
a
benefits
transfer
for
children s
health
values.
The
last
section
discusses
the
implications
of
using
the
benefits
transfer
method
to
estimate
values
for
children s
health.
The
scarcity
and
state
of
existing
child­
oriented
health
valuation
literature
suggests
that
it
may
be
necessary
to
transfer
adult­
oriented
values
to
estimate
child­
related
values
(
Neumann
and
Greenwood
1999).
However,
the
results
of
this
paper
suggest
that
transfer
of
these
value
estimates
to
children
at
best
provides
estimates
for
a
scoping
analysis.
In
cases
where
these
scoping
exercises
indicate
that
children's
health
values
may
be
a
crucial
component
in
the
policy
analysis,
primary
research
should
be
undertaken
to
estimate
child­
related
values.

1)
Assess
the
appropriateness
of
transferring
health
benefit
values
estimated
for
adult
populations
to
children.
Describe
the
specific
issues
that
arise
in
these
transfers.
Which
variables
or
situations
improve
or
decrease
the
appropriateness
of
transferring
benefit
values
from
adults
to
children?

2)
Does
the
analysis
support
the
proposition
that
the
value
of
children s
health
effects
should
be
estimated
differently
than
adult
health
effects?
Why
or
why
not?

3)
Identify
issues
for
further
research
that
would
improve
our
ability
to
estimate
values
for
children s
health
effects.
Peer
Review
Handbook
Page
C­
9
CHARGE
EXAMPLE
6
­
Economics
­
Study
of
Municipal
Government
Costs
and
Financial
Impacts
from
Environmental
Regulations
Background
Municipalities
play
a
major
role
in
supplying
environmental
services.
Local
governments
have
taken
responsibility
for
providing
drinking
water,
sewage
treatment,
and
waste
disposal
in
a
majority
of
communities.
Over
the
past
fifteen
to
twenty
years,
most
of
the
mandates
found
in
the
federal
environmental
legislation
enacted
in
the
early
1970s
have
been
met.
The
increase
in
the
number
of
people
served
and
improvements
in
the
quality
of
local
environmental
services
have
been
considerable,
as
has
the
investment
in
public
infrastructure
to
meet
these
laws.

Recent
revisions
to
the
environmental
legislation
have
established
a
broader
and
more
stringent
set
of
standards
to
be
met
by
suppliers
of
environmental
services.
As
a
result,
many
local
governments
are
now
faced
with
having
to
maintain
all
or
some
part
of
their
public
services
at
a
higher
level
of
performance.
To
meet
these
new
standards
will
require
additional
investments
in
capital,
and
increases
in
rates
charged
to
customers
for
environmental
services.

Improvements
in
environmental
services
are
but
one
of
several
demands
being
made
of
local
public
infrastructure.
Studies
prepared
on
public
infrastructure
needs
and
the
availability
of
funds
to
meet
these
needs
indicate
that
there
will
be
an
excess
demand
for
money
to
rebuild
and
improve
upon
the
existing
stock
of
public
infrastructure.
Therefore,
it
is
important
to
recognize
that
additional
environmental
requirements
will
have
to
compete
with
other
infrastructure
needs
(
e.
g.,
highways,
bridges),
as
well
as
other
public
services
(
e.
g.,
police,
education,
health
and
welfare
programs)
provided
at
the
local
level.

Given
the
increasing
demand
for
public
services,
this
study
examines
what
additional
investments
the
new
environmental
legislation
will
require
local
governments
to
undertake,
and
the
likelihood
that
they
will
face
difficulties
raising
the
necessary
funds
through
capital
markets
and
revenues
from
customers.
The
economic
impacts
of
individual
EPA
actions
are
considered
during
the
regulatory
process
in
those
situations
permitted
by
environmental
statutes.
The
unique
feature
of
this
study
is
its
attempt
to
estimate
the
cumulative
costs
and
impacts
of
meeting
a
combined
set
of
EPA
requirements,
and
to
determine
whether
they
will
place
a
significant
burden
on
the
fiscal
conditions
of
local
governments,
and
require
them
to
significantly
increase
existing
charges
for
improved
environmental
services.

Please
find
attached
a
copy
of
the
draft
study
and
appendices
for
your
review.
This
version
of
the
report
reflects
Agency
comments
received
on
an
earlier
draft.
We
expect
that
this
version
of
the
report,
with
some
additional
minor
modifications,
will
be
the
final
version.
Your
comments
will
be
useful
in
preparing
the
final
version
of
the
report
and
discussing
the
findings
of
the
reports
with
the
public.
Page
C­
10
Peer
Review
Handbook
To
assist
in
your
review
of
the
report,
we
ask
that
you
pay
particular
attention
to
the
following
questions:

1)
Do
state
and
local
governments
and
financial
markets
consider
household
costs
(
measured
as
a
percent
of
household
income)
and
selected
municipal
financial
information
(
debt
service
to
general
revenues
or
taxable
property
values)
when
evaluating
the
ability
of
enterprise
systems
and
municipalities
to
issue
bonds
or
obtain
loans?

2)
Having
selected
a
series
of
financial
indicators,
are
the
criteria
used
in
the
analysis
acceptable?

3)
Do
the
results
support
our
conclusions?
What
additional
conclusions
can
be
reached
from
the
analysis?

4)
What
modifications
would
you
suggest
be
made
to
the
recommendation
section?
Peer
Review
Handbook
Page
C­
11
CHARGE
EXAMPLE
7
­
Economics
­
Valuation
of
Fatal
Cancer
Risks
Background
The
Science
Advisory
Board
 
Environmental
Economics
Advisory
Committee
(
SABEEAC
review
of
the
draft
Guidelines
for
Preparing
Economic
Analyses
(
Guidelines)
helped
the
Agency
to
identify
valid
and
sound
economic
procedures
to
use
when
conducting
benefit­
cost
analyses
of
environmental
policies.
Both
the
Guidelines
and
the
SAB­
EEAC
review
(
EPA­
SAB­
EEAC­
99­
020,
dated
September
1999)
recognize
that
economic
theory
will
evolve
and
new
empirical
literature
will
be
forthcoming
that
may
necessitate
revisiting
the
analytic
procedures
contained
in
the
Guidelines.
There
may
also
be
situations
where
the
Guidelines
do
not
provide
sufficient
detail
to
respond
to
analytic
questions
whose
answers
have
broad
implications
for
the
conduct
of
economic
analysis
at
the
EPA.
Both
circumstances
have
recently
arisen,
making
it
necessary
for
the
Agency
to
submit
for
SAB­
EEAC
review
a
document
that
examines
the
 
benefit
transfer 
issues
that
arise
when
using
the
value
of
statistical
life
(
VSL)
literature
that
is
based
on
accidental
risks
to
estimate
the
economic
benefits
of
environmental
policies
that
reduce
fatal
cancer
risks.

The
Guidelines
provide
information
and
guidance
on
the
valuation
of
reduced
mortality
risks
(
Chapter
7,
pp.
37­
43).
The
Agency
Guidelines
conclude
­
and
we
understand
the
SABEEAC
to
have
concurred
in
their
review
on
this
subject
­
that
one
practical
and
well­
supported
means
to
value
changes
in
mortality
risks
is
to
use
the
Value
of
a
Statistical
Life
(
VSL)
approach.
Further,
in
response
to
the
SAB­
EEAC
review,
the
Guidelines
describe
a
number
of
important
factors
to
consider
in
applying
benefit
transfer
approaches
using
VSL
estimates
from
the
empirical
literature
on
wage­
risk
tradeoffs.
Recognizing
that
this
is
an
important
benefit
category,
the
Agency
Guidelines
stated
that
the
EPA
would
 
continue
to
conduct
annual
reviews
of
the
risk
valuation
literature 
and
 
reconsider
and
revise
the
recommendations
in
these
guidelines
accordingly. 
Furthermore,
the
EPA
would
 
seek
advice
from
the
Science
Advisory
Board
as
guidance
recommendations
are
revised. 

The
Agency
needs
to
return
to
the
SAB­
EEAC
and
obtain
additional
counsel
on
this
subject.
Some
economists
within
the
government
have
suggested
some
particular
approaches
to
dealing
with
the
benefit­
transfer
issues.
Since
the
Guidelines
were
drafted,
a
few
relevant
articles
have
been
published
that
examine
benefit
transfer
issues
surrounding
the
use
of
VSL
estimates
when
there
is
a
passage
of
time
(
or
latency
period)
between
the
pollution
exposure
and
harm,
or
when
fatal
cancer
risks
are
involved.
The
importance
of
these
issues
was
articulated
in
a
recently
proposed
regulation
to
reduce
human
health
risks
from
radon
in
drinking
water.
The
proposed
rule
estimated
the
number
of
reduced
fatal
cancers
resulting
from
different
regulatory
options.
The
Agency
presented
information
on
the
economic
values
for
the
reductions
in
fatal
cancer
risks,
along
with
other
quantified
benefits.
A
brief
discussion
of
some
of
the
benefit
transfer
issues
involved
in
this
estimation
was
published
in
the
preamble
to
the
proposed
rule
for
setting
standards
for
exposure
to
radon
from
drinking
water
sources
(
Federal
Register,
Page
C­
12
Peer
Review
Handbook
November
2,
1999
volume
64,
Number
211,
pages
59245­
59378).
Quoting
from
the
Federal
Register
notice
requesting
SAB
review:

Latency
is
one
of
a
number
of
adjustments
or
factors
that
are
related
to
an
evaluation
of
potential
benefits
associated
with
this
rule,
how
those
benefits
are
calculated,
and
when
those
economic
benefits
occur.
Other
factors
which
may
influence
the
estimate
of
economic
benefits
associated
with
avoided
cancer
fatalities
include
(
1)
a
possible
 
cancer
premium 
(
i.
e.,
the
additional
value
or
sum
that
people
may
be
willing
to
pay
to
avoid
the
experiences
of
dread,
pain
and
suffering,
and
diminished
quality
of
life
associated
with
cancer­
related
illness
and
ultimate
fatality);
(
2)
the
willingness
of
people
to
pay
more
over
time
to
avoid
mortality
risk
as
their
income
rises;
(
3)
a
possible
premium
for
accepting
involuntary
risks
as
opposed
to
voluntary
assumed
risks;
(
4)
the
greater
risk
aversion
of
the
general
population
compared
to
the
workers
in
the
wage­
risk
valuation
studies;
(
5)
 
altruism 
or
the
willingness
of
people
to
pay
more
to
reduce
risk
in
other
sectors
of
the
population;
and
(
6)
a
consideration
of
health
status
and
life
years
remaining
at
the
time
of
premature
mortality.
Use
of
certain
of
these
factors
may
significantly
increase
the
present
value
estimate.
EPA
therefore
believes
that
adjustments
should
be
considered
simultaneously.
The
Agency
also
believes
that
there
is
currently
neither
a
clear
consensus
among
economists
about
how
to
simultaneously
analyze
each
of
these
adjustments
nor
is
there
adequate
empirical
data
to
support
definitive
quantitative
estimates
for
all
potentially
significant
adjustment
factors.
As
a
result,
the
primary
estimates
of
economic
benefits
presented
in
the
analysis
of
this
rule
rely
on
the
unadjusted
$
5.8
million
estimate.
However,
EPA
solicits
comment
on
whether
and
how
to
conduct
these
potential
adjustments
to
economic
benefits
estimates
together
with
any
rationale
or
supporting
data
commenters
wish
to
offer.
Because
of
the
complexity
of
these
issues,
EPA
will
ask
the
Science
Advisory
Board
(
SAB)
to
conduct
a
review
of
these
benefits
transfer
issues
associated
with
economic
valuation
of
adjustments
in
mortality
risks.
In
its
analysis
of
the
final
rule,
EPA
will
attempt
to
develop
and
present
an
analysis
and
estimate
of
the
latency
structure
and
associated
benefits
transfer
issues
outlined
previously
consistent
with
the
recommendations
of
the
SAB
and
subject
to
resolution
of
any
technical
limitations
of
the
data
and
models. 
(
page
59326)

In
the
process
of
responding
to
reviews
prepared
during
deliberations
on
the
proposed
radon
rule,
the
Agency
found
that
the
Guidelines
lack
sufficient
detail
on
how
to
fully
evaluate
and
characterize
the
different
risk
attributes
that
are
central
to
a
complete
understanding
of
the
benefit­
cost
implications
of
this
rule.
For
example,
time
can
pass
between
the
point
of
initial
exposure
to
a
carcinogen,
the
biological
manifestation
or
onset
of
cancer
in
the
body,
the
medical
diagnosis
of
cancer,
and
death
caused
by
the
cancer.
During
development
of
policies
affecting
cancer
risks,
suggestions
have
been
made
to
discount
the
VSL
estimate
(
i.
e.,
$
5.8
million
recommended
in
the
Guidelines)
to
account
for
latencies,
or
the
delay
in
time
between
reduced
exposure
and
when
the
cancer
death
would
have
occurred
absent
the
exposure
reduction.

Others
argued
that
a
suitable
approach
for
valuing
benefits
from
reduced
cancer
risks
must
consider
simultaneously
all
of
the
benefit
transfer
factors
related
to
valuing
cancer
risks
to
ensure
a
careful
and
full
treatment
of
benefits.
There
is
evidence
in
the
economics
literature
Peer
Review
Handbook
Page
C­
13
regarding
many
such
factors
(
e.
g.,
potential
premiums
ascribed
to
cancer
risk
reductions
due
to
a
higher
willingness
to
pay
to
avoid
the
dread,
pain
and
suffering,
morbidity
effects,
and
other
features
of
cancer
endpoints)
that
may
suggest
introducing
upward
adjustments
factors
which
offset
any
potential
downward
adjustments
caused
by
accounting
for
cancer
latency.
In
addition,
proponents
argue
that
adjustments
for
the
age
of
population
at
risk,
income,
altruism
and
other
risk
characteristics
(
e.
g.,
controllability,
voluntariness)
can
all
have
some
potential
influence
on
the
value
of
a
statistical
cancer
fatality
(
VSCF)
and
therefore
need
to
be
reflected
in
the
quantitative
benefit
assessment.

While
developing
the
primary
benefit
estimates
for
reduced
fatal
cancer
risks
in
the
proposed
radon
rule,
questions
arose
regarding
the
implementation
of
adjustments
for
some
factors,
but
not
others.
For
example,
would
it
ever
be
appropriate
to
adjust
only
for
latency
periods,
and
not
other
factors,
in
the
valuation
of
reduced
cancer
deaths?
To
help
answer
this
and
related
questions
regarding
the
valuation
of
cancer
risks,
the
Agency
seeks
the
SAB­
EEAC s
counsel.
We
further
ask
that
your
guidance
reflect
the
typical
uncertainties
facing
EPA
economists,
including
those
surrounding
the
underlying
risk
assessments,
the
prediction
(
or
lack
thereof)
of
latency
periods
for
cancers,
and
the
risk
characteristics
associated
with
the
VSL
approach.

Therefore,
the
Agency
proposes
to
seek
review
of
a
 
white
paper 
and
list
of
charge
questions
by
the
SAB­
EEAC
on
the
valuation
and
benefit
transfer
practices
arising
in
the
calculation
of
the
economic
benefits
of
reduced
fatal
cancer
risks.
The
Agency
seeks
SABEEAC
review
of
the
treatment
and
presentation
of
quantitative
and
qualitative
information
for
these
types
of
benefits.
Numeric
case
studies
are
included
in
the
white
paper,
to
both
identify
and
present
prospective
approaches
to
address
these
issues.
The
results
of
the
SAB­
EEAC
review
of
this
document,
and
responses
to
the
specific
charge
questions,
will
be
considered
by
the
Agency
during
future
revisions
to
the
Guidelines,
consistent
with
the
Agency s
commitment
to
credible
and
consistent
economic
analysis
in
support
of
the
policy
making
process.

Charge
Questions:

As
the
Committee
considers
the
charge
questions,
it
is
asked
to
keep
in
mind
the
differing
situations
relating
to
differing
degrees
of
data
availability
or
uncertainty
in
key
parameters.

1)
Does
the
white
paper
accurately
describe
the
empirical
economic
literature
relevant
to
the
benefit
transfer
issues
that
ensue
when
using
the
VSL
literature
to
estimate
the
VSCF
in
a
benefit­
cost
analysis?

2)
Does
the
white
paper
present
the
important
risk
and
demographic
factors
that
can
affect
benefit
transfer
approaches
that
use
VSL
estimates
for
VSCF?
Page
C­
14
Peer
Review
Handbook
3)
Does
the
white
paper
accurately
describe
attempts
in
the
economic
literature
to
measure
VSCF
directly?

4)
There
are
two
numeric
case
studies
of
environmental
cancer
risks
developed
for
the
white
paper.
Each
presents
risk
assessment
information
that
forms
the
basis
for
quantifying
the
number
of
statistical
cancer
fatalities
that
will
be
reduced
as
a
consequence
of
a
hypothetical
proposed
environmental
policy.
The
case
studies
are
then
used
to
illustrate
the
outcome
of
using
direct
measures
of
the
VSCF
and
benefit
transfer
adjustments
to
VSL
estimates
in
order
to
calculate
the
VSCF.

a)
Which
of
the
valuation
approaches
applied
to
the
case
study
designated
as
ALPHA
are
valid
to
use?
Does
this
case
study
omit
any
credible
alternative
protocols
for
valuing
reductions
in
fatal
cancer
risks
for
benefit­
cost
analyses
of
environmental
programs?

b)
Which
of
the
valuation
approaches
applied
to
the
case
study
designated
as
OMEGA
are
valid
to
use?
Does
this
case
study
omit
any
credible
alternative
protocols
for
valuing
reductions
in
fatal
cancer
risks
for
benefit­
cost
analyses
of
environmental
programs?

5)
Which
economic
methods
illustrated
with
the
case
studies,
or
additional
methods
identified
by
the
Committee
under
charge
questions
4.
a
and
4.
b,
serve
as
credible
protocols
for
the
Agency
to
use
in
representing
quantitative
data,
qualitative
information,
and
sensitivity
analyses
for
the
economic
value
of
reduced
fatal
cancer
risks
reported
in
benefit­
cost
analyses?
Peer
Review
Handbook
Page
C­
15
CHARGE
EXAMPLE
8
­
Hudson
River
PCBs
Site
Reassessment
RI/
FS
Modeling
Approach
Charge:
Peer
Review
1
Members
of
this
peer
review
will
be
tasked
to
determine
whether
the
models
being
used
to
support
the
decision­
making
process
for
the
Reassessment,
and
the
assumptions
therein,
are
appropriate.
The
peer
reviewers
will
base
their
assessment
on
the
review
the
Preliminary
Model
Calibration
Report
(
PMCR),
an
updated
Technical
Scope
of
Work
for
the
Baseline
Modeling
Report
(
Appendix
B
of
the
PMCR)
and
the
responses
to
selected
comments
received
from
stakeholders
during
the
public
comment
period
on
the
PMCR.

In
October
1996,
EPA
released
the
Preliminary
Model
Calibration
Report
(
PMCR),
which
described
the
models,
datasets
and
assumptions
being
used
as
part
of
the
Hudson
River
PCB
Reassessment
RI/
FS.
The
PMCR
represents
the
status
of
the
preliminary
PCB
modeling
effort
as
of
Fall
1995.
Datasets,
database
corrections
and
other
pertinent
information
which
became
available
after
October
1995
were
not
incorporated
within
the
fate
and
transport
modeling
presented
in
the
PMCR.
The
PMCR
was
an
interim
document
prepared
to
describe
work
in
progress
and
was
not
intended
to
be
a
conclusive
report.
In
particular
the
HUDTOX
model
presented
in
the
PMCR
was
not
intended
to
be
used
as
a
predictive
tool
to
assess
remedial
action
scenarios.
In
addition,
while
time­
varying
mechanistic
models
of
bioaccumulation
will
be
used
along
with
other
models
to
predict
fish
body
burdens,
these
models
are
not
described
in
the
PMCR.

The
PMCR
was
not
formally
peer
reviewed
at
the
time
of
publication,
but
was
distributed
to
interested
parties
who
were
invited
to
submit
comments
and
questions.
Written
responses
were
made
to
all
of
these
comments
and
questions.
In
addition,
the
work
plan
contained
in
Appendix
B
of
the
PMCR
has
been
revised
to
reflect
the
ongoing
work
being
conducted
as
part
of
the
Baseline
Modeling
effort.
Results
from
this
effort
will
be
presented
in
a
Baseline
Modeling
Report
that
will
be
formally
peer
reviewed.

The
peer
reviewers
are
requested
to
determine
whether
the
models
being
used
to
support
the
decision­
making
process
for
the
Reassessment
RI/
FS,
and
the
assumptions
therein,
are
appropriate.
The
peer
reviewers
are
not
being
asked
whether
they
would
conduct
the
work
in
the
same
manner,
only
whether
the
work
being
conducted
will
yield
scientifically
credible
conclusions.

It
is
suggested
that
the
reviewer
first
read
the
PMCR.
The
Responses
to
Comments
provides
information
on
the
context
of
the
PMCR
within
the
overall
modeling
effort
and
additional
details
beyond
the
PMCR
results.
The
current
work
plan
as
revised
in
June
1998
reflects
the
ongoing
Baseline
Modeling
effort
and
revisions
to
some
of
the
original
modeling
tasks
proposed
in
Appendix
B
of
the
PMCR.
In
addition,
the
USEPA/
TAMS
Phase
2
database
has
been
considerably
revised.
New
datasets
have
been
added
and
some
earlier
datasets
have
been
extensively
revised.
Page
C­
16
Peer
Review
Handbook
The
peer
reviewers
are
asked
to
comment
on
the
following:

1)
Is
EPA
using
appropriate
models,
datasets
and
assumptions
on
which
to
base
a
scientifically
credible
decision?

2)
Will
the
models,
with
the
associated
datasets
and
assumptions,
be
able
to
answer
the
following
principal
study
questions
as
stated
in
the
PMCR?

a)
When
will
PCB
levels
in
the
fish
population
recover
to
levels
meeting
human
health
and
ecological
risk
criteria
under
No
Action?

b)
Can
remedies
other
than
No
Action
significantly
shorten
the
time
required
to
achieve
acceptable
risk
levels?

c)
Are
there
contaminated
sediments
now
buried
and
effectively
sequestered
from
the
food
chain
which
are
likely
to
become
 
reactivated 
following
a
major
flood,
resulting
in
an
increase
in
contamination
of
the
fish
population?

3)
Specific
questions:

a)
Are
the
modeling
approaches
suitable
for
developing
quantitative
relationships
between
external
forcing
functions
(
e.
g.,
hydraulic
flows,
solids
and
PCB
loads,
sediment
initial
conditions,
etc.)
and
PCB
concentrations
in
the
water
column,
sediments
and
fish?
Are
the
models
adequate
for
discriminating
between
water­
related
and
sediment­
related
sources
of
PCBs?

b)
Are
the
spatial
and
temporal
scales
of
the
modeling
approaches
adequate
to
answer
the
principal
study
questions?
If
not,
what
levels
of
spatial
and
temporal
resolution
are
required
to
answer
these
questions?
What
supporting
data
are
required
for
calibration/
validation
of
these
spatial
and
temporal
scales?

c)
It
is
contemplated
that
PCB
concentrations
in
fish
will
be
estimated
using
several
modeling
approaches:
an
empirical
probabilistic
model
derived
from
Hudson
River
data,
a
steady
state
model
that
takes
into
account
mechanisms
of
bioaccumulation
body
burdens,
and
a
time­
varying
mechanistic
model
(
not
included
in
the
PMCR).
A
bi­
variate
statistical
model
may
also
be
used
to
provide
insight
into
accumulations.
This
multi­
model
approach
is
being
contemplated
because
of
the
uncertainties
associated
with
any
individual
model.
Is
this
a
reasonable
approach
or
should
predictions
be
made
using
a
single
 
best 
model?
Peer
Review
Handbook
Page
C­
17
d)
Is
the
level
of
process
resolution
in
the
models
adequate
to
answer
the
principal
study
questions?
If
not,
what
processes
and
what
levels
of
resolution
are
required
to
answer
these
questions?
What
supporting
data
(
such
as
data
to
support
specifications
of
a
mixed
depth
layer,
solids
and
scour
dynamics,
groundwater
inflow,
etc.)
are
required
for
these
processes
and
levels
of
resolution?

e)
The
results
of
the
modeling
effort
will
be
used,
in
part,
to
support
human
and
ecological
risk
assessments.
In
your
judgment,
will
the
models
provide
estimates
adequate
for
this
purpose?

4)
Are
there
any
changes
to
the
work
effort
outlined
in
the
revised
work
plan
that
would
significantly
improve
the
outcome?

5)
In
terms
of
evaluating
the
overall
and
specific
effects
and
behavior
of
PCBs
in
the
Hudson
River,
are
there
any
serious
flaws
in
the
modeling
approach
(
theory,
structure,
physical
parameters,
etc.)
that
would
limit
or
invalidate
any
conclusions
or
further
work
based
upon
the
results
of
these
models?

Recommendations
Based
on
your
reading
and
analysis
of
the
information
provided,
please
identify
and
submit
an
explanation
of
your
overall
recommendation
for
the
modeling
effort
for
the
Hudson
River
PCB
Reassessment
RI/
FS:

1)
Acceptable
as
is
2)
Acceptable
with
minor
revision
(
as
indicated)
3)
Acceptable
with
major
revision
(
as
outlined)
4)
Not
acceptable
(
under
any
circumstance)
Page
C­
18
Peer
Review
Handbook
CHARGE
EXAMPLE
9
­
IRIS
Pilot
Program
­
Instructions
to
Peer
Reviewers
for
Reviewing
IRIS
Summaries
and
Supporting
Documentation
The
U.
S.
EPA
is
conducting
a
peer
review
of
the
scientific
basis
supporting
the
health
hazard
and
dose
response
assessments
for
the
subject
chemical
that
will
appear
on
the
Agency s
online
database,
the
Integrated
Risk
Information
System
(
IRIS).
Materials
to
be
reviewed
include
the
summary
information
that
will
appear
on
IRIS
(
the
inhalation
reference
concentration
[
RfC],
oral
reference
dose
[
RfD],
and
cancer
assessment)
and
the
supporting
document,
the
Toxicological
Review,
which
will
also
be
made
available
to
the
public.

A
listing
of
Agency
Guidelines
and
Methodologies
that
were
used
in
the
development
of
these
hazard
and
dose­
response
assessments
included
the
following:
The
Risk
Assessment
Guidelines
(
1986),
the
(
new)
Proposed
Guidelines
for
Carcinogen
Risk
Assessment
(
1996),
Guidelines
for
Developmental
Toxicity
Risk
Assessment,
(
proposed)
Interim
Policy
for
Particle
Size
and
Limit
Concentration
Issues
in
Inhalation
Toxicity,
(
proposed)
Guidelines
for
Neurotoxicity
Risk
Assessment,
Methods
for
Derivation
of
Inhalation
Reference
Concentrations
and
Application
of
Inhalation
Dosimetry,
Recommendations
for
and
Documentation
of
Biological
Values
for
Use
in
Risk
Assessment
and
Use
of
the
Benchmark
Dose
Approach
in
Health
Risk
Assessment.
Copies
of
these
documents
(
and/
or
their
relevant
Sections)
will
be
made
to
the
reviewer
upon
request.

Peer
review
is
meant
to
ensure
that
science
is
used
credibly
and
appropriately
in
derivation
of
these
dose­
response
assessments.
You
have
been
chosen
as
an
expert
on
the
chemical
under
consideration,
on
a
scientific
discipline
related
to
at
least
one
of
the
assessments,
or
in
the
field
of
risk
assessment.
At
least
three
peer
reviewers
per
chemical
are
being
chosen
to
review
the
scientific
basis
of
these
draft
dose­
response
assessments
before
they
are
forwarded
on
to
the
EPA's
Consensus
Process
for
final
approval
and
adoption
by
the
EPA.
These
hazard
and
dose­
response
assessments
will
then
appear
on
IRIS
and
become
available
as
Agency
consensus
health
effect
information.

The
primary
function
of
the
peer
reviewer
should
be
to
judge
whether
the
choice,
use,
and
interpretation
of
data
employed
in
the
derivation
of
the
assessments
is
appropriate
and
scientifically
sound.
This
review
is
not
of
the
recommended
Agency
risk
assessment
guidelines
or
methodologies
used
to
derive
cancer
or
RfD/
C
assessments
as
these
have
been
reviewed
by
external
scientific
peers,
the
public,
and
EPA
Science
Advisory
Boards.
The
reviewer s
comments
on
the
application
of
these
guidelines/
methodologies
within
the
individual
assessments
is,
however,
welcomed
and
encouraged.
For
example,
the
reviewer
may
ascertain
whether
or
not
there
is
data
sufficient
to
support
use
of
other
than
default
assumptions
for
areas
such
as
sensitive
subpopulations
or
linear
cancer
extrapolation.
The
reviewer
may
also
have
opinions
on
other
areas
of
uncertainty
such
as
subchronic
to
chronic
duration
(
when
only
a
subchronic
study
is
available)
or
an
incomplete
data
base
but
should
focus
on
the
specific
area
of
uncertainty
rather
than
on
the
magnitude
of
the
overall
estimate.
Peer
Review
Handbook
Page
C­
19
Below
are
two
groups
of
questions
regarding
this
review.
The
first
is
a
set
of
general
questions
that
are
meant
to
guide
you
through
your
review.
It
is
not
imperative
that
you
specifically
answer
each
question
of
this
group.
The
second
group
of
questions,
however,
are
specific
for
the
chemical
assessments
and
deal
with
areas
of
scientific
controversy
or
uncertainty
in
which
the
Agency
may
have
to
make
a
scientific
judgment.
Your
input
to
this
set
of
questions
is
considered
vital
to
the
review
process.

Questions
for
IRIS
Peer
Reviewers
­
General
1)
Are
you
aware
of
any
other
data/
studies
that
are
relevant
(
i.
e.,
useful
for
the
hazard
identification
or
dose­
response
assessment)
for
the
assessment
of
the
adverse
health
effects,
both
cancer
and
noncancer,
of
this
chemical?
Please
explain
fully.

2)
For
the
RfD
and
RfC,
has
the
most
appropriate
critical
effect
been
chosen
(
i.
e.,
that
adverse
effect
appearing
first
in
a
dose­
response
continuum)?
For
the
cancer
assessment,
are
the
tumors
observed
biologically
significant?
relevant
to
human
health?
Points
relevant
to
this
determination
include
whether
or
not
the
choice
follows
from
the
dose­
response
assessment,
whether
the
effect
is
considered
adverse,
and
if
the
effect
(
including
tumors
observed
in
the
cancer
assessment)
and
the
species
in
which
it
is
observed
is
a
valid
model
for
humans.

3)
Have
the
noncancer
and
cancer
assessments
been
based
on
the
most
appropriate
studies?
These
studies
should
present
the
critical
effect/
cancer
(
tumors
or
appropriate
precursor)
in
the
clearest
dose­
response
relationship.
If
not,
what
other
study
(
or
studies)
should
be
chosen
and
why?

4)
Studies
included
in
the
RfD
and
RfC
under
the
heading
"
Supporting/
Additional
studies"
are
meant
to
lend
scientific
justification
for
the
designation
of
critical
effect
by
including
any
relevant
pathogenesis
in
humans,
any
applicable
mechanistic
information,
any
evidence
corroborative
of
the
critical
effect,
or
to
establish
the
comprehensiveness
of
the
data
base
with
respect
to
various
endpoints
(
such
as
reproductive/
developmental
toxicity
studies).
Should
other
studies
be
included
under
the
"
Supporting/
Additional"
category?
Should
some
studies
be
removed?

5)
For
the
noncancer
assessments,
are
there
other
data
that
should
be
considered
in
developing
the
uncertainty
factors
or
the
modifying
factor?
Do
you
consider
that
the
data
support
use
of
different
(
default)
values
than
those
proposed?

6)
Do
the
Confidence
statements
and
weight­
of­
evidence
statements
present
a
clear
rationale
and
accurately
reflect
the
utility
of
the
studies
chosen,
the
relevancy
of
the
effects
(
cancer
and
noncancer)
to
humans,
and
the
comprehensiveness
of
the
Page
C­
20
Peer
Review
Handbook
data
base?
Do
these
statements
make
sufficiently
apparent
all
the
underlying
assumptions
and
limitations
of
these
assessments?
If
not,
what
needs
to
be
added?

Questions
for
IRIS
Peer
Reviewers
­
Chemical
Specific
[
example:
cumene]

1)
Based
on
the
information
noted
in
the
Principal
study
currently
designated
(
Cushman
et
al.,
1995)
is
the
discounting
of
the
renal
effects
in
males
justified?
Is
sufficient
rationale
given
to
let
stand
the
organ
weight
changes
in
female
rats
as
a
critical
effect?

2)
Is
the
information
in
the
Toxicological
Review
sufficient
to
consider
cumene
as
having
a
low
potential
for
causing
reproductive
effects?
Please
explain
fully.

RECOMMENDATIONS
Based
on
your
reading
and
analysis
of
the
information
provided,
please
identify
your
overall
recommendation
for
the
IRIS
materials
you
have
reviewed
as
1)
Acceptable
as
is
2)
Acceptable
with
minor
revision
(
as
indicated)
3)
Acceptable
with
major
revision
(
as
outlined)
4)
Not
acceptable
Peer
Review
Handbook
Page
C­
21
CHARGE
EXAMPLE
10
­
Charge
to
Reviewers
for
the
WTI
Draft
Final
Risk
Assessment
The
draft
final
WTI
risk
assessment
is
divided
into
several
volumes
covering
the
scientific
disciplines
of
toxicology,
environmental
fate
and
transport,
combustion
engineering,
atmospheric
modeling,
exposure
assessment,
ecological
risk
assessment,
and
accident
analysis.
As
a
reviewer
of
the
WTI
draft
final
risk
assessment,
you
should
use
your
best
technical
knowledge
and
professional
judgment
to
comment
on
the
technical
accuracy,
completeness
and
scientific
soundness
of
the
assessment.
Each
reviewer
is
asked
to
focus
on
several
specific
issues
in
his
or
her
area
of
expertise
with
comments
on
other
areas
invited
but
optional.
Your
comments
will
be
considered
in
finalizing
the
risk
assessment.

For
the
peer
review
workshop
reviewers
will
be
organized
into
5
work
groups:
Combustion
Engineering,
Air
Dispersion
and
Deposition
Modeling
and
Accident
Analysis,
Toxicology,
Exposure
Assessment,
and
Ecological
Risk
Assessment.
All
reviewers
should
be
familiar
with
the
Executive
Summary
(
Volume
I)
and
the
Facility
Background
(
Volume
II)
Sections
of
the
draft
risk
assessment.
In
addition,
each
work
group
should
focus
on
specific
Volumes
as
specified
below:

Workgroup
Risk
Assessment
Volumes
Combustion
Engineering
Volume
III
­
Facility
Emissions
Air
Dispersion
and
Volume
IV
­
Atmospheric
Volume
VII
­
Accident
Deposition
Modeling
and
Dispersion
and
Deposition
Analysis
Accident
Analysis
Modeling
Toxicology
Volume
V
­
Human
Health
Risk
Assessment
Volume
VII
­
Accident
Analysis
Exposure
Assessment
Volume
V
­
Human
Health
Risk
Assessment
Volume
VII
­
Accident
Analysis
Ecological
Risk
Assessment
Volume
VI
­
Screening
Ecological
Risk
Assessment
While
reviewing
these
Sections
of
the
document,
please
address
the
following
general
issues.
Page
C­
22
Peer
Review
Handbook
1)
Comment
on
the
organization
of
the
risk
assessment
document.
Does
the
layout
follow
a
logical
format?
Is
the
presentation
of
information
in
the
document
clear,
concise
and
easy
to
follow?

2)
Does
the
executive
summary
accurately
reflect
the
data
and
methodologies
used
and
the
conclusions
derived
in
the
risk
assessment?

3)
Were
the
major
recommendations
of
the
1993
peer
review
workshop
for
the
risk
assessment
plan
addressed?

4)
As
with
any
risk
assessment,
there
are
always
additional
data
and
method
development
efforts
that
could
be
undertaken
to
reduce
the
level
of
uncertainty.
However,
are
there
any
major
data
or
methodological
gaps
that
would
preclude
the
use
of
this
risk
assessment
for
decision
making?
If
so,
how
should
they
be
addressed?

5)
What
long­
term
research
would
you
recommend
that
could
improve
risk
assessments
of
this
type
in
the
future?

In
addition,
the
following
workgroup
specific
issues
should
be
addressed.

Emissions
Characterization
Emissions
characterization
includes
identification
of
substances
of
concern
and
the
development
of
emission
rates
for
these
contaminants.
Emission
rates
were
developed
through
a
combination
of
site
specific
stack
test
data
and
models.
Please
comment
on
the
following
issues
with
respect
to
this
aspect
of
the
draft
risk
assessment.

1)
To
characterize
the
nature
of
the
emissions,
waste
stream
profiles
were
developed
and
entered
into
a
database.
Several
refinements
and
adjustments
(
e.
g.,
the
Subtraction
Correction
Factor
for
chlorinated
compounds)
were
applied
to
the
profiles
before
substances
of
concern
were
identified.
Please
comment
on
whether
or
not
these
adjustments
are
appropriate.
What
is
the
anticipated
effect
on
the
risk
assessment?

2)
Comment
on
the
selected
chemicals
of
concern.
Have
important
chemicals
been
missed
due
to
the
selection
technique?

3)
Comment
on
the
approaches
used
to
estimate
stack
emission
rates
(
e.
g.,
use
of
the
95%
UCL
of
the
arithmetic
mean
or
the
maximum
detected
value,
whichever
is
smaller
,
for
high
end
emission
rates).
Are
the
approaches
appropriate?
Are
their
effects
on
the
risk
assessment
adequately
characterized?
Comment
on
the
Peer
Review
Handbook
Page
C­
23
adjustment
made
to
PCDD/
PCDF
emission
rates
to
account
for
brominated
dioxin­
like
compounds.
Also,
comment
on
the
approach
to
characterizing
emission
rates
from
fugitive
sources
(
e.
g.,
use
of
the
TANKS
2
model
for
the
Carbon
Adsorption
Bed).

4)
Comment
on
the
identified
sources
of
fugitive
emissions.
Was
the
approach
used
to
select
these
sources
appropriate?
Have
important
sources
been
missed?
Have
emissions
from
process
upsets
been
given
appropriate
consideration?

5)
There
have
been
a
number
of
controlled
burns
at
the
WTI
facility.
Please
comment
on
the
adequacy
of
these
data
in
estimating
potential
exposure.
Please
comment
on
the
assumptions
made
from
the
tests
in
regard
to
composition
of
wastes
received
at
WTI
and
emissions
when
the
plant
operates
in
the
future.

6)
Comment
on
the
use
of
emission
factors
from
coal
burning
to
estimate
the
emission
rate
of
fly
ash
from
WTI.
Are
the
factors
used
to
adjust
the
coal
emission
rate
appropriate?
Are
the
uncertainties
introduced
from
this
approach
adequately
characterized?

7)
Overall,
is
the
identification
of
the
key
assumptions
used
in
characterizing
the
nature
and
magnitude
of
emissions
thorough?
Are
the
magnitude
and
direction
of
effect
of
these
assumptions
on
the
overall
risk
assessment
accurately
characterized?
Is
the
uncertainty
and
variability
inherent
in
this
analysis
adequately
discussed?
Does
the
sensitivity
analysis
cover
the
major
parameters
expected
to
have
an
effect
on
the
risk
assessment?

Dispersion
and
Deposition
Modeling
To
develop
this
risk
assessment,
computer
models
have
been
used
with
site
specific
data
on
emission
rates
and
meteorological
conditions
to
simulate
the
air
concentrations
and
deposition
rates
for
contaminants
potentially
emitted
from
the
WTI
facility.
The
models
used
include
the
Industrial
Source
Complex
­
Complex
Terrain
Deposition
(
ISC­
COMPDEP),
the
CALPUFF,
and
the
INPUFF
models.
In
your
review,
please
address
the
following
issues.

1)
Since
the
1993
peer
review
of
the
risk
assessment
plan,
a
number
of
efforts
have
been
completed
to
reduce
the
uncertainty
associated
with
the
air
dispersion
and
deposition
modeling.
These
efforts
include
the
collection
of
site­
specific
data
for
emission
rates
and
meteorological
conditions.
Also,
a
wind
tunnel
study
was
conducted
to
evaluate
the
effects
of
the
complex
terrain
surrounding
the
WTI
facility.
Does
the
risk
assessment
document
adequately
summarize
these
activities?
Is
the
link
between
these
data
collection
efforts,
the
air
dispersion
models,
and
the
risk
assessment
clearly
established?
Page
C­
24
Peer
Review
Handbook
2)
The
results
of
12
sets
of
sensitivity
tests
indicate
that
geophysical
variables
(
e.
g.,
terrain)
are
more
likely
to
affect
dispersion
and
deposition
than
emission
variables
(
e.
g.,
stack
temperature).
Were
these
sensitivity
analyses
adequate?
Comment
on
the
conclusions
reached.
To
further
examine
the
effect
of
geophysical
variables,
wind
tunnel
testing
was
conducted
to
model
the
terrain
induced
flow
effects
expected
near
WTI.
It
was
concluded
that
changes
in
peak
concentrations
attributed
to
these
effects
are
relatively
minor
and
that
the
ISCCOMPDEP
model
is
sufficiently
conservative.
Comment
on
this
conclusion.
Have
these
analyses
helped
to
characterize
and/
or
reduce
the
uncertainty
in
the
air
dispersion
modeling
associated
with
the
complex
terrain
surrounding
WTI.

3)
The
ISC­
COMPDEP
model
does
not
allow
for
non­
steady
state
conditions
such
as
calm
winds
and
strong
temperature
inversions.
Therefore,
CALPUFF
was
used
to
estimate
air
dispersion
and
deposition
under
these
conditions.
However,
CALPUFF
gave
similar
peak,
24
hour,
and
annual
average
concentrations
as
ISCCOMPDEP
Comment
on
the
adequacy
of
this
analysis.
Comment
on
the
conclusions
reached.
Has
this
analysis
helped
to
characterize
and/
or
reduce
the
uncertainty
in
the
air
dispersion
modeling
associated
with
non­
steady
state
meteorological
conditions?

4)
Atmospheric
dispersion
modeling
was
used
to
estimate
air
concentrations
of
hazardous
chemicals
for
the
accident
analysis.
The
SLAB
model
was
used
for
vapor
releases
from
spills
and
the
mixing
of
incompatible
wastes.
ISCCOMPDEP
was
used
for
releases
associated
with
fires.
Comment
on
the
selection
of
the
models
and
inputs.
Are
they
appropriate
selections?

5)
Overall,
have
adequate
sensitivity
tests
been
conducted
to
demonstrate
the
magnitude
of
variation
in
concentrations
and
deposition
estimates
with
model
inputs?
Please
explain
fully.

Human
Health
Risks
Human
Health
Risk
Assessment
includes
hazard
identification,
dose­
response
evaluation,
exposure
assessment,
and
risk
characterization.
To
develop
the
risk
assessment,
potentially
exposed
populations
have
been
identified
and
the
magnitude,
frequency,
and
duration
of
their
exposure
quantified.
This
information
was
then
integrated
with
the
hazard
identification
and
dose
response
evaluation
for
the
risk
characterization.
For
this
risk
assessment,
both
carcinogenic
and
non­
carcinogenic
health
effects
have
been
evaluated.
In
your
review,
please
comment
on
the
following
issues.
Peer
Review
Handbook
Page
C­
25
Exposure
1)
EPA's
Exposure
Assessment
Guidelines
identify
certain
exposure
descriptors
that
should
be
used
to
characterize
exposure
estimates.
The
Guidelines
define
high
end
exposure
estimates
as
those
representing
individuals
above
the
90th
percentile
on
the
exposure
distribution
but
not
higher
than
the
individual
in
the
population
who
has
the
highest
exposure.
Bounding
exposure
estimates
are
those
that
are
higher
than
the
exposure
incurred
by
the
person
in
the
population
with
the
highest
exposure.
Central
tendency
exposure
estimates
are
defined
as
the
best
representation
of
the
center
of
the
exposure
distribution
(
e.
g.,
arithmetic
mean
for
normal
distributions).
Comment
on
whether
or
not
the
WTI
exposure
assessment
properly
characterizes
each
of
the
exposure
estimates
in
terms
of
these
descriptors.

2)
The
factors
that
go
into
estimating
a
central
tendency
or
high
end
exposure,
once
the
population
has
been
defined,
include
the
environmental
media
concentration,
the
intake
rate,
and
the
duration
and/
or
frequency
of
exposure.
Comment
on
whether
or
not
the
WTI
exposure
assessment
does
an
adequate
job
of
describing
the
logical
procedure
of
combining
these
factors
to
develop
central
tendency,
high
end,
and/
or
bounding
estimates
of
exposure
for
each
of
the
exposed
subpopulations.

3)
An
important
factor
in
an
exposure
assessment
is
identifying
all
of
the
important
exposure
sources.
Please
comment
on
the
adequacy
of
the
WTI
assessment
in
identifying
the
important
sources
and
pathways
of
exposure.

4)
Have
the
key
assumptions
for
estimation
of
chemical
concentration
and
for
estimation
of
exposure
been
identified?
Are
the
magnitude
and
direction
of
effect
correct
for
the
assumptions
that
have
been
identified?

5)
Supposedly,
conservative
assumptions
have
been
applied
in
this
assessment
to
account
for
uncertainty.
Are
the
conservative
assumptions
appropriately
factored
into
the
ultimate
characterization
of
what
descriptor
best
applies
to
each
exposure
estimate?
Please
comment
on
whether
the
uncertainties
were
confronted
in
an
adequate
manner.
If
they
were
not,
please
state
what
should
be
done
differently.

Hazard
Identification/
Dose
Response
and
Risk
Characterization
1)
To
select
surrogate
compounds
for
quantitative
risk
assessment,
a
two
step
process
was
used
in
which
chemicals
were
ranked
on
the
basis
of
emission
rate,
toxicity
(
both
cancer
and
non­
cancer),
and
bioaccumulation
potential.
Please
comment
on
this
selection
process.
Are
the
ranking
factors
appropriate?
Could
Page
C­
26
Peer
Review
Handbook
important
compounds
have
been
omitted
from
the
analysis
based
on
the
ranking
procedure?

2)
For
the
majority
of
the
chemicals
of
concern,
traditional
approaches
to
dose
response
evaluation
were
employed
(
e.
g.,
use
of
a
slope
factor
for
cancer
and
use
of
a
RfD/
RfC
for
non­
cancer).
However
for
certain
chemicals
or
groups
of
compounds
a
different
methodology
was
used.
Specifically,
dioxins,
furans,
PAHs,
lead,
mercury,
nickel,
chromium,
acid
gases,
and
particulate
matter
were
given
special
consideration.
Please
comment
on
the
methodology
used
for
these
compounds.
Was
it
appropriate?
Have
the
uncertainties
associated
with
the
methodology
been
adequately
characterized?
Comment
on
the
assumptions
used
due
to
a
lack
of
chemical
specific
data.

3)
Please
comment
on
the
selection
of
the
overall
population
and
the
various
subpopulations
at
risk.
Were
site
specific
data,
such
as
the
informal
home
gardening
survey,
properly
utilized
to
identify
these
subpopulations?

4)
It
is
stated
in
the
risk
assessment
that
average
risk
estimates
are
based
on
average
emission
rates,
average
air
dispersion/
deposition
within
a
subarea,
and
typical
exposure
factors.
Further,
maximum
risks
are
based
on
average
emission
rates,
typical
exposure
factors,
and
the
maximum
air
concentration
within
a
subarea.
Please
comment
on
this
use
of
the
terms
average
and
maximum
risks.
Are
these
descriptive
terms
appropriate
given
the
parameters
used
to
derive
each?
Please
explain
fully.

5)
Comment
on
whether
or
not
the
non­
cancer
risks
of
chemicals
of
concern
have
been
adequately
addressed
by
the
risk
assessment?
For
example,
has
an
adequate
discussion
of
endocrine
disrupters
been
provided
which
either
characterizes
their
risks
or
clearly
explains
why
their
risks
cannot
be
characterized?
Further,
have
non­
cancer
chronic
toxicities
of
dioxins
and
furans
been
adequately
addressed
in
the
risk
assessment?

6)
Please
comment
on
whether
or
not
the
uncertainties
associated
with
the
additivity
and/
or
synergy
of
risks
from
pollutants
emitted
together
from
the
WTI
facility
are
adequately
discussed
in
the
risk
assessment.

7)
Have
the
key
assumptions
for
estimation
of
dose
and
risk
been
identified?
Are
the
magnitude
and
direction
of
effect
correct
for
the
assumptions
that
have
been
identified?
Please
comment
on
whether
the
uncertainties
were
confronted
in
an
adequate
manner.
If
they
were
not,
please
state
what
should
be
done
differently.
Peer
Review
Handbook
Page
C­
27
8)
Please
comment
on
the
overall
adequacy
of
the
risk
characterization.
Does
the
risk
characterization
include
a
statement
of
confidence
in
the
risk
assessment
including
a
discussion
of
the
major
uncertainties.
Are
the
hazard
identification,
dose­
response
assessment,
and
exposure
assessment
clearly
presented?
Have
sufficient
risk
descriptors
which
include
important
subgroups
been
presented
and
discussed?

Screening
Ecological
Risk
Assessment
As
with
the
human
health
risk
assessment,
the
ecological
risk
assessment
pulls
together
elements
of
exposure
analysis
and
dose­
response
evaluations
to
develop
a
risk
characterization.
For
the
Screening
level
Ecological
Risk
Assessment
(
SERA),
Ecological
Chemicals
of
Concern
(
ECOC)
and
indicator
species
have
been
identified
to
provide
conservative
estimates
of
risk.
Please
address
the
following
issues
in
your
review.

1)
Are
there
any
components
of
the
SERA
which
you
feel
undermine
the
scientific
validity
of
the
assessment?
If
so,
what
are
they
and
can
you
provide
suggestions
to
strengthen
the
identified
components?

2)
Is
the
organization
of
the
document
clear
and
does
it
present
the
material
in
a
clear
and
concise
manner
consistent
with
the
Framework
for
Ecological
Risk
Assessment
(
EPA,
1992)?
Please
explain
fully.

3)
Uncertainties
are
discussed
in
numerous
sections
of
the
SERA
and
compose
Section
VIII
of
the
SERA.
In
each
case,
do
these
discussions
cover
all
relevant
and
important
aspects
of
the
uncertainties
which
you
think
should
be
addressed
in
the
SERA?

4)
In
your
opinion,
what
is
the
weakest
and
what
is
the
strongest
aspect
of
the
SERA?
Can
you
make
any
suggestions
on
how
the
weakest
parts
can
be
strengthened
by
the
Agency?

5)
In
Section
II,
are
the
stressors,
ecological
effects,
and
both
the
assessment
and
measurement
endpoints
adequately
characterized?
Are
the
five
emission
scenarios
adequate
to
characterize
the
exposures
for
the
WTI
facility?
Are
there
other
emission
scenarios
which
you
think
should
be
included
in
the
SERA?

6)
In
Section
III,
is
the
site
characterization
adequate
to
support
the
SERA?
Why
or
why
not?

7)
In
Section
IV,
is
the
tiered
process
used
to
identify
the
ecological
chemicals
of
concern
(
ECOC)
from
the
initial
list
of
potential
chemicals
considered
Page
C­
28
Peer
Review
Handbook
scientifically
defensible?
Does
application
of
this
tiered
approach
support
the
statement
made
in
the
SERA
"
by
focusing
on
the
potential
risk
from
the
selected
ECOCs,
the
SERA
provides
a
thorough
screening­
level
evaluation
for
the
WTI
facility?"

8)
In
Sections
V
and
VI,
are
the
exposure
and
ecological
effects
adequately
characterized?
Are
the
most
appropriate
estimation
techniques
available
used?
Are
the
assumptions
clearly
stated?
Please
explain
fully.

9)
In
Section
VIII,
are
there
any
major
elements
missing
from
the
risk
characterization
which
you
think
need
to
be
included
or
which
would
strengthen
the
risk
characterization?
Does
the
risk
characterization
support
the
summary
and
conclusions
presented
in
Section
IX?

10)
In
Section
IX,
given
the
assumptions
made
and
the
processes
used
to
select
and
evaluate
chemicals,
receptors,
and
exposure
pathways,
do
you
think
the
SERA
adequately
met
its
objective
of
not
inadvertently
underestimating
risk?

Accident
Analysis
The
Accident
Analysis
for
the
WTI
incinerator
involves
evaluating
the
probability
of
an
emergency
incident
occurring
which
results
in
the
release
of
hazardous
waste.
The
consequences
of
this
release
are
also
evaluated
using
exposure
and
human
health
effects
information.
Unlike
the
human
health
risk
assessment
which
has
a
primary
goal
of
quantifying
risks,
the
accident
analysis
typically
provides
information
that
can
be
used
to
reduce
the
likelihood,
extent
and
impact
of
possible
accidents.
Please
comment
on
the
following
issues
in
your
review
of
this
aspect
of
the
risk
assessment.

1)
The
WTI
accident
assessment
selected
five
scenarios
for
quantitative
evaluation
that
were
considered
to
be
of
primary
concern.
The
scenarios
are
an
on­
site
spill,
an
on­
site
fire,
an
on­
site
mixing
of
incompatible
waste,
an
off­
site
spill,
and
an
off­
site
spill
and
fire.
Please
comment
on
the
selection
of
these
scenarios.
Were
any
significant
scenarios
missed?

2)
Specific
chemicals
were
selected
to
evaluate
each
scenario.
Please
comment
on
the
selections.
Would
other
chemicals
have
been
more
appropriate?

3)
Chemical
specific
release
rates
are
calculated
for
each
scenario.
Please
comment
on
the
procedures
used
to
estimate
the
release
rates.
Was
an
appropriate
approach
used?
Peer
Review
Handbook
Page
C­
29
4)
Atmospheric
dispersion
modeling
was
used
to
estimate
air
concentrations
of
hazardous
chemicals.
Specifically,
the
SLAB
model
was
used
for
vapor
releases
from
spills
and
the
mixing
of
incompatible
wastes.
ISC­
COMPDEP
was
used
for
releases
associated
with
fires.
Comment
on
the
selection
of
the
models
and
in­
puts.
Are
they
appropriate
selections?
Should
other
models
or
inputs
been
used?

5)
Please
comment
on
the
assessment's
conclusions
on
the
severity
of
consequences
and
probability
of
occurrence.
Has
the
report
correctly
categorized
the
severity
of
the
consequences
of
the
different
accident
scenarios?
Has
the
assessment
adequately
justified
the
reported
probability
of
occurrence
of
each
of
the
accident
events?

6)
Key
assumptions
were
made
in
the
identification
of
accident
scenarios
and
the
description
of
the
conservative
and
typical
events.
Included
were
a
description
of
the
magnitude
of
the
effect
of
the
assumptions
and
direction
of
the
effect.
Please
comment
on
the
assumptions.
Are
they
justified?
Are
the
descriptions
of
the
magnitude
and
directions
of
the
effects
correct?
Has
the
accident
assessment
adequately
confronted
the
uncertainties
involved
in
doing
this
type
of
analysis?
If
not,
what
else
should
be
done?

7)
Comment
on
the
appropriateness
of
using
IDLH
values
for
characterizing
the
severity
of
consequences
in
the
accident
analysis.
Comment
on
the
appropriateness
of
using
10
X
LOC
for
chemicals
for
which
IDLH
values
have
not
been
established.

8)
In
the
accident
analysis,
IDLH
(
or
10
X
LOC)
values
were
used
to
determine
the
downwind
distances
over
which
adverse
human
health
effects
might
occur.
To
evaluate
the
uncertainty
introduced
by
using
the
IDLH,
a
sensitivity
analysis
was
conducted
where
these
distances
were
recalculated
using
the
LOC
(
a
more
stringent
health
criteria).
Other
sources
of
uncertainty
that
are
identified
in
the
accident
analysis
include
concentration
averaging
times,
chemical
concentrations,
emission
rates,
and
meteorological
conditions.
For
most
of
these
parameters
it
is
stated
that
conservative
assumptions
were
used
to
avoid
underestimating
risks.
Have
the
uncertainties
inherent
in
the
accident
analysis
been
adequately
characterized?
For
those
parameters
where
sensitivity
analyses
were
not
conducted,
is
the
conclusion
that
conservative
assumptions
have
avoided
underestimation
valid?
Peer
Review
Handbook
Page
D­
1
APPENDIX
D
­
GUIDANCE
ON
REQUESTING
A
REVIEW
BY
THE
US
EPA
SCIENCE
ADVISORY
BOARD
(
SAB)
Page
D­
2
Peer
Review
Handbook
Guidance
on
Requesting
a
Review
By
the
US
EPA
Science
Advisory
Board
(
SAB)

Summary
The
US
EPA
Science
Advisory
Board
(
SAB
or
the
 
Board )
annually
solicits
proposals
for
review
projects
every
spring
for
the
following
fiscal
year.
This
appendix
provides
guidance
to
Programs
and
Regions
to
help
them
submit
requests
for
SAB
reviews.
Requests
should
be
submitted
to
the
SAB
in
both
hard
copy
and
electronic
versions
usually
by
mid­
June
for
the
following
fiscal
year.
The
requests
may
be
part
of
the
annual
submissions
that
respond
to
the
peer
review
activities
of
the
Agency,
or
they
may
be
submitted
directly
to
the
Board.
Although
providing
requests
at
one
defined
time
in
the
spring
helps
with
SAB
planning,
we
recognize
that
projects
also
come
to
light
during
other
parts
of
the
year.
Please
contact
the
SAB
staff
for
details
on
making
submissions
during
the
remainder
of
the
year
(
see
end
of
this
document
for
contacts).

Background
A
key
priority
for
the
Administrator
is
to
base
Agency
actions
on
sound
scientific
data,
analyses,
and
interpretations.
The
Administrator
issued
the
Agency's
Peer
Review
Policy
to
increase
the
quality
of
the
technical
foundations
upon
which
EPA's
regulatory
structures
are
built.
The
SAB
is
a
key
scientific
peer
review
mechanism
available
to
Programs
and
Regions
in
implementing
the
Peer
Review
Policy.
However,
because
the
Board
has
finite
resources
it
cannot
conduct
all
reviews.
This
document
is
designed
to
help
Programs
and
Regions
determine
which
projects
to
submit
to
the
SAB.
Note
particularly
that
the
SAB
focuses
on
the
technical
underpinnings
of
Agency
positions;
i.
e.,
risk
assessment
issues,
in
contrast
to
risk
management
issues.

The
topics
that
are
best
suited
for
the
Board's
agenda
are
those
that
satisfy
several
of
the
following
criteria:

1)
Integrate
science
into
Agency
actions
in
new
ways.
2)
Influence
long­
term
technological
developments.
3)
Impact
overall
environmental
protection.
4)
Address
novel
scientific
problems
or
principles.
5)
Address
problems
that
transcend
federal­
agency
or
other
organizational
boundaries.
6)
Strengthen
the
Agency's
basic
capabilities.
7)
Serve
Congressional
or
other
leadership
interests.
8)
Deal
with
controversial
issues.

In
suggesting
issues
for
SAB
involvement,
Programs
and
Regions
should
note
the
breadth
of
SAB
activities:
Peer
Review
Handbook
Page
D­
3
Historically,
most
of
the
outputs
of
the
Board
are
in
the
form
of
"
full"
reports.
They
present
the
findings
of
peer
reviews
of
Agency
document(
s)
and
contain
considerable
detail
about
the
findings
and
recommendations
of
the
Board.
They
also
address
the
specific
questions
posed
by
the
Charge
to
the
Board.
"
Letter"
reports
fulfill
the
same
function
as
reports,
but
are
simply
shorter
in
length.
Due
to
the
need
to
be
more
responsive
with
advice,
the
Board
has
recently
begun
to
produce
more
short
letter
reports
than
full
reports,
as
they
can
be
produced
and
finalized
in
less
time.

The
SAB
has
also
introduced
the
"
Consultation 
as
a
means
of
conferring­­
in
public
session­­
with
the
Agency
on
a
technical
matter
before
the
Agency
has
begun
substantive
work
on
that
issue.
The
goal
is
to
leaven
EPA's
thinking
on
an
issue
by
brainstorming
a
variety
of
approaches
to
the
problem
very
early
in
the
development
process.
There
is
no
attempt
or
intent
to
express
an
SAB
consensus
or
to
generate
an
SAB
report.
The
Board,
via
a
brief
letter
simply
notifies
the
Administrator
that
a
Consultation
has
taken
place.

More
recently,
the
Board
introduced
a
new
vehicle
for
communicating
with
its
clients
­­
the
"
Advisory 
­­
which
provides,
via
a
formal
SAB
consensus
report,
critical
input
on
technical
issues
that
arise
during
the
Agency's
issue
development
process.
The
Advisory
generally
involves
a
review
of
a
multi­
year
Agency
project.
The
intent
is
to
provide
some
mid­
course
assessment
to
see
if
the
Agency
is
heading
in
a
scientifically
credible
direction.
In
order
to
maintain
an
objective,
arms­
length
relation
with
the
Agency
and
its
projects,
the
SAB
review
of
the
final
product
at
some
point
in
the
future
will
include
experts
who
did
not
participate
in
producing
the
Advisory.

The
Agenda
Setting
Process
Each
Assistant
Administrator
and
Regional
Administrator
is
normally
asked
to
submit
a
list
of
candidate
topics
for
SAB
action/
review.
A
"
project
sheet"
(
see
attached
example)
is
used
to
define
each
topic
that
is
nominated
for
SAB
review.
The
project
sheet
is
prepared
by
the
requesting
office
and
contains
the
following
information:

1)
Project
title/
subject
(
Descriptive
short
title
of
project)

2)
Requesting
Organization/
Office
(
Primary
office
requesting
review,
AA/
RA
level)

3)
Requesting
Official
(
Name
and
position
of
senior
official
requesting
review,
usually
office
or
division
level
­
this
is
the
person
who
may
receive
a
summary
briefing
from
the
Chair
following
the
review)

4)
Program
Contact
(
Name/
phone
number/
mail
code
­
this
is
the
principal
contact
for
SAB
Staff
to
interact
with
during
development
of
the
SAB
review)
Page
D­
4
Peer
Review
Handbook
5)
Background
(
brief
history
of
the
project
and
why
it
is
important)

6)
Tentative
Charge
(
what
the
SAB
is
being
asked
to
comment
on
­
usually
a
set
of
questions)

7)
Tentative
Schedule
and
Committee
(
when
the
review
is
expected
to
be
conducted,
e.
g.,
Winter
2001;
and
which
SAB
committee
is
appropriate
for
the
review
­­
final
choice
as
to
the
review
committee
is
at
the
discretion
of
the
Board)

8)
Budget
Estimate
(
Rough
estimates
of
Agency
funding
for
the
subject
over
the
past
5
years
(
if
applicable)
and
for
the
next
2
years
(
if
applicable).
The
SAB
Executive
Committee
has
asked
for
this
information
to
help
it
better
appreciate
the
level
of
Agency
involvement
in
and
commitment
to
the
issue)

9)
Preparer
(
name,
phone,
office
of
preparer
of
Project
Sheet
and
the
date
prepared)

The
proposed
topics
will
be
examined
and
discussed
in
a
number
of
forums:

1)
The
individual
SAB
Committees
­
Throughout
the
late
Spring
and
Summer,
the
SAB
Committees
will
be
examining
options
for
the
following
fiscal
year,
including
all
suggestions
made
by
the
Agency.

2)
The
Science
Policy
Council­
Steering
Committee
(
SPC­
SC)
­
The
SPC­
SC
usually
meets
in
early
summer
to
examine
the
proposals
for
each
fiscal
year.
The
goal
is
to
provide
cross­
office
critique/
integration
of
the
proposals.
The
SPC­
SC
will
be
used
as
a
forum
for
continuing
discussion
throughout
the
process
and
throughout
the
year
as
new
topics
emerge.

3)
The
Deputy's
perspective
­
In
the
summer,
the
Deputy
Administrator
will
review
the
requests
and
provide
insights
on
priorities.

4)
The
SAB
Executive
Committee
­
During
its
summer
meeting,
the
SAB s
EC
will
examine
the
nominated
topics,
adding
its
own
perspective
on
an
appropriate
agenda,
using
its
selection
criteria.

5)
The
Administrator
­
In
September,
the
list
of
proposed
topics
will
be
delivered
to
the
Administrator
for
information
and
added
insights.

The
completed
project
sheets
should
be
submitted
electronically
to
the
SAB
Deputy
Staff
Director
(
fowle.
jack@
epa.
gov)
and
in
signed
hard
copy
(
mail
code
1400A).
Peer
Review
Handbook
Page
D­
5
For
general
information,
or
to
contact
a
specific
SAB
Staff
member,
please
call
the
SAB
main
phone
line:
(
202)
564­
4533.
Please
ask
to
be
connected
to
the
Staff
person
handling
the
major
review
area
you
wish
to
inquire
about
(
e.
g.,
human
health
and
exposure,
drinking
water/
water
quality,
ecology,
engineering,
modeling,
air
quality,
economics,
research,
and
radiation).
Page
D­
6
Peer
Review
Handbook
US
EPA
Science
Advisory
Board
Proposed
Project
Project
title/
subject:
Proposed
Amendments
to
the
Risk
Assessment
Guidelines
for
Carcinogens
Requesting
Organization/
Office:
Office
of
Research
and
Development
(
ORD)

Requesting
Official:
Name,
Title,
Office/
Organization
Program
Contact:
Name,
Title,
Office/
Organization,
202­
260­
xxxx
Background:
EPA's
Health
Risk
Assessment
Guidelines
provide
generic
science
and
science
policy
guidance
on
risk
assessment
issues
for
use
in
all
Agency
offices.
EPA
has
currently
issued
or
proposed
nine
guidelines
(
or
amendments)
in
this
series,
all
of
which
have
been
submitted
to
the
Science
Advisory
Board
for
review.

The
current
guidelines
for
carcinogen
risk
assessment
were
reviewed
by
the
SAB
and
issued
as
final
guidance
in
1986.
In
1988,
the
Forum
initiated
a
public
process
for
considering
amendments
to
these
guidelines.
A
Risk
Assessment
Technical
Panel
considered
submissions
from
the
public
as
well
as
information
developed
by
experts
at
two
public
workshops
in
revising
these
guidelines.

Tentative
Charge:
Review
the
amended
and
expanded
guidance,
with
special
emphasis
on
(
a)
weight­
of­
evidence
issues,
(
b)
a
new
classification
system,
(
c)
dose
response
modeling,
and
(
d)
the
use
of
pharmacokinetic
and
metabolic
data.
A
more
detailed
charge
will
be
negotiated
with
SAB
at
a
later
date.

Tentative
Schedule
and
Committee:
Winter,
2001,
Environmental
Health
Committee
Budget:
FY
1997
­
$
xxx
and
yy
FTE
FY
1998
­
$
xxx
and
yy
FTE
FY
1999
­
$
xxx
and
yy
FTE
FY
2000
­
$
xxx
and
yy
FTE
FY
2001
­
estimated
costs
of
$
xxx
and
yy
FTE
each
year
Preparer:
Name,
Title,
Office/
Organization,
202­
260­
xxxx
Date:
June
1,
2000
Peer
Review
Handbook
Page
E­
1
APPENDIX
E
­
EXAMPLE
STATEMENTS
OF
WORK
FOR
CONTRACTS
Users
of
this
Peer
Review
Handbook
need
to
be
aware
that
the
examples
contained
in
this
Appendix
are
generalized
statements
of
work
prepared
(
and
in
some
cases
modified)
to
emphasize
certain
important
features
(
e.
g.,
attention
to
conflicts
of
interest,
responsibilities
of
contractors
or
contracting
officers,
development
and
use
of
the
charge
to
peer
reviewers).
Please
be
sure
that
proper
and
currently
approved
contract
language
is
used
in
any
EPA
contract
document
at
the
time
of
award
(
for
example,
see
EPAAR
1552.212­
71,
alternate
I).
Page
E­
2
Peer
Review
Handbook
STATEMENT
OF
WORK
­
EXAMPLE
1
­
Statement
of
Work:
Technical
Review
Contractor
for
Panel
Review
of
Assistance
Agreement
or
Fellowship
Applications
1)
Purpose
The
purpose
of
this
contract
is
to
purchase
peer
review
services
of
a
contractor
with
expertise
in
Exploratory
Research;
Environmental
Chemistry.
The
services
are
for
peer
reviewing
applications
received
in
response
to
the
Office
of
Research
and
Development's
(
ORD's)
1997
Science
to
Achieve
Results
program.
These
reviews
shall
be
completed
and
the
evaluation
sheets
shall
be
prepared
prior
to
the
reviewer's
participation
in
a
3­
day
panel
discussion
to
be
held
in
Washington,
D.
C.
on
May
5
­
7,
1997
.

2)
Statement
of
Work
ORD's
National
Center
for
Environmental
Research
and
Quality
Assurance
(
NCERQA)
is
responsible
for
overseeing
the
recently
expanded
research
grants
and
fellowships
programs.
Each
year
NCERQA
(
alone
or
in
conjunction
with
other
organizations)
solicits
applications
in
each
of
these
programs.
The
applications
to
be
reviewed
under
this
contract
were
submitted
in
response
to
the
solicitation
for
the
1997
Science
to
Achieve
Results
program.
As
part
of
the
selection
process,
NCERQA
must
conduct
a
peer
review
that
is
designed
to
evaluate
the
scientific
quality
of
each
application;
this
is
accomplished
through
the
ad
hoc
use
of
technical
experts.

The
peer
review
services
required
by
this
contract
necessitate
the
independent
review
of
a
maximum
of
10
applications
and
the
preparation
of
a
typed
evaluation
summary
and
an
overall
rating
for
each
of
these
applications.
Each
evaluation
summary
shall
support
and
be
consistent
with
the
overall
rating
that
is
assigned;
it
also
shall
be
completed
prior
to
the
contractor s
participation
in
the
panel
discussions.
After
the
panel
discussions
for
the
applications
assigned
to
the
contractor,
the
contractor
shall
submit
all
completed
evaluation
summaries
to
the
designated
Science
Review
Administrator
(
SRA).

The
contractor
also
shall
serve
as
the
panel's
rapporteur
for
approximately
6
­
8
of
the
applications
assigned.
As
rapporteur,
the
contractor
shall
be
responsible
for
preparing
a
typed
evaluation
summary
(
on­
site
typing
support
will
be
provided
by
NCERQA)
that
reflects
the
panel's
discussion
of
the
respective
application
as
well
as
the
panel's
overall
rating
(
the
criteria
for
the
panel's
overall
rating
are
the
same
as
those
for
each
peer
reviewer's
overall
rating).
As
rapporteur,
the
contractor
shall
submit
these
panel
evaluation
summaries
to
the
designated
SRA
prior
to
leaving
the
panel
meeting.
Peer
Review
Handbook
Page
E­
3
Before
the
contractor
shall
be
allowed
to
participate
in
the
review
process,
the
contractor
shall
have
disclosed
any
actual
or
potential
conflicts
of
interest
and
shall
have
signed
and
submitted
to
NCERQA
a
Conflict
of
Interest/
Confidentiality
Form.
The
contractor
is
directed
to
assure
that
none
of
the
conflicts
disclosed
are
so
direct
and
substantial
as
to
rule
out
a
particular
reviewer.
Upon
receipt
of
an
approved
Purchase
Order
(
PO),
NCERQA
will
send
the
following
items
to
the
contractor:

a)
A
copy
of
the
Purchase
Order
or
the
Purchase
Order
number
b)
The
applications
assigned
to
the
contractor
c)
For
grants,
a
set
of
abstracts
for
all
the
applications
being
reviewed
by
the
panel
d)
For
persons
reviewing
grant
applications,
a
copy
of
the
pertinent
section(
s)
of
the
solicitation
package
to
provide
background
information;
for
persons
reviewing
fellowship
applications,
information
on
how
to
access
the
solicitation
package
on
the
Internet
e)
A
sample
evaluation
form
to
help
the
contractor
prepare
an
acceptable
evaluation
form
for
each
assigned
application
f)
A
blank
evaluation
form
for
each
assigned
application
and
the
criteria
for
completing
the
form
and
determining
the
overall
rating
g)
A
blank
and
sample
invoice
as
well
as
instructions
for
completing
and
submitting
the
invoice
to
EPA
h)
Information
on
the
points
of
contact
for
additional
information
(
e.
g.,
NCERQA's
SRA)
i)
Logistics
information
on
the
location
and
time
of
the
panel
discussions
NCERQA
will
transmit
the
above
items
under
a
cover
letter.
In
this
cover
letter,
NCERQA
will
provide
additional
details
about
each
item,
including
(
as
needed)
more
specific
instructions
for
the
set
of
applications
assigned
to
the
reviewer.

Each
contractor
shall
be
responsible
for
making
his/
her
own
travel
reservations
for
hotel
and
transportation.

3)
Reviewer
Tasks
a)
Review
the
assigned
applications
using
the
guidance
provided
with
NCERQA's
evaluation
form.

b)
Submit
completed
evaluation
forms
to
the
SRA
designated
in
the
cover
letter
immediately
following
the
panel
discussions
for
the
applications
assigned
to
the
contractor.
THE
COMPLETED
FORMS
MUST
BE
TYPED,
AND
THE
EVALUATION
SUMMARY
FOR
EACH
APPLICATION
MUST
SUPPORT
AND
BE
CONSISTENT
WITH
THE
OVERALL
RATING
THAT
IS
ASSIGNED
BY
THE
CONTRACTOR.
IN
SITUATIONS
Page
E­
4
Peer
Review
Handbook
WHERE
THESE
CONDITIONS
ARE
NOT
MET,
THE
SRA
WILL
ASK
THE
CONTRACTOR
TO
REDO
THE
FORM.

c)
For
those
applications
for
which
the
contractor
is
serving
as
the
panel's
rapporteur,
submit
a
completed
panel
evaluation
summary
to
the
SRA
designated
in
the
cover
letter
prior
to
leaving
the
panel
meeting.
THE
PANEL'S
EVALUATION
SUMMARY
MUST
BE
TYPED
(
ON­
SITE
TYPING
SUPPORT
WILL
BE
PROVIDED
BY
NCERQA)
AND
BE
CONSISTENT
WITH
THE
PANEL'S
OVERALL
RATING.
IN
SITUATIONS
WHERE
THESE
CONDITIONS
ARE
NOT
MET,
THE
SRA
WILL
ASK
THE
CONTRACTOR
TO
REDO
THE
FORM.

d)
Make
own
airline
and
hotel
accommodations
for
participation
in
the
panel
review
meeting.
Round­
trip
air
fare
must
be
a
commercial
REFUNDABLE
ticket.
Peer
Review
Handbook
Page
E­
5
STATEMENT
OF
WORK
­
EXAMPLE
2
­
Peer
Review
of
Prioritization
Tool
Report
Work
Assignment
No.:

Title:
Peer
Review
of
Prioritization
Tool
Report
Work
Assignment
Manager
(
WAM):

Name:
John
Q.
Government
Employee
Address:
Office
of
Solid
Waste
Phone
No.:
(
202)
260­
XXXX
Background:

The
Waste
Minimization
Branch
(
WMB)
in
the
Office
of
Solid
Waste
(
OSW)
is
in
the
process
of
implementing
the
Waste
Minimization
National
Plan,
announced
by
the
Agency
on
November
18,
1994.
The
Plan
reaffirms
the
Agency's
commitment
to
promote
source
reduction
over
waste
management,
in
keeping
with
the
policy
stated
in
the
1984
amendments
to
the
Resource,
Conservation,
and
Recovery
Act
(
RCRA)
and
in
the
1990
Pollution
Prevention
Act
(
PPA).
The
Plan
outlines
major
goals,
objectives,
and
action
items
to
achieve
national
reductions
in
the
generation
of
hazardous
wastes.

One
of
the
objectives
of
the
Plan
is
to:
"
develop
a
framework
for
setting
national
priorities;
develop
and
distribute
a
flexible
screening
tool
for
identifying
priorities
at
individual
facilities;
[
and]
identify
constituents
of
concern."
This
objective
is
a
key
building
block
in
implementing
subsequent
objectives
of
the
Plan.

In
September
1995,
WMB
formed
the
Waste
Minimization
Prioritization
Team,
which
includes
representatives
from
EPA
regions
and
states,
to
implement
this
objective.
The
Team
has
worked
to
assess
stakeholder
needs
for
prioritization
tools
and
to
evaluate
prioritization
tools
that
are
currently
available.
The
Team
plans
to
summarize
this
work,
along
with
its
recommendations,
in
a
report
(
referred
to
herein
as
the
Prioritization
Tool
report)
that
would
be
available
in
draft
form
in
July
1996.

WMB
and
the
Team
wish
to
obtain
independent
peer
review
of
the
Prioritization
Tool
report
prior
to
briefing
EPA
management.
The
report
is
being
prepared
with
the
support
of
ICF,
Inc.;
therefore,
for
the
peer
review
to
be
considered
independent,
it
must
be
performed
by
another
contractor.
Page
E­
6
Peer
Review
Handbook
Purpose
and
Scope
of
Work:

The
purpose
of
this
work
assignment
is
to
provide
support
to
WMB
and
the
Team
in
finalizing
the
Prioritization
Tool
report
by
conducting
an
independent
peer
review
of
the
report.

Work
Statement:

Task
1
­
Management
work
plan
and
budget
Within
15
days
of
CO
approval
of
this
work
assignment,
the
contractor
shall
deliver
a
management
work
plan
including
a
proposed
level
of
effort,
schedule,
and
budget
for
all
tasks.

Task
2
­
Provide
independent
peer
review
of
Prioritization
Tool
report
The
contractor
shall
provide
support
to
WMB
and
the
Team
in
preparing
the
Prioritization
Tool
report
by
performing
an
independent
peer
review
of
the
report.
The
contractor
shall
establish
a
panel
of
peer
reviewers
including
three
senior­
level
persons
who
collectively
have
extensive
expertise
in
particular
areas
to
be
identified
by
the
WAM
upon
approval
of
the
work
assignment.

Within
three
weeks
of
work
assignment
approval,
receipt
from
the
WAM
of
the
necessary
qualifications
of
peer
reviewers
(
in
a
TD),
and
receipt
from
the
WAM
of
the
peer
review
 
charge 
(
in
a
TD),
whichever
comes
latest,
the
contractor
shall
identify
the
three
peer
reviewers
and
prepare
a
memo
that
lists
the
names
of
the
peer
reviewers
and
their
affiliations
and
includes
the
peer
reviewers 
bio s.
Within
five
weeks
of
WAM
approval
of
the
of
the
peer
reviewers
(
via
a
TD)
and
receipt
of
the
draft
Prioritization
Tool
report
from
the
WAM
(
via
a
TD),
whichever
comes
later,
the
contractor
shall
conduct
the
peer
review,
assemble
the
peer
review
comments
and
recommendations
in
a
peer
review
report
organized
by
charge
question,
prepare
an
introduction
to
the
peer
review
report
with
a
clear
and
concise
overview
of
the
comments,
and
attach
to
the
peer
review
report
any
marginal
comments
the
peer
reviewers
had
on
the
Prioritization
Tool
report.

It
is
not
necessary
that
the
peer
reviewers
jointly
reach
consensus
on
their
findings
and
recommendations,
since
there
may
be
limited
overlap
in
the
peer
reviewers'
areas
of
expertise
and
in
the
charge
questions
that
they
focus
on.
The
contractor
shall
assume,
for
the
purpose
of
estimating
costs,
that
the
draft
Prioritization
Tool
report
is
roughly
100
pages
in
length
with
200
pages
of
appendices,
and
that
each
peer
reviewer
will
spend
40
hours
in
reviewing
the
report
and
writing
comments.
EPA
plans
to
provide
the
report
to
the
contractor
in
mid­
July.
Peer
Review
Handbook
Page
E­
7
Deliverables
and
Schedule:

Task
Deliverable
Schedule
1
Work
plan
and
budget
Within
15
days
of
CO
ap
proval
of
work
assignment
2
Memo
identifying
peer
reviewers
Within
3
weeks
of
wo
rk
assignment
approval,
receipt
of
peer
reviewer
qualifications
from
WAM,
and
receipt
of
charge
from
WAM,
whichever
comes
latest
3
Peer
review
report
Within
five
weeks
of
WAM
approval
of
peer
reviewers
and
receipt
of
draft
Prioritization
Tool
report
from
WAM,
whichever
comes
later
Other
Requirements:

CONTRACTOR
COMMUNICATIONS
Upon
approval
of
the
Work
Plan,
the
contractor
shall
maintain
at
least
weekly
communications
with
the
Work
Assignment
Manager
regarding
the
status
of
work
on
the
Work
Assignment.

CONFLICT
OF
INTEREST
(
COI)
[
Revised
Section]

The
contractor
must
adhere
to
the
following
requirements:

a)
Upon
receipt
of
a
Work
Assignment,
QRT,
or
similar
tasking
document,
and
prior
to
commencement
of
any
work,
notify
both
the
CO
and
PO
of
any
actual
or
potential
organizational
or
personal
conflicts
of
interest.

b)
Provide
a
written
certification,
within
20
days
of
receipt
of
a
Work
Assignment,
QRT,
or
similar
tasking
document,
that:

1)
Either
all
conflicts
of
interest
have
been
reported
to
the
CO
or
that
no
conflicts
of
interest
exist.
The
contractor
is
directed
to
assure
that
none
of
the
conflicts
disclosed
are
so
direct
and
substantial
as
to
rule
out
a
particular
reviewer.

2)
All
personnel
who
perform
work
under
this
Work
Assignment
or
relating
to
this
Work
Assignment
have
been
informed
of
their
obligation
to
report
personal
and
organizational
conflicts
of
interest
to
the
CO.
Page
E­
8
Peer
Review
Handbook
3)
The
Contractor
recognizes
its
continuing
obligation
to
identify
and
report
any
conflicts
of
interest
arising
during
performance
of
this
Work
Assignment.

c)
If
a
conflict
of
interest
is
identified
during
performance
under
this
Work
Assignment,
the
Contractor
shall
immediately
make
a
full
disclosure
in
writing
to
the
CO.
The
disclosure
shall
include
a
description
of
action
which
the
Contractor
has
taken
or
proposes
to
take,
after
consultation
with
the
CO,
to
avoid,
mitigate,
or
neutralize
the
conflict
of
interest.

d)
After
selecting
the
peer
reviewers
but
before
starting
the
peer
review,
submit
to
EPA
documentation
that
shows
that
the
contractor
has
determined
if
the
peer
review
candidates:

1)
have
a
conflict
of
interest
or
a
situation
that
could
create
the
appearance
of
a
lack
of
impartiality
in
relation
to
the
work
product
2)
have
had
or
presently
have
a
financial
relationship
with
EPA
and
summarizes
for
EPA
its
efforts
to
identify
and
propose
resolution
of
these
concerns
with
peer
review
candidates.

EXPENDITURE
OF
FUNDS/
HOURS
In
addition
to
the
requirements
of
the
contract,
the
contractor
shall
notify
both
the
Project
Officer
and
the
Work
Assignment
Manager
when
75%
of
funds
or
hours
for
this
Work
Assignment
have
been
expended.

INFORMATION
COLLECTION
Any
other
provision
of
this
Work
Assignment
notwithstanding,
the
contractor
shall
not
proceed
with
any
information
collection
where
the
same
or
similar
information
will
be
collected
from
ten
or
more
public
respondents
until
written
approval
is
received
from
the
Contracting
Officer.
This
approval
will
cite
an
approval
number
from
the
Office
of
Management
and
Budget
as
required
by
the
Paperwork
Reduction
Act
(
PRA).

Only
Federal
agencies
and
their
employees
are
exempt
from
the
PRA
definition
of
"
public
respondent."
State
agencies
and
their
employees
are
classified
as
"
public
respondents."

Soliciting
similar
information
applies
to
any
collection
method,
i.
e.,
written,
oral,
electronic,
etc.,
and
utilizing
any
approach,
i.
e.,
surveys,
phone
calls,
focus
groups,
TQM,
etc.
The
PRA
applies
equally
to
"
willing
participants"
and
participation
that
is
mandated
by
law.
Peer
Review
Handbook
Page
E­
9
Any
question
of
applicability
of
the
PRA
shall
be
resolved
by
submitting
a
complete
description
of
the
circumstances
in
a
written
request
to
the
Contracting
Officer.
No
collection
shall
be
undertaken
until
the
Contracting
Officer
provides
written
notice
to
the
Contractor
as
to
the
applicability
of
the
PRA.
If
the
PRA
is
determined
to
be
applicable,
the
Contractor
shall
not
initiate
any
collection
until
the
requisite
approval
is
received.

The
General
Services
Administration
(
GSA),
under
FIRMR
Bulletin
B­
2
administers
the
Interagency
Reports
Management
Program
as
derived
from
44
U.
S.
C.
Chapters
29
and
31.
All
work
performed
under
this
Work
Assignment
involving
federal
interagency
reporting
must
be
done
in
full
compliance
with
these
GSA
procedures.

CONFIDENTIAL
BUSINESS
INFORMATION
If
this
Work
Assignment
requires
use
of
RCRA
Confidential
Business
Information
(
CBI),
the
contract
must
specifically
authorize
the
contractor
to
have
access
to
RCRA
CBI
and
the
contractor
shall
abide
by
all
RCRA
CBI
requirements
and
stipulations
found
in
the
RCRA
CBI
Security
Manual
and
in
the
contract.
The
contractor
shall
identify
in
the
Work
Plan
budget
all
estimated
costs
for
dealing
with
CBI
requirements.
All
CBI
must
be
returned
to
EPA
as
soon
as
it
is
no
longer
needed
under
this
Work
Assignment
or
before
the
expiration
of
the
Work
Assignment,
whichever
occurs
first.

PRINTING
AND
DUPLICATION
The
contractor
is
prohibited
from
performing
any
printing
under
the
Government
Printing
and
Binding
Regulations.
Duplication
is
allowed
to
the
extent
it
does
not
exceed
5,000
impressions
of
a
single­
page
document
or
25,000
impressions
of
a
multiple­
page
stand­
alone
document,
is
limited
to
one
color
(
black)
copies,
and
does
not
exceed
the
maximum
image
size
of
10
3/
4
by
14
1/
4
inches.
For
all
duplication
jobs
in
excess
of
5,000
impressions,
the
EPA
WAM
will
determine
in
advance
if
the
work
can
be
performed
more
cost
effectively
and
under
the
job
or
time
constraints
at
the
EPA
Print
Shop.
If
the
total
number
of
photocopies
for
this
Work
Assignment
exceeds
5,000
impressions,
the
contractor
shall
identify
in
their
Work
Plan
the
photocopying
costs
by
task
and
deliverable.

WORK
ASSIGNMENT/
WORK
PLAN
BUDGETS
The
contractor
shall
not
exceed
either
the
dollar
or
PL
hour
budget
contained
in
the
approved
Work
Plan.
In
addition,
on
Quick
Response
Tasks
(
QRTs)
the
contractor
shall
not
exceed
the
PL
hour
budget
of
the
QRT.
Page
E­
10
Peer
Review
Handbook
TECHNICAL
DIRECTION
The
Designated
Work
Assignment
Manager
(
WAM)
on
this
Work
Assignment
is
authorized
to
provide
technical
direction
to
the
extent
allowed
under
EPAAR
(
1552.237­
71)
(
APR
1984)
(
DEVIATION).
Other
than
the
Designated
WAM,
only
the
Project
Officer
and
the
Contracting
Officer
are
authorized
to
provide
technical
direction.

Technical
direction
includes:

(
1)
Direction
to
the
contractor
which
assists
the
contractor
in
accomplishing
the
Statement
of
Work
(
2)
Comments
on
and
approval/
acceptance
of
reports
or
other
deliverables
Technical
direction
must
be
within
the
contract
and
the
Work
Assignment
statement
of
work.
The
Project
Officer
and
the
WAM
do
not
have
the
authority
to
issue
technical
direction
which
(
1)
institutes
additional
work
outside
the
scope
of
either
the
contract
or
this
Work
Assignment;
(
2)
constitutes
a
change
as
defined
in
the
"
Changes"
clause;
(
3)
causes
an
increase
or
decrease
in
the
estimated
cost
of
the
contract
or
Work
Assignment;
(
4)
alters
the
period
of
performance
or
deliverable
due
dates;
or
(
5)
changes
any
of
the
other
express
terms
or
conditions
of
the
contract
or
Work
Assignment.

Technical
direction
will
be
issued
in
writing
or
confirmed
in
writing
within
five
(
5)
calendar
days
after
verbal
issuance.
The
technical
direction
memorandum
will
be
provided
to
the
contractor
and
copies
will
be
forwarded
to
the
Contracting
Officer
and
the
Project
Officer.
If
the
contractor
has
not
received
written
confirmation
within
five
(
5)
calendar
days
of
a
oral
issuance,
the
contractor
must
so
notify
the
Project
Officer.

INHERENTLY
GOVERNMENTAL
FUNCTIONS
The
contractor
shall
not
perform
any
inherently
governmental
functions
(
IGF)
under
this
Work
Assignment.
If
during
the
course
of
developing
the
plan
of
work,
through
receipt
of
technical
direction,
or
in
carrying
out
the
assignment
any
portion
of
the
effort
is
considered
to
possibly
be
an
inherently
governmental
function,
the
contractor
must
immediately
notify
the
Project
Officer
and
the
Contracting
Officer.

OCCUPATIONAL
HEALTH
AND
SAFETY
Facility
site
visits
conducted
under
a
Work
Assignment
that
include
on­
site
inspections
or
sampling
must
be
conducted
in
full
compliance
with
the
Department
of
Labor,
Occupational
Safety,
and
Health
Administration
rules
under
29
CFR
Part
1910
and
EPA
Order
1440
(
Occupational
Health
and
Safety
Manual).
Peer
Review
Handbook
Page
E­
11
TRAVEL
COSTS
The
contractor
shall
follow
the
requirements
of
Subpart
31.2
of
the
FAR
and
the
Federal
regulations
in
incurring
allowable
travel
costs
under
this
Work
Assignment,
and
correspondingly
must
at
all
times
seek
and
obtain
Government
rates
whenever
available
and
observe
current
subsistence
ceilings.

QUICK
RESPONSE
TASKS
Each
Quick
Response
Task
(
QRT)
shall
be
confirmed
in
writing
and
approved
by
the
Project
Officer.
The
contractor
shall
respond
by
letter
to
the
PO
with
copies
to
the
WAM
and
the
CO
within
two
working
days,
giving
a
brief
description
of
the
plan
of
work,
including
best
estimate
of
hours
(
by
P­
level)
and
a
break­
out
of
costs
to
accomplish
the
task.

No
task
shall
exceed
a
duration
of
30
calendar
days
from
start
date
to
completion
date.
The
level
of
effort
for
each
task
shall
be
limited
to
a
maximum
of
250
labor
hours.

Quick
Response
Task
Requests
do
not
change
the
dollar
or
professional
labor
hour
budgets
of
a
Work
Assignment.
Page
E­
12
Peer
Review
Handbook
STATEMENT
OF
WORK
­
EXAMPLE
3
­
External
Peer
Review
of
Protozoa
Method
Development
Criteria
Document
Period
of
Performance:
Work
Plan
Approval
to
August
1,
1997
Work
Assignment
Manager:
Sally
Q.
Government
Employee
Office
of
Water
U.
S.
Environmental
Protection
Agency
LOE:
196
hours
SOW:
2.4
BACKGROUND
INFORMATION:

The
United
States
Environmental
Protection
Agency
(
EPA),
Office
of
Water
is
charged
with
protecting
public
health
and
the
environment
from
adverse
exposure
to
chemicals
and
microbials
in
water
media,
such
as
ambient
and
drinking
waters,
wastewater/
sewage
sludge
and
sediments.
In
support
of
this
mission
OW s
Office
of
Science
and
Technology
(
OST)
develops
health
standards,
health
criteria,
health
advisories,
and
technical
guidance
documents
for
water
and
water­
related
media.
Under
this
work
assignment,
documents
prepared
by
OST
are
to
undergo
peer
review.

Peer
review
is
an
important
component
of
the
scientific
process.
It
provides
a
focused,
objective
evaluation
of
a
research
proposal,
publication,
risk
assessment,
health
advisory,
guidance
or
other
document
submitted
for
review.
The
criticism,
suggestions
and
new
ideas
provided
by
the
peer
reviewers
stimulate
creative
thought,
strengthens
the
reviewed
document
and
confer
credibility
on
the
product.
Comprehensive,
objective
peer
reviews
leads
to
good
science
and
product
acceptance
within
the
scientific
community.

Under
this
work
assignment,
the
contractor
will
receive
one
document
(
Protozoa
Method
Development
Criteria
Document)
for
peer
review
which
is
related
to
human
health
and
ecological
effects.

STATEMENT
OF
WORK:

Task
1.
The
contractor
shall
develop
a
work
plan
to
address
all
tasks
in
this
work
assignment.
The
work
plan
shall
describe
the
steps
that
will
be
taken
by
the
contractor
to
provide
for
peer
review,
including
selection
of
peer
reviewer
candidates
with
appropriate
expertise,
determining
absence
of
conflict
of
interest,
document
and
reference
distribution,
establishing
schedules,
preparing
the
peer
review
report,
and
submittal
of
the
peer
review
package.
Curriculum
vitae
for
all
persons
assigned
to
complete
this
work
assignment
shall
be
provided.
All
P
Peer
Review
Handbook
Page
E­
13
levels,
hours
and
total
costs
for
each
task
will
be
provided
and
costs
greater
than
$
100.00
shall
be
itemized
in
detail.

Task
2.
The
contractor
shall
select
a
group
of
peer
reviewers
and
determine
their
availability
for
the
task
and
absence
of
conflict
of
interest,
and
establish
a
schedule
for
the
peer
review.
The
contractor
is
directed
to
assure
that
none
of
the
conflicts
disclosed
are
so
direct
and
substantial
as
to
rule
out
a
particular
reviewer.
Three
peer
reviewers
shall
participate
in
the
review.
No
single
peer
reviewer
may
charge
more
than
40
hours
to
this
task.
It
is
fully
acceptable
for
peer
reviewers
to
commit
to
less
than
40
hours.
The
peer
review
will
be
conducted
for
the
Protozoa
Method
Development
Criteria
Document.
Reviewers
selected
by
and
working
for
the
contractor
shall
be
approved
by
the
EPA
Project
Officer
in
writing
prior
to
their
beginning
work.
Minimally,
all
peer
reviewers
shall
be
accomplished
in
protozoan
methods
for
sample
recovery
and
analysis
from
water.
Approval
submissions
shall
include
the
reviewers 
names
and
curriculum
vitae.

Task
3.
The
contractor
shall
arrange
for
the
selected
peer
reviewers
to
review
the
EPA
document.
Prepare
the
charge
to
the
peer
reviewers
based
on
technical
direction
received
from
the
EPA
WAM.
Provide
the
peer
reviewers
with
copies
of
the
candidate
report
and
all
relevant
references
and
instruct
the
selected
peer
reviewers
to
undertake
the
review.
The
WAM
will
provide
the
contractor
with
the
final
version
of
the
document
to
be
reviewed.

Task
4.
The
contractor
shall
monitor
peer
reviewers 
progress
to
assure
timely
completion.
The
contractor
shall
collate
peer
review
comments,
and
organize
the
comments
in
the
peer
review
 
for
comments 
document.
Provide
the
peer
review
document
and
all
materials
submitted
by
the
peer
reviewers
to
the
EPA
WAM.

SCHEDULE
AND
DELIVERABLES:

Task
1.
(
Work
Plan)
15
days
after
receipt
of
work
assignment
Task
2.
1
week
after
work
plan
approval
Task
3.
1
week
after
selection
of
peer
reviewers
Task
4.
1
week
after
receiving
comments
from
the
peer
reviewers
TRAVEL:
No
travel
is
anticipated
under
this
work
assignment.
Any
travel
directly
chargeable
to
this
work
assignment
must
be
submitted
and
approved
by
the
project
officer.
Peer
Review
Handbook
Page
F­
1
APPENDIX
F
­
USEFUL
FORMS
Peer
Review
Conflict
of
Interest
Inquiry
Peer
Review
Checklist
for
Determining
Whether
a
Work
Product
Needs
Peer
Review
(
Template
Provided
by
Region
V)

Peer
Review
Checklist
for
Conducting
a
Peer
Review
(
Template
Provided
by
Region
V)

Volunteer
Service
Program
Participation
Agreement
(
EPA
Form
3100­
14)
Page
F­
2
Peer
Review
Handbook
SUBJECT:
PEER
REVIEW
CONFLICT
OF
INTEREST
INQUIRY
Dear
(
Peer
Reviewer
­­
insert
name):

You
have
been
requested
by
EPA
to
serve
as
a
Peer
Reviewer
for______
__________(
name
of
project)____________.
Your
participation
in
this
review
will
be
greatly
appreciated.
However,
it
is
possible
that
your
personal
affiliations
and
involvement
in
particular
activities
could
pose
a
conflict
of
interest
or
create
the
appearance
that
you
lack
impartiality
in
your
involvement
for
this
peer
review.
Although
your
involvement
in
these
activities
is
not
necessarily
grounds
for
exclusion
from
the
peer
review,
you
should
consult
the
contact
named
below
or
other
appropriate
official
to
discuss
these
matters.
Affiliations
or
activities
that
could
potentially
lead
to
conflicts
of
interest
might
include:

a)
work
or
arrangements
concerning
future
work
in
support
of
industries
or
other
parties
that
could
potentially
be
affected
by
regulatory
developments
or
other
actions
based
on
material
presented
in
the
document
(
or
review
materials)
that
you
have
been
asked
to
review;

b)
your
personal
benefit
(
or
benefit
of
your
employer,
spouse
or
dependent
child)
from
the
developments
or
other
actions
based
on
the
document
(
or
review
materials)
you
have
been
asked
to
review;

c)
any
previous
involvement
you
have
had
with
the
development
of
the
document
(
or
review
materials)
you
have
been
asked
to
review;

d)
any
financial
interest
held
by
you
(
or
your
employer,
spouse
or
dependent
child)
that
could
be
affected
by
your
participation
in
this
matter;
and
e)
any
financial
relationship
you
have
or
have
had
with
EPA
such
as
research
grants
or
cooperative
agreements.

Please
contact
___________________
(
name
and
contact
info
for
Agency
peer
review
official
or
primary
contractor)
to
discuss
any
potential
conflict
of
interest
issues
at
your
earliest
convenience,
but
no
later
than
_________.

[
Be
sure
to
date
and
sign
this
inquiry]
Peer
Review
Handbook
Page
F­
3
PEER
REVIEW
CHECKLIST
FOR
DETERMINING
WHETHER
A
WORK
PRODUCT
NEEDS
PEER
REVIEW
Instructions:
This
checklist
is
based
on
the
Agency s
Peer
Review
Handbook
and
the
October
2000
Region
5
Order
 
U.
S.
EPA
Region
5
Improved
Policies
and
Procedures:
Peer
Review,
Records
Management,
and
Work
Product
Authorization
of
Scientific
and
Technical
Work
Products 
which
constitute
Region
5 
s
standard
operating
procedures
for
peer
review.
If
you
have
any
questions
about
peer
review
or
need
clarification
when
completing
this
checklist,
please
refer
to
the
Handbook,
available
via
the
internet
at
http://
www.
epa.
gov/
ord/
spc/
2peerrev.
htm.
Figure
1
on
page
2
of
the
Handbook
includes
a
useful
flow
chart
and
cross
references
to
specific
sections
of
the
Handbook
that
are
applicable
to
this
checklist.
You
are
also
encouraged
to
consult
with
your
Division
or
Office
Peer
Review
Coordinator.
The
Division/
Office
Peer
Review
Coordinators
will
periodically
request
information
from
this
checklist
in
order
to
update
the
National
Peer
Review
Database.

1.
Title
of
Work
Product:

2.
Product
Description:

3.
Project
Manager:

Name,
Organization
and
Phone
Number
Please
circle
the
4.
Determination
if
Work
Product
is
Scientific
or
Technical:
appropriate
answer
a)
Is
the
work
product
a
scientific,
engineering,
economic,
social
yes
no
science,
or
statistical
document?
(
Examples
of
such
documents
include:
risk
assessments,
technical
studies
and
guidance,
analytical
methods,
scientific
database
designs,
technical
models,
technical
protocols,
statistical
surveys/
studies,
technical
background
materials,
and
research
plans
and
strategies.)
b)
Is
the
work
product
a
scientific
or
technical
document
resulting
yes
no
from
a
grant,
contract
or
cooperative
agreement?
c)
Will
the
work
product
be
used
to
support
a
research
agenda,
yes
no
regulatory
program,
policy
position,
or
other
Agency
position
or
action?

If
you
answered
 
no 
to
all
of
these
questions,
your
work
product
is
not
subject
to
EPA s
peer
review
policy
for
scientific
or
technical
work
products
and
does
not
need
to
be
placed
on
any
of
the
peer
review
lists.
Please
proceed
to
#
7
of
this
checklist.
If
you
answered
 
yes 
to
any
of
these
questions,
your
work
product
might
need
peer
review;
please
continue
on
to
#
5
of
this
checklist.
Page
F­
4
Peer
Review
Handbook
5.
Determination
if
Work
Product
is
a
Major
Work
Product:
Determination
of
whether
a
work
product
is
 
major 
will
largely
be
on
a
case­
by­
case
basis.
As
the
continuum
of
work
products
covers
the
range
from
the
obviously
major
to
those
products
that
clearly
don t
need
peer
review,
there
is
no
one
single,
easy
yes/
no
answer
to
the
test
of
 
major 
(
see
Handbook,
Section
2.2.3).
There
also
is
no
single
definition
of
 
significant. 
Determination
of
 
major 
and
 
significant 
are
the
responsibility
of
the
Division
or
Office
Director
who
is
the
official
Decision
Maker.
Please
circle
the
appropriate
answer
a)
Does
the
work
product
establish
a
significant
precedent,
yes
no
model,
or
methodology?
b)
Does
the
work
product
address
significant
controversial
yes
no
issues?
c)
Does
the
work
product
focus
on
significant
emerging
or
yes
no
 
cutting
edge 
issues?
d)
Does
the
work
product
have
significant
cross­
Agency
or
yes
no
inter­
agency
implications?
e)
Does
the
work
product
involve
a
significant
investment
of
yes
no
agency
resources?
f)
Does
the
work
product
consider
an
innovative
approach
or
yes
no
application
for
a
previously
defined
problem,
process
or
methodology?
g)
Is
the
work
product
required
to
be
peer
reviewed
by
statute
yes
no
or
other
legal
mandate?
h)
Does
the
work
product
support
a
regulatory
decision,
yes
no
policy
or
guidance
of
major
impact?
(
Major
impact
can
mean
that
it
will
have
applicability
to
a
broad
spectrum
of
regulated
entities
and
other
stakeholders,
or
that
it
will
have
narrower
applicability,
but
with
significant
consequences
on
a
smaller
geographic
or
practical
scale.)
i)
Is
the
work
product
an
application
of
or
modification
to
yes
no
an
existing,
adequately
peer
reviewed
methodology
or
model
that
departs
significantly
from
the
situation
it
was
originally
designed
to
address?

If
you
answered
 
yes 
to
any
of
these
questions,
your
work
product
needs
peer
review
unless
special
circumstances
exist;
please
continue
on
to
#
6.
If
you
answered
 
no 
to
all
of
these
questions,
your
work
product
probably
does
not
need
peer
review.
However,
peer
review
can
always
be
done
to
improve
the
quality
of
the
work
product.
Please
proceed
to
#
7
of
this
checklist.
Peer
Review
Handbook
Page
F­
5
6.
Determination
Whether
Circumstances
Exist
Where
a
Major
Work
Product
Would
Not
Be
Peer
Reviewed:
Please
circle
the
appropriate
answer
a)
Was
the
work
product
previously
reviewed
by
recognized
yes
no
experts
or
an
expert
body?
(
Note:
Peer
review
of
an
EPA
work
product
by
a
recognized
refereed
journal
strengthens
the
scientific
credibility
of
the
work
product
but
does
not
eliminate
the
need
to
have
the
work
product
itself
peer
reviewed
for
issues
and
concerns
to
support
an
Agency
action.
See
Sections
2.4.4
and
2.4.5
of
the
Handbook
for
more
details.)
b)
Are
the
scientific
or
technical
methodologies
or
information
yes
no
being
used
commonly
accepted
in
the
field
of
expertise?
c)
Has
the
regulatory
activity
or
action
which
the
work
product
yes
no
supports
been
terminated
or
canceled?
d)
Is
there
a
statutory
or
court
ordered
deadline,
or
a
time
yes
no
constraint
which
may
limit
or
preclude
peer
review
of
the
work
product?

If
you
answered
 
yes 
to
any
of
these
questions,
your
work
product
probably
does
not
require
peer
review.
This
decision
with
the
justification
needs
to
be
concurred
with
and
signed
off
by
the
Division/
Office
Director.
The
decision
with
the
justification
must
be
retained
in
the
peer
review
files
and
noted
in
Peer
Review
Work
Product
List
C
in
the
National
Peer
Review
Database.
Continue
on
to
#
7.
If
you
answered
 
no 
to
all
of
these
questions,
proceed
to
#
8.

7.
Next
Steps
For
Work
Products
That
Will
Not
Be
Peer
Reviewed:
a)
Division/
Office
Director
concurs
with
the
decision
that
the
work
product
should
not
be
peer
reviewed.

Signature
of
Division/
Office
Director
and
Date
Signed
b)
A
copy
of
this
completed
checklist
has
been
given
to
the
Div/
Off
Peer
Review
Coordinator
and
put
in
the
official
peer
review
files
in
the
Division/
Office.

Signature
of
Div/
Off
Peer
Review
Coordinator
and
Date
Signed
Location
of
Div/
Off
Peer
Review
Files
Page
F­
6
Peer
Review
Handbook
c)
Work
product
has
been
placed
on
Peer
Review
Work
Product
List
C
in
the
National
Peer
Review
Database.
(
Note:
This
only
applies
to
those
work
products
subject
to
the
peer
review
policy.)

Signature
of
Div/
Off
Peer
Review
Coordinator
and
Date
Signed
If
all
of
the
necessary
information
is
complete,
you
are
done.
You
don t
need
to
proceed
any
further
with
this
checklist.

8.
Next
Steps
For
Work
Products
That
Will
Be
Peer
Reviewed:
a)
Division/
Office
Director
has
been
consulted
and
concurs
with
the
decision
that
the
product
should
be
peer
reviewed.

Signature
of
Division/
Office
Director
and
Date
Signed
b)
A
copy
of
this
completed
checklist
has
been
given
to
the
Division/
Office
Peer
Review
Coordinator
and
put
in
the
official
peer
review
files
in
the
Division/
Office.

Signature
of
Div/
Off
Peer
Review
Coordinator
and
Date
Signed
Location
of
Div/
Off
Peer
Review
Files
c)
Work
product
has
been
placed
on
Peer
Review
Work
Product
List
B
in
the
National
Peer
Review
Database.

Signature
of
Div/
Off
Peer
Review
Coordinator
and
Date
Signed
Because
your
work
product
will
be
peer
reviewed,
you
need
to
complete
a
second
checklist
entitled
 
Peer
Review
Checklist
for
Conducting
a
Peer
Review. 
Peer
Review
Handbook
Page
F­
7
PEER
REVIEW
CHECKLIST
FOR
CONDUCTING
A
PEER
REVIEW
Instructions:
This
checklist
is
based
on
the
Agency s
Peer
Review
Handbook
and
the
October
2000
Region
5
Order
 
U.
S.
EPA
Region
5
Improved
Policies
and
Procedures:
Peer
Review,
Records
Management,
and
Work
Product
Authorization
of
Scientific
and
Technical
Work
Products 
which
constitute
Region
5 
s
standard
operating
procedures
for
peer
review.
If
you
have
any
questions
about
peer
review
or
need
clarification
when
completing
this
checklist,
please
refer
to
the
Handbook,
available
via
the
internet
at
http://
www.
epa.
gov/
ord/
spc/
2peerrev.
htm.
Pages
2­
4
of
the
Handbook
contain
useful
flowcharts
and
cross
references
to
specific
sections
of
the
Handbook
that
are
applicable
to
this
checklist.
You
are
also
encouraged
to
consult
with
your
Division
or
Office
Peer
Review
Coordinator.
The
Division/
Office
Peer
Review
Coordinators
will
periodically
request
information
from
this
checklist
in
order
to
update
the
National
Peer
Review
Database.

1.
Title
of
Work
Product:

2.
Product
Description:

3.
Project
Manager:
Name,
Organization
and
Phone
Number
Check
the
box
when
4.
Up­
front
Considerations
for
Planning
the
Peer
Review:
a)
The
Div/
Office
Director
has
chosen
a
peer
review
leader
for
the
project.
(
Note:
The
project
manager
and
peer
review
leader
can
be
the
same
person.)
Name
of
Peer
Review
Leader:
item
is
completed
Phone
Number:
Organization:
b)
The
peer
review
leader
has
obtained
appropriate
peer
review
training
before
conducting
the
peer
review.
c)
Key
questions
and
issues
have
been
identified
to
include
in
the
charge
to
the
peer
reviewers.
d)
The
Div/
Office
Records
Coordinator
has
been
consulted
to
insure
that
all
the
files,
including
electronic
records,
will
be
created,
maintained,
retained,
and
disposed
of
appropriately
and
in
accordance
with
Div/
Office
and
Agency
procedures.
e)
A
formal
peer
review
record
or
file
has
been
established,
and
provisions
have
been
made
to
store
any
electronic
records
associated
with
the
work
product
and
peer
review.
Location
of
Record/
File:
Provisions
for
Electronic
Records:
Page
F­
8
Peer
Review
Handbook
Check
the
bo
x
when
item
is
completed
or
circle
the
appropriate
answer
(
NA
=
not
applicable)
f)
There
is
a
source
of
adequate
funding
to
pay
for
external
peer
review
if
external
peer
review
is
necessary
and
funding
is
needed.
(
Note:
Contracts
NA
can
be
used
for
peer
review
services.
However,
special
management
controls
are
required
to
ensure
proper
use
of
these
contracts.
See
Section
3.6
of
the
Handbook
for
details.)
Source
of
Funding:
g)
Resource
limitations
may
restrict
the
peer
review.
(
If
 
yes 
was
Yes
No
selected,
a
limited
peer
review
might
be
considered.
However,
only
in
very
rare
circumstances
should
resource
limitations
restrict
peer
review.
Peer
review
must
be
planned
for
as
part
of
a
project s
budget.)
h)
Amount
of
time
needed
for
peer
review(
s)
has
been
allotted
given
existing
constraints
of
potential
peer
reviewers,
deadline
for
the
final
work
product,
logistics
for
the
peer
review,
etc.
Length
of
Time
Needed:

5.
Develop
the
Charge
to
the
Peer
Reviewers:
a)
A
clear,
focused
charge
has
been
formulated
that
identifies
recognized
issues,
asks
specific
questions,
and
invites
comments
or
assistance.
b)
The
charge
has
been
included
in
the
peer
review
record.

6.
Select
the
Peer
Review
Mechanism:
a)
The
work
product
is
novel,
complex,
controversial,
or
has
great
Yes
No
cost
implications.
(
If
the
answer
is
 
yes 
to
any
of
the
above,
serious
thought
should
be
given
to
conducting
an
external
peer
review.
If
the
answer
is
 
no 
to
all
of
the
above,
internal
peer
review
is
probably
sufficient.)
b)
A
determination
has
been
made
regarding
which
components
or
stages
of
the
work
product
will
be
peer
reviewed.
(
Note:
Generally,
peer
review
is
recommended
for
each
stage
of
a
product s
development.)
Components
to
be
peer
reviewed:

c)
A
peer
review
mechanism
(
e.
g.,
internal,
external
or
a
combination
of
both)
has
been
chosen
for
the
work
product
or
stages
of
the
work
product.
Mechanism:
d)
The
work
product
either:
1)
has
been,
or
is
being,
generated
as
part
Yes
No
of
administrative
or
civil
enforcement
activities
by
U.
S.
EPA,
or
2)
likely
will
be
used
in
the
future
to
support
administrative
or
civil
enforcement
activities
by
U.
S.
EPA.
(
If
the
answer
is
 
yes 
to
either
item
above,
then
the
Office
of
Regional
Council
(
ORC)
must
be
consulted
if
the
Peer
Review
Leader
believes
an
external
peer
review
is
needed
or
is
preferable.
ORC
concurrence
should
be
obtained.)
Peer
Review
Handbook
Page
F­
9
Check
the
box
when
item
is
com
pleted,
or
circle
yes
or
no
e)
The
work
product
is
going
to
be
peer
reviewed
via
a
refereed,
Yes
No
scientific
journal.
(
If
the
answer
is
 
yes, 
the
work
product
still
should
be
considered
for
peer
review
because
journal
peer
review
may
not
cover
issues
and
concerns
that
the
Agency
would
want
peer
reviewed
in
order
to
support
an
Agency
action.)
f)
Logistics
for
conducting
the
peer
review
(
e.
g.,
written
comments
will
be
received
by
mail,
or
will
be
collected
at
a
meeting)
have
been
included
in
the
peer
review
record.
g)
The
Div/
Off
Director
has
concurred
with
the
recommended
method
of
peer
review.
Date
of
Div/
Off
Director
Concurrence:
h)
The
concurrence
of
the
Div/
Off
Director
has
been
included
in
the
peer
review
record.

7.
Determine
the
Specific
Time
Line
for
the
Peer
Review:
a)
A
start
date
for
the
peer
review
has
been
selected.
Start
Date:
b)
The
amount
of
time
the
peer
reviewers
will
be
given
to
conduct
the
peer
review
has
been
determined.
Number
of
Days
for
Review:
c)
A
due
date
for
comments
from
the
reviewers
has
been
selected.
Due
Date:
d)
The
amount
of
time
necessary
to
incorporate
comments
from
th
e
peer
reviewers
into
the
work
product
has
been
determined.
Number
of
Days
for
Revision:
e)
A
deadline
for
final
completion
of
the
work
product
has
been
determined.
Due
Date:

8.
Select
the
Peer
Reviewers:
a)
Advice
was
sought
in
developing
a
list
of
potential
peer
reviewer
candidates
who
are
independent
of
the
work
product
and
have
appropriate
scientific
and
technical
expertise.
b)
The
expertise
required
for
the
peer
review
has
been
determined.
c)
In
reviewing
the
candidates,
a
balance
and
a
broad
enough
spectrum
of
expertise
were
considered.
d)
In
reviewing
the
candidates,
any
potential
conflicts
of
interest
were
considered.
Page
F­
10
Peer
Review
Handbook
Check
the
box
when
item
is
completed
e)
The
peer
reviewers
have
been
selected
and
the
process
for
selecting
the
reviewers,
including
inquiries
and
resolution
of
potential
conflicts
of
interest,
has
been
documented
and
included
in
the
peer
review
record/
file.
(
Note:
Conflict
of
Interest
Inquiry
Forms
are
available
from
the
Regional
and
Div/
Off
Peer
Review
Coordinators.)

9.
Obtain
and
Transmit
Materials
for
Peer
Review:
a)
Instructions
have
been
given
to
the
peer
reviewers
which
ask
for
written
comments
in
a
specified
format
by
the
specified
deadline
that
are
responsive
to
the
charge.
b)
The
peer
reviewers
have
been
provided
with
the
essential
documents,
data,
and
information
to
conduct
their
review.
Date
Peer
Reviewers
Given
Charge/
Materials:
c)
The
peer
reviewers
have
been
instructed
not
to
disclose
draft
work
products
to
the
public.
d)
The
peer
review
record/
file
contains
all
the
materials
given
to
the
peer
reviewers.

10.
Conduct
the
Peer
Review:
a)
Written
comments
have
been
received
from
all
peer
reviewers.
Date
all
comments
were
received:
b)
All
clarification
or
additional
information
necessary
from
the
peer
reviewers
is
received.
c)
The
validity
and
objectivity
of
the
comments
have
been
evaluated.
d)
Appropriate
experts/
staff/
managers
have
been
consulted
on
the
potential
impacts
of
the
comments
on
the
final
work
product,
the
project
schedule,
and
budget.
e)
The
peer
review
comments
have
been
included
in
the
peer
review
record/
file.

11.
Consider
the
Peer
Review
Comments:
a)
Decisions
have
been
made
regarding
which
comments
are
accepted
and
will
be
incorporated
into
the
final
work
product,
and
which
comments
will
not
be
incorporated.
b)
A
memo
or
other
written
record
has
been
prepared
which
responds
to
the
peer
review
comments
and
specifies
acceptance
or,
where
thought
appropriate,
rebuttal
and
non­
acceptance.
c)
The
Div/
Off
Director
has
concurred
with
the
decisions
and
written
record
on
how
to
incorporate
the
peer
reviewers
comments
in
the
work
product
and
on
which
comments
will
not
be
incorporated.
Date
of
Div/
Off
Director
concurrence:
Peer
Review
Handbook
Page
F­
11
Check
the
box
when
item
is
com
pleted,
or
circle
yes
or
no
d)
The
concurrence
of
the
Div/
Off
Director
has
been
included
in
the
peer
review
record/
file.
e)
The
memo
or
written
record
documenting
how
comments
were
handled
and
how
the
work
product
was
revised
has
been
included
in
the
peer
review
record/
file.
f)
The
work
product
has
been
revised
to
incorporate
the
acceptable
comments.
g)
The
peer
review
performed
during
the
process
of
developing
the
work
product
has
been
summarized
and
included
in
the
work
product.
h)
It
is
necessary
to
send
the
revised
work
product
back
to
the
peer
Yes
No
reviewers.
(
If
the
answer
is
 
yes, 
proceed
to
item
#
11i.
If
the
answer
is
no,
proceed
to
item
#
12.)
i)
Additional
comments
are
received,
evaluated,
and
incorporated
into
the
work
product,
and
placed
in
the
peer
review
record.

12.
Consider
Other
Comments:
a)
Prior
to
finalization,
the
document
needs
additional
internal
Yes
No
and/
or
external
programmatic
review.
(
If
the
answer
is 
yes, 
go
to
#
12b.
If
the
answer
is
 
no, 
proceed
to
#
13.)
b)
Written
comments
by
programmatic
reviewers
have
been
received.
c)
Final
decisions
have
been
made
regarding
which
comments
are
accepted
and
will
be
incorporated
into
the
final
work
product,
and
which
ones
will
not
be
incorporated.
d)
A
memo
or
other
written
record
has
been
prepared
which
responds
to
the
programmatic
review
comments
and
specifies
acceptance
or,
where
thought
appropriate,
rebuttal
and
non­
acceptance.
e)
Div/
Off
Director
has
concurred
with
the
decisions
and
written
record
on
how
to
incorporate
the
programmatic
comments.
Date
of
Div/
Off
Director
concurrence:
f)
The
memo
or
written
record
has
been
included
in
the
peer
review
record/
file.
g)
The
work
product
has
been
revised
to
incorporate
the
acceptable
programmatic
comments.

13.
Finalize
Work
Product
and
Close
Out
Peer
Review:
a)
The
work
product
has
been
completed.
b)
The
Div/
Off
Director
has
approved
the
work
product.
Date
of
Div/
Off
Director
Approval:
c)
The
Div/
Off
Director
approval
has
been
included
in
the
peer
review
record/
file.
Page
F­
12
Peer
Review
Handbook
Check
the
box
when
item
is
completed,
or
circle
yes
or
no
d)
The
Div/
Off
Director
has
judged
the
work
product
to
be
sufficiently
Yes
No
controversial,
of
significant
enough
interest
to
outside
parties,
or
of
wide
enough
distribution,
such
that
it
should
also
be
authorized
by
the
Regional
Administrator
(
RA),
or
the
Deputy
RA
(
DRA).
(
If
the
answer
is
 
yes, 
proceed
to
#
13e.
If
the
answer
is
 
no, 
proceed
to
#
13f.)
e)
The
RA
or
DRA
has
authorized
the
work
product.
Date
of
RA
or
DRA
Authorization:
f)
The
final
work
product
has
been
included
in
the
peer
review
record/
file.

14.
Publication
and
Release
of
Reports:
a)
The
Div/
Off
Director
has
approved
publication
or
release
of
the
work
product.
b)
The
written
approval
by
the
Div/
Off
Director
has
been
included
in
the
peer
review
record/
file.
c)
The
Div/
Off
Director
has
judged
the
work
product
to
be
sufficiently
Yes
No
controversial,
of
significant
enough
interest
to
outside
parties
or
of
wide
enough
distribution,
such
that
its
distribution
or
release
should
also
be
authorized
by
the
RA
or
DRA.
If
the
answer
is
 
yes, 
proceed
to
#
14d.
If
the
answer
is
 
no, 
proceed
to
#
15.
(
Note:
The
Div/
Off
Director s
decision
to
elevate
to
the
RA
or
DRA
can
be
made
concurrently
with
item
#
13d.)
d)
The
RA
or
DRA
has
authorized
distribution
or
release
of
the
work
product.
Date
of
RA
or
DRA
Authorization:

15.
Retention
of
Peer
Review
Files
and
Records:
a)
The
Div/
Off
official
procedures
for
administrative
records
and
the
Agency s
record
retention
schedules
have
been
examined
to
determine
how
long
the
peer
review
record/
file,
including
electronic
records,
should
be
retained.
(
Note:
The
required
time
of
retention
for
final
reports
and
supporting
data
varies
depending
upon
the
nature
of
the
report,
however,
final
reports
which
are
mission
related
or
have
an
EPA
number
and
receive
external
distribution
are
generally
permanent
federal
records.)
b)
The
Div/
Off
Records
Officer
or
the
Regional
Records
Officer
has
been
consulted
to
help
determine
how
long
the
peer
review
record/
file,
including
electronic
records,
should
be
retained.
Peer
Review
Handbook
Page
F­
13
Check
the
box
when
item
is
completed
c)
A
location
for
the
completed
peer
review
record/
file
has
been
identified,
and
provisions
have
been
made
to
retain
electronic
records
associated
with
the
work
product
and
peer
review.
(
Note:
This
can
be
the
same
location
and
provisions
as
identified
in
#
4e.)
Location
of
Record/
File:
Provisions
for
Electronic
Records:

d)
Someone
has
been
assigned
the
responsibility
for
maintaining
the
record/
file
and
electronic
records,
and
ensuring
that
they
are
either
archived
or
destroyed
appropriately.
(
Note:
This
can
be
the
same
person
as
identified
in
#
4a.)
Contact
Name
and
Phone
No:
Organization:

16.
Closeout
of
Checklist:
a)
Items
#
1­
15
of
checklist
have
been
completed.

Signature
of
Peer
Review
Leader
and
Date
Signed
b)
A
copy
of
the
completed
checklist
has
been
given
to
the
Div/
Off
Peer
Review
Coordinator.

Signature
of
Div/
Off
Peer
Review
Coordinator
and
Date
Signed
c)
The
completed
checklist
has
been
included
in
official
peer
review
record/
file.

d)
The
work
product
has
been
moved
from
Peer
Review
Work
Product
List
B
to
List
A
in
the
National
Peer
Review
Database.
Date
Product
moved
to
List
A:
Page
F­
14
Peer
Review
Handbook
U.
S.
Environmental
Protection
Agency
VOLUNTEER
SERVICE
PROGRAM
PARTICIPATION
AGREEMENT
Name
of
Program
Participant
Citizen
of
U.
S.

 
Yes
 
No
Project
Supervisor
Proposed
Organizational
Agreement
(
Include
Geographical
Location)

Proposed
Arrival
Date
Actual
Arrival
Date
Proposed
Termination
Actual
Termination
Sponsored
by
Educational
Institution
 
Yes
 
No
Name
and
Address
of
Institution
(
if
 
Yes 
)

Describe
Project
(
s)
on
Which
Par
ticipant
Will
Work,
Including
Scope
and
Anti
cipated
Hour
s
per
Week
Facilities
and
Equipment
to
be
Made
Available
by
EPA
Degree
of
Supervision
to
be
Exercised
by
EPA
Assistance
and
Degree
of
Cooperation
Required
by
other
Agency
Personnel
by
Participant
Project
Supervisor
Certifies
That
Services
to
be
Produced
by
the
Program
Participant
are
not
Services
Provided
for
Through
EPA
Agency
Operations
______________________________________________
Signature
of
Supervisor
THE
VOLUNTEER
AGREES
THAT:
a.
l
claims
for
compensation
from
the
Government
of
the
United
States
are
waived
for
any
services
performed;
b.
ernment
has
a
non­
exclusive
royalty­
free
license
to
use
or
reproduce
and
patent
or
copyright
material
which
is
developed
as
part
of
and
during
participation
in
this
program;
and
c.
to
the
administrative
instructions
and
requirements
of
the
agency
while
on
EPA
premises.

______________________________________________
Participant
Signature
PERSONNEL
OFFICE
SIGNEE
CONCURRENCE
EPA
Form
3100­
14
(
Rev.
b
30)
Replaces
the
7­
72
edition
and
the
12­
77
edition
which
was
entitled
 
Unpaid
Work­
Study
Program 
Any
and
al
The
Gov
He/
she
will
adhere
SIGNATURE
OF
PERSONNEL
OFFICER
OR
DE
DATE
Peer
Review
Handbook
Page
G­
1
APPENDIX
G
­
REFERENCES
CONCERNING
PEER
REVIEW
American
Chemical
Society
and
the
Conservation
Foundation
(
1985)
Issues
in
Peer
Review
of
the
Scientific
Basis
for
Regulatory
Decisions,
Washington,
DC,
November
1985.

Browner,
C.
(
1994)
Peer
Review
Program,
Washington,
DC,
Memorandum
issued
June
7,
1994.
(
NOTE:
Attached
as
Appendix
A
to
this
Handbook)

Chubin,
D.
(
1994)
Grants
peer
review
in
theory
and
practice,
Evaluation
Review
18:
20­
30.

Chubin,
D.
and
E.
Hackett
(
1990)
Peerless
Science:
Peer
Review
and
Science
Policy,
Albany,
NY:
State
University
of
New
York
Press.

Jasanoff,
S.
(
1990)
The
Fifth
Branch:
Science
Advisors
as
Policymakers,
Cambridge,
MA:
Harvard
University
Press.

Kostoff,
R.
(
1996)
Peer
Review
in
Selected
Federal
Agencies,
presented
at
AAAS
Annual
Meeting,
Baltimore,
MD,
February
9,
1996.

National
Academy
of
Sciences
(
1999)
Evaluating
Federal
Research
Programs:
Research
and
the
Government
Performance
and
Results
Act,
Washington,
DC:
National
Academy
Press,
January
1999.

National
Environmental
Policy
Institute
(
1996)
Enhancing
the
Integrity
and
Transparency
of
Science
in
the
Regulatory
Process,
Washington,
DC:
National
Environmental
Policy
Institute,
Fall
1996.

National
Research
Council
(
1995)
Interim
Report
of
the
Committee
on
Research
and
Peer
Review
in
EPA,
Washington,
DC:
National
Academy
Press,
March
1995.

National
Research
Council
(
2000)
Strengthening
Science
at
the
U.
S.
Environmental
Protection
Agency:
Research
Management
and
Peer
Review
Practices,
Washington,
DC:
National
Academy
Press.

Reilly,
W.
(
1993)
Peer­
review
Policy,
Washington,
DC,
Memorandum
issued
January
19,
1993.

Rennie,
D.
(
1999)
The
Development
and
Rationale
of
Peer
Review,
In:
Peer
Review
in
Health
Sciences,
Fiona
Godlee
and
Tom
Jefferson,
eds.,
BMA
Books,
London.
Page
G­
2
Peer
Review
Handbook
Science
Advisory
Board
(
1999)
An
SAB
Report:
Review
of
the
Peer
Review
Program
of
the
Environmental
Protection
Agency,
EPA­
SAB­
RSAC­
00­
002,
U.
S.
Environmental
Protection
Agency,
Science
Advisory
Board,
Washington,
DC,
November
1999.

Spitzer,
H.
(
1995)
Peer
Review
Practices
in
the
Federal
Government,
Bethesda,
MD:
Environmental
Network,
report
prepared
for
the
American
Industrial
Health
Council,
April
26,
1995)

USEPA
(
1992)
Safeguarding
the
Future:
Credible
Science,
Credible
Decisions,
U.
S.
Environmental
Protection
Agency,
Washington,
DC,
March
1992.

USEPA
(
1998)
Science
Policy
Council
Handbook:
Peer
Review,
EPA
100­
B­
98­
001,
U.
S.
Environmental
Protection
Agency,
Science
Policy
Council,
Washington,
DC,
January
1998.

USEPA
(
1999)
EPA s
Selection
of
Peer
Reviewers,
EPA
OIG
Report
No.
1999­
P­
217,
U.
S.
Environmental
Protection
Agency,
Office
of
the
Inspector
General
(
OIG),
Washington,
DC,
September
1999.

U.
S.
General
Accounting
Office
(
1994)
Peer
Review:
EPA
Needs
Implementation
Procedures
and
Additional
Controls,
GAO/
RCED­
94­
89,
Washington,
DC:
U.
S.
Government
Printing
Office,
February
1994.

U.
S.
General
Accounting
Office
(
1996)
Peer
Review:
EPA s
Implementation
Remains
Uneven,
GAO/
RCED­
96­
236,
Washington,
DC:
U.
S.
Government
Printing
Office,
September
1996.

U.
S.
General
Accounting
Office
(
1999)
Federal
Research:
Peer
Review
Practices
at
Federal
Science
Agencies
Vary,
GAO/
RCED­
99­
99,
Washington,
DC:
U.
S.
Government
Printing
Office,
March
1999.
Peer
Review
Handbook
Notes
and
Comments
Peer
Review
Handbook
Notes
and
Comments
Peer
Review
Handbook
Notes
and
Comments
Peer
Review
Handbook
Notes
and
Comments
