Charge
for
the
BOSC
Subcommittee
on
Particulate
Matter
and
Ozone
Research
1.0
Objective.
The
objective
of
this
review
is
to
evaluate
the
relevance,
quality,
performance,
as
well
as
the
scientific
and
managerial
leadership
of
ORD's
Particulate
Matter
and
Ozone
Research
(
PM­
O3)
Programs.
The
panel's
evaluation
and
recommendations
will
provide
guidance
to
the
Office
of
Research
and
Development
to
help:

$
plan,
implement,
and
strengthen
the
Program;

$
make
research
investment
decisions
over
the
next
five
years;

$
refine
the
integration
of
the
ORD
Program
with
those
of
other
federal
agencies
$
prepare
EPA's
performance
and
accountability
reports
to
Congress
under
the
Government
Performance
and
Results
Act;
and
$
respond
to
evaluations
of
federal
research
such
as
those
conducted
by
the
Office
of
Management
and
Budget
(
OMB
highlights
the
value
of
recommendations
from
independent
expert
panels
in
guidance
to
federal
agencies1,2).

2.0
Background
Information.
Independent
expert
review
is
used
extensively
in
industry,
federal
agencies,
Congressional
committees,
and
academia.
The
National
Academy
of
Science
has
recommended
this
approach
for
evaluating
federal
research
programs.
3
Because
of
the
nature
of
research,
it
is
not
possible
to
measure
the
creation
of
new
knowledge
as
it
develops
 
or
the
pace
at
which
research
progresses
or
scientific
breakthroughs
occur.
Demonstrating
research
contributions
to
outcomes
is
very
challenging4
when
federal
agencies
conduct
research
to
support
regulatory
decisions,
and
then
rely
on
third
parties5
 
such
as
state
environmental
agencies
 
to
enforce
the
regulations
and
demonstrate
environmental
improvements.
Typically,
many
years
may
be
required
for
practical
research
applications
to
be
developed
and
decades
may
be
required
for
some
research
outcomes
to
be
achieved.

Most
of
EPA's
environmental
research
programs
investigate
complex
environmental
problems
and
processes
 
combining
use­
inspired
basic
research6,7
with
applied
research,
and
integrating
several
scientific
disciplines
across
a
conceptual
framework8
that
links
research
to
environmental
decisions
or
environmental
outcomes.
In
multi­
disciplinary
research
programs
such
as
these,
progress
toward
outcomes
can
not
be
measured
by
outputs
created
in
a
single
year.
Rather,
research
progress
occurs
over
several
years,
as
research
teams
explore
hypotheses
with
individual
studies,
interpret
research
findings,
and
then
develop
hypotheses
for
future
studies.

In
designing
and
managing
its
research
programs,
ORD
emphasizes
the
importance
of
identifying
priority
research
questions
to
guide
the
research.
Similarly,
ORD
recommends
that
its
programs
develop
a
small
number
of
performance
goals
that
serve
as
indicators
of
progress.
Short­
term
outcomes
are
accomplished
when
research
is
2
applied
by
specific
clients
to
strengthen
environmental
decisions
or
regulations.
These
decisions
and
resulting
actions
(
e.
g.,
the
reduction
of
contaminant
emissions
or
the
reduction
of
uncertainties
in
risk
assessment)
ultimately
contribute
to
improved
environmental
quality
and
health.

In
a
comprehensive
evaluation
of
science
and
research
at
EPA,
the
National
Research
Council
recommended9
that
the
agency
substantially
increase
its
efforts
to
explain
the
significance
of
its
research
products
and
to
assist
clients
inside
and
outside
the
agency
in
applying
them.
In
response
to
this
recommendation,
ORD
has
engaged
science
advisors
from
client
organizations
to
serve
as
members
of
its
research
program
teams.
These
teams
help
identify
research
contributions
with
significant
decision­
making
value
and
help
plan
for
their
transfer
and
application.

For
EPA's
environmental
research
programs,
periodic
retrospective
analysis
at
intervals
of
four
or
five
years
is
needed
to
characterize
research
progress,
to
identify
when
clients
are
applying
research
to
strengthen
environmental
decisions,
and
to
evaluate
client
feedback
about
the
research.
Conducting
program
evaluation
at
this
interval
enables
assessment
of
research
progress,
the
scientific
quality
and
decision­
making
value
of
the
research,
and
whether
research
progress
has
resulted
in
short­
term
outcomes
for
specific
clients.

In
1998,
Congress
augmented
the
budget
for
PM­
related
research
and
mandated
the
establishment
of
a
NRC
Committee
to
assess
the
research
needs
PM.
The
NRC
Committee
has
since
published
four
reports
of
Research
Priorities
for
Airborne
Particulate
Matter,
with
volume
IV
published
in
October,
2004.10
The
four
volumes
have
provided
guidance
to
the
PM
Research
Program
in
the
form
of
an
initial
ten
(
and
eventually
twelve)
priority
research
areas
(
needs).
In
these
same
reports,
the
NRC
has
submitted
peer­
expert
evaluations
of
the
Agency's
PM
Research
Program
including
its
strengths
and
productivity,
its
short­
comings,
as
well
as
identifying
challenges
for
the
future.
Since
1998,
ORD
has
aligned
its
research
program
with
the
NRC
priorities,
evolving
the
relative
emphases
on
these
priorities
with
the
development
of
the
science,
client
needs,
and
frequent
peer
reviews
of
all
or
selected
parts
of
the
Program.

It
is
essential
to
appreciate
that
the
ORD
PM
Program
comprises
an
intramural
research
program
in
health
and
implementation,
as
well
as
an
extramural
(
grant
funded)
program,
which
is
complimentary
and
integrated
by
design
to
meet
the
client
Air
Office
needs.
In
completing
the
final
report
(
IV),
the
NRC
provided
its
assessment
of
the
PM
Research
Program
and
its
accomplishments,
and
delineated
a
series
of
challenges
for
the
years
ahead.
These
challenges
were
provided
in
the
presentation
of
the
Committee
Chair,
Dr.
Jonathan
Samet
of
Johns
Hopkins
University,
to
ORD
at
the
completion
of
Report
IV.
The
charge
to
the
NRC
Committee
reviewing
the
PM
Program
is
now
complete,
and
the
formal
Committee
will
cease
to
exist;
but
it
is
expected
that
ad
hoc
committees
will
be
convened
at
points
in
the
future
to
revisit
the
Program
priorities
and
directions.

Beginning
in
1997,
ORD
gradually
redirected
its
long­
standing
Ozone
Research
Program,
initially
in
health
and
ecology,
to
allow
for
the
growth
and
emphasis
in
health
3
research
in
PM.
Agency
supported
ozone­
specific
research
in
these
areas
is
currently
minimal.
More
recently,
an
analogous
adjustment
has
occurred
in
the
implementation
program
as
well.
In
the
latter
case,
the
atmospheric
science
research
in
ozone
and
PM
were
merged
in
research
of
atmospheric
processes
and
modeling,
as
they
are
inextricably
linked
in
the
air
environment.
With
the
disinvestment
in
the
ozone­
specific
research
and
its
emergence
in
a
more
integrated
form
within
the
PM
Program
efforts
in
atmospheric
chemistry
and
co­
pollutant
health
research,
these
two
largely
independent
research
programs
have
been
fully
merged,
and
is
evidenced
by
plans
to
revise
the
Mult­
Year
Plans
(
MYP)
for
PM
and
Ozone
into
a
merged,
single
MYP.

In
2003,
the
PM
Program
underwent
review
by
OMB
using
a
novel
approach
to
assess
program
success.
This
review
is
termed
the
Program
Assessment
Rating
Tool
(
PART),
which,
in
brief,
focused
on
(
1)
the
relevance
of
the
PM
Program
to
its
clients,
(
2)
the
clarity
and
specificity
of
its
long­
term
goals
and
resultant
outcomes
that
could
be
linked
explicitly
with
measurable
improvements
in
health
and
the
environment,
(
3)
research
progress
and
performance,
and,
finally,
(
4)
the
resource
management
ensuring
high­
quality
research.
Overall,
the
PM
Program
scored
well
in
this
process,
except
in
the
areas
designated
in
(
2)
requiring
demonstrated
measurable
outcomes
in
public
health
and
the
environment.
As
the
PM
and
Ozone
Programs
have
been
merged
and
enter
a
second
PART
review,
the
intent
is
to
use
this
BOSC
assessment
of
the
Program's
new
structure,
its
management
and
leadership,
as
well
as
its
scientific
achievements
and
directions
to
guide
preparations
for
the
PART
(
spring/
summer
2004).

3.0
Draft
Charge
Questions
for
ORD's
Particulate
Matter
­
Ozone
Research
Program
The
following
charge
questions
will
help
evaluate
the
relevance,
quality,
performance,
as
well
as
management
and
scientific
leadership
of
ORD's
Particulate
Matter
­
Ozone
Research
Program:

1.
Program
Design
and
Demonstrated
Leadership
$
Does
the
new
draft
PM­
O3MYP
structure
reflect
the
identified
science
needs
of
the
Program
and
show
integration
and
leveraging
of
human
and
fiscal
resources?

S
Is
the
PM­
O3
MYP
structure
strategic
by
design,
implementation
and
review?

S
Does
the
PM­
O
MYP
structure
provide
a
reasonable
"
road­
map"
of
the
Program
demonstrating
a
well­
thought­
out
plan,
identifying
critical
paths,
clear
goals,
priorities
and
schedules?

S
Is
the
extramural
Program
adequately
integrated
into
the
Program
MYP
and
goals?

S
Does
the
PM­
O3MYP
structure
reflect
an
"
outcome"
orientation
that
provides
measures
demonstrating
the
true
impact
on
public
health
and
the
environment?

S
Is
the
ORD
PM­
O3
Program
responsive
to
the
recommendations
of
the
NRC
in
terms
of
products
and
outputs?
4
S
Are
the
near
and
long­
term
visions
of
the
Program
consistent
with
the
NRC­
noted
"
challenges
for
the
future"?

$
Is
ORD
sufficiently
coordinating
research
across
categories
of
the
risk
assessment
paradigm
(
source,
exposure,
health,
assessment,
and
management)?

S
Is
the
work
within
the
Labs
and
Centers
integrated
to
maximize
resource
investment?

$
Is
EPA­
ORD
providing
evident
and
appropriate
science
leadership
and
Program
management?

S
Are
there
changes
or
refinements
in
management
or
science
leadership
that
are
needed
to
improve
the
Program?

$
Is
the
EPA­
ORD
sufficiently
communicating
its
results
to
its
clients
and
the
broader
scientific
community?

S
What
can
be
done
to
improve
communication
and
access
to
information
by
regulatory
and
science
communities?

$
Are
there
important
interagency
or
extramural
collaborations
that
should
and
can
be
improved
to
advance
the
Agency's
research
agenda?

S
To
what
extent
has
the
EPA
established
and
utilized
other
organizations
(
inside
and
outside
governments)
in
advancing
the
EPA's
research
agenda?

S
Is
the
interaction
and
leadership
role
of
EPA­
ORD
with
other
federal
agencies
through
the
CENR
effectively
providing
national
coordination?

2.
Science
Quality
$
Is
the
science
being
conducted
by
EPA­
ORD
research
Labs
and
Centers
of
recognized
high
quality
and
appropriate
to
the
perceived
needs?

$
Is
program
integration
across
Labs,
Centers,
and
science
discipline
making
full
advantage
of
research
opportunities?

$
Does
the
program
ensure
high
quality
research
through
competitive,
merit­
based
funding?
If
funds
are
not
competitively
awarded,
what
process
does
the
program
use
to
allocate
funds?
Does
this
process
ensure
that
quality
is
maintained?

3.
Relevance
$
Does
the
PM­
O3MYP
structure
and
Research
Program
clearly
reflect
its
focus
and
the
rationale
behind
its
research
direction
and
out­
year
emphasis?

$
Are
the
potential
public
benefits
in
term
of
public
health
protection
and
pollution
abatement
clearly
articulated?

$
Has
the
PM­
O3
Program
effectively
engaged
stakeholders
in
its
assessment
processes,
and
provided
useful
information
and
tools
in
a
timely
manner?

$
Has
the
Program
begun
to
establish
a
process
for
using
the
results
of
assessments,
along
with
stakeholder
feedback,
to
identify
key
research
gaps
and
to
update
the
Program's
research
agenda?

4.
Demonstrated
Outcomes
$
Does
the
Program
have
a
limited
number
of
specific
long­
term
performance
measures
that
focus
on
outcomes
and
meaningfully
reflect
the
purpose
of
the
program?
5
$
Has
the
Program
made
significant
progress
in
the
conduct
of
the
planned
research
and
in
answering
the
key
science
questions
related
to
public
health
and
pollution
abatement?

$
Does
the
Program
have
ambitious
targets
and
time­
frames
for
long­
term
measures?

S
Has
the
Program
made
adequate
progress
in
meeting
its
long­
term
goals?

S
Are
there
baselines
and
appropriate
targets
for
the
Program's
annual
measures?

S
Have
the
Program's
research
products
been
consistent
with
the
program's
goals
and
supportive
of
client
needs?

S
Are
the
research
program's
findings
incorporated
into
regulations
and
standards,
published
in
the
peer­
reviewed
literature,
or
do
they
otherwise
demonstrate
superior
scientific
quality?

$
Do
independent
evaluations
of
sufficient
scope
and
quality
indicate
that
the
Program
is
effective
and
is
achieving
results?

S
Does
the
Program
demonstrate
improved
efficiencies
and
cost
effectiveness
in
achieving
Program
goals
each
year?

$
Do
EPA­
ORD
and
Program
leadership
make
adjustments
in
the
Program's
science
and
emphasis
to
meet
the
evolving
science
and
research
needs?

S
Is
the
Program
appropriately
structured
to
allow
for
flexibility
in
direction
and
emphasis?

4.0
Potential
BOSC
Approach
for
Program
Review
$
Hold
conference
call/
s
in
the
month
preceding
a
face­
to­
face
meeting.



Goal:
Familiarize
Subcommittee
with
Review
objectives,
introduce
review
materials,
and
make
assignments
for
face
to
face
meeting.
1.
The
DFO
distributes
background
materials
and
documents
requested
by
the
Subcommittee
four
weeks
in
advance
of
the
first
conference
call.
2.
ORD
presents
background
materials
to
the
Subcommittee
during
the
first
call
for
initial
orientation;
3.
The
Subcommittee
reviews
and
comments
on
the
charge;
and
4.
The
Subcommittee
asks
clarifying
questions
about
the
program
under
review
5.
The
Subcommittee
Chair
makes
review
and
writing
assignments
to
Subcommittee
members
in
advance
of
a
face­
to­
face
meeting.

$
Hold
a
2
to
3
day
face­
to­
face
meeting
for
the
program
review
at
a
location
where
a
critical
mass
of
ORD
scientists
is
located.



Goal:
A
draft
Subcommittee
report
is
available
for
circulation
and
comment
at
the
end
of
the
face­
to­
face
meeting
that
thoroughly
addresses
all
charge
questions.
1.
The
first
segment
of
the
meeting:
ORD
presentations
and
poster
sessions,
Subcommittee
questions
and
discussion,
identification
of
issues
for
further
resolution.
6
2.
The
second
segment
of
meeting:
the
subcommittee
discusses
prepared
written
assignments
in
context
of
presentations
and
discussion,
identifies
and
agrees
to
areas
for
change,
elaboration
or
other
adjustment
of
the
text
as
necessary.
3.
The
third
segment
of
meeting:
the
subcommittee
revises
written
assignments
and
assembles
them
into
a
draft
report.

$
As
necessary,
hold
one
to
two
conference
calls
to
complete
the
draft
report
in
the
month
following
the
face­
to­
face
meeting.



Goal:
A
report
approved
by
the
Subcommittee
is
available
for
BOSC
Executive
Committee
discussion/
approval
at
the
May
2005
BOSC
Executive
Committee
Meeting,
with
final
draft
completed
by
the
end
of
April.
7
References
1
Budget
Data
Request
04­
31.
Executive
Office
of
the
President,
Office
of
Management
and
Budget.
March
22,
2004.
"
Completing
the
Program
Assessment
Rating
Tool
(
PART)
for
the
FY06
Review
Process,"
pages
50­
56.

2
Memorandum
for
the
Heads
of
Executive
Departments
and
Agencies.
Executive
Office
of
the
President,
Office
of
Management
and
Budget.
June
5,
2003.
"
FY
2005
Interagency
Research
and
Development
Priorities,"
pages
5­
10.

3
Evaluating
Federal
Research
under
the
Government
Performance
and
Results
Act
(
National
Research
Council,
1999).

4
The
House
Science
Subcommittee.
Letter
to
Dr.
Bruce
Alberts,
President
of
the
National
Academy
of
Sciences,
from
F.
James
Sensenbrenner,
Jr.
and
George
E.
Brown.
October
23,
1997.

5
The
Government
Performance
and
Results
Act:
1997
Government
wide
Implementation
Will
Be
Uneven.
U.
S.
General
Accounting
Office.
(
GAO/
GGD,
1997).

6
Building
a
Foundation
for
Sound
Environmental
Decisions.
(
National
Research
Council,
1997).

7
"
Renewing
the
Compact
between
Science
and
Government,"
Stokes,
D.
E.,
in
1995
Forum
Proceedings,
Vannevar
Bush
II
 
Science
for
the
21st
Century.
Pages
15­
32.
Sigma
Xi,
1995.

8
Risk
Assessment
in
the
Federal
Government:
Managing
the
Process.
(
National
Research
Council,
1983).

9
Strengthening
Science
at
the
U.
S.
Environmental
Protection
Agency.
(
National
Research
Council,
2000,
p
141).

10
National
Research
Council
of
the
National
Academies:
Research
Priorities
for
Airborne
Particulate
Matter
IV,
The
National
Academies
Press,
Washington,
DC.
2004
(
www.
nap.
edu;
Report
IV
­
http://
www.
nap.
edu/
books/
0309091993/
html/)
8
Appendix
I
OSTP/
OMB
Research
and
Development
Investment
Criteria
included
as
a
PDF
file.
