Using Probabilistic Methods to Enhance the Role of Risk Analysis in
Decision-Making 

With

Case Study Examples 

Prepared by Risk Assessment Forum 

PRA Technical Panel Working Groups

EPA/100/R-09/001

“THIS INFORMATION IS DISTRIBUTED SOLELY FOR THE PURPOSE OF
PRE-DISSEMINATION PEER REVIEW UNDER APPLICABLE INFORMATION QUALITY
GUIDELINES. IT HAS NOT BEEN FORMALLY DISSEMINATED BY EPA. IT DOES NOT
REPRESENT AND SHOULD NOT BE CONSTRUED TO REPRESENT ANY AGENCY
DETERMINATION OR POLICY.”Disclaimer

This document is a peer review draft. It has not been formally released
by the U.S. Environmental Protection Agency and should not at this stage
be construed to represent Agency policy. It is being circulated for
comments on its technical merit and policy implications. Mention of
trade names or commercial products does not constitute endorsement or
recommendation for use.

This document was produced by a Technical Panel of the EPA Risk
Assessment Forum.  The authors drew on their experience in doing
probabilistic assessments and interpreting them to improve risk
management of environmental and health hazards.  Interviews,
presentations, and dialogues with risk managers conducted by the
Technical Panel have contributed to the insights and recommendations in
this summary and the associated White Papers.Foreword

Throughout most of the Environmental Protection Agency's program offices
and regions, various forms of probabilistic methods have been used to
answer questions about exposure and risk, to humans and other organisms,
and the environment.  EPA risk assessors, risk managers and others,
particularly within the scientific and research divisions have
recognized that sophisticated statistical and mathematic approaches
could be still more fully utilized to enhance the quality and accuracy
of Agency risk assessment and risk management.  Various stakeholders,
inside and outside the Agency, have called for a more comprehensive
characterization of risks, including uncertainties, in protecting more
sensitive or vulnerable populations and life stages.

Therefore, the Office of the Science Advisor of the EPA, together with
the Science Policy Council and members of the Risk Assessment Forum
(RAF), identified a need to examine the use of probabilistic approaches
in Agency risk assessment and risk management.  An RAF Technical Panel
developed papers (this paper and a managers' summary) which provide a
general overview of the value of probabilistic analyses and similar or
related methods, and some examples of current applications across the
Agency.

The goal of these papers is not only to describe potential and actual
uses of these tools in the risk decision process, but also to encourage
their further implementation in human, ecological and environmental risk
analysis and related decision making.  The enhanced use of probabilistic
analyses to characterize uncertainty in assessments would not only
reflect external scientific advice on how to further advance EPA risk
assessment science, but will also help to address specific challenges
faced by managers and improve confidence in Agency decisions

Kevin Teichman

Acting EPA Science Advisor

Acknowledgments

We would like to acknowledge the scientists and risk assessors who
assisted in the preparation of this paper.

Contributors and Reviewers of this Document

Chris Frey, John Paul, and Pasky Pascual (Leads)

Gary Bangs, Mike Clipper, Kathryn Gallagher, Bob Hetes, Michael Messner,
Keeve Nachman, Haluk Ozkaynak, Zachary Pekar, and Woodrow Setzer

The PRA Technical Panel

Table of Contents

  TOC \o "1-3" \h \z \u    HYPERLINK \l "_Toc221604244"  Using
Probabilistic Methods to Enhance the Role of Risk Analysis in
Decision-Making	  PAGEREF _Toc221604244 \h  4  

  HYPERLINK \l "_Toc221604245"  1. Introduction: Relevance of
Uncertainty to Decision making: How Probabilistic Approaches Can Help	 
PAGEREF _Toc221604245 \h  5  

  HYPERLINK \l "_Toc221604246"  1.0. What Is Probabilistic Risk
Analysis, and How Does It Address Variability and Uncertainty?	  PAGEREF
_Toc221604246 \h  5  

  HYPERLINK \l "_Toc221604247"  1.1. Goals and Intended Audience	 
PAGEREF _Toc221604247 \h  5  

  HYPERLINK \l "_Toc221604248"  1.2. Overview of This Document	  PAGEREF
_Toc221604248 \h  5  

  HYPERLINK \l "_Toc221604249"  1.3. What Are Common Challenges Facing
EPA Risk Decision makers?	  PAGEREF _Toc221604249 \h  6  

  HYPERLINK \l "_Toc221604250"  1.4. What Are Key Questions Often Asked
by Decision makers?	  PAGEREF _Toc221604250 \h  7  

  HYPERLINK \l "_Toc221604251"  1.5. Why Is the Implementation of
Probabilistic Risk Analysis Important?	  PAGEREF _Toc221604251 \h  7  

  HYPERLINK \l "_Toc221604252"  1.6. How Does EPA Typically Address
Scientific Uncertainty and Communicate Variability?	  PAGEREF
_Toc221604252 \h  8  

  HYPERLINK \l "_Toc221604253"  1.7. What Are the Limitations of Relying
on Default-Based Deterministic Approaches?	  PAGEREF _Toc221604253 \h  9
 

  HYPERLINK \l "_Toc221604254"  1.8. What Is EPA’s Experience with the
Use of Probabilistic Risk Analysis?	  PAGEREF _Toc221604254 \h  10  

  HYPERLINK \l "_Toc221604255"  2. Probabilistic Risk Analysis	  PAGEREF
_Toc221604255 \h  12  

  HYPERLINK \l "_Toc221604256"  2.1. What Are Variability and
Uncertainty, and How Are They Relevant to Decision-making?	  PAGEREF
_Toc221604256 \h  12  

  HYPERLINK \l "_Toc221604257"  2.1.1. Variability	  PAGEREF
_Toc221604257 \h  12  

  HYPERLINK \l "_Toc221604258"  2.1.2. Uncertainty	  PAGEREF
_Toc221604258 \h  12  

  HYPERLINK \l "_Toc221604259"  2.2. When Is Probabilistic Risk Analysis
Applicable or Useful?	  PAGEREF _Toc221604259 \h  13  

  HYPERLINK \l "_Toc221604260"  2.3. How Can Probabilistic Risk Analysis
Be Incorporated into Assessments?	  PAGEREF _Toc221604260 \h  13  

  HYPERLINK \l "_Toc221604261"  2.4. What Are the Scientific
Community’s Views on Probabilistic Risk Analysis, and What Is the
Institutional Support for Its Use?	  PAGEREF _Toc221604261 \h  14  

  HYPERLINK \l "_Toc221604262"  2.5. How Can Probabilistic Risk
Assessment Provide More Comprehensive, Rigorous Scientific Information
in Support of Regulatory Decisions?	  PAGEREF _Toc221604262 \h  14  

  HYPERLINK \l "_Toc221604263"  2.6. Are There Additional Advantages of
Using Probabilistic Risk Analysis?	  PAGEREF _Toc221604263 \h  15  

  HYPERLINK \l "_Toc221604264"  2.7. What Are the Challenges to
Implementation of Probabilistic Analyses?	  PAGEREF _Toc221604264 \h  15
 

  HYPERLINK \l "_Toc221604265"  2.8. How Can Probabilistic Risk Analysis
Support Specific Regulatory Decision making?	  PAGEREF _Toc221604265 \h 
16  

  HYPERLINK \l "_Toc221604266"  2.9. Does Probabilistic Risk Analysis
Require More Resources Than Default-Based Deterministic Approaches?	 
PAGEREF _Toc221604266 \h  16  

  HYPERLINK \l "_Toc221604267"  2.10. Doesn’t Probabilistic Risk
Analysis Require More Data Than Conventional Approaches?	  PAGEREF
_Toc221604267 \h  17  

  HYPERLINK \l "_Toc221604268"  2.11. Can Probabilistic Risk Analysis Be
Used To Screen Risks or Only in Complex or Refined Assessments?	 
PAGEREF _Toc221604268 \h  17  

  HYPERLINK \l "_Toc221604269"  2.12. Does Probabilistic Risk Analysis
Present Unique Challenges to Model Evaluation?	  PAGEREF _Toc221604269
\h  17  

  HYPERLINK \l "_Toc221604270"  2.13. How Do You Communicate Results of
Probabilistic Risk Analysis?	  PAGEREF _Toc221604270 \h  18  

  HYPERLINK \l "_Toc221604271"  2.14. Are the Results of Probabilistic
Risk Analysis Difficult To Communicate to Decision-makers and
Stakeholders?	  PAGEREF _Toc221604271 \h  19  

  HYPERLINK \l "_Toc221604272"  3. Findings and Recommendations	 
PAGEREF _Toc221604272 \h  21  

  HYPERLINK \l "_Toc221604273"  3.1 Findings: How Probabilistic Risk
Analysis and Related Analyses Can Improve the Decision-making Process at
EPA	  PAGEREF _Toc221604273 \h  21  

  HYPERLINK \l "_Toc221604274"  3.2. Recommendations for Enhanced
Utilization of PRA in EPA	  PAGEREF _Toc221604274 \h  21  

  HYPERLINK \l "_Toc221604275"  3.3 Guidance and Policy	  PAGEREF
_Toc221604275 \h  22  

  HYPERLINK \l "_Toc221604276"  3.4 Challenges	  PAGEREF _Toc221604276
\h  23  

  HYPERLINK \l "_Toc221604277"  Appendix A: An Overview of Some of the
Techniques Used in Probabilistic Risk Analysis	  PAGEREF _Toc221604277
\h  24  

  HYPERLINK \l "_Toc221604278"  A.1. What Is the General Conceptual
Approach in Probabilistic Risk Analysis?	  PAGEREF _Toc221604278 \h  24 


  HYPERLINK \l "_Toc221604279"  A.2. What Are the Multiple Types of
Probabilistic Risk Analyses, and How Are They Used?	  PAGEREF
_Toc221604279 \h  25  

  HYPERLINK \l "_Toc221604280"  A.3. What Are Some Specific Aspects of
and Issues Related to Methodology for Probabilistic Risk Analysis?	 
PAGEREF _Toc221604280 \h  27  

  HYPERLINK \l "_Toc221604281"  A.3.1. Developing a Probabilistic Risk
Analysis Model	  PAGEREF _Toc221604281 \h  27  

  HYPERLINK \l "_Toc221604282"  A.3.2. Conducting the Probabilistic
Analysis	  PAGEREF _Toc221604282 \h  27  

  HYPERLINK \l "_Toc221604283"  Appendix B: Glossary	  PAGEREF
_Toc221604283 \h  31  

  HYPERLINK \l "_Toc221604284"  Appendix C: References	  PAGEREF
_Toc221604284 \h  36  

  HYPERLINK \l "_Toc221604285"  Bibliography	  PAGEREF _Toc221604285 \h 
38  

  HYPERLINK \l "_Toc221604286"  Probabilistic Risk Analysis
Methodology(General	  PAGEREF _Toc221604286 \h  38  

  HYPERLINK \l "_Toc221604287"  Probabilistic Risk Analysis and Decision
Making	  PAGEREF _Toc221604287 \h  38  

  HYPERLINK \l "_Toc221604288"  Probabilistic Risk Analysis
Methodology(Specific Aspects	  PAGEREF _Toc221604288 \h  38  

  HYPERLINK \l "_Toc221604289"  Sensitivity Analysis	  PAGEREF
_Toc221604289 \h  39  

  HYPERLINK \l "_Toc221604290"  Case Study Examples of Probabilistic
Risk Analysis(EPA (see also the PRA Case Studies White Paper)	  PAGEREF
_Toc221604290 \h  39  

  HYPERLINK \l "_Toc221604291"  Case Study Examples of Probabilistic
Risk Analysis(Other	  PAGEREF _Toc221604291 \h  40  

  HYPERLINK \l "_Toc221604292"  Appendix D: Case Study Examples of the
Application of Probabilistic Risk Analysis in U.S. Environmental
Protection Agency Regulatory Decision-Making	  PAGEREF _Toc221604292 \h 
42  

  HYPERLINK \l "_Toc221604293"  1. Introduction	  PAGEREF _Toc221604293
\h  46  

  HYPERLINK \l "_Toc221604294"  2. Overall Approach to Probabilistic
Risk Analysis at the U.S. Environmental Protection Agency	  PAGEREF
_Toc221604294 \h  47  

  HYPERLINK \l "_Toc221604295"  2.1. U.S. Environmental Protection
Agency Guidance and Policies on Probabilistic Risk Analysis	  PAGEREF
_Toc221604295 \h  47  

  HYPERLINK \l "_Toc221604296"  2.2. Categorizing Case Studies	  PAGEREF
_Toc221604296 \h  47  

  HYPERLINK \l "_Toc221604297"  2.2.1. Group 1 Case Studies	  PAGEREF
_Toc221604297 \h  48  

  HYPERLINK \l "_Toc221604298"  2.2.2. Group 2 Case Studies	  PAGEREF
_Toc221604298 \h  49  

  HYPERLINK \l "_Toc221604299"  2.2.3. Group 3 Case Studies	  PAGEREF
_Toc221604299 \h  49  

  HYPERLINK \l "_Toc221604300"  3. Case Study Summaries	  PAGEREF
_Toc221604300 \h  53  

  HYPERLINK \l "_Toc221604301"  Group 1 Case Studies	  PAGEREF
_Toc221604301 \h  53  

  HYPERLINK \l "_Toc221604302"  Case Study 1: Sensitivity Analysis of
Key Variables in Probabilistic Assessment of Children’s Exposure to
Arsenic in Chromated Copper Arsenate (CCA) Pressure-Treated Wood	 
PAGEREF _Toc221604302 \h  53  

  HYPERLINK \l "_Toc221604303"  Case Study 2: Assessment of Relative
Contribution of Atmospheric Deposition to Watershed Contamination	 
PAGEREF _Toc221604303 \h  55  

  HYPERLINK \l "_Toc221604304"  Group 2 Case Studies	  PAGEREF
_Toc221604304 \h  57  

  HYPERLINK \l "_Toc221604305"  Case Study 3: Probabilistic Assessment
of Angling Duration Used in Assessment of Exposure to Hudson River
Sediments via Consumption of Contaminated Fish	  PAGEREF _Toc221604305
\h  57  

  HYPERLINK \l "_Toc221604306"  Case Study 4: Probabilistic Analysis of
Dietary Exposure to Pesticides for use in Setting Tolerance Levels	 
PAGEREF _Toc221604306 \h  58  

  HYPERLINK \l "_Toc221604307"  Case Study 5: One-Dimensional
Probabilistic Risk Analysis of Exposure to Polychlorinated Biphenyls
(PCBs) via Consumption of Fish from a Contaminated Sediment Site	 
PAGEREF _Toc221604307 \h  60  

  HYPERLINK \l "_Toc221604308"  Case Study 6: Probabilistic Sensitivity
Analysis of Expert Elicitation of Concentration-Response Relationship
Between Particulate Matter (PM2.5) Exposure and Mortality	  PAGEREF
_Toc221604308 \h  64  

  HYPERLINK \l "_Toc221604309"  Case Study 7: Environmental Monitoring
and Assessment Program (EMAP): Using Probabilistic Sampling Techniques
To Evaluate the Nation’s Ecological Resources	  PAGEREF _Toc221604309
\h  66  

  HYPERLINK \l "_Toc221604310"  Group 3 Case Studies	  PAGEREF
_Toc221604310 \h  66  

  HYPERLINK \l "_Toc221604311"  Case Study 8: Two-Dimensional
Probabilistic Risk Analysis of Cryptosporidium in Public Water Supplies,
with Bayesian Approaches to Uncertainty Analysis	  PAGEREF _Toc221604311
\h  68  

  HYPERLINK \l "_Toc221604312"  Case Study 9: Two-Dimensional
Probabilistic Model of Children’s Exposure to Arsenic in Chromated
Copper Arsenate (CCA) Pressure-Treated Wood	  PAGEREF _Toc221604312 \h 
70  

  HYPERLINK \l "_Toc221604313"  Case Study 10: Two-Dimensional
Probabilistic Exposure Assessment of Ozone	  PAGEREF _Toc221604313 \h 
72  

  HYPERLINK \l "_Toc221604314"  Case Study 11: Analysis of
Microenvironmental Exposures to Particulate Matter (PM2.5) for a
Population Living in Philadelphia, PA	  PAGEREF _Toc221604314 \h  75  

  HYPERLINK \l "_Toc221604315"  Case Study 12: Probabilistic Analysis in
Cumulative Risk Assessment of Organophosphorus Pesticides	  PAGEREF
_Toc221604315 \h  77  

  HYPERLINK \l "_Toc221604316"  Case Study 13: Probabilistic Ecological
Effects Risk Assessment Models for Evaluating Pesticide Uses	  PAGEREF
_Toc221604316 \h  79  

  HYPERLINK \l "_Toc221604317"  Case Study 14: Expert Elicitation of
Concentration-Response Relationship Between Particulate Matter (PM2.5)
Exposure and Mortality	  PAGEREF _Toc221604317 \h  81  

  HYPERLINK \l "_Toc221604318"  Case Study 15: Expert Elicitation of
Sea-Level Change Resulting from Global Climate Change	  PAGEREF
_Toc221604318 \h  Error! Bookmark not defined.  

  HYPERLINK \l "_Toc221604319"  Case Study 16: Expert Elicitation for
Bayesian Belief Network Model of Stream Ecology	  PAGEREF _Toc221604319
\h  83  

  HYPERLINK \l "_Toc221604320"  4. References to Case Studies	  PAGEREF
_Toc221604320 \h  87  

  HYPERLINK \l "_Toc221604321"  List of Acronyms and Abbreviations	 
PAGEREF _Toc221604321 \h  88  

 

Using Probabilistic Methods to Enhance the Role of Risk Analysis in
Decision-Making

Executive Summary

Probabilistic risk assessment (PRA) is a group of techniques that
provide estimates of the range and likelihood of hazard, exposure or
risk, rather than a single point estimate.  Stakeholders, inside and
outside the Agency, have recommended a fuller characterization of risks,
including uncertainties, in protecting more sensitive or vulnerable
populations and life stages.

The goal of this white paper is to explain how EPA can achieve broader
use of probabilistic methods and address uncertainty and variability by
capitalizing on the wide array of tools and methods that comprise PRA. 
The information contained in this document is intended for both risk
analysts and managers faced with determining when and how to apply these
tools in their decisions.  This paper begins with a decision-maker’s
perspective, proceeds to a more technical discussion, and finally gives
a number of illustrative examples of actual EPA applications of
probabilistic analyses.  

The white paper describes challenges faced by EPA decision makers,
defines and explains the basic principles of probabilistic analysis,
briefly highlights instances where these techniques have been used in
EPA decisions, and describes criteria that may be useful in determining
whether application of probabilistic methods may be useful and/or
applicable to a specific decision.  The white paper also describes
commonly employed methods to address variability and uncertainty,
including those used in the consideration of uncertainty in scenarios,
uncertainty in models, and variability and uncertainty in the inputs and
outputs of models.  A general description is provided of the range of
methods from simple to complex, rapid to more time-consuming, and least
to most resource-intensive, and opportunities for utilization.  More
detailed examples of applications of these methods are provided, in
Appendix D titled “Case Study Examples of the Application of
Probabilistic Risk Analysis in U.S. Environmental Protection Agency
Decision Making.”

This document does not prescribe a specific approach but, rather,
describes the various stages and aspects of an assessment or decision
process in which probabilistic assessment tools may add value.  This
white paper provides answers to common questions regarding PRA,
including key concepts such as scientific and institutional motivations
for use of PRA, and challenges in the application of probabilistic
techniques.  The white paper describes how PRA can both enhance the
Agency’s credibility and improve decision making.

1. Introduction: Relevance of Uncertainty to Decision making: How
Probabilistic Approaches Can Help

1.0. What Is Probabilistic Risk Analysis, and How Does It Address
Variability and Uncertainty?  

Probabilistic analyses include techniques that can be applied formally
to address both variability and uncertainty. Probability is used in
sciences, business, economics, and other fields to examine existing data
and estimate the chance of an event, from health effects to rain to
metal fatigue. One can use probability (chance) to quantify the
frequency of occurrence or the degree of belief in information. For
variability, probability distributions are interpreted as representing
the relative frequency of a given state of the system (i.e., that the
data are distributed a certain way), whereas, for uncertainty, they
represent the degree of belief or confidence that a given state of the
system exists (i.e., that we have the appropriate data) (Cullen and
Frey, 1999). PRA often is defined narrowly to mean a statistical or
thought process used to analyze and evaluate the variability of
available data or to look at uncertainty across data sets.

For the purposes of this document, PRA is a term used to describe a
process that uses probability to incorporate the variability in data
sets, and/or the uncertainty in information such as data or models, into
analyses that support environmental risk-based decision making. PRA is
used here broadly to include both quantitative and qualitative methods
for dealing with scenario, model, and input uncertainty. Probabilistic
techniques can be used with other types of analysis, such as
benefit-cost analysis, regulatory impact analysis, and engineering
performance standards and, thus, is used for a variety of applications
and by experts in many disciplines.

1.1. Goals and Intended Audience

The goals of this white paper are to introduce probabilistic analysis
(PRA) and how it can be used to better inform and improve the
decision-making process, and to provide case  studies where it has been
used in human health and ecological analyses at EPA (Appendix D). A
secondary goal of this paper is to bridge communication gaps regarding
PRA among analysts of various disciplines, between these analysts and
Agency decision makers, and affected stakeholders. The white paper is
also intended to serve as a communication tool to help introduce key
concepts and background information on approaches to risk analysis that
incorporate uncertainty and provide a more comprehensive treatment of
variability. Risk analysts, risk managers, and affected stakeholders can
benefit from understanding the potential uses of PRA. PRA and related
approaches can be used to identify further research that can decrease
uncertainty and more thoroughly characterize variability in a risk
assessment. This white paper will explain how PRA is well suited to
enhancing the decision-making processes in EPA by addressing inherent
uncertainties faced by managers involved in that process.

1.2. Overview of This Document

This document reviews EPA’s interest in and experience with addressing
uncertainty and variability using probabilistic methods; identifies key
questions asked or faced by Agency decision makers; shows how
conventional deterministic approaches to risk analysis may not answer
these questions fully; provides examples of applications; and shows how
and why “probabilistic risk analysis” (broadly defined) provides
added value with regard to regulatory decision making by more fully
characterizing risk estimates. For the purposes of this white paper, PRA
and related tools for both human health and ecological assessments
include a range of approaches from statistical tools, such as
sensitivity analysis, to multi-dimensional Monte Carlo models,
geospatial approaches, and expert elicitation. Key points addressed by
this document include definitions and key concepts pertaining to PRA,
the need for PRA, benefits and challenges of PRA, a general conceptual
framework for PRA, conclusions regarding products and insights obtained
from PRA, and examples where EPA has used PRA in human health and
ecological analyses. A glossary and a bibliography are provided.

1.3. What Are Common Challenges Facing EPA Risk Decision makers?

EPA decision makers face scientifically complex problems that are
compounded by varying levels of uncertainty and variability.  In
reality, uncertainty in risk decisions is unavoidable, since we cannot
perfectly model or predict real world situations, but uncertainty can be
reduced or better characterized through knowledge.  Variability is
inherent in natural systems, and therefore cannot be reduced, but can
also be examined and described. Decision makers often want to know who
is at risk and by how much, the tradeoffs between alternative actions or
decisions, and the likely or possible consequences of decisions. To this
end, it is particularly useful to decision makers to understand the
distribution of risk across potentially impacted populations and
ecological systems. It can be important to know the number of
individuals experiencing different magnitudes of risk, the differences
in risk magnitude experienced by individuals in different life stages or
populations, or the probability of an event which may lead to
unacceptable levels of risk. Given the limitations of data, traditional
methods of risk analyses are not well suited to produce such estimates.
Probabilistic analytical methods are capable of addressing these
shortcomings and can contribute to a more thorough recognition of the
impact of data gaps on the projected risk estimates.

A defensible decision process explicitly takes into account
uncertainties and variability and the rationale or factors influencing
how a decision maker addresses these. Factors such as economics, equity,
feasibility, stakeholder input, and other considerations may also be
part of the decision-making process. Decision making typically contend
with several key factors, including multiple, conflicting objectives,
uncertainty, and alternative regulatory options available to a decision
maker. In addition, decision analysis provides a theoretical foundation
for estimating the value of collecting more information to allow for
more informed decisions.  In the face of uncertainty, decision making is
determined not only by science but also by Agency policy.  Where not
prohibited by statute, the relative costs and benefits of regulatory
alternatives may be considered in making decisions.

If uncertainty and variability have not been well characterized or
acknowledged, potential complications arise in the process of
decision-making that seeks to achieve a balance between over- and
under-regulation. Increased uncertainty can make it more difficult to
determine with reasonable confidence the balance point between costs of
regulation and the implications for avoiding damages and producing
benefits. Characterization of these factors, facilitated by
probabilistic analyses, can provide insight in weighing the relative
costs and benefits of varying levels of regulation and also assist in
risk communication activities.

1.4. What Are Key Questions Often Asked by Decision makers?

Determining decision-maker concerns is a critical first step toward
developing a useful and responsive risk assessment. For example, the
appropriate focus and level of detail of the analysis should be
commensurate with decision maker and stakeholder needs as well as
appropriate use of science. Often, analyses are conducted at a level of
detail dictated by the issue being addressed, the breadth and quality of
the available information upon which to base an analysis, and the
significance surrounding this decision. The analytical process tends to
be an iterative one, and, even though a guiding set of questions may
frame the initial analyses, additional questions can arise that further
direct or even reframe the analyses.

Based on a series of discussions with Agency decision makers and risk
assessors, some questions that are typically posed about risk analyses
include the following:

How representative or conservative is the estimate, (e.g., what is the
variability around an estimate)?

What are the major gaps in knowledge, and what are the major assumptions
used in the assessment? How reasonable are the assumptions?

Would my decision be different if the data were different?  Would
additional data collection and research likely lead to a different
decision? How long will it take to collect the information, how much
would it cost, and would the resulting decision be significantly
different?

Will the use of additional resources, such as a probabilistic approach,
impact the decision making in a timely manner (i.e., better characterize
uncertainties, better identify variability, impact timelines, etc.)?

What are the liabilities/consequences of making a decision under the
current level of  knowledge and uncertainty?  

What is the percentile of the population to be protected?

How do the different alternative decision choices and the interpretation
of uncertainty and variability impact the target population?

The questions that arise concerning analyses, including PRA, change
depending on the stage and nature of the decision making and analysis,
from planning and scoping through risk management. The utility of
various levels of analysis and levels of sophistication in answering
these questions are illustrated in the case studies section described in
Section 1.8 and presented in Appendix D.

1.5. Why Is the Implementation of Probabilistic Risk Analysis Important?

The principal reason for the inclusion of PRA as an option in the risk
assessor’s toolbox is PRA’s ability to refine and improve the
information leading to decision making by incorporating known
uncertainties. Beginning as early as the 1980s (NRC, 1983), expert
scientific advisory groups have been recommending that risk analyses
include a clear discussion of the uncertainties in risk estimation. The
National Research Council (NRC, 1994) stated the need to describe
uncertainty and to capture variability in risk estimates. The
Presidential/Congressional Commission on Risk Assessment and Risk
Management (PCRARM, 1997) recommended against a requirement or need for
a “bright line,” or single number, level of risk (see Section 2.5
for more on the scientific community’s opinion on use of PRA).
Regulatory science often requires selection of a limit for a
contaminant, yet that limit always contains uncertainty as to how
protective it is. PRA and related tools quantitatively describe the very
real variations in natural systems and living things and how they
respond to stressors and the uncertainty in estimating those responses.
Risk characterization became EPA policy in 1995, and the principles of
transparency, clarity, consistency, and reasonableness are explicated in
the 2000 Risk Characterization Handbook (EPA, 2000). Transparency,
clarity, consistency, and reasonableness criteria require decision
makers to describe and explain the uncertainties, variability, and known
data gaps in the risk analysis and how they affected resulting
decision-making processes (USEPA, 2000, 1992, 1995). 

The use of probabilistic methods also has received support from some
decision makers within the Agency, and these methods have been
incorporated into a number of Agency decisions to date. Program offices
such as the Offices of Pesticides, Solid Waste and Emergency Response,
Air and Radiation, and Water, as well as the Office of Research and
Development have utilized probabilistic approaches in different ways and
to varying extents, for both human exposure and ecological risk
analyses. In addition, the Office of Solid Waste and Emergency Response
has provided explicit guidance on the use of probabilistic approaches
for exposure analysis (EPA, 2001). Some program offices have held
training sessions on Monte Carlo simulation (MCS) software frequently
used in probabilistic analyses.

Where it is useful to refine risk estimates, the use of PRA can help in
the characterization and communication of uncertainty, variability, and
the impact of data gaps in risk analyses, for assessors, decision
makers, and stakeholders including the target population.

1.6. How Does EPA Typically Address Scientific Uncertainty and
Communicate Variability?

Environmental assessments can be complex, such as exposures to multiple
chemicals in multiple media for a wide ranging population. The Agency
often has developed simplified approaches to characterizing risks
through the use of point estimates for model variables or parameters.
Such an approach typically produces point estimates of risks (e.g., 10-5
lifetime probability of cancer risk for an individual). These are often
called “deterministic” assessments. As a result of the use of point
estimates for variables in model algorithms, deterministic risk results
usually are reported as what are assumed to be either average or
worst-case estimates and do not contain any quantitative estimate of the
uncertainty in that estimate, nor report to what percentile of the
exposed population the estimate applies. However, the methods typically
used in EPA risk assessments rely on a combination of point values with
potentially varying levels of conservatism and certainty, yielding a
point estimate of exposure that may be at some unknown point in the
range of possible risks.

Because   SEQ CHAPTER \h \r 1 uncertainty is inherent in all risk
assessments, it is important that the risk assessment process handle
uncertainties in a logical way that is transparent and scientifically
defensible, consistent with the Agency’s statutory mission, and
responsive to the needs of decision makers (NRC, 1994). Thus, when data
are missing, EPA often uses several options to provide some bounds on
uncertainty and variability, in an attempt to avoid risk
underestimation; attempting to give a single quantification of how much
confidence there is in the risk estimate may not be informative or
feasible.  For example, in exposure assessment, the practice at EPA is
to collect new data where needed and where time and resources allow.
Alternatives include narrowing the scope of the assessment, using
screening level default assumptions that include upper-end values and/or
central tendency values that are generally combined to generate risk
estimates that fall within the higher end of the population risk range
(USEPA, 2004), using models to estimate missing values, using surrogate
data (e.g., data on a parameter that come from a different region of the
country than the region being assessed), or using professional judgment.
The use of individual assumptions can range from qualitative (e.g.,
assuming one is tied to the residence location and does not move through
time or space) to more quantitative (e.g., using the 95th percentile of
a sample distribution for an ingestion rate). This approach also can be
applied to the practice of hazard identification and dose-response
assessment when data are missing. Identifying the sensitivity of
exposure or risk estimates to key inputs can help in focusing efforts to
reduce uncertainty by collecting more data.

1.7. What Are the Limitations of Relying on Default-Based Deterministic
Approaches?

Deterministic risk analysis often is considered a traditional approach
to risk analysis because of the existence of established guidance and
procedures regarding its use, the ease with which it can be performed,
and its limited data and resource needs. The use of defaults supporting
deterministic risk assessment provides a procedural consistency that
allows for risk assessments to be feasible and tractable. Risk managers
and members of the public tend to be relatively familiar with
deterministic risk assessment, and use of such an approach addresses
assessment-related uncertainties primarily through the incorporation of
predetermined default values and conservative assumptions. It addresses
variability by combining input parameters intended to be representative
of typical or higher end exposure (considered to be conservative
assumptions). The intention is often to provide a margin of safety or to
construct a screening level estimate of high-end exposure and risk
(i.e., an estimate representative of more highly exposed and susceptible
individuals).

Deterministic risk assessment provides an estimation of exposures and
resulting risks that addresses uncertainties and variabilities in a
qualitative manner. The methods typically used in EPA deterministic risk
assessments rely on a combination of point values(some conservative and
some typical(yielding a point estimate of exposure that is at some
unknown point in the range of possible risks. Such an approach is
believed to more likely overestimate than underestimate risks. Although
this conservative bias (more likely to over- than underestimate risks)
aligns with the public health mission of EPA (USEPA, 2004), the degree
of conservatism in these risk estimates (and in any concomitant
decision) cannot be estimated well or communicated (Hattis and
Burmaster, 1994). This results in unquantified uncertainty in risk
statements.

Estimates generated using these methods are unaccompanied by
quantitative information regarding their precision or potential
systematic error and do not account for the distribution of exposures,
effects, and resulting risks across different members of an exposed
population. Although deterministic risk assessments may present
qualitative information regarding the robustness of the estimates, the
impact of data and model limitations on the quality of the results
cannot be quantified. Reliance on deterministically derived estimations
of risk can result in decision making based solely on point estimates
with an unknown degree of conservatism, which can complicate comparison
of risks or management options.   SEQ CHAPTER \h \r 1 The use of
conservative defaults has long been the target of criticism and has led
to the presumption by critics that EPA assessments are overly
conservative and unrealistic. This criticism may reduce the overall
perceived credibility of an Agency decision.

Deterministic risk assessment is not as well suited as PRA for more
complex assessments, including those of aggregate and cumulative
exposures and time-dependent individual exposure, dose, and effects
analyses. Identification and prioritization of contributory sources of
uncertainty can be difficult and time consuming when using deterministic
methods, leading to difficulties in model evaluation and the subsequent
appraisal of risk estimates (Cullen and Frey, 1999). These comprehensive
quantitative analyses of model sensitivities are essential for the
prioritization of key uncertainties(a critical step in identifying steps
for data collection or research to improve exposure or risk estimates.

1.8. What Is EPA’s Experience with the Use of Probabilistic Risk
Analysis?

To assist with the growing number of probabilistic analyses of exposure
data, EPA issued Guiding Principles for Monte Carlo Analysis (EPA,
1997). Probabilistic analysis techniques such as Monte Carlo analysis,
given adequate supporting data and credible assumptions, can be viable
statistical tools for analyzing variability and uncertainty in risk
assessments. The EPA policy for use of probabilistic analysis in risk
assessment, released in 1997, is inclusive of human exposure and
ecological risk assessments, but does not rule out probabilistic health
effects analyses. Subsequently, EPA’s Science Advisory Board (SAB) and
Scientific Advisory Panel have reviewed PRA approaches to risks by EPA
Offices such as Air and Radiation, Pesticides, and others. Several
programs have developed specific guidance on use of PRA, including
Pesticides and Solid Waste and Emergency Response (EPA, 1998, 2001).

To illustrate the practical application of PRA to problems relevant to
the Agency, several example case studies are briefly described here.
Appendix D, Case Study Examples of the Application of Probabilistic Risk
Analysis in U.S. Environmental Protection Agency Regulatory Decision
Making, discusses these and other case studies in greater detail,
including the procedures and outcome. The examples are intended to
illustrate how some of EPA’s programs and offices currently utilize
PRA. They demonstrate how information from probabilistic analyses,
including  sensitivity analysis, MCS, and other techniques, were used in
decision making. Some of the approaches that are profiled can be used
easily in the planning and scoping of risk assessments and risk
management. Other more complex approaches are used to answer more
specific questions and provide a richer description of the risks. Most
show that PRA can improve or expand information generated by
deterministic methods. The case studies illustrate that the Agency
already has applied the science of PRA to ecological risk and human
exposure estimation and has begun using PRA to describe health effects.
Some of the applications have used existing “off the shelf”
software, whereas others have required significant effort and resources.
Once developed, however, some of the more complex models have been used
many times for different assessments. All have stood the test of
internal and external peer review. A list of the case study examples
presented in Appendix D are provided in  Table 1 including
categorizations based on type of assessment (i.e., human health or
ecological risk assessment); PRA tools used in the assessment; and
program office or region responsible for the assessment.  In several
cases, the examples presented represent components of the overall risk
assessment that demonstrate use of multiple PRA techniques.

A few examples that illustrate the variety of PRA uses in EPA are: 

Hudson River PCB-Contaminated Sediment Site: Region 2 evaluated the
variability in risks to anglers who consume recreationally caught fish
contaminated with PCBs from sediment contamination in the Hudson River.
(Case Study 5)

EMAP program:  The Office of Research and Development (ORD) developed
and Office of Water (OW) adopted an applied probabilistic sampling
techniques to evaluate nation’s aquatic resources under CWA Section
305(b) (Case Study 7)

Chromated Copper Arsenate Risk Assessment: ORD and the Office of
Pesticide Programs (OPP) conducted a probabilistic exposure assessment
of children’s exposure (addressing both variability and uncertainty)
to arsenic and chromium from contact with CCA-treated wood playsets and
decks. (Case Study 9)

Evaluating Ecological Effects of Pesticide Uses: OPP developed a
probabilistic model which evaluates acute mortality levels in generic
and specific ecological species for user-defined pesticide uses and
exposures.  (Case Study 13)

PM2.5 Health Impacts: The Office of Air and Radiation (OAR) used expert
elicitation to more completely characterize, both qualitatively and
quantitatively, the uncertainties associated with the relationship
between reduction in PM2.5 and benefits of reduced PM2.5-related
mortality.  (Case Study 14)

2. Probabilistic Risk Analysis

2.1. What Are Variability and Uncertainty, and How Are They Relevant to
Decision-making?

The concepts of variability and uncertainty are introduced here, and the
relevance of these concepts to decision making is discussed.

2.1.1. Variability

Variability refers to real differences over time, space, or members of a
population and is a property of the system being studied (e.g., drinking
water rates for each of the many individual adult residents living in a
specific location) (Cullen and Frey, 1999). Variability can arise from
inherently random processes, such as variations in wind speed over time
at a given location or from true variation across members of a
population that, in principle, could be explained but that, in practice,
may not be explainable using currently available models or data (e.g.,
the range of blood lead levels in 6-year-old children following a
specific degree of lead exposure). Of particular interest in human
health risk assessment is inter-individual variability, which typically
refers to differences between members of the same population in either
behavior related to exposure (e.g., dietary consumption rates for
specific food items) or biokinetics related to chemical uptake or toxic
response (e.g., gastrointestinal uptake rates for lead following
intake).

2.1.2. Uncertainty

Uncertainty is the lack of knowledge of the true value of a quantity or
relationships among quantities. For example, there may be a lack of
information regarding the true distribution variability between
individuals for consumption of certain food items. There are a number of
types of uncertainties for both risk analyses and risk management. The
following description of types of uncertainty (drawn from Cullen and
Frey, 1999) addresses uncertainties that arise during risk analyses.
These uncertainties can be separated broadly into three categories: (1)
scenario uncertainty, (2) model uncertainty, and (3) input (or data)
uncertainty. Each of these is explained below.

Scenario uncertainty refers to errors, typically of omission, resulting
from incorrect or incomplete specification of the risk scenario to be
evaluated. The risk scenario refers to a set of assumptions regarding
the situation to be evaluated, such as (a) the specific sources of
chemical emissions or exposure to be evaluated (one industrial facility
or a cluster of varied facilities impacting the same study area), (b)
the specific receptor populations and associated exposure pathways to be
modeled (e.g., indoor inhalation exposure, track-in dust, or consumption
of home-produced dietary items), and (c) the times or activities to be
considered (e.g., exposure only at home, or consideration of workplace
or commuting exposure). Misspecification of the risk scenario can result
in underestimation, overestimation, or other mischaracterization of
risks.  For instance, underestimation may occur because of exclusion of
relevant situations or inclusion of irrelevant situations with respect
to a particular analysis.  Overestimation may occur because of the
inclusion of unrealistic or irrelevant situations (e.g., assuming
continuous exposure to an intermittent airborne contaminant source
rather than accounting for mobility throughout the day.) 

Model uncertainty refers to limitations in the mathematical models or
techniques that are developed to represent the system of interest and
often stems from: (a) simplifying assumptions, (b) exclusion of relevant
processes, (c) misspecification of model boundary conditions (i.e.,
range of input parameters), or (d) misapplication of a model developed
for other purposes. Model uncertainty typically arises when the risk
model relies on missing or improperly formulated processes, structures,
or equations.   Refer to the glossary for additional information.

Input or Parameter uncertainty typically refers to errors in
characterizing empirical values used as inputs to the model (e.g.,
engineering, physical, chemical, biological, or behavioral variables).
Input uncertainty can stem from random or systematic errors involved in
measuring a specific phenomenon (e.g., biomarker measurements such as
the concentration of mercury in human hair); from statistical sampling
errors associated with small sample sizes (if the data are based on
samples selected with a random, representative sampling design); from
the use of surrogate data instead of directly relevant data, or the
absence of an empirical basis for characterizing an input (e.g., absence
of measurements for fugitive emissions from an industrial facility); or
from the use of summary measures of central tendency rather than
individual observations. Nonlinear random processes can exhibit a
behavior that, for small changes in input values, produces large
variation in results.

2.2. When Is Probabilistic Risk Analysis Applicable or Useful?

PRA is useful in the following types of situations (Cullen and Frey,
1999).

When a screening level deterministic risk assessment indicates that
risks are possibly higher than a level of concern, and, therefore, a
more refined assessment is needed

When the consequences of using potentially biased point estimates of
risk are unacceptably high

To estimate the value of collecting additional information to reduce
uncertainty

When significant equity issues are raised by inter-individual
variability

To identify promising critical control points and critical levels when
evaluating risk management alternatives

To rank exposure pathways, sites, contaminants, and so on for purposes
of prioritizing model development or further research

PRA typically is not necessary in the following types of situations
(Cullen and Frey, 1999; EPA 1997).

When a screening-level deterministic risk assessment indicates that
risks are negligible, presuming that the assessment is known to be
biased to produce overestimates of risk

When the cost of averting the exposure and risk is smaller than the cost
of probabilistic analysis

When there is little uncertainty or variability in the analysis (This is
a rare situation.)

2.3. How Can Probabilistic Risk Analysis Be Incorporated into
Assessments?

As illustrated in the accompanying case studies (Appendix D),
probabilistic approaches can be incorporated into any stage of a risk
assessment, from problem formulation or planning and scoping to analysis
of alternative risk management decisions. In some situations, PRA can be
used selectively for components of an assessment. It is common in
assessments that some model inputs are known with high confidence (i.e.,
based on site-specific measurements), whereas values for other inputs
are less certain (i.e., based on surrogate data collected for a
different purpose). For example, an exposure modeler may determine that
there is relevant air quality monitoring data but a lack of detailed
information of human activity patterns in different microenvironments.
Thus, an assessment of variability in exposure to airborne pollutants
might be based on direct use of the monitoring data, whereas assessment
of uncertainty and variability in the inhalation exposure component
might be based on statistical analysis of surrogate data or use of
expert judgment. The uncertainties are likely larger for the latter than
the former component of the assessment; thus, efforts to characterize
uncertainties associated with pollutant exposures would focus on the
latter.

2.4. What Are the Scientific Community’s Views on Probabilistic Risk
Analysis, and What Is the Institutional Support for Its Use?

The NRC recently emphasized its long-standing advocacy for probabilistic
risk assessment (NRC, 2007a,b). Dating from its 1983 Risk Assessment in
the Federal Government (NRC, 1983)—which first formalized the risk
assessment paradigm—through various reports released in the late
1980s, all during the 1990s, and through the early 2000s, various NRC
panels have consistently maintained that—because risk analysis
involves substantial uncertainties—these uncertainties should be
evaluated within a risk assessment. These panels noted,

in 1989, that when evaluating total population risk, EPA should consider
the distribution of exposure and sensitivity of response in the
population (NRC, 1989);

in 1991, that when assessing human exposure to air pollutants, EPA
should present model results along with estimated uncertainties (NRC,
1991);

in 1993, that when conducting ecological risk assessments, EPA should
discuss thoroughly uncertainty and variability within the assessment
(NRC, 1993); and,

in 1994, an NRC report, Science and Judgment in Risk Assessment, stated
that “uncertainty analysis is the only way to combat the ‘false
sense of certainty,’ which is caused by a refusal to acknowledge and
[attempt to] quantify the uncertainty in risk predictions” (emphasis
in the original) (NRC, 1994). In 2002, another report suggested that
EPA’s estimation of health benefits were not wholly credible because
EPA failed to deal formally with uncertainties in its analyses (NRC,
2002). 

Asked to recommend improvements to the Agency’s human health risk
assessment practices, EPA’s Science Advisory Board echoed the NRC’s
sentiments and urged the Agency to characterize variability and
uncertainty more fully and more systematically and to replace
single-point uncertainty factors with a set of distributions using
probabilistic methods (Parkin and Morgan, 2007). The key principles of
risk assessment cited by the Office of Science and Technology Policy
(OSTP) and the Office of Management and Budget (OMB) include
“explicit” characterization of the uncertainties in risk judgments;
they go on to cite the National Academy of Science’s 2007
recommendation to address “variability of effects across potentially
affected populations” (OSTP/OMB, 2007).

2.5. How Can Probabilistic Risk Assessment Provide More Comprehensive,
Rigorous Scientific Information in Support of Regulatory Decisions?

External stakeholders in the past have used the Administrative Procedure
Act and the Data Quality Act to challenge the Agency for a lack of
transparency and consistency or for not fully analyzing and
characterizing the uncertainties in risk assessments or risk management
decisions (see Fisher et al., 2006). The more complete implementation of
PRA and related approaches to deal with uncertainties in decision-making
lends support to the overall Agency risk-based decision-making process.

The results of any assessment, including PRA, are dependent on the
underlying methods and assumptions. Accompanied by the appropriate
documentation, PRA may communicate a more robust representation of risks
and corresponding uncertainties. This characterization may take shape in
the form of a range of possible estimates as opposed to the more
traditionally presented single-point values. Depending on the needs of
the assessment, ranges can be derived for variability and uncertainty
(or a combination of the two) in both model inputs and resulting
estimations of risk.

2.6. Are There Additional Advantages of Using Probabilistic Risk
Analysis?

PRA quantifies how exposures, effects, and risks differ among
individuals and provides an estimation of the degree of confidence with
which these estimates may be made, given the current uncertainty in
scientific knowledge and available data. A 2007 NRC panel stated that
the objective of PRAs is not to decide “how much evidence is
sufficient” to adopt an alternative but, rather, to describe the
scientific bases of proposed alternatives so that scientific and policy
considerations may be more fully evaluated (NRC, 2007a). EPA’s SAB
similarly noted that PRAs provided more “value of information”
through quantitative assessment of uncertainty, and clarify the science
underlying Agency decisions (EPA, 2007).

The SAB proposed a number advantages Agency decision-makers could reap
from utilization of probabilistic methods (Parkin and Morgan, 2007).

A probabilistic reference dose could help reduce the potentially
inaccurate implication of zero risk below the RfD. 

By understanding and explicitly accounting for uncertainties underlying
a decision, EPA can estimate formally the value of gathering more
information. By doing so, two benefits follow: (1) EPA can better
prioritize its research needs by investing in areas that yield the
greatest information value; and (2) when making decisions, the Agency
can eschew the less-than-helpful rationale of “too much scientific
uncertainty.” By candidly acknowledging uncertainty’s ubiquity, EPA
can base a decision on more intellectually robust concepts of comparing
present risks against the costs of gathering more information.

By adopting PRA, EPA sends the appropriate signal to the intellectual
marketplace, thereby encouraging analysts to gather data and to develop
methodologies necessary for assessing uncertainties.

2.7. What Are the Challenges to Implementation of Probabilistic
Analyses?

Currently, EPA is using PRA in a variety of programs to support
decisions, but challenges remain regarding the expanded use of these
tools within the Agency; these include those that follow.

A lack of understanding of the value of PRA to decision making;

The perception, if not reality, that PRA requires additional resources;

Limited resources (staff, time, training, or methods) to conduct PRA;

A lack of clear directive or requirement to utilize PRA in many cases;

Lack of understanding of how to incorporate the results of probabilistic
analyses into decision making and how to establish action levels based
on the scope of the assessment.

The complexity of communicating probabilistic analysis results.

Fuller characterization of the uncertainties in a risk estimate through
use of PRA can lead to more difficult decision making or more
complicated risk communication

The ability to simulate and reanalyze numerous scenarios using various
models and input data could lead to prolonged analyses and delay
decisions

These challenges are discussed in more detail below.

2.8. How Can Probabilistic Risk Analysis Support Specific Regulatory
Decision making?

Decision makers sometimes perceive that the binary nature of regulatory
decisions (e.g. Does an exposure exceed a reference dose or not? Do
emissions comply with Agency standards or not?) precludes the use of a
range of uncertainty, compared with the use of point estimates.
Generally, it is legally necessary to explain the rationale underlying a
particular decision. PRA’s primary purpose is to provide information
so that decisions are based on the best available science; it is not to
necessarily displace legally mandated decisions with a range of
alternatives. By doing a sensitivity analysis of the influence of the
uncertainty on the decision-making process, it can be determined how or
if PRA can help improve the process.

2.9. Does Probabilistic Risk Analysis Require More Resources Than
Default-Based Deterministic Approaches?

PRA can generally be expected to require more resources than standard
Agency default-based deterministic approaches. There is extensive
experience within EPA in conducting and reviewing deterministic risk
assessments. These assessments tend to follow standardized methods that
minimize the effort required to conduct them and to communicate the
results. Probabilistic assessments often entail a more detailed
analysis, and, as a result, there exists a common perception that these
assessments require substantially more resources than do deterministic
approaches.

Appropriately trained staff and the availability of adequate tools,
methods, and guidance are essential for application of PRA. Proper
application of probabilistic methods requires not only software and data
but also guidance and training for both analysts using the tools and for
managers and decision makers tasked with interpreting and communicating
the results. In most circumstances, probabilistic assessments may take
more time and effort to conduct than conventional approaches, primarily
because of the comprehensive inclusion of available information on model
inputs.

An upfront increase in resources needed to conduct a probabilistic
assessment can be expected, but development of standardized approaches
and/or methods can lead to the routine incorporation of PRA in Agency
approaches (e.g., the Office of Pesticide Programs’ use of DEEM, a
probabilistic dietary exposure model). The initial and, in some cases,
ongoing resource cost (e.g., that for development of site-specific
models for site assessments) may be offset by a more informed decision
than a comparable deterministic analysis. Probabilistic methods are
useful for identifying effective risk management options and in
prioritizing additional data collection or research aimed at improving
risk estimation, ultimately resulting in management options that enable
improved environmental protection while, simultaneously, conserving
greater resources.

2.10. Doesn’t Probabilistic Risk Analysis Require More Data Than
Conventional Approaches?

There are differences in opinion within the technical community as to
whether PRA requires more data than other types of analyses. Although
some emphatically believe that PRA requires more data, others argue that
probabilistic assessments make better use of all of the available data
and information. Stahl et al. (2005) discuss when and how much data are
necessary for a decision. PRA can benefit from more data than might be
used in a deterministic risk assessment. For example, where
deterministic risk assessments may employ selected point estimates (such
as mean or 95th percentile values) from available datasets for use in
model inputs, PRA facilitates the use of frequency-weighted data
distributions, allowing for a more comprehensive consideration of the
available data. In many cases, the data that were used to develop the
presumptive 95th percentile can be employed in the development of
probabilistic distributions.

Restriction of PRA to data-rich situations may prevent its application
where it is most useful. Because PRA incorporates information on data
quality, variability, and uncertainty into risk models, the influence of
these factors on the characterization of risk can become a greater focus
of discussion and debate.

A key benefit of using PRA is its ability to reveal limitations, as well
as strengths, of data that often are masked by a deterministic approach.
In doing so, PRA can help inform research agendas, as well as support
regulatory decision making, based on the state of the best available
science. In summary, PRA typically requires more time for developing
input assumptions than a corresponding deterministic risk assessment,
but when incorporated in the relevant steps of the risk assessment
process, PRA can demonstrate real added benefits.  In some cases PRA can
provide additional interpretations that compensate for the additional
efforts.

2.11. Can Probabilistic Risk Analysis Be Used To Screen Risks or Only in
Complex or Refined Assessments?

Probabilistic methods typically are not necessary where traditional
default-based deterministic methods are adequate for screening risks.
Such methods are relatively low cost, are intended to produce
conservatively biased estimates, and are useful for identifying
situations in which risks are so low that no further action is needed.
The application of probabilistic methods can be targeted to situations
in which a screening approach indicates that a risk may be of concern or
when the cost of managing the risk is high, creating a need for
information to help inform risk management decisions. PRA fits directly
into a graduated hierarchical approach to risk analysis. PRA also could
be used to more fully examine the existing default-based methods based
on the current state of information and knowledge to determine if such
methods are truly conservative and adequate for screening.

2.12. Does Probabilistic Risk Analysis Present Unique Challenges to
Model Evaluation?

The concept of “validation” of models used for regulatory decision
making has been a topic of heated discussion. In a recent report on the
use of models in environmental regulatory decision making, NRC
recommended the use of the notion of model “evaluation” rather than
“validation,” suggesting use of a process that encompasses the
entire life cycle of the model and recognizes the spectrum of interested
parties in the application of the model, which often extends beyond the
model builder and decision maker. Such a process can be designed to
ensure that judgment of the model application is based not only on its
predictive value determined from comparison with historical data but
also on its comprehensiveness, rigor in development, transparency, and
interpretability (NRC, 2007b).

Model evaluation is important in all risk assessments. In the case of
PRA, there is an additional question as to the validity of the
assumptions made regarding probability and frequency distributions for
model inputs and their dependencies. Probabilistic information can be
accounted for during evaluation analyses by considering the range of
uncertainty in the model prediction and whether such a range overlaps
with the “true” value based on independent data. Thus, probabilistic
information can aid in characterizing the precision of the model
predictions and whether a prediction is significantly different from a
benchmark of interest.  For example, comparisons of probabilistic model
results and monitoring data were done for multiple models in developing
the cumulative pesticide exposure model.  There are also published
concurrent PRA model evaluations using a Bayesian analysis (Clyde,
2000).

2.13. How Do You Communicate Results of Probabilistic Risk Analysis?

The approaches for reporting results from PRA vary depending on the
assessment objective and the intended audience. Beyond the basic 1997
principles and the policy from the same year, the Risk Assessment
Guidance for Superfund: Volume III also provides some guidance on the
quality and criteria for acceptance as well as communication basics
(EPA, 2001). There have been limited studies of how information from PRA
regarding variability and uncertainty can or should be communicated to
key audiences such as decision makers and stakeholders (e.g., Morgan and
Henrion, 1990; Bloom et al., 1993; Krupnick et al., 2006). Among the
analyst community, there is often an interest in visualization of the
structure of a scenario and model using influence diagrams and depiction
of the variability and uncertainty in model inputs and outputs using
probability distributions in the form of cumulative density functions or
probability 

 

Figure 1. Graphical Description of the Likelihood (Probability of Risk)
of Toxicity 

(Fitted data distribution and confidence intervals)

Source: Frey, H.C. (2004)

distribution functions (Figure 1). Sensitivity of the model output to
variability and uncertainty in model inputs can be depicted using
graphical tools.

In some cases, these graphical methods can be useful for those less
familiar with PRA, but in many cases there is a need to translate the
quantitative results into a message that extracts the key insights
without burdening the decision maker with obscure technical details. In
this regard, the use of ranges of values for a particular metric of
decision-making relevance (e.g., range of uncertainty associated with a
particular estimate of risk) may be adequate. The presentation of PRA
results to a decision maker may be conducted best as an interactive
discussion, in which a principal message is conveyed, followed by
exploration of issues such as the source, quality, and degree of
confidence associated with the information. There is a need for
development of recommendations and a communication plan regarding how to
communicate the results of PRA to decision makers and stakeholders,
building on the experience of various programs and regions in this area.

2.14. Are the Results of Probabilistic Risk Analysis Difficult To
Communicate to Decision-makers and Stakeholders?

Research has shown that the ability of decision makers to deal with
concepts of probability and uncertainty is variable. Bloom et al. (1993)
surveyed a group of senior managers at EPA and found that many could
interpret information about uncertainty if it was communicated in an
appropriate manner that was responsive to decision-maker interests,
capabilities, and needs. In a more recent survey of ex-EPA officials,
Krupnick et al. (2006) concluded that most had difficulty understanding
information on uncertainty, and that certain formats used to present
uncertainty information were more effective than others. The findings of
these studies highlight the need for practical strategies for
communication of results of PRA and uncertainty information between risk
analysts and decision makers, as well as between decision makers and
other stakeholders. The Office of Emergency and Remedial Response has
compiled guidance to assist analysts and managers in understanding and
communicating the results of PRA (EPA, 2001).

3. Findings and Recommendations

3.1 Findings: How Probabilistic Risk Analysis and Related Analyses Can
Improve the Decision-making Process at EPA

PRA is an analytical methodology capable of incorporating information
regarding uncertainty and/or variability in analyses to provide insight
regarding the degree of certainty of a risk estimate and how the risk
estimate varies among different members of an exposed population.
Traditional approaches often report risks as “central tendency” or
“high end (e.g., 90th percentile or above),” or “maximum
anticipated exposure” and PRA can be used to more fully describe
uncertainty surrounding such estimates and identify the key contributors
to variability or uncertainty in predicted exposures or risk estimates.
This information then can be used by decision makers to achieve a
science-based level of safety, to weigh alternative risk management
options, or to invest in researching areas which have the greatest
uncertainty and impact on the risk estimates.

Using PRA, one can obtain insight regarding whether one risk management
strategy is more likely to reduce risks compared to another, and by how
much. The methodology facilitates the investigation of potential changes
in decisions that may result from the collection of additional
information that could better characterize variability and potentially
reduce uncertainty and helps determine how expenses incurred by
activities to reduce uncertainty are offset by improved decision-making
capabilities gained from the acquisition of that knowledge. PRA can
facilitate the construction and simultaneous consideration of multiple
model alternatives.  Probabilistic methods offer a number of tools
designed to promote robust management and increased confidence in
decision making through the incorporation of input variability and
uncertainty characterization and prioritization in risk analyses. For
example, sensitivity analyses can be used to identify influential
knowledge gaps involved in the estimation of risk, allowing for improved
transparency and the ability to more clearly communicate or articulate
the most relevant information to decision makers and stakeholders.
Ultimately, PRA can enhance the Agency’s credibility in its approach
to science-based decision making.

3.2. Recommendations for Enhanced Utilization of PRA in EPA

The various tools and methods discussed in this paper can be used at all
stages of risk analysis and also aid the decision-making process by
characterizing inter-individual variability and uncertainties.
Probabilistic analyses and related methods are in use in varying degrees
across the Agency:

The use of Monte Carlo or other probability based techniques to derive a
range of possible outputs from uncertain inputs is a fairly
well-developed approach within EPA. 

Although basic guidance exists at EPA on the use and acceptability of
PRA for risk estimation, implementation varies greatly within programs,
offices and regions.

Although highly sophisticated human exposure assessment and ecological
risk applications have been developed, use of PRA models to assess human
health effects and dose-response has been somewhat limited.

Enhanced use of PRA and consistent applications in support of EPA
decision-making requires improved internal capacity for conducting these
assessments, as well as interpreting and communicating such information
in the context of decisions.  Such improvements of internal capacity
could be accomplished through sharing of experiences, knowledge, and
training, improved policy and guidance, and increased availability of
tools and methods.    

Some steps to improve implementation include:

Inform risk managers about the advantages and disadvantages in using PRA
techniques in their decision-making process through lectures, webinars,
and communication regarding the techniques and their use in EPA.

Train risk assessors so that they can learn about the various tools
available, their applications, software and review considerations, and
resources for additional information (e.g., experts and support services
within the Agency).   

Meetings and discussions of PRA techniques and their application with
both managers and assessors will aid in providing greater consistency
and transparency to EPA’s risk assessment process and in developing
EPA’s internal capacity. 

Demonstrate through informational opportunities and resource libraries
the various tools and methods that can be used at all stages of risk
analysis and also aid the decision-making process by characterizing
inter-individual variability and uncertainties. 

Promote the sharing of experience, knowledge, models, and best practices
via meetings of risk assessors and risk managers; electronic exchanges,
such as the EPA Portal Environmental Science Connector; and more
detailed discussions regarding the case studies.  

Provide easily available, flexible, modular training for all levels of
experience to familiarize employees to the menu of tools and their
capacities.

Provide introductory as well as advanced training open to all offices.

Provide live and recorded seminars and Webinars for introductory and
supplemental education, as well as periodic, centralized hands-on
training sessions on how to utilize software programs. 

Risk assessors, risk managers, and decision makers need to be provided
the information and training necessary so that they can better utilize
these tools. Education and experience will generate familiarity with
these tools that will then lead analysts and decision makers to better
understand the techniques and consider more fully utilizing these
techniques.

3.3 Guidance and Policy

Additional guidance can be developed to help analysts and decision
makers decide which statistical tools to use and when to use them, and
how probabilistic information can help to inform the basis of those
decisions. Both deterministic and probabilistic approaches and other
statistical methods may be useful at any stage of the risk analysis and
decision-making process, from planning and scoping to characterizing and
communicating uncertainty. Such bodies as the EPA’s Science Policy
Council can play a role in directing guidance development to help
implement probabilistic and related tools.  Examples of guidance needed
include:

Probabilistic approaches to evaluating health effects data

Probabilistic approaches to ecological risk assessment

Integrating probabilistic exposure and risk estimates and communicating
uncertainty and variability.

3.4 Challenges

In general terms, while PRA techniques are currently available which
would help inform EPA decision-making process, research, as well as
guidance, is needed to further improve these methods for more complete
implementation of PRA in human health and ecological risk assessment.  
Some examples include:

Although highly sophisticated human exposure assessment and ecological
risk applications have been developed, use of PRA models to evaluate
toxicity data has been very limited. Scientific, technical, and science
policy discussions are needed in this area.

There is no consensus on any one well-accepted general methodology for
dealing with model uncertainty, although there are various examples of
efforts to do so. Thus, additional research on formal methods for
treating model uncertainties will be valuable. 

As noted in Appendix A.3, there are significant challenges to properly
account for variability and uncertainty when multiple models are coupled
together to represent the source-to-outcome continuum (e.g., the
OPP-Environmental Fate and Effects Division’s aquatic and terrestrial
models). Moreover, the coupling of multiple models (e.g., emissions, air
quality, exposure, dose, effect) may need to involve inputs and
corresponding uncertainties that are incorporated into more than one
model, potentially resulting in complex dependencies (e.g., ambient
temperature affects emission rates, air quality, and human activity that
influence total emissions and exposures). 

There may be mismatches in the temporal and spatial resolution of each
model, which confound the ability to propagate variability and
uncertainty from one model to another. For some models, the key
uncertainties may be associated with inputs, whereas, for other models,
the key uncertainties may be associated with structure or
parameterization alternatives

Appendix A: An Overview of Some of the Techniques Used in Probabilistic
Risk Analysis

A.1. What Is the General Conceptual Approach in Probabilistic Risk
Analysis?

PRA includes several major steps, which parallel the accepted
environmental health risk assessment process. These include (1) problem
and/or decision criteria identification, (2) getting information, (3)
interpreting the information, (4) selecting and applying models and
methods for quantifying variability and/or uncertainty, (5) quantifying
inter-individual or population variability and uncertainty in metrics
relevant to decision-making, (6) sensitivity analysis to identify key
sources of variability and uncertainty, and (7) reporting of results.

Problem identification deals with identifying the assessment end points,
or issues, that are relevant to the decision-making process, as well as
to other stakeholders, and that can be addressed in a scientific
assessment process. Following problem identification, information is
needed from stakeholders and experts regarding the scenarios to
evaluate. Based on the scenarios and assessment endpoints, the analysts
select or develop models, which in turn leads to identification of model
input data requirements and acquisition of data or other information
(e.g., expert judgment encoded as the result of a formal elicitation
process) from which to quantify inputs to the models. The data or other
information for model inputs is interpreted in the process of developing
probability distributions to represent variability, uncertainty, or both
for a particular input. Thus, the steps (1) through (4) listed above are
highly interactive and iterative in that the data input requirements and
how information is to be interpreted depend on the model formulation,
which depends on the scenario, which in turn depends on the assessment
objective. The assessment objective may have to be refined depending on
the availability of information.

Once a scenario, model, and inputs are specified, the model output is
estimated. A common approach is to use Monte Carlo or other
probabilistic methods for generating samples from the probability
distributions of each model input, run the model based on one random
value from each probabilistic input, and produce one corresponding
estimate of the model outputs. This process is repeated typically
hundreds or thousands of times to create a synthetic statistical sample
of model outputs. These output data are interpreted as a probability
distribution of the output of interest. Sensitivity analysis can be
performed to determine which model input distributions are most highly
associated with the range of variation in the model outputs. The results
may be reported in a wide variety of forms depending on the intended
audience, ranging from qualitative summaries to tables, graphs, and
diagrams.

A.2. What Are the Multiple Types of Probabilistic Risk Analyses, and How
Are They Used?

There are multiple levels for conducting risk assessments. Graduated
approaches to analysis are widely recognized (e.g., EPA, 1997; EPA,
2001; WHO, 2007). The idea of a graduated approach is to choose a level
of detail and refinement for an analysis that is appropriate to the
assessment objective, data quality, information available, and
importance of the decision (e.g., resource implications).

Detailed introductions to PRA methodology are available elsewhere, such
as Ang and Tang (1984), Cullen and Frey (1999), EPA (2001), Morgan and
Henrion (1990) and EPA, 2001. A few key aspects of PRA methodology are
briefly mentioned here. However, readers who seek more detail should
consult these references and see the bibliography for additional
references.

The deterministic risk assessment approaches described in Section 1.6
are examples of lower levels in a graduated approach to analysis, in
which risk at the lower levels of analysis is assessed by conservative,
bounding assumptions. If the risk estimate is found to be very low
despite use of conservative assumptions, then there exists a great deal
of certainty that the actual risks to the population of interest for the
given scenario are below levels of concern and, thus, that no further
intervention is required, assuming the model specification is correct.
However, when a conservative deterministic risk assessment indicates
that a risk may be high, it is possible that the risk estimate is
biased, and the actual risk may be lower. In such a situation, depending
on the resource implications of risk management, it may be appropriate
to proceed with a more refined, or higher level, analysis. If the cost
of intervention is less than the cost of further analysis, then it may
be appropriate to simply proceed to the risk management decision as a
preventive measure that is also expedient. In some deterministic
assessments, for instance, for ecological risks, the assumptions are not
well assured of conservatism and the estimated  risks might be biased to
appear lower than the unseen actual risk

A more refined analysis could involve applications of deterministic risk
assessment methods but with alternative sets of assumptions intended to
characterize central tendency and reasonable upper bounds of exposure,
effects, and risk estimates, such that the estimates could be for an
actual individual in the population of interest (rather than a
hypothetical maximally exposed individual). However, such analyses are
not likely to provide quantification regarding the proportion of the
population at or below a particular exposure or risk level of concern,
uncertainties for any given percentile of the exposed population, nor
priorities among input assumptions with respect to their contributions
to variability and uncertainty in the estimates.

To more fully answer the questions often asked by decision-makers, the
analysis can be further refined by incorporating quantitative
comparisons of alternative modeling strategies (to represent structural
uncertainties associated with scenarios or models), by quantifying
ranges of variability and uncertainty in model outputs, and by providing
the corresponding ranges for model outputs of interest. When performing
probabilistic analyses such as these, choices are made regarding whether
to focus on quantification of variability only, uncertainty only, both
variability and uncertainty co-mingled (representing a randomly selected
individual), or variability and uncertainty distinguished (e.g., in a
two-dimensional depiction of probability bands for estimates of
inter-individual variability) (see Figure 2). The simultaneous but
distinct propagation of variability and uncertainty in a two dimensional
framework enables quantification of uncertainty in the risk for any
percentile of the population.  For example, one could estimate the range
of uncertainty in the risk faced by the median member of the population
or the 95th percentile member of the population. Such information can be
used by a decision maker (for example) to gauge the confidence that
should be placed in any particular estimate of risk, as well as to
determine whether additional data collection or information might be
useful to reduce the uncertainty in the estimates. The OPP assessment of
chromated copper arsenate treated wood (see Appendix D) used such an
approach.  

Figure 2: Diagrammatic comparison between three alternative
probabilistic approaches for the same exposure assessment. In option 1,
only variability is quantified. In option 2, both variability and
uncertainty are propagated together. In option 3, variability and
uncertainty are propagated separately. MC = Monte Carlo. 1D = one
dimensional; 2D = two dimensional.   Source: WHO (2008)

When conducting an analysis for the first time, it may not be known or
clear, prior to analysis, which components of the model or which model
inputs contribute the most to the estimate risk or its variability and
uncertainty. However, as a result of completing an analysis, the analyst
often gains insight into both strengths and weaknesses of the models and
input information. Probabilistic analysis and sensitivity analysis can
be used together to identify the key sources of quantified uncertainty
in the model outputs to inform decisions regarding priorities for
additional data collection. Ideally, time should be allowed for
collecting such information and refining the analysis to arrive at a
more representative and robust estimate of variability and uncertainty
in risk. Thus, the notion of iteration in developing and improving an
analysis is widely recommended.

The notion of iteration can be applied broadly to the risk assessment
framework. For example, a first effort to perform an analysis may lead
to insight that the assessment questions might be impossible to address,
or that there are additional assessment questions that may be equally or
more important. Thus, iteration can include reconsideration of the
initial assessment questions and the corresponding implications for
definition of scenarios, selection of models, and priorities for
obtaining data for model inputs. Alternatively, in a time-limited
decision environment, such probabilistic and sensitivity analyses may
offer insight into the effect of risk management options on risk
estimates.

A.3. What Are Some Specific Aspects of and Issues Related to Methodology
for Probabilistic Risk Analysis?

This section briefly touches on a few key aspects of PRA, model
development, and associated uncertainties. Detailed introductions to PRA
methodology are available elsewhere, such as Ang and Tang (1984), Cullen
and Frey (1999), EPA (2001), and Morgan and Henrion (1990). Readers who
seek more detail should consult these references and see the
bibliography for additional references.

A.3.1. Developing a Probabilistic Risk Analysis Model

There are a number of key issues that should be considered in developing
a PRA model. Some of these are outlined below.

Structural Uncertainty in Scenarios

A potentially key source of uncertainty in an analysis is the scenario,
which includes specification of pollutant sources, transport pathways,
exposure routes, timing and locations, geographic extent, and related
issues. As yet, there appears to be no formalized methodology for
dealing quantitatively with uncertainty and variability in scenarios.
Decisions regarding what to include or exclude from a scenario could be
recast as hypotheses regarding which agents, pathways,
microenvironments, and so on contribute significantly to the overall
exposure and risk of interest. In practice, however, the use of
qualitative methods tends to be more common, given the absence of a
formal quantitative methodology.

Coupled Models

For source-to-outcome risk assessments, it is often necessary to work
with multiple models, each of which represents a different component of
a scenario. For example, there may be separate models for emissions, air
quality, exposure, dose, and effects. Such models may have different
spatial and temporal scales. When conducting an integrated assessment,
there may be significant challenges and barriers to coupling such models
into one coherent framework. Sometimes, the coupling is done dynamically
in a software environment. In other cases, the output of one model might
be processed manually to prepare the information for input to the next
model. Furthermore, there may be feedbacks between components of the
scenario (e.g., poor air quality might affect human activity, which, in
turn, could affect both emissions and exposures) that are incompletely
captured or not included at all. Thus, the coupling of multiple models
can be a potentially significant source of structural uncertainty.

A.3.2. Conducting the Probabilistic Analysis

Quantifying Variability and Uncertainty in Model Inputs and Parameters

Once the models are selected or developed to simulate a scenario of
interest, attention typically turns to development of input data for the
model. There is a substantial amount of literature regarding the
application of statistical methods for quantifying variability and
uncertainty in model inputs and parameters based on empirical data
(e.g,. Ang and Tang, 1984; Cullen and Frey, 1999; Morgan and Henrion,
1990; EPA 2001). For example, a commonly used method for quantifying
variability in a model input is to obtain a sample of data, select a
type of parametric probability distribution model to fit to the data
(e.g., normal, lognormal, or other form), estimate the parameters of the
distribution based on the data, critique the goodness-of-fit using
graphical (e.g., probability plot) and statistical methods (e.g.,
Anderson-Darling, Chi-Square, or Kolmogorov-Smirnov tests), and choose a
preferred fitted distribution. This methodology can be adjusted to deal
with various types of data, such as data that are samples from mixtures
of distributions or that contain nondetected (censored) values.
Uncertainties can be estimated based on confidence intervals for
statistics of interest, such as mean values, or the parameters of
frequency distributions for variability.  Various texts and guidance
documents, both Agency and programmatic, describe these approaches,
including the Guiding Principles for Monte Carlo Analysis (EPA, 1997)
and the internet site learner.org.

The most commonly used method for estimating a probability distribution
in the output of a model, based on probability distributions specified
for model inputs, is Monte Carlo Simulation  (MCS) (Cullen and Frey,
1999; Morgan and Henrion, 1990). MCS is popular because it is very
flexible. MCS can be used with a wide variety of different types of
probability distributions as well as different types of models. The main
challenge for MCS is that it requires repetitive model calculations to
construct a set of pseudo-random numbers for model inputs and the
corresponding estimates for model outputs of interest. There are
alternatives to MCS that are similar but more computationally efficient,
such as Latin Hypercube Sampling (LHS). Techniques are available for
simulating correlations between inputs in both MCS and LHS. For models
with very simple functional forms, it may be possible to use exact or
approximate analytical calculations, but, in practice, such situations
are encountered infrequently.

There may be situations in which the data do not conform to a
well-defined probability distribution. For such situations, Markov Chain
Monte Carlo is an algorithm that samples the data iteratively and
randomly to estimate a so-called “likelihood function” (i.e., the
probability distribution and parameter estimates that provide the most
likely explanation of the data). The likelihood function is a key
component of   HYPERLINK  \l "_Glossary"  Bayesian  inference and,
therefore, serves as the basis for some of the analytical approaches to
variability and uncertainty described below.

The use of empirical data presumes that the data are a representative,
random sample. However, if there are known biases or other data quality
problems, or if there is a scarcity or absence of relevant data, then
reliance on available empirical data is likely to lead to misleading
inferences in the analysis. Alternatively, estimates of variability and
uncertainty can be encoded, using formal protocols, based on elicitation
of expert judgment (e.g., Morgan and Henrion, 1990). Elicitation of
expert judgment for subjective probability distributions is used in
situations where there are insufficient data to support a statistical
analysis of uncertainty but in which there is sufficient knowledge on
the part of experts to make an inference regarding uncertainty. For
example, EPA has recently conducted an expert elicitation study on the
concentration-response relationship between annual average ambient PM2.5
exposures and annual mortality (IEC, 2006; see also Case Studies 6 and
14). Subjective probability distributions that are based on expert
judgment can be “updated” with new data as they become available
using Bayesian statistical methods.

Structural Uncertainty in Models

There may be situations in which it proves useful to evaluate not just
the uncertainties in inputs and parameter values, but also uncertainties
regarding whether a model adequately captures, in a hypothesized,
mathematical, structured form, the relationship under investigation. A
qualitative approach to evaluating the structural uncertainty in a model
is to describe critical assumptions within a model, the documentation of
a model, or model quality. Quantitative approaches to evaluating
structural uncertainty in models are manifold. These include
parameterization of a general model that can be reduced to alternative
functional forms (e.g., Morgan and Henrion, 1990), enumeration of
alternative models in a probability tree (e.g., Evans et al., 1994),
comparing alternative models by evaluating likelihood functions (e.g.,
Royall, 1997; Burnham and Anderson, 2002), pooling results of model
alternatives using Bayesian updating (e.g., Hoeting et al., 1999), or
testing the causal relationships within alternative models using
Bayesian Networks (Pearl 2000).

Sensitivity Analysis: Identifying the Most Important Model Inputs

Sensitivity analysis is complementary to probabilistic methods. There
are many types of sensitivity analysis methods, including, for example,
simple techniques that involve changing the value of one input at a time
and assessing the effect on an output and statistical methods that
evaluate which of many simultaneously varying inputs contributes the
most to the variance of the model output. Sensitivity analysis can
answer the following key questions.

What is the impact of changes in input values on model output?

How can variation in output values be apportioned among model inputs?

What are the ranges of inputs associated with best or worst outcomes?

What are the key controllable sources of variability?

What are the critical limits (e.g., emission reduction target for a risk
management strategy)?

What are the key contributors to the output uncertainty?

Thus, sensitivity analysis can be used to inform decision making
regarding research priorities and risk management.

Probabilistic methods typically focus on the forward propagation of
uncertainty or variability in the input to a model with respect to
uncertainty or variability in a model output. However, once a
probabilistic analysis is completed, sensitivity analysis typically
takes the perspective of looking backwards to evaluate how much of the
variation in the model output is attributable to individual model inputs
(e.g., Frey and Patil, 2002; Mokhtari et al., 2006; Saltelli et al.,
2004).

Iteration

There are two major types of iteration in risk assessment modeling. One
is iterative refinement of the type of analysis, perhaps starting with a
relatively simple deterministic risk assessment as a screening step in
an initial level of analysis and proceeding to more refined types of
assessments as needed in subsequent levels of analysis. Examples of more
refined levels of assessment include application of sensitivity analysis
to deterministic risk assessment; the use of probabilistic methods to
quantify variability only, uncertainty only, or co-mingled variability
and uncertainty (to represent a randomly selected individual); or the
use of two-dimensional probabilistic methods for distinguishing and
simultaneously characterizing both variability and uncertainty.

The other type of iteration occurs within a particular level and
includes iterative efforts to formulate a model, obtain data, and
evaluate the model to prioritize data needs. For example, a model may
require a large number of input assumptions. To prioritize efforts of
specifying distributions for variability and uncertainty for model
inputs, it is useful to determine which model inputs are most
influential with respect to the assessment end point. Therefore,
sensitivity can be used based on preliminary assessments of ranges or
distributions for each model input to determine which inputs are the
most important to the assessment. Refined efforts to characterize
distributions then can be prioritized to the most important inputs.

Appendix B: Glossary

Analysis. Examination of anything complex to understand its nature or to
determine its essential features (WHO IPCS Risk Assessment Terminology)

Assessment. The analysis and transformation of data into policy-relevant
information that can assist decision making and action.

Assessment end point. 1. Quantitative or qualitative expression of a
specific factor or metric with which a risk may be associated, as
determined through an appropriate risk assessment. 2. An explicit
expression of the environmental value that is to be protected,
operationally defined by an ecological entity and its attributes. For
example, salmon are valued ecological entities; reproduction and age
class structure are some of their important attributes. Together, salmon
“reproduction and age class structure” form an assessment end point.

Bayesian probability.  An approach to probability, representing a
personal degree of belief that something will occur..

Critical control point. A controllable variable that can be adjusted to
reduce exposure and risk. For example, a critical control point might be
the emission rate from a particular emission source. The concept of
critical control point is from the hazard assessment and critical
control point concept for risk management that is used in space and food
safety applications, among others.

Critical limit. A numerical value of a critical control point at or
below which risk is considered to be acceptable.

Ecological risk assessment. An ecological risk assessment evaluates the
potential adverse effects that human activities have on the plants and
animals that make up ecosystems. The risk assessment process provides a
way to develop, organize, and present scientific information, so that it
is relevant to environmental decisions. When conducted for a particular
place, such as a watershed, the ecological risk assessment process can
be used to identify vulnerable and valued resources, prioritize data
collection activity, and link human activities with their potential
effects.

Ecosystem. The interacting system of a biological community (plants and
animals) and its nonliving environment.

Expert Elicitation.  Expert elicitation (EE) is a systematic process of
formalizing and quantifying, typically in probabilistic terms, expert
judgments about uncertain quantities.  

Environment. The sum of all external conditions affecting the life,
development, and survival of an organism.

Frequentist (or frequency) probability. A view of probability that
concerns itself with the frequency of events in a long series of trials,
or is based upon a data set.

Inputs. Quantities that are input to a model.

Model. 1. A set of constraints restricting the possible joint values of
several quantities. 2. A hypothesis or system of belief regarding how a
system works or responds to changes in its inputs. 3. A mathematical
function with parameters that can be adjusted so the function closely
describes a set of empirical data. A mechanistic model usually reflects
observed or hypothesized biological or physical mechanisms and has model
parameters with real-world interpretation. In contrast, statistical or
empirical models selected for particular numerical properties are best
fits to data; model parameters may or may not have real-world
interpretation. When data quality is otherwise equivalent, extrapolation
from mechanistic models (e.g., biologically based dose-response models)
often carries higher confidence than extrapolation using empirical
models (e.g., logistic models).

Modeling. 1. Development of a mathematical or physical representation of
a system or theory that accounts for all or some of its known
properties. Models often are used to test the effect of changes of
components on the overall performance of the system. 2. Use of
mathematical equations to simulate and predict real events and
processes. 3. Development or application of conceptual or graphical
methods to depict the structure and organization among major elements of
the system to be modeled.

Model uncertainty (sources of).

Model structure. Reflects competing sets of conceptual, scientific, or
technical assumptions available to develop a model for a particular
phenomenon. An example of model structure uncertainty is the use of
epidemiologically derived concentration-response functions for modeling
cancer risk in humans versus the use of toxicological (animal)-based
functions employing human-to-animal extrapolation factors.

Model detail. Reflects simplifying assumptions used to make modeling
tractable. For example, complex nonlinear behavior of chemical uptake
from the gastrointestinal system into the blood stream may be replaced
by a rather simplified linear model, especially for specific exposure
(intake) ranges.

Extrapolation. Use of models outside of the parameter space used in
their derivation may result in erroneous predictions. For example, a
threshold for health effects may exist at exposure levels below those
covered by a particular epidemiological study. If that study is used in
modeling health effects at those lower levels (and it is assumed that
the level of response seen in the study holds for lower levels of
exposure), then disease incidence may be overestimated.

Resolution. Selection of spatial or temporal resolution (i.e., grid
size) typically reflects a balance between a desired level of precision
and resources required to model the system. If the grid size selected is
too small, then key patterns of behavior reflected in the smaller step
size may be missed altogether, or the behavior of the system may be
misrepresented. For example, efforts to capture realistic high-end (near
upper-bound) risks to farmers around an incinerator may necessitate a
geographical-information-system-based modeling framework precise enough
to model exposures for individual farms. If a less resolved exposure
model is used, then risk to the most exposed farm may be underpredicted.

Model boundaries. Decisions regarding the time, space, number of
chemicals, etc., used in guiding modeling of the system. Risks can be
understated or overstated if the model boundary is misspecified. For
example, if a study area is defined to be too large and includes a
significant number of low-exposure areas, then a population-level risk
distribution can be diluted by including less exposed individuals, which
can, in turn, result in a risk-based decision that does not protect
sufficiently the most exposed individuals in the study area.

Parameter. 1. A variable, measurable property whose value is a
determinant of the characteristics of a system (e.g., Temperature,
pressure, and density are parameters of the atmosphere.). 2. A constant
or variable term in a function that determines the specific form of the
function but not its general nature, as “a” in f(x) = ax, where
“a” determines only the slope of the line described by f(x). 3. A
variable entering into the mathematical form of any probability
distribution model such that the possible values of the variable
correspond to different distributions.

Probability. 1. Frequentist approach/ The frequency with which samples
are obtained within a specified range or for a specified category (e.g.,
the probability that an average individual with a particular mean dose
will develop an illness). 2. Bayesian approach. The degree of belief
regarding the different possible values of a quantity.

Probabilistic risk analysis. Application of a computational method,
often based on a randomized sampling of available data or information,
to produce a probability distribution to more fully describe the data
than selecting a single point in the distribution, e.g., the mean.

Risk. 1. Risk includes consideration of exposure to the possibility of
an adverse outcome, the frequency with which one or more types of
adverse outcomes may occur, and the severity or consequences of the
adverse outcomes if such occur. 2. The potential for realization of
unwanted, adverse consequences to human life, health, property, or the
environment. 3. The probability of adverse effects resulting from
exposure to an environmental agent or mixture of agents. 4. The combined
answers to (1) What can go wrong? (2) How likely is it? and (3) What are
the consequences?

Risk analysis. 1. A process for identifying, characterizing,
controlling, and communicating risks in situations where an organism,
system, subpopulation, or population could be exposed to a hazard. Risk
analysis is a process that includes risk assessment, risk management,
and risk communication (WHO). 2. A detailed examination, including risk
assessment, risk evaluation, and risk management alternatives, performed
to understand the nature of unwanted, negative consequences to human
life, health, property, or the environment; an analytical process to
provide information regarding undesirable events; the process of
quantification of the probabilities and expected consequences for
identified risks.

Risk assessment. 1. A process intended to calculate or estimate the risk
to a given target organism, system, subpopulation, or population,
including the identification of attendant uncertainties following
exposure to a particular agent, taking into account the inherent
characteristics of the agent of concern, as well as the characteristics
of the specific target system (WHO). 2. The evaluation of scientific
information on the hazardous properties of environmental agents (hazard
characterization), the dose-response relationship (dose-response
assessment), and the extent of human exposure to those agents (exposure
assessment) (NRC, 1983). The product of the risk assessment is a
statement regarding the probability that populations or individuals so
exposed will be harmed and to what degree (risk characterization)
(USEPA, 2000). 3. Qualitative and quantitative evaluation of the risk
posed to human health or the environment by the actual or potential
presence or use of specific pollutants.

Risk-informed decision making. An approach to decision making in which
insights from probabilistic risk analyses are considered with other
insights and factors.

Risk management. A decision making process that takes into account
environmental laws, regulations, political, social, economic,
engineering, and scientific information, including a risk assessment, to
weigh policy alternatives associated with a hazard.

Scenario. 1. An outline or model of an expected or supposed sequence of
events. 2. A set of facts, assumptions, and inferences about how
exposure takes place and regarding how exposures translate into adverse
effects that aides the analyst in evaluating, estimating, or quantifying
exposures and risks. Scenarios might include identification of
pollutants, pathways, exposure routes, and modes of action, among
others.

Sensitivity analysis. A study of how the variation in data inputs
(including inputs to models) affect the outputs of a model or choice
among potential decision options.

Levels. Refers to various hierarchical levels of complexity and
refinement for different types of modeling approaches that can be used
in risk assessment. A deterministic risk assessment with conservative
assumptions is an example of a lower level type of analysis that can be
used to determine whether exposures and risks are below levels of
concern. Examples of progressively higher levels include the use of
deterministic risk assessment coupled with sensitivity analysis, the use
of probabilistic techniques to characterize either variability or
uncertainty only, and the use of two-dimensional probabilistic
techniques to distinguish between but simultaneously characterize both
variability and uncertainty.

Two-dimensional probabilistic analysis. A modeling approach in which
inter-individual variability in exposure and risk is characterized using
frequency distributions, and in which uncertainty in the estimates of
statistics of the frequency distributions (e.g., the mean, median,
standard deviation, percentiles) are characterized using probability
distributions.

Uncertainty. Occurs because of a lack of knowledge. It is not the same
as variability. For example, a risk assessor may be very certain that
different people drink different amounts of water but may be uncertain
about how much variability there is in water intakes within the
population. Uncertainty often can be reduced by collecting more and
better data, whereas variability is an inherent property of the
population being evaluated. Variability can be better characterized with
more data but it cannot be reduced or eliminated. Efforts to clearly
distinguish between variability and uncertainty are important for both
risk assessment and risk characterization, although they both may be
incorporated into an assessment.[I agree with Harvey’s comment about
this sentence.]

Uncertainty analysis. A detailed examination of the systematic and
random errors of a measurement or estimate; an analytical process to
provide information regarding uncertainty.

Value of information. A quantitative measure of the value of knowing the
outcome of an uncertain variable prior to making a decision. Decision
theory provides a means for calculating the value of both perfect and
imperfect information. The former value, informally known as the value
of clairvoyance, is an upper bound for the latter. Obtaining meaningful
value-of-information measurements requires an awareness of important
restrictions (concerning the nature of free will) on the validity of
this kind of information.

Variability. Refers to true heterogeneity or diversity, as exemplified
in natural variation . For example, among a population that drinks water
from the same source and with the same contaminant concentration, the
risks from consuming the water may vary. This may result from
differences in exposure (i.e., different people drinking different
amounts of water and having different body weights, different exposure
frequencies, and different exposure durations), as well as differences
in response (e.g., genetic differences in resistance to a chemical
dose). Those inherent differences are referred to as variability.
Differences among individuals in a population are referred to as
inter-individual variability, differences for one individual over time
is referred to as intra-individual variability.Appendix C: References

Ang, A. H-S., and W.H. Tang. 1984. Probability Concepts in Engineering
Planning and Design, Volume 2: Decision, Risk, and Reliability. John
Wiley & Sons, New York.

Bloom, D.L., D.M. Byrne, and J.M. Andreson. 1993. Communicating Risk to
Senior EPA Policy-Makers: A Focus Group Study. Prepared by Bloom
Research and the Office of Air Quality Planning and Standards. U.S.
Environmental Protection Agency, Research Triangle Park, NC.

Burnham, K.P., and D.R. Anderson. 2002. Model Selection and Inference: A
Practical Information-Theoretic Approach (2nd ed.). Springer-Verlag, New
York.

Cullen, A.C., and H.C. Frey. 1999. Probabilistic Exposure Assessment: A
Handbook for Dealing with Variability and Uncertainty in Models and
Inputs. Plenum Press, New York.

Clyde, M. et al.  2000.  Effects of Ambient Fine and Coarse Particles on
Mortality in Phoenix, Arizona.  Duke University, Durham, N.C.

ON MORTALITY IN PHOENIX, ARIZONA1

EPA, 1995.  1995.  Policy for risk characterization.  Science Policy
Council, Washington, DC.  (http://www.epa.gov/osp/spc/rcpolicy.htm).

EPA. 1997a. Guiding Principles for Monte Carlo Analysis.
EPA/630/R-97/001, Risk Assessment Forum. U.S. Environmental Protection
Agency, Washington.

EPA. 1998. Guidance for Submission of Probabilistic Human Health
Exposure Assessments to the Office of Pesticide Programs. Office of
Pesticide Programs. U.S. Environmental Protection Agency, Draft,
11/4/98,   HYPERLINK
"http://www.epa.gov/fedrgstr/EPA-PEST/1998/November/Day-05/6021.pdf" 
http://www.epa.gov/fedrgstr/EPA-PEST/1998/November/Day-05/6021.pdf 
(accessed August 8, 2007).

EPA. 2000. EPA Science Policy Council Handbook, Risk Characterization.
EPA 100-B-00-002.

EPA. 2001. Risk Assessment Guidance for Superfund: Volume III - Part A,
Process for Conducting Probabilistic Risk Assessment. EPA 540-R-02-002,
Office of Emergency and Remedial Response, U.S. Environmental Protection
Agency, Washington,   HYPERLINK
"http://www.epa.gov/oswer/riskassessment/rags3adt/" 
http://www.epa.gov/oswer/riskassessment/rags3adt/  (accessed August 8,
2007).

EPA. 2007. Letter from M.G. Morgan and R.T. Parkin (SAB) to S. Johnson
(EPA), February 28, 2007. EPA-SAB-07-003. 

EPA. 2008. Case Study Examples of the Application of Probabilistic Risk
Analysis in U.S. Environmental Protection Agency Regulatory
Decision-Making (in review). Risk Assessment Forum, U.S. Environmental
Protection Agency, Washington.

Evans, J.S., J.D. Graham, G.M. Gray, and R.L. Sielken. 1994. A
Distributional Approach to Characterizing Low-Dose Cancer Risk. Risk
Analysis, 14(1):25-34.

Fisher, L., P. Pascual, and W. Wagner. 2006. Mapping the Role of Models
in US and EU Risk Regulation: A Legal Background Paper.

Frey, H.C. 2004. Presentation at the EPA Colloquium on Probabilistic
Risk Assessment, April 2004 (unpublished).

Frey, H.C., and S.R. Patil. 2002. Identification and Review of
Sensitivity Analysis Methods. Risk Analysis, 22(3):553-578.

Hattis, D.B., and D.E. Burmaster. 1994. “Assessment of Variability and
Uncertainty Distributions for Practical Risk Analyses,” Risk Analysis,
14(5):713-730.

Hoeting, J.A.; D. Madigan, A.E. Raftery, and C.T. Volinsky. 1999.
Bayesian Model Averaging: A Tutorial. Statistical Science,
14(4):382-417.

IEC. 2006. Expanded Expert Judgment Assessment of the
Concentration-Response Relationship Between PM2.5 Exposure and
Mortality, Final Report. Prepared by Industrial Economics, Incorporated
for Office of Air Quality Planning and Standards, U.S. Environmental
Protection Agency, Research Triangle Park, NC,   HYPERLINK
"http://www.epa.gov/ttn/ecas/regdata/Uncertainty/pm_ee_report.pdf" 
http://www.epa.gov/ttn/ecas/regdata/Uncertainty/pm_ee_report.pdf .

Krupnick, A., R. Morgenstern, M. Batz, P. Nelson, D. Burtraw, J. Shih,
and M. McWilliams. 2006. Not a Sure Thing: Making Regulatory Choices
Under Uncertainty. Resources for the Future, Washington.

Mokhtari, A., H.C. Frey, and J. Zheng. 2006. Evaluation and
Recommendation of Sensitivity Analysis Methods for Application to
Stochastic Human Exposure and Dose Simulation (SHEDS) Models. Journal of
Exposure Science and Environmental Epidemiology, 16(6):491-506.

Morgan, M.G., and M. Henrion. 1990. Uncertainty: A Guide to Dealing with
Uncertainty in Quantitative Risk and Policy Analysis. Cambridge
University Press, New York.

NRC. 1983. Risk Assessment in the Federal Government: Managing the
Process. Committee on the Institutional Means for Assessment of Risks to
Public Health, National Academy Press, Washington.

NRC. 1989. Risk Assessment in the Federal Government: Managing the
Process. National Research Council, National Academy Press, Washington.

NRC. 1991. Rethinking the Ozone Problem in Urban and Regional Air
Pollution. National Research Council, National Academy Press,
Washington.

NRC. 1993. Issues in Risk Assessment. National Research Council,
National Academy Press, Washington.

NRC. 1994. Science and Judgment in Risk Assessment. National Research
Council, National Academy Press, Washington.

NRC. 2002. Estimating the Public Health Benefits of Proposed Air
Pollution Regulations. National Research Council, National Academy
Press, Washington.

NRC. 2007a. Scientific Review of the Proposed Risk Assessment Bulletin
from the Office of Management and Budget. National Research Council,
National Academy Press, Washington.

NRC. 2007b. Models in Environmental Regulatory Decision Making. National
Research Council, National Academy Press, Washington.

OSTP/OMB. 2007. Memorandum (M-07-24) for the Heads of Executive
Departments and Agencies; from Susan E. Dudley, Administrator, Office of
Information and Regulatory Affairs, Office of Management and Budget, to
Sharon L. Hays, Associate Director and Deputy Director for Science,
Office of Science and Technology Policy: Updated Principles for Risk
Analysis, September 19, 2007.

Parkin, R.T., and M.G. Morgan. 2007. Consultation on Enhancing Risk
Assessment Practices and Updating EPA’s Exposure Guidelines.
EPA-SAB-07-003, Science Advisory Board, U.S. Environmental Protection
Agency, Washington.

PCRARM. 1997. Framework for Environmental Health Risk Management Final
Report, Volume 1. Presidential/Congressional Commission on Risk
Assessment and Risk Management,   HYPERLINK
"http://www.riskworld.com/nreports/1997/risk-rpt/pdf/EPAJAN.PDF" 
http://www.riskworld.com/nreports/1997/risk-rpt/pdf/EPAJAN.PDF .

Pearl, J. 2000. Causality: models, Reasoning, and Inference. Cambridge
University Press, New York.

Royall, R.M. 1997. Statistical Evidence: A Likelihood Paradigm. Chapman
& Hall, London.

Saltelli A, S. Tarantola, F. Campolongo, et al. 2004. Sensitivity
Analysis in Practice. John Wiley & Sons, New York.

Stahl, C.H., and A.J. Cimorelli. 2005. How Much Uncertainty Is Too Much
and How Do We Know? A Case Example of the Assessment of Ozone Monitor
Network Options.   HYPERLINK "javascript:void(0);"  Risk Analysis, Vol.
25, No. 5, pp. 1109-1120, October 2005 .

WHO. 2008. Guidance on Characterizing and Communicating Uncertainty in
Exposure Assessment (draft). The International Programme on Chemical
Safety, World Health Organisation, Geneva.

Bibliography

Probabilistic Risk Analysis Methodology(General

Cullen, A.C., and H.C. Frey. 1999. The Use of Probabilistic Techniques
in Exposure Assessment: A Handbook for Dealing with Variability and
Uncertainty in Models and Inputs. Plenum, New York.

EPA. 1997. Guiding Principles for Monte Carlo Analysis.
EPA/630/R-97/001, U.S. Environmental Protection Agency, Washington,
March 1997.

Frey, H.C., D. Crawford-Brown, J. Zheng, and D. Loughlin. 2003.
Hierarchy of Methods to Characterize Uncertainty: State of Science of
Methods for Describing and Quantifying Uncertainty, Draft. Prepared for
E.H. Pechan and Associates, Inc., Springfield, VA, for submittal to U.S.
Environmental Protection Agency, Research Triangle Park, NC, August 12,
2003.

Frey, C., J. Penman, L. Hanle, S. Monni, and S. Ogle. 2006.
Uncertainties, Chapter 3 in Volume 1, General Guidance and Reporting,
2006 IPCC Guidelines for National Greenhouse Gas Inventories. National
Greenhouse Gas Inventories Programme, Inter-Governmental Panel on
Climate Change, Technical Support Unit, Kanagawa, Japan.

Loughlin, D., H.C. Frey, K. Hanisak, and A. Eyth. 2003. Implementation
Requirements for the Development of a Sensitivity/Uncertainty Analysis
Tool for MIMS, Draft. Prepared by Carolina Environmental Program and
N.C. State University for U.S. Environmental Protection Agency, Research
Triangle Park, NC, May 6, 2003.

Morgan, M.G., and M. Henrion. 1990. Uncertainty: A Guide to Dealing with
Uncertainty in Quantitative Risk and Policy Analysis. Cambridge
University Press, New York.

NARSTO. 2005. Improving Emission Inventories for Effective Air Quality
Management Across North America, A NARSTO Assessment. NARSTO-05-001.

NRC. 1994. Science and Judgment in Risk Assessment. National Research
Council, National Academy Press, Washington.

Probabilistic Risk Analysis and Decision Making

Bloom, D.L., D.M. Byrne, and J.M. Andreson. 1993. Communicating Risk to
Senior EPA Policy-Makers: A Focus Group Study. Prepared by Bloom
Research and the Office of Air Quality Planning and Standards, U.S.
Environmental Protection Agency, Research Triangle Park, NC.

EPA. 1997.   HYPERLINK "http://www.epa.gov/OSA/spc/pdfs/probcovr.pdf_" 
Memo  from Fred Hansen entitled Use of Probabilistic Techniques
(including Monte Carlo Analysis) in Risk Assessment, dated May 15, 1997.
U.S. Environmental Protection Agency, Washington.

Krupnick et al. 2006. Not a Sure Thing: Making Regulatory Choices Under
Uncertainty. Resources for the Future, Washington.

McGarity, T.O., and W.E. Wagner. 2003. Legal Aspects of the Regulatory
Use of Environmental Modeling. Environmental Law Reporter, Volume 33,
10751-10774.

NRC. 1996. Understanding Risk: Informing Decisions in a Democratic
Society. National Research Council, National Academy Press, Washington.

Pascual, P., N. Steiber, and E. Sunderland. 2003. Draft Guidance on the
Development, Evaluation, and Application of Regulatory Environmental
Models. Prepared by the Council for Regulatory Environmental Modeling,
Office of Science Policy, U.S. Environmental Protection Agency,
Washington, November.

Thompson, K.M., and J.D. Graham. 1996. Going Beyond the Single Number:
Using Probabilistic Risk Assessment to Improve Risk Management. Human
and Ecological Risk Assessment, 2(4):1008-1034.

Probabilistic Risk Analysis Methodology(Specific Aspects

Abdel-Aziz, A., and H.C. Frey. 2003b. Development of Hourly
Probabilistic Utility NOx Emission Inventories Using Time Series
Techniques: Part I-Univariate Approach. Atmospheric Environment,
37:5379-5389.

Abdel-Aziz, A., and H.C. Frey. 2003c. Development of Hourly
Probabilistic Utility NOx Emission Inventories Using Time Series
Techniques: Part II-Multivariate Approach. Atmospheric Environment,
37:5391-5401.

Abdel-Aziz, A., and H.C. Frey. 2004. Propagation of Uncertainty in
Hourly Utility NOx Emissions Through a Photochemical Grid Air Quality
Model: A Case Study for the Charlotte, NC Modeling Domain. Environmental
Science and Technology, 38(7):2153-2160.

Casman, E.A., M.G. Morgan, and H. Dowlatabadi. 1999. Mixed Levels of
Uncertainty in Complex Policy Models. Risk Analysis, 19(1):33-42.

Crawford-Brown and Brown. 1987. A Framework for Assessing the
Rationality of Judgments in Carcinogenicity Hazard Identification. Risk:
Health, Safety and Environment, Fall, 307-337.

Draper, D. 1995. Assessment and Propagation of Model Uncertainty.
Journal of Royal Statistical Society 57:45-97.

Evans, J.S., J.D. Graham, G.M. Gray, and R.L. Sielken. 1994. A
Distributional Approach to Characterizing Low-Dose Cancer Risk. Risk
Analysis, 14(1):25-34.

Frey, H.C., and D.E. Burmaster. 1999. Methods for Characterizing
Variability and Uncertainty: Comparison of Bootstrap Simulation and
Likelihood-Based Approaches. Risk Analysis, 19(1):109-130 (February).

Frey, H.C., and D.S. Rhodes. 1996. Characterizing, Simulating, and
Analyzing Variability and Uncertainty: An Illustration of Methods Using
an Air Toxics Emissions Example. Human and Ecological Risk Assessment:
an International Journal, 2(4):762-797 (December).

Frey, H.C., and Y. Zhao. 2004. Quantification of Variability and
Uncertainty for Air Toxic Emission Inventories With Censored Emission
Factor Data. Environmental Science and Technology, 38(22):6094-6100.

Zheng, J., and H.C. Frey. 2004. Quantification of Variability and
Uncertainty Using Mixture Distributions: Evaluation of Sample Size,
Mixing Weights and Separation between Components. Risk Analysis,
24(3):553-571 (June).

Zheng, J., and H.C. Frey. 2005. Quantitative Analysis of Variability and
Uncertainty with Known Measurement Error: Methodology and Case Study.
Risk Analysis, 25(3):663-676.

Sensitivity Analysis

Agro, K.E., C.A. Bradley, and N. Mittmann. 1997. Sensitivity Analysis in
Health Economic and Pharmacoeconomic Studies - an Appraisal of the
Literature. Pharmacoeconomics, 11(1):75-88.

Cukier, R.I., H.B. Levine, and K.E. Shuler. 1978. Nonlinear Sensitivity
Analysis of Multi-parameter Model Systems. Journal of Chemical Physics,
26(1):1-42.

Frey, H.C., and R. Patil. 2002. Identification and Review of Sensitivity
Analysis Methods. Risk Analysis, 22(3):553-577.

Frey, H.C., A. Mokhtari, and J. Zheng. 2004. Recommended Practice
Regarding Selection, Application, and Interpretation of Sensitivity
Analysis Methods Applied to Food Safety Process Risk Models. Prepared by
N.C. State University for U.S. Department of Agriculture, Washington,
January 30, 2004.  HYPERLINK "http://www.ce.ncsu.edu/risk"   Available
at   HYPERLINK "http://www.ncsu.edu/risk"  http://www.ncsu.edu/risk  .

Helton, J.C., and F.J. Davis. 2002. Illustration of Sampling-Based
Methods for Sensitivity Analysis. Reliability Engineering and System
Safety, 81(1):23-69.

Mokhtari, A., and H.C. Frey. 2005a. Recommended Practice Regarding
Selection of Sensitivity Analysis Methods Applied to Microbial Food
Safety Process Risk Models. Human and Ecological Risk Assessment,
11(3):591-605.

Mokhtari, A., H.C. Frey, J. Zheng. 2006. Evaluation and Recommendation
of Sensitivity Analysis Methods for Application to EPA-Stochastic Human
Exposure and Dose Simulation (SHEDS) Models. Journal of Exposure Science
and Environmental Epidemiology, 16(6):491-506.

Saltelli, A., K. Chan, and M. Scott. 2000. Sensitivity Analysis,
Probability and Statistics Series. John Wiley & Sons, New York.

Sobol, I.M. 1993. Sensitivity Estimates for Nonlinear Mathematical
Models. Mathematical Modeling and Computation, 1(4):407-414.

Xue, J., V.G. Zartarian, H. Özkaynak, W. Dang, G. Glen, L. Smith, and
C. Stallings. 2006. A Probabilistic Arsenic Exposure Assessment for
Children Who Contact Chromated Copper Arsenate (CCA)-Treated Playsets
and Decks, Part 2: Sensitivity and Uncertainty Analyses. Risk Analysis,
26(2):533-541.

Case Study Examples of Probabilistic Risk Analysis(EPA (see also the PRA
Case Studies White Paper)

Babendreier, J.E., and K.L. Castleton. 2005. Investigating Uncertainty
and Sensitivity in Integrated Multimedia Environmental Models: Tools for
FRAMES-3MRA. Environmental Modeling and Software, 20:1043-1055.

Blancato, J.N., F.W. Power, R.N. Brown, and C.C. Dary. 2004. Exposure
Related Dose Estimating Model (ERDEM) for Assessing Human Exposure and
Dose, EPA/600/R-04/060, U.S. Environmental Protection Agency, Research
Triangle Park, NC, December 2004.

Burke, J. 2005. EPA Stochastic Human Exposure and Dose Simulation for
Particulate Matter (SHEDS-PM), User’s Guide. U.S. Environmental
Protection Agency, Research Triangle Park, NC.

Frey, H.C. 2003. Evaluation of an Approximate Analytical Procedure for
Calculating Uncertainty in the Greenhouse Gas Version of the Multi-Scale
Motor Vehicle and Equipment Emissions System. Prepared for Office of
Transportation and Air Quality, U.S. Environmental Protection Agency,
Ann Arbor, MI, May 30, 2003.

Frey, H.C., and Y. Zhao. 2003. Development of Probabilistic Emission
Inventories of Benzene, Formaldehyde and Chromium for the Houston
Domain. Prepared by N.C. State University for U.S. Environmental
Protection Agency, Research Triangle Park, NC, September 2003.

Frey, H.C., R. Bharvirkar, and J. Zheng. 1999. Quantitative Analysis of
Variability and Uncertainty in Emissions Estimation. Prepared by N.C.
State University for U.S. Environmental Protection Agency, Research
Triangle Park, NC, July 1999.

Frey, H.C., J. Zheng, Y. Zhao, S. Li, and Y. Zhu. 2002. Technical
Documentation of the AuvTool Software for Analysis of Variability and
Uncertainty. Prepared by N.C. State University for Office of Research
and Development, U.S. Environmental Protection Agency, Research Triangle
Park, NC, February 2002.

Zartarian, V.G., J. Xue, H.A. Ozkaynak, W. Dang, G. Glen, L. Smith, and
C. Stallings. 2005. A Probabilistic Exposure Assessment for Children Who
Contact CCA-Treated Playsets and Decks Using the Stochastic Human
Exposure and Dose Simulation Model for the Wood Preservative Scenario
(SHEDS-WOOD), Final Report. U.S. Environmental Protection Agency,
Washington, EPA/600/X-05/009. Available at   HYPERLINK
"http://www.epa.gov/heasd/sheds/cca_treated.htm" 
http://www.epa.gov/heasd/sheds/cca_treated.htm .

Zartarian, V.G., J. Xue, H. Özkaynak, W. Dang, G. Glen, L. Smith, and
C. Stallings. 2006. A Probabilistic Arsenic Exposure Assessment for
Children Who Contact Chromated Copper Arsenate (CCA)-Treated Playsets
and Decks, Part 1: Model Methodology, Variability Results, and Model
Evaluation. Risk Analysis, 26(2):515-532.

Zheng, J., and H.C. Frey. 2002. AuvTool User’s Guide. Prepared by N.C.
State University for Office of Research and Development, U.S.
Environmental Protection Agency, Research Triangle Park, NC, February
2002.

Case Study Examples of Probabilistic Risk Analysis(Other

Bloyd, Cary, et al. 1996. Tracking and Analysis Framework (TAF) Model
Documentation and User’s Guide, 1996. Argonne National Laboratory,
ANL/DIS/TM-36 (December).

Frey, H.C., and S. Li. 2003. Quantification of Variability and
Uncertainty in AP-42 Emission Factors: Case Studies for Natural
Gas-Fueled Engines. Journal of the Air & Waste Management Association,
53(12):1436-1447 (December).

Hanna, S.R., Z. Lu, H.C. Frey, N. Wheeler, J. Vukovich, S. Arunachalam,
M. Fernau, and D.A. Hansen. 2001. Uncertainties in Predicted Ozone
Concentrations due to Input Uncertainties for the UAM-V Photochemical
Grid Model Applied to the July 1995 OTAG Domain. Atmospheric
Environment, 35(5):891-903.

Munns, W.R., Jr., H.A. Walker, and J.F. Paul. 1989. An Ecological Risk
Assessment Framework for Examining the Impacts of Oceanic Disposal
Status and Trends Program. In: Oceans ‘89, Volume 2: Ocean Pollution,
Marine Technology Society and Oceanic Society of the Institute of
Electrical and Electronics Engineers, IEEE Publication No. 89CH2780-5,
pp. 664-669.

Munns, W.R., Jr., H.A. Walker, J.F. Paul, and J.H. Gentile. 1996. A
Prospective Assessment of Ecological Risks to Upper Water Column
Populations from Ocean Disposal at the 106-Mile Dumpsite. Journal of
Marine Environmental Engineering, 3:279-297.

Nocito, J.A., H.A. Walker, J.F. Paul, and C. Menzie. 1988. Application
of a Risk Assessment for Marine Disposal of Sewage Sludge at Midshelf
and Offshelf Sites. In: Proceedings of the Seventh International Ocean
Disposal Symposium, Environment Canada, pp. 644-663.

Nocito, J.A., H.A. Walker, J.F. Paul, and C.A. Menzie. 1989. Application
of a Risk Assessment Framework for Marine Disposal of Sewage Sludge at
Midshelf and Offshelf Sites. In: Aquatic Toxicology and Environmental
Fate: Eleventh Volume, ASTM STP 1007, G.W. Suter II and M.A. Lewis
(eds.), American Society for Testing and Materials, Philadelphia, pp.
101-120.

Paul, J.F., H.A. Walker, and V.J. Bierman, Jr. 1983. Probabilistic
Approach for the Determination of the Potential Area of Influence for
Waste Disposed at the 106-Mile Ocean Disposal Site. In: 106-Mile Waste
Disposal Site Characterization Update Report, J.B. Pearce, D.C. Miller,
and C. Berman (eds.), NOAA Technical Memorandum NMFS-F/NEC-26, National
Marine Fisheries Service, Woods Hole, MA, pp. A-1 to A-8.

Paul, J.F., V.J. Bierman, Jr., W.R. Davis, G.L. Hoffman, W.R. Munns,
C.E. Pesch, P.F. Rogerson, and S.C. Schimmel. 1988. The Application of a
Hazard Assessment Research Strategy to the Ocean Disposal of a Dredged
Material: Exposure Assessment Component. In: Oceanic Processes in Marine
Pollution, Vol. 5, D.A. Wolfe and T.P. O’Connor (eds.), Kreiger
Publishing Co., Melbourne, FL, pp. 123-135.

Prager, J.C., V.J. Bierman, Jr., J.F. Paul, and J.S. Bonner. 1986.
Sampling the Oceans for Pollution: A Risk Assessment Approach to
Evaluating Low-Level Radioactive Waste Disposal at Sea. Dangerous
Properties of Industrial Materials Report, Vol. 6, No. 3, pp. 2-26.

Walker, H.A., J.F. Paul, J.A. Nocito, and J.H. Gentile. 1988. Ecological
and Human Health Risks for Sewage Sludge Disposal at the 106-Mile Site.
In: Proceedings of the Seventh International Ocean Disposal Symposium,
Environment Canada, pp. 625-643.



Appendix D: Case Study Examples of the Application of Probabilistic Risk
Analysis in U.S. Environmental Protection Agency Regulatory
Decision-Making

Prepared by Risk Assessment Forum

PRA Working Group 2

Allison Hess, David Hrdy, John Langstaff, Elizabeth Margosches, Michael
Messner, and Marian Olsen

Disclaimer

This document is a preliminary draft. It has not been formally released
by the U.S. Environmental Protection Agency and should not at this stage
be construed to represent Agency policy. It is being circulated for
comments on its technical merit and policy implications. Mention of
trade names or commercial products does not constitute endorsement or
recommendation for use.



Foreword

The U.S. Environmental Protection Agency’s (EPA’s) Risk Assessment
Forum was directed by the Science Policy Council in the Office of the
Science Advisor to consider how to better and more fully implement
probabilistic risk analysis (PRA) and related tools in the EPA
decision-making process. A technical panel of senior scientists gathered
and identified several ways of moving the agenda to fully implement PRA.
This Appendix focuses on examples of how PRA approaches have been used
at EPA to inform regulatory decisions.

This White Paper was prepared by representatives from various EPA
program offices and regions currently involved in the development and
application of PRA techniques. The workgroup selected the case study
examples based on the workgroup’s knowledge of the specific PRA
procedures, the types of techniques demonstrated, availability to the
reader through the internet, peer-reviewed, and illustrative of a
spectrum of PRA used in EPA. The case studies are not designed to
provide an exhaustive discussion of the wide variety of applications of
PRA used within the Agency but to highlight specific examples reflecting
the range of approaches currently applied within EPA.	



Acknowledgments

	We would like to acknowledge the scientists and risk assessors who
performed the original analyses on which the summaries of these case
studies are based.  The names of the points of contact are included for
each study, and many more contributors were involved and acknowledged in
the original work.

	Contributors and Reviewers to this Document

Gary Bangs, Jonathan Chen, Lisa Conner, Kathryn Gallagher, and Valerie
Zartarian

The PRA Technical Panel

The EPA Risk Assessment Forum

Executive Summary

This Appendix is designed to serve as a resource for risk managers faced
with decisions regarding when to apply PRA techniques to inform
environmental decisions and for exposure and risk assessors who may not
be familiar with the wide range of available PRA approaches. The
document outlines categories of PRAs classified by the complexity of
analysis to aid the decision-making process. This approach identifies
various PRA tools that include techniques ranging from a simple
sensitivity analysis (e.g., identification of key exposure parameters or
data visualization) requiring limited time, resources, and expertise to
develop (Group 1); to probabilistic approaches, including Monte Carlo
analysis, that provide tools for evaluating variability and uncertainty
separately and require more resources and specialized expertise (Group
2); and to sophisticated techniques of expert elicitation that generally
require significant investment of employee time, additional expertise,
and external peer-review (Group 3).

This document describes case studies wherein PRA techniques have been
used within this ranked framework to provide additional information for
risk managers. PRA is a scientific tool to help describe the data or the
risk and is one of many inputs considered by the risk manager in the
decision-making process. The case study summaries are provided in a
format designed to highlight how the results of the PRAs were considered
in decision-making. These summaries include specific information on the
conduct of the analyses as an aid to determining what tools might be
appropriate for developing specific exposure or risk assessments for
other assessments.

The case studies range from examples of less resource-intensive analyses
that might assist in identifying key exposure parameters or the need for
more data to more detailed and resource-intensive approaches. Tools
include Monte Carlo modeling, sensitivity analyses, and application of
expert elicitation. Examples of applications in human health and
ecological risk assessment include the exposure of children to chromated
copper arsenate treated wood, the relation between particulates in air
and health, dietary exposures to pesticides, modeling sea level change,
sampling watersheds, and modeling bird and animal exposures.



1. Introduction

 methods typically used in EPA deterministic risk assessments rely on a
combination of point values―some conservative and some
typical―yielding a point estimate of exposure that is at some unknown
point in the range of possible risks (EPA, 2004).

The development of sophisticated computational tools over the past 10
years has prompted an increased interest in analyses that evaluate the
variability and uncertainty in the risk assessments; these include the
use of tools such as probabilistic risk analysis, or PRA (EPA, 2001,
2004). These analyses provide the results of the risk assessment as a
probability or likelihood of different risk levels in a population
(describing variability) or to characterize uncertainty in risk
estimates.

This Appendix presents case studies of PRA conducted by EPA over the
past 10 to 15 years. Table 1 summarizes the case studies by title,
technique demonstrated, classification based on Human Health and
Ecological Risk Assessment, and the program office responsible for
developing the case studies. This document, by illustration, provides a
“snapshot” of utilization of PRA across various programs in EPA.



2. Overall Approach to Probabilistic Risk Analysis at the U.S.
Environmental Protection Agency

2.1. U.S. Environmental Protection Agency Guidance and Policies on
Probabilistic Risk Analysis

The case studies presented here build on the principles of PRA outlined
in EPA’s 1996 Policy (EPA, 1996) and Guiding Principles for Monte
Carlo Analysis (EPA, 1997b) and subsequent guidance documents on
developing and using PRA.  Guidance has been developed for the Agency as
well as for individual programs that refers to the use of PRA, including
the Risk Assessment Guidance for Superfund Part III (EPA, 2001); Risk
Assessment Forum Framework for Ecological Risk Assessment (EPA, 1992b);
Guidelines for Ecological Risk Assessment (EPA, 1998); Guidance for Risk
Characterization (EPA, 1995a); Policy for Risk Characterization (EPA,
1995a); Policy on Evaluating Health Risks to Children (EPA, 1995b);
Policy for Use of Probabilistic Analysis in Risk Assessment (EPA,
1997a); Guidance on Cumulative Risk Assessment. Part 1. Planning and
Scoping (EPA, 1997c ; and Risk Characterization Handbook (EPA, 2000).

As shown in the individual case studies, the range and scope of the PRA
will depend on the overall objectives of the decision that the analysis
will inform. The Guiding Principles for Monte Carlo Analysis lay out the
general approach that should be taken in all cases, beginning with 
defining the problem and scope of the assessment, so that the best tools
and approach may be selected.  The Guiding Principles also describe the
process of estimating and characterizing variability and uncertainty
around the risk estimates. Stahl and Cimorelli (2005) and the Risk
Assessment Guidance for Superfund Volume III (EPA, 2001) highlight the
importance of communication between risk assessor and manager. Stahl and
Cimorelli (2005) and Jamieson (1996) indicate it is important to
determine whether a particular level of uncertainty is acceptable or
not. The authors also suggest this decision is a matter of context,
values, and regulatory policy. The Risk Assessment Guidance for
Superfund Part III (Chapter 2 and Appendix F in EPA, 2001) describes a
process for determining the appropriate level of PRA using a ranked
approach from the less resource- and time-intensive approaches to more
sophisticated analyses (Chapter 2 in EPA, 2001). Further, the Risk
Assessment Guidance for Superfund Part III  outlines a process for
developing a PRA work plan and a checklist for PRA reviewers (Chapter 2
and Appendix F in EPA, 2001). This guidance also provides information
regarding how to communicate PRA results to risk managers and
stakeholders (Chapter 6 in EPA, 2001).

The guidance and policies on uncertainty and variability, and
application of the principles of PRA, all highlight the ongoing need for
communication between the risk assessor and risk manager. The ongoing
communication is important in determining the appropriate levels of
analysis for the specific decision.

2.2. Categorizing Case Studies

The ranked approach used for categorization is a process for a
systematic, informed progression to increasingly more complex risk
assessment methods of PRA that is outlined in the Risk Assessment
Guidance for Superfund (EPA, 2001). The use of categories provides a
framework for evaluating the various techniques of PRA.  Higher
categories reflect increasing complexity and, in many cases, will
require more time and resources. Higher categories also reflect
increasing characterization of variability and uncertainty in the risk
estimate, which may be important for making specific risk management
decisions. Central to the approach is a systematic, informed progression
using an iterative process of evaluation, deliberation, data collection,
planning and scoping, development and updates to the work plan, and
communication. All of these steps focus on deciding

whether or not the risk assessment, in its current state (i.e.,
deterministic risk analysis), is sufficient to support risk management
decisions (a clear path to exiting the process is available at each
step); and

if the assessment is determined to be insufficient, whether or not
progression to a higher group of complexity (or refinement of the
current analyses) would provide a sufficient benefit to warrant the
additional effort of performing a PRA.

This paper groups case studies according to level of effort and
complexity of the analysis, and the increasing sophistication of the
methods used (Table 1).  Although each group generally represents
increasing effort and cost, this may not always be true.  The groups are
intended to also reflect the progression from simple to complex analysis
that is determined by the interactive planning and scoping efforts of
the risk assessors and managers.  The use of particular terms to
describe the groups, including tiers, was avoided due to specific
programmatic and regulatory connotations.

2.2.1. Group 1 Case Studies

Assessments within this group typically involve a sensitivity analysis
and serve as an initial screening step in the risk assessment.
Sensitivity analyses identify important parameters in the assessment
where additional investigation may be helpful (Kurowicka and Cooke,
2006). Sensitivity analysis can be simple or involve more complex
mathematical and statistical techniques such as correlation and
regression analysis to determine which factors in a risk model
contribute most to the variance in the risk estimate.

Within the sensitivity analyses, a range of techniques is available:
simple, “back of the envelope” calculation, where the risk
parameters are evaluated using a range of exposure parameters to
determine the parameter that contributes most significantly to the risk
(Case Study 1); analyses to rank relative contributions of variables to
the overall risk (Case Study 2); and data visualization using graphical
techniques to array the data or Monte Carlo simulations (e.g., scatter
plots).

More sophisticated analyses may include sensitivity ratios (i.e.,
elasticity); sensitivity scores (i.e., weighted sensitivity ratios);
correlation coefficient or coefficient of determination, r2 (e.g.,
Pearson product moment, Spearman rank); normalized multiple regression
coefficient; and goodness-of-fit tests for subsets of the risk
distribution (EPA, 2001).

The sensitivity analyses typically require limited resources and time to
conduct. Results of the sensitivity analyses are useful in identifying
key parameters where additional Group 2 or 3 analyses may be
appropriate. Sensitivity analyses are also helpful in identifying key
parameters where additional research will have the most impact on the
risk assessment.

2.2.2. Group 2 Case Studies

Case studies within this group include more sophisticated application of
probabilistic tools, including PRA of specific exposure parameters (Case
Studies 3 and 4), one-dimensional analyses (Case Study 5), and
probabilistic sensitivity analysis (Case Studies 6 and 7).

The Group 2 case studies require larger time commitments for
development, specialized expertise, and additional analysis of exposure
parameter data sources. Depending on the nature of the analysis, peer
involvement or peer review may be appropriate to the evaluation of the
products of the analysis.

2.2.3. Group 3 Case Studies

Assessments within this group are the most resource- and time-intensive
analyses of the three categories. Risk analyses include two-dimensional
Monte Carlo analysis that evaluate model variability and uncertainty
(Case Studies 8 through 10); Microexposure Event Analysis, in which
long-term exposure of an individual is simulated as the sum of separate
short-term exposure events (Case Study 11); and Probabilistic Analysis
(Case Studies 12 and 13).

Other types of analyses within this group include the expert elicitation
method that is a systematic process of formalizing and quantifying, in
terms of probabilities, experts’ judgments about uncertain quantities
(Case Studies 14 and 15); Bayesian statistics that is a specialized
branch of statistics that views the probability of an event occurring as
the degree of belief or confidence in that occurrence (Case Study 16);
and geostatistical analysis, which is another specialized branch of
statistics that explicitly takes into account the geo-referenced context
of data and the information (i.e., attributes) attached to the data.

The Group 3 analyses require additional time and expertise in the
planning and  analysis of the assessment. Within this group, the level
of expertise and resource commitments may vary with techniques such as
expert elicitation requiring significantly longer time for planning,
identification of experts, and meetings, when compared with the other
techniques.

.

Table 1.  Case Study Titles, Description, Type of Assessment (Human
Health or Ecological Risk Assessment) and Program Office that Developed
Assessment.

Study

No.	Title and Case Study Description	HH/

Eco1	Program

oup 2―Probabilistic Risk Analysis, One-Dimensional Monte Carlo
Analysis, and

Probabilistic Sensitivity Analysis

Probabilistic Risk Analysis

3	Probabilistic Assessment of Angling Duration Used in Assessment of
Exposure to Hudson River Sediments via Consumption of Contaminated Fish.
A probabilistic analysis of one parameter in an exposure
assessment―the time an individual fishes in a large river system.
Development of site-specific information regarding exposure, with an
existing data set for this geographic area, was needed to represent this
exposed population. This information was used in the one-dimensional PRA
described in Case Study 5.	HH	Region 2/

Superfund

4	Probabilistic Analysis of Dietary Exposure to Pesticides for Use in
Setting Tolerance Levels. The probabilistic Dietary Exposure Evaluation
Model (DEEM) provides more accurate information on the range and
probability of possible exposures.	HH	OPP

1 HH = Human Health Risk Assessment; Eco = Ecological Risk Assessment.

2 OPP, Office of Pesticides Programs; ORD, Office of Research and
Development; OAR, Office of Air and Radiation; OW, Office of Water.



Study

No.	

Title and Case Study Description	HH/

Eco	Program

Office

Group 2 (cont’d.)

One-Dimensional Monte Carlo Analysis

5	One-Dimensional Probabilistic Risk Analysis of Exposures to
Polychlorinated Biphenyls (PCBs) via Consumption of Fish from a
Contaminated Sediment Site. An example of a one-dimensional PRA (1-D
Monte Carlo analysis of the variability of exposure as a function of the
variability of individual exposure factors.) to evaluate the risks to
anglers who consume recreationally caught fish from a PCB-contaminated
river.	HH	Region 2/

Group 3―Advanced Probabilistic Risk Analysis―Two-Dimensional Monte
Carlo Analysis Including Microexposure Modeling, Bayesian Statistics,
Geostatistics, and Expert Elicitation

Two- Dimensional Probabilistic Risk Analysis

8	Two-Dimensional Probabilistic Risk Analysis of Cryptosporidium in
Public Water Supplies, with Bayesian Approaches to Uncertainty Analysis.
An analysis of variability in the occurrence of Cryptosporidium in raw
water supplies and in the treatment efficiency, as well as the
uncertainty in these inputs. This case study includes an analysis of the
dose-response relationship for Cryptosporidium infection.	HH	OW

9	Two-Dimensional Probabilistic Model of Children’s Exposure to
Arsenic in Chromated Copper Arsenate (CCA) Pressure-Treated Wood. A
two-dimensional model that addresses both variability and uncertainty in
the exposures of children to CCA pressure-treated wood. The analysis was
built on the sensitivity analysis described in Case Study 2.	HH	OPP/ORD



Study

No.	

Title and Case Study Description	HH/

Eco	Program

Office

Group 3 (cont’d.)

10	Two-Dimensional Probabilistic Exposure Assessment of Ozone

A probabilistic exposure assessment that addresses short-term exposures
to ozone.  Population exposure to ambient ozone levels was evaluated
using EPA’s Air Pollutants Exposure (APEX) model, also referred to as
the Total Risk Integrated Methodology/Exposure (TRIM.Expo) model.	HH	OAR

Group 3―Microexposure Event Modeling

Microexposure Event Analysis

11	Analysis of Microenivironmental Exposures to Particulate Matter
(PM2.5) for a Population Living in Philadelphia, PA. A microexposure
event analysis to simulate individual exposures to PM2.5 in specific
microenvironments including the outdoors, indoor residences, offices,
schools, stores, and a vehicle.	HH	Region 3

Group 3―Expert Elicitation and Bayesian Belief Network

Expert Elicitation

14	Expert Elicitation of Concentration-Response Relationship Between
Particulate Matter (PM2.5) Exposure and Mortality. An expert elicitation
used to derive probabilistic estimates of the uncertainty in one element
of a cost-benefit analysis used to support the PM2.5 regulations.	HH
ORD/

OAR

15	Expert Elicitation of Sea-Level Change Resulting from Global Climate
Change. An example of a PRA that describes the probability of sea level
rise and parameters that predict sea level change.	Eco

	16	Expert Elicitation for Bayesian Belief Network Model of Stream
Ecology. An example of a Bayesian belief network model of the effect of
increased fine-sediment load in a stream on macroinvertebrate
populations.	Eco	ORD



3. Case Study Summaries

Group 1 Case Studies

Case Study 1: Sensitivity Analysis of Key Variables in Probabilistic
Assessment of Children’s Exposure to Arsenic in Chromated Copper
Arsenate (CCA) Pressure-Treated Wood

This case study provides an example of the application of sensitivity
analysis to identify important variables for population exposure
variability for a Group 2 assessment (Case Study 9) and to indicate
areas for further research.  Specifically, EPA’s Office of Research
and Development (ORD), in collaboration with the Office of Pesticide
Programs (OPP) used sensitivity analyses to identify the key variables
in children’s exposure to CCA treated wood.   

Approach.  The sensitivity analyses used two approaches.  The first
approach estimated baseline exposure by running the exposure model with
each input variable set to its median (50th percentile) value.  Next,
alternative exposure estimates were made by setting each input to its
25th or 75th percentile value while holding all other inputs at their
median values.  The ratio of the exposure estimate calculated when an
input was estimated at its 25th or 75th percentile to the exposure
estimate calculated when the input was at its median value provided a
measure of that input’s importance to the overall exposure assessment.
 The second approach applied multiple stepwise regression analysis to
the data points generated from the first approach.  The correlation
between the input variables and the exposure estimates provided an
alternative measure of the input variable’s relative importance in the
exposure assessment.  These two approaches were used in tandem to
identify the critical inputs to the exposure assessment model.

Results of Analysis.  The two sensitivity analyses together identified
six critical input variables that most influenced the exposure
assessment.  The critical input variables were: wood surface
residue-to-skin transfer efficiency, wood surface residue levels,
fraction of hand surface area mouthed per mouthing event, average
fraction of nonresidential outdoor time spent playing on a CCA-treated
playset, frequency of hand-washing, and frequency of hand-to-mouth
activity.

Management Considerations:  The results of the sensitivity analyses were
used to identify the most important input parameters in the treated wood
risk assessments.  The process also identified critical areas for future
research.  In particular, the assessment pointed to a need to collect
data on the amount of dislodgeable residue that is transferred from the
wood surface to a child’s hand upon contact, and to better
characterize the amount of dislodgeable residue that exists on the wood
surface.  



Document Availability.  

Title:  The final report on the probabilistic exposure assessment of
CCA-treated wood.

Zartarian, V.G., J. Xue, H. A. Ozkaynak, W. Dang, G. Glen, L. Smith, and
C. Stallings. A Probabilistic Exposure Assessment for Children Who
Contact CCA-treated Playsets and Decks Using the Stochastic Human
Exposure and Dose Simulation Model for the Wood Preservative Scenario
(SHEDS-WOOD), Final Report. U.S. Environmental Protection Agency,
Washington, DC, EPA/600/X-05/009.   HYPERLINK
"http://www.epa.gov/heasd/sheds/cca_treated.htm" 
http://www.epa.gov/heasd/sheds/cca_treated.htm 

See also: Xue, J., Zartarian, V.G., Özkaynak, H., Dang, W., Glen, G.,
Smith, L., and Stallings, C.   A probabilistic arsenic exposure
assessment for children who contact chromated copper arsenate
(CAA)-treated playsets and decks, Part 2:  Sensitivity and uncertainty
analyses.  Risk Analysis 26:533, 2006.

Contact Person: Dr. Jianping Xue, EPA's Office of Research and
Development, 919-541-7962, xue.jianping@epa.gov

Case Study 2: Assessment of Relative Contribution of Atmospheric
Deposition to Watershed Contamination

Watershed contamination can result from several different sources,
including direct release into a water body, input from upstream water
bodies, and deposition from airborne sources.  Efforts to control water
body contamination begin with an analysis of the environmental sources
in order to identify those parameters providing the greatest
contribution and to determine where mitigation and/or analysis resources
should be directed.  

Approach. This case study provides an example of a back-of-the envelope
analysis of the contribution of air deposition to overall watershed
contamination to identify uncertainties and/or data gaps as well as to
target resource expenditures. (Group 1: Deterministic Analysis).
Nitrogen inputs have been studied in several east and Gulf Coast
estuaries due to concerns about eutrophication.  Nitrogen from
atmospheric deposition is estimated to be as high as 10 to 40% of the
total input of nitrogen to many of these estuaries and perhaps higher in
a few cases. For a watershed that has not already been studied, a
back-of-the envelope calculation could be prepared based on information
based on nitrogen deposition rates measured in a similar area.  To
estimate the deposition load directly to the waterbody, one would
multiply the nitrogen deposition rate by the area of the waterbody.  
The analyst could then estimate the nitrogen load from other sources,
(e.g., point source discharges and runoff) to estimate a total nitrogen
load for the waterbody.  The estimate of loading due to atmospheric
deposition could then be divided by the total nitrogen load for the
waterbody to estimate the percent contribution directly to the waterbody
from atmospheric deposition.

The May 2003 report by the Casco Bay Air Deposition Study Team titled
“Estimating Pollutant Loading from Atmospheric Deposition Using Casco
Bay, Maine” is an analysis using the methodology described above.  The
Casco Bay Estuary, located in the southwestern Maine, is used as a case
study. The paper also includes the results of a field air deposition
monitoring program conducted in Casco Bay (1998 - 2000) and favorably
compares the estimates developed for rate of deposition of nitrogen,
mercury and PAHs to the field monitoring results. The estimation
approach is a useful starting point for understanding the sources of
pollutants entering water bodies that cannot be accounted for through
run-off or point source discharges.

Results of Analysis. The approach outlined above was applied to the
Casco Bay Estuary in Maine.  Resources, tools and strategies for
pollution abatement can be effectively targeted at priority sources if
estuaries are to be protected. Understanding the sources and annual
loading of contaminants to an estuary guides good water quality
management by defining the range of controls of both air and water
pollution needed to achieve a desired result. The cost of conducting
monitoring to determine atmospheric loading to a water body can be
prohibitively high. Also, collection of monitoring data is a long-term
undertaking, since a minimum of three                                
years of data is advisable in order to “smooth out” inter-annual
variability. The estimation techniques described in this paper can serve
as a useful and inexpensive “first-cut” at understanding the
importance of the atmospheric as a pollution source, and can help to
pinpoint those areas where field measurements are needed to guide future
management decisions.

Management Consideration: If a review of information on air deposition
available for the analysis indicates a wide range of potential
deposition rates, then further study of this input would lead to better
characterization of the air contribution to overall contamination.  If
the back-of-the envelope analysis suggests that air deposition is very
small relative to other inputs, then resources should be targeted at
studying or reducing other inputs before proceeding with further
analysis of the air inputs.  

Document Availability.  The back-of-the envelope calculation is outlined
in Frequently Asked Questions about Atmospheric Deposition: A Handbook
for Watershed Managers (available at   HYPERLINK
"http://www.epa.gov/air/oaqps/gr8water/handbook/airdep_sept.pdf" 
http://www.epa.gov/air/oaqps/gr8water/handbook/airdep_sept.pdf  ).  

Further analysis is available in Deposition of Air Pollutants to the
Great Waters - Third Report to Congress (available at   HYPERLINK
"http://www.epa.gov/air/oaqps/gr8water/3rdrpt/index.html" 
http://www.epa.gov/air/oaqps/gr8water/3rdrpt/index.html  )

The Casco Bay Estuary examples is available at:    HYPERLINK
"http://www.epa.gov/owow/airdeposition/index.html" 
www.epa.gov/owow/airdeposition/index.html .  

Contact Person.  Gail Lacy at (919) 541-5261 at   HYPERLINK
"mailto:lacy.gail@epa.gov"  lacy.gail@epa.gov .

Contact for Casco Site is:  Diane Gould at gould.diane@epa.gov.

Group 2 Case Studies

Case Study 3: Probabilistic Assessment of Angling Duration Used in
Assessment of Exposure to Hudson River Sediments via Consumption of
Contaminated Fish

In assessing the health impact of contaminated Superfund sites, exposure
duration typically is assumed to be the same as the length of time an
individual lives in a specific area (i.e., residence duration). In
conducting the human health risk assessment for the Hudson River PCB
Superfund Site, however, there was concern that exposure duration based
on residence duration may underestimate the time spent fishing (i.e.,
angling duration).

Risk Analysis. An individual may move from one residence to another and
continue to fish in the same location, or an individual may choose to
stop fishing irrespective of the location of his or her home. EPA Region
2 developed a site-specific distribution of angling duration using the
fishing patterns reported in a New York State-wide angling survey
(Connelly et al., 1991) and migration data for the five counties
surrounding the 40-plus miles of the Upper Hudson River collected as
part of the U.S. Census.

Results of Analysis. The 50th and 95th percentile values from the
distribution of angling durations were higher than the default values
based on residence duration using standard default exposure assumptions
for residential scenarios and were used as bases for the central
tendency and reasonable maximum exposure point estimates, respectively,
in the deterministic assessment.

Management Considerations. The information provided in this analysis was
used in the point estimate analysis. The full distribution was used in
conducting a Group 2 PRA for the fish consumption pathway, which is
presented as Case Study 6.

Document Availability. The final risk assessment was released in
November 2000 (available at   HYPERLINK
"http://www.epa.gov/hudson/reports.htm" 
http://www.epa.gov/hudson/reports.htm ).

Contacts. Remedial Project Manager, Alison Hess, 212-637-3959; Risk
Assessor, Marian Olsen, 212-637-4313

Case Study 4: Probabilistic Analysis of Dietary Exposure to Pesticides
for use in Setting Tolerance Levels

Under the Federal Food, Drug, and Cosmetic Act (FFDCA), EPA may
authorize a tolerance or exemption from the requirement of a tolerance,
to allow a pesticide residue in food, only if the Agency determines that
such residues would be “safe”.  This determination is made by
estimating exposure to the pesticide and comparing the estimated
exposure to a toxicological benchmark dose (i.e., a dose where there is
reasonable certainty of no harm).  Until 1998, Office of Pesticide
Programs (OPP) used a software program called the Dietary Risk
Evaluation System (DRES) to conduct its acute dietary risk assessments
for pesticide residues in foods. Acute assessments conducted with DRES
assumed that 100% of a given crop with registered uses of a pesticide
was treated with that pesticide and that all such treated crop items
contained pesticide residues at the maximum legal (tolerance) level
matching this to a reasonably high consumption value (around 95th
percentile). The resulting DRES acute risk estimates were considered
"high-end" or "bounding" estimates.   However, it was not possible to
know where the pesticide exposure estimates from the DRES software fit
in the overall distribution of exposures due to the limits of the tools
being used. 

Approach:  To address these deficiencies, OPP has developed an acute
probabilistic dietary exposure guidance in order to use a model to
estimate exposure to pesticides in the food supply.  Rather than the
crude "high-end," single point estimates provided by deterministic
assessments, the probabilistic Dietary Exposure Evaluation Model (DEEM)
provides specific information on the range and probability of possible
exposures and depending upon the characterization of the input, 95th
percentile regulation generally for lower tiers that do not include
percent crop treated, to the 99.9th percentile for the more refined
assessments which would include percent of crop treated information.  

Probabilistic Analysis.  This case study provides an example of a
one-dimensional probabilistic risk assessment of dietary exposure to
pesticides (Group 2).  The DEEM generates acute, probabilistic dietary
exposure assessments using data on (1) the distribution of daily
consumption of specific commodities (e.g., wheat, corn, apples, etc.) by
specific individuals, and (2) the distribution of concentrations of a
specific pesticide in those food commodities.  Data on commodity
consumption are collected by USDA in its Continuing Survey of Food
Intake by Individuals (CSFII). Pesticide residue concentrations on food
commodities are generally obtained from crop field trials, USDA’s
Pesticide Data Program (PDP) data, Food and Drug Administration (FDA)
monitoring data, or market basket surveys conducted by the registrants. 
Using these data, DEEM is able to calculate an estimate of the risk to
the general U.S. population in addition to 26 population subgroups,
including five subgroups for infants and children (infants less than 1,
children 1-2, children 3-5, youth 6-12 and teen 13-19).

Results of Analysis.  DEEM has been used in risk assessments to support
tolerance levels for several pesticides (e.g., phosalone) and as part of
cumulative risk assessments for organophosphorus compounds (see Case
Study 11) and other pesticides.  

Management Considerations:  Using the DRES, risk management decisions
were being made without a full picture of the distribution of risk among
the population, and also without full knowledge of where in the
distribution of risk the DRES risk estimate lay.  This was of concern
not only for regulators interested in public health protection, but also
for the pesticide registrants who could argue that the Agency was being
arbitrary in selecting the level at which to regulate. For most cases
reviewed by OPP to date, estimated exposure at the 99.9th percentile
calculated by DEEM probabilistic techniques is significantly lower than
exposure calculated using DRES-type deterministic assumptions at the
unknown percentile.

Document Availability.  

Link to DEEM model available at   HYPERLINK
"http://www.epa.gov/oppsrrd1/cumulative/methods_tools.htm" 
http://www.epa.gov/oppsrrd1/cumulative/methods_tools.htm   

Contact Person.

David Hrdy at (703) 305-6990 or hrdy.david@epa.gov.

Case Study 5: One-Dimensional Probabilistic Risk Analysis of Exposure
to Polychlorinated Biphenyls (PCBs) via Consumption of Fish from a
Contaminated Sediment Site

EPA Region 2 conducted a preliminary deterministic human health risk
assessment at the Hudson River PCBs Superfund site. The deterministic
risk analysis showed that consumption of recreationally caught fish
provided the highest exposure among relevant exposure pathways and
resulted in cancer risks and noncancer health hazards that exceeded
regulatory benchmarks.

Probabilistic Analysis. Because of the size, complexity, and high level
of public interest in this site, EPA Region 2 implemented a Group 2
probabilistic assessment to characterize the variability in risks
associated with the fish consumption exposure pathway. The analysis was
a 1-Dimensional Monte Carlo analysis of the variability of exposure as a
function of the variability of individual exposure factors.  Uncertainty
was assessed using sensitivity analysis of the input variables. Data to
characterize distributions of exposure parameters were drawn from the
published literature (e.g., fish consumption rate) or from existing
databases such as the U.S. Census data (e.g., angling duration, see Case
Study 3). Mathematical models of the environmental fate, transport, and
bioaccumulation of PCBs in the Hudson River previously developed were
used to forecast changes in PCB concentration over time.

Results of Analysis. The results of the PRA were in line with the
deterministic results. For the Central Tendency individual, point
estimates were near the median (50th percentile). For the Reasonable
Maximum Exposure individual, point estimate values were at or above the
95th percentile of the probabilistic analysis. The deterministic and
probabilistic risk analyses were the subject of a formal peer review by
a panel of independent experts. 

The Monte Carlo base case scenario is the one from which point estimate
exposure factors for

fish ingestion were drawn, thus the point estimate RMEs and the Monte
Carlo base case estimates can be compared. Similarly, the point estimate
central tendency (average) and the Monte Carlo base case midpoint (50th
percentile) are comparable. For cancer risk, the point estimate RME for
fish ingestion (1 x 10-3) falls approximately at the 95th percentile
from the Monte Carlo base case analysis. The point estimate central
tendency value (3 x10-5) and the Monte Carlo base case 50th
percentile value (6 x10-5) are similar. For non-cancer health
hazards, the point estimate RME for fish ingestion (104 for young child)
falls between the 95th and 99th percentiles of the Monte Carlo base
case. The point estimate central tendency hazard index (HI) (12 for
young child) is approximately equal to the 50th percentile of the Monte
Carlo base case HI of 11.  Figures 1 and 2 provide a comparison of
results from the probabilistic analysis with that of the deterministic
risk analysis for cancer risks and non-cancer health hazards.  

Management Considerations. Early and continued involvement of the
community improved public acceptance of the results. In addition,
careful consideration of the methods used to present the probabilistic
results to the public lead to greater understanding of the findings.

Document Availability. The final risk assessment was released in
November 2000 (available at   HYPERLINK
"http://www.epa.gov/hudson/reports.htm" 
http://www.epa.gov/hudson/reports.htm ).

Contacts. Remedial Project Manager, Alison Hess, 212-637-3959; Risk
Assessor, Marian Olsen, 212-637-4313

A comparison of results from the probabilistic analysis with that of the
deterministic risk analysis for cancer risks and non-cancer health
hazards.  Figures 1 and 2 plot percentiles for 72 combinations of
exposure variables (e.g., distributions from creel angler surveys;
residence duration; fishing locations; cooking losses, etc.) of the
non-cancer Hazard Index values and the cancer risks, respectively.  In
each of these figures, the variability of cancer risk or non-cancer HIs
for anglers within the exposed population is plotted on the y axis for
particular percentiles within the population.  This variability is a
function of the variations in fish consumption rates, fishing duration,
differences in fish species ingested, etc.  The uncertainty in the
estimates is indicated by the range of either cancer risk or non-cancer
HI values plotted on the x-axis.  This uncertianty is a function of the
72 combinations of the exposure factor inputs examined in the
sensitivity analysis.  This analysis provides a semi-quantitative
confidence interval for the cancer risks and HI values at any
particulate percentile.  As these figures show, the intervals span
somewhat less than two orders of magnitude (e.g., < 100 fold). The
vertical lines indicate the deterministic endpoints.

 

Figure 2.  Monte Carlo Non-Cancer Hazard Index Summary Based on a
One-Dimensional Probabilistic Risk Analysis of Exposure to
Polychlorinated Biphenyls (PCBs) via Consumption of Fish from a
Contaminated Sediment Site.  From Phase 2 Reprot:  Further Site
Characterization and Analysis.  Volume 2F – Revised Human Health Risk
Assesment, Hudson River PCBs Reassessment RI/FS.  U.S. EPA, November
2000.

 

Case Study 6: Probabilistic Sensitivity Analysis of Expert Elicitation
of Concentration-Response Relationship Between Particulate Matter
(PM2.5) Exposure and Mortality

In 2002, the National Research Council (NRC) recommended that EPA
improve its characterization of uncertainty in the benefits assessment
for proposed regulations of air pollutants. NRC recommended that
probability distributions for key sources of uncertainty be developed
using available empirical data or through formal elicitation of expert
judgments. In response to this recommendation, EPA conducted an expert
elicitation evaluation of the concentration-response relationship
between PM2.5 exposure and mortality, a key component of the benefits
assessment of the PM2.5 regulation. Further information on the expert
elicitation procedure and results is provided in Case Study 12. To
evaluate the degree to which the results of the assessment depended on
individual experts’ judgments or on the methods of expert elicitation,
a probabilistic sensitivity analysis was performed of the results

concentrations from 12 to 11 μg/m3. The 12 individual distributions of
estimated avoided deaths were then pooled using equal weights to create
a single overall distribution reflecting input from each expert. This
distribution served as the baseline for the sensitivity analysis, which
compared the means and standard deviations of the baseline distribution
with several variants.

Results of Analysis. The first analysis examined sensitivity of the mean
and standard deviation of the overall mortality distribution to the
removal of individual experts’ distributions. In general, the results
suggested a fairly equal split between those experts whose removal
shifted the distribution mean up and those who shifted it down and
relatively modest impacts of individual experts. The standard deviation
of the combined distribution also was not affected strongly by removal
of individual experts. The second analysis evaluated whether the use of
parametric or nonparametric approaches affected the overall results. The
results suggested that the use of parametric distributions led to
distributions with similar or slightly increased uncertainty compared
with distributions provided by experts who offered percentiles of a
nonparametric distribution. The last analyses evaluated whether
participation in the Pre- or Post-elicitation Workshops impacted the
results. Participation in either workshop did not appear to have a
significant effect on experts’ judgments, based on measures of change
in the baseline distribution. Overall, the sensitivity analyses
demonstrated that the assessment was robust, with little dependence on
individual experts’ judgments or on the specific elicitation methods
evaluated.

Management Considerations. The sensitivity analysis demonstrated the
robustness of the PM 2.5 expert elicitation-based assessment by showing
that the panel of experts was generally well balanced and that
alternative elicitation methods would not have markedly altered the
overall results.  

Document Availability. The details of this analysis are provided in the
IEC document titled: “Expanded Expert Judgment Assessment of the
Concentration-Response Relationship Between PM2.5 Exposure and
Mortality” Final Report, September 21, 2006 (  HYPERLINK
"http://www.epa.gov/ttn/ecas/regdata/uncertainty/pm_ee_report.pdf" 
www.epa.gov/ttn/ecas/regdata/uncertainty/pm_ee_report.pdf ).

The expert elicitation assessment, along with the Regulatory Impact
Analysis (RIA) of the PM2.5 standard, is available at   HYPERLINK
"http://www.epa.gov/ttn/ecas/ria.html" 
http://www.epa.gov/ttn/ecas/ria.html .

Contact. Lisa Conner, 919-541-5060,   HYPERLINK
"mailto:conner.lisa@epa.gov"  conner.lisa@epa.gov 

Case Study 7: Environmental Monitoring and Assessment Program (EMAP):
Using Probabilistic Sampling to Evaluate the Condition of the Nation’s
Aquatic Resources

Monitoring is a key tool used to identify where the environment is in
healthy biological condition and requires protection, and where
environmental problems are occurring and need remediation.  However,
most monitoring is not currently done in a way that allows for
statistically-valid assessments of water quality conditions in
unmonitored waters (GAO 2000).  States thus cannot adequately measure
the status and trends in water quality in their waters as required by
Clean Water Act Section 305(b).  

EMAP’s focus has been to develop unbiased statistical survey design
frameworks, and sensitive indicators that allow the condition of aquatic
ecosystems to be assessed at state, regional, and national scales.  A
cornerstone of EMAP has been the use of probabilistic sampling to allow
representative, unbiased, cost-effective condition assessments for
aquatic resources over large areas.  EMAP’s statistical survey methods
are very efficient, requiring relatively few sample locations to make
valid scientific statements about the condition of aquatic resources
over large areas (e.g., the condition of all the wadeable streams in the
Western US).  

Probabilistic Analysis. This research program had a number of case
studies using probabilistic sampling designs for different aquatic
resources (estuaries, streams, and rivers). An EMAP probability-based
sampling program provides an unbiased estimate of the condition of an
aquatic resource over a large geographic area from a small number of
samples. The principal characteristics of a probabilistic sampling
design are:  the population being sampled is unambiguously described;
every element in the population has the opportunity to be sampled with a
known probability; and sample selection is carried out by a random
process. This approach allows statistical confidence levels to be placed
on the estimates and provides the potential to detect statistically
significant changes and trends in condition with repeated sampling. In
addition, this approach permits the aggregation of data collected from
smaller areas to predict the condition of a large geographic area.

The EMAP design framework allows the selection of unbiased,
representative sampling sites and specifies the information to be
collected at these sites. The validity of the overall inference rests on
the design and subsequent analysis to produce regionally representative
information. The EMAP uses   HYPERLINK
"http://www.epa.gov/nheerl/arm/documents/presents/grts_ss.pdf" 
Generalized Random Tessellation Stratified (GRTS) Spatially-Balanced
Survey Designs for Aquatic Resources .  The spatially-balanced aspect
spreads out the sampling locations geographically, but still ensures
that each element has an equal chance of being selected.

Results of the Analysis.  Data collected using the EMAP approach has
allowed the Agency to make scientifically defensible assessments of the
ecological condition of large geographic areas for reporting to Congress
under CWA 305(b).  The EMAP approach has been used to provide the first
reports on the condition of the nation’s estuaries, streams, rivers
and lakes, and it is scheduled to used for wetlands.  EMAP findings have
been included in EPA’s Report on the Environment, and the Heinz
Center’s The State of the Nation’s Ecosystems.  Data collected
through an EMAP approach improve the ability to assess ecological
progress in environmental protection and restoration, and provide
valuable information for decision-makers and the public. The use of
probabilistic analysis methods allows meaningful assessment and regional
comparisons of aquatic ecosystem conditions across the United States.
Finally, the probabilistic approach provides scientific credibility for
the monitoring network and aids in identifying data gaps.

Management Considerations.  Use of an EMAP approach addresses criticisms
from the General Accounting Office, the National Academies of Sciences
(NAS), the Heinz Center (a nonprofit environmental policy institution),
and others who noted the nation lacked the data to make scientifically
valid characterizations of water quality regionally and across the
United States. The program provides cost-effective, scientifically
defensible, and representative data, to permit the evaluation of
quantifiable trends in ecosystem condition, to support performance-based
management, and to facilitate better public decisions regarding
ecosystem management. EMAP’s approach has now been adopted by the
EPA’s Office of Water (OW) to collect data on the condition of all the
nation’s aquatic resources.  OW, Office of Air and Radiation and
Office of Prevention, Pesticides, and Toxic Substances now have
environmental accountability endpoints using EMAP approaches in their
Agency performance goals.

Document Availability. Available at   HYPERLINK
"http://www.epa.gov/emap/index.html"  http://www.epa.gov/emap/index.html
.

U. S. EPA. 2002. Research Strategy. Environmental Monitoring and
Assessment Program. U.S. EPA, Office of Research and Development,
National Health and Environmental Effects Research Laboratory. U.S. EPA,
Research Triangle Park, NC. Available at   HYPERLINK
"http://www.epa.gov/emap/html/pubs/docs/resdocs/emap_research_strategy.p
df"  www.epa.gov/emap/html/pubs/docs/resdocs/emap_research_strategy.pdf
.

Information on EMAP monitoring designs is available at 

http://www.epa.gov/nheerl/arm/designpages/monitdesign/monitoring_design_
info.htm

Contacts. Michael McDonald, 919-541-7973,   HYPERLINK
"mailto:mcdonald.michael@epa.gov"  mcdonald.michael@epa.gov ; Tony
Olsen, 541-754-4790,   HYPERLINK "mailto:olsen.tony@epa.gov" 
olsen.tony@epa.gov 

Group 3 Case Studies

Case Study 8: Two-Dimensional Probabilistic Risk Analysis of
Cryptosporidium in Public Water Supplies, with Bayesian Approaches to
Uncertainty Analysis

Probabilistic assessment of the occurrence and health effects associated
with Cryptosporidium bacteria in public drinking water supplies was used
to support the economic analysis of the final Long-Term 2 Enhanced
Surface Water Treatment Rule (LT2).  EPA’s Office of Ground Water and
Drinking Water (OGWDW) conducted this analysis and established a
baseline disease burden attributable to Cryptosporidium in Public Water
supplies that use surface water sources.  Next, it models the source
water monitoring and finished water improvements that will be realized
as a result of the Rule.  Post-Rule risk is estimated and the Rule’s
health benefit is the result of subtracting this from the baseline
disease burden.

Probabilistic Risk Analysis.  Probabilistic assessment was used for this
analysis as a means of addressing the variability in the occurrence of
Cryptosporidium in raw water supplies, the variability in the treatment
efficiency, as well as the uncertainty in these inputs and in the
dose-response relationship for Cryptosporidium infection.  This case
study provides an example of a PRA that evaluates both variability and
uncertainty at the same time and is referred to as a two-dimensional
probabilistic risk assessment.    The analysis also included
probabilistic treatments of uncertain dose-response and occurrence
parameters.  Markov Chain Monte Carlo samples of parameter sets filled
this function.  This  Bayesian approach (treating the unknown parameters
as random variables) differs from classical treatments, which would
regard the parameters as unknown, but fixed (Group 3: Advanced PRA). 
The risk analysis used existing datasets (e.g., occurrence of
Cryptosporidium and treatment efficacy) to inform the variability of
these inputs.  Uncertainty distributions were characterized based on
professional judgment or by analyzing data using Bayesian statistical
techniques.  

Results of Analysis. The risk analysis identified the Cryptosporidium
dose-response relationship as the most critical model parameters in the
assessment, followed by the occurrence of the pathogen and treatment
efficiency.  By simulating implementation of the Rule using imprecise,
biased measurement methods, the assessment provided estimates of the
number of public water supply systems that would require corrective
action and the nature of the actions likely to be implemented.  This
information afforded a realistic measure of the benefits (in reduced
disease burden) expected with the LT2 rule.  In response to SAB
comments, additional Cryptosporidium dose-response models were added to
more fully reflect uncertainty in this element of the assessment.

Management Considerations. The rule underwent external peer review,
review by EPA’s Science Advisory Board (SAB) and intense review by the
Office of Management and Budget (OMB).  Occurrence and dose-response
components of the risk analysis model were communicated to stakeholders
during the Rule’s Federal Advisory Committee Act (FACA) process.  Due
to the rigor of the analysis and the signed FACA “Agreement in
Principle”, the OMB review was straight-forward.

Document Availability. The final assessment of occurrence and exposure
to Cryptosporidium was released in December 2005 (available at  
HYPERLINK
"http://www.epa.gov/safewater/disinfection/lt2/regulations.html" 
http://www.epa.gov/safewater/disinfection/lt2/regulations.html ).

Contact. Michael Messner, 202-564-5268,   HYPERLINK
"mailto:messner.michael@epa.gov"  messner.michael@epa.gov 

Case Study 9: Two-Dimensional Probabilistic Model of Children’s
Exposure to Arsenic in Chromated Copper Arsenate (CCA) Pressure-Treated
Wood

Probabilistic models were developed in response to EPA’s October 2001
Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA) Scientific
Advisory Panel (SAP) recommendations to use probabilistic modeling to
estimate children’s exposures to arsenic from chromated copper
arsenate (CCA) treated playsets and home decks.  

Probabilistic Risk Analysis.  EPA’s Office of Research and Development
(ORD), in collaboration with the Office of Pesticide Programs (OPP)
developed and applied a probabilistic exposure assessment of
children’s exposure to arsenic and chromium from contact with
CCA-treated wood playsets and decks.  This case study provides an
example of the use of two-dimensional (i.e., addressing both variability
and uncertainty) probabilistic exposure assessment (Group 3: Advanced
PRA).  The two-dimensional assessment employed a modification of the
ORD’s SHEDS (Stochastic Human Exposure and Dose Simulation) model to
simulate children’s exposure to arsenic and chromium from CCA-treated
wood playsets and decks, and surrounding soil.  Staff from both ORD and
OPP collaborated in the development of the SHEDS-Wood model.  

Results of Analysis.  A draft of the probabilistic exposure assessment
received SAP review in December, 2003; the final report was released in
2005.  The results of the probabilistic exposure assessment were
consistent with or in the range of the results of deterministic exposure
assessments conducted by several other organizations.  The model results
were used to compare exposures under a variety of scenarios, including
cold vs. warm weather activity patterns, use of wood sealants to reduce
the availability of contaminants on the surface of the wood, and
different hand-washing frequencies.  The modeling of alternative
mitigation scenarios indicated that the use of sealants could result in
the greatest exposure reduction, while increased frequency of
hand-washing could also reduce exposure.  

OPP used the SHEDS-Wood exposure results in their probabilistic
children’s risk assessment for CCA (EPA, 2008).  This included
recommendations for risk reduction (use of sealants and careful
attention to children’s hand-washing) to homeowners with existing CCA
wood structures.  In addition, the exposure assessment was used to
identify areas for further research, including: the efficacy of wood
sealants in reducing dislodgeable contaminant residues, the frequency
with which children play on or around CCA wood, and transfer efficiency
and residue concentrations.  In order to better characterize the
efficacy of sealants in reducing exposure, EPA and the Consumer Product
Safety Commission conducted a 2-year study of how dislodgeable
contaminant residue levels changed with the use of various types of
commercially-available wood sealants.  

Management Considerations.  The SHEDS-wood model was one of Agency’s
first probabilistic modeling assessments for regulatory purposes.  The
OPP used SHEDS results directly in their final risk assessment for
children playing on CCA treated playground equipment and decks. The
model enhanced risk assessment and management decisions and prioritized
data needs related to wood preservatives.  The modeling product was
pivotal in the risk management and re-registration eligibility decisions
for CCA, and in advising the public how to minimize health risks from
existing treated wood structures. Industry is also using SHEDS to
estimate exposures to CCA and other wood preservatives. Some states are
using the risk assessment as guidance in setting their regulations for
CCA related playground equipment.

Document Availability.  

The model results were included in the final report on the probabilistic
exposure assessment of CCA-treated wood surfaces: 

Zartarian, V.G., J. Xue, H. A. Ozkaynak, W. Dang, G. Glen, L. Smith, and
C. Stallings. A Probabilistic Exposure Assessment for Children Who
Contact CCA-treated Playsets and Decks Using the Stochastic Human
Exposure and Dose Simulation Model for the Wood Preservative Scenario
(SHEDS-WOOD), Final Report. U.S. Environmental Protection Agency,
Washington, DC, EPA/600/X-05/009.   HYPERLINK
"http://www.epa.gov/heasd/sheds/cca_treated.htm" 
http://www.epa.gov/heasd/sheds/cca_treated.htm 

The final probabilistic risk assessment based on the SHEDS-Wood exposure
assessment can be found at:
http://www.epa.gov/oppad001/reregistration/cca/final_cca_factsheet.htm

Results of the sealant studies were released in January, 2007 (available
at   HYPERLINK
"http://www.epa.gov/oppad001/reregistration/cca/index.htm#reviews" 
http://www.epa.gov/oppad001/reregistration/cca/index.htm#reviews  ).

The results of the analysis were published as:  

Zartarian, V.G., Xue, J., Özkaynak, H., Dang, W., Glen, G., Smith, L.,
and Stallings, C.   A probabilistic arsenic exposure assessment for
children who contact chromated copper arsenate (CAA)-treated playsets
and decks, Part 1:  Model methodology, variability results, and model
evaluation.  Risk Analysis 26:515, 2006.

Contact.  Valerie G. Zartarian, Ph.D., EPA's Office of Research and
Development, 617-918-1541,   HYPERLINK
"mailto:zartarian.valerie@epa.gov"  zartarian.valerie@epa.gov  

Case Study 10: Two-Dimensional Probabilistic Exposure Assessment of
Ozone

As part of EPA’s recent review of the ozone National Ambient Air
Quality Standards (NAAQS), the Office of Air Quality Planning and
Standards (OAQPS) conducted detailed probabilistic exposure and risk
assessments in evaluating potential alternative standards for ozone. At
different stages of this review, the Clean Air Scientific Advisory
Committee (CASAC) Ozone Panel (an independent scientific review
committee of EPA’s SAB) and the public reviewed and provided comments
on analyses and documents prepared by EPA. A scope and methods plan for
the exposure and risk assessments was developed in 2005 (EPA, 2005).
This plan was intended to facilitate consultation with CASAC, as well as
public review, and to obtain advice on the overall scope, approaches,
and key issues in advance of the completion of the analyses. This case
study describes the probabilistic exposure assessment, addressing
short-term exposures to ozone. The exposure estimates were used as an
input to the health risk assessment for lung function decrements in all
children and asthmatic school-aged children based on exposure-response
relationships derived from controlled human exposure studies.

Probabilistic Exposure Analysis. Population exposure to ambient ozone
levels was evaluated using EPA’s APEX model, also referred to as the
Total Risk Integrated Methodology/Exposure (TRIM.Expo) model. Exposure
estimates were developed for recent ozone levels, based on 2002 to 2004
air quality data, and for ozone levels simulated to just meet the
existing 0.08 ppm, 8-h ozone NAAQS and several alternative ozone
standards, based on adjusting 2002 to 2004 air quality data. Exposure
estimates were modeled for 12 urban areas located throughout the United
States for the general population, all school-age children, and
asthmatic school-age children. This exposure assessment is described in
a technical report (EPA, 2007b). The exposure model, APEX, is documented
in a user’s guide and technical document (EPA, 2006a,b). A Monte Carlo
approach was used to produce quantitative estimates of the uncertainty
in the APEX model results, based on estimates of the uncertainties for
the most important model inputs. The quantification of model input
uncertainties, a discussion of model structure uncertainties, and the
results of the uncertainty analysis are documented in Langstaff (2007).

Results of Analysis. Uncertainty in the APEX model predictions results
from uncertainties in the spatial interpolation of measured
concentrations, the microenvironment models and parameters, people’s
activity patterns, and, to a lesser extent, model structure. The
predominant sources of uncertainty appear to be the human activity
pattern information and the spatial interpolation of ambient
concentrations from monitoring sites to other locations. The primary
policy-relevant finding was that the uncertainty in the exposure
assessment is small enough to lend confidence to the use of the model
results for the purpose of informing the Administrator’s decision on
the ozone standard.

The following figure illustrates the uncertainty distribution for one
model result, the percent of children with exposures above 0.08 ppm-8hr
while at moderate exertion.  The “point estimate” of 20 percent is
the result from the APEX simulation using the best estimates of the
model inputs. The corresponding result from the Monte Carlo simulations
ranges from 17 to 26 percent, with a 95 percent uncertainty interval
(UI) of 19 to 24 percent.  Note that the uncertainty intervals are not
symmetric since the distributions are skewed.

Management Considerations. The exposure analysis also provided
information on the frequency with which population exposures exceeded
several potential health effect benchmark levels that were identified
based on evaluation of health effects in clinical studies.

The exposure and risk assessments are summarized in Chapters 4 and 5,
respectively, of the Ozone Staff Paper (EPA, 2007a). The exposure
estimates over these potential health effect benchmarks were part of the
basis for the Administrator’s March 27, 2008, decision to revise the
ozone NAAQS from 0.08 to 0.075 ppm, 8-h average (see 73 FR 16436).

Document Availability.

 Langstaff, J. E. (2007). Analysis of Uncertainty in Ozone Population
Exposure Modeling. OAQPS Staff Memorandum to Ozone NAAQS Review Docket
(OAR-2005-0172). Available at   HYPERLINK
"http://www.epa.gov/ttn/naaqs/standards/ozone/s_o3_cr_td.html" 
http://www.epa.gov/ttn/naaqs/standards/ozone/s_ozone_cr_td.html .

EPA (2007a). Review of National Ambient Air Quality Standards for Ozone:
Assessment of Scientific and Technical Information - OAQPS Staff Paper.
OAQPS, U.S. EPA, RTP, NC. Available at   HYPERLINK
"http://www.epa.gov/ttn/naaqs/standards/ozone/s_o3_cr_sp.html" 
http://www.epa.gov/ttn/naaqs/standards/ozone/s_ozone_cr_sp.html .

EPA (2007b). Ozone Population Exposure Analysis for Selected Urban
Areas. OAQPS, U.S. EPA, RTP, NC. Available at   HYPERLINK
"http://www.epa.gov/ttn/naaqs/standards/ozone/s_o3_cr_td.html" 
http://www.epa.gov/ttn/naaqs/standards/ozone/s_ozone_cr_td.html .

EPA (2006a,b). Total Risk Integrated Methodology (TRIM) - Air Pollutants
Exposure Model Documentation (TRIM.Expo / APEX, Version 4) Volume I:
User’s Guide; Volume II: Technical Support Document. OAQPS, U.S. EPA,
RTP, NC. June 2006. Available at   HYPERLINK
"http://www.epa.gov/ttn/fera/human_apex.html" 
http://www.epa.gov/ttn/fera/human_apex.html .

EPA (2005). Ozone Health Assessment Plan: Scope and Methods for Exposure
Analysis and Risk Assessment. OAQPS, U.S. EPA, RTP, NC. Available at  
HYPERLINK "http://www.epa.gov/ttn/naaqs/standards/ozone/s_o3_cr_pd.html"
 http://www.epa.gov/ttn/naaqs/standards/ozone/s_o3_cr_pd.html .

Contact. John E. Langstaff, EPA’s Office of Air and Radiation,
919-541-1449,   HYPERLINK "mailto:Langstaff.John@epa.gov" 
Langstaff.John@epa.gov 

Case Study 11: Analysis of Microenvironmental Exposures to Particulate
Matter (PM2.5) for a Population Living in Philadelphia, PA

This case study used the Stochastic Human Exposure and Dose Simulation
model for particulate matter (SHEDS-PM) developed by EPA’s National
Exposure Research Laboratory (NERL) to prepare a probabilistic
assessment of population exposure to particulate matter (PM) in
Philadelphia, PA.  This case study simulation was prepared to accomplish
three goals:  1) to estimate the contribution of PM of ambient (outdoor)
origin to total PM exposure, 2) to determine the major factors that
influence personal exposure to PM, and 3) to identify factors that
contribute the greatest uncertainty to model predictions.  

Probabilistic Risk Analysis.  The two-dimensional probabilistic
assessment used a microexposure event technique to simulate individual
exposures to PM in specific microenvironments (outdoors, indoor
residence, office school, store, restaurant or bar, and in a vehicle). 
The assessment used spatially and temporally interpolated ambient PM2.5
measurements from 1992-93 and 1990 U.S. Census data for each census
tract in Philadelphia.  This case study provides an example of both
two-dimensional (variability and uncertainty) probabilistic assessment
and microexposure event assessment (Group 3: Advanced PRA).  

Results of Analysis.  Results of the analysis showed that that human
activity patterns did not have as strong an influence on ambient PM2.5
exposures as was observed for exposure to indoor PM2.5 sources. 
Exposure to PM2.5 of ambient origin contributed a significant percent of
the daily total PM2.5 exposures, especially for the segment of the
population without exposure to environmental tobacco smoke in the
residence. Development of the SHEDS-PM model using the Philadelphia
PM2.5 case study also provided useful insights into data needs for
improving inputs to the SHEDS-PM model, reducing uncertainty and further
refinement of the model structure.  Some of the important data needs
identified from the application of the model include: daily PM
measurements over multiple seasons and across multiple sites within an
urban area, improved capability of dispersion models to predict ambient
PM concentrations at greater spatial resolution and over a year time
period, measurement studies to better characterize the physical factors
governing infiltration of ambient PM2.5 into residential
microenvironments, further information on particle-generating sources
within the residence, and data for the other indoor microenvironments
not specified in the model.  

  

Management Considerations:  The application of the SHEDS-PM model to the
Philadelphia population gave insights into data needs and areas for
model refinement.  The continued development and evaluation of the
SHEDS-PM population exposure model are being conducted as part of
EPA/ORD’s effort to develop a source-to-dose modeling system for PM
and air toxics. This type of exposure and dose modeling system is
considered to be important for scientific and policy evaluation of the
critical pathways, as well as exposure factors and source types
influencing human exposures to a variety of environmental pollutants,
including particulate matter.



Document Availability.  

The assessment is available at   HYPERLINK
"http://www.epa.gov/heasd/pm/pdf/exposure-model-for-pm.pdf" 
http://www.epa.gov/heasd/pm/pdf/exposure-model-for-pm.pdf .

 

The results of the analysis were published as:  

Burke, J., Zufall, M., and Özkaynak, H.  A population exposure model
for particulate matter: Case study results for PM2.5 in Philadelphia,
PA.  Journal of Exposure Analysis and Environmental Epidemiology 11:
470, 2001. 

Georgepoulos,  P.G., Wang, S.-W., Vyas, V.M., Sun, Q., Burke, J.,
Vedantham, R., McCurdy, T., and Ozkayanak, H.  A source-to-dose
assessment of poulation exposure to fine PM and ozone in Philadelphia,
PA, during a summer 1999 episode.  J. Expos. Analysis and Environm. Epi.
1-19, 2005. 

Contact:.  Janet M. Burke, EPA’s Office of Research and Development,
919-541-0820,   HYPERLINK "mailto:burke.janet@epa.gov" 
burke.janet@epa.gov 

Case Study 12: Probabilistic Analysis in Cumulative Risk Assessment of
Organophosphorus Pesticides

In 1996, Congress enacted the Food Quality Protection Act (FQPA), which
requires EPA to consider “available evidence concerning the cumulative
effects on infants and children of such residues and other substances
that have a common mechanism of toxicity” when setting pesticide
tolerances (i.e., the maximum amount of pesticide residue legally
allowed to remain on food products). FQPA also mandated that EPA
completely reassess the safety of all existing pesticide tolerances
(those in effect since August 1996) to ensure that they are supported by
up-to-date scientific data and meet current safety standards. Because
organophosphorus pesticides (OPs) were assigned priority for tolerance
reassessment, these pesticides were the first “common mechanism”
group identified by EPA’s OPP. The ultimate risk management goal
associated with this cumulative risk assessment (CRA) was to establish
safe tolerance levels for this group of pesticides, while meeting the
FQPA standard for protecting infants and children.

Probabilistic Risk Analysis. This case study provides an example of an
advanced probabilistic assessment (Group 3). In 2006, EPA analyzed
exposures to 30 OPs through food consumption, consumption of drinking
water, and exposure via pesticide application. EPA used Calendex, a
probabilistic computer software program (available at   HYPERLINK
"http://www.exponent.com/calendex_software" 
http://www.exponent.com/calendex_software ), to integrate various
pathways, while simultaneously incorporating the time dimensions of the
input data. Based on the results of the exposure assessment, EPA
calculated margins of exposure (MOEs) for the total cumulative risk from
all pathways.

The food component of the OPs CRA was highly refined, as it was based on
residue monitoring data from the USDA’s PDP and supplemented with
information from the FDA’s Surveillance Monitoring Programs and Total
Diet Study. The residue data were combined with actual consumption data
from USDA’s Continuing Survey of Food Intakes by Individuals using
probabilistic techniques. The CRA evaluated drinking water exposures on
a regional basis. The assessment focused on areas where combined OP
exposure is likely to be highest within each region. Primarily, the
analysis used probabilistic modeling to estimate the co-occurrence of OP
residues in water. Monitoring data were not available with enough
consistency to be the sole basis for the assessment; however, they were
used to corroborate the modeling results. Data sources for the water
component of the assessment included USDA Agricultural Usage Reports for
Field Crops, Fruits, and Vegetables; USDA Typical Planting and
Harvesting Dates for Field Crops and Fresh Market and Processing
Vegetables; local sources for refinements; and monitoring studies from
the U.S. Geological Survey and other sources. Finally, exposure via the
oral, dermal, and inhalation routes resulting from applications of OPs
in and around homes, schools, offices, and other public areas were
assessed probabilistically for each of the seven regions. The data
sources for this part of the assessment included information from
surveys and task forces, special studies and reports from published
scientific literature, EPA’s Exposure Factors Handbook (USEPA, 1997),
and other sources.

Results of Analysis. The OPs CRA presented potential risk from
single-day (acute) exposures across a year and from a series of 21-day
rolling averages across the year. MOEs at the 99.9th percentile were
approximately 100 or greater for all populations for the 21-day average
results. The only exception is a brief period (roughly 2 weeks) in which
drinking water exposures resulting from phorate use on sugarcane result
in MOEs near 80 for children aged 1 to 2 years. Generally, exposures
through the food pathway dominated total MOEs, and exposures through
drinking water were substantially lower throughout most of the year.
Residential exposures were substantially smaller than exposures through
both food and drinking water.

The OPs CRA was very resource intensive. Work began in 1997 with the
preparation of guidance documents and the development of a CRA
methodology. Over 2 to 3 years, more than 25 people spent 50 to 100% of
their time working on the assessment, with up to 50 people working on
the CRA at critical periods. EPA has spent approximately $1 million on
this assessment (e.g., for computers, models, and contractor support).

Management Considerations.  The OP CRA was a landmark demonstration of
the feasibility of a regulatory level assessment of the risk from
multiple chemicals.  On its completion, EPA presented the CRA at
numerous public technical briefings and FIFRA SAP meetings, and made all
of the data inputs available to the public. OPP’s substantial effort
to communicate methodologies, approaches, and results to the
stakeholders aided in the success of the OPs CRA. The stakeholders
expressed appreciation for the transparent nature of the OPs CRA and
recognized the innovation and hard work that went into developing the
assessments.

Document Availability. The 2006 assessment and related documents are
available at   HYPERLINK
"http://www.epa.gov/pesticides/cumulative/common_mech_groups.htm#op" 
http://www.epa.gov/pesticides/cumulative/common_mech_groups.htm#op .

Contact.  David Miller, 703-305-5352;   HYPERLINK
"mailto:miller.davidj@epa.gov"  miller.davidj@epa.gov  .

Case Study 13: Probabilistic Ecological Effects Risk Assessment Models
for Evaluating Pesticide Uses

As part of the process of developing and implementing a probabilistic
approach for ecological risk assessment, an illustrative case was
completed in 1996. The illustrative case involved both deterministic and
probabilistic risk analysis for effects of a hypothetical chemical X on
birds and aquatic species. Following the recommendations of the SAP and
in response to issues raised by OPP risk managers, the Agency began an
initiative to refine the ecological risk assessment process for
evaluating the effects of pesticides to terrestrial and aquatic species
within the context of FIFRA, the main statutory authority for regulating
pesticides at the Federal level. Among the key goals and objectives of
EPA’s initiative were to:

incorporate probabilistic tools and methods to provide an estimate on
the magnitude and probability of effects;

build on existing data requirements for registration;

utilize, wherever possible, existing databases and create new ones from
existing data sources to minimize the need to generate additional data;
and

focus additional data requirements on reducing uncertainty in key areas.

After proposing a four-level risk assessment scheme, with higher levels
reflecting more refined risk assessment techniques, the Agency developed
pilot models for both terrestrial and aquatic species. Refined risk
assessment models (Level II) were then developed and used in a generic
chemical case study that was presented to the SAP in 2001.

Probabilistic Analysis. This case study describes an advanced
probabilistic model for ecological effects of pesticides (Group 3). The
terrestrial Level II model (version 2.0) is a multimedia
exposure/effects model that evaluates acute mortality levels in generic
or specific avian species over a user-defined exposure window. The
spatial scale is at the field level, which includes the field and
surrounding area. Both areas are assumed to meet the habitat
requirements for each species, and contamination of edge or adjacent
habitat from drift is assumed to be zero. For each individual bird
considered in a run of the Level II model, a random selection of values
is made for the major exposure input parameters to estimate an external
oral dose equivalent for that individual. The estimated dose equivalent
is compared to a randomly assigned tolerance for the individual
preselected from the dose/response distribution. If the dose is greater
than the tolerance, the individual is scored “dead,” if not, the
individual is scored “not dead.” After multiple iterations of
individuals, a probability density function of percent mortality is
generated.

May 29-31, 1996, the Agency presented two ecological risk assessment
case studies to SAP for review and comment. Although recognizing and
generally reaffirming the utility of EPA’s current deterministic
assessment process, SAP offered a number of suggestions for improvement.
Foremost among their suggestions was a recommendation to move beyond the
existing deterministic assessment approach by developing the tools and
methodologies necessary to conduct a probabilistic assessment of
effects. Such an assessment would estimate the magnitude and probability
of the expected impact and define the level of certainty and variation
involved in the estimate, information that risk managers within EPA’s
OPP also had requested in the past.

The aquatic Level II model is a two-dimensional Monte Carlo risk model
consisting of three main components: (1) exposure, (2) effects, and (3)
risk. The exposure scenarios used at Level II are intended to provide
estimates for vulnerable aquatic habitats across a wide range of
geographical conditions under which a pesticide product is used. The
Level II risk evaluation process yields estimates of likelihood and
magnitude of effects for acute exposures. For the estimate of acute
risks, a distribution of estimated exposure and a distribution of lethal
effects are combined through a two-dimensional Monte Carlo analysis to
obtain a distribution of joint probability functions. For the estimate
of chronic risks, a distribution of exposure concentrations is compared
to a chronic measurement endpoint. The risk analysis for chronic effects
provides information only on the probability that the chronic end point
assessed will be exceeded, not on the magnitude of the chronic effect
expected.

Results of Analysis. As part of the process of developing and
implementing a probabilistic approach for ecological risk assessment, a
case study was completed. The case study involved both deterministic and
probabilistic risk analyses for effects of ChemX on birds and aquatic
species. The deterministic screen conducted on ChemX concluded
qualitatively that it could pose a high risk to both freshwater fish and
invertebrates and showed that PRA was warranted. Based on the
probabilistic analysis, it was concluded that the use of ChemX was
expected to infrequently result in significant freshwater fish
mortalities but routinely result in reduced growth and other chronic
effects in exposed fish. Substantial mortalities and chronic effects to
sensitive aquatic invertebrates were predicted to routinely occur after
peak exposures.

Management Considerations.  In its review of the case study, the FIFRA
SAP congratulated the Agency on the effort made to conduct PRA of
pesticide effects in ecosystems. The panel commented that the approach
had progressed greatly from earlier efforts, and that the intricacy of
the models was surprisingly good, given the time interval in which the
Agency had to complete the task.  Following the case study, the Agency
refined the models based on the SAP comments. In addition, the
terrestrial Level II model was refined to include dermal and inhalation
exposure.

Document Availability. An overview of the models is available at

  HYPERLINK
"http://www.epa.gov/oppefed1/ecorisk/fifrasap/rra_exec_sum.htm#Terrestri
al" 
http://www.epa.gov/oppefed1/ecorisk/fifrasap/rra_exec_sum.htm#Terrestria
l .

Contact: Donna Randall, 703-605-1298,   HYPERLINK
"mailto:randall.donna@epa.gov"  randall.donna@epa.gov 

Case Study 14: Expert Elicitation of Concentration-Response
Relationship Between Particulate Matter (PM2.5) Exposure and Mortality

In 2002, the NRC recommended that EPA improve its characterization of
uncertainty in the benefits assessment for proposed regulations of air
pollutants. NRC recommended that probability distributions for key
sources of uncertainty be developed using available empirical data or
through formal elicitation of expert judgments.  A key component of
EPA’s approach for assessing potential health benefits associated with
air quality regulations targeting emissions of PM2.5 and its precursors
is the effect of changes in ambient PM2.5 levels on mortality. Avoided
premature deaths constitute, on a dollars basis, between 85 and 95% of
the monetized benefits reported in EPA’s retrospective and prospective
Section 812A benefit-cost analyses of the Clean Air Act (EPA, 1997 and
1999) and in Regulatory Impact Analysis (RIAs) for rules such as the
Heavy Duty Diesel Engine/Fuel Rule (EPA, 2000) and the Non-road Diesel
Engine Rule (EPA, 2004). In response to the National Research Council
(NRC) recommendation, EPA conducted an expert elicitation evaluation of
the concentration-response relationship between PM2.5 exposure and
mortality.

Probabilistic Risk Analysis. This case study provides an example of the
use of expert elicitation (Group 3) to derive probabilistic estimates of
the uncertainty in one element of a cost-benefit analysis. Expert
elicitation uses carefully structured interviews to elicit from each
expert a best estimate of the true value for an outcome or variable of
interest, as well as their uncertainty about the true value. This
uncertainty is expressed as a subjective probabilistic distribution of
values and reflects each expert’s interpretation of theory and
empirical evidence from relevant disciplines, as well as their beliefs
about what is known and not known about the subject of the study. For
the PM2.5 expert elicitation, the process consisted of development of an
elicitation protocol, selection of experts, development of a briefing
book, conducting elicitation interviews, the use of expert workshops
prior to and following individual elicitation of judgments, and the
expert judgments themselves. The elicitation involved personal
interviews with 12 health experts who have conducted research on the
relationship between PM2.5 exposures and mortality.

The main quantitative question asked each expert to provide a
probabilistic distribution for the average expected decrease in U.S.
annual, adult, and all-cause mortality associated with a 1-μg/m3
decrease in annual average PM2.5 levels. When answering the main
quantitative question, each expert was instructed to consider that the
total mortality effect of a 1-μg/m3 decrease in ambient annual average
PM2.5 may reflect reductions in both short-term peak and long-term
average exposures to PM2.5. Each expert was asked to aggregate the
effects of both types of changes in their answers.

The experts were given the option to integrate their judgments about the
likelihood of a causal relationship or threshold in the
concentration-response function into their own distributions or to
provide a distribution “conditional on” one or both of these
factors.

Results of Analysis. The project team developed the interview protocol
between October 2004 and January 2006. Development of the protocol was
informed by an April 2005 symposium held by the project team, where
numerous health scientists and analysts provided feedback; by detailed
pretesting with independent EPA scientists in November 2005; and by
discussion with the participating experts at a pre-elicitation workshop
in January 2006. The elicitation interviews were conducted between
January and April 2006. Following the interviews, the experts reconvened
for a post-elicitation workshop in June 2006, in which the project team
anonymously shared the results of all experts with the group.

The median estimates for the PM2.5 mortality relationship were generally
similar to estimates derived from the two epidemiological studies most
often used in benefits assessment. However, in almost all cases, the
spread of the uncertainty distributions elicited from the experts
exceeded the statistical uncertainty bounds reported by the most
influential epidemiologic studies, suggesting that the expert
elicitation process was successful in developing more comprehensive
estimates of uncertainty for the PM2.5 mortality relationship. The
uncertainty distributions for PM2.5 concentration-response resulting
from the expert elicitation process were used in the RIA for the revised
NAAQS standard for PM2.5 (promulgated on September 21, 2006). Because
the NAAQS are exclusively health based standards, this RIA played no
part in EPA’s determination to revise the Pm2.5 NAAQS.  Benefits
estimates in the RIA were presented as ranges and included additional
information on the quantified uncertainty distributions surrounding the
points on the ranges, derived from both epidemiological studies and the
expert elicitation results. OMB’s review of the RIA was completed in
March 2007.

Management Considerations.  The NAAQS are exclusively health-based
standards, so these analyses were not used in any manner by EPA in
determining whether to revise the NAAQS for PM2.5.  Moreover, the
request to engage in the expert elicitation did not come from the Clean
Air Scientific Advisory Committee, or CASAC, the official peer review
body for the NAAQS, so that a decision to conduct the analyses does not
reflect CASAC advice that such analyses inform NAAQS determinations. 
The analyses addressed comments from the National Research Council that
recommended that probability distributions for key sources of
uncertainty be addressed. The analyses were used in EPA’s
retrospective and prospective Section 812A benefit-cost analyses of the
Clean Air Act (EPA, 1997 and 1999) and in RIAs for rules such as the
Heavy Duty Diesel Engine/Fuel Rule (EPA, 2000) and the Non-road Diesel
Engine Rule (EPA, 2004). In response to the NRC recommendation, EPA
conducted an expert elicitation evaluation of the concentration-response
relationship between PM2.5 exposure and mortality.  

Document Availability. The assessment is available at   HYPERLINK
"http://www.epa.gov/ttn/ecas/ria.html" 
http://www.epa.gov/ttn/ecas/ria.html .

Contact: Lisa Conner, 919-541-5060,   HYPERLINK
"mailto:conner.lisa@epa.gov"  conner.lisa@epa.gov 

Case Study 15: Expert Elicitation of Sea-Level Rise Resulting from
Global Climate Change

The United Nations Framework Convention on Climate Change requires
nations to implement measures for adapting to rising sea level and other
effects of changing climate. To decide on an appropriate response,
coastal planners and engineers weigh the cost of these measures against
the likely cost of failing to prepare, which depends on the probability
of the sea rising a particular amount. The U.S. National Academy of
Engineering recommended that assessments of sea level rise should
provide probability estimates. Coastal engineers regularly employ
probability information when designing structures for floods, and courts
use probabilities to determine the value of land expropriated by
regulations. This case study describes the development of a probability
distribution for sea level rise, using models employed by previous
assessments, as well as the expert opinions of 20 climate and glaciology
reviewers about the probability distributions for particular model
coefficients.

Probabilistic Analysis. This case study provides an example both of an
analysis describing the probability of sea level rise, as well as an
expert elicitation of the likelihood of particular models and
probability distributions of the coefficients used by those models to
predict future sea level rise (Group 3). The assessment of the
probability of sea level rise used existing models describing the change
in five components of sea level rise associated with
greenhouse-gas-related climate change (thermal expansion, small
glaciers, polar precipitation, melting and ice discharge from Greenland,
ice discharge from Antarctica). To provide a starting point for the
expert elicitation, initial probability distributions were assigned to
each model coefficient based on the published literature.

Once the initial probabilistic assessment was completed, the draft
report was circulated to expert reviewers considered most qualified to
render judgments on particular processes for revised estimates of the
likelihood of particular models and the model coefficients’
probability distributions. Experts representing both extremes of climate
change science (those who predicted trivial consequences and those who
predicted catastrophic effects; those whose thinking had been excluded
from previous assessments) were included. The experts were asked to
provide subjective assessments of the probabilities of various models
and model coefficients. These subjective probability estimates were used
in place of the initial probabilities in the final model simulations.
Different reviewer opinions were not combined to produce a single
probability distribution for each parameter, but, rather, each
reviewer’s opinions were used in independent iterations of the
simulation. The group of simulations resulted in the probability
distribution of sea level rise.

Results of Analysis. The analysis, completed with a budget of $100,000,
provided a probabilistic estimate of sea level rise for use by coastal
engineers and regulators. The results suggested that there is a 65%
chance that sea level will rise 1 mm/year more rapidly in the next 30
years than it has been rising in the last century. Under the assumption
that nonclimatic factors do not change, the projections suggested that
there is a 50% chance that global sea level will rise 45 cm, and a 1%
chance of a 112-cm rise by the year 2100. The median prediction of sea
level rise was similar to the midpoint estimate of 48 cm published by
the Intergovernmental Panel on Climate Change (IPCC, 2006) shortly
thereafter.. The report also found a 1% chance of a 4-5 meter rise over
the next two centuries.  

Management Considerations:  Both reports (EPA 1995; Titus and Narayanan
1996) discuss several uses of the results of this study.  By providing a
probabilistic representation of sea level rise, the assessment allows
coastal residents to make decisions with recognition of the uncertainty
associated with predicted change.   Rolling easements that vest when the
sea rises to a particular level can be properly valued in both
arms-length transaction sales or when calculating the allowable
deduction for a charitable contribution of the easement to a
conservancy.   Cost-benefit assessments of alternative infrastructure
designs—which already consider flood probabilities—can also consider
sea level rise uncertainty.  Assessments of the benefits of preventing
an acceleration of sea level rise can more readily include
low-probability outcomes, which can provide a better assessment of the
true risk when the damage function is nonlinear, which is often the
case.  

Document Availability. 

 EPA 1995.   The Probability of Sea Level Rise.    Washington, D.C.: 
Climate Change Division.

http://epa.gov/climatechange/effects/coastal/slrmaps_probability.html

IPCC (1996).   Climate Change 1995: The Science of Climate Change. 
Contribution of Working Group I to the Second Assessment of the
Intergovernmental Panel on Climate Change.

  HYPERLINK "http://uk.cambridge.org/"  Cambridge University Press , 
Cambridge CB2 2RU ENGLAND.

 

Titus, J. G. and V Narayanan.  1996.    HYPERLINK
"http://epa.gov/climatechange/effects/coastal/Risk_of_rise.html"  The
Risk of Sea Level Rise: A Delphic Monte Carlo Analysis in which Twenty
Researchers Specify Subjective Probability Distributions for Model
Coefficients within their Respective Areas of Expertise  - Climatic
Change, 33: 151-212 (1996).   
http://epa.gov/climatechange/effects/coastal/Risk_of_rise.html

Contact:  James G. Titus, 202-343-9307;   HYPERLINK
"mailto:jtitus@erols.com"  titus.jim@epa.gov 

Case Study 16: Expert Elicitation for Bayesian Belief Network Model of
Stream Ecology

The identification of the causal pathways leading to stream impairment
is a central challenge to  understanding ecological relationships.
Bayesian belief networks (BBNs) are a promising tool for modeling
presumed causal relationships, providing a modeling structure within
which different factors describing the ecosystem can be causally linked,
and uncertainties expressed for each linkage.

BBNs can be used to model complex systems that involve several
interdependent or interrelated variables. In general, a BBN is a model
that evaluates situations where some information is already known, and
incoming data are uncertain or partially unavailable. The information is
depicted with influence diagrams that present a simple visual
representation of a decision problem, for which quantitative estimates
of effect probabilities are assigned. As such, BBNs have the potential
for representing ecological knowledge and uncertainty in a format that
is useful for predicting outcomes from management actions or for
diagnosing the causes of observed conditions. Generally, specification
of a BBN can be performed using available experimental data, literature
review information (secondary data), and expert elicitation. Attempts to
specify a BBN for the linkage between fine sediment load and
macroinvertebrate populations using data from literature reviews have
failed because of the absence of consistent conceptual models and lack
of quantitative data or summary statistics needed for the model. In
light of these deficiencies, an effort was made to use expert
elicitation to specify a BBN for the relationship between fine sediment
load resulting from human activity and populations of
macroinvertebrates. The goals of this effort were to examine whether
BBNs might be useful for modeling stream impairment and to assess
whether expert opinion could be elicited to make the BBN approach useful
as a management tool.

Probabilistic Risk Analysis. This case study provides an example of
expert elicitation in the development of a BBN model of the effect of
increased fine sediment load in a stream on macroinvertebrate
populations (Group 3). For the purpose of this study, a stream setting
(a Midwestern, low-gradient stream) and a scenario of impairment
(introduction of excess fine sediment) were used. Five stream ecologists
with experience in the specified geographic setting were invited to
participate in an elicitation workshop. An initial model was depicted
using influence diagrams, with the goals of structuring and specifying
the model using expert elicitation. The ecologists were guided through a
knowledge elicitation session in which they defined factors that
described relevant chemical, physical, and biological aspects of the
ecosystem. The ecologists then described how these factors were
connected and were asked to provide subjective, quantitative estimates
of how different attributes of the macroinvertebrate assemblage would
change in response to increased levels of fine sediment. Elicited input
was used to restructure the model diagram and to develop probabilistic
estimates of the relationships among the variables.

Results of Analysis. The elicited input was compiled and presented in
terms of the model as structured by the stream ecologists and their
model specifications. The results were presented both as revised
influence diagrams and with Bayesian probabilistic terms representing
the elicited input. The study yielded several important lessons. Among
these were the elicitation process takes time (including an initial
session by teleconference as well as a 3-day workshop), defining a
scenario with an appropriate degree of detail is critical, and
elicitation of complex ecological relationships is feasible.

Management Considerations. The study was considered successful for
several reasons. First, the feasibility of the elicitation approach to
building stream ecosystem models was demonstrated. The study also
resulted in the development of new graphical techniques to perform the
elicitation. The elicited input was interpreted in terms of predictive
distributions to support fitting a complete Bayesian model. Further, the
study was successful in bringing together a group of experts in a
particular subject area for the purpose of sharing information and
learning about expert elicitation in support of model building. The
exercise provided insights into how best to adapt knowledge elicitation
methods to ecological questions and informed the assembled stream
ecologists on the elicitation process and on the potential benefits of
this modeling approach. The explicit quantification of uncertainty in
the model not only enhances the utility of the model predictions but
also can help guide future research.

Document Availability. Black, P and Stockton, T. 2005. Using Knowledge
Elicitation to Inform a Bayesian Belief Network Model of a Stream
Ecosystem. Neptune and Company, Inc. July.

Yuan, L. TI: A Bayesian Approach for Combining Data Sets to Improve
Estimates of Taxon Optima, AGU, 86(18), Jt. Assem. Suppl., Abstract #
NB41E-04, 2005.

Contact:  Lester Yuan, 703-347-8534,   HYPERLINK
"mailto:yuan.lester@epa.gov"  yuan.lester@epa.gov 



4. References to Case Studies

Jamieson, D. (1996). Scientific uncertainty and the political process.
Ann. Am. Acad. Pol. Soc. Sci., 545:35-43.

Kurowicka, D. and Cooke, R. (2006). “Uncertainty Analysis With High
Dimensional Dependent Modeling.” Wiley Series in Probability and
Statistics, May, 2006.

Stahl, C.H., Cimorelli, A.J. (2005). How much uncertainty is too much
and how do we know? A case example of the assessment of ozone monitor
network options. Risk Anal, 25:1109-1120.

U. S. Environmental Protection Agency (EPA). (1992a). Guidelines for
exposure assessment. EPA 600Z-92/001. Risk Assessment Forum, Washington,
DC, 170 pp. (  HYPERLINK
"http://cfpub.epa.gov/ncea/raf/recordisplay.cfm?deid=559070" 
http://cfpub.epa.gov/ncea/raf/recordisplay.cfm?deid=559070 ).

U. S. Environmental Protection Agency (EPA). (1992b). Framework for
ecological risk assessment. EPA/630/R-92/001. Washington, DC.

U. S. Environmental Protection Agency (EPA). (1995a). Policy for risk
characterization. Science Policy Council, Washington, DC (  HYPERLINK
"http://www.epa.gov/osp/spc/rcpolicy.htm" 
http://www.epa.gov/osp/spc/rcpolicy.htm ).

U. S. Environmental Protection Agency (EPA). (1995b). Policy on
evaluating health risks to children. Science Policy Council, Washington,
DC (  HYPERLINK "http://www.epa.gov/osp/spc/memohlth.htm" 
http://www.epa.gov/osp/spc/memohlth.htm ).

U. S. Environmental Protection Agency (EPA). (1997a). Policy for use of
probabilistic analysis in risk assessment at the U.S. Environmental
Protection Agency. Fred Hansen, Deputy Administrator. Science Policy
Council, Washington, DC (  HYPERLINK
"http://www.epa.gov/osp/spc/probpol.htm" 
http://www.epa.gov/osp/spc/probpol.htm ).

w

x

y

z

.

/

8

9

L

q

x

y

z

/

8

9

ꐔx␷㠀$⑈愀Ĥ摧塉		̀Ĥ␱愀Ĥ摧塉	

j¸

j;

j¬

j/

j



㜀$␸䠀$

h

㜀$␸䠀$

਀&䘋

h

ЀS摧৻G

ž

Ÿ

hþv

 h*@

摧৻G

h*@

h0

 h0

h0

hþv

B*

h

  h

h

h

–

—

™

«

a

y

˜

™

␃愁Ĥ摧⶙ÿᔀ™

«

¬

摧⶙ÿᘀ«

¬

±

º

 hÿ

	㜀$␸䠀$摧⶙ÿ

x



‚

OJ

QJ

摧⶙ÿ	㜀$␸䠀$摧⶙ÿ

^J

01. Risk Assessment Forum, Office of Research and Development,
Washington, DC.

U. S. Environmental Protection Agency (EPA). (1997c). Guidance on
cumulative risk assessment. Part 1: Planning and scoping. Science Policy
Council, Washington, DC (  HYPERLINK
"http://www.epa.gpv/osp/spc/cumrisk2.htm" 
http://www.epa.gpv/osp/spc/cumrisk2.htm ).

U. S. Environmental Protection Agency (EPA). (1998). Guidelines for
ecological risk assessment. EPA/630/R-95/002F. Risk Assessment Forum,
Washington, DC, 171 pp. (  HYPERLINK
"http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=12460" 
http://cfpub.epa.gov/ncea/cfm/recordisplay.cfm?deid=12460 ).

U. S. Environmental Protection Agency (EPA). (2000). Science Policy
Council Handbook: Risk Characterization Handbook. EPA 100-B00-002.
Science Policy Council, Washington, DC, December (  HYPERLINK
"http://www.epa.gov/osp/spc/rchandbk.pdf" 
http://www.epa.gov/osp/spc/rchandbk.pdf ).

U. S. Environmental Protection Agency (EPA). (2001). Risk assessment
guidance for Superfund: Volume III—Part A, Process for conducting
probabilistic risk assessment. EPA 540-R-02-002. Office of Emergency and
Remedial Response, Washington, DC, December (  HYPERLINK
"http://www.epa.gov/superfund/programs/risk/rags3a/index.htm" 
http://www.epa.gov/superfund/programs/risk/rags3a/index.htm ).

U. S. Environmental Protection Agency (EPA). (2004). An Examination of
EPA Risk Assessment Principles and Practices. U.S. EPA, Office of the
Science Advisor, Office of Research and Development, Washington, DC.
EPA/100/B-04/001 March 2004 (  HYPERLINK
"http://www.epa.gov/OSA/ratf.htm"  http://www.epa.gov/OSA/ratf.htm ).



List of Acronyms and Abbreviations

APEX		Air Pollutants Exposure Model

BBN		Bayesian belief network

CASAC	Clean Air Scientific Advisory Committee

CCA		chromated copper arsenate

CRA		cumulative risk assessment

CSFII		Continuing Survey of Food Intake by Individuals

DEEM		Dietary Exposure Evaluation Model

DRES		Dietary Risk Evaluation System

Eco		ecological risk assessment

EMAP		Environmental Monitoring and Assessment Program

EPA		U.S. Environmental Protection Agency

FACA		Federal Advisory Committee Act

FDA		Food and Drug Administration

FIFRA		Federal Insecticide, Fungicide, and Rodenticide Act

FQPA		Food Quality Protection Act

HH		human health

LT		long-term

MOEs		margins of exposure

NAAQS	National Ambient Air Quality Standards

NRC		National Research Council

OAQPS	Office of Air Quality Planning and Standards

OAR		Office of Air and Radiation

OGWDW	Office of Groundwater and Drinking Water

OMB		Office of Management and Budget

OP		organophosphorous pesticide

ORD		Office of Risk Analysis

OW		Office of Water

PCB		polychlorinated biphenyl

PDP		Pesticide Data Program

PM		particulate matter

PRA		probabilistic risk analysis

RIA		Regulatory Impact Analysis

SAB		Science Advisory Board

SAP		Scientific Advisory Panel

SHEDS	Stochastic Human Exposure and Dose Simulation

TRIM		Total Risk Integrated Methodology

TRIM.Expo	Total Risk Integrated Methology/Exposure Model

UI		Uncertainty Interval

USDA		U.S. Department of Agriculture

EXTERNAL REVIEW DRAFT – DO NOT CITE OR QUOTE

 PAGE   

  PAGE  i 

DRAFT INTERNAL TO EPA RISK ASSESSMENT FORUM DO NOT CITE OR QUOTE v 
SAVEDATE  \@ "M/d/yyyy h:mm:ss am/pm"  \* MERGEFORMAT  6/18/2009 9:24:00
AM 

 PAGE   

 PAGE   37 

DRAFT INTERNAL TO EPA DO NOT CITE OR QUOTE

Levels of Analyses

Sensitivity analysis

Monte Carlo analysis of variability in

Exposure data

Human health or ecological effect data

Monte Carlo analysis of uncertainty 

“Cumulative” PRA(multi-pathway or multi-chemical

Two-dimensional PRA of uncertainty and variability

Decision uncertainty analysis

Geospatial analysis

Expert elicitation

Central Tendency

 Value

RME Value

 

Range of Non-Cancer Hazard Index (HI) Estimates for Fish Ingestion

Fraction of Anglers with 

HI (  Indicated Value

Central Tendency

 Value

RME 

Value

 

Range of Cancer Risk Estimates 

for Fish Ingestion

Fraction of Anglers with 

Risk ( than Indicated Value

