         U.S. EPA FEDERAL INSECTICIDE, FUNGICIDE, AND RODENTICIDE ACT 

                     SCIENTIFIC ADVISORY PANEL (FIFRA SAP)
                                PUBLIC MEETING
                                       
                                       
PEER REVIEW FOR THE APPROACHES FOR QUANTITATIVE USE OF SURFACE WATER MONITORING DATA IN PESTICIDE DRINKING WATER ASSESSMENTS
                                       
                  DOCKET NUMBER: EPA - HQ - OPP - 2019 - 0417
             TSCA SACC WEBSITE http://www.epa.gov/tsca-peer-review
                                       
                                       
                                       
                       HOLIDAY INN ROSSLYN AT KEY BRIDGE
                             1900 FORT MYER DRIVE
                              ARLINGTON, VA 22209
                             November 19-21, 2019

FIFRA SAP Chair


ROBERT E. CHAPIN, PH.D.
FORMER SENIOR RESEARCH FELLOW (RETIRED) 
PFIZER GLOBAL RESEARCH AND DEVELOPMENT 
GROTON, CONNECTICUT
DESIGNATED FEDERAL OFFICIAL

TAMUE L. GIBSON, M.S.
BIOLOGIST AND DESIGNATED FEDERAL OFFICIAL FIFRA SCIENTIFIC ADVISORY PANEL 
OFFICE OF SCIENCE COORDINATION AND POLICY 
U.S. ENVIRONMENTAL PROTECTION AGENCY
FIFRA SCIENTIFIC ADVISORY PANEL MEMBERS

GEORGE B. CORCORAN, PH.D.
CHAIRMAN AND PROFESSOR
DEPARTMENT OF PHARMACEUTICAL SCIENCES 
WAYNE STATE UNIVERSITY
SONYA K. SOBRIAN, PH.D.
DEPARTMENT OF PHARMACOLOGY
HOWARD UNIVERSITY COLLEGE OF MEDICINE
CLIFFORD P. WEISEL, PH.D.
PROFESSOR 
RUTGERS UNIVERSITY
RAYMOND S.H. YANG, PH.D.
PROFESSOR (EMERITUS) 
COLLEGE OF VETERINARY MEDICINE AND BIOMEDICAL SCIENCES 
COLORADO STATE UNIVERSITY
FQPA SCIENCE REVIEW BOARD MEMBERS

CLAIRE BAFFAUT, PH.D.
RESEARCH HYDROLOGIST 
UNITED STATES DEPARTMENT OF AGRICULTURE 
AGRICULTURAL RESEARCH SERVICE 
CROPPING SYSTEM AND WATER QUALITY RESEARCH UNIT
VERONICA BERROCAL, PH.D.
ASSOCIATE PROFESSOR 
DEPARTMENT OF STATISTICS
UNIVERSITY OF CALIFORNIA
TERRY COUNCELL, M.S.
COORDINATOR
TOTAL DIET STUDY
UNITED STATES FOOD AND DRUG ADMINISTRATION
TIMOTHY GREEN, PH.D.
RESEARCH AGRICULTURAL ENGINEER/HYDROLOGIST 
UNITED STATES DEPARTMENT OF AGRICULTURE 
AGRICULTURAL RESEARCH SERVICE 
CENTER FOR AGRICULTURAL RESOURCES RESEARCH
IAN KENNEDY, PH.D.
SENIOR EVALUATION OFFICER
HEALTH CANADA
PEST MANAGEMENT REGULATORY AGENCY
REBECCA KLAPER, PH.D.
PROFESSOR 
SCHOOL OF FRESHWATER SCIENCES
UNIVERSITY OF WISCONSIN-MILWAUKEE
ANDREW MIGLINO, PH.D.
PHYSICAL SCIENTIST
UNITED STATES FOOD AND DRUG ADMINISTRATION 
CENTER FOR VETERINARY DRUGS
LISA NOWELL, PH.D.
RESEARCH CHEMIST
UNITED STATES GEOLOGICAL SURVEY
THOMAS POTTER, PH.D.
PRESIDENT AND PRINCIPAL SCIENTIST 
INNOVATIVE SYSTEMS FOR EDUCATION AND ENVIRONMENTAL TECHNOLOGY
KENNETH PORTIER, PH.D. (RETIRED)
(FORMERLY OF THE NATIONAL CANCER SOCIETY)
CONSULTING BIOSTATISTICIAN
JOHN RODGERS, JR., PH.D.
PROFESSOR OF AQUATIC TOXICOLOGY AND ECOTOXICOLOGY
CLEMSON UNIVERSITY
JAMES SADD, PH.D.
PROFESSOR OF ENVIRONMENTAL SCIENCE
DEPARTMENT OF GEOLOGY AND ENVIRONMENTAL SCIENCE
OCCIDENTAL COLLEGE
XUYANG ZHANG, PH.D.
SENIOR ENVIRONMENTAL SCIENTIST (SPECIALIST)
CALIFORNIA DEPARTMENT OF PESTICIDE REGULATION
ENVIRONMENTAL MONITORING BRANCH




PRESENTERS

MARIETTA ECHEVERRIA, M.S.
DIRECTOR
OCSPP/OPP/EFED

DANA SPATZ, M.S.
BRANCH CHIEF, OCSPP/OPP/EFED
ROCHELLE BOHATY, PH.D.
SENIOR CHEMIST 
OCSPP/OPP/EFED/ERB3
SARAH HAFNER, PH.D.
CHEMIST, 
OCSPP/OPP/EFED
ALDO VECCHIA, PH.D.,
STATISTICIAN EMERITUS (RETIRED)
U.S. GEOLOGICAL SURVEY
MATTHEW BISCHOF, M.S.
NATURAL RESOURCE SCIENTIST 

            WASHINGTON STATE DEPARTMENT OF AGRICULTURE 

CHRISTINE HARTLESS, PH.D.
BIOLOGIST, OCSPP/OPP/EFED
CHARLES PECK, M.S.,
SENIOR ENVIRONMENTAL 
ENGINEER, OCSPP/OPP/EFED
JAMES HOOK, M.S.
ECOLOGIST, OCSPP/OPP/EFED
KATRINA WHITE, PH.D.
SENIOR BIOLOGIST, OCSPP/OPP/EFED
JESSICA JOYCE, M.S.,
PHYSICAL SCIENTIST, OCSPP/OPP/EFED
PUBLIC COMMENTERS

DANIEL PERKINS, PH.D
WATERBORNE ENVIRONMENTAL, INC.
PAUL MOSQUIN, PH.D
RTI INTERNATIONAL
JEREMEY ALDWORTH, PH.D.
RTI INTERNATIONAL
MANOJIT BASU, PH.D.
CROPLIFE AMERICA

                                       

TABLE OF CONTENTS
OPENING OF MEETING	6
INTRODUCTION OF PANEL MEMBERS	11
INTRODUCTION AND WELCOME	18
OPP TECHNICAL PRESENTATION  -  OVERVIEW OF THE APPROACHES FOR QUANTITATIVE USE OF SURFACE WATER MONITORING DATA IN DRINKING WATER ASSESSMENTS	23
FRAMEWORK FOR CONDUCTING PESTICIDE DRINKING WATER ASSESSMENTS FOR SURFACE WATER	37
ANALYZING AND INTERPRETING SURFACE WATER PESTICIDE MONITORING DATA	57
EVALUATION OF SEAWAVE-QEX AS AN IMPUTATION TECHNIQUE FOR ESTIMATING DAILY PESTICIDE CONCENTRATIONS FROM PESTICIDE MONITORING DATA PART 1	70
EVALUATION OF SEAWAVE-QEX AS AN IMPUTATION TECHNIQUE FOR ESTIMATING DAILY PESTICIDE CONCENTRATIONS FROM PESTICIDE MONITORING DATA PART 2	113
DEVELOPMENT AND EVALUATION OF A SAMPLING BIAS FACTOR PROGRAM PART 1: SHORT-TERM	142
DEVELOPMENT AND EVALUATION OF A SAMPLING BIAS FACTOR PROGRAM PART 2: LONG-TERM	186
WATERSHED EXTRAPOLATION AND WEIGHT-OF-EVIDENCE APPROACH	198
DRINKING WATER ASSESSMENT CASE STUDY   -   A PESTICIDE WITH SHORT-TERM EXPOSURE CONSIDERATION	220
DRINKING WATER ASSESSMENT CASE STUDY  -  A PESTICIDE WITH LONG-TERM EXPOSURE CONSIDERATIONS	238
DEVELOPMENT OF SURFACE WATER MONITORING PROGRAM	263
SUMMARY AND WRAP-UP	296
PUBLIC COMMENTS	299
OPENING OF MEETING - DAY 2	322
PREVIOUS DAY FOLLOW UP	322
CHARGE QUESTION 1	325
CHARGE QUESTION 1(a)	327
CHARGE QUESTION 1(b)	395
CHARGE QUESTION 1(c)	422
CHARGE QUESTION 1(d)	450
CHARGE QUESTION 1(e)	468
CHARGE QUESTION 1(f)	484
CHARGE QUESTION 2	504
CHARGE QUESTION 2 (a)	505
CHARGE QUESTION 2(b)	529
CHARGE QUESTION 2(c)	534
CHARGE QUESTION 2(d)	547
CHARGE QUESTION 2(e)	549
CHARGE QUESTION 3	558
CHARGE QUESTION 3(a)	559
OPENING OF MEETING - DAY 3	589
PREVIOUS DAY FOLLOW-UP	589
CHARGE QUESTION 3(b)	607
CHARGE QUESTION 4	624
CHARGE QUESTION 4(a)	625
CHARGE QUESTION 4(b) - CASE STUDY 1	653
CHARGE QUESTION 4(c) - CASE STUDY 2	671




                              OPENING OF MEETING 
                  
                  MS. TAMUE GIBSON:  Good morning and thank you everyone.  I am Tamue Gibson and I'll be serving as the designated federal official for the Federal Insecticide and Rodenticide Act Scientific Advisory Panel for this meeting.  
                  I want to thank our chair, Dr. Chapin, for agreeing to serve as the chair for FIFRA SAP meeting.  I also want to thank both the members of the panel and the public for attending this important meeting.  We appreciate your time and effort of the panel members in preparing for this meeting and taking account of your busy schedule.
                  In addition, I want to thank the EPA's Office of Pesticide Programs and my colleagues and staff at EPA for their very hard work in preparing for this important review of the approaches for quantitative use of surface water monitoring data and pesticide drinking water assessments.  You all should be commended in the amount of effort that you all had taken to prepare for this meeting.
                  By way of background, the FIFRA SAP is a federal advisory committee that provides independent scientific peer review and advice to the agency on pesticide and pesticide-related issues regarding the impact proposed regulatory actions on human health and the environment.  The FIFRA SAP only provides advice and recommendations to EPA.  Decision-making and implementation authority remains with the agency.
                  The FIFRA SAP consists of seven members which is called the permanent panel members.  The expertise of these members is augmented through the science review board which at ad hoc expert nominees that are considered and selected on an as-needed basis.  For the present meeting, five ad hoc -- excuse me.  The ad hoc expert peer reviewers have been added and we appreciate the contributions of these experts.  
                  As the DFO for this meeting, I serve as a liaison between the FIFRA SAP and the agency.  I am also responsible for insurant provisions of the Federal Advisory Committee Act are met.  The Federal Advisory Committee Act of 1972 established a system that governs the creation, operation, and termination of executive branch advisory committees.  FIFRA SAP meetings are subject to all FACCO requirements.  These include an open meeting, timely public notice of meetings, and document availability which is provided via the Office of Pesticide Programs public docket at www.regulations.gov.
                  As a designated federal official for this meeting, a critical responsibility is to work with appropriate agency officials to ensure that all appropriate ethics regulations are satisfied.  In that capacity, panel members receive training on provisions of the federal conflict interest laws.  In addition, each participant has filled a standard government financial disclosure report.  I, along with our deputy ethics official for the Office of Science Coordination and Policy and in consultation with the Office of General Counsel, have reviewed these reports to ensure that all ethics requirements are met.  A sample copy of this form is available on the FIFRA SAP website.  The address for the website is noted on the meeting agenda. 
                  The FIFRA SAP will review challenging scientific issues over the next few days.  We do have a full agenda and meeting times are approximate.  Thus, we may not keep to exact times as noted due to panel discussions and public comments.  We strive to ensure adequate time for agency presentations, public comments, and panel deliberations.  
                  For all presenters, panel members, and public commenters, I would like to ask that you please identify yourselves and speak directly into the microphone provided since this meeting is being webcasted, recorded, and transcribed.  Copies of all EPA presentation materials and written public comments are available in the public docket at regulations.gov.  Any copies of presentations given during this meeting by public commenters will be available in the public docket within the next week.
                  Members of the panel are encouraged to fully consider all written and all public comments submitted for this meeting.  For members of the public that have not preregistered for public comments, please notify either myself or another member of the FIFRA SAP staff if you're interested in making a comment.  
                  At this time, the agenda has been set; however, as we move through the proceedings, if time allows, we may be able to accommodate additional brief comments of five minutes or less.  
                  As I mentioned previously, there is a public docket for this meeting.  All background materials, questions posed by the panel, by the agency, and other documents related to this meeting are available in the public docket.  Some documents are also available on the EPA FIFRA SAP website.  Again, the docket number and website are noted on the meeting agenda.
                  For members of the press, EPA media relations staff are available to answer your questions about this meeting.  Please address all questions to Robert Daguillard, EPA media contact.  His number is 202-564-6618.  
                  At the conclusion of the meeting, the FIFRA SAP will prepare a report as a response to questions by the Agency, background materials, presentations and public comments.  The final report also serves as meeting minutes.  We anticipate the final report and meeting minutes will be completed in approximately 90 days after the meeting. 
                  Again, I wish to thank the panel for your avid participation, the program office for your diligent work, and my colleagues at the Office of Science Coordination and Policy and the public for your attendance.  I'm looking forward to both the Office of Pesticide Programs presentation and at this time, I turn the meeting over to our chair, Dr. Chapin.
                  DR. ROBERT CHAPIN:  I know his names' here someplace.  Thank you, Dr. Gibson.
                  All right.  So my name is Bob Chapin.  Welcome everyone.  Thank you for being here.  
                  So at this point, let's go around the room and get our -- introduce the panelists and our EPA colleagues and with one phrase of our expertise and background and what brings us to the table.
                  My name is Bob Chapin.  I do developmental and reproductive toxicology and I spent my career at the National Toxicology Program and at Pfizer.  
                  
INTRODUCTION OF PANEL MEMBERS
                  
                  DR. SONYA SOBRIAN:  Good morning.  I'm Sonya Sobrian.  I'm at the Howard University College of Medicine Department of Pharmacology.  My expertise is developmental neuropharmacology and toxicology.
                  DR. CLIFFORD WEISEL:  I'm Cliff Weisel from the Environmental and Occupational Health Science Institute at Rutgers University.  I work on exposure science. 
                  DR. RAYMOND YANG:  Good morning.  I'm Ray Yang, retired professor from Colorado State University in toxicology and cancer biology.  And my interest -- research interest is in toxicology of chemical mixture.  As a result of that, I'm interested in biologically based computer modeling.  Thank you.
                  DR. ANDREW MIGLINO:  Hello.  My name is Andrew Miglino.  I work at the U.S. Food and Drug Administration Center for Veterinary Medicine.  I do environmental fate modeling of pesticides and veterinary drugs.  
                  DR. REBECCA KLAPER:  My name is Rebecca Klaper.  I'm a professor at the School of Freshwater Sciences at University of Wisconsin-Milwaukee.  I study -- I'm an ecotoxicologist and I study the effects of emerging contaminants on freshwater organisms and also look at the presence and distribution of emerging contaminants in freshwater systems. 
                  DR. GEORGE B. CORCORAN:  Good morning.  My name is George Corcoran.  I'm a professor of pharmaceutical sciences in the Eugene Applebaum College of Pharmacy and Health Sciences of Wayne State University in Detroit.  My area of specialty is chemical and drug toxicities.  I'm a toxicologist and my organ of greatest interest is the liver, but other organs as well. 
                  DR. JOHN RODGERS JR.:  Good morning.  My name is John Rodgers.  I'm at Clemson University in South Carolina, professor of ecotoxicology.  And the area I'm interested in is fate and effects as well as the toxicology of materials as they come from our surfaces, our ag surfaces.
                  DR. LISA NOWELL:  Hello.  My name is Lisa Nowell.  I'm with the U.S. Geological Survey.  I'm a research chemist.  I've worked with the National Water Quality Assessment Program or project since the beginning of NAWQA actually.  And I've specialized in pesticides and other contaminants, monitoring and assessment, in particular implications for human health and ecological health. 
                  DR. TIMOTHY GREEN:  Hi.  I'm Tim Green.  I'm with the USDA Agricultural Research Service in Fort Collins, Colorado.  I'm a hydrologist working in special variability and modeling.
                  MR. CHARLES PECK:  Good morning.  My name is Chuck Peck.  I'm a senior fate scientist with the Office of Pesticide Programs, Environmental Fate and Effects Division.  I deal with aquatic modeling as well as spray drift and volatility.  
                  DR. CHRISTINE HARTLESS:  Good morning.  My name is Christine Hartless.  I'm a biologist in the Office of Pesticide Programs, the Environmental Fate and Effects Division.  I specialist in ecotoxicology and statistics with a smattering of working in lots of different areas of need within the Environmental Fate and Effects Division.  
                  MR. MATTHEW BISCHOF:  Good morning.  I'm Matthew Bischof with the Washington State Department of Agriculture.  My background is in aquatic ecology and I've been monitoring for pesticides in surface water in the state of Washington since 2013.  I've been involved with SEAWAVE-QEX and I'm also bringing to the table experience in the field -- field research as well.
                  DR. SARAH HAFNER:  My name is Sarah Hafner.  I'm at the EPA.  I'm a chemist in the Environmental Fate and Effects Division and I've been with the Agency since 2017.  
                  DR. ROCHELLE BOHATY:  Hi.  Good morning everyone.  I'm Rochelle Bohaty.  I'm a senior chemist in the Environmental Fate and Effects Division.  
                  MR. DANA SPATZ:  My name is Dana Spatz.  I'm a branch chief in the Environmental Fate and Effects Division.  I manage a group of scientists who conduct environmental fates assessments, ecological risk assessments, drinking water exposure assessments.  
                  MS. MARIETTA ECHEVERRIA:  Good morning.  I'm Marietta Echeverria.  I'm the director of the Environmental Fate and Effects Division of the Office of Pesticide Programs EPA.
                  DR. ANNA LOWIT:  Good morning.  My name is Anna Lowit.  By training, I'm a toxicologist, but for the last several years, I've had the honor of being the science advisor in the EPA's Office of Pesticide Programs.
                  MS. JESSICA JOYCE:  Good morning.  I'm Jessica Joyce.  I am with the Environmental Fate and Effects Division at EPA as a physical scientist.
                  DR. KATRINA WHITE:  Katrina White, fate scientist, Environmental Fate and Effects Division.
                  DR. THOMAS POTTER:  Can you hear me from here?  Good morning.  I'm Thomas Potter.  I'm currently an independent consultant.  My background and training is in environmental chemistry.  Prior to my work as a consultant, I served for 20 years as a research chemist with the Agricultural Research Service in Tifton, Georgia, the Southeast Watershed Laboratory where I led a series of programs looking at pesticide, environmental fate, monitoring, et cetera.  
                  DR. JAMES SADD:  Hello.  I'm James Sadd, Professor of Environmental Science at Occidental College in Los Angeles and a research associate at the University of Southern California Program for Environmental and Regional Equity.  My particular expertise is in GIS and other GS spatial approaches and spatial statistics in addressing questions of environmental justice.  
                  MR. TERRY COUNCELL:  I'm Terry Councell and in former agencies that I've worked for, I did a lot of water quality monitoring in community water systems for pesticides and worked as a hydrologist for the U.S. Geological Survey.
                  DR. IAN KENNEDY:  I'm Ian Kennedy.  I work for the Pest Management Regulatory Agency in Canada and I specialize in environmental fate and modeling of pesticides.
                  DR. CLAIRE BAFFAUT:  Good morning.  I'm Claire Baffaut.  I work for the USDA Agriculture Research Service in Columbia, Missouri.  I'm a hydrologist.  I do modeling and monitoring.  Thank you.  
                  DR. VERONICA BERROCAL:  Hello.  My name is Veronica Berrocal.  I'm an associate professor in the Department of Statistics at University of California Irvine.  My expertise is in spatial statistics and statistical modeling of environmental exposure and environmental health. 
                  DR. KENNETH PORTIER:  Good morning.  I'm Ken Portier, retired biostatician with 40 years' experience in agriculture environment, environmental health, and public health.   
                  DR. ROBERT CHAPIN:  And we have Dr. Zhang on the phone.
                  DR. XUYANG ZHANG:  Good morning.  This is Xuyang Zhang.  I'm a senior environmental scientist with California Department of Pesticide Regulation, Environmental Monitoring Branch.  My area of specialty is pesticide transport and fate modeling, and now effective management of pesticide monitoring data and GIS.
                  DR. ROBERT CHAPIN:  Okay.  Thank you all.  An august group.  I'm looking forward to learning a bunch in the next couple days.  
                  All right.  So I think we're ready to go.  We've got -- the largest part of today is going to be presentations from our EPA colleagues about sort of setting the stage for the questions they're going to be asking us.  We have some bio-breaks built in, but when -- if you have your own sense of urgency, I encourage you to address that.  Just do what needs to be done.  
                  Let's see.  So I guess without -- and I'll just remind us that we need to -- because the meeting is being recorded and then transcribed, please get very close to the microphone and speak into it and just give your name.  You don't need your affiliation and your expertise and all that stuff, just your name.  That helps with the transcription.
                  So with that, I think we'll turn it over to Dr. Echeverria and she can have at it.
                   
INTRODUCTION AND WELCOME
                  
                  MS. MARIETTA ECHEVERRIA:  Great.  Good morning.  My name's Marietta Echeverria and I am the director of the Environmental Fate and Effects Division of the Office of Pesticides Programs.  It is my honor to welcome you to this consultation of the FIFRA Scientific Advisory Panel.
                  The Environmental Fate and Effects Division, or EFED like we like to call ourselves, is charged with conducting ecological risk assessments and drinking water assessments which feed into a human health risk assessment for conventional pesticides as part of the registration and registration review programs.  Our work provides critical information to agency decision-makers ensuring that pesticides, when used as directed, meet safety standards and to not cause unreasonable adverse effects to human health or the environment.  
                  I want to welcome and thank you.  I want to thank Dr. Chapin, the chair, for this meeting.  I also want to thank the permanent panel members as well as the ad hoc panel members for being here today.  I thank you for the work that you've done in preparation for this meeting, and I thank you for the input and advice that you'll provide throughout our time together over the next couple of days.
                  I also want to thank the public for participating in this process.  This is a meeting that's open to the public and I'm looking forward the public comments that we will get at the end of the day.  
                  I also want to thank my good friend, Ms. Tamue Gibson, the designated federal official.  I've known Tamue for a very long time and it has been a pleasure working with you in this role.  I thank you, Tamue, and I thank the SAP staff for all the work that they've done organizing this meeting and for serving as the liaison between the agency and the panel. 
                  The feedback we receive as part of the scientific advisory panel is very important as EPA moves forward with integrating surface water monitoring data in pesticide risk assessments.  We've been working with considerable determination to advance our methods for estimating pesticide concentrations in drinking water and the tools and approaches that we are presenting today are related to our goal, a goal that is shared by many of our stakeholders which is to make better use of surface water monitoring data quantitatively in our drinking water exposure assessments.
                  There's a full agenda today and you will be hearing a series of presentations from my team on how we conduct drinking water assessments using a tiered approach, the use of predictive models in monitoring data to estimate exposure, and how new tools and approaches, specifically SEAWAVE-QEX and sampling bias factors can improve our ability to use surface water monitoring data in refined assessments.  We will also present two case studies that illustrate the use of these new tools in the context of an actual assessment.  
                  I want to thank my team for the excellent work they have done in preparing for this meeting and in particular, for developing the White Paper, the drinking water assessment framework document, and the SEAWAVE-QEX SOP and the case studies, all of which are in the docket for this meeting.  
                  At this time, I am very happy to introduce the team that is presenting today as follows:  First, we have Mr. Dana Spatz here on my right.  Dana has served as the management lead for this project.  Dana will kick things off with an overview of the approaches that you'll hear about throughout the day, followed by Dr. Rochelle Bohaty to Dana's right.  Dr. Bohaty has been the technical lead for this project.  She will present on the framework followed by a presentation on the interpretation of monitoring data.
                  Next, we will have Dr. Sarah Hafner next to Rochelle.  Sarah is going to present on our evaluation of SEAWAVE-QEX, followed by two presentations from Dr. Christine Hartless and Chuck Peck, all the way to my right over here.  Both of whom are presenting on sampling bias factors.  
                  Following the sampling bias factors presentations, Dr. Hartless with make another presentation on our efforts to extrapolate across watersheds and a weight of evidence approach for evaluating relevancy.  Then we will have Dr. Katrina White, all the way over here on the left, and Ms. Jessica Joyce who will present the two case studies.  
                  And finally, we are very fortunate to have Dr. Matthew Bischof over here next to Sarah, I believe, representing Washington State Department of Agriculture.  Matthew brings an important perspective to this meeting.  He has on the ground experience and he will be discussing Washington State's program followed by the work he did in collaboration with the EPA team on the use of the tools in designing monitoring programs generally. 
                  I also want to take a minute here to thank some additional contributors, specifically Dr. Aldo Vecchia from the USGS who developed the SEAWAVE-QEX model.  Dr. Vecchia is participating with us over the phone today.  I also have Dr. Anna Lowit, the OPP's senior science advisor and Ms. Jan Matuszko over here, the acting EFED deputy director, both of whom provided excellent guidance and support to this team as they were preparing for this meeting.
                  I want to thank Mr. Trip Hook who I believe is participating with us over the phone who provided GIS assistance and to the many other staff members in EFED and OPP, some of whom are here today who provided input towards this efforts.  Again, welcome.  We look forward to your input, your advice, and the discussion.  Thank you.  
                  I'll turn it over to Dana now, I believe.  
                  
OPP TECHNICAL PRESENTATION  -  OVERVIEW OF THE APPROACHES FOR QUANTITATIVE USE OF SURFACE WATER MONITORING DATA IN DRINKING WATER ASSESSMENTS
                  
                  MR. DANA SPATZ:  Good morning.  My name is Dana Spatz and I'm a branch chief in the Environmental Fate and Effects Division where I manage a group of scientists that conduct ecological risk assessments, drinking water exposure assessments.  I led the team of scientists that will be presenting here today.  
                  I'm going to provide you with some background and context to the project you will be hearing about this morning, what brought us to this point, why this effort is important for evaluating the potential exposure to pesticides in drinking water --  
                  PARTICIPANT:  A little louder please.
                  MR. DANA SPATZ:  -- recommendations from previous FIFRA SAP panels, and the great value the meeting this week with bring going forward. 
                  The Office of Pesticide Programs regulates the distribution, sale, and use of pesticides and establishes maximum levels for pesticide residues in food.  The Federal Insecticide, Fungicide, and Rodenticide Act or FIFRA, provides the federal regulation of pesticide distribution sale and use.  With some limited exceptions, all pesticides distributed or sold in the United States must be registered by EPA.  
                  Before EPA may register a pesticide under FIFRA, the applicant must show, among other things, that using the pesticide according to label directions will not generally cause unreasonable adverse effects on the environment.  FIFRA defines the term "unreasonable adverse effects on the environment" to mean, one, any unreasonable risk to man or the environment, taking into account the economic, social, and environmental costs and benefits of the use of any pesticide or two, a human dietary risk from residues that result from a use of a pesticide in or on any food inconsistent with the standard under Section 408 of the Federal Food, Drug, and Cosmetic Act.
                  Section 408 of the Federal Food, Drug, and Cosmetic Act, or FFDCA, authorizes EPA to set tolerances or maximum residue limits for pesticide residues on food.  Under the FFDCA, EPA must determine that aggregate exposure to the pesticide residues is safe, that is, there is a reasonable certainty of no harm from aggregate exposure to the pesticide before issuing a tolerance.  As part of this determination, EPA must consider aggregate risk from exposure to a pesticide from multiple sources, food, water, residential, and other non-occupational sources.  
                  The Agency has developed approaches for assessing drinking water exposure and for refining those assessments to meet its obligations to assess aggregate exposure to the pesticide chemical residue under the FFDCA.
                  The Office of Pesticide Programs conducts roughly 100 to 150 drinking water assessments per year.  These assessments are done to support various regulatory actions including when a new pesticide is first submitted to EPA for registration, when subsequent new uses are requested to be registered, and in registration review where all registered pesticides are reevaluated every 15 years.
                  As you will hear about later, the Office of Pesticide Programs uses a tiered process for conducting drinking water assessments.  The level of effort needed to complete a drinking water assessment varies for each case and can range from a few days to several months depending on the exposure and toxicity profile, the number of uses considered, and previous work completed for the pesticide under evaluation. 
                  Concentrations of pesticides in both surface water and ground water sources are provided in drinking water assessments.  The concentration estimate recommended for use in the human health risk assessment considers both model estimated and measured pesticide concentrations.  The results are then integrated into the human health dietary risk assessment.  This can be done using a single estimated concentration and a deterministic assessment or distribution of concentrations for probabilistic assessment.
                  Most drinking water assessments conducted by the Office of Pesticide Programs have relied primarily on model estimated concentrations while monitoring data, if available, was summarized and used to characterize the potential for contamination of drinking water.  The reasons why monitoring data have not been used quantitatively include inadequate sampling frequency, limited number of years sampled at a site, and geographical scope of sampling compared with the pesticide use pattern.  
                  While there have been a few cases where monitoring data was considered robust enough to use quantitatively in drinking water assessments, most stakeholders and the Office of Pesticide Programs would like to make better use of the available monitoring data and additional monitoring programs going forward.  Together with the sophisticated surface water modeling tools currently in use, the Office of Pesticide Programs wants to use surface water monitoring data to estimate pesticide concentrations in drinking water to best reflect actual exposure, increasing confidence in risk conclusions.
                  We believe the recently developed tools you are going to hear about today can be used to estimate pesticide concentrations between sampling events in available monitoring data to address uncertainty in having non-daily sampling data.  In addition, we will present a weight of evidence approach we believe can be used to describe potential exposure across the landscape to help address uncertainty in having limited geographical representation. 
                  When reviewing the methods presented here, it is important to consider that models must be of appropriate quality and rigor for risk assessment and risk management applications; that is, we do not want a complex model to use as a screen at lower tiers.  It's just not practical when conducting -- when considering the workload of 100 to 150 drinking water assessments each year.
                  Before models are implemented for use, the Office of Pesticide Programs requests input and peer review from the scientific community to make sure we got it right.  Models used in a regulatory process need to be publicly available including the underlying code so that anyone can check our work.  And finally, we need a way to easily update the models we use to incorporate evolving risk assessment methodology.  
                  The Office of Pesticide Programs has developed and implemented tools and methods to estimate pesticide concentrations for use in drinking water assessments.  These tools and methods have undergone incremental improvements over the last 20 plus years, specifically 18 different scientific advisory panel meetings have focused on tools and methods specific to the conduct of drinking water assessments.  This includes seven on the use of monitoring data in pesticide risk assessments.  These panels of experts have included representatives from academia, industry, and government over the last two decades.  
                  The most notable scientific advisory panel meetings on this topic include those related to atrazine where they discuss methods for estimating exposure between measured samples.  Generally, the SAPs supported the concept of sampling bias factors and encouraged exploring the USGS tool SEAWAVE for estimating exposure between sampling events and for developing additional daily pesticide chemographs for further development of bias factors.  
                  Here's a quote from an SAP meeting in 2010 on the need for frequent sampling if one wants to use surface water monitoring data quantitatively in a drinking water assessment.  The concept behind this important recommendation has driven much of the work we are presenting here today, especially with the evaluation of the SEAWAVE-QEX model, particularly for pesticides with acute or short term toxicity endpoints of concern, the sampling frequency must be often enough to be able to compare exposure values with toxicity levels of concern.
                  The first bullet on this slide emphasizes the idea that it may not be appropriate to combine monitoring results from one site with those of another as each individual community water system site is unique, thus pooling of data across sites can lead to erroneous conclusions.  
                  In 2012, the Scientific Advisory Panel supported EPA's work in developing sampling bias factors and encouraged use to incorporate as much additional monitoring data as possible before attempting to correlate or predict sampling bias factors with watershed characteristics.  We followed up on this recommendation and will be presenting the work later this morning.
                  The SAPs in 2011 and 2012 agreed with EPA that the use of statistical time series models, such as the SEAWAVE-Q model, that use a covariate such stream flow may help to overcome the limited sampling frequency issue characteristic of many surface water monitoring programs.  Other methods, such as kriging, were explored and discussed but the Scientific Advisory Panel favored SEAWAVE-Q as being more practical for use in EPA drinking water assessments.
                  The development of USGS -- the development by USGS of SEAWAVE-QEX and our evaluation of the statistical model, which we will go into detail later today, was in direct response to the recommendations made by earlier scientific advisory panels.
                  So as you've seen, it's been several years that EPA has been looking at ways in which to make more use of surface water monitoring data in its risk assessments and there is a long history of Scientific Advisory Panel meetings that have gotten use to where we are today.  What you've seen in the White Paper and what you will hear presented today, we believe, follows up on the valuable input we've received over the years from the scientific advisory panel and our stakeholders.  We are grateful for all of those who have provided recommendations and we have taken those recommendations seriously and incorporated then into the work presented here.  
                  The feedback we receive this week with help inform several important and far reaching decisions the Office of Pesticide Programs will need to make when it comes to the future of using surface water monitoring data quantitatively in drinking water assessments.  These include minimum data quality and quantity criteria, design of future monitoring programs, and future steps in method development. 
                  Thank you for serving in this FIFRA Scientific Advisory Panel and we look forward to a vigorous and insightful discussion.  It's important that we get this right as we continue the work to provide growers tools to help produce a healthy food supply while at the same time, protecting our drinking water.  Thank you.  
                  DR. ROBERT CHAPIN:  Any questions from the panel?  
                  DR. CLAIRE BAFFAUT:  You mentioned that currently the way that the drinking water assessments are conducted are based on predictive models.  And I'm asking this question because in the document that we've read, there's several instances where it stated that the SEAWAVE-QEX would improve on current methods, but I -- could you explain a little more what those current predictive models are and how they're used?
                  MR. CHARLES PECK:  So this is Chuck Peck with EFED.  
                  So currently in our drinking water modeling, we use a model that we call the pesticide and water calculator.  It looks at different fate and chemical parameters, degradation in soil, degradation in water.  It looks at different watershed characteristics, runoff parameters, as well as meteorological data that could be representative of different areas of the country.  And we use that information combined with label information, label rates, retreatment intervals, the maximum amount that can be applied in a particular year, to develop a -- what we call an estimated drinking water concentration in modeling.  
                  Again, this looks at maximum label rates, minimum retreatment intervals, and what we're trying to do with the SEAWAVE-QEX model is to take information, monitoring data that's provided on the ground by different sampling agencies, as well as covariate data to try and inform those estimates that we develop using our pesticide and water calculator model.
                  DR. THOMAS POTTER:  Hello.  This is Thomas Potter.  I guess I'm going to have to stand or lean for this.  I have a follow-up question to that, and it came to mind as I was reading through the documents is, you know, how often do you run into the circumstance where the models don't give you appropriate answer and when you need to move to another tier or another level such as in using tools like SEAWAVE-QEX or whatever?
                  MR. DANA SPATZ:  So it's not that the model gives an inappropriate answer, it's that it's a screening level assessment that as we go through the tiers it becomes more and more refined using more and more real world data.  Generally, we're at Tier 2 using the pesticide and water calculator.  If we still have risks of concern, we move to Tier 3 where we'll be using typical application rates and more site-specific information.  And what we're looking to do with monitoring data is start looking at that more intensely using sampling bias factors at Tier 3 and then into Tier 4 if we still need to refine.
                  DR. THOMAS POTTER:  Can I follow up with that?  Yeah, perhaps I misspoke when I said inappropriate.  I said it didn't necessarily answer the question of whether or not there was a risk.  But what I wanted to get at is, you know, and I'm trying to figure out how critical it is to have this tool, SEAWAVE-QEX, in your risk assessment tool kit and that relates to basically coming down to registration decisions and how often do you think something like SEAWAVE-QEX would be critical in terms of coming up with an answer in terms of a particular registration of an active ingredient? 
                  MR. DANA SPATZ:  We've estimated approximately 10 percent of the chemicals would benefit from going through a higher tier assessment.  
                  DR. THOMAS POTTER:  Okay.  Good.  Thanks.  Exactly what I'm looking for.
                  DR. ROBERT CHAPIN: Okay.  Next up.  
                  DR. TIMOTHY GREEN:  May I ask a quick question?
                  DR. ROBERT CHAPIN:  A quick question, just for clarification.
                  DR. TIMOTHY GREEN:  Okay.  Just -- well, I'm sorry.  This is historical context.  There was a lag from 2000 to 2007 then almost annual meetings and then haven't had another one of these since 2012.  I just wondered if there was a reason.  
                  MR. DANA SPATZ:  There was a lot of focus on atrazine at the time and we were using atrazine because of the robust monitoring data to help develop the tools and with other priorities and things.  We've been working on it for a number of years and it's, you know, mature enough to bring to you today.  
                  DR. ROBERT CHAPIN:  Okay.  Dr. Bohaty.
                  
FRAMEWORK FOR CONDUCTING PESTICIDE DRINKING WATER ASSESSMENTS FOR SURFACE WATER
                  
                  DR. ROCHELLE BOHATY:  Hi.  Good morning.  Again, my name is Rochelle Bohaty.  I'm a senior chemist in the Environmental Fate and Effects Division and I will present the drinking water assessment framework used by the Office of Pesticide Programs to conduct drinking water assessments for pesticides in surface water.  I think this presentation will help address some of these questions that we had just a few minutes ago.
                  So during my presentation you will hear the purpose of documenting the process we use to conduct drinking water assessments.  In addition, I will provide you with an overview of the framework including a high-level summary of each assessment tier.
                  The purpose of the drinking water assessment framework is to document standard and longstanding practices for conducting drinking water assessments by creating a clear and concise yet comprehensive document that describes the tiered process.  In addition, we hope the document will increase consistency across assessments and foster better cross-division coordination within the Office of Pesticide Programs.  
                  The document also provides transparency to stakeholders on the conduct of drinking water assessments.  Lastly, the document integrates new methods for interpreting surface water monitoring data that you will hear about today that will help address spatial and temporal uncertainties in available surface water monitoring data.
                  As Mr. Spatz mentioned in his presentation, the Office of Pesticide Programs conducts roughly 100 to 150 drinking water assessments per year.  As such, the Office of Pesticide Programs has come up with a tiered assessment approach to prioritize resources.  Lower tiers are easy to use, require simple input, and produce conservative output.  Higher tiers require more input and produce more detailed and realistic output.
                  The drinking water assessment refinement process or framework consists of four tiers and a scoping phase.  I want to point out that the refinement process doesn't necessarily mean that concentrations go down but that we have considered more data and have a better understanding of the values increasing our confidence in the assessment conclusions.  Progression through the tiering process only proceeds until concentrations are below the level of concern or a risk management decision can be made.  We don't continue to refine because we can or the data support refinement.  We only refine until we can stop; that is meaning a risk management decision can be made. 
                  Determining the need to advance through the tiers is done by comparing the estimated pesticide concentrations to a drinking water level of concern, also sometimes referred to as a drinking water level of comparison or a DWLOC.  Simply put, a drinking water level of concern is used as a benchmark of how much space in the risk cup is available for residues in drinking water for a pesticide under evaluation.  
                  EPA developed the concept of a risk cup to facilitate risk refinement when considering aggregate human health risk to a pesticide.  The risk cup is the total exposure considering multiple pathways or routes of exposure including through food, shown on this image in blue, residential shown in orange in this figure, and water, the remaining top third of the cup in this example, allowed for a pesticide considering its toxicity and the required toxicity safety factors.  Exposures exceeding the risk cup are of potential concern.   
                  When drinking water assessments are completed for pesticides, both model-estimated as well as measured concentrations are considered; however, it should be noted that these two data sources can tell you very different things.  For example, monitoring data reflect typical use while modeling can reflect maximum label use rates, which are often different than the typically used rates.
                  The aggregated exposure assessments can be deterministic, using a point estimate or a probabilistic assessment by using the entire distribution of possible pesticide concentrations as mentioned earlier.  Drinking water assessments are completed on different spatial scales from national to regional or sub-regional levels, depending on the pesticide use under evaluation.  
                  Most assessments are completed on a national scale; that is, one high end estimate covers the entire country for all use sites.  However, regional or sub-regional scale assessments are conducted when needed.  
                  As this figure shows for a national scale assessment, generally a Tier 1 or a Tier 2 level assessment provides a single upper end pesticide concentration as a starting point.  This could be based on solubility or a model-estimated concentration.  This value is expected to rarely, if ever, be exceeded.  The estimate doesn't represent a concentration that everyone across the country is exposed or an average concentration across the country, but rather a pesticide concentration that could occur in a vulnerable pesticide use area.
                  If necessary, the assessment may be refined to a regional scale drinking water assessment.  In doing this, the United States is subdivided into smaller areas, typically water resource regions or hydrologic unit code 2 to focus on areas where the pesticide concentrations may be higher than the drinking water level of concern.  This can be done as part of a Tier 2 or Tier 3 level assessment.
                  A sub-regional scale drinking water assessment provides generally the highest spatial resolution of all assessment scales, further zooms in on vulnerable pesticide use locations where the estimated pesticide concentrations may be higher than the drinking water level of concern.  These spatially refined assessments occur at the Tier 3 or the Tier 4 level assessment.
                  I'm going to briefly walk you through the tiered assessment process beginning with scoping.  Scoping is the first step in any drinking water assessment.  It is also a process that is done in consultation with the entire pesticide team that includes members from several divisions within the office of pesticide programs.  This is the step where we gather all the information we have about a pesticide to determine the level of effort necessary to conduct the drinking water assessment.  For example, this is when the health effects division provides the Environmental Fate and Effects Division the drinking water level of concern.  
                  Scoping also occurs through the tiered assessment process.  So for example, the assessment team will scope out next steps following each tier and skipped tiers as appropriate.  
                  A tier 1 drinking water assessment provides bounds on potential pesticide concentrations in drinking water.  A tier 1 assessment is not intended to present actual drinking water exposure concentrations.  Tier 1 assessments are used to screen out pesticides having low risk potential or identify pesticides that require additional refinement. 
                  Since it is not feasible to directly measure pesticide concentrations for every location and time for a pesticide of interest, EPA uses data from one or more localized sites and scenarios from across the country to develop conclusions about pesticide concentrations across the landscape.  At Tier 2, this includes the use of aquatic models to derive upper bound pesticide concentrations based on maximum label rates and for vulnerable pesticide application locations.
                  We know that concentrations of pesticides are a local or regional phenomenon, but temporal variation based on pesticide characteristics, use, and environmental conditions; however, use of the upper bound estimate is expected to provide a protective measure of all uses across the United States.  Monitoring data are summarized at a very high level for characterization and generally, only to make sure that an important exposure pathway is not missed using modeling.
                  Tier 2 assessments are designed to identify pesticides with low potential for risk from residues in drinking water on a national basis.  Most of the drinking water assessments completed by the Office of Pesticides are Tier 2 level assessments.
                  Tier 3 assessments are less common and require more resources.  For example, usage data, that is data on the actual use of the pesticide, can be considered and is often done so on a regional basis.  Often pesticides are not applied at the maximum single or maximum yearly application rate as specified on the label.  This ultimately impacts the real concentrations of these pesticides in drinking water.  Tier 3 is used to further identify pesticides in corresponding uses or use sites where exposure in drinking water is not a concern and may be done on a more localized spatial scale, such as on a water resource region or HUC-2 basis.
                  Tier 3 is where EPA is looking to introduce methods presented here as a first step in estimated pesticide concentrations using surface water monitoring data.  This includes use of sampling bias factors from four pesticides as a screen to identify pesticides that exposure in drinking water is not likely a concern.  You will learn more about sampling bias factors in upcoming presentations by Dr. Hartless and Mr. Peck.
                  DR. ROBERT CHAPIN:  Dr. B., quick question.  
                  DR. VERONICA BERROCAL:  Yes.  Dr. Berrocal here.  I just had a quick clarification.  So no data is used in tier 1?  No modeled or no monitoring data is used for tier 1?
                  DR. ROCHELLE BOHATY:  So currently we don't have a Tier 2 model and so we usually go to a higher screen at Tier 2, but we could use data such as solubility as a tier 1 screen to see if there is a potential concern.  
                  Tier 4 level assessments provide the highest temporal and spatial resolution with watershed and scale-based concentrations.  Tier 4 assessments utilize both model estimated concentrations as well as monitoring data to determine pesticide concentrations in drinking water.  It further defines regions or areas of the country with a high degree of confidence where there may be a risk concern versus where risks to human health from pesticide residues in drinking water is unlikely.
                  Tier 4 is also where we are looking to introduce methods presented in the White Paper to estimate pesticide concentrations from available surface water monitoring data.  This includes use of SEAWAVE-QEX to estimate daily pesticide concentrations at specific locations that can be used directly in drinking water assessments to estimate pesticide concentrations in drinking water. 
                  You will learn more about SEAWAVE-QEX in upcoming presentations by Dr. Hafner.  In addition, the estimated SEAWAVE-QEX chemographs can be used in the development of pesticide-specific sampling bias factors.  
                  Finally, in Tier 4, we would use a weight of evidence approach to determine the relevancy of available monitoring data to source drinking water.  
                  To demonstrate the drinking water assessment framework, Dr. White and Ms. Joyce will be providing presentations on two drinking water assessment case studies illustrating how EPA conducts drinking water assessments for pesticides.  These case studies will focus on showing how sampling bias factors, SEAWAVE-QEX, and a weight of evidence approach can be used to quantitatively incorporate surface water monitoring data into drinking water assessments.  In addition, these case studies serve to provide a better understanding of the monitoring data typically available for pesticide drinking water assessments including the quality and the quantity of the data.  
                  We as you, the panel, to think about the case studies in light of the drinking water assessment and implementation of the tools for making better use of available surface water monitoring data.
                  With that, I can take any questions you have before I move on to my second talk.
                  DR. KENNETH PORTIER:  So I was thinking of our risk cup and Tier 2 where you kind of use one number for the drinking water and I guess you're thinking in terms of a maximum kind of drinking water exposure concentration.  Does that concept follow over to the residential and food exposures at the same time?  So is that cup a real conservative estimate, you know, the maximum potential food concentration, the maximum potential residential exposure, the maximum surface water?
                  DR. ANNA LOWIT:  Anna Lowit.  So the Health Effects Division, who is sister division to EFED that sits here who conducts the dietary, the foot assessment and actually does what we call aggregate, to add up the food, the water, and the residential when it does occur, and we do have a standard tiering process for the food assessment that's largely parallel to what you see from EFED where we began with very high end assumptions like 100 percent of the crop is treated around the United States, whereas in higher tier we'll do a percent crop treated for example.  Or we may start with the type of residues we use in the more screening level assessments as opposed to a later probabilistic assessment we would use while monitoring data at the point of sale.
                  For dietary, we have a nearly identical side-by-side process so that the goal in time is to get them more aligned so a Tier 4 across the divisions match.  It's a work in progress, but we do have conceptionally similar situation.
                  DR. KENNETH PORTIER:  So when you get to something like Tier 4, you're really dealing with three time series and looking at coinciding or noncoinciding exposures where food exposure may be down or water exposures might be high before the crop is gathered and goes into commercial use, so you're water might drop but your food consumption going -- are you at that level of complexity?
                  DR. ANNA LOWIT:  Most of the time, not.  So in most of our single chemical risk assessments, we make assumptions on the food side of things that there is very little annual changes in variability in food.  We know with Thanksgiving coming, a lot of people will eat turkey and cranberry and things like that, but for most of the year, most food consumption programs have demonstrated that people eat berries year-round.  
                  And keep in mind, we also get food monitoring data from imports, so USDA and FDA and others are monitoring the food throughout the year.  People eat strawberries year-round, even if their only grown in certain parts of the country in certain times of the year.  So we generally assume for most of the year that food is a constant from a time point of view.  
                  There are situations, like a couple of our very complicated accumulative risk assessments, which you sat on a few of those panels I think, where we have done a far more sophisticated time course of food consumption.  Because when you add up 30 organophosphates in a single risk assessment, you have to throw every tool known to risk assessors at that to get a realistic estimate.  But in most cases, we don't.  
                  DR. ROBERT CHAPIN:  Ray.
                  DR. RAYMOND YANG:  Ray Yang.  Thank you very much for your presentation.  
                  I have three little questions for clarification okay.  You good folks do 100 to 150 assessments a year of which, what is the percentage of tier 1, 2, 3, 4, roughly?  
                  MR. DANA SPATZ:  Obviously, it would depend on which chemicals we're looking at in that year, but the vast majority would be a Tier 2.  Some would go into Tier 3, but the vast majority is Tier 2.
                  DR. RAYMOND YANG:  All right.  All right.  Second question, a follow-up to Tom's question, do you consider inhalation and dermal exposure at all?
                  DR. ANNA LOWIT:  The short answer is yes.  So if you go back to the risk cup, there is that, I think it's pink or orange segment that would be considered residential.  So our Health Effects Division also conducts residential and occupational exposure assessment where we do evaluate workers in the field who may be handling pesticides, either they're spraying or their picking fruit for example, and those may have inhalation or dermal contact.  
                  We also conduct assessments of children and adults who apply pesticides around their homes, to pets, ant baits in the house, turf uses, the whole gamut, mosquito use.  We do public health spraying assessments.  So yes, we are counting for those routes.  But the focus of this meeting is just on that drinking water piece from the risk cup.  Not on how it's all put together, but the tools of how we want to refine them.
                  And I think one comment, I think a few of you are having questions about the frequency that we're going to need to use these tools.  So it may not be that often as a percent basis, but keep in mind that when we're going to need to use these tools are also some of the most important tools to our growers and are some of the most heavily used pesticides in the country.  So although there's frequency that may be not very often, when we need these tools, they're extremely important to us.  And as we get in the last couple of years of this 15-year review cycle for registration review, some of our most complicated assessments are yet to come in the next two years where we would really like to apply these tools to these really complicated cases.
                  DR. RAYMOND YANG:  Thank you.  Third question, from your presentation I gathered the modeling started to come in at level 2, Tier 2; am I correct?
                  DR. ROCHELLE BOHATY:  So historically, we did have a tier 1 model, but as of the last few years, we stopped using that tier 1 model because often times when we went to Tier 2, the concentrations were higher and so we needed to come up with a better screen at the tier 1 level.  So currently we don't start modeling until Tier 2.
                  DR. RAYMOND YANG:  Okay.  So when you start to do modeling, are you dealing with SEAWAVE?
                  DR. ROCHELLE BOHATY:  No.  So where we're proposing to integrate SEAWAVE would be in a Tier 4 level assessment.  
                  DR. RAYMOND YANG:  Okay.  Thank you.
                  DR. ROBERT CHAPIN:  One more question would be okay.  Yep.
                  DR. VERONICA BERROCAL:  This is Dr. Berrocal.  I was just confused about this drinking water level of concern.  If I'm looking at these pictures, it seems to me that everything is basically estimated or extrapolated.  What is the food exposure, there is a natural exposure, so is there any uncertainty that is considered when this drinking water level of concern is established?
                  DR. ANNA LOWIT:  So the drinking water level concern as it notes there is a benchmark.  In our lower tier assessments, that benchmark is a deterministic benchmark.  So literally it's just some simple math of the hazard assessment point of departure accounted for the uncertainty factors defines the size of that.  So the size of an organophosphate risk cup is actually quite small, whereas something like glyphosate, the risk cup is actually quite large because as determined by the size of that point of departure and the number of uncertainty factors.
                  And so the rest of it's just math.  The amount of food that we have calculated in the tier 1 or Tier 2 assessment plus those determined as residential and it's essentially just a subtraction exercise.  What's available left.  
                  As we move up to the higher tiers and the more complicated Tier 3s and the Tier 4s, we're doing more distributional probabilistic assessments, so the distributions of the water are brought into the distribution of the food, so the mathematics become a lot more complicated.  
                  We don't, however, in our risk assessments, provide error bars on margins of exposure, things like that when we provide the assessments to the risk managers.  They will provide a probabilistic distribution of different exposures of different percentiles, for example, but not error bars on those.  
                  DR. VERONICA BERROCAL:  So, follow-up question.  I guess my curiosity, or my misunderstanding is whether this drinking water level of concern, you are using it to determine which pesticide will go to Tier 2, right?  So is that right? 
                  DR. ANNA LOWIT:  Yeah.  So for example, let's say, one of the -- I think one of the case studies that you'll see later this afternoon demonstrated in lower tiers that the food had been calculated.  There's residential exposures to one of the examples that we're showing.  So if you essentially in a lower tier subtract out what's left for water, that becomes a DWLOC and so as they do modeling, you'll see in some of the graphs that they'll show later that those model estimates exceed that level, that we crossed that benchmark.  When we cross that benchmark, that's where it began to trigger the conversations between the Health Effects Division and EFED on where is the quickest and fastest to do refinements to get back within the risk cup.  And then if we make -- often, you know, as we move in the latter parts of the reg review, we're going to be -- both divisions are going to be making heavy refinements on a number of chemicals to do probabilistic assessments at these higher tiers.  So it becomes far more complex.  It's not necessarily even a linear approach because the divisions are working together to be efficient, because we have to get through 100 a year, so we have to be efficient with how we use our time.
                  DR. ROBERT CHAPIN:  Tom.
                  DR. THOMAS POTTER:  Tom Potter.  I just had the quick question, where are safety factors applied?
                  DR. ANNA LOWIT:  So generally, the workflow for risk assessment is that the uncertainty factors are applied through the hazard assessment, the hazard characterization level.  So a point of departure is selected from toxicology studies generally, and in most cases, the Pesticide Office uses animal studies with the standard 10X safety factor for intraspecies and interspecies.  
                  We generally do two different types of risk metrics.  One is a reference dose approach and one is a margin of exposure.  Essentially, they're inverses of the other, but the uncertainty factors come from that hazard assessment side.  
                  DR. ROBERT CHAPIN:  All right, Dr. B., back to you.  
                  DR. ROCHELLE BOHATY:  Thanks.  Before I start my next presentation, I want to provide a little bit more context to make sure that you guys are following the process in terms of modeling.  So the current modeling, or the pesticide water calculator as Chuck mentioned earlier, it doesn't consider monitoring data.  It relies on environmental fate data that's submitted to the agency that's conducted in a laboratory.  And so based on the tier that we're at, we can make those, the input parameters into the model more or less conservative.  But I just wanted to say, because that's very different than SEAWAVE where you would actually input like measured concentration data.
                  Does that introduce any other follow-up clarifying questions?
                  
ANALYZING AND INTERPRETING SURFACE WATER PESTICIDE MONITORING DATA

                  DR. ROCHELLE BOHATY:  Okay.  So next I'm going to talk about how the Office of Pesticide Programs analyzes and interprets surface water monitoring data.  During this presentation, I will provide you with some background information to provide context around the drinking water assessments conducted by the Office of Pesticide Programs.  This includes a summary of drinking water sources as well as factors that drive pesticide concentrations in surface water.  I will also highlight data considerations and finish with providing an overview of the methods that you will hear about today.
                  So more than 282 million people, that's 87 percent of the total population, receive drinking water via community water systems.  In 2015, the total amount of drinking water derived from surface water was about 61 percent of all drinking water.  Surface water sources of drinking water include rivers, streams, lakes, and impoundments such as reservoirs.  Community water systems rely on surface water are distributed across the 21 major water resource regions located in the United States, 18 of which are in the contiguous states.
                  This figure highlights how the community drinking water systems vary across the landscape; that is, including watershed size, percent agriculture, and we know how environmental conditions vary across the landscape.  
                  This figure begins to illustrate how drinking water is a local phenomenon.  No one is drinking an average pesticide concentration derived from across the landscape and there are many situations where the water the person is drinking is coming from the same location.  On the short-term basis, for example, someone could go to home, work, or school, and still be consuming drinking water from the same source; however over longer periods of time, for example, someone may live in the same community over time whereas other folks may choose to move or travel elsewhere.  
                  Further complicating things is the fact that pesticide concentrations vary across the landscape for reasons other than previously described.  Factors driving pesticide concentrations in surface water include use patterns, that is the application rate, the retreatment, the timing, the practices that's used by the farmer, and those drive things like runoff and drift.  Physical chemical properties such as volatilization and solubility, environmental fate properties including transformation, sorption, or leaching, and waterbody characteristics including size, type, and flow, and soil and rainfall characteristics also play a part.  These factors need to be kept in mind when evaluating monitoring data. 
                  Monitoring data are often available from several sources including federal, state, and academic, as well as other sources such as registrant submitted data.  Often the design and the implementation of the programs vary dramatically and as such are designed to be used differently.  Furthermore, the quality of the data also varies.
                  Often we are criticized for the fact that there is 30,000 or 50,000 samples, for example, available for a given pesticide when we're conducting a drinking water assessment, and yet we only use the data for characterization purposes.  This is due to the uncertainty and the temporal and spatial occurrence of the sampling.  For example, there are many sites where only one sample has ever been collected or one sample was collected once per year and maybe for only a few years.
                  Therefore, temporal and spatial information needs to be considered including pesticide usage, varying environmental conditions across the landscape and across time, as well as the program design to better understand the relevancy of the data before integrating it into a drinking water assessment.
                  To give you a better idea of the impact of sample frequency, this figure here presents a real pesticide chemograph, that is, daily pesticide concentrations measured from grab sample.  These data are shown with the black dashes.  The Y-axis is concentration while the sampling date is show on the X-axis.  The orange dots illustrate a hypothetical sampling program that occurs every 14 days.  It is important to note that often we rarely have access to data sampled this frequently.
                  You can see that these hypothetical samples all miss the peak or the highest measured concentrations of the original daily chemograph shown with the black dashes.  As a result, these subsample data would generally underestimate the real exposure.  The less frequent the sampling becomes, the more and more likely you are to miss peak concentrations.
                  We have done this same type of analysis with samples coinciding with peak measured concentrations and while you capture the one-day peak concentrations, you can underestimate as well as overestimate concentrations depending on the concentration duration of interest.  To address the uncertainty and non-daily sampling data, we need a robust method for estimating concentrations between sampling events such that we don't bias the estimates compared to the real concentrations.  
                  As important as being able to estimate concentrations between sample events is the ability to estimate concentrations across space, or at least understand how concentrations change across space.  Use and environmental characteristics are major drivers.  This figure showing total precipitation across the United States is one example to show how various parameters affecting pesticide concentrations in surface water change across the landscape.  Going with this example, exposure estimates in one location may be completely different that another location. 
                  One could simply be due to the difference in rainfall, or it may be less direct as the difference in pest pressure or the corresponding pesticide applications due to rainfall.  In this figure, the darker brown shading indicates the dryer the area while the darker turquois, the wetter the area.  Conditions such as precipitation can vary on large scale, that is across the country, or on a much smaller scale.  For exampling, looking at the state of Washington or Orgon, you have the rain shadow from the Cascade Mountains.  The precipitation varies as much within this HUC-2 region as it does across the country.
                  So you can see how these parameters impacting pesticide concentrations in surface water can quickly become complicated when trying to understand how it impact pesticide concentrations in surface water.  
                  Two prior FIFRA Scientific Advisory Panels recommended that EPA consider SEAWAVE-Q, a USGS model that fits a regression to pesticide concentration data from using stream flow samples to assess variability in the trends, to impute pesticide concentrations with less than daily sampling.  The panel, however, had concerns that peak concentrations may not have been captured by SEAWAVE-Q.  
                  USGS made modifications to SEAWAVE-Q so that peak concentrations could be captured and recently, in 2018, released a new model called SEAWAVE-QEX.  This model imputes concentrations using a daily covariate such as flow.  It is specifically designed to capture higher end concentrations.  It produces multiple equally probably pesticide chemographs as shown on the figure, the different colored lines.  The Y-axis in this figure is concentration with the sampling here shown on the X-axis.  You can see that this tool is able to account for the seasonal nature of pesticide occurrence.  See the repeating peak exposure periods on the figure.  Stay tuned for more in depth description as well as the evaluation of SEAWAVE-QEX in Dr. Hafner's presentation.  
                  Although many of you may be familiar with the concept of sampling bias factors, I am going to give you a basic definition.  A sampling bias factor is a multiplier for monitoring data to account for the uncertainty associated with non-daily sample frequency.  
                  As you saw on the earlier slide, a reduction in sample frequency can have a substantial impact on the various summary statistics calculated from the data, depending on the number of samples and when sampling occurs.  A sampling bias factor can be applied to a summary statistic such as an annual or a 21-day rolling average concentration from less than daily monitoring data to ensure that at least some percentage of the time the sampling bias factor adjusted concentrations is equal to or higher than true unknown concentrations.  Currently, the sampling bias factors are developed such that the adjusted values will cover the true unknown maximum concentration 95 percent of the time.
                  While you will hear that SEAWAVE is a useful tool, it does have minimal data requirements as previously mentioned.  Often available surface water monitoring data are not suitable for use in SEAWAVE-QEX.  As such, sample bias factors are needed to best integrate these data into drinking water assessments.  
                  The method used to develop short-term sampling bias factors was previously presented to a scientific advisory panel while the method used to develop long-term sampling bias factors has not.  You will hear more about the development and evaluation of sampling bias factors in presentations later today by Dr. Hartless and Mr. Peck.
                  There are several different sources of surface water monitoring of data available to the Office of Pesticide Programs and it's important to remind you that not all data are created equal.  For the work you will hear about today, we selected and used several different sources of monitoring data which I have listed on this slide including the National Center for Water Quality Research, commonly referred to as Heidelberg University, the U.S. Geological Survey, Washington State Department of Agriculture, the Atrazine Monitoring Program, the Atrazine Ecological Monitoring Program.  
                  These programs have different study designs including sample frequency, the number of years sampled, the relevancy to pesticide use and source drinking water, and site characteristics.  Each program is summarized in the White Paper in more detail and you will hear more about these data and how they were used in our work as we go through the next several presentations.
                  On this slide, I have displayed a flow chart to help orient you with the workflow process for the development and evaluation of the tools you're to hear about today.  This chart, which you will see in the next several presentations, illustrates how the different methods that will be presented fit together in the overall process.  This chart also follows the outline of the White Paper.  I should note though, that this process is different than how we propose to implement the use of the tools in our tiered assessment approach as I described in my previous presentation.
                  This chart begins with non-daily pesticide concentration data and covariate data such as flow.  These data are processed through SEAWAVE-QEX, producing daily pesticide concentration data or chemographs.  These estimated concentrations, or if data are available on a daily time scale, you can input these into the sampling bias factor programs, to develop either short or long-term sampling bias factors.  In the end, regression analysis was completed for sampling bias factors once developed.  
                  I can take any of your questions. 
                   
                  DR. THOMAS POTTER:  Tom Potter here.  Can you expand a little bit more on the regression analysis of the sample bias factors, or are we going to hear about that later?
                  DR. ROCHELLE BOHATY:  Yeah, you will hear about that later.  
                  DR. THOMAS POTTER:  Okay.
                  DR.  CLIFFORD WEISEL:  Cliff Weisel.  So, you indicated that not all monitoring data can be used, which I certainly understand.  But do you have a specific set of criteria that you can share with us to make a decision, or is it you look at them and put your best guess at that point?
                  DR. ROCHELLE BOHATY:  So, in the White Paper we did put forth some data quality -- or data quantity information for you guys to weigh in on.  Both for the use of SEAWAVE-QEX as well as the use of sampling bias factors.  Up until now, it's been based on the assessor's best judgement.  
                  DR. VERONICA BERROCAL:  Dr. Berrocal here.  Can you go one slide back?  So, does it exist, pesticide data that is measured daily?  My understanding from reading the White Paper was that that data does not exist.  
                  DR. ROCHELLE BOHATY:  So, there are a few pesticides where we do have daily data, it's just not many.  Atrazine is one example.  There are a few other chemicals where we do have daily pesticide data, but it's not extensive; which is why we're looking for methods to be able to expand.  
                  DR. CLAIRE BAFFAUT:  Claire Baffaut here.  Do I understand correctly that if the best -- if you have daily pesticide data, when does the sampling bias factor equal 1?  I mean, why do we have a sampling bias factor if we have daily data?
                  DR. ROCHELLE BOHATY:  Yeah, so if you had daily data, you would use it, but I think this question is probably better addressed in the sampling bias factors when they talk about the development of it.  But you could develop a bias factor based on that to be applied elsewhere where you wouldn't have the daily data.  
                  DR. RAYMOND YANG:  Just a quick clarification as long as we have this -- oh, I'm Ray Yang.  As long as we have this on the screen, now, in your presentation, you mentioned something about probabilistic assessment, okay.  Is it somewhere here or later on in the risk assessment process?
                  DR. ROCHELLE BOHATY:  So the probabilistic component that we've talked about now has been more in the context of how we would use the data in the human health risk assessment.  
                  DR. RAYMOND YANG:  Later?
                  DR. ROCHELLE BOHATY:  After we give the number to our sister division.
                  DR. RAYMOND YANG:  Okay.  Thank you.  
                  DR. ROBERT CHAPIN:  We good?
                  DR. JAMES SADD:  James Sadd.  You mentioned or you introduced the idea of the covariates and how they would be used.  One of the problems with the monitoring data is that it is only available on the level or on the scale or on the temporal relation that is available.  How about the covariate data?  Does that also vary where you don't have enough or the proper covariate data in order to produce a sampling bias factor?
                  DR. ROCHELLE BOHATY:  That's correct.  
                  DR. ROBERT CHAPIN:  Okay.  So I've got a little break now; is that right?  Are we -- okay.  Let's take 10 minutes so we logged at 10:25 so let's meet at 10:35.  We'll reconvene in 10 minutes.  Thank you. 
                                    [BREAK]
                                       
                  DR. ROBERT CHAPIN:  Okay.  We're going to resume now with presentations from Dr. Hafner and Dr. Bohaty, right?  Am I right?  Okay.  
                  DR. SARAH HAFNER:  I don't think the slide is working.
                    
EVALUATION OF SEAWAVE-QEX AS AN IMPUTATION TECHNIQUE FOR ESTIMATING DAILY PESTICIDE CONCENTRATIONS FROM PESTICIDE MONITORING DATA PART 1
                  
                  DR. SARAH HAFNER:  Good morning.  My name is Sarah Hafner and I'm a chemist with the Environmental Fate and Effects Division.  This morning I'll be talking about the evaluation of SEAWAVE-QEX which we've broken into two presentations to allow for clarifying questions on the methods before discussing the results.
                  So this is a brief outline of the first presentation on the SEAWAVE-QEX.  First I'll cover some of the background of the model and the objectives of the evaluation.  Next, I'll give a brief overview of some of the key aspects of the model, and finally, I'll discuss the data preparation and methods we use for evaluating the model.
                  So if you recall this process overview from Rochelle Bohaty's presentation, the use of SEAWAVE-QEX is a critical step in a large project investigating tools for using monitoring data quantitatively in risk assessment which combines non-daily measured pesticide data with a daily covariate like streamflow to estimate daily pesticide concentrations.  
                  In addition to proposing the use of SEAWAVE-QEX for direct estimates of pesticide concentrations in drinking water assessments, SEAWAVE-QEX-generated chemographs are being used to developing sampling bias factors for pesticides without daily monitoring data, a process which will be discussed more in later presentations by Christine Hartless and Chuck Peck.  
                  Daily pesticide concentrations are needed for estimating short-term exposures and to calculate sampling bias factors.  As a result, the use of monitoring data quantitatively through daily monitored concentrations has been limited to a few chemicals.  To account for uncertainty from infrequent sampling, previous SAPs in 2011 and 2012 recommended that EPA consider the model SEAWAVE-Q to interpolate or fill in daily pesticide concentrations for data sets with less than daily sampling; however, those SAPs expressed concerns that SEAWAVE-Q may not adequately estimate peak concentrations for direct use in short-term exposure assessments.  
                  In response, the USGS released a new time series regression model called SEAWAVE-QEX in 2018.  Like SEAWAVE-Q this model interpolates sparse pesticide concentrations using a daily covariate such as streamflow; however, this model differs from SEAWAVE-Q in that it is designed to capture higher end concentrations.  Additionally, SEAWAVE-QEX can produce multiple equally probable estimates of the daily concentrations.  With each chemograph constrained by the measured input data.
                  It is important to note that this input data must meet some minimum requirements to get a proper fit by the model.  The model generally needs at least three years of sampling, at least 12 samples per year, and a detection frequency greater than 30 percent.  There is some flexibility in these requirements based on a condition such as chemical properties and sampling structure as discussed in the publication which is released along with SEAWAVE-QEX in 2018 and we are looking for feedback on data characteristics to consider when using flexibility in these minimum requirements.
                  The broad objectives of this evaluation were to determine if SEAWAVE-QEX can produce daily concentrations with reasonable accuracy such that the peak concentrations are not underestimated or grossly overestimated and if so, to determine how to appropriately adapt the tool to our drinking water assessment process.  
                  SEAWAVE-QEX consists of a simple code that's run in R with several modifications that are available to adjust for poor model runs.  Like SEAWAVE-Q, there are two inputs to SEAWAVE-QEX, measured pesticide data and continuous streamflow data.  When USGS gate stations are used from streamflow, the user only needs to supply the pesticide data as SEAWAVE-QEX will download UGSG streamflow data based on the gate station number.  
                  The two most important output files produced by the model for risk assessors are the diagnostic plots for evaluating the model fit and the file of daily concentrations or chemographs.  SEAWAVE-QEX can produce as many chemographs as the user specifies with the default value being 100 chemographs.
                  Diagnostic plots are used to determine if the model fits appropriately which is essential to having confidence in the estimated concentrations.  If not, some default input parameters in the model can be changed to improve fit.  If it is determined the model fit well and no further changes are needed, then the chemographs can be used.  In addition to these files, the model provides documentation of the model fitting parameters for each run.  
                  To understand the basics of how the model works -- 
                  Question?
                  DR. VERONICA BERROCAL:  Yes, Dr. Berrocal here.  So I just wanted to ask about the pesticide data.  Is that like an average daily concentration or it is what referring to the White Paper graph samples which I think is just referring to a specific time point.  
                  DR. SARAH HAFNER:  Usually they're graph samples.  
                  DR. VERONICA BERROCAL:  So the pesticide data is basically specific to a given time point while the streamflow data is some sort of measurement that is representative of the entire day?
                  DR. SARAH HAFNER:  Yes.
                  DR. VERONICA BERROCAL:  So they're not on the same time scale, those two measurements?
                  DR. SARAH HAFNER:  Yes.
                  DR. VERONICA BERROCAL:  Okay.
                  DR. SARAH HAFNER:  Okay.  To understand the basics of how the model works, it's helpful to understand how SEAWAVE-QEX uses the covariate data.  The covariate is used to calculate midterm and short-term flow anomalies which are unitless measures of the variability in the covariate, in this example, streamflow.
                  The midterm flow anomaly represents seasonal variability and is calculated from the streamflow data for the proceeding 30 days.  An example is given in this figure which shows the seasonal variation in streamflow over the years of a SEAWAVE-QEX simulation.  
                  On the other hand, the short-term flow anomaly captures high frequency variability and flow as can be seen in the figure on this slide in comparison to the midterm flow anomaly from the previous slide.  As a result in flashy systems that have a quick response to precipitation events, the short-term flow anomaly will tend to be positively skewed compared to the midterm flow anomaly.  The use of both the midterm and short-term flow anomalies in the model allows for estimation of concentrations based on seasonality as well as estimations of upper end concentrations to capture all variability of pesticide concentrations. 
                  So over the next few slides, I'm going to walk through a few examples of diagnostic plots including examples of acceptable model fits as well as an example where more work is needed.  This figure is an example of one of the diagnostic plots generated by the model and shows just one of the chemographs in gray with concentrations in log-based ten scale on the Y-axis and years of the simulation on the X-axis.  The daily concentration chemographs produced by SEAWAVE-QEX are constrained by the measured input data which are shown here as red circles.  
                  DR. KENNETH PORTIER:  Excuse me.  On this chart, does the graph for this one realization, does it go right through the red dots? 
                  DR. SARAH HAFNER:  Yes.
                  DR. KENNETH PORTIER:  In other words, it doesn't predict the uncertainty of that dot.  It goes right to that sample value, right?
                  DR. SARAH HAFNER:  Yes.  They do.
                  DR. KENNETH PORTIER:  Okay.  
                  DR. SARAH HAFNER:  The -- every chemograph is constrained by the measured input values.  
                  DR. VERONICA BERROCAL:  Dr. Berrocal here.  So what is -- I was think -- I was confusing when reading the White Paper was the censored value or the censoring limit.  There was no explanation of what the censoring limit is.  Is the same as limit of detection, or what is the definition?
                  DR. SARAH HAFNER:  We do talk about that in the standard operating procedure in a little bit more detail, but it is the limit of detection that you input into your measured data.  So it could be a limit of detection that comes with the data that you download.  
                  The horizontal red line in this graph represents the censoring limit that was specified in the input data.  The blue line within each blue box shows the estimated maximum concentrations for each year which is the average of the maximum concentration from each simulation for that year.  The blue boxes are 80 percent error bounds on this average so that smaller blue boxes indicate more confidence in the estimated maximum concentration for that year across all chemographs and large blue boxes indicate more variability in the estimated maximum concentration across chemographs.
                  DR. VERONICA BERROCAL:  Sorry to be annoying.  So can I know how those blue boxes are calculated?  Is that like the 10th and 90th percentile of the condition of simulated values?  Or is it using normal approximation to compute those confidence bands?  So you have 100 simulations.  You compute the average maximum of all this simulation that you think 10 lowest and the 90th highest and that's how you compute your bands?
                  DR. SARAH HAFNER:  I believe it is the 10th and 90th percentiles.  
                  DR. VERONICA BERROCAL:  Okay.
                  DR. SARAH HAFNER:  For use in risk assessment, a line indicating the drinking water level of concern can also be added to the plot as shown in this example.  The relationship between the drinking water level of concern in the blue boxes, that is, the error bounds on the estimated maximum, can help risk-assessors determine the likelihood that a site might be a risk concern.  For example, if the error bounds are small and overlap with the drinking water level of concern, or if the drinking water level of concern is lower than the estimated maximum concentrations, there's high confidence that the site poses risk to exceeding the level of concern.  This concept will be discussed in a later presentation on the short-term case study by Katrina White. 
                  As the name implies, SEAWAVE-QEX fits the data to a seasonal wave model.  This diagnostic plot shows an example of a seasonal wave model fit to the data with all of the years of data in the simulation compressed to one calendar year so that the seasonality of concentrations can be seen.  In this example, the season of pesticide occurrence is from about May to early July which is highlighted in gray shading.  The red dots are, again, the measured input data and open red circles are censored data points.  Note that the Y-axis is, again, in log-based ten scale.  The black line is the seasonal wave fit to the data and the dashed lines represent the seasonal standard deviations around that wave.  In this example, the data are contained within the bounds of the seasonal standard deviations indicating that the model assumptions are verified for this plot.   
                   There are several key features of the diagnostic plots that are important for having confidence in the estimate chemographs, and there are choices that our user can made to improve the model fit as needed.  We have documented these in a standard operating procedure for using SEAWAVE-QEX.
                  One feature to highlight is the presence of a secondary season without data to support it, demonstrated in the figure here.  There is a setting in SEAWAVE-QEX that will restrict the peak for a partial record to make sure that the peak occurs during the sampling season and essentially limit the model to selecting a single season instead of two.  Generally, this is recommended when all of the data for the time period being analyzed falls within a six-month window.  For example, the figure here shows all of the data from the years 2009 to 2014 in one calendar year and all of the sampling occurred between early March and early September which is shaded in blue.  And this indicates that a partial record restriction may need to be applied.
                  In the figure on the right, the partial record restriction has been added to the model and a single wave is selected.  While this setting can be preselected based on the data structure, it may not be practical for risk assessment to predetermine the sample structure on a site by site basis, especially given the short amount of time available to process the data for risk assessment.  
                  Therefore, we have only recommended rerunning the model with this setting when the diagnostic plots indicate a problem with the default so that sites can be run in batch and only rerun select sites with a partial record restriction when a two-wave model is selected without data in the second wave.
                  DR. CLAIRE BAFFAUT:  Claire Baffaut here.  Just a small clarification, is this what you called optimization in the White Paper?
                  DR. SARAH HAFNER:  I'm not sure if we -- 
                  DR. CLAIRE BAFFAUT:  There is several instances, for example, all the SBF discussions, there's a caveat that says that the model was not optimized.  
                  DR. SARAH HAFNER:  Yes.
                  DR. CLAIRE BAFFAUT:  Is that what you're talking about?
                  DR. ROCHELLE BOHATY:  So the optimization that you're referring to in the White Paper is specific to the sampling bias factors and not to SEAWAVE-QEX.
                  DR. CLAIRE BAFFAUT:  So, no.  
                  DR. ROCHELLE BOHATY:  So we'll be talking about the sampling bias factors -- 
                  DR. CLAIRE BAFFAUT:  Okay
                  DR. ROCHELLE BOHATY:  -- in a couple of presentations.
                  DR. CLAIRE BAFFAUT:  All right. 
                  DR. SARAH HAFNER:  And that optimization will be discussed separately.  
                  So now that we've done a high-level review of the basics of SEAWAVE-QEX, we'll move into how we evaluated the model.  The evaluation focused on three parts to ensure that the model was robust enough for the types of data sets that we anticipate in countering in risk assessment. 
                  First, we evaluated the accuracy of the model using both streamflow and estimated precipitation as covariates by subsampling near daily measured concentrations running the subsample data sets in SEAWAVE-QEX and comparing the SEAWAVE-QEX estimates to the original measured data.  
                  Second, we evaluated the use of alternative covariates besides streamflow, qualitatively, by comparing how the flow anomalies were generated and used in the model.  These covariates included estimated precipitation, measured precipitation, and stream stage or height.  
                  Third, we evaluated the use of SEAWAVE-QEX with low to no flow systems by viewing the model fits in the diagnostic plots.  
                  Accuracy was assessed in a quantitative way by comparing SEAWAVE-QEX estimates to measured data which required preparing the data before use.  These treated data sets were used for determining how much SEAWAVE-QEX underestimates concentrations as well as for evaluating the accuracy of point estimates and distributions of SEAWAVE-QEX estimates using both flow and model precipitation as covariates.
                  Data sets used for another analyses, like the alternative waterbody analysis, did not have daily monitoring and so a qualitative approach was taken by evaluating the diagnostic plots only.  Those data sets were not subsampled before use.  
                  For the quantitative analyses, near daily measured data was subsampled to create new data sets with infrequent sampling more typical of what is observed in water monitoring programs.  These subsamples were also meant to test close to the lower bounds of the SEAWAVE-QEX requirements for sample number by only pulling 12 measured samples per year.  Each subsample data set had a random seed start date early in the year with samples pulled every 7 or 14 days until 12 samples were reached for the year.  Then this process was repeated for the following years.  For example, for the seven day stratified samples, this results in data sets with 12 samples per year with all 12 samples occurring within a 12-week window within each year; however, across years, the total number of months over which sampling occurs will vary because of the random start dates for each year.
                  This subsampling process was repeated five times per data set to obtain replicates for combination of variables including pesticide site, covariate, and subsampling strategy.  These include atrazine and metolachlor evaluated with flow and precipitation at four sites for the NCWQR data.  
                  The figures on this slide are one example of subsampling a measured data set and one site with a seven-day stratified sampling strategy where the gray circles represent concentrations and the red circles are censored data points.  Note that in the full measured data set on the left, concentrations range up to 90 micrograms per liter, but many of these higher concentrations are removed in the subsampling process leaving the maximum concentration of the subsample data set at about 30 micrograms per liter.  By using only 12 samples per year in the subsample data sets, we're not only evaluating how well SEAWAVE-QEX can fit a model to sparse data, but also if it can capture the peak.
                  DR. KENNETH PORTIER:  So, can I ask a clarifying question?  
                  DR. SARAH HAFNER:  Yes.
                  DR. KENNETH PORTIER:  So you have a random start seed, right, and then in your stratified sample, say you're doing seven-day subsamples, you just start with that seed and do a systemic sample every seven days?  You take the value from the time series that date and that concentration.  You do that till you get 12 or 13 points and that's one of your five replicants; is that correct?  Or, 
                  DR. SARAH HAFNER:  Or --
                  DR. KENNETH PORTIER:  Or you take those seven days and you take a random day, and do you snap to seven days.  Which one -- because it's not clear in the White Paper for this experiment exactly which one of those you did.
                  DR. SARAH HAFNER:  We did not use snap for this.  We just did the random start date and did every seven days until we hit 12.  But the next year has a different random start date.
                  DR. KENNETH PORTIER:  And you chose your start date from the beginning of sampling to the midpoint of the season; is that correct?  I mean, it was kind of a wide range of almost two months from which you could draw your starting date from; is that correct?
                  DR. SARAH HAFNER:  Yeah.
                  DR. KENNETH PORTIER:  Yeah, I think it is.  
                  And then what's the justification for five replicates?  Just five.  Why didn't you do 50 or 10?  I just wondered.  
                  DR. ROCHELLE BOHATY:  Yeah, it was time.
                  DR. SARAH HAFNER:  Time, resources.  
                  DR. ROBERT CHAPIN:  Quick question.  Tom.
                  DR. THOMAS POTTER:  Tom Potter.  I heard you say that you used both precipitation and flow as covariates.  Were they used in a combined form or were they used independently?
                  DR. SARAH HAFNER:  They were independent.  
                  DR. THOMAS POTTER:  And do you have separate breakout of how well precipitation performed as opposed to flow?
                  DR. SARAH HAFNER:  Yeah, they're discussed later.  
                  DR. THOMAS POTTER:  Okay.  
                  DR. CLAIRE BAFFAUT:  Claire Baffaut.  You said --
                  DR. ROBERT CHAPIN:  Claire, could you get this a little closer to you please?  Thank you.
                  DR. CLAIRE BAFFAUT:  You said at the beginning of your presentation that the goal was to evaluate the model and you said whether it was simulating the extreme concentration with reasonable accuracy which then you defined as not being underestimated or not grossly overestimated.  Do you have any criteria to decide what's underestimated or overestimated?  
                  DR. SARAH HAFNER:  We discuss that a little bit later in this presentation.  
                  DR. CLAIRE BAFFAUT:  Okay.  I seem to -- 
                  DR. ROBERT CHAPIN:  One last thing and that's it.  
                  MS. TAMUE GIBSON:  I apologize for those members of the public that are joining us online, if you could please mute your lines.  We are getting feedback from discussions or conversations.  So please mute your lines at this time.  Thank you.  
                  DR. ROBERT CHAPIN:  Back to you.
                  DR. XUYANG ZHANG:  Hi.  This is Xuyang Zhang.  I have a question about you subsampling.  
                  DR. SARAH HAFNER:  Right.
                  DR. XUYANG ZHANG:  So you mentioned that the beginning date was always somewhere in the spring, right?  
                  MS. TAMUE GIBSON:  Dr. Zhang, we could not hear you.  Could you please speak and repeat your statement please?
                  DR. XUYANG ZHANG:  Okay.  So can you hear me now?  
                  MS. TAMUE GIBSON:  Just a little bit louder.
                  DR. XUYANG ZHANG:  Okay.  I think I might have reduced the volume of my line.  Yeah, if you could increase my volume.  
                  MS. TAMUE GIBSON:  Yes, go right ahead?
                  DR. XUYANG ZHANG:  Hello?
                  MS. TAMUE GIBSON:  Okay.  
                  DR. XUYANG ZHANG:  So my question is regarding the subsampling.  So the beginning date is always somewhere between January and March in the springtime; is that right?
                  DR. SARAH HAFNER:  Yes.  
                  DR. XUYANG ZHANG:  Okay.  So, even with the 14-day sampling strategy, does that mean that the latest month in the year that can be sampled is somewhere around October -- I mean, September and August.  And the samples that occurred later than that, say October, November, and December were not -- appear in the subsampling.  Is that right?
                  MR. CHARLES PECK:  So this is Chuck Peck.  I'm the one that developed the random sort of sampling routine.  What we designed it to do was take a look at the data that were available, the set that we had, split it in half and sort of pick a start date, a random seed from that and then systematically either do 7 day or 14-day sampling.  So typically in a lot of cases, yes, the March is when it started, but it could have moved anywhere to later in the summertime, April, May, June sort of timeframe at which point, if you were doing a 14-day, it would then shift some of your sampling to later in the calendar year.  
                  DR. XUYANG ZHANG:  Okay.  Thank you.  
                  DR. VERONICA BERROCAL:  Dr. Berrocal.  I have a question about -- so in the model, in the SEAWAVE-QEX, the residuals are modeled to be temporarily correlated and there is that correlation time scale factor.  Did you calculate the correlation times scale factor before deciding on 7 and 14 day sampling strategy?  I'm wondering because if the correlation time scale was small then by doing this 7 or 14-day sampling strategy, you're basically making your data independent when in reality it is temporally correlated.  
                  DR. SARAH HAFNER:  No, we didn't calculate it beforehand.  Okay. 
                  The sampling process resulted in this data structure for each combination of variables.  For example, for one chemical at one site using flow as a covariate, there are five data sets using seven-day stratified sampling.  These five new subsample data sets are then run in SEAWAVE-QEX to estimate daily concentrations resulting in 100 chemographs for each subsample data set.  
                  The SEAWAVE-QEX estimates from the subsample data sets were first briefly evaluated to determine how frequently and how much the model underestimated the measured concentrations in order to address concerns from previous SAPs that SEAWAVE-Q underestimates peak concentrations.  It is important that we have confidence that our drinking water concentration estimates are not underestimating exposure as we are charged with ensuring reasonable certainty of no harm from these concentrations.
                  For each subsample data set, the maximum 1 for 21 and 365-day average was determined across all 100 SEAWAVE-QEX estimated chemographs for each year.  For the one-day average, this would be the maximum concentration that SEAWAVE-QEX estimated in any of the 100 chemographs for a given year in a data set.  These estimated concentrations were then compared against the associated maximum average from the daily measured data set that was not subsampled.  The frequency of underestimation was evaluated by calculating the number of subsampled data sets that underestimated as well as the number of years within each SEAWAVE-QEX run that underestimated.  
                  For example, in this diagram, there are five subsample data sets at a site using a seven-day stratified sampling strategy.  For each subsample data set, there are six years of data which totals 30 years of data across the five subsamples.  Only three of these years underestimated the maximum concentration for that year which was equated to a 10 percent rate of underestimation for those data.  
                  Next, summary statistics were derived from the SEAWAVE-QEX estimates to compare against measured concentrations to determine which might be appropriate to use as a point estimate in a deterministic risk assessment.  A single percentile can't be used since there are 100 SEAWAVE-QEX chemographs for every data set.  Instead, summary statistics were developed by first deriving a percentile for each of the 100 chemographs.  In this example, the 99th percentile concentration was selected for each year of each chemograph.
                  Another summary statistic was taken across those one-hundred 99th percentiles.  In this case, it is the maximum of the 99th percentile values.  These summary statistics were then compared to the maximum measured concentration for each year. 
                  For probabilistic assessment, the entire distribution of concentrations would be used.  To evaluate our ability to use SEAWAVE-QEX in this capacity, we've compared distributions of measured concentrations to all 100 SEAWAVE-QEX chemographs from the five subsample data sets for each chemical, site, and covariate combination; however, the SEAWAVE-QEX distributions needed to be truncated for a fair comparison since the measured data was not completely daily.  
                  The figures here show this process over one year of samples.  First, we started with the full near daily measured data set from which 12 samples were taken for that year shown in yellow.  That subsample data set was interpolated using SEAWAVE-QEX to produce daily chemographs which all included the measured input data.  For the final comparison of distributions, the 12 input measured concentrations from each year are removed and concentrations from the SEAWAVE-QEX chemographs from the same days as the original measured data set are compared.  
                  I should note that SEAWAVE-QEX is designed to capture the statistical properties of the data and not to predict concentrations on a given date.  For example, estimated concentrations from June 5th should not be compared with measured concentration from June 5th; however, for the purposes of thinning data and creating and comparing distributions of concentrations, using the same dates is considered acceptable.
                  And this concludes the methods part of our SEAWAVE-QEX presentation.  And I'll pause now for any clarifying questions.
                  DR. KENNETH PORTIER:  So this is two times series, right?  So how did you compare them?  What's -- did you just take the difference and look at maximum difference or integrated difference or --
                  DR. SARAH HAFNER:  I'll show that in the next presentation, but visual mostly.
                  DR. ROBERT CHAPIN:  Right.  Ray Yang.
                  DR. RAYMOND YANG:  Thank you for your presentation.  My first question for you is really a reflection of my ignorance of this area, okay.  So be patient with me.  Basically, what I would like to do is for you and your colleagues to educate me so I can do a better job, okay.  
                  When I think about a model, I think about either a statistical model which could be a first order or higher order polynomial equation, okay, which has coefficient becomes parameter and so on and so forth.  Or it could be compartmental modeling which in this case, could be ecologically-based compartmental modeling whereby all the factors influencing the flow and exposure rates and so on and so forth would be taken into consideration then each compartment will have parameters and probably mass balance differential equations and then you do modeling, okay.  
                  So far and throughout the White Paper, I don't see anything like that of a model.  Model is a nebulous seeing behind so I don't understand what's going on, all right.  So are we going to have somebody actually talk about a model so we know which factor, which parameter is involved in this process, that process and so on so we can make assessments?  That's question number one.  
                  MR. DANA SPATZ:  So the compartmental model you're describing describes the pesticide and water calculator, the Tier 2 modeling that we do where what Sarah's talking about, the SEAWAVE-QEX model, is the statistical tool.  Today we're not focusing so much on the pesticide and water calculator compartment model.
                  DR. RAYMOND YANG:  So it is a polynomial equation for the model.  Am I correct or --
                  DR. CHRISTINE HARTLESS:  You take the idea of the polynomial regression equation that you're thinking of and add several layers of complication to it and then you get the SEAWAVE-QEX model that's buried inside that R-code.  But yes, it's much more of a statistical model than what you're thinking of in terms of ecological or compartmental models.  
                  DR. RAYMOND YANG:  And do we -- will we discuss that or is there anywhere that you're going to let us see something along that line?  In other words, it seems to me this SEAWAVE now is something in the background.  Trust me, it is working.  
                  DR. ROBERT CHAPIN:  Well, you're going to see some examples, right.  We're going to see some case studies in a while.  Do you want to see the code?
                  DR. RAYMOND YANG:  Well, case study -- I looked through the case study.  I didn't see anything either.  
                  DR. ROBERT CHAPIN:  See the code, is that what you want?
                  DR. RAYMOND YANG:  Well, the code is derived from the model.  Model is representing a concept.  
                  DR. ROBERT CHAPIN:  It makes the model.
                  DR. VERONICA BERROCAL:  There is a document with actually the model that -- yes.
                  DR. RAYMOND YANG:  Okay.  Thank you.  Thank you.
                  DR. VERONICA BERROCAL:  There is a document with the model, the Vecchia 2018 USGS paper as the model, the equation and -- 
                  DR. RAYMOND YANG:  Okay, okay.
                  DR. VERONICA BERROCAL:  Yes.
                  DR. RAYMOND YANG:  Now let me ask you a less troublesome question, okay.  In your presentation, there was a slide that says SEAWAVE-Q, the earlier panel considered the estimate was too low.  I assume you folks have updated and correct that area and so on with QEX, all right?  And I look at some of the graphics.  They look beautiful, very nice.  You know, you -- 
                  DR. ROBERT CHAPIN:  What's your question, Ray?
                  DR. RAYMOND YANG:  The question, the question is Crop Life America.  This public written statement.  They say EPA's estimate mostly too high.  So what's going on?  Are we talking about differences of opinion or different interpretation or what?  
                  DR. ROCHELLE BOHATY:  So Rochelle Bohaty here.  I'll take that question.  
                  So I think it's probably a little bit of both, but I think to help better inform this discussion I think Sarah's presentation on the results and how we interpreted the data would be useful.  
                  DR. RAYMOND YANG:  All right.  
                  DR. VERONICA BERROCAL:  Yes.  I have a clarifying question on the statement that you made just at the end of your presentation.  You said that the SEAWAVE-QEX model is not the -- should not be used to generate estimates of pesticide concentration for a given day.  I'm just confused about this statement because the model is set up where your response variable is the daily concentration and the covariate is again something that refers to the day, whether it is the streamflow or precipitation.  So it seems to me -- and then the results of the fact that there is -- it was recounted for temporal correlation.  So I don't understand why the output or the estimated concentrated of the SEAWAVE-QEX model gives for a given day cannot be interpreted as what is the estimated concentration on that day.  Can you clarify why you made that statement?
                  DR. SARAH HAFNER:  That's how it's been explained to us by the model developer, that it was not designed to predict a specific concentration for a date, but to encompass the properties so it can give these distributions of concentrations.  So it's -- you know, you're getting equally probable chemographs for each date, but it's not designed to give you a concentration for a specific date to compare it to measured concentrations.
                  DR. ROBERT CHAPIN:  Yeah, just get close to the mic please.
                  DR. IAN KENNEDY:  So correct me if I -- 
                  DR. ROBERT CHAPIN:  And identify yourself for folks.
                  DR. IAN KENNEDY:  Oh, Ian Kennedy.
                  DR. ROBERT CHAPIN:  Thank you.
                  DR. IAN KENNEDY:  Right.  What you do basically is you take the input data, you run the SEAWAVE-QEX, and it gives you some parameters for model and then does generally 100 possible outcomes of that or 100 chemographs for each year of the run.  And then, according to what you have a couple of slides before, you're taking the 99th percentile of each of those runs and taking the maximum of that; is that correct?
                  DR. ROBERT CHAPIN:  So is there a question there?
                  DR. IAN KENNEDY:  My question is if that's what's going on, how repeatable is that?  If I did it with another hundred runs, because you've got some random number generator in there somewhere that is generating the differences between these hundred runs and so you're sampling from distribution and then looking at the maximum of those samples.  I'm just wondering if there's some issue with repeatability?
                  DR. SARAH HAFNER:  So generally if you take the same data set and run it again, you will get different chemographs, but it should be fit to the same model.  So you may not get -- you won't get exactly the same number when you develop this summary statistic, but it should be very similar.  
                  DR. IAN KENNEDY:  How does that work in the regulatory context then if you generate one number and someone else can do exactly the same thing and generate another number, even if the numbers are reasonably close?
                  DR. VERONICA BERROCAL:  Can I try and explain here?  So when the model is fit, the standard deviation of this residual is estimated.  And so, every simulation that is done is basically, yes, simulating random numbers, but it's bounded so that these random numbers have the standard deviation that is estimated from the data.  
                  So they are not -- they are taking this percentile and basically characterizing the distribution.  It's not the actual value.  So even if you repeat this experiment many times, it's true that the residuals will not be the same, but the standard deviation of these residuals will be the same because the data that you're feeding to the -- that you're using to feed the model is fixed.  It doesn't change.  
                  DR. IAN KENNEDY:  Right.  I think my question is more one of -- is that a sufficient number to be sure that those differences are small enough that you won't get some issue when someone else does it and gets a different value?  
                  DR. VERONICA BERROCAL:  Well, then the question may be would be is 100 chemographs enough.  Or to make sure that this percentile is estimated accurately versus should you do 1,000 chemographs or 10,0000 chemographs?  So, here you're going into how to quantify uncertainty in the percentile of a distribution; but I don't think that the procedure in itself is invalid.  This is what we're doing statistically, Monte Carlo simulations.
                  DR. IAN KENNEDY:  Yeah.
                  DR. ROBERT CHAPIN:  Okay.  Dr. Sadd.
                  DR. JAMES SADD:  Jim Sadd.  A couple of slides back you showed some measured concentrations as points and then the estimated chemograph.  And the chemograph -- a little farther back please.  One more maybe, one more.  Yes, right there.
                  And the chemograph actually shows concentrations that are higher than the measured samples.  And I understand that you can't measure the peak, so that is something that's estimated from the covariates.  So if you have a covariate, say it's precipitation, in an arid climate that it's contribution to a waterbody or to a river is going to be very much influenced by infiltration and that could vary seasonally because of, you know, soil moisture and soil dryness.  And in a humid climate, it's going to be contribution to the river from groundwater flow and there's going to be a lag there too.  So are those sorts of covariates included in consideration of precipitation?  What I'm trying to say is it's not just the rainfall, it's how the water actually gets into the waterbody and that is complicated by those other factors.
                  DR. ROCHELLE BOHATY:  So I think the best answer is that we're exploring for -- we would explore for different sites the best covariate in that work and you'll hear some more in some of the subsequent presentations, particularly by Mr. Bischof on how he looked at other covariates because his sites didn't work so well using flow as a predictor for the concentrations. 
                  DR. THOMAS POTTER:  Tom Potter here.  I need to use -- ask my same question earlier and you can -- maybe you're going to have the same answer which is I'm going to hear about it later.  But my question is, you did a systematic evaluation of the two covariates, precipitation and flow.  That's what I understand to be the case and so are we going to hear that later, or is this embedded in this analysis here?
                  DR. SARAH HAFNER:  You will hear about precipitation in this -- the results part of this.  
                  DR. THOMAS POTTER:  This -- your current presentation?
                  DR. SARAH HAFNER:  Presentation. 
                  DR. THOMAS POTTER:  Okay, sorry.  I just needed to get -- 
                  DR. SARAH HAFNER:  It's still, yeah.
                  DR. THOMAS POTTER:  I needed to be reassured.  Thank you.
                  DR. ROBERT CHAPIN:  Okay.  We good?  Any other -- 
                  DR. REBECCA KLAPER:  I had a couple questions.  This is Rebecca Klaper.  Dr. Hafner, are there -- so you guys were talking about the Tier 2 -- 
                  DR. ROBERT CHAPIN:  Rebecca, I'm sorry.  Could you move the mic so it's -- thank you.
                  DR. REBECCA KLAPER:  Better?  You were talking about the Tier 2 evaluation and all of the considerations you make in your model looking at potential pesticide affects including all the properties of the chemicals.  Do you -- have you -- I know you've given us two examples, and I'm not totally clear what those are because they're mystery chemicals, but does it make a difference or have you evaluated other chemicals based on those chemical properties to see how well the models fit based on the properties of the pesticide whether it's more water soluble or not or whether it absorbs to soil or not or those kinds of things?  That was my first question.  
                  DR. ROCHELLE BOHATY:  So I think the extent that we have done that is when we use the USGS chemicals and we ran them through SEAWAVE-QEX and evaluated the output plots for that.  We won't specifically get into the results of that, even though it's provided, but we do use that data in the subsequent development of the sampling bias factors.  
                  DR. REBECCA KLAPER:  So does it --
                  DR. ROCHELLE BOHATY:  So the data are available in the supplemental package that was provided.
                  DR. REBECCA KLAPER:  Okay.  So it does make some difference and you've integrated that into whatever the cofactors are for the model in order to make better estimations then?
                  DR. ROCHELLE BOHATY:  So for SEAWAVE-QEX, you don't actually put in any independent environmental fate data.  It's just simply flow and the measured pesticide concentrations.  The environmental fate parameters that you're talking about, those would be in our Tier 2 model which is the pesticide water calculator so they're completely independent.  
                  DR. REBECCA KLAPER:  Okay.  But no matter which of those chemicals you chose -- do all those pesticides, and I'm sorry I don't know the KOD values or whatever of all the pesticides, are they all very similar?  So they would all behave the same way in the -- 
                  DR. ROCHELLE BOHATY:  No, they're very different.
                  DR. REBECCA KLAPER:  -- environmentals And they still all held up equally in the same well system?
                  DR. ROCHELLE BOHATY:  I mean, you could run the tools, sure, depending on the site and the pesticide combination, we would have more confidence in the results based on the individual runs and diagnostic plots.  But we thought overall that the model performed well for what we needed it to do.  
                  DR. REBECCA KLAPER:  Okay.  Then I just have one more question too.  How are you -- and maybe you're getting into this a little bit later -- but how are you dealing with the change of the flow rate over a day?  It is integrated or is it one median value of the flow for that particular day or -- because, you know, especially in some of the smaller order streams they're very flashy, et cetera.  So how do you deal with that flashiness?  
                  MR. MATTHEW BISCHOF:  It's just one value for the day.  
                  DR. REBECCA KLAPER:  So it's like an average or a median or something or --
                  MR. MATTHEW BISCHOF:  Yeah, I believe it's average for the day.
                  DR. REBECCA KLAPER:  Okay.  Because the other part of that question that I have then too is that, you know, when we go -- I do environmental sampling too -- when you go out, you can't go sample at the high point of a stream or when the first flush happens coming off of a field and so you're missing the highest points of the data.  What kind of confidence then do you have that you're estimating the highest concentration when most environmental samples are missing that high point of a pesticide concentration in a stream?  Do you set some kind of variability or confidence interval so that you can say, you know, 10 percent more et cetera in order to make sure that you're safe basically?  Thank you.
                  DR. ROCHELLE BOHATY:  No, we just take the data as they are.  
                  DR. ANDREW MIGLINO:  Hi.  Andrew Miglino.  I just had a couple questions, hopefully easy questions, and maybe they're in Vecchia and I just didn't delve too far into it.  So sorry if it's explained somewhere else.  But these are about the fitted parameters.  So some of these are static, so like, season and rate are the same across years.  Is that a reasonable assumption, and maybe this is just ignorance on the pesticide world, but, is that a reasonable assumption to assume I apply the same timeframe at the same rate year after year for all these simulations?
                  DR. ROCHELLE BOHATY:  So Rochelle Bohaty.  It depends on the chemical so that's important to note.  One of the utilities of SEAWAVE is that you can, if you notice a change in the use profile over time, you could split up the years --
                  DR. ANDREW MIGLINO:  Okay.
                  DR. ROCHELLE BOHATY:  -- and run them, as long as it still met the data quality and quantity criteria.  
                  DR. ANDREW MIGLINO:  Sure, so like a 12-year sampling split into four years each -- 
                  DR. ROCHELLE BOHATY:  Yeah.
                  DR. ANDREW MIGLINO:  -- or something like that.  Okay.  
                  And then the other one is about censored data.  So I noted in the White Paper you talk about SEAWAVE gives a random value to censored data.  Is that a static fixed value across all hundred realizations, or does it give it a new value for each one of those?
                  DR. SARAH HAFNER:  It's a new value for each realization.
                  DR. ANDREW MIGLINO:  Okay.  And is there like a bounding on that?  I mean, obviously, the upper bound, but is there a lower bound on there?
                  DR. SARAH HAFNER:  It is somehow constrained to the model, but I couldn't tell you the specifics at this moment.
                  DR. ANDREW MIGLINO:  I'll go delving into Vecchia.  Okay.  Thank you.  
                  DR. ANNA LOWIT:  Anna Lowit.  Just a couple of things from a few minutes ago.  
                  So Dr. Yang, when you asked us about the opinion of Crop Life, I think that's a question for them and I don't think we'll cover at any point what Crop Life is going to think.  So I just -- make sure he that wasn't expecting use to comment on their comments.
                  And as a federal regulator, it makes sense to start asking implementation questions, like, how are going to use it?  The number is going to change ever time, you know.  There are a lot of policy and implementation questions that become obvious when you think about this, but understand that where we are right this second is that we've been working with the developers of SEAWAVE-QEX and the thought of moving more fruitfully towards sampling bias factors and so we're really in the methods development with this evaluation and how that would actually in a real assessment through public comment and to do a risk mitigation decision using these tools is really not part of what we're talking about today, although it's obvious in your mind of how it would actually work in process that, you know, a lot of the conversations about the, you know, the way the model is working and the number of simulations.  These are the kind of recommendations we're really looking for, but, you know, we're not going to have a lot of answers on how you're going to use it and what's it going to look like in the risk assessment because, you know, I think we're still working through that, you know, if we're honest.
                  DR. VERONICA BERROCAL:  Yes.  Dr. Berrocal.  So, I have a question about the chemograph.  I understand that the model is fit to the data, the parameters are estimated so those regression coefficients and then the standard deviation of the residuals and the correlation approximation.  
                  And this simulated condition of simulation are generated by simulating the residuals, right?  But there is uncertainty in the parameters of the models.  Do you generate also random values for these parameters using the, you know, distribution that you get when you do maximal likelihood; when you estimate the parameter there is some standard data attached to it, or you just keep them fixed and you just simulate the residuals?  Am I being too technical?
                  DR. CHRISTINE HARTLESS:  This is Christine Hartless and no, we're not doing any kind of a nested Monte Carlo in going back and evaluating the distributions and simulating from the distributions of the estimated parameters within SEAWAVE.
                  And I also want to add, further clarification, in that when we're evaluating the SEAWAVE and you fit the data from one particular site using the SEAWAVE-QEX model, and this component is included in the SOP and there's also additional information in the Vecchia paper about the steps that we go through and it is recommended to go through to evaluate the fit of the model and determine whether or not these data are appropriate for using SEAWAVE-QEX.  There are some monitoring sites where we decide these data just don't fit and we're not going to be able to utilize SEAWAVE-QEX and there's some additional tools and guidance provided in those.  And I don't think that any of that was incorporated in today's presentations because it's getting very down into the weeds and we wanted to try to present to you all the bigger picture of what we're doing.  But that information is included in all of the affiliated SAP documentation.  
                  DR. ROBERT CHAPIN:  Okay.  I think we're kind of going to be done with questions for now, and we'll turn it back over to Dr. Hafner to lead us through the second part of her business here.
                  
EVALUATION OF SEAWAVE-QEX AS AN IMPUTATION TECHNIQUE FOR ESTIMATING DAILY PESTICIDE CONCENTRATIONS FROM PESTICIDE MONITORING DATA PART 2
                  
                  DR. SARAH HAFNER:  Okay.  So I'll now continue with part two of the SEAWAVE-QEX evaluation which is the result. 
                  So this is the outline for the second part.  First, we will go over the results for the accuracy evaluation and next discuss the alternative covariate evaluation and the use of SEAWAVE-QEX for alternative waterbodies.  We'll end with a summary of the results, discussion of the challenges the EPA faces in adopting SEAWAVE-QEX, and EPA's proposed approach for using the model.
                  So as described earlier in the methods of data preparation, there were three parts to the accuracy evaluation of SEAWAVE-QEX.  First, to determine how frequently concentrations are underestimated by the model compared to the measured concentrations.  Second, determine summary statistics of estimated concentrations that correspond with measured concentrations.  And third, plot the distributions of estimated concentrations with the measured concentrations.
                  First, the underestimation rate was viewed by combining replicates and determining the number of years that SEAWAVE-QEX underestimated the measured maximum for each year.  For this presentation, I'm going to walk through a few comparisons of the data showing the underestimation rate for the 1-day and 365-day averages at two sites, Honey Creek and Maumee River.
                  Although the 4-day and 21-day averages are also presented in the White Paper as well as data for Rock Creek and Sandusky River sites, the data are truncated for the purposes of this presentation.  
                  The first comparison is between using streamflow and precipitation as covariates.  The 1-day average is shown in blue and the 365-day average is shown in black with streamflow on the left in solid and precipitation on the right in cross-hatch.  This slide is showing metolachlor results.
                  When a bar appears to be missing, as is often the case with precipitation here on the right, it means that the underestimation rate is zero percent or that SEAWAVE-QEX never underestimated these values.  This data suggests that SEAWAVE-QEX is underestimating somewhat more frequently with streamflow than precipitation for this set of data although the results are more similar between the two covariates for atrazine as is shown on this line, where the rate of underestimation for streamflow on the left is similar to that for precipitation on the right. 
                  The next comparison I'm showing is the 1-day against the 365-day averages with flow as covariate for atrazine on the left and metolachlor on the right.  As before, the one-day average is in blue and 365 is in black.  
                  A broad trend can be seen in the data that can be seen through these figures that the 365-day average seems to be underestimating more frequently than the other averaging periods which is seen by the black bars being larger than the blue bars more frequently; however, this may be an artifact of the interpolation method that was used to derive a 365-day average for the measured data since the data was robust but not completely daily year-round.  
                  While the 365-day average underestimated more frequently than other average periods, the magnitude of underestimation was at most 50 percent or half the concentration of the measured maximum.  This is similar to the magnitude of underestimation that was observed for the 4-day and 21-day averages.  When the daily average was underestimated, the estimated concentrations were above 20 percent of the measured maximum.
                  The final comparison here is between the 7-day stratified sampling on the left and the 14-day stratified sampling on the right shown here for metolachlor.  The 14-day stratified sampling strategy is somewhat less likely to underestimate than the 7-day which may be related to the 14-day sampling strategy representing a greater part of the year since both strategies were capped at 12 samples per year regardless of the sampling interval, meaning that the 7-day sampling strategy would span 12 weeks each year whereas the 14-day sampling strategy spanned 24 weeks in a year.  It is important to note that this difference would not be expected if the sampling windows were a fixed length rather than fixing sample number.
                  So viewing all the data combined broadly, the rate of underestimation was less than 30 percent for all average periods for both atrazine and metolachlor across sites using either flow or precipitation as a covariate.  With few exceptions, the rate was less than 15 percent.  This suggests that SEAWAVE-QEX is capable of capturing peak concentrations most of the time.  The magnitude of overestimation is not considered at this point as overestimation is considered more easily corrected than underestimation.  
                  The next part of the accuracy evaluation was to overlay summary statistics from SEAWAVE-QEX estimates with maximum concentrations from the measured data.  The following slides will be showing an example of the measured maximum concentrations for each year compared to the summary statistic described earlier.
                  DR. LISA NOWELL:  This is Lisa Nowell.  Sorry to backtrack just a little bit.  I wanted to ask for clarification of something I think you said.  Correct me if I'm misstating this.  It had to do with -- I think you said that the underestimation was worse for the 365-day average than the 1-day and that that may be an artifact of the method you used to interpolate your measure data.  Is that what you said?
                  DR. SARAH HAFNER:  Yes.
                  DR. LISA NOWELL:  And would you mind clarifying that?
                  DR. SARAH HAFNER:  Yes.  So if you -- well, so for the data that we used for this, it's daily sampling in the summer months, but in the off-season it's biweekly sampling.  So for the 1-day, 4-day and 21-day averages, the maximum for year is generally in the summer during the peak season and so that's calculated mostly from daily sampling.  But to calculate the 365-day average, we did have to do some infilling.  So we did log linear interpolation for the 365-day measured concentration that we're comparing SEAWAVE against.  
                  Does that answer your question?
                  DR. LISA NOWELL:  Thank you.
                  DR. SARAH HAFNER:  Okay.
                  DR. THOMAS POTTER:  Oh, question over here.  
                  DR. SARAH HAFNER:  Where?
                  DR. THOMAS POTTER:  Tom Potter.  I want to go back.  I don't really know a whole lot about the particular watershed you're looking at.  I assume they're quite runoff prone since precipitation is seemingly performing equally to flow so there must be a reasonably strong relationship there.  Is there data out there that describes what the hydrologic characteristics are of these watersheds are and we can delve into that in terms of seeing why this relationship exists here and whether it could be extended elsewhere?
                  DR. SARAH HAFNER:  There is a lot of information out there on these sites.  The Heidelberg NCWQR.
                  DR. THOMAS POTTER:  Yeah, I'm aware of where they're coming from and, you know, I guess part of it is let me ask a question.  Did you look in some detail here to kind of do some ground truthing here as to why you're seeing this, what appears to be a very good relationship with precipitation, which I have to say, I remain a skeptic.  So --
                  DR. SARAH HAFNER:  No, we didn't.  
                  DR. THOMAS POTTER:  Okay. 
                  DR. SARAH HAFNER:  We just used the best available data.  
                  DR. THOMAS POTTER:  And we'll get into this later on, but, you know, best available data for rainfall is really, you know, I have to say, a can of worms.
                  DR. SARAH HAFNER:  Oh, I meant pesticide data.
                  DR. THOMAS POTTER:  Yeah.
                  DR. ROBERT CHAPIN:  Okay.  Here we go.  Dr. Hafner.
                  DR. SARAH HAFNER:  Okay.  So in the next slides I'll be walking through an example of the measured maximum concentration for each year compared to the summary statistic that was described earlier at the maximum of the 99th percentile for all subsample data sets.  A model that estimates concentrations well should neither consistently over or underestimate measured concentrations and so the most reasonable estimate of the true concentration could be viewed as having the most balance between over and underestimation of the measured annual maximum across replicate data sets with few outliers.  This is demonstrated for the maximum of the 99th percentile concentrations and generally observed in the data for other sites covariates in both chemicals. 
                  So I'm going to briefly walk through an example figure that demonstrates this concept.  In this figure, concentrations on the Y-axis and years of the simulation on the X-axis.  The yellow bars are the maximum measured concentration for each year.  Each blue circle within each year represents a different seven-day stratified subsample data set based on the subsample process described earlier so that there are five total seven-day subsample data sets overlaid for each year.  The same is true for the 14-day stratified subsamples shown as green diamonds.  
                  When the annual summary statistic for all of the subsample data sets are overlaid on the maximum measured concentrations, there is not consistent over or underestimation of the measured maximum as there are estimated concentrations both above and below the yellow lines.
                  It should be noted that this approach looks broadly at the patterns of estimated versus measured concentrations for many subsamples in the same data set and that a risk assessor would only get one data set from a site.  And so in some cases using this approach, a single data set may miss the peak concentration.  
                  Similar figures were developed for rolling averages; that is, 4-day, 21-day, and 365-day averages.  These are presented in the White Paper. 
                  The last part of the accuracy evaluation was comparing the full distribution of raw SEAWAVE-QEX estimates to the measured concentrations.  This figure shows an example distribution for metolachlor at Maumee River.  The orange-colored band is made up of 100 distributions of SEAWAVE-QEX estimates from one of the subsample data sets.  The black line overlaid on top of the colored band is the single distribution of the measured concentrations.  
                  Estimated distributions that fall to the left of the measured concentrations are underestimated and those to the right are overestimated relative to the measured.  To walk through an example, when looking at the one microgram per liter concentration on the X-axis, this corresponds to the 70th percentile on the distribution for the measured concentrations.  This indicates that 70 percent of the concentrations in the original measured data set were less than one microgram per liter.
                  Next, look at one of the SEAWAVE-QEX estimated distributions that falls to the right of the measured distribution and you can see that these cross the one microgram per liter line below the 70th percentile line.  For example, the distribution indicated here with an arrow is at the 60th percentile indicating that only 60 percent of those estimated concentrations were less than one microgram per liter and so overestimating concentrations relative to the measured.  
                  Conversely, for a distribution to the left of the measure, the one microgram per liter line crosses above the 70th percentile meaning that more than 70 percent of the estimated concentrations are less than one microgram per liter and so overall, the concentrations are underestimated relative to the measured.  
                  For the distribution analysis, all five subsample data sets for each chemical, site, and covariate combination were overlaid with the measured data distribution.  This is shown here with each colored band representing one of the five subsample data sets in each subsample data set representing distributions from 100 SEAWAVE-QEX chemographs.  Generally, the distributions matched well with the measured concentrations across chemical sites and covariates.  Often when there was variability between the five subsample data sets, it occurred at low concentrations such as below the limit of detection.  
                  Both flow and precipitation were found to be suitable covariates.  Occasionally, there was some extreme tailing in the distribution as seen here.  Both flow and precipitation were subject to tailing on some distributions though using precipitation tended cause a few more extreme concentrations.
                  In addition to the more extensive quantitation comparison of flow and model precipitation, the diagnostic plots were evaluated when using various other covariates available.  The two plots that were evaluated in particular relate to mid-term and short-term flow anomalies to the estimated concentrations in SEAWAVE-QEX.  Note that the concentration on the Y-axis is in log base 10-scale and the flow anomaly on the X-axis is a unit-less value.
                  We compared the similarity of these plots created when using flow as a covariate to those when stream stage or precipitation was used as a covariate.  For model precipitation, which was a more extensive comparison because of the number of data inputs, we found that both flow anomalies behaved similarly as with streamflow.  This supports the quantitative evaluation that model precipitation is a viable covariate.  
                  DR. RAYMOND YANG:  Ray Yang.  What are the open circle and solid circle?
                  DR. SARAH HAFNER:  The closed circles represent the measured data inputs to SEAWAVE-QEX and the open circles are still the censored data points.
                  DR. RAYMOND YANG:  The open circle is what?
                  DR. SARAH HAFNER:  The censored data that SEAWAVE is -- 
                  DR. RAYMOND YANG:  Censored data.  An the model is the straight line?
                  DR. SARAH HAFNER:  That's just a regression line.  
                  DR. KENNETH PORTIER:  In the material I gave you -- this is Ken Portier, Dr. Yang.  In the material I gave you, so that the slope of that line is the Beta 2 or Beta 3 term in that regression.  That's all they're showing here is that part of the decomposition of the whole SEAWAVE model.
                  DR. RAYMOND YANG:  All right.  
                  DR. SARAH HAFNER:  Stream stage and measured precipitation were also evaluated, though on fewer data sets.  Stream stage matched with streamflow very well which is expected as streamflow is often correlated with stage measurements.  Measured precipitation was also similar to flow in this evaluation, although deriving the data is much more difficult than monitoring stage or estimating precipitation.
                  The use of SEAWAVE-QEX on waterbodies with intermittent flow or no flow was assessed with a few data sets from the atrazine monitoring program or AMP data.  These sites are of particular interest to use since water monitoring in reservoirs would be highly relevant to drinking water assessments.  
                  For this assessment, the diagnostic plots were evaluated for an appropriate model fit.  Despite the lack of seasonality in some of the data sets, SEAWAVE-QEX diagnostic plots were reasonable for most of the sites.  Sites that did not have reasonable plots typically had an underlying uncertainty in the data that might explain poor fits.  This is primarily because the samples were collected in such a way that the source water watershed is not always known as several sampling sites have multiple intakes from which treatment plants may draw at a given time.
                  It is unclear at times from viewing the data if a sudden change in concentration is real or if the source is switched which can affect the model results; however, for some sites, it's clear that SEAWAVE-QEX fits the data well and could be used to estimate concentration on these types of waterbodies.
                  Overall, throughout our analysis, SEAWAVE-QEX proved to be a robust tool with minimal data inputs.  The model does not frequently underestimate the measured maximum in each year and the magnitude of underestimation is low for most averaging periods.  Our method of selecting summary statistics from the SEAWAVE-QEX chemographs shows promise for estimating the true concentration, although when choosing a summary statistic, there is a tradeoff in the frequency of overestimating versus capturing the peak in every data set.  The distributions also matched most of the measured data while with some tendency to overestimate at the tails, particularly with modeled precipitation.  
                  As for choosing a covariate, streamflow is always preferable if it is appropriate.  Cases when streamflow would not be appropriate may include waterbodies with periods of stagnation or backflow.  Stream stage would be the next best alternative covariate followed by measured precipitation and model precipitation.  
                  While SEAWAVE-QEX can be used on waterbodies with irregular flow patterns, this is less straightforward than flowing systems and in systems with higher frequency sampling with low or no flow, there may not be much added value to using the model to produce daily concentrations if there is high correlation between sampling events, for example, with persistent pesticides in low flowing systems.  
                  While SEAWAVE-QEX is a powerful tool, there are several challenges to adopting it, particularly in a regulatory setting.  SEAWAVE-QEX requires more pesticide data than is available for most monitoring sites needing three or more years of data from individual sites with roughly 12 or more samples per year at 30 percent detection rate.  Additionally, while SEAWAVE-QEX is simple to run, there's a steep learning curve to analyzing the diagnostic plots and optimizing the input parameters which leads to subjectivity and its use for risk assessment.
                  SEAWAVE-QEX was also designed for use with specific types of data such as flowing systems and daily grab samples which is not representative of all water monitoring sources available for inclusion in drinking water assessments.  Monitoring databases include a range of waterbody types including flowing and non-flowing sources such as reservoirs which are highly relevant for drinking water, and a range of waterbody flashiness.  Monitoring programs also vary in sample collection methods and complimentary data such as the availability of streamflow data.  
                  A large obstacle to consider when utilizing any monitoring data is the time and resources allocated to cleaning up the data.  This includes filtering and formatting large data sets without excluding valuable data by filtering too broadly or including inappropriate data such as non-surface water samples, passive samples, duplicate data sets that are reported by different agencies, and data with incorrect units.  This data processing is made more challenging when trying to determine if a data set can meet the minimum data requirements for SEAWAVE-QEX since there is flexibility in these requirements.  
                  There's additional time used for processing or deriving covariate data before actually running the model and ultimately, time is needed to describe the site relevance based on the vulnerability of the site to a pesticide of interest.
                  Because SEAWAVE-QEX would be a valuable tool to have available for use in drinking water assessments, we've taken several steps to address the challenges that we expect to most hinder our use of the model.  First, to address the challenge of subjectivity in using the model, we've developed a standard operating procedure for running SEAWAVE-QEX and interpreting the diagnostic plots.
                  We have also developed a weight of evidence approach for determining site relevance for use in drinking water assessments which will be discussed in a later presentation by Christine Hartless.  
                  In order to maximize data that could be used in SEAWAVE-QEX, the model was only evaluated on flowing systems with streamflow as a covariate, but also using alternative covariates to streamflow including precipitation and stream stage.  The model was also tested on systems with irregular flow including low flow and non-flowing systems.  
                  Our use of the model showed that although SEAWAVE-QEX requires some effort to use, it can be adapted to the risk assessment fairly easily with summary statistics from daily concentrations used in deterministic assessments and distributions used in probabilistic assessments.  Additionally, surrogate covariates can be used when streamflow is not available for historical data or when streamflow is not appropriate.
                  Overall, due to the level of effort and spatial resolution require, SEAWAVE-QEX is considered a tool for use in Tier 4 drinking water assessments for pesticides needing the highest level of refinement.
                  And I can now take any more clarifying questions.
                  DR. CLIFFORD WEISEL:  This is Cliff Weisel.  I have two that I want to pose.  One is I thought I heard you say that was easy to deal with when you overestimate rather than underestimate.  Is that easier to deal with from a perspective that you care less about those or is it easier to deal with mathematically? 
                  DR. SARAH HAFNER:  More mathematically.
                  DR. CLIFFORD WEISEL:  Why is it easier to deal with?
                  DR. SARAH HAFNER:  By considering the summary statistic, it's easier to choose a different percentile, but you can't get higher than the highest concentration that you have.
                  DR. CLIFFORD WEISEL:  Okay.  The other question I have has to do with you're talking about when you have multiple sources and how that might affect the data.  All right, so this is a -- I'm beginning to understand what's going on as a statistical fit of your measurement data trying to incorporate something else that would help you evaluate that.  So if you do have multiple sources, that's a real situation, and I don't think it's a rare situation.  So what do you think might be the approach?  Would more monitoring data be useful?  Would it be data that helps understand why you have multiple sources and what might you need to do to try to get at that better?
                  DR. ROCHELLE BOHATY:  So that particular comment comes up for the monitoring data set that was relevant to community water systems.  So it's not uncommon for community water systems to have multiple intakes.  And so from my understanding, that data set, the sample is taken at like the point of entry for the plant.  And so maybe for a better use of our tool would be to sample at the intake location so that that way you know exactly where the data were coming from.
                  DR. CLIFFORD WEISEL:  Yeah, but many streams -- I mean, you're looking at surface water in general as opposed to the surface water at the plant, intakes for plants, or is the latter really what you want to think about more?  Because they're not the same and if you're trying to -- the way this was presented was to look at surface water and then as a secondary situation when you reach the point that there may be a problem, okay, let's also see is it really impacting what's going through the drinking water system?  So those are slightly different questions.
                  DR. ROCHELLE BOHATY:  So we pulled in that extra data source because a lot of times the questions that we get relevant to the other sources is what is the relevancy to community water systems.  So that data set is specific for community water systems.  So we started looked at that data set.  We looked at the flowing.  It seemed like they worked well for those flowing systems and then we moved on to looking at those, like non-flowing systems and low-flow.  
                  And so you're right.  When the model may not be appropriate and likely isn't appropriate for use if we are getting measured values at the intake location in terms of fitting the model, but the relevancy to drinking water would remain for those measured concentrations.
                  DR. TIMOTHY GREEN:  I was essentially going to ask a similar question, but a follow-up --
                  DR. ROBERT CHAPIN:  Your name, Tim, your name.
                  DR. TIMOTHY GREEN:  Tim Green, sorry.  Many drinking water sources are coming from reservoirs or low-flow and non-flowing sources.  So to what extent have you tested -- you've shown cases of primarily rivers and streams.  What's happening with reservoirs and lakes, and what is your confidence in SEAWAVE, I guess?
                  DR. SARAH HAFNER:  So that data set, the AMP data, is the only one that we've used for the non-flowing system.  And unfortunately, it does have those issues where the samples were taken at the treatment plant so there is some uncertainty in that data.  But for the ones that seem to work, and it doesn't have this concentration, or it doesn't look like the source is being switched, the model does seem to work, but it's a very flat wave.  You can look in the supplemental data to look at it if you'd like.  So it almost seems that you may not, at least for the frequency of that sampling -- for that program, it's weekly sampling -- that the tool may not, for a persistent chemical, be as relevant.
                  DR. TIMOTHY GREEN:  Thank you.  So is there, going forward, a strategy that differs between flowing and non-flowing if we call them that?
                  DR. SARAH HAFNER:  I believe we have charge question asking if you guys know of any more tools.  
                  DR. TIMOTHY GREEN:  Well, that's why I'm asking the question is to understand your direction and what your plans are before we make up input to you.  
                  DR. VERONICA BERROCAL:  Dr. Berrocal here.  So I wanted to understand a little bit more how SEAWAVE-QEX enters into this Tier 4.  So my understanding is that you have a site, you construct this chemograph, and you get your summary statistics.  You're going to do this for several sites where the pesticide data is good enough or it fits the criteria for using SEAWAVE-QEX and then how does this then move on onto the Tier 4?   I guess I want to know what is the long-term goal here.  I understand what happens side by side.  I just don't understand how that then gets all together to make a decision about the pesticide itself.
                  DR. SARAH HAFNER:  If you could hold on a little bit, we will talk about that in the case studies.  There's an example of how the SEAWAVE sites would be used in a drinking water assessment.  
                  DR. ROBERT CHAPIN:  Terry.
                  MR. TERRY COUNCELL:  All right.  Terry Councell here.  So the two examples that you gave are atrazine and metolachlor you used.  From the data sets that you got that data from, those both compounds have significant metabolites, oxalinic acids or ethane sulfonic acids and all that.  So did you use just the parent compound?  Did you stoichiometrically add the metabolites back in or how did you deal with that?
                  DR. SARAH HAFNER:  We just used the parent compounds for this.
                  DR. ROBERT CHAPIN:  Cliff.
                  DR. CLIFFORD WEISEL:  Cliff Weisel.  This might be too broad right now, but what's happening with our water stream systems is they're being more and more controlled as to when things are being released intentionally in some places which will change our streamflow rate quite a bit.  So I have two edges to that question.  One is, is that being considered and how do you consider that?  And the second edge is reverse from what you're predicting for certain areas, do you think this might be a tool for helping advise how that might be done in the future?
                  DR. SARAH HAFNER:  Well, I can answer your first question.  Mr. Bischof will be talking about some of his sites later today that have some of this issue with streamflow and that's where he was able to use measured precipitation as a covariate for those sites that worked for him.  
                  As for if this could be used --
                  DR. CLIFFORD WEISEL:  It's just a thought.  Looking at how we control our water systems in the future so I'm just putting this out as a thought.  If this really works, that might be another thing to be using, to worry about, because this is really trying to predict what is - the measured value and as Dr. Klaper pointed out, that's not always the real value that's there, but that's what we have.  But that's getting more and more tenuous as people try to control water systems.
                  DR. REBECCA KLAPER:  Rebecca Klaper.  I just had one question about variability too.  I understand -- maybe you can get around this with -- by considering the order of the stream, but with the climate change issues happening now where we have differences in variability of rainstorms and flow rates, is there an evaluation of when you have -- as it's gone through time.  I guess you only have like four years there, but, you know, especially over the last five, there's been a lot more variability in water flow.  How does the model handle when you have more consistent versus a variable rain flow events, you know, large storms and overflows and things like that?
                  DR. SARAH HAFNER:  Yeah, I guess we didn't do that analysis, but it would impact the short-term flow anomaly probably more than the other parts of the model, but we have focused on more recent data and if the run indicated, you could split it up into different runs based on that intensity.  
                  DR. ROBERT CHAPIN:  So let me just point out -- Dr. Yang, first, let me just point out that we're eating into our lunch hour.  We're going to be rejoining here at 1:00 so if you've got important questions that -- 
                  DR. RAYMOND YANG:  Very brief, okay?  
                  Ray Yang.  Once again, thank you for your presentation.  With Tom's help, I have a far better understanding of what you're doing, and, in fact, the model is, in fact, a polynomial equation with a variety of coefficients.  And I do have one question for Tamue.  Tamue, are we going to continue to get all these slides and so on?  These are very important information.  
                  MS. TAMUE GIBSON:  I've already emailed them to the panel.
                  DR. RAYMOND YANG:  Okay.  Thank you.
                  MS. TAMUE GIBSON:  Yes, yes.  
                  DR. ROBERT CHAPIN:  Andrew.
                  DR. ANDREW MIGLINO:  Andrew Miglino.  This will be, I hope, very quick.  So to get back to something Cliff said earlier about overestimation and choosing or fixing overestimation, in the real world, right, we're not going to know if we're overestimated.  So how do you make the decision to pick a different percentile in that case?
                  DR. SARAH HAFNER:  So that decision would be made based on these test data sets and based on the feedback from today.  
                  DR. ANDREW MIGLINO:  So you see the four-day is consistently overestimated at the 99th, you pick 75th or something like that and that's now the de facto standard for the four-day?
                  DR. SARAH HAFNER:  That's the idea.  
                  DR. ANDREW MIGLINO:  Okay.  
                  DR. RAYMOND YANG:  May I make a recommendation very quickly?  Okay.  Ray Yang.
                  I would recommend to EPA colleagues, this equation that Tom gave me actually is from the paper of Vecchia 2018.  And because SEAWAVE-QEX is a very, very important aspect of this whole meeting, you should have this information, the equation, which is model, and the explanation of all the parameters and so on, in your White Paper and your presentation, okay.  
                  DR. KENNETH PORTIER:  You stole my first comment.
                  DR. RAYMOND YANG:  I'm sorry.
                  DR. ROBERT CHAPIN:  All right.  Are we done with this part?  Okay.  We'll come back and do SBFs after lunch.  We're going to reconvene at 1:00.  It's 12:15.  See you in 45 minutes.  
                                    [BREAK]
                  DR. ROBERT CHAPIN:  All right.  Thank you all.  Thanks for being back here on time.  And now let's start doing some SBFs.  
                  Dr. Hartless, I think you're up.
                  
DEVELOPMENT AND EVALUATION OF A SAMPLING BIAS FACTOR PROGRAM PART 1: SHORT-TERM
                  
                  DR. CHISTINE HARTLESS:  Good afternoon, everyone.  I'm Christine Hartless and I'll be presenting talking about the development and further optimization of our sampling bias factors.  For this presentation, I will be focusing on the short-term sampling bias factors.  Mr. Peck, next to me, will be presenting on the long-term sampling bias factors next.
                  My presentation is structured to start with the purpose of this methodology, move to the development and evaluation approach, results, and finally implications for pesticide drinking water assessments.
                  So this is a flow diagram showing the overall picture of what we'll be presenting to you today in terms of our method development and evaluation of SEAWAVE-QEX and sampling bias factors that Dr. Bohaty presented earlier today.  And just to kind of remind you where we are, Dr. Hafner presented earlier about the SEAWAVE-QEX and that evaluation and so what she has done there is look at the monitoring data that were available and used SEAWAVE-QEX as tool, a regression-based tool, to help fill in those missing data and create realization such that we can have good picture of what's going on over a daily time step for those particular sites.
                  And there was a lot of discussion this morning about some optimization and what we were doing and there are some tools, diagnostic plots, and guidance that are included in the SOP documentation as well as in Dr. Vecchia's report that helped us to make sure that we are making the best use of our data and not over-using the data, not driving the tools too hard, and making sure that we're understanding the uncertainty that are present in our data.
                  So at this point, we're now going to be working with sampling bias factors, so I will assume that for the purposes of this presentation, what the -- the data that will be going into the development of these general sampling bias factors will be data that came out of SEAWAVE-QEX.  It had gone through all of the appropriate evaluation and that we have fair confidence in what those estimates are telling us about the specific sampling sites.
                  So these data, like from SEAWAVE-QEX, like daily monitoring data can be used in the development of sampling bias factors.  This provides this opportunity for us to greatly expand the number of pesticides for which sampling bias factors can be developed.
                  Short-term sampling bias factors allow the EPA to use sparse data sets, for example, those that may be composite samples collected on a weekly to a monthly basis over the course of a year, to be used quantitatively in drinking water assessments where short-term exposures would be of concern.  Additionally, they allow EPA to use the available monitoring data to the maximum extent possible permitting us to include more sites than if we were only able to use data from SEAWAVE-QEX alone.
                  Although many of you may be familiar with the use of sampling bias factors as they have been previously presented in several science advisory panels, I'm going to start with the basic definition.  A sampling bias factor is a protective multiplier for monitoring data to account for the uncertainty associated with reductions in sampling frequency.  As you've seen in some of the earlier presentations with Dr. Hafner, a reduction in the sampling frequency can potentially have a huge impact on various summary statistics calculated from the data, most specifically if we end up missing those peak observations -- peak concentrations.
                  So the sampling bias factor can be applied to summary statistics like the annual daily maximum or the annual 21-day rolling average derived from less than daily monitoring data to ensure that for some percentage of time, the sampling bias factor adjusted monitoring concentration is equal to or higher than the true unknown parameter.  Currently, the sampling bias factor has been developed such that the adjusted values will cover the true unknown maximum 95 percent of the time.  
                  The impetus behind this approach was to allow for the use of monitoring data that is limited by the number of samples while still being applicable to the largest possible number of monitoring locations and maintaining a simple, straightforward methodology.
                  To build on our work from earlier Scientific Advisory Panels with short-term sampling bias factors, we used the USGS surface water monitoring data from the four pesticides listed here, the atrazine, carbaryl, chlorpyrifos, and fipronil.  These pesticides represent different use patterns across the United States and different environmental fate and transport properties.  These data were released as part of the SEAWAVE-QEX publication from USGS and included additional QAQC steps compared to the data directly available in the water quality portal.  These USGS data then were processed through SEAWAVE-QEX to obtain the daily chemographs.  
                  And this is where I noted earlier, we've gone through and evaluated those data based on all of the various SOPs and the additional diagnostic plots in order to develop the best regression models for each of those particular sites and chemicals included in this analysis.
                  The SEAWAVE-QEX chemographs from atrazine USGS data were used to evaluate the different sampling and mathematical methods within the bootstrapping algorithm for the short-term bias factors.  The root mean square error was used to evaluate quality of fit and the data were visually evaluated.  Based on those results, we then moved on to estimate the short-term sampling bias factors for three additional pesticides based also on this USGS data set.  
                  Finally, we did an independent evaluation of the sampling bias factor methodologies using monitoring data from the National Center for Water Quality Research data set.  This is the same data set that Dr. Hafner presented earlier which is also known as the Heidelberg data set.  
                  This graphic here shows a high-level overview of the steps for generating a short-term sampling bias factor from the daily pesticide concentration data.  Over the next few slides, I will be going over each of these steps in more detail.  It is important to note that the methods here are only for the short-term sampling bias factors as the long-term sampling bias factors were developed with a different approach and will be discussed by Mr. Peck in the next presentation.  
                  So first, we'll start with one year of daily concentrations of data in this upper left-hand graph in this figure -- in this slide.  This could be data from a monitoring program where samples were collected daily or it could be output from the SEAWAVE-QEX program.  In this case, we were utilizing output from SEAWAVE-QEX.  This graph displays the concentration on the Y-axis in micrograms per liter over the course of one calendar year.  
                  The next step here is to calculate the parameter of interest from this full one-year data set.  Say, in this example, I'll be looking at the on-day annual maximum and here it's 18.4 micrograms per liter as you can see depicted on the slide.  
                  Next, to create the subsample data, we're going to first choose a sampling intensity and then choose a sampling methodology.  Here, the sampling intensity that we selected was 13 samples per year in this figure and a random sampling methodology.  The other sampling methodologies that we -- the other sampling frequencies that we evaluated were 52, 26, 17, and 13 samples per year which roughly equilibrates to weekly, bi-weekly, and monthly.  And then we also could choose between either the random or the stratified.  So on the right-hand graphic, I have randomly drawn 13 daily samples from that year, from the graph on the left.  
                  Next, we choose one of the three methodologies to infill these daily data.  This is the same graphic that we saw on the previous slide, to infill the daily data between the 13 sample time points in out bootstrap sample.  The methods that we chose from were linear, log linear, or stairstep.  
                  The first of these methods we evaluated here was the linear interpolation.  In non-technical terms, we're just basically connecting the dots.  This is shown in this example here.  And what this means is that we assume a linear relationship between neighboring points and use that simple equation to calculate a concentration for any of the dates that happen to fall in that interval.  So in this example, the infilled concentration following the red line on day six would be 1.25 micrograms per liter.
                  The second approach that we used was log linear which is very similar; however, first, the concentration data are transformed to the log scale and the linear relationship is estimated in that transformed scale.  After the values are estimated for the non-sampled days using that same approach as with the linear data, those estimates are then quote, back-transformed, to the original scale.  Please note that there is no figure for the log linear interpolation shown on this slide.  
                  Finally, for stairstep, the third interpolation method that we evaluated, daily concentrations on the non-sampled days are determined by using the concentration on the most recent preceding sample day.  For this example shown in the lower figure, days two through nine would have concentrations equal to the concentration measured on day one of 0.75 micrograms per liter.  
                  An important note for all of these methodologies is that a limitation is that the infilled concentrations will never be higher than the highest of the highest sample concentrations from that annual chemograph.  
                  So now we're moving to the next step in our bias factor flow chart.  Remember that the graph on the left here represents the 13 monitoring values that we sampled from the full initial chemograph.  And now in this figure here, we've used log linear interpolations to infill the values between those 13 sample dates as shown on the figure on the right.
                  So now based on this new time series, which is, again, seen on the left-hand graph, one calculates a summary statistic of interest, say for example, the 1-day, the 4-day, or the 21-day rolling averages.  For here for this one iteration of the bias factor development, the maximum one for 21-day rolling averages were 8.2, 8.1, and 7.9 micrograms per liter.  Then this whole process, the subsampling, the developing the new chemograph, and infilling between those sampled points, calculating the summary statistics is repeated 10,000 times.  One year of data from that site, from a given site, then results in 10,000 values estimated for a maximum 1-day, 4-day, and 21-day rolling average.  
                  So now we've got this distribution and again, I'm going to stick to talking about the one-day annual maximum values for this example as we step through.  We have a distribution of 10,000 one-day annual maximum values.  And here in this hypothetical example, the one-day annual maximums for the first three simulations that we happen to have run were 8.2, 9.3, and 15.9.  And remember then, for the initial daily chemograph, the one-day maximum measured value was 18.4 micrograms per liter.  
                  So now to calculate the one-day sampling bias factor for that particular site-year combination, then we're going to take the one-day maximum measured value, which was here at 18.4 micrograms per liter, and divide it by the fifth-centile from the distribution of the one-day annual maximums.  In this example, the fifth-centile was determined to be 9.5 micrograms per liter and therefore the one-day sampling bias factor was estimated to be 18.4 divided by 9.5.  So the sampling bias factor then results as 1.9.  
                  This process was done for each of the available site-year combinations of data remembering that for each site-year for utilizing data from SEAWAVE-QEX, we would have multiple realizations, so it would be actually for each site-year realization of data that we're calculating and estimating a bias factor.
                  So now what do these bias factors look like once they're calculated?  Here are the distributions for three sites based on some of the atrazine data.  A reminder here that each of the box plots that you see is contracted from the sampling bias factors that were generated from the multiple realizations of SEAWAVE-QEX data.  For this component of the work, we were using just 50 realizations of each site-year of data.  So each box and whisker plot that you see here represents 50 realizations.  So along the X-axis is the year for each of those three sites and along the Y-axis is the one-day sampling bias factor and that value is a unitless value.
                  So several things to note here as you're looking at this figure is that while all these sites had some years of overlap, some sites in our world of sites that we use from the USGS data only have three years' worth of data.  And sometimes there was less temporal overlap than as seen here.  It's also important to note that the Y-axis here is reported in log base ten scale and if these data were plotted in the original scale, you would see how strongly right skewed they are, meaning that vast majority of the data would be in the lower end of the distribution, but there would still be a significant percentage of data points with very high values.  
                  Finally, when looking at the first site as shown in red, one can see that the variation in sampling bias factors within each year is quite large and that the median value, the horizontal bar in the middle of the colored box tends to vary quite a bit from year to year.  The bottom and the top of the colored boxes here represent the 25th and 75th percentiles respectively.
                  When one looks at the second site shown in green, the within year variation is much smaller, but there still is a moderate level of year to year variation on that site as well.  Finally, the third site shown in blue, tends to have the lowest and most consisting variation from year to year, and there is little change in the median or any measure of central tendency over time.  These data just represent a subset of what we see when we're looking across all of the different sites.
                  So for each of these realizations of daily chemographs that are available for each site year, all combinations of sampling intensity, sampling method, and interpolation method were used to generate sampling bias factors.  Concurrent to calculating the sampling bias factors, the root mean square error, the formula that is shown on the slide here, was also calculated.  This provides a measure of goodness of fit of the daily chemograph generated within the sampling bias factor program with the original data and here these data were from the SEAWAVE-QEX chemograph.  
                  Interpolation methods and sampling methodologies were compared using a generalized linear mixed model and the resulting interpretations from that analysis were that the log linear interpolation performed better than linear or stairstep interpolations when looking at the root mean square error as an evaluation term, and that there were minimal differences amongst sampling methods when evaluating the fit of the data for the bias factors. These results held across all of the different sampling intensities. 
                  Oh, is there a question?
                  DR. ROBERT CHAPIN:  Quick question.
                  DR. VERONICA BERROCAL:  Yeah, sorry.  Can I understand the rationale behind this procedure?  I'm just very confused about the steps that have been implemented here.  So there is minor pesticide concentration data that is not available for every day.  So you apply SEAWAVE-QEX and you get this daily chemograph which is one of many possible data chemograph.  Then you subsample values from those so -- which might be observed values but might also be estimated values that are caught into that daily chemograph and then you interpolate with either a line or this step function.  Why do that?  Why not just use the daily chemograph several times?  I just don't understand it.  Moreover, there is obvious limitation that with the step or the line you would never be able to estimate the concentration greater than what you sampled.
                  DR. CHISTINE HARTLESS:  The idea behind using the kind of development of the sampling bias factors was to utilize site data from sites where we do have adequate data to develop those sampling bias factors to get a feel for if we have really good data and then subsample from that, we can get a kind of -- true isn't the right word, but a more confident kind of level of what that is, if we were to generate the sampling bias factor.  Then what we would do is we would take those sampling bias factors, which is basically kind of looking at a ratio of what we would expect as that parameter or choice, say a one-day maximum, if we have the optimum data set, which is daily data, versus data that was not an optimum data set, and we would kind of know what that ratio was.  
                  Then let's say, for example, we have data from another site -- monitoring data from another site, where we might have 13 samples across the year, but we can't use SEAWAVE-QEX on it because we only have one or two years' worth of data.  SEAWAVE-QEX in the best of situations requires a minimum of three years' worth of data.  So this allows us to use that very minimally sampled data of, say, 12 observations per year, to be able to kind of estimate and come up with a better feel for where our uncertainty would be if we had better data to adjust for that uncertainty there.  Does that help?
                  DR. VERONICA BERROCAL:  I'm still very confused because the SEAWAVE-QEX model is supposed to try to represent what is the trajectory of this pesticide concentration.  So I don't understand one, why the sampling bias factor could not have been computed by using one daily chemograph over all the possible of 10,000 values that you got from 10,000 chemographs.  You get your sampling bias factor for the site that has enough data and then extrapolate to the site that don't have enough data.  I just don't understand where the -- maybe there is another logical reason why?  
                  DR. KENNETH PORTIER:  So I think there's a misunderstanding and I had the same thing reading the White Papers.  If you go back to your slide where you describe -- you're using SEAWAVE --
                  DR. CHISTINE HARTLESS:  Which?
                  DR. KENNETH PORTIER:  Right there is fine.  Go back.  There.  
                  So you're not using SEAWAVE here.  
                  DR. CHISTINE HARTLESS:  Correct.
                  DR. KENNETH PORTIER:  You only use SEAWAVE in an almost full data set to kind of fill in the missing -- it's as an interpolation tool.  And then SEAWAVE is done.  They take the complete time series and draw a subsample from that time series, interpolate it, calculate their statistic, their five percentile, that's one, and then do it again.  Draw another sample, interpolate, get an -- do that 10,000 times.
                  DR. VERONICA BERROCAL:  So I understand that, but I guess -- sorry, but those 13 dots, where all 13 measure concentration, I think I wouldn't have this issue, but it's the fact that of these 13 dots, maybe 9 of them have been estimated using the data chemographs.  So why do this estimation twice and in one case using a function that you had -- you have some logical reason to believe that the pesticide concentration has that wave shape versus this one that is like a step function.  I'm just very confused. 
                  DR. KENNETH PORTIER:  But they only SEAWAVE once.  
                  DR. ROCHELLE BOHATY:  So --
                  DR. KENNETH PORTIER:  Yeah.  I think they're just using it once.  
                  DR. ROCHELLE BOHATY:  So maybe just to add a little bit of context is that we have very few pesticides out there where we have daily data.  So in order to develop bias factors for more pesticides, we need to fine a best way to infill the concentrations between non-daily data, right?  And so we're trying to use SEAWAVE to do that.  And then from that, because we're only going to, as you'll see in the case studies, have maybe 30, 100 sites depending on the pesticide, right, where we can develop sampling bias factors, but we have maybe a thousand other sites where we could apply bias factors.  And so we're trying to leverage as much of the available data that we have to make some sort of conclusions about the potential exposure.
                  DR. JAMES SADD:  Jim Sadd.  So you extract the points and then interpolate them.  And the three interpolation techniques that you use are all deterministic.  Did you think about or explore the possibility of using non-deterministic interpolation method which would better recognize the uncertainty in each of the estimated measurements just from sampling and analysis and also, perhaps, uncertainty from the way in which they were selected and that would, perhaps, be a better way to recognize that the sample points you have or the estimate sample points you have don't capture the peak.  
                  DR. CHISTINE HARTLESS:  We did not include any additional methodologies in this piece of our work to look at different methods of interpolation for the development of these bias factors.  
                  And the other thing that I'll add in too, because -- and it took me a while to get to understand where the differences between when you're using SEAWAVE and when you're using the bias factors -- remember, the SEAWAVE-QEX, if you think about kind of a general framework presentation that was presented earlier, on a chemical by chemical analysis when we're going through and doing risk assessments, SEAWAVE-QEX would typically only be used at the Tier 4 level for an individual chemical when you're looking at individual sites and you're doing very highly refined analysis.  That's a fairly big level of lift in terms of level of effort to go through and do that.  
                  The develop -- what I'm -- with the bias factors what we're developing here is to trying to develop something that's perhaps a bit more generic in that we're going to be eventually -- we're looking at it across a couple of chemicals to look at how much variation there is in these bias factors amongst chemicals to start looking at that.  And this is something that we're hoping that we can be able to apply at the Tier 3 level which would impact more chemicals and it's a lesser lift to, if we've got these bias factors, to use those.
                  So I think you have to kind of put in context what we're doing here.  And this is really gut-wrenching detailed work as was mentioned earlier of why are we doing all this subsampling when we have perfect data?  We're doing all of this so that we can develop something to use on a broader scale in Tier 3 when we have less than perfect monitoring data.
                  DR. JAMES SADD:  This is Jim Sadd again.  I understand.  If you look at the graph on the right, the interpolated data never goes above one of those sample points and we know in reality it probably does.  So my suggestion or my question is, if you don't use a deterministic interpolator, you could have a curve that goes above or below your sampling points.  And I'm wondering if that would be a more realistic way to interpolate the estimate?
                  DR. CHISTINE HARTLESS:  We've not looked at that specifically.  What we did to account for that was run those 10,000 Monte Carlo simulations where we're sampling a different set of 13 points each of those 10,000 times.  And so hopefully by doing that, we would capture that broad range of distributions and be able to capture where all those peaks were and then using a percentile off of that distribution.  
                  DR. ROBERT CHAPIN:  Okay.  Question for clarification only, right?  Clarifying questions.  We're here for clarifying questions if we're going to interrupt their presentation.
                  DR. CLIFFORD WEISEL:  All right.  Well, the reason I brought it is there was an initial statement that you're doing these so you can do it for other chemicals, and you clarified that you're doing -- when you look at other chemicals -- you're trying to understand something about the properties.  Are also trying to understanding something about the properties of the site, how that interacts with the chemicals because that's -- 
                  DR. CHISTINE HARTLESS:  Yes.  Stay tuned for later this afternoon.  
                  DR. ROBERT CHAPIN:  Okay.  Go.  
                  DR. CHISTINE HARTLESS:  All right.  Let me -- All right.  I'm trying to remember where I am.  That's okay.  
                  DR. ROBERT CHAPIN:  I think you were back one.
                  DR. CHISTINE HARTLESS:  All right.  There.  Okay.
                  So to summarize what we have so far, the distribution of these sampling bias factors that we've developed within a site-year combination are right skewed.  
                  Sampling bias factors can be highly variable within a site year and among site years within a site.  Log linear interpolation performs better than linear or stairstep interpolation and minimal differences in sampling bias factors were identified when looking at either random or stratified sampling methodologies.  
                  So for further work in this stage of the process, we're going to limit ourselves to looking only at the log linear interpolation and random sampling methodologies.  Our next step then was to take the sampling bias factors that we generated and obtain a summary statistic for each site.  Remember that for each site, a sampling bias factor was generated for each year realization of the SEAWAVE-QEX data and that's quite a few values if you remember from the previous box plot figures.  For example, if there were eight years of data for a given site and SEAWAVE-QEX was parameterized to generate 50 realizations, then we have 8 times 50, or 400 sampling bias factor estimates for that one site based on the data that we have available.
                  So to simplify this down to look at one value per site while accounting for the skewed distributions, varying time windows, both in the number of years represented at a site and the starting and ending of the sampling windows for the monitoring data, the following approach was used.  The first step was to calculate the median sampling bias factor for each year within a given site.  So in this example here, we would have a median sampling bias factor for 2009 for 2010, 2011, et cetera.  Then using those annual medians from the first step, we would calculate an overall site median to obtain one sampling bias factor per site.  
                  So now that we have evaluated and honed in on some of the methods that we will be using to estimate the short-term sampling bias factors and have an approach to summarize data for each site, we went back to the SEAWAVE-QEX data generated earlier for the other three chemicals from the USGS data set and developed sampling bias factors for those pesticides as well.  As a reminder, we were focusing on the log linear interpolation and random sampling for the sampling bias factors.  And what is pictured here is the results based on 26 samples per year.  
                  Each data point here represents one value per monitoring site and each box and whisker plot here contains a single value for each site.  So remember that atrazine had approximately 100 sites within that USGS data set so there are approximately 100 points within that -- the dark blue box and whisker plots that are presented up above.  The other three pesticides had approximately 30 sites for each pesticide.  And again, these sampling bias factors had right skewed distributions noting that the Y-axis here is in log scale and we also see that the differences amongst the pesticides were consistent across the various different averaging windows looking at 1-day maximum, 4-day rolling average, or 21-day rolling averages.  And then as the averaging window increasing, the sampling bias factor will decrease.  
                  These relationships amongst the pesticides in the averaging windows that you see here are similar to those as if a different sampling intensity was used.  We've done those work, but I just did not present them in this slide.
                  So lastly, we're going to do some further evaluation of these sampling bias factors.  We're going to compare in two different examples.  We're going to compare the short-term sampling bias factors derived from this USGS sites to those developing using the National Center for Water Quality Research data set.  Again, these are the Heidelberg data, which were subsampled to create a sparse data set which to better represent the data typically available to assessors when completing drinking water assessments.
                  So there were four watersheds that were used for this evaluation.  This is Honey Creek, Rock Creek, Maumee, and Sandusky.  All four of these sites had data for atrazine.  So the data were run through SEAWAVE-QEX to create annual chemographs for use in the short-term sampling bias factor program.  Sampling bias factors were derived for 13, 17, 26, and 52 annual sampling intensities using the random sampling method and log linear interpolation and then were compared to the estimates derived from the atrazine from those USGS sites, that will be the hundred sites that were shown on the figure from the previous slide.  
                  The minimum, median, and maximum values were plotted for the USGS data set represented by the circles on the figure here and the values for the Heidelberg data set are represented by the diamonds.  It's hard to distinguish there, but the diamonds are the more orange-red color.  All the Heidelberg values fell within the values derived from the USGS sites meaning that they fell between the maximum and the median except for the flashiest system when the minimal sampling intensity was selected.  This provides some general indication that the method for developing short-term sampling bias factors appears to work for these particular sparse data sets.  
                  Finally, to --
                  DR. XUYANG ZHANG:  Xuyang Zhang here.  I have a question.  
                  DR. ROBERT CHAPIN:  Go ahead.
                  DR. XUYANG ZHANG:  Can you go back to the previous slide?  Yes.  So you mentioned that the -- looks like both USGS data sets and the NAWQA data set has atrazine.  Is that -- do they have any overlap in terms of, you know, the same sampling site and the same time between the two data sets for the same chemical?
                  DR. CHISTINE HARTLESS:  There is overlap in the temporal construct.  The USGS data sites -- the USGS did not include the same sampling sites as were included in the Heidelberg data set.  
                  DR. XUYANG ZHANG:  Okay.  So they are different?
                  DR. CHISTINE HARTLESS:  Yes, they are different.  
                  DR. XUYANG ZHANG:  Okay. Thank you.  
                  DR. CHISTINE HARTLESS:  Finally, to compliment the analysis done earlier as part of the SEAWAVE-QEX evaluation, sampling bias factors were compared using chemographs derived from flow or precipitation as a covariate in SEAWAVE-QEX.  Here you see the data for metolachlor in Rock Creek.  The closed circles are the sampling bias factors when flow was used as a SEAWAVE-QEX covariate for those five sparse data sets that Dr. Hafner discussed earlier and the open circles represent the sampling bias factors when precipitation was used as a SEAWAVE-QEX covariate.  The straight line represents the sampling bias factor that was estimated using the full metolachlor data set for that particular site.
                  We recognize that this is a small exploratory analysis as only one site was evaluated, but it does demonstrate that use of flow or precipitation as a covariate can provide comparable sampling bias factors.  
                  So to present some of our -- to summarize with some results and conclusions, the analysis using the USGS monitoring data showed that the log linear interpolation provided the best fit for the sampling bias factor estimation.  There were small differences in sampling bias factor estimates between the random and stratified sampling, but we chose to move with the random sampling methodology.  The site-specific sampling bias factor was calculated as a median within a year and then median across years.
                  Use of the Heidelberg monitoring data provided an independent evaluation of the sampling bias factor estimation methodology and as few as 13 samples per year were needed to apply a sampling bias factor to an estimated upper end short-term average concentration.  
                  So finally to introduce some implications for drinking water assessments, the short-term sampling bias factors were derived for four pesticides at between 30 to 100 sites and they permit the use of a value, the maximum across sites, to be used as a screen at Tier 3 assess potential exposure concerns.  The pesticide-specific sampling bias factors then can be used at Tier 4 to capture the range of potential concentrations not measured between sampling events.  Sometimes these large sampling bias factors -- sometimes, large sampling bias factors are obtained suggesting uncertainty in the available data and more data and further exploration into the specific factors for that particular site may be warranted.
                  Thank you, and I'll take any clarifying questions at this point.
                  DR. ROBERT CHAPIN:  Okay.  
                  DR. CLAIRE BAFFAUT:  On your -- 
                  DR. ROBERT CHAPIN:  And your name?
                  DR. CLAIRE BAFFAUT:  -- table where you had -- 
                  DR. ROBERT CHAPIN:  Claire, just state your name.
                  DR. CLAIRE BAFFAUT:  Oh, Claire Baffaut.  On your table where you had explained how you combined the factors for each year to calculate one for the site --
                  DR. CHISTINE HARTLESS:  Yep.  I went too fast.  That one.
                  DR. CLAIRE BAFFAUT:  Okay.  So you explain here for the median and you take the median for each year and then the median of the median basically.  
                  DR. CHISTINE HARTLESS:  Correct.
                  DR. CLAIRE BAFFAUT:  But in the White Paper, you say you used the same method for all the other percentiles, the 90th percentile, 99th percentile, so I can understand how you calculate the 99th percentile, for example, for each year, but then how do you combine all of those values into one?  So if you have ten years of data, that gives you ten 99th percentile. 
                  DR. CHISTINE HARTLESS:  Correct.
                  DR. CLAIRE BAFFAUT:  How do you combine those?
                  DR. CHISTINE HARTLESS:  You would probably -- you would use the median of those 99th percentiles.  You would not use the 99th percentile.  
                  DR. CLAIRE BAFFAUT:  Well --
                  DR. CHISTINE HARTLESS:  I understand where the confusion would be.  Does that help?  
                  DR. CLAIRE BAFFAUT:  Yeah.
                  DR. CHISTINE HARTLESS:  Okay. 
                  DR. CLAIRE BAFFAUT:  Thanks.
                  DR. ROBERT CHAPIN:  Ray.
                  DR. RAYMOND YANG:  Ray Yang.  I have two questions.  One is how realistic is the risk of one-day and four-day exposure?  Do you actually have data to indicate that somebody have one-day exposure and get sick?  I remember earlier on, I think maybe Rochelle's presentation, there was a large percentage of the sample or below one part per billion.  So are we talking about overkill? 
                  DR. ANNA LOWIT:  So the rolling -- so the days, the number of days that they're doing, the one-day, the four-day, come from the different durations of exposure we derive from the Human Health Risk Assessments that are largely derived from the toxicology from the pesticides.  So in a standard Human Health Risk Assessment, we would do acute dietary.  So we do one-day assessments for food, and that would include food plus water.  We would also do what we call short-term which is generally 30 days, approximately, but in some cases like atrazine and the triazines, for example, we actually had a series of SAPs on this back around 2011, 2012 as we talked about this morning, we did five in 18 months.  It was an insane period, but the four-days applied to the triazines is derived from that work of how LH attenuation in the rats is then brought over to the human experience.  That's where the four-day is actually applied specifically to the triazines.  
                  So, what we're doing here is to try to match the mode of action in toxicology to the exposure duration, so we have a seamless duration across.  So for example, in methylcarbamate where we do only single-day peak assessments.  So, in methylcarbamate, cholinesterase peaks after about an hour and a couple hours later it's mostly recovered.  So, all of our assessments are only one-day peak assessments.  So we need to match that highest peak in the water to the peak in the cholinesterase inefficient.  That's where the numbers come from.
                  DR. RAYMOND YANG:  Okay.  Well, I think, you know, the point I raise this -- the reason I raise this is that maybe EPA should think about something like this rather than, you know, this is the way we've been doing for the last decade and so on and so forth, okay.  
                  So second question, if we go back to slide 102, this is, again, for education purposes.  Okay, 102.  Okay.  You got this bracket, the three-step process, all right.  I'm interested in the bootstrapping method of resampling, all right.  Am I correct that, you see, you've got this profile there.  That represents the actual sampling monitoring data, right?  And now you resampled it certain times, certain -- once a month or what have you.  When you do that bootstrap resampling, are you sampling from a probability distribution for that point, from earlier results and so on?
                  DR. CHISTINE HARTLESS:  I'm going to slide back to this slide here where I'm actually presenting the bootstrap sample.  So the figure -- the graph on the left represents the daily monitoring data, be it actually observed out the field and measured or be it generated using SEAWAVE-QEX.  And then what we'll do is sample from that and that's where those 13 samples come from and we walked that all the way through that process to get one value at the end which say, for example, might be the one-day maximum value and the four-day rolling average.  And then I'm going to resample again from the original complete daily chemograph that's shown on the left to come up with a new set of 13 values.  That's the piece that I'm doing the bootstrapping with and sampling 100 times, but it's not based on a probability distribution.
                  DR. RAYMOND YANG:  It's not?
                  DR. CHISTINE HARTLESS:  No.  
                  DR. RAYMOND YANG:  All right.  Thank you.
                  DR. ROBERT CHAPIN:  Tom.
                  DR. THOMAS POTTER:  Hey, Tom Potter.  A couple of clarifying questions.  One, I recall reading in the White Paper, and perhaps I'm having some tunnel vision here right now, but that you only use SEAWAVE-QEX with flow as a covariate in this analysis?
                  DR. CHISTINE HARTLESS:  For this piece of it, yes.
                  DR. THOMAS POTTER:  Yeah, okay.  Now, was there -- what was your rationale there?  Is that -- you have greater confidence in the result?
                  DR. CHISTINE HARTLESS:  It was really a function of time. 
                  DR. THOMAS POTTER:  Okay.
                  DR. CHISTINE HARTLESS:  And we did do some bits and pieces where we looked at some, and one of the very last slides that I presented was a small subset -- was a small look at comparing bias factors that were generated using SEAWAVE-QEX with flow as a covariate versus bias factors that were generated using SEAWAVE-QEX with precipitation as a covariate.  I believe that was probably the last figure slide that I presented.
                  But that was, again, just a very small -- 
                  DR. THOMAS POTTER:  In this presentation here as opposed to the White Paper or the --
                  DR. CHISTINE HARTLESS:  It's in the White Paper.  I don't remember the page number.
                  DR. THOMAS POTTER:  I'll -- maybe I'll dig in there again tonight and take a look, but I just wanted to be clear on that.  
                  So the second clarifying is, these are all flowing systems; is that correct?
                  DR. CHISTINE HARTLESS:  All the USGS -- 
                  DR. THOMAS POTTER:  All your analyses were based on flowing system only.  You didn't look at any of the static system scenarios.  
                  DR. CHISTINE HARTLESS:  No.  There would have been quite a range in the amount of flow that each of these systems may have had just based on what's in the USGS data set, yes.
                  DR. THOMAS POTTER:  Okay.  And finally, just to wrap it up, and again, maybe we'll defer to later on this, but there seemed to be some real differences in responses across sites.  So I'm assuming you folks are looking at this and trying to gain insight as to what drives that, whether it's the size of the watershed, the flashiness, et cetera.  Is that ongoing work?
                  DR. CHISTINE HARTLESS:  Actually, I'll speak some to that after Mr. Peck's presentation.  
                  DR. THOMAS POTTER:  Okay, super.  Thank you.  
                  DR. ROBERT CHAPIN:  Last one.
                  DR. VERONICA BERROCAL:  Yes.  Can I --
                  DR. XUYANG ZHANG:  Xuyang Zhang here.
                  DR. VERONICA BERROCAL:  Dr. Berrocal.  Can you explain again how you do the sampling bias factor evaluation?  I guess you went through the slide and I didn't see it.  Can I -- can you clarify?
                  DR. CHISTINE HARTLESS:  Which -- the whole --
                  DR. VERONICA BERROCAL:  No, no, no.  Like, there was a slide where you explained how you evaluated the sampling bias factor.  You had a statement at the end that said, the log linear random sampling was the best, so I just wanted to know how did you determine best?  That's what I was talking about.
                  DR. CHISTINE HARTLESS:  I was using the variable that I was evaluating there, was the root mean square error term and I used a mixed -- generalized linear mixed model to evaluate that.
                  DR. ROBERT CHAPIN:  All right.  Dr. Zhang.
                  DR. XUYANG ZHANG:  Yeah, I have a question on slide 108.  Yes.  So here, we -- I think you guys are deriving the median value from the 50 different chemograph simulations for each year.  And my question is, how variable, you know, what's the variation among these different chemographs because atrazine data is probably one of the best data sets that we can find for now and I'm curious about their ability among the 50 chemographs that it generates.
                  DR. CHISTINE HARTLESS:  There is a fair bit of variation amongst the chemographs.  One of the things that I looked at in developing that -- the generalized linear mixed model was incorporate variants terms to account for that variability amongst the chemographs in years.  I don't remember what those variant parameter estimates are off the top of my head though.  
                  DR. XUYANG ZHANG:  Okay.  That's a good idea.  Thank you.
                  DR. ROBERT CHAPIN:  Okay.  We done with the short-term bias factors?  
                  DR. CLIFFORD WEISEL:  I think it's the next slide or the one that you showed the comparisons of the different water.  Just, no.  Keep going.  We have the NCWRQ (sic) and can you tell me what you're using this for because I see you have a range?  What are you trying to explain out of this?
                  DR. CHISTINE HARTLESS:  The intent of why we looked at this was to look at doing -- it's a very small.  I wouldn't consider it a full validation of the methodology, but a very small validation of the methodology where I looked -- we developed quite a large data set using the USGS data where I had roughly a hundred sites.  And then I said, well, is that going to kind of potentially encompass the world of other sites?  And we had data available for four sites that we then calculated comparable -- 
                  DR. CLIFFORD WEISEL:  Well, that's -- it says minimum, max, and medium.  That's three, not different sites, but -- 
                  DR. CHISTINE HARTLESS:  That's from the USGS data where I had the hundred sites.
                  DR. CLIFFORD WEISEL:  Right.
                  DR. CHISTINE HARTLESS:  And so I took the minimum when I looked at --
                  DR. CLIFFORD WEISEL:  These are minimum.  That's the -- 
                  DR. CHISTINE HARTLESS:  The minimum from one site.
                  DR. CLIFFORD WEISEL:  Oh, okay.
                  DR. CHISTINE HARTLESS:  So I took -- 
                  DR. CLIFFORD WEISEL:  That's what I was trying to -- 
                  DR. CHISTINE HARTLESS:  So I had my 100 sites --
                  DR. CLIFFORD WEISEL:  Okay.  Now I understand.  That's what I was trying to understand where -- 
                  And I have a similar question for, I think, it's the next slide, where you're looking at the bias.  Yeah.  Again, how are you trying to -- explain to me what you're trying to say from this data so I can see a number of things on things on here.
                  DR. CHISTINE HARTLESS:  Again, it's a very, very small, it's not a full validation, of the methodology.  But looking at comparing, if I was to develop sampling bias factors using data from SEAWAVE-QEX that were developed using flow as a covariate or if I used precipitation as a covariate for that same monitoring data, do the bias -- are the bias factors comparable?
                  DR. CLIFFORD WEISEL:  Between the two of them?
                  DR. CHISTINE HARTLESS:  Correct.
                  DR. CLIFFORD WEISEL:  Not compared to your full record?
                  DR. CHISTINE HARTLESS:  And this one I also looked at -- this one we also have the full record for that one site.  I did not look at this with relative to the entire -- all of the USGS data set.
                  DR. CLIFFORD WEISEL:  All right.  The reason I ask that is the range I'm seeing is in order of magnitude across these and -- yeah, pretty -- well I see -- if we look on the left-hand side, I don't know what the lower value is, but it looks like it's closer to one to me.  And --
                  DR. CHISTINE HARTLESS:  This one -- remember, this one is not presented in log scale.  This is --
                  DR. CLIFFORD WEISEL:  No, I realize it's not a log scale, but I'm trying to understand a variation of the order of the levels that you're seeing here in these factors.  What do they mean and how do they translate to the variation in predicting what the concentrations would be?
                  DR. CHISTINE HARTLESS:  All right.  So the set of five points that you see in a vertical column there are the sampling bias factors that would be developed from each -- one of those from each of those five sparse data sets that Dr. Hafner presented -- talked about earlier.  And so you can get a feel for the variability based on that prescribed subsampling that was done in terms of how that impacts the sampling -- how that carryover impacts the sampling bias factors. 
                  DR. CLIFFORD WEISEL:  All right.  So --
                  DR. CHISTINE HARTLESS:  Is that what you're trying -- 
                  DR. CLIFFORD WEISEL:  You know, this gives a -- then you would always take the maximum you would get out of this or the ---
                  DR. CHISTINE HARTLESS:  One would not typically do this in practice because this is --
                  DR. CLIFFORD WEISEL:  Well, that's my concern because now you've done it.  In practice, you would only do one.  You get a single value.
                  DR. CHISTINE HARTLESS:  So this is just to show that there is variation in those and so that's part of why we're accounting for that uncertainty.  And I think one of our questions to you is in looking at the various percentiles that we're proposing for these -- for the sampling bias factors kind of at the end stage of our process -- 
                  DR. CLIFFORD WEISEL:  Right.
                  DR. CHISTINE HARTLESS:  -- is that going to adequately cover the variability that we see in looking at all of these different pieces of the variant.
                  DR. CLIFFORD WEISEL:  And that's what I see when I'm looking at it and that's what worries me.  I don't know enough to answer that question, but hopefully -- that's why I'm bringing it up is something that hopefully -- 
                  DR. ROBERT CHAPIN:  Okay.  Last one.  Speak into the mic please and identify yourself.
                  DR. IAN KENNEDY:  Ian Kennedy.  I just would like you to possibly confirm that what I think you're doing is what I think you're doing, and that is you're taking a SEAWAVE-QEX-generated chemograph and you're taking random samples from that repeatedly and trying to kind of the fifth percentile worst possible random sample that you would have and comparing that to the maximum.  
                  DR. CHISTINE HARTLESS:  I don't like the characterization of it.  
                  MR. CHARLES PECK:  So we're taking the chemograph.  We're subsampling it.  We're generating another chemograph using log linear, linear, whatever.
                  DR. IAN KENNEDY:  Right.
                  MR. CHARLES PECK:  And then from that we're calculating a statistic, whether it's the maximum 1-day concentration, 4-day, 21-day.
                  DR. IAN KENNEDY:  Right.
                  MR. CHARLES PECK:  We're setting that aside.  We're going that 10,000 times using the fifth percentile and coming up with --
                  DR. IAN KENNEDY:  Right.  It's the fifth percentile of those 10,000 times.
                  MR. CHARLES PECK:  Correct.
                  DR. IAN KENNEDY:  So you're taking 10,000 possible -- if that was the real chemograph, these are 10,000 possible ways that someone might have sampled it.
                  MR. CHARLES PECK:  Correct.  
                  DR. IAN KENNEDY:  And then you're basically taking the fifth percentile of all the maximum --
                  MR. CHARLES PECK:  Correct.
                  DR. IAN KENNEDY:  -- of those.  Okay.  Thanks. 
                  DR. ROBERT CHAPIN:  I'm going to declare victory and we'll move on to Mr. Peck and talk about long-term sampling bias factors.  
                  And may the Lord have mercy on you.
                  
DEVELOPMENT AND EVALUATION OF A SAMPLING BIAS FACTOR PROGRAM PART 2: LONG-TERM
                  
                  MR. CHARLES PECK:  Hopefully, this will be a little bit easier, but good afternoon.  Again, my name is Chuck Peck, and I'll be discussing the development of the long-term sampling bias factors and how they will be applied in drinking water assessments.  
                  For this portion of the SAP, we'll look at why we're developing long-term sampling bias factors, how they were developed and evaluated, and their implications in drinking water assessments completed by the Office of Pesticide Programs.
                  So just to reorient and remind folks where we're at, at this point, we have daily chemographs, either generated by SEAWAVE-QEX or that luckily, somebody's gone out and monitored daily for at least a year and developed a chemograph.  We then take that, and we put it into a sampling bias factor program to develop a long-term sampling bias factor.  And that sampling bias factor program and algorithm is what I'm going to talk about here.
                  So just a little bit about why we're doing this.  Okay, certain pesticides at EPA valuate have long-term exposure durations.  For these pesticides, EPA typically uses modeled estimates, annual averages for chronic concerns, and 30-year averages for pesticides with cancer concerns in our drinking water assessments.  And the long-term sampling bias factors would allow EPA to develop annual and 30-year averages using monitoring data that we could then use in the human risk assessments.  They allow EPA to use sparse data sets comprised of four to ten samples per year.  As many community water systems typical conduct monitoring on a quarterly basis, it would be really desirable for EPA to use long-term sampling bias factors on these data sets to adjust the samples and obtain an annual average for use in a drinking water assessment.
                  Lastly, long-term sampling bias factors allow EPA to use the available monitoring data to the maximum extent possible permitting us to include more sites than would have been considered using SEAWAVE-QEX alone.  Specifically, they provide EPA with an easy method for estimating an annual average and a 30-year average concentrations that would account for temporal uncertainty in non-daily surface water monitoring data.  
                  So as Dr. Hartless had presented, I'm going to sort of walk you through my process here.  Okay, and this is sort of a visual representation of what we're doing.  And I just need to emphasize that this process is going to be different than Dr. Hartless' for a couple different reasons.  One, I mean, for short-term sampling bias factors, after we do sampling and interpolation, you take the maximum rolling average from that for the exposure duration you're in interested in.  But when you're looking at an annual average, you're only going to get one value each time you do that.  And so trying to develop, you know, the maximum rolling average, you're just going to get an annual average.  
                  The other thing is that when we're doing this, the tails.  When we start to subsample, you lose the tails in that.  So quite honestly, you're not able to estimate a 365-day average using the preceding technique.  So what we've done is we've come up with somewhat simpler method to develop a long-term sampling bias factor.  
                  So the process starts with a chemograph here on the left for a chemical and depending on the number of samples you want to develop the sampling bias factor for, either four, six, eight, or ten, and the sampling method you want to use that you want to evaluate, whether it's a random or a stratified sampling method, you would then go in and you would do what's called estimating the 90th percentile upper confidence value around the mean of those values.  
                  You would then repeat this process 10,000 times, okay, getting 10,000 90th percentile upper confidence values around the mean, and then you would then take the fifth percentile of that distribution and you would divide that into the actual 365-day average that you have to develop your long-term sampling bias factor.  In this case, the 365-day average from the chemograph was 1.8 micrograms per liter.  When we did our distribution, our 10,000 runs, the fifth percentile was 1.4 so you would end up with a long-terms sampling bias factor of 1.3 and that would be the sampling bias factor for that site, for that realization, and for that year of data.
                  So this slide here presents the equation that we use to estimate the 90th percentile upper confidence value around the mean using the number, the average, standard deviation of the samples, and the one-sided student T-value at an alpha 0.1.  
                  It was discussed earlier by Dr. Hartless to simplify this down to one value per site while accounting for the skewed distributions and the varying time windows, EPA calculated the median sampling bias factor across all 50 equally probable annual chemographs for a site and year, then calculated the median sampling bias factor for each year within a given site, then calculated the median sampling bias factor for that site.  
                  To determine the feasibility and use of long-term sampling bias factors, EPA used the existing SEAWAVE-QEX analysis presented earlier for the four pesticides from the USGS data set to, one, evaluate the number of samples that would be compatible for use in a long-term sampling bias factor process.  Two, develop long-term sampling bias factors for both the random and stratified sampling methods to three, see how different the results would be, and four, compared these long-term sampling bias factors to those using sparse or minimally in-field sampling derived from the National Center for Water Quality Research Data Set commonly referred to as the Heidelberg data set, to evaluate the impact of different data sets on long-term sampling bias factors.
                  So the first step was to determine how many samples we would need in order to effectively use the long-terms sampling bias factors.  So using 502 site years of USGS chemographs for atrazine, EPA estimated the 90th percentile upper confidence value around the mean 10,000 times for between 2 and 15 samples and created a distribution for each of the number of samples.  EPA then compared the value for N samples to the value of N plus one samples to estimate the percent gain in adding a sample.  This graph depicts the results as one moves from an N of 2 to an N of 14.  
                  For two samples, the percent gain moving from two to three samples range from minus 30 to 169 percent with an average gain of 111 percent.  So moving to three samples is a significant improvement.  Once you get to four samples, the average gain falls to about 16 percent.  At 10 or more samples, the average gain is five percent or less so there isn't much gain in developing long-term sampling bias factors for more than 10 samples.
                  It's important to note that this figure does not mean that the long-term sampling bias factors are below one.  Okay.  It simply means that the increase in the number of samples doesn't impact the value calculated for the sampling bias factor.  Based on EPA's analysis, four samples were selected as the lower bound for the long-term sampling bias factors and 10 samples were selected as the upper bound.
                  Next, EPA developed long-term sampling bias factors for four USGS pesticides using four, six, eight, and ten samples and the random sampling method and compared them across pesticide sites and years.  An example of this analysis is provided in the box and whisker plots for three sites for carbaryl.  Please note that the Y-axis is on the log scale.  This can be seen in the figure long-term sampling bias factors can vary across sites as can be seen in the differences between the values for sites one and two, the pink and green boxes respectively, compared to the values for site three, the blue boxes, and across years as can be seen in the spread of the box plots for sites one and two.  Although it is not shown in this figure, there's also differences in the long-term sampling bias factors across the four pesticides that we evaluated. 
                  Lastly, EPA compared the long-term sampling bias factors derived for the USGS sites to those developed using the Heidelberg data sets which were minimally infilled using SEAWAVE-QEX.  Three watershed sites were selected for assessment, Honey Creek, Rock Creek, and Sandusky River.  All three sites had data for atrazine, so the minimally in-field data were run through the long-term sampling bias factor program and sampling bias factors were derived for four, six, eight, and ten samples using the random sampling method.  The results were then compared to the estimates derived for atrazine at the USGS sites.  The minimum, median, and maximum values were plotted for both data sets and the values for the three Heidelberg sites, the orange diamonds, fell within the values derived for the USGS data set, the hollow circles, indicating the method for developing long-term sampling bias factors appeared to work for other data sets.
                  Based on our valuation, EPA believes that using SEAWAVE-QEX pesticide concentrations, long-term sampling bias factors can be developed to quantify the range of potential concentrations occurring between samples.  And as few as four samples per year are needed to apply a sampling bias factor to estimate upper end long-term average concentrations. 
                  With regards to using long-term sampling bias factors in drinking water assessments, EPA believes that sampling bias factors can be used to quantify the uncertainty in available surface water monitoring data.  Specifically, EPA has developed a range of long-term sampling bias factors for four pesticides in 30 to 100 USGS sites.  These values can be used to adjust monitoring data as part of the screening assessment conducted at the Tier 3 level on a drinking water assessment which can then be compared to the drinking water level of concern to see if more refinements are needed at the Tier 4 level. 
                  EPA also believes that the pesticide specific long-term sampling bias factors can be developed and used at the Tier 4 level of a drinking water assessment to adjust concentrations and capture the range of potential concentrations that were not measured during sampling events. 
                  This concludes my presentation on the long-term sampling bias factors, and I'm happy to consider any of your clarifying questions at this time.
                  DR. KENNETH PORTIER:  Ken Portier.  This might be an easy question.  In calculating the mean and the standard deviation for your average, how did you handle no-detects or censored observations?  I'm trying to get at if you did half the detection limit or something like that?
                  MR. CHARLES PECK:  So, because we were using the SEAWAVE-QEX sampling data set, we didn't go back in and adjust the values that may have been below a detection limit.  We just used the values that came out of the chemographs.
                  DR. KENNETH PORTIER:  So the whole experiment assumes no no-detects.  It assumes everything's detected down to infinitely small?
                  MR. CHARLES PECK:  That's correct. 
                  DR. ROBERT CHAPIN:  Tom.
                  DR. THOMAS POTTER:  Tom Potter.  On your comment of regarding the use in Tier 3, I'm really not clear as to how you're approaching this.  Are you taking the maximum sample bias factor for one of the pesticides or -- I mean, you're making some intercomparison between pesticides of different classes, et cetera, so you know, can you kind of amplify that a little bit?
                  MR. CHARLES PECK:  So this is sort of into the regulatory how we apply things, at this point, we're just trying to see that can we use them, but I would envision -- and this is me speaking personally -- I would envision that we would look at sort of the chemical properties of the chemical being evaluated, look at the four chemicals we have, potentially see how the fate properties are consistent with them, and then possibly use it that way.  But I think also we may end up talking about this in the next presentation when we look at the regressions that we did as well as the way to evidence approach that we're considering as well. 
                  DR. THOMAS POTTER:  Well, I'm kind of remembering this storm again.  I'm kind of remembering the outcome of those regressions and they weren't very encouraging.
                  MR. CHARLES PECK:  Exactly.
                  DR. THOMAS POTTER:  Wade into that.
                  MR. CHARLES PECK:  So one other thing when we get to the case studies, you'll sort of see an indication of how we intend to use it as well there.  
                  DR. ROCHELLE BOHATY:  This is Rochelle.  We're also asking that question on that.  
                  DR. VERONICA BERROCAL:  This is Dr. Berrocal.  Can I ask a clarifying comment on the previous slide where you said something about, yes, as few as four samples per year are needed to apply a sampling bias factor to estimate upper end long-term average concentration?  Why upper end?  You're just -- can you explain why you're talking about upper end?  
                  MR. CHARLES PECK:  So again, so the bias factors are designed to account for the uncertainty where you haven't sampled and so you're always going to be applying a number and trying to get a higher number with the bias factor.  And so generally when we do our drinking water assessments, we're looking at sort of, upper end, upper bound concentrations.
                  DR. ROBERT CHAPIN:  Okay.  
                  DR. REBECCA KLAPER:  This is Rebecca Klaper.  Just a clarifying question.  So the factors that you're calculating, would they be then used on a watershed by watershed basis or is it an aggregate one that gets used then across all of them?
                  MR. CHARLES PECK:  So we're going to start to talk about that in the next section I think.
                  DR. ROBERT CHAPIN:  All right.  So we've had three introductions -- three mentions of the next talk so it's time to do that.  
                  No, too late.  All right.  That's right.  Dr. Hartless.
                  
WATERSHED EXTRAPOLATION AND WEIGHT-OF-EVIDENCE APPROACH

                  DR. CHISTINE HARTLESS:  Thank you, Mr. Peck.  
                  Again, I'm Christine Hartless and I'll be talking about the watershed regression analysis that we looked at and the weight of evidence component of our work today.
                  I'll start by talking about the watershed regression, its goals, methods, and outcomes, then move to introducing the weight of evidence approach and its implications for drinking water assessments.
                  As a reminder of where we are in the overall method development process, after looking at and evaluating our short-term and long-term sampling bias factors, we're looking now to utilize that information as well as available watershed characteristics and fate characteristics of the pesticides that we have the sampling bias factors for to develop prediction equations for sampling bias factors that could then be used at sites with little monitoring data.  
                  In following up from previous science advisory panels, our goal here was to identify watershed characteristics and possibly fate characteristics that could be reliably used to estimate sampling bias factors for watersheds.  Then those identified characteristics and corresponding regression equations would be used to predict sampling bias factors for those watersheds with inadequate monitoring data for site-specific bias factor estimation.  
                  The work here focused on the data from the USGS sites that were used for earlier sampling bias factor development and evaluation.  Atrazine, carbaryl, chlorpyriphos, and fipronil had 98, 33, 30, and 28 sites, respectively.  There were 102 unique monitoring sites as some sites had multiple pesticides measured.  This is a slightly reduced number relative to the sampling bias factor analysis because there was some limited availability of some of the regression -- the regressor variables for a few of sites.
                  Watershed, catchment, landscape, and environmental characteristics were obtained from the StreamCat database.  These included such variables as watershed area, percent area classified as agriculture, urban area, and open water, et cetera.  We also included a few variables from outside the StreamCat data set such as average precipitation during May and June.
                  Prior to any regression analysis, we noted that there was a very large number of potential regressors.  We had 66 variables total and many of them were highly correlated with each other or had little to no variation across all of the selected sites.  Preliminary work was done using principle component analysis, correlation analysis, and graphics to reduce the initial set of potential regressor variables for use in the step-wide regression to a set of only 36 variables.
                  The pair-wise correlations between the 1-day, 4-day, and 21-day sampling bias factors and the pair-wise correlations of the sampling bias factors across all the different sampling intervals of 13, 17, 26, and 52 samples per year were all very high.  All of these pair-wise correlations were greater than 0.9; therefore, we elected to start with just one set of bias factors for the initial regression explorations.  We started with one-day sampling bias factor developed from random sampling and log linear interpolation with 28 samples per year which is comparable to a 14-day sampling interval.
                  For each of the four pesticides, step-wise regression was used to initially -- to identify initially important predictor variables.  Further exploration of the initial regressions was done using plots and influence statistics.  Final regression equations were focused on trying to find the best fitting, most robust model.  And note, for this stage of the regression analysis, we're only looking at the short-term bias factors.  
                  For example here, the best fitting regression with the highest R² was for atrazine with two predictor variables, percent agriculture in the catchment, as calculated in 2006, and the average precipitation in May and June in the catchment.  The figures show these two variables on the X-axis and the one-day sampling bias factor on the Y-axis, each of the points on the graph represents one of the USGS atrazine sites that were included in the regression analysis.  The orange point here is one site that was identified as being highly influential in the regression analysis with a very high sampling bias factor.  The blue line approximately represents the fitted regression line for these data with the influential site removed from the analysis.  The R² for this resulting model after removing the influential site was only 0.26.
                  Of the four evaluated pesticides, atrazine results as shown here in the regression, resulted with the highest R²; however, we did not have confidence that these regressions were adequate for prediction purposes.  We're looking for feedback from the panel on this conclusion as well as recommendations on how and if it is worthwhile to continue to explore this process.  
                  Another idea was to look across both sites and pesticides and use the pesticide fate properties for prediction of sampling bias factors.  These data sets here were very limited as the analysis was restricted to those site years where all four pesticides had sampling bias factors.  There were only five of 102 sites where this occurred.  If we relax the restriction to include those site years with any three of the four pesticides, we were able to include an additional 15 sites.  
                  This exploratory work here shows that there is some potential that the fate characteristics may be useful to predict sampling bias factors, but the data are too limited for strong confidence at this time.  The two characteristics with the highest potential for use in regression equations were aerobic aquatic metabolism half-life, which was negatively correlated with the bias factors as shown here, and the minimum reported terrestrial field dissipation half-life which was positively correlated with the bias factor.  But again, we don't have confidence in using any of these at this point.
                  Given the limited usefulness of the regression-based watershed characteristics and the regression based on fate characteristics for the one-day sampling bias factors, we chose not to explore these regressions for other short-term or long-term bias factors at this time.  We're looking for feedback from the panel on this conclusion as well as recommendations on how and if it is worthwhile to continue to explore watershed and fate characteristics as a means for predicting sampling bias factors.  
                  Specifically, some of the concerns that we would like for you to contemplate are sampling bias factors for each site were summarized over a several year window and those windows may not overlap across the sites included in the regression analysis.  In addition, the windows of data from each site may not overlap with some of the potential regressor variables that were included, specifically landscape type percent cover variables were based on data from 2006 imagery.  While some classes of landscape may not change much, like percent open water, other classes may change a meaningful percentage, a meaningful amount over a 20- to 30-year time span.  Some variables like average annual rainfall or average rainfall during May and June were based on data over a range of years.  In this analysis, we utilize data on average from 1981 to 2010.  
                  Finally, for this preliminary work with pesticide fate properties, is it promising enough to warrant the development of sampling bias factors for additional pesticides at these USGS sites to further explore these relationships.  
                  With no quantitative methods, meaning the regression analysis, to use to predict the sampling bias factors at sites with insufficient monitoring data, EPA moved to develop a more qualitative weight of evidence approach to evaluate the monitoring data.  This approach could be used for evaluating the spatial relevancy of monitoring sites and the associated monitoring data determined if it was applicable for use in estimating drinking water pesticide concentrations, applicability and usefulness of developing the SEAWAVE-QEX chemographs, and in the use of sampling bias factors to adjust existing monitoring data.  This weight of evidence approach is intended to be used as an interim process until more quantitative approaches are developed for sites with these minimal monitoring data.
                  In short, this weight of evidence approach was designed to answer these three questions when considering monitoring data in a drinking water assessment.  Are the sufficient monitoring data available to quantify concentrations in high usage areas?  This may include addressing questions such as, are the monitoring data collected from areas of known usage?  Are they collected during the timeframe when runoff into the waterbodies would be expected?  And are the data collected with a frequency such that the timeframe of concern could be adequately represented?
                  Second, can sites with monitoring data be used as surrogates for areas without monitoring data?  Here questions might be that are they adequate -- are there adequate data to developing sampling bias factors over that relevant timeframe?  And finally, are spatially relevant sampling bias factors available for use in adjusting these monitoring data?  Here one might ask if existing sampling bias factors from other sites may be relevant.
                  As a result, EPA feels that the following concepts or lines of evidence can be used to focus the evaluation of spatial relevancy of monitoring sites.  First, potential pesticide use sites and usage information can help EPA focus on areas where the pesticide is likely to be applied and potentially enter waterbodies.  EPA distinguishes pesticide use as the sites where the pesticide can legally be applied while pesticide usage as sites were observed pesticide applications have occurred.  
                  Pesticide usage data can be obtained based on review -- excuse me.  Pesticide use data can be obtained based on review of the pesticide labels and geographic data identifying where different crops are grown.  Pesticide usage data can be obtained through survey databases or national and state public resources.  
                  Second, EPA will consider a watershed and waterbody properties conducive to a pesticide reaching the drinking water intake.  Such evaluation could include the proximity of the monitoring site to the community water system, hydrologic connectivity of the monitoring site to the drinking water intake, soil properties and geology of the monitoring site that might impact the likelihood of runoff, and finally climatic factors such as precipitation.
                  Lastly, EPA will also consider the properties of the pesticide including how likely it is to be dissolved in water and move as opposed to how likely it is to bind to the soil and sediment.  In many cases, EPA will be able to employ available GIS technology and databases to help assist assessors in this weight of evidence approach.  
                  So here's a hypothetical example of how EPA might be able to employ this weight of evidence approach.  The figure to the right depicts a typical community watershed with the drinking water intake depicted by the yellow dot.  Two green dots depict the monitoring sites identified within the watershed boundaries where the pesticide has been detected.  Next, we can evaluate these data and this information by stepping through those various lines of evidence and questions outlined on previous slides.
                  First, we looked at the pesticide use and usage.  For this hypothetical example, the red depicts the agricultural areas where the pesticide can legally be applied according to the label.  Pesticide usage data, which was not shown in the figure, indicated the pesticide applications occurred in close proximity to those green monitoring sites.  
                  Second, evaluation of the NHD plus flow lines indicate that the water flows from the two sites -- the water from the two sites flows directly to the drinking water intake.  The estimated time of travel indicates that suspended pesticide in the flowing could reach the intake in a relatively short time, say, two to three days.
                  Third, for this hypothetical scenario, pesticide fate information indicated that half-life and water would be estimated at five to seven days.  That combined with the time of travel indicated that the pesticide in the water at the monitoring location could reach the drinking water intake location.  
                  Given this information, EPA would conclude that the two monitoring sites are spatially relevant with regards to evaluating pesticide concentrations at that drinking water intake.  Further evaluation of any available monitoring data from these sites would be done at the Tier 4 of the drinking water framework.
                  A second example, again, here is the drinking water intake and a new monitoring site was identified in the watershed boundaries where different pesticide of interested was detected, down in the very bottom of the figure, of the graphic.  
                  So next, we can again evaluate these data by stepping through the various lines of evidence justifications as outlined in the previous steps.  So the red here depicts the agriculture areas where this pesticide can be legally applied according to the label and pesticide usage data indicated that the pesticide -- however, if you'll see here that this one -- that the pesticide is not used in near immediately adjacent to that location.  
                  Second, the water from this new monitoring location does flow into the drinking water intake, but the travel time is expected to be much longer based on the flow data, estimated seven to eight days.  
                  And finally for this pesticide, fate information indicates that the half-life in water is estimated at only one to three days.  That combined with the time of travel indicated that the pesticide in the water at the monitoring location is less likely to reach the drinking water intake.  
                  Given this information then, EPA would conclude that this monitoring site is not likely spatially relevant with regards to evaluating pesticide concentrations at the drinking water intake.  
                  In conclusion, with regards to the implications of using watershed regressions and weight of evidence in drinking water assessments, the watershed regressions are not ready for use quantitatively in drinking water assessments; however, the weight of evidence approach can be used immediately to assist EPA in better understanding the spatial relevancy of the available surface water monitoring data.  This approach would be used at the Tier 4 level due to resource requirements and would allow EPA, using GIS databases and tools, to expand the use of monitoring sites and their data in drinking water assessments, whether, the data were used directly in SEAWAVE-QEX or in the development or application of sampling bias factors.  
                  After walking through this weight of evidence approach and a couple simplified examples, we're asking for the panel to provide feedback on ways our posed approach could be implemented in our drinking water assessments.  
                  Thank you and I'll take any clarifying questions.  
                  DR. ROBERT CHAPIN:  Or any other questions.  Cliff.
                  DR. CLIFFORD WEISEL:  Cliff Weisel.  So a regression analysis, the way I think I saw it, assumes linearity.  A lot of the properties of these chemicals and the absorption properties are not going to be linear.  Did you consider how you might start using what you're learning from the way of evidence as to what might be some of the properties and see if you can do a semi-mechanistic understanding of what you should be doing in the regression?
                  DR. CHRISTINE HARTLESS:  I think that that would be a valid approach.  At this time, we only have the sampling bias factors developed for pesticides so there's not -- we don't have a lot to go on at this point.  So I think one the questions that we're asking to the panel is, do you think that there is enough going on with this approach and enough that we need to do more work and perhaps look at more pesticides to further look at that or are we spending a lot of resources for something that's not going to gain us much in the end?  
                  DR. XUYANG ZHANG:  Xuyang Zhang.  I have I question.  
                  DR. LISA NOWELL:  I'm Lisa Nowell.  Oh.  Ms. Zhang, want to go?
                  DR. ROBERT CHAPIN:  Hold on a second, Dr. Zhang.  
                  Go ahead.
                  DR. XUYANG ZHANG:  Okay.  Sure.
                  DR. LISA NOWELL:  Lisa Nowell speaking.  I just wondered if you had tried -- I realize there are only four pesticides, but did you look at multivari- you tried look at more than one variable at a time, combined, not just simple univariate correlations but look for -- try to build a multivariate model?
                  DR. CHRISTINE HARTLESS:  With the fate properties, no, because we only had four pesticides.  You're basically connecting the dots at that point.  
                  DR. ROBERT CHAPIN:  Okay, Dr. Zhang.
                  DR. XUYANG ZHANG:  In your regression analysis, I noticed that the pesticide usage is not included as one of the factors while pesticide usage was mentioned in the weight of evidence.  So is there a particular reason that you don't include that in your regression?
                  DR. CHRISTINE HARTLESS:  We do not have pesticide usage for specific pesticides on a watershed or catchment scale across the country for all of the different monitoring sites.  That's available on fairly -- some states have much better data than other states, and we also have that data available in general for overall pesticide usage, but not for specific pesticides at a national scale.
                  DR. XUYANG ZHANG:  Have you tried to -- instead of conducting regression analysis, using pie (phonetic), which is located in a watershed that has pesticide usage data, like for example, in California?
                  DR. CHRISTINE HARTLESS:  We did not look at that specific regression analysis.  We were trying to focus on looking at more broader scale national available -- looking at a broader scale.  I think that would be something that we could look at in the future in the panel suggests that is a path forward.
                  DR. XUYANG ZHANG:  Okay.  Thanks.  
                  DR. ROCHELLE BOHATY:  I think it -- Rochelle Bohaty here.  I think it's also important to add that when we get down to a state level where we might have more detailed use information, like from the state of California, we have very few sites to include in that analysis.  
                  DR. TIMOTHY GREEN:  Tim Green.  So the proximity of the monitoring station to water supply to me seems to come out of the blue.  If you had a good regression analysis, would you consider proximity?  It seems like you have two different things going on so I'm not sure why one replaces the other instead of somehow complimenting.  
                  DR. CHRISTINE HARTLESS:  I think our initial hope was that the regression analyses would be able to be used.  And as we move through the work, that was not playing out the way we had hoped in terms of having good strong predictive abilities for sites with less data.  So we were trying to figure out what are the best ways to move forward and make the most use of the data that we had and this was kind of our approach trying to figure out how to do that and fitting it into the broader -- in something that would also fit naturally into the broader framework of drinking water assessment, that tiered approach that we have -- that was presented earlier.
                  DR. TIMOTHY GREEN:  Okay.  And to follow up, so if you had a really nice regression relationship with SEAWAVE, would you then look at proximity in addition to that?
                  DR. CHRISTINE HARTLESS:  So the regression work that I presented here was using the sampling bias factors, not SEAWAVE.  I think that had that regression analysis panned out really well that the weight of evidence would be -- I think as we kind of develop through our process and I think the weight of evidence would certainly have a lot of added value, even if the regression equations were more predictive and we felt more comfortable using them.
                  DR. LISA NOWELL:  Lisa Nowell.  I know you already know this, but I just wanted to mention that the USGS has pesticide agricultural use estimates on the county's fate scale that could, with a fair amount of work, be adapted.  You could overlay the base boundaries, et cetera, but I think the fact that it requires a lot of work is part of the problem.  Also, it's just ag use so it leaves a big portion of the non-agricultural applications unaddressed.  
                  DR. REBECCA KLAPER:  Rebecca Klaper.  I'm just wondering if there's some way to improve your regression model, considering or dividing up the streams by the type of, you know, what order they are or base flow conditions?  I don't see those in there in the list of things that you did the regression against each of those things, the partial regressions.  
                  DR. CHRISTINE HARTLESS:  I think the only chemical that we had adequate data for to kind of do any subdividing would be atrazine.  The other chemicals we did not have.  We only had roughly 30 sites for each of those other three chemicals.  We wanted to try to find something that was more broadly applicable, but I think that looking at some further subgrouping for the atrazine may be a way to try to tease something out of these data.  
                  DR. REBECCA KLAPER:  Yeah, just because the stream characteristics can be so different even with all of the other factors that you're talking about based on the base flow of the stream.  Or have you tried something like decision tree analysis or something like that instead of doing the partial regression analysis?
                  DR. CHRISTINE HARTLESS:  I had done some decision tree analysis early in the work and I think that's where I was discovering that how many factors were so highly correlated.
                  DR. REBECCA KLAPER:  Right.
                  DR. CHRISTINE HARTLESS:  And that kind of approach allowed me to minimize my data set such that I had much less issues with correlation.  I did not go back and utilize the decision tree at that point.  That could be something that could be done with some further work.
                  DR. REBECCA KLAPER:  Thank you.
                  DR. THOMAS POTTER:  I probably could find this out by looking back on your list of regression parameters, but I thought it's just better to ask you because I think it's an important issue that will come up later on.  Irrigation.  Was that one of your regression factors?
                  DR. CHRISTINE HARTLESS:  I don't remember that one off the top of my head.  
                  DR. THOMAS POTTER:  It hasn't been discussed yet in our dialogue.  We're talking about flow and rain, but we haven't got to irrigation and I think we need to.  And we'll talk about that, I guess, in the answer to the charge questions.  But the other part is a management issue, a land management issue was tile drainage.  A lot of what we're looking at is in the Midwest where a lot of the acreage is tiled drained and that certainly could be a dominating factor in some watersheds and I just wondered if you'd taken a look at that?
                  DR. CHRISTINE HARTLESS:  I did not break the variables --
                  DR. THOMAS POTTER:  Yeah.
                  DR. CHRISTINE HARTLESS:  I did not break it down on tile drainage.  If you, as part of the panel, have good data sets that we could utilize to grab that information, we'd really appreciate it.
                  DR. THOMAS POTTER:  Yeah, I think probably some of it can be excavated somewhere.  
                  DR. ROBERT CHAPIN:  All right.  Look at your watch.  We have 10-minute bio-breaks.  So 10 minutes from right now we come back and dive in.  
                  
                   [BREAK]
                  
                  DR. ROBERT CHAPIN:  All right ladies and gentlemen.  Let's do this thing.  
                  All right.  We're going to have a case study now and we're going to have Dr. White from the EPA give us a walk-through of that pesticide with short-term exposure with considerations.
                  Katrina White.
                  
DRINKING WATER ASSESSMENT CASE STUDY   -   A PESTICIDE WITH SHORT-TERM EXPOSURE CONSIDERATION

                  DR. KATRINA WHITE:  It helps if you turn the microphone on.
                  Good afternoon.  My name is Katrina White and I'm a senior scientist with the Environmental Fate and Effects Division.  I'll be talking with you today about the case studies.  Jessica Joyce and Chuck Peck also contributed to the case study work.  
                  First, I wanted to go over an outline of our presentation.  I will start by talking about the purpose of the case studies and then discuss our approach and the methods used in the case studies.  Next, I will discuss the results and conclusions of the case study with short-term exposure considerations.  Jessica Joyce will then discuss the case study with long-term exposure considerations, and finally, Mrs. Joyce will also discuss the implications for drinking water assessments and lessons learned from the case studies.
                  Throughout the talks, for the short-term exposure considerations case study, we refer to the pesticide as Chem 1 and for the long-term exposure consideration case study, we refer to the pesticide as Chem 2.
                  We conducted the case studies to provide the panel with a practical example of how EPA conducts drinking water assessments using the framework and tiering process and to provide an example of the use of SEAWAVE-QEX in short-term sampling bias factors in our fighting drinking water assessments.  We also wanted to provide an understanding of the monitoring data available for analysis and how it can be used in risk assessment.  Finally, the case studies provide examples of the use of weight of evidence approach for assessing and integrating surface water monitoring data into drinking water assessments.
                  On the next slide, I will talk about how these case studies relate to the framework of conducting drinking water assessments.  Recall that Dr. Bohaty presented the framework for conducting drinking water assessments earlier today.  One of the charge questions relates to the clarity and organization of the framework and one reason the case studies are provided is to help understand the framework.
                  Recall that the framework presents a tiered process where at lower tiers you utilize simple inputs and conservative outputs and move to more detailed and realistic estimates of exposure at higher tiers.  As you move into higher tiers, the analysis becomes more regionally specific.  
                  In the case studies, we focus on Tier 3 and Tier 4 where new methods are provided.  In Tier 3, we propose applying sampling bias factors calculated for four other pesticides to monitoring results without daily sampling to count for the temporal uncertainty in the data, that is, not having daily sampling frequency.  
                  In Tier 4, we introduce the use of SEAWAVE-QEX to estimate daily concentrations from data with non-daily sampling.  This provides an exposure estimate for the site with enough data to run SEAWAVE-QEX and chemographs to calculate pesticide-specific sampling bias factors that can be applied to data that is not robust enough for running SEAWAVE-QEX.  Finally, a weight of evidence approach was developed to better understand the spatial relevance of our monitoring.  
                  For the case studies, we selected one pesticide with acute toxicity endpoint of concern and another with a chronic and cancer toxicity endpoint of concern.  We selected pesticides that required refinements at higher tiers and pesticides that had a monitoring data set available where SEAWAVE-QEX and sampling bias factors could be applied.  We wanted the case studies to be illustrative of the process but not too complicated as there are a lot of materials supporting the SAP.
                  To simplify the case study, we selected a subset of use patterns to evaluate and focused on the tiers of the assessment where the new tools we present here today are utilized, Tier 3 and Tier 4.  In Tier 4 of the drinking water assessments, we focused on two regions.  The case studies do not represent complete drinking water assessments and while the case studies are mostly based on a true story, some items were altered for simplicity.
                  Now I will change gears and focus on the short-term exposure case study.  The first step in the framework is understanding the background as part of scoping for a pesticide to determine what will be needed for a drinking water assessment.  
                  Chem 1 is an insecticide applied to residential areas, apples, citrus, and soybeans with most usage occurring in residential areas.  It has an acute drinking water level of concern of 34.5 micrograms per liter and previous drinking water assessments required higher tiered assessments.  Chem 1 has a lot of monitoring data available; however, data has non-daily sampling frequency and is unlikely to capture the peak concentration that occurred at the site.
                  In previous drinking water assessments, we characterized these monitoring data but could not rely on it quantitatively because it was not likely to capture those peak concentrations.  So today I'm going to walk you through how we can use SEAWAVE-QEX and sampling bias factors to quantify the uncertainty in this monitoring data, to use it quantitatively in the exposure assessment.  It is important to consider that while we have a lot of data, the majority of it is not robust enough that we can quantify the amount of uncertainty in the data.  We can only apply the tools we present today to about 25 percent of the monitoring data for Chem 1.  
                  Based on the environmental fate properties, Chem 1 is moderately mobile and not persistent in water.  Scoping indicated that the need for at least a Tier 3 exposure assessment.  At this tier, modeling is refined with the pesticide and water calculator and monitoring data are adjusting with sampling bias factors calculated for four other pesticides to account for the uncertainty in the data.  As the sampling bias factors are not specific to the pesticide of interest, this is considered a screen.
                  First, I will discuss the Tier 3 PWC modeling results.  PWC is pesticide in water calculator.  In Tier 3, the pesticide and water calculator is used to estimate potential Chem 1 concentrations in surface water.  We modeled typical application rates and numbers of applications where usage of Chem 1 is reported for citrus, apples, and soybeans.  The estimated drinking water concentrations are on the Y-axis and the use sites are on the X-axis.  Results indicated that estimated drinking water concentrations may exceed the drinking water level of concern for all crops evaluated.
                  Recall that Dr. Bohaty presented that we have more regional specificity as you move to the higher tiers in the assessment.  In Tier 3, monitoring data are analyzed for each HUC-02 region.  To account for uncertainty in the monitoring data with non-daily sampling, measured concentrations are adjusted using sampling bias factors.
                  So let me walk you through the process of the Tier 3 monitoring data analysis.  First, we filter the data by HUC-02 region.  We consider the site years with detections with at least 13 samples per year in the analysis because these are the data where we can quantify the uncertainty and apply sampling bias factors.  
                  Next, we categorize the data based on the number of samples collected per year.  Recall that Dr. Hartless explained that sampling bias factors are calculated for different sampling frequencies which were translated into a sample number per year, such as 13 to 16 samples per year and so on.  
                  Finally, measured concentrations are multiplied by the appropriate sample bias factor determined based on the sample number category and averaging period, that is, daily, 4-day, or 21-day average, to provide a concentration in which we assume that about 95 percent of the time the concentration would be below that value for the site year.
                  We use the sampling bias factor calculated for the four USGS pesticides presented by Dr. Hartless to estimate a daily average concentration.  Monitoring analysis in Tier 3 is considered a screen as the Chem 1 specific sampling bias factors may be different.
                  The results of the monitoring data with sampling bias factor adjusted daily average concentrations are graphed here.  The sampling bias factor adjusted daily concentrations are on the Y-axis in log scale and separated by region on the X-axis and compared to the drinking water level of concern, the red line, to focus the assessment on areas of the country where there could be potential concern.
                  Looking at this analysis, all HUCs with adjusted concentrations above the red line would move to Tier 4.  Those below the red line would not necessarily be passing because there may be too much uncertainty in available monitoring data to rely on the monitoring data.  For example, if there are very limited number of site years with data in the region, these regions would rely primarily on modeling; however, if adjusted concentrations were below the red line and there was enough relevant data, that region would not move to Tier 4.  For this case study, 15 regions exceed the drinking water level of concern and would move to Tier 4. 
                  In the short-term case study, we evaluated HUC-02 Regions 3 and 17 to simplify the case study.  These regions have high usage and sampling bias factor adjusted concentrations exceeded the drinking water level of concern.  Tier 4 requires the highest level of refinement.  Uncertainty in monitoring data is characterized using SEAWAVE-QEX in Chem 1 sampling bias factors.  Then a weight of evidence analysis is conducted.
                  So let's go through the monitoring analysis process in Tier 4.  The first step is to run SEAWAVE-QEX for the sites that meet the criteria that Dr. Hafner discussed earlier.  Next, the chemographs developed using SEAWAVE-QEX are used to calculate sampling bias factors.  As we only have a few sites where we could run SEAWAVE-QEX, results from all sites across the country are used to calculate the sampling bias factors, even if the site is in a region that did not move to Tier 4.  Here the assumed variability of across a region is assumed to be similar to the range of available data.  After calculating the sampling bias factors, they are applied to the monitoring results for site years with at least 13 samples per year for regions that move to Tier 4.  Finally, a weight of evidence analysis is conducted for the regions and sites with predicted drinking water level of concern exceedances.  This helps to understand the relationship of monitoring to usage in drinking water and whether monitoring data are available in vulnerable areas.
                  Of the roughly 8,000 sites with available monitoring data, only 32 sites, the pink dots on the map, could be evaluated using SEAWAVE-QEX.  Fortunately, most SEAWAVE-QEX sites are located in use areas and in drinking water watersheds.  There are two sites were SEAWAVE-QEX predicted daily concentrations are above the drinking water level of concern shown with the blue dots on the map.  One is in northern Virginia in an urban area and one in Alabama. 
                  Next, I will talk about how we determined whether SEAWAVE-QEX predicted a drinking water level of concern exceedance for a site.  
                  Recall we are using SEAWAVE-QEX to capture the peak concentrations that could be missed due to infrequent sampling.  Here is an example of the diagnostic plots for one site.  We show the drinking water level of concern using a dashed red line and the blue boxes are the uncertainty around the maximum estimated concentration across chemographs.  When the blue boxes are above the drinking water level of concern, there is a potential for exceedances at the site.  
                  For example, in 1999, the blue box is completely above the drinking water level of concern.  We have more confidence that this site might have a drinking water level of concern exceedance.  For 1997 and 1998, there is some potential for a drinking water level of concern exceedance, but it is less certain on whether it is likely to occur.  Finally for 2000, an exceedance was not predicted.  The site was determined to have an unlikely but possible drinking water level of concern exceedance. 
                  SEAWAVE-QEX results are used to calculate Chem 1-specific sampling bias factors.  The approach to determine one sampling bias factor for each site is to take the median across chemographs and the median across years.  This figure shows the short-term sampling bias factors on the Y-axis and each sample number category on the X-axis.  After calculating the Chem 1-specific sampling bias factor for the short-term case study, we apply the maximum sampling bias factor across sites to the monitoring results at other sites with at least 13 samples collected per year.
                  As expected for Chem 1, the higher the number of samples collected at the site, the lower the sampling bias factor.  Also note that some sampling bias factors are large, up to 288, indicating a lot of uncertainty in the monitoring data without daily sampling.  
                  Now we will move to the weight of an evidence analysis for Region 17 as an example.  Here we are looking at a map of the monitoring data with apple growing areas in red and where about 170,000 pounds of Chem 1 is applied per year.  Drinking water watersheds are outlined in blue.  Of the 55 sites with data, the 6 SEAWAVE-QEX sites shown in pink in the region did not predict a drinking water level of concern exceedance; however, 8 sampling bias factor adjusted concentrations shown in green exceeded the drinking water level of concern in the region.  Those sites were at high usage areas and we relevant to drinking water.  Based on this information, there's high confidence in potential drinking water level of concern exceedances in the area.  As some measured concentrations were just below the drinking water level of concern.  The tools we have presented today allow us to quantify our uncertainty in the available data and draw this conclusion.
                  Today I walked you through how the SEAWAVE-QEX and sampling bias factors were used to quantify the uncertainty in monitoring data.  The first step was to run SEAWAVE-QEX and we were able to do that for 32 sites.  In step two, we calculated the Chem 1-specific sampling bias factors which ranged from 2.3 to 288.  This highlights that we can have high uncertainty in monitoring results with infrequent sampling.  
                  Calculation of the sampling bias factors allowed us to consider data from 55 sites in Region 17.  When applying the sampling bias factors in step three, eight sites in Region 17 had a drinking water level of concern exceedance and some with little uncertainty.  When conducting the weight of evidence analysis, the sites were exceedances were in high usage areas and at sites that were relevant to drinking water.  Quantifying uncertainty in the available monitoring data provided us with confidence that there could be potential drinking water level of concern exceedances in Region 17.
                  Finally, I wanted to talk about some of what we learned about available monitoring data from this case study.  For Chem 1, we started out with a lot of data, but we could only use a small portion of that data with the tools we have developed to quantify the uncertainty in the data.  Across the United States of the 9,000 sites, we could use SEAWAVE-QEX for 32 sites and could apply sampling bias factors to results from 900 sites.
                  Another important item we found is that while the majority of data is in the most uncertain data category, there is too much data to discount these data but too much uncertainty to use these data quantitatively.  These data are considered with the understanding that these values may underestimate the potential for exposure.
                  We are asking the panel to provide feedback on how we used SEAWAVE-QEX in sampling bias factors in the case studies.  Now I will take clarifying questions on the short-term case study.
                  DR. RAYMOND YANG:  Ray Yang.  Could you go back to slide 152?  All right.  So we have these crops.  Let me just you a soybean as example because it covers most of the Midwest.  You say these are all above, all exceeded the DWLC.  Now, are these different chemicals combined or you're talking about one chemical, say atrazine, okay?
                  DR. KATRINA WHITE:  So this slide is one we did pesticide in water calculator modeling for one pesticide, and we --
                  DR. RAYMOND YANG:  Just one pesticide?  
                  Now am I correct, for any of this crop, it is likely that multiple pesticides are used?  In other words, we could be dealing with chemical mixture and if, let's just say soybean, we use three pesticide.  Does EPA do each of them separately and then use additivity to consider possible interaction or what do you do?
                  DR. KATRINA WHITE:  So I think we had a few questions there, so I'll start with the first one.  I think that there are multiple pesticides used on a single crop and we do expect that.  
                  I think the second one was, do we estimate exposure and consider exposure for mixtures in the dietary assessment?  And so for most chemicals, we consider them individually and do not do any type of additive analysis except there is a subcategory of assessments where we would consider, for chemicals that have a similar mode of action they would do what is considered a cumulative assessment.  So for the organophosphates, for example, they would do a different -- they would consider combined exposure for those and those would be a very specific assessment.  But those are outside of the scope of what we're doing today.
                  DR. RAYMOND YANG:  Thank you.
                  DR. ROBERT CHAPIN:  Claire.
                  DR. CLAIRE BAFFAUT:  This is Claire Baffaut.  Can you go back to slide 158?  All right and in Bullet 2 here, you say that there -- you assume that the variability across the nation is similar to the variability across the region.  I think I missed it, but could you explain a little bit how you used that statement?
                  DR. KATRINA WHITE:  So let me go to a map somewhere.  So we have 32 pink dots on that map, and we need to estimate exposure for all of those HUC-02 regions.  And if you go to any one individual region, there's a very limited number of pink dots.  And so we didn't think that we could only consider the data from one region to estimate exposure for that region because there are so few number of dots.  And that really simplified things.  And there is quite a bit of variate.  There are so many factors that influence exposure.  There can be a wide variability across a region and across the United States.  I think so for precipitation and HUC-17, I think Rochelle has often presented an example where precipitation is going to be very different across HUC-17.   
                  Did that answer your question?
                  DR. CLAIRE BAFFAUT:  Yes.  So to rephrase, you're using those 32 dots to calculate the range of sampling bias factors?
                  DR. KATRINA WHITE:  And using all of those and applying it to data and looking at a regional analysis.
                  DR. ROBERT CHAPIN:  Tom.
                  DR. THOMAS POTTER:  I'm still a little bit confused about Tier 3 so I'm going to go back to that.  When you applied the sampling bias factors for the four pesticides where you generated data using SEAWAVE-QEX, how did you decide which value to use for your compound-specific analysis for Chem 1?
                  DR. KATRINA WHITE:  So for Chem 1, and this is a case study example.  This doesn't necessarily mean this is what we would do.  It's just an example of how it could be done.  For Chem 1 for the short-term, we took the median of the median to get one SBF for each site.  And in the White Paper, they calculated SBFs, one value for each site for four chemicals, 100 sites for atrazine, 30 for the rest, and we took the maximum.  In this graph, the lower end of the bar is the minimum for across those SBFs in the White Paper, and the max is that highest value for the region times the maximum SBF.  Across all chemicals, all sites.
                  DR. THOMAS POTTER:  And the assumption I guess is that this is protected in from a regulatory perspective. 
                  This is Tom again.  Again, the assumption is that that approach is protected from a regulatory perspective?
                  DR. KATRINA WHITE:  That would be the assumption, yes. 
                  DR. ROBERT CHAPIN:  Okay.  Ran through that gauntlet line.  Okay, excellent.  Thank you very much.
                  All right.  And Ms. Joyce for the longer-term exposure considerations, a case study.
                  
DRINKING WATER ASSESSMENT CASE STUDY  -  A PESTICIDE WITH LONG-TERM EXPOSURE CONSIDERATIONS

                  MS. JESSICA JOYCE:  Good afternoon.  I might want to get us on the right slide.  All right.
                  Again, my name is Jessica Joyce and I am presenting the second drinking water assessment case study which highlights the quantitative use of monitoring data for developing long-term exposure estimates.  As with the acute case study, the first step in our drinking water assessment is scoping.  I will begin with some background information on the pesticide.  
                  We are calling this pesticide of interest Chem 2 which is applied primarily to weeds and citrus and other orchards which is depicted by orange and yellow on the pie chart.  The chronic and cancer drinking water level of concern for Chem 2 is eight micrograms per liter.  Chem 2 has a lot of non-daily monitoring data.  It may indicate drinking water level of concern exceedances though the majority of the data are not robust enough that we can quantify the amount of uncertainty in the data.  Therefore, based on previous modeling and preliminary evaluation of the monitoring data, a higher tiered assessment where tools, such as SEAWAVE-QEX, and sampling bias factors will be needed to reduce the uncertainty in the monitoring data.
                  Based on the pesticide fate properties, Chem 2 is moderately mobile and persistent in water and soil.  Scoping indicating the need for at least a Tier 3 exposure assessment.  At this tier, modeling is refined with a pesticide and water calculator and monitoring data are adjusted with sampling bias factors calculated for four other pesticides, the chlorpyrifos, atrazine, fipronil, and carbaryl, the USGS pesticides, to account for uncertainty in the data.  This is considered a screen as the sampling bias factors are not specific to Chem 2.
                  First, I will discuss the Tier 3 modeling results.  This is using the pesticide in water calculator and is used to estimate potential Chem 2 concentrations in surface water.  The estimated drinking water concentrations are on the Y-axis and the various use sites are on the X-axis.  Tier 3 modeling indicates that both the maximum 1- and 10 year annual concentration and the 30 year annual average concentration exceed the drinking water level of concern of eight micrograms per liter, that red line, for all uses indicating the need for more refinement. 
                  Now I will discuss the monitoring data.  At Tier 3, monitoring data are refined on a regional basis.  Pictured here are the hydrologic unit code 2 regions across the country.  To address the uncertainty and non-daily monitoring data, EPA could use the long-term sampling bias factor derived for the four pesticides from the USGS data.  
                  EPA applied sampling bias factors to the monitoring data in this case study by first filtering the monitoring sites by region.  We consider site years with detections with at least four samples per year in the analysis because these are the data where we can quantify uncertainty and apply sampling bias factors.  
                  Second, we categorize the data based on the number of samples collected per year.  Recall that Mr. Peck explained that sampling bias factors are calculated for different sampling frequencies which were translated into a sample number per year.  
                  Lastly, we adjust the measured concentrations by multiplying by the appropriate sampling bias factor, using factors developed from the four USGS pesticides.  This ultimately provides a concentration in which we assume that 95 percent of the time the adjusted concentration would be below that value for a site.  This step is considered a screen as the Chem 2 specific sampling bias factors may be different.
                  The results of the monitoring data with sampling bias factor adjusted annual concentrations are graphed here.  The sampling bias factor adjusted annual concentrations are on the Y-axis in log scale and separated by region on the X-axis and compared to the drinking water level of concern, the red line, to focus the assessment on areas of the country where there could be potential concern.  This figure indicates that there are potential exceedances in the drinking water level of concern, those that are above the red line in Regions 3, 15, 17, and 18; however, values that fall below the red line may not necessarily be passing if there's too much uncertainty in the monitoring data, such as when there are only a few years of site data.  Since these four regions exceed the drinking water level water of concern, they would move on to Tier 4.
                  The following conclusions can be made at Tier 3:  refined model estimated concentrations still exceed the drinking water level of concern in most regions for all uses.  The sampling bias factor adjusted concentrations are higher than the level of concern in Regions 3, 15, 17, and 18; therefore, these regions would move on to Tier 4.  
                  Now onto Tier 4.  Tier 4 requires the highest level of refinement.  For this case study, EPA focused on use of SEAWAVE-QEX, pesticide-specific sampling bias factors, and a weight of evidence analysis.  In Tier 4, the first step of processing the monitoring data is to run SEAWAVE-QEX for the sites that met the criteria that Dr. Hafner discussed earlier.  
                  Second, the chemographs developing using SEAWAVE-QEX were used to calculate Chem 2 specific sampling bias factors.  Since there were only a few sites where we could run SEAWAVE-QEX, results from all the sites across the country were used with the assumption that variability across the nation is similar to the variability across a region.  After calculating the sampling bias factors, they are then applied to the monitoring results in relevant regions that move to Tier 4 and had at least four samples per year to develop annual and 30-year averages.
                  Finally, a weight of evidence analysis is conducted.  This aids in understanding the relationship of monitoring data to usage locations and drinking water watersheds.  
                  The first step in processing the monitoring data at Tier 4 is to rune SEAWAVE-QEX.  As a reminder, it's designed to capture higher end concentrations with less than daily sampling.  Of the roughly 6,000 sites with monitoring data for Chem 2, only a fraction of these sites, 27 noted by the pink dots, had the data requirements to run SEAWAVE-QEX to generate chemographs and annual averages.  After running SEAWAVE-QEX for these sites, EPA determined that the annual average concentrations from these sties did not exceed the drinking water level of concern.  
                  Next, we explored pesticide-specific sampling bias factors.  These bias factors allow for more monitoring data to be used than with SEAWAVE-QEX alone and are calculated using the SEAWAVE-QEX chemographs.  This figure demonstrates the long-term sampling bias factors on the Y-axis and each sample number category, four to five, six to seven, and so on, on the X-axis.  The approach to determine one sampling bias factor for each site is to take the median across chemographs and the median across years.  And then the median sampling bias factor across sites was selected for use.  As seen in the figure, the higher the number of samples collected out of site, greater than or equal to 10 for example, the lower the sampling bias factor or uncertainty.  
                  Next, EPA conducted a weight of evidence approach to qualitatively evaluate spatial relevancy of monitoring sites to drinking water sources.  This weight of evidence analysis focuses where monitoring data overlap with potential use areas in drinking water watersheds.  For the purposes of this presentation, only Region 18, which covers the state of California, is discussed.  
                  This is a map of Chem 2 monitoring data with citrus growing areas in yellow and other orchards in red.  Of the 229 site years of data, three sites exceed the drinking water level of concern which is depicted by the green circles, two of which overlap in the top left-hand corner and one down below.  All three of these sites are located in a drinking water watershed that contain a large area of citrus and other orchards where potential use could occur.  Based on this information, there is high confidence in potential exceedances of the drinking water level of concern in the area.
                  In conclusion, when modeling suggests risk concern and in lieu of daily monitoring data, the first step is to fun SEAWAVE-QEX for sites that meet the criteria.  In this case study, only 27 sites could be run, and none had drinking water level of concern exceedances.  In step two, we calculated Chem 2 specific sampling bias factors which ranged from 1.1 to 7.  In step three, we applied pesticide-specific sampling bias factors to site years with at least four samples per year which allowed us to consider data from 154 sites in Region 18.  In step four, when conducting the weight of evidence analysis, the sites with exceedances were in high usage areas and at sites that were relevant to drinking water.
                  Ultimately, quantifying the uncertainty in the available monitoring data provided us with confidence that there could be potential drinking water level of concern exceedances in Region 18.  But before we switch gears, I will follow up with what we have learned about the available monitoring data from this case study.
                  On a national level, there are a lot of monitoring data for Chem 2, over 6,000 sites, and only 27 of them met the criteria to run SEAWAVE-QEX and about 1,500 sites could be adjusted with sampling bias factors.  The majority of the sites with detections of Chem 2 had less than four samples per year as noted by the 70 percent portion of the pie chart.  This is too much data to discount but has too much uncertainty to use quantitatively which highlights the importance of considering all relevant monitoring data.
                  Now on to lessons learned from both case studies.  For drinking water assessments conducted for pesticides with short-term and long-term exposure concerns and a lot of non-daily monitoring data, EPA considers SEAWAVE-QEX a useful tool to address uncertainty and non-daily monitoring data for pesticides with seasonal applications.  But the number of sites nationally that meet the minimum data requirements are limited and are far fewer on a regional basis.
                  SEAWAVE-QEX allowed for the calculation of short-term and long-term sampling bias factors.  Application of sampling bias factors permitted the quantitative use of much more data than previously could be used in drinking water assessments or by using SEAWAVE-QEX alone.  
                  A weight of evidence approach can be used in absence of a quantitative approach to address spatial uncertainty in the available data.  Additional monitoring data will be needed to reduce the uncertainty associated with available surface water monitoring data.
                  And this concludes the presentation on the case studies, and I will now take any clarifying questions.
                  DR. IAN KENNEDY:  Ian Kennedy.  Did you have any non-detects and if so, how did you deal with those?  Did they count in, say, the minimum of four samples?  
                  MR. CHARLES PECK:  So when you're looking at the monitoring data and the application of the bias factors, when we took the average we used the detection, the limit of detection that was provided with the monitoring data.  If we're looking at SEAWAVE-QEX and calculating the average, it was basically the values that were provided in the chemographs.  So if they were below the limit of detection, we didn't go back and correct for the limit of detection.
                  DR. IAN KENNEDY:  SEAWAVE-QEX allows for non-detects, correct?
                  MR. CHARLES PECK:  So SEAWAVE -- yes.  What it does is if it estimates that a value is going to be below the limit of detection, it pulls a random value from that down.
                  DR. IAN KENNEDY:  Thanks.
                  DR. ROBERT CHAPIN:  Cliff.
                  DR. CLIFFORD WEISEL:  Cliff Weisel.  If I heard you correctly, when you're doing Tier 3, you said you used some sort of estimate of factor with the recognition there was a screening system.  But how do you decide what factor to use because if you really underestimate it, you can get them all below your screening saying everything is good but might be wrong.  I'm sure you've figured that out, but I'm just --
                  MR. CHARLES PECK:  So in Tier 3, again, we go back to the four chemicals that we had in the USGS data set and we used the maximum concentration that was detected, or the maximum annual concentration that was derived in a HUC-02 and applied the minimum and the maximum sampling bias factors that we derived for the for chemicals to sort of get a bound of, were we above or below the DWLOC.  And I agree, in some cases we may be below, but at this point, we're considering that the values that we derived for those four chemicals would be protected enough and we're looking for your feedback on whether you agree with that or whether, you know, we need to derive more of them.
                  DR. CLIFFORD WEISEL:  Do you have any sense as to what are the properties of the chemicals that most drive these bias factors?  And I don't know, and so that's how I help make the decision whether the range you have are right or what you should be looking for.
                  MR. CHARLES PECK:  So we tried to do the regression analysis with the fate properties.  It didn't quite work.  We do know the properties in general of the chemicals, the four chemicals that we looked, and that is kind of one of the questions we would pose to the panel is, you know, looking at those properties and looking at the properties of the chemical we're assessing, would it be more prudent if a chemical, let's say, binds to soil and sediment if we used the bias factor for that as opposed to one that may be more soluble. 
                  DR. ROCHELLE BOHATY:  So to add on to that, if you go back to the sampling bias factor presentations, we had a graph by each of the chemicals were represented by a different bar and it showed the distribution.  You could generally see, for example, that insecticides had higher sampling bias factors.  You would expect that from the use profile.  And then you also need to consider the dissipation of those and generally the more persistent the chemical is the lower the bias factor is as well.  So the non-persistent chemicals would have higher sampling bias factors.  So if you had an insecticide that was less persistent, you're likely to have a much higher sampling bias factor.
                  DR. CLIFFORD WEISEL:  So if I heard you right, what category -- usage is one thing to consider.  Are the physical chemical properties another is persistent in the environment might be a third.  You have a beautiful table and I appreciate -- I think it was Table 6.1 or something like that, which is very important that might be something that we can looking at to help, but that's going to have to go back.  Thank you.
                  DR. ROBERT CHAPIN:  Ray.
                  DR. RAYMOND YANG:  Ray Yang.  Thanks for your presentation.  I have two questions.  First one, you have four example chemicals.  Now, as a toxicologist, of the four, I would have worried the most about chlorpyrifos.  That's because of my understanding of toxicology of these chemicals and so on.  Even if it's lower in terms of monitoring results, okay, does EPA take that sort of factor into consideration, you know, toxicity?
                  DR. ROCHELLE BOHATY:  So all of the information you're hearing about today is just in the context of developing and evaluating the models, except for the two case studies.  And so as they presented the way we would take into account the toxicity, it would be represented by the benchmark of the drinking water level of concern that was used in each one of those case studies.  But again, these methods and tools could be used in other pesticide risk assessments.
                  DR. RAYMOND YANG:  All right.  Second question, it's a follow-up of what I asked this morning in terms of the distribution of tier 1 to Tier 4 and so on.  So we are almost at the end of 2019, okay.  So in the year of 2019, how many chemicals EPA checked up to Tier 4 assessment?
                  DR. ROCHELLE BOHATY:  So today we have done a high level Tier 3-like assessment where we've delved into monitoring data to this kind of level and it's only been for two pesticides just to give you context.  It's not to say that it wouldn't apply to other pesticides.  That's why we're here today is to bring these methods forward before we start using them for other pesticides.
                  DR. RAYMOND YANG:  So 1 out of 150 or 100?
                  DR. ROCHELLE BOHATY:  Two so far.
                  DR. RAYMOND YANG:  Two.  All right.  Thanks.  
                  DR. ROBERT CHAPIN:  Tom.
                  DR. THOMAS POTTER:  Hey, this is Tom Potter.  Thank you for those two case study presentations.  They certainly clarified a lot of issues for me and I thought they were very well do so again, thanks for doing that.
                  One thing I've been wondering about in the Tier 4 analysis is -- and I say this because I've sat on the panel that first looked at SAM some years ago, so I'd like to know where SAM is hiding, and if indeed, he's going to become part of the Tier 4 process.  That's the Spatial Aquatic Model for those of you who we're on the panel or familiar with it.  
                  And then the follow-up is in the weight of evidence approach.  When you look at, for example, I think you had a HUC-17 was your target area, do you do a detailed analysis of use and use amounts and use profile, timing application and that sort of thing to further strengthen your insights into whether or not you have a DWLOC exceedance or not, et cetera?  So does that come into play?
                  DR. ROCHELLE BOHATY:  I can take the first question.  If you look in the framework in the Tier 4 section, we do mention that's where the Spatial Aquatic Model would be integrated into the framework.  Tier 4.  Rochelle Bohaty.  I didn't mention my name.  I'm sorry.
                  DR. KATRINA WHITE:  This is Katrina White.  I think you definitely would -- well, we would look more into the usage data and how it related to the monitoring and timing of when the monitoring occurred and that was presented by Dr. Hartless when she talked about the weight of evidence analysis.  
                  We do -- most of the usage data that we use is only available at the state level.  So for HUC-17 in this case study, there's a high usage on orchards.  So in 170,000 pounds, we know there's usage in that area and we have a pretty good idea of that.  But if the percent crop treated might be a lot lower and we only had it available on a state level, we might not know if there's really specific usage in the watershed and the timing of the applications we often might not know the exact timing.  So to the extent that we can and that we know when something would be applied, we would consider that, but we don't always know when and where exact applications are occurring.  
                  DR. THOMAS POTTER:  Can I just follow up?  This is Tom Potter again.  I think going back to that SAM assessment we did we spent several days talking about those issues, and I think some good ideas were put on the table as to how to approach timing of application.  We've heard earlier about use data being available nationally through USGS or the state of California and their program so, you know, I think there's a path forward there and I certainly would like to see the agency take it.  
                  DR. ROBERT CHAPIN:  Dr. Rodgers.  
                  DR. JOHN RODGERS JR.:  John Rodgers.  Very good job on the case studies.  I'd like to ask about is there is a similar effort going on the DWLOC, drinking water LOC side of the equation because these have got to come together some day or is all the effort going on the exposure side?
                  DR. KATRINA WHITE:  I'm not a dietary assessor and I wish Anna were here, but they do have a tiered assessment for the dietary assessment on the food as well.  And I don't have a good understanding of the type of refinement work they're doing at this time.  I'm not sure if someone else does.
                  DR. JOHN RODGERS JR.:  And I guess to follow that up, the basis for that question is the effort in terms of risk assessment, you know, you boil it down to one number.  We've taken some 90th percentiles here and some max values here and then you multiply those and then -- I mean, we do that in risk assessment all the time -- and we worry about compounding conservativism because these values are often linked correlative or whatever.  And so you worry about losing realism and I just wanted to know if an equal effort is being put into -- if we're going to do this ratio thing, or cook it down to a ratio where we've got a number that we've pulled from a model or pulled from monitoring data and a number that we're using to indicate potential for harm in terms of a DWLOC, how much confidence do we have in those numbers?  It seems to me like you need equal effort.
                  DR. KATRINA WHITE:  So they start at a higher tier assessment where they would use some simpler exposure estimates on the dietary side.  And then they'll move to measured concentrations in the food and they'll look at what -- conservative assumptions of what people are consuming and move to more specific assumptions on what people consume.  So there is that refinement and moving to a higher realism in the dietary side as well.  
                  Did anyone have something to add?
                  DR. ROCHELLE BOHATY:  I was just going to add that earlier when Anna mentioned -- she talked about the HED, the Health Effects Division also has a tiered process that tries to mirror what we're doing in our framework.  
                  DR. JOHN RODGERS JR.:  Very good.  I was also curious about the weight of evidence and use of that term because usually other branches use the term and they argue about whether you've got lines of evidence and whether there's a weight of evidence call.  And I realize that that's not necessarily a quantitative interpretation in this situation and that's fine.  But could this analysis be trumped if I went to a drinking water system and pulled a sample and measured a pesticide concentration in excessive of the drinking water LOC that the drinking water intake or at a customer tap?  
                  DR. KATRINA WHITE:  I think that would be a yes.  That's a solid line of evidence that we might have a concern, right?
                  DR. JOHN RODGERS JR.:  I just think I would consider the weight of evidence for a hierarchy of data.  I mean, in one case, you've got real data coming in from the field and in another case you've got estimates from a model and in maybe a higher tiered case or a higher line of evidence, you've got, it's in my cup.  I'm drinking it.  
                  So, I was thinking about that as I went through all of this and, you know, ultimately that's the call when we start pulling those samples.  Because the drinking water intake samples that I've pulled, a lot of them they're not in Lodec (phonetic) system so they're not well mixed.  They are stratified quite often.  The pesticides are not uniformly distributed at all.  You could certainly hit some sometime or miss some completely sometime and you know they're there.  You just can't find them, and you say, what kind of situation is that?  
                  But we have EPA-approved aquatic pesticides that people like me try go out and try to track.  I mean, we know where they're going, we know when they're going in the water.  We want to know whether they get the drinking water or not so we kind of got to jump on things a little bit.  We know how much went out there and so on.  
                  So I just was curious about the analysis that you did in those case studies in terms of the juxtaposition of sampling versus where the drinking waters' intake were and that was very enlightening.  
                  DR. KATRINA WHITE:  One of the things we considered the case studies was the time of travel based on the flow and you could also consider the fate properties in water.  So with the persistent pesticide, you could think about the time of travel in relation to that.
                  DR. JOHN RODGERS JR.:  It's very useful.  Thank you.
                  DR. ROBERT CHAPIN:  Dr. Nowell.
                  DR. LISA NOWELL:  Lisa Nowell.  Reiterate, thanks for the great presentations today.  They've all been really helpful in trying to process all of the reading I've been doing over the last while.
                  I have a question, I guess, thinking about the use of usage data or use data.  That's all you have in the case studies or in the weight of evidence.  Can you tell me a little more what you have to work with in terms of non-agricultural applications?  Do you have any market survey data or are you going with labels or you have to go back to the states for that?
                  DR. KATRINA WHITE:  So nationally its generally -- we consider the reliable at the state level.  It's based on market survey data.  And then in California, we have a robust status set for licensed applicators.  And then some states we might be able to ask for more specific usage data and they might -- some other states have some data that they could share with us, but we'd have to -- it's not necessarily available readily in the database.  
                  For non-agricultural usage, there are some surveys for that, but it's not as -- they aren't as robust as they are for the agricultural side.  And sometimes we might ask for some sales data or some more specific information from registrants, but that's much more difficult to get and much more uncertainty in that data.  
                  DR. LISA NOWELL:  May I ask a follow-up?  So Lisa Nowell again.
                  So would that be part of the weight of evidence stage basically?  And I'd love to be reminded about your case studies.  You had, I think, Chem 1 had substantial residential use and Chem 2 did not.  How did you factor those in separately in the case studies?  
                  DR. KATRINA WHITE:  So for Chem 1, there was estimated about 3.8 million pounds applied in residential areas and about, I think -- I'm going to get the numbers wrong -- about 100,000 pounds -- well, about 700,000 pounds, I think, applied in ag.  So most of the usage was in residential areas.  And what we consider is that we have a lot of uncertain in the usage data and so it's hard to say where that 3.8 million pounds is being applied and we didn't have that information available in the case study.
                  DR. ROBERT CHAPIN:  Dr. Yang.
                  DR. RAYMOND YANG:  Ray Yang.  Quick question -- 
                  DR. KATRINA WHITE:  Can I do a little bit more follow-up?  For Chem 1, the sampling, there's much higher detection frequency in urban areas and all but two of the sites, SEAWAVE sites, where we had detections were -- they had less than 20 percent ag in the watershed.  So the usage data and detection frequency for Chem 1 help to tell a story.
                  DR. RAYMOND YANG:  All right.  I'm going back to my question.  Quick question about -- does it require fair amount of competency in computer programming for a scientist to run this SEAWAVE-QEX modeling?  
                  DR. SARAH HAFNER:  No, it's pretty simple and straightforward to run.  It's the interpretation of the diagnostic plots that takes some work.  
                  DR. RAYMOND YANG:  Thank you. 
                  DR. ROBERT CHAPIN:  Okay.
                  DR. SARAH HAFNER:  And also the data preparation takes time, but it's not difficult.  
                  DR. RAYMOND YANG:  Is it programmed in R -- Ray Young again.
                  DR. SARAH HAFNER:  Yes.
                  DR. RAYMOND YANG:  Okay.  Thanks.
                  DR. ROBERT CHAPIN:  All right.  We have Dr. Bischof.  Mr. Bischof you get the Job award for patience today.
                  All right.  So we're going to move to a surface water monitoring program from Washington.
                  DR. ROCHELLE BOHATY:  Okay.  I'm going to start.  This is Rochelle Bohaty back to introduce Mr. Matthew Bischof with the Natural Resources Assessment Section of the Washington State Department of Agriculture.  We've been working with Mr. Bischof for more than two years.  He's been an integral part of our team contributing primarily to the evaluation of SEAWAVE-QEX using alternative covariates as well as the development of SEAWAVE-QEX standard operating procedures.
                  In his role at Washington State Department of Agriculture, Mr. Bischof monitors for pesticides in eastern Washington and he's going to provide you with an overview of the state's monitoring program as an example.  His presentation is not an endorsement that all monitoring programs need or should resemble their program, but we invited him here to present as he is part of our team and he offers a unique technical perspective having used the tools you heard about today and offers a practical experience with the conduct of surface water monitoring sampling and program design.  
                  
DEVELOPMENT OF SURFACE WATER MONITORING PROGRAM

                  MR. MATTHEW BISCHOF:  Thank you, Rochelle.  Thanks for the opportunity to speak here.
                  So you've just heard about two pesticide case studies that demonstrated the use of SEAWAVE-QEX.  
                  Can everybody hear me fine?  Oh, sorry.  Okay.  There.
                  All right.  So the case studies demonstrated the use of SEAWAVE-QEX in the sampling bias factors in drinking water assessments.  I will now present how these tools could be use in the design of the surface water monitoring program.
                  So on this slide, I've outlined my presentation beginning with the purpose of my involvement followed by describing an example monitoring program.  I will then broaden the discussion to general considerations when designing a surface water monitoring program and how SEAWAVE-QEX and sampling bias factors can be used to help overcome monitoring program challenges.
                  So my role has been to provide technical guidance on the conduct of surface water monitoring programs as well as support the evaluation work on SEAWAVE-QEX.  To this end, I will describe Washington State's surface water monitoring program as an example of an ongoing pesticide monitoring program.  I will then step back to highlight important surface water monitoring program considerations.  I will also bring forward practical limitations in the conduct of a monitoring program and how SEAWAVE-QEX and sampling bias factors can help augment these challenges as well as highlight the utility of these tools for Washington State.
                  Washington State Department of Agriculture Pesticide Monitoring Program consists of ambient monitoring during the typical pesticide use season.  Sampling occurs weekly or biweekly from March to August or into September/October, depending on the year, resources, as well as watershed.  
                  So this slide shows a map of Washington state with the 2019 monitoring sites as red dots including nine sites in eastern Washington and seven sites in western Washington.  The green overlay on the map represents agricultural crop land.  There's also influence from urban and mixed-use areas depending on the sampling site.  
                  Now our sampling site locations are driven by some Salmonid co-occurrence.  We are currently monitoring for about 150 different analytes, and this includes pesticides as well as some transformation products.  So note the lack of monitoring sites in far eastern Washington.  The goal of our program is to add additional monitoring sites further east but is currently limited by budget.
                  SEAWAVE-QEX has influenced our program by identifying ways we can improve our program.  The Washington State Department of Agriculture uses a tiered approach to determine sampling frequency depending on its detected pesticide concentrations.  While this tiered approach is different from the one you heard about in the drinking water assessment framework, the concept is the same, to conserve and best allocate resources.  
                  The Tier 1 when a new site is added to the program, the site is sampled every other work during the application season for three years.  The site is moved to Tier 2 sampling if at least one detection is greater than Washington State Department of Agriculture benchmark which is based off of an EPA or other water quality benchmark with a safety factor applied to it.  
                  At Tier 2, samples are collected weekly during the time period when higher concentrations are expected.  Tier 3 sampling is dependent on program budget and happens when best management practices are implemented.  So 2-3 monitoring is used to assess the effectiveness of best management practices.
                  Now as a result of using SEAWAVE-QEX, some sites have been established as long-term monitoring sites and are in agriculturally important regions.  The site is sampled weekly and may be monitored for 10 or more years for the purpose of identifying pesticide trends and changes in pesticide use or agricultural practices.  
                  Stepping back from Washington State's program, there are several factors to consider when designing a surface water monitoring program that will influence if, when, and how the data can be used by the conducting organization as well as other organizations.  Listed on this slide are a few considerations that are important when developing a monitoring program.  
                  There are different ways to collect samples, composite, grab, or passive.  The most common method is a grab sample; however, a collection of composite samples is increasing due to reduction in costs associated with sample automation.  A composite sample is where multiple samples are collected throughout a specific time period, for example, daily or weekly and combined to develop an average concentration over that duration.  
                  Site characteristics that may influence when or how often samples need to be collected include precipitation, soil vulnerability, use profile, and land practices such as irrigation.  The relevance of the sampling site to areas where you're trying to protect or represent also needs to be considered.  
                  The number of analytes needs to be thought through.  Often this needs to be a balance decision-making progress that considers resources as well as available methods and detection limits.  It is important for a program to document quality assurance and quality control protocols along with the analytical methods.  It is also important to make sure the detection limit is below the level of concern. 
                  There are several considerations when determining the ideal sample frequency, as I mentioned previously.  Site characteristics play a role as does the exposure duration of concern.  For example, one or more -- for example, more intensive sampling may be needed for pesticides with acute or short-term durations of concern, for example, 1-, 4- and 21-day average compared to chronic or long-term durations of concern such as 365-day average.  So when considering sample timing, it is important to think about application timing and environmental conditions that impact runoff.
                  In some cases, the sampling window needs to extend beyond the application season because, for example, the rainfall event necessary to derive runoff does not occur until much later in the year compared to the application season.
                  So monitoring programs need to target use areas that could impact source drinking water.  Furthermore, programs are also more robust when all residues of toxicological concern are considered.  For example, if the parent pesticide is expected to transform quickly in the environment to a more persistent and mobile transformation product, having data for the transformation product is imperative.  
                  Other considerations for monitoring programs include when the samples are collected compared to the seasonal pesticide occurrence window in collecting data for other covariates such as flow.  Taken all together, the sum of these factors can quickly snowball into an intensive sampling program that agencies or organizations may not have the resources to do; however, the SEAWAVE-QEX can be used to reduce the burden of intensive sampling at sites of interest between interpolating between sparse sampling to estimate daily pesticide concentrations.  In cases where SEAWAVE-QEX cannot be used, sampling bias factors can also help inform monitoring decisions.
                  First, I'm going to discuss how SEAWAVE-QEX can be used to help inform the design of surface water monitoring program and there are a few things to consider in a program design when planning to supplement the data using SEAWAVE-QEX.  One consideration is that SEAWAVE-QEX requires at least three years of monitoring at a given site, although non-consecutive years is acceptable if the timeframe and the covariate data matches the pesticide data; however, more years of data will help capture the range in variability and weather end also use patterns.
                  If you have many years of data, it may be useful to split the data for SEAWAVE-QEX to get better model fit.  As described in the SEAWAVE-QEX standard operating procedure, it is important for the book end or anchor years to have robust data.
                  The minimum number of samples that need to be collected for use in SEAWAVE-QEX each year and the detection rate will likely vary by pesticide properties, use profile among other things, and therefore needs to be reviewed periodically.  Factors that can influence these decisions include the likelihood of the pesticide to run off versus absorb to soil, the persistence of the pesticide in soil, and water and the residues of concern. 
                  SEAWAVE-QEX generally performs better when the measure samples represent more of the year rather than collecting, for example, 12 or more samples within a short timeframe; however, for some pesticides, the detection rate will drop too low when sampling outside of the usage season or too long after precipitation events.  Therefore, if the sampling program is planning to use SEAWAVE-QEX on a specific pesticide then the properties will need to be carefully considered.  If a non-targeted approach is taken and the sample number is limited to the minimum, that is about 12 samples, then focusing the sampling to the use seasons and likely increases the number of pesticides from those samples that may be usable in SEAWAVE-QEX.
                  SEAWAVE-QEX requires samples that are on a daily time-step, like the covariate data, which could be accomplished with either grab samples or daily composite samples; however, composite samples over more than a day would not be suitable for SEAWAVE-QEX.  Additionally, while passive samplers are a useful monitoring tool, they are not recommended to be use in SEAWAVE-QEX currently.
                  Similarly, the flashiness of a waterbody, and that is a waterbody having rapid or short-lasting changes in streamflow following runoff, needs to be factored in how frequently a site is sampled so that these sites are sampled more frequently or that there is flexibility in the sampling routine, such as possible precipitation event focused sampling.  This becomes more important for site and pesticides that have a significant correlation between concentration in the short-term flow anomaly when running SEAWAVE-QEX.  
                  The last output figure of the SEAWAVE-QEX diagnostic plots provides a correlation time scale, or CTS.  This parameter can be used as a rough gage as a relative comparison of the duration of time which concentrations are correlated.  This can inform the sample frequency necessary for a site and pesticide combination.  This parameter captures all the variables that impact the concentration of the pesticide over time at a given site, for example, flow an environmental and transport properties.
                  Pesticide use patterns should also influence the timing of sampling whether there is consistent use season or sporadic use driven by past pressure.  This includes considering the use sites, for example, urban uses compared to agricultural uses.  Also, to use SEAWAVE-QEX, a daily covariate needs to be collected.  As we saw in the evaluation of SEAWAVE-QEX, streamflow is the preferable covariate for use followed by staged then precipitation, noting that precipitation might be needed for certain systems where streamflow and stage aren't appropriate.
                  The covariate needs to be daily during the time period that the pesticide samples are taken but are not required for the full year.  For example, if pesticide sampling takes place from April to October, streamflow does not need to be collected after October; however, for the midterm flow anomaly to be calculated, the covariate data needs to be collected 30 days prior to the first sampling event.
                  So one of the biggest challenges for adopting SEAWAVE-QEX for some programs is collecting continuous flow data.  Continuous flow data is expensive and time consuming to collect and ensure accuracy.  While streamflow is preferred, if necessary, collecting stream stage may be a cheaper alternative for use in SEAWAVE-QEX if the covariate would otherwise not be collected.  
                  Additionally, data from USGS data stations or other agency stations can be leveraged when possible.  Other challenges with flow data are that it may not be the best predictor of pesticide concentrations at all sites, as not all sites exhibit a natural flow behavior, and may include periods of back flow or no flow.  In these situations and other covariates of just precipitation maybe an option or better. 
                  As with all daily field sampling, there are unavoidable challenges, such as equipment malfunctions or natural causes, like freezing streams that prevent a complete daily record of streamflow and may result in that streamflow record needing to be infilled before it can be used in SEAWAVE-QEX. 
                  Now, there are various methods for infilling occasional missing flow data for use in SEAWAVE-QEX.  One is a method used by SEAWAVE-QEX on flow data pulled from USGS gage stations from the R package waterData.  The figure on this slide shows an example of infilling flow using a waterData package.  
                  Flow in cubic feet per second is shown on the Y-axis and the record day is shown in the X-axis.  You can see here at the yellow arrow is an example of one segment of the incomplete flow input that was infilled using the waterData program.  
                  Now, when using USGS flow data in SEAWAVE-QEX, the model automatically pulls the data and interpolates missing flow data using this package.  However, if you're using flow data that you have collected yourself, you must interpolate the missing days with a method, such as waterData before it can be used in SEAWAVE-QEX. 
                  Additionally, use of this package keeps the flow data infilling consistent across covariate data sources so that external covariate data and USGS gage station data are treated similarly.   
                  So these next couple of slides highlight the work we did in identifying SEAWAVE-QEX alternative covariates to streamflow. 
                  This slide is showing SEAWAVE-QEX seasonal wave fits to the same pesticide data using two different covariates, streamflow and stream stage, as shown on the left and right of the slide, respectively. 
                  The site used to produce these seasonal waves is a stream located in western Washington, so on the wet side of the state.  That exhibits a natural flow pattern.  There is some irrigation used in the watershed that contributes to the stream source from ground water or pulled from a creek.  But in general, it follows a natural flow pattern. 
                  So the monitoring site has a gage station that provides continuance streamflow and stream stage data.  We compared these covariates in SEAWAVE-QEX with the monitoring data and found that stream stage is a good surrogate for streamflow. 
                  The results from these simulations were evaluated as Dr. Hafner described in her presentation.  However, for simplicity, these figures illustrate the similarities in the model selected by SEAWAVE-QEX.  Where both covariates is shown by the similar wave shapes and the position of the season shown in gray in the two figures. 
                  We also looked at measured precipitation as a covariate compared with streamflow.  And the results for streamflow are shown in the figure on the left and results, the measured precipitation are shown on the right.  
                  The site used for this comparison to make the seasonal waves shown here is in eastern Washington.  It's on the dry side of the state.  And so the site is influenced by irrigated agriculture and sourced by canals and laterals. 
                  This site does not follow a natural flow pattern.  Precipitation used for the covariate comparison here came from agweather.net, which is a weather station maintained by Washington State University.  Precipitation data was collected from four weather stations within this watershed. Precipitation data from these sites was used to calculate daily average watershed precipitation and considering the type of travel.  
                  Again, comparing measured precipitation to streamflow in SEAWAVE-QEX showed that measured precipitation may be a suitable covariate to use when streamflow is not available or if it is not appropriate given the hydrology of the site. 
                  In this example, while the seasonal wave looks similar in both plots shown, when you look at the other diagnostic plots, precipitation is a better covariate compared to flow for this site. 
                  Although SEAWAVE-QEX can be a valuable tool for leveraging resources, it may not always be possible to collect enough samples to meet the minimum requirements for the model at every site. 
                  Additionally, even if you meet the data requirements, some pesticides may be more difficult to fit using SEAWAVE-QEX.  In these cases, sampling bias factors can be used to make more use of the data.  Sampling bias factors can be used as a screen, requiring fewer years of data to determine which sites may require more intensive marketing.  If there is a sampling bias factor concentration that exceeds the level of exposure concern. 
                  When using sampling bias factors inform a monitoring program, the pesticide and the associated duration of exposure certainly needs to be considered.  The pesticide of interest has a longer duration of exposure concern, less sampling may be possible if sampling bias factors are considered.  This strategy could also be used if the program wanted to increase the number of sampling sites.  
                  There are many benefits to incorporating SEAWAVE-QEX in sampling bias factors into monitoring programs to better understand real-world exposure levels without intensive sampling. 
                  SEAWAVE-QEX can be used to estimate concentrations occurring between infrequent sampling events, that is nondaily.  This is particularly useful for pesticides with short-term exposure durations where the sample frequency needs to be high, yet is resource prohibited. 
                  Sampling bias factors can be used to help understand the range of uncertainty in available data for infrequent sampling.  Sampling bias factors can also be used to gage the added value in increasing the sample frequency at a site. 
                  Taken together, these tools can be used to optimize monitoring programs.  For example, in some cases, fewer samples need to be collected, and as a result, resources can be reallocated to increase the number of sites that are monitored. 
                  Washington State Department of Agriculture is considering this approach for the purpose of expanding our monitoring program further east in Washington.  
                  In addition, these tools can be used to assess the added value in increasing the frequency of monitoring at a given site. 
                  SEAWAVE-QEX can also be used for trend analysis to leverage the data we have and gain insights into what is happening in the environment and what outreach is to growers is effective.  These tools can increase the utility of the data beyond the organization, collects the data as well as for use by other organizations.  
                  All right.  Questions? 
                  DR. KENNETH PORTIER:  One of the things I've been thinking about is why the regression between the site characteristics and the SBFs have such low R².  One of the ways to increase, at least your competence in the regression would be to sample more in the extremes, to increase the range of the characteristics you look at.  And that kind of tells me that you'd need to go to site characteristics sites that you aren't currently monitoring, because they may not be the sites where you expect to see a lot of action, right.  Right now, you're design is places where you know you have a lot of ag use.  You have stream flow.  You put a monitoring station.  So one of the take homes from this whole modeling process might be requiring you to go somewhere else.  
                  Is that something that a state would look at, like, Washington, say, I don't know I'm going to put a million dollars in a marginal site over here?  I'm just wondering. 
                  MR. MATTHEW BISCHOF:  Yeah.  A million dollars, that'd be nice.  Well, the thing is, we do also devote a lot of time and energy into outreach to growers because they don't know what's in the water unless we're sampling and then we're able to tell them and explain it to them.  
                  So we look for watersheds where we're likely going to see pesticides where outreach can be effective.  So those are the sites where we can make a difference and that's where we put our value, I guess.  I mean, it would be nice to monitor in beautiful mountain streams too, but the biggest difference we can make is at sites where we're likely going to see pesticides. 
                  MR. KENNETH PORTIER:  Yeah, but sometimes knowledge is not gained by going where you currently are, right.  Sometimes you have to go where you aren't.  And I know that's a hard communication issue, but I think I hear the answer to your question.  The answer is no. 
                  MR. MATTHEW BISCHOF:  Yeah.  Yeah.  Good question though.  Thanks. 
                  DR. CLAIRE BAFFAUT:  Hi.  This is Claire Baffaut.  So I have three questions, actually.  Is there -- so you talked about missing flow rate, missing discharge values.  Is there a way to deal with streams that go dry? 
                  MR. MATTHEW BISCHOF:  Well, I guess none of our streams really go dry.  I mean, then you just don't have samples to collect, I guess.  
                  I mean, we do partial years, but then you run into the issue of, you know, not having enough samples to fulfill the requirements of SEAWAVE-QEX.  
                  DR. CLAIRE BAFFAUT:  Sure, but I can think of several streams that, you know, every year maybe in August or September, they'll go dry for two weeks.  
                  MR. MATTHEW BISCHOF:  Oh, okay.  
                  DR. CLAIRE BAFFAUT:  And then it rains again, and it flows again. 
                  MR. MATTHEW BISCHOF:  Yeah.  Yeah, that's interesting to think about.  I can't think of -- we don't have any sites that do go dry. 
                  DR. KENNETH PORTIER:  This is Ken Portier.  The 2012 atrazine study, we spent at least a day and a half talking about dry run streams and what you can and can't do with them.  And of course, the answer is you can't do much.  I mean, you run out of water.  
                  MR. MATTHEW BISCHOF:  Yeah.  
                  DR. CLAIRE BAFFAUT:  Okay.  So my second question was have you thought about running SEAWAVE-QEX for wet years together or drier years together? 
                  MR. MATTHEW BISCHOF:  No.  I have not.  But that can be a good idea to, I mean, for further investigation because especially in the west, we have had periods of dryness.  
                  DR. CLAIRE BAFFAUT:  Yeah.  I also think that it might change the way the pesticides are applied and when they are applied.  So it might actually change the wave itself. 
                  MR. MATTHEW BISCHOF:  Yeah.  In 2015, we had a very particularly dry -- well, it was a warm winter, but we had near normal precipitation but no snowpack.  And so snowpack in Washington is relied upon as basically a reservoir for feeding the streams.  And so we had very little influence on those streams and we were expecting to see some differences in pesticide detections, but it seems like -- it seems like we didn't -- there wasn't a big contrast from year to year, with looking at 2015 compared to others. 
                  But in terms of looking at chunks of years separately, wet versus dry, I haven't done that, but that could be worth looking into. 
                  DR. CLAIRE BAFFAUT:  Finally, my last question is you or anybody else has anyone used SEAWAVE-QEX for fall applied pesticides, like pesticides that are applied now, basically, October or November? 
                  DR. SARAH HAFNER:  So the only one that we have or the only example that we have was cyanazine, which is in the White Paper briefly.  And it's a poor example.  Not because it's fall applied, but just as a chemical it had a summer peak and a fall peak.  But other than that, we haven't looked at fall applications, specifically.  
                  DR. ROBERT CHAPIN:  Tom. 
                  DR. THOMAS POTTER:  Well, in southern Georgia and northern Florida, we actually have some rivers that go underground at certain times of year, but I'm not going to ask you how to deal with those. 
                  MR. MATTHEW BISCHOF:  Thank you.  
                  DR. THOMAS POTTER:  What I am going to ask you is that I've been digging through the White Paper and over several days and I still have not been able to find where you take into account time of travel in the context of precipitation data.  Is that -- can you guide me to that, or can you tell me, you know, what it is verbally? 
                  DR. SARAH HAFNER:  If you give me a moment, I could give you the page number. 
                  DR. THOMAS POTTER:  Yeah. 
                  DR. ROBERT CHAPIN:  Do you have a specific question about that? 
                  DR. THOMAS POTTER:  Yeah, we do.  Yeah, what is it?  I mean, what's the time of travel?  I mean, there are -- actually, there is another approach to using precipitation data in SEAWAVE-Q that was used by Johnson et al in 2011.  And they actually combined flow and precipitation with the wave regression and seemed to get very good data fits.  They looked at Washington state data, a lot of California data and I'll read the citation into the question and answer period tomorrow.  But I just think there may be a lot of other ways to think about this.  And so my specific question is, how did you handle this -- and I'll go back -- I need to go back, I guess if it's in the White Papers while I'm here I'll dig into it tonight and we'll look at it. 
                  But I also wanted to explore the broader question, you know, other ways of incorporating precipitation into the model, including combining it with flow to get better data fits.  
                  DR. SARAH HAFNER:  So section 9.1.3 discusses the model precipitation and there's a figure in 9.3 that talks about the time of travel. 
                  DR. THOMAS POTTER:  Okay.  Thank you.  
                  DR. ROBERT CHAPIN:  Ray.  
                  DR. RAYMOND YANG:  Excuse me.  Ray Yang.  Thanks very much for your presentation.  Thank you also for collaborating with EPA.  I have a question.  This is a two-part question.  I'm asking you as well as EPA colleagues, so you don't have to answer if you don't want to, okay.  
                  We all know the state of California is having multiple, multiple wildfires, and in large area and some are encroaching upon crop land, okay.  Combustion of pesticide residues from my perspective, is extreme oxidation reaction, okay.  Therefore, new chemicals might be synthesized.  
                  Now, in a situation like this, and this will happen again and again, okay.  Part 1, how are you going to deal with this in terms of monitoring?  Part 2 is SEAWAVE-QEX flexible enough such that the parameter could be modified to accommodate this type of scenario? 
                  DR. ANNA LOWIT:  Sorry.  I was gone for an hour, guys.  I do apologize for being away from the table for an hour.  
                  DR. ROBERT CHAPIN:  That's all right.  we had a wild party, and somebody has to clean up the room. 
                  DR. ANNA LOWIT:  I know you did.  I heard it was fabulous and I missed a lot of important things.  So that is a practical question and one that I know a lot of people put some thought into.  But I think it's important that we don't -- regulatory science is about doing the best we can at a moment in time with what we have in front of us.  And we've brought you a White Paper with a lot of thought into it and a lot of hard work. 
                  And going down the rabbit hole of what if this and what if that, and you know what if a meteor comes in and the world explodes.  And I'm being a little bit facetious and I apologize for that, but the fire will change the fate properties, which will fundamentally change how we assess a chemical.  There's just fundamental changes in a situation like that, that really have nothing to do with SEAWAVE.  That would really fundamentally change how we might evaluate the fate properties, whether it's volatilization and movement in the air.  
                  There's a lot of other things that go with these fires that would, in terms of air quality and everything else.  So although, that's a very thoughtful question, I think we would respectfully ask that we maybe stick to the questions that we've asked.
                  DR. CLIFFORD WEISEL:  This is Cliff Weisel.  Again, thank you for your presentation.  And this is probably due to my lack of knowledge.  When we're looking at the precipitation versus streamflow, one of the fundamental properties you need to understand precipitation is to make sure the watershed that you are defining only goes to the stream that you're thinking about and not others within the state of Washington or elsewhere.  How good do you really know that?  And how variable might it be?  What might you miss looking at that from a water -- just a water balance system? 
                  MR. MATTHEW BISCHOF:  So I use precipitation data from four weather stations within this watershed, which is four weather stations is quite a few within a watershed.  You're not always going to find that many.  
                  DR. CLIFFORD WEISEL:  Now, that's to say how good you know the rainfall.  My question is a little different.  How good do you know the watershed for a stream, which is a simpler, might be a simpler question?  Do you really know for this stream, this is my entire watershed going into it and it doesn't go to another one, or there might be another area that -- if you know 90 percent then that tells me a lot.  If there are times that it's maybe you're 70 percent sure of it, that has a quite a bit of uncertainty to precipitation as to this.  And maybe the Washington is not the right answer to look at that.  
                  DR. REBECCA KLAPER:  Doesn't the USGS has a pretty good map of all the surface watersheds, anyway.  Groundwater is a little bit more difficult, but --  
                  DR. LISA NOWELL:  Yeah, and there are definitely watersheds where there's big water transfers.  You know, in the arid west, you're going to get a lot of trouble setting a drainage area.  So there definitely could be complications in some areas.  I'm sure Matt knows about that more than I do on that. 
                  MR. MATTHEW BISCHOF:  Yeah, so for this site where I did look at precipitation, this one particular site it receives excess canal water periodically, so irrigation water is delivered through canals.  And this one, particular, some of the canals it can have a six day time in travel, so you'll have growers periodically taking water from this canal and they may have to -- they need to fulfill the needs for the growers.  And so a grower might need water and where they're pulling water to feed into the canal, way up at the top, the grower may not have time to wait for that additional water to come through.  So they have different reservoirs throughout that also feed into the canal.  
                  So periodically there's excess canal water and that will spill into this one stream that I was looking at that doesn't -- and that's why it doesn't exhibit a natural flow pattern because of excess irrigation water that feeds into it, so you can have a spike in the hydrograph and it's not due to a rain fall event or anything.  It's just excess canal water that's feeding into it. 
                  DR. CLIFFORD WEISEL:  Any idea how that might affect your ability to predict what the pesticide concentrations in the water system are, either by stream or by the precipitation? 
                  MR. MATTHEW BISCHOF:  Well, it certainly makes it a little more difficult to identify a relationship between the hydrograph and the chemical because this canal water, it feeds growers that are both conventional and organic.  So you don't really want -- you don't want chemicals in this canal water. 
                  And so which is why sometimes you see a negative relationship between pesticide concentration and the hydrograph.  So and in the case that I showed, you actually saw a positive relationship between the precipitation with the time in travel incorporated into it and that chemical.  
                  So yeah, in irrigated areas, you are going to have a lot of complexities with hydrology of some of these sites.  
                  DR. ROBERT CHAPIN:  Dr. Zhang? 
                  DR. XUYANG ZHANG:  Hi.  Yeah, I second that.  I want to comment.  I think the same thing occurred in California, especially in the central valley.  Like the San Joaquin River watershed because of all this canals and the direction of the water flow doesn't really follow the elevation.  So that adds more complexity to probably the StreamCat data set that EPA uses and the regression analysis of sampling bias factors, I think because the NHD watershed boundary may not be accurate.  Especially in areas with very low relief, very flat areas.  And the boundary of the watershed doesn't really -- what really follow the elevation.  So that's just a comment. 
                  And I have another question regarding the case study.  So you used to extrapolate the sampling bias factors based on a few sites that can apply with SEAWAVE-QEX model.  And then use that sampling bias factor to extrapolate to the rest of monitoring sites within the region, right? 
                  So I think there's a large uncertainty involved in this extrapolation and I wanted to understand more about that.  And do you know the range of variability among different sites in the USGS dataset that you have calculated the sampling bias factor using the four pesticides?  What's the range of variability among different sites? 
                  DR. KATRINA WHITE:  This is Katrina.  I think we agree that there's uncertainty that we're taking sampling bias factors taken for some specific sits and applying it to other sites.
                  And I'm going to ask Christine or Chuck who did the sampling bias factor work if they want to talk about some of the variation in the sampling bias factors they calculated for the USGS chemicals.  
                  DR. CHRISTINE HARTLESS:  So this is Christine Hartless.  So certainly when we're doing the estimating the sampling bias factors either for the short term or for the long term, there was quite a bit of variability in those values that were estimated both within a site year, where you're looking amongst those realizations within a site year as well as across sites.  Or excuse me, across years within a site and as well as across sites.  And you could see that in some of the graphs that we showed earlier today.  And it goes into much more detail in the White Paper.  
                  And I think -- 
                  DR. XUYANG ZHANG:  I was trying to -- yeah, I was trying to see if there's like a number showing the magnitude of differences.  But I searched through the White Paper.  I couldn't find any information about that.  If like for example 10, 10 fold or 100 fold in difference? 
                  DR. CHRISTINE HARTLESS:  I think it depends upon whether or not you're talking about the variation across, say all realizations or not.  In the White Paper, probably for the short-term sampling bias factors, a place where you would find that, would be where we looked at the mixed model analysis and there should be some summary statistics in there looking at the -- that would summarize what the variability is for those different variance parameters.  
                  DR. ANNA LOWIT:  So this is Anna Lowit.  This seems like a question; it might be helpful if we just looked in the paper overnight and follow up first thing in the morning to give a more accurate answer to the question.  If that's okay with the Chair. 
                  DR. XUYANG ZHANG:  Okay.  That will be fine. 
                  DR. ROBERT CHAPIN:  Perfect.  
                  DR. XUYANG ZHANG:  Yes.  Thank you.  
                  DR. ROCHELLE BOHATY:  If there is a specific question -- this is Rochelle Bohaty -- in the White Paper.  If you let us know the page that you're looking at, that would be helpful.  
                  DR. KENNETH PORTIER:  This is Ken Portier.  You might also look in the supplemental files under the USGS regression results is where you can see the SAS results for the mixed-model analysis, that Dr. Hartless was talking about.  So you can actually find those estimates there.  
                  DR. XUYANG ZHANG:  Do you have the page number? 
                  DR. KENNETH PORTIER:  No.  It's in the supplemental files.  So you go to supplemental, USGS, regression, and somewhere in there is the SAS program and then the SAS output.  
                  DR. ROBERT CHAPIN:  It's why God made a glass of wine.  Sit down with a glass of wine and your computer.  
                  DR. KENNETH PORTIER:  this tells you how much I've spent, time. 
                  DR. ROBERT CHAPIN:  I am completely impressed. 
                  All right.  We're going to move on.  All right.  Let's see, Dr. Bohaty, can you wrap us up for this afternoon?  This impressive series of presentations? 
                  
SUMMARY AND WRAP-UP

                  DR. ROCHELLE BOHATY:  Hi.  Rochelle Bohaty here again.  So I want to thank you all for listening to all of our presentations.  And I did have several slides that I planned to previously give, but I'm going to skip those for the sake of time.  And also, I think our active listening skills, at least mine, are starting to wear thin a little bit. 
                  But I want to quickly wrap up.  So you heard about a couple of tools, sampling bias factors, SEAWAVE-QEX as well as a weight of evidence that we can add to our toolbox.  And that these would just be another couple or few tools to add into that toolbox among others that we already have and use on an ongoing basis in our drinking water assessment. 
                  We're trying to do our best with the tools and the data we have available to make use of the available surface water monitoring data.  And as Dr. Lowit mentioned earlier, we're in the later stages of our 15-year review cycle.  As such, we need some practical feedback from you folks on how we can best integrate the tools that we have and make the best use of the available data that we have. 
                  We do have several challenging pesticides remaining that we need to conduct drinking water assessments for as part of that review process.  
                  So aside from that, we're interested -- that's a short-term need -- but we're also interested in your recommendations on what we can consider on a long-term basis. 
                  That can be including improving the methods and the tools that we use in our drinking water assessments, as well as recommendations you have on improving monitoring data so that we have better monitoring data to use in our drinking water assessments in the future.
                  So with that, if you have any clarifying questions, I can take those.  
                  DR. ROBERT CHAPIN:  No clarifying questions.  Okay.  That's -- sorry, that's a rabbit hold we've been down already today.  So we're done with clarifying questions. 
                  Okay.  So the -- I'm going to use the chairman's prerogative and say, break, shmake, we're late.  We're going to move to public comments.  If you need a bio break, take it yourself.  We're going to move to public comments. 
                  Dr. Lowit, I think you're sitting in the seat that is going to be occupied by some of the -- 
                  DR. ANNA LOWIT:  Yes, I will happily move back again.  
                  DR. ROBERT CHAPIN:  There you go. 
                  Okay so -- 
                  DR. ANNA LOWIT:  So should our -- normally, our whole team vacates so the public commenters come up. 
                  DR. ROBERT CHAPIN:  Okay. 
                  DR. ANNA LOWIT:  So I think that's what we'll do.  Our whole team will vacate.  
                  DR. ROBERT CHAPIN:  Okay.  
                  DR. ANNA LOWIT:  And they'll just sit here. 
                  DR. ROBERT CHAPIN:  Cool.  
                                       
                                    [BREAK]
                  
                  DR. ROBERT CHAPIN:  Okay.  So the first person up is going to be Dr. Perkins from Waterborne Environmental.  And as soon as the space is available, you can work yourself to the table.  I want to thank our EPA colleagues for number one, a buttload of work, making the White Paper and all the stuff that got sent us out to us.  And then you're calm, collected, presentations today covering a lot of ground.  I said break shmake.  Daniel, all right.  So Dr. Perkins.
                  So as soon as your slides appear, as if by magic, then you can -- there we go.  Cool.  Sweet.  Technology is wonderful when it works.  All right.  The microphone is yours.    
PUBLIC COMMENTS

                  DR. DANIEL PERKINS:   All right.  Thank you very much.  My name is Daniel Perkins and I hold a Ph.D. from the University of Florida, Gainesville.  I spent my professional career as a regulatory scientist for two different patricide registrants and now as senior consultant at Waterborne Environmental. 
                  We recognize the tremendous effort and thought that has been dedicated to the means by which monitoring data will be included in the FIFRA Drinking Water Assessment as well as to the risk assessment process, overall.  
                  We submit that the work presented by the agency falls into two categories.  First, how to interpret the value of monitoring data in order to quantitatively use it in the risk assessment.  And two, how that value compares with other lines of evidence towards risk base decision making. 
                  I will address these two categories specifically.  Related to the first category, how to interpret the value of monitoring data.  Several comments will be provided.  
                  First, tools such as SEAWAVE-QEX were presented in the EPA docket materials to demonstrate that upper end concentrations for monitoring data could be estimated by SEAWAVE-QEX.  More information could be provided to define the utility of SEAWAVE-QEX as related to prediction on certainty associated with different monitoring data sampling characteristics and design, given that many analysis we've seen today have been termed preliminary.  
                  Additionally, some data required by SEAWAVE-QEX may unnecessarily prohibit its application to some monitoring data, specifically related to streamflow requirements.  In our limited SEAWAVE-QEX testing on relatively high variability concentration time series data sets.  Results suggest that the source of streamflow data is less important than including streamflow at all. 
                  In fact, the SEAWAVE-QEX requirement to include streamflow data as a covariate may be unnecessary in some systems.  This may create opportunities to use alternative time series imputation techniques for which streamflow is not required or available, such as ordinary and universal kriging as published in the literature on this topic.  
                  It remains to be seen if the addition of covariates that are not colinear with flow might improve prediction accuracy.  However, the effort required to explore additional covariates that would replace streamflow is perhaps not cost effective considering that the highest potential payoff might only apply to an unclear number of relatively sparse monitoring data sets.  
                  We also support further testing of SEAWAVE-QEX be completed toward its ability to adequately represent temporal trends associated with reservoirs.  Reservoirs are relatively common with approximately 25 percent of all flowing waters having a reservoir.  And likely exhibiting hydrodynamic behavior similar to a lake a pond rather than a flowing stream. 
                  We urge further SEAWAVE-QEX model performance evaluation against concentration data that reservoir, measured at reservoirs that represent drinking water sources.  In relatively large data sets that contain both nontargeted and targeted monitoring data, the upper percentile concentrations are nearly always represented by targeted monitoring programs in highly vulnerable areas. 
                  Additional guidance on the amount, type, and quality of data that would be acceptable for decision making would be helpful as there is likely a point at which there may be a limited benefit of even long term targeted monitoring. 
                  While this SAP focuses on monitoring data specifically, we'd like to better understand the degree of detail and complexity that HED or the Human Effects Division will be able to accommodate in higher tier aggregate risk assessments.  It is clear that EFED will generate exposure estimates across a range of data quality and processing techniques. 
                  It seems as though, HED may need to update risk assessment techniques in their higher tier assessments.  But this was not clear in the information provided to date.  We support the idea that all monitoring data be used to its potential and the characterization of drinking water exposure, as well as in the aggregate human health risk assessment.
                  Beyond statistical and interpolation tools to support the use of monitoring data and risk assessment, other approaches are available and can even offer additional information beyond imputation techniques presented by the Agency for this SAP. 
                  Specifically, physically based modeling can be used to identify the value of monitoring data and support monitoring data through more technical calibration or traditional calibration and validation models that account for dynamic and static physical processes. 
                  Physically based modeling approaches also offer the advantage of not only characterizing the value of monitoring data within its temporal bounds, but also simultaneously providing a framework to estimate long term exposure profiles.  They are especially useful when applied to high vulnerability areas that are protective of other lower risk areas. 
                  Second, related to the second category, more specific guidance on the prioritization schemes of different information and data sources would be helpful in the weight of evidence approach. 
                  It seems obvious that the highest certainty data would be ranked to have the most weight in a weight of evidence framework.  Thus it is our opinion that high certainty monitoring data in context with acceptable uncertainty, should supersede scenario based modeling assessments for risk decision making. 
                  Furthermore, data obtained from targeted monitoring specifically designed to address a regulatory concern should be the highest priority in the weight of evidence framework and if designed properly, characterize the highest vulnerability in use areas that can be representative and protective of other geographies.  
                  Finally, nontargeted monitoring data may still be weighted or ranked in the context of the standard EPA scenario modeling and sample bias factor adjusted monitoring values, monitoring data. 
                  And we think that the weighting of information that is considered in the weight of evidence approach should be an explicit parallel topic.  Thank you. 
                  DR. ROBERT CHAPIN:  Excellent.  Thank you very much.  
                  Okay.  Dr. Mosquin from RTI. 
                  DR. PAUL MOSQUIN: Thank you.  Oh it's the first slide.  Okay.  All right.  Before I discuss any issues related to bias factors and modeling, it's worth considering other methods which take advantage of available data to characterize extremes outside your concentration profiles. 
                  For example, our method, which is based on survey sampling techniques estimates medians of distributions of site year percentiles using most of the available data, including site years where there is insufficient data for a direct estimate. 
                  For example, the method could use either of the data sets depicted here where sampling is too infrequent to estimate either the 90th or the 99th percentile reliably.  Analysis such as these may be useful at earlier tiers in the assessment processes as they can use much more of the available monitoring data. 
                  Moving on to bias factors.  The next slide provides a simple definition of the bias factor.  Given the sampling distribution for theta, an estimator for a target quantity theta, the bias factor theta over c if the c is the 5th percentile of the sampling distribution. The product bias factor in theta then provides the limit of a upper one-sided 95 percent confidence interval for theta.
                  Alternatively to put it another way, on your repeated sampling, which is not possible in practice, the successive interval is zero through the first bias factor adjusted value and then zero through the second bias factor adjusted value and so on, will contain theta 95 percent of the time, both for nominal and actual coverage. 
                  And important point here is that this is the definition of a random interval, not a plane estimate.  Of course, exact calculation of the sampling bias factor requires the sampling distribution to be known, which is not true in practice.  
                  In practice, the true bias factor is unknown and must be estimated using other sources of information.  
                  This gets to our first point.  Although sometimes correctly describes as interval estimates, the White Paper also describes bias factor adjusted values using point estimate terminology. 
                  For example, point estimates, protective estimates, or estimate with confidence.  Although suggestive of point estimation, this sampling bias factor adjusted values can be more than an order of magnitude larger than a simple point estimate, which can confuse readers. 
                  Furthermore, interval estimates, if considered as point estimates have sample size dependent bias, decreasing with n.  For example, using an upper 95 percent normal interval for the mean, we have u -- if we were to take that as an estimator for the mean -- is equal to x-bar, plus a bias depending upon n. 
                  The existence of sample size dependent bias is suppressed when bias factor adjusted value represented as a single number.  What is n?  Is it five?  Is it 150?  Or a bias factor adjusted value is a varying n are summarized as a group.  For example, the bending of samples of size 26 to 51.  Thus if numerical values are to be reported, we recommend that the following standard of statistical practice, bias factor adjusted values be stated as interval estimates.  That you provide both point and interval estimates, both should be useful in an assessment. 
                  Also, any associated decision role for screening should be stated in terms of the interval and not a protective estimate.  A terminology that we think should be avoided as it is unnecessary.  
                  In addition, if presented as interval estimates, we note that actual coverage may be poor for the four reasons listed here.  If also accompanied by a point estimate, we note that the analogous method for creating a point estimate of multiplicative factors, as we called it, was shown in our earlier study to be outperformed in many cases by log-linear interpolation of the sample data. 
                  For these reasons, we only recommend the use of bias factor adjusted values for screening at this time.  
                  If bias factor adjusted values are used for screening, certain sampling design and target quantity combinations may have extremely high false positive rates.  Because the current White Paper approach uses a maximum of medians of bias factors, it is expected that false positive rates are higher than in our earlier paper. 
                  In separate work, we found the White Paper approach to be approximately equivalent to using the 90th percentile of a reference set of bias factors, so you calculate a set of bias factors and then based upon that set of bias factors, you choose some value which you're going to apply to the sample data as your sample bias factor. 
                  And in previous works is that value has been the mean or the median and the method that's been proposed in the White Paper, according to this analysis, which is not provided, is -- we have found it to be approximately equivalent to the 90th percentile of that reference set. 
                  To illustrate a scenario of many false positives, the X-axis of the figure gives true target quantity values for a test set of 46 site years.  The Y-axis gives bias factor adjusted values. 
                  For a given site year target quantity value, all 90 possible bias factor adjusted values are arising from the systematic every 90 days sampling design are plotted.  
                  The vertical and horizontal lines indicate the decision boundary of ten parts per billion, which divides the region into four screening outcomes.  A false positive region for true negatives.  A false negative region for true positives, and two correct result regions. 
                  Clearly in this case, the false positive rate is very high.  More generally, the next figure gives false positive and false negative rates for varying values of the decision boundary for annual maximum using the same test set of 46 site years.  The X-axis provides the possible decision boundary values and the Y-axis, the overall false positive or false negative rates.  Plotted points are true target quantity values in the test set.  So those are the black dots in the lower left of the plot. 
                  Notably, there can be very high false positive rate within the distribution of true values and this rate may persist well above the distribution of true values.  We also see that infrequent sampling such as every 90 days, tends to have higher false positive rates.  
                  And note that this approach, which was described in detail on the right hand side because I only had five minutes, is -- does not depend in any way on the SEAWAVE-QEX estimation.  That's been eliminated here because we're using EMP data, which is everyday sampling within the summary so you can calculate what you think are these true bias factor values exactly more or less. 
                  The next figure shows the same plot for the maximum 21 day rolling average.  False positive rates are somewhat lower, but still high within the distribution of true values and can be persistent beyond. 
                  Finally, the last plot shows the results for the annual mean.  Again, there are high false positive rates within the range of the data extending beyond the extreme outlier at about nine parts per billion. 
                  I note that these false positive rates must have contribution for more than the extreme site year.  And that is that if you see the plotted values beyond nine parts per billion, since those false positive rates are greater than 1 over 46, I believe then that indicates there's contribution from some of the other sites, which are down much further in the distribution. 
                  The next slide -- 
                  DR. ROBERT CHAPIN:  Okay.  So you're well over your five, so can you bring it to a close, please?
                  DR. PAUL MOSQUIN:  Yeah.  I'm actually getting there right now.  
                  DR. ROBERT CHAPIN:  Excellent. 
                  DR. PAUL MOSQUIN:  Although I -- all right.  My next two slides provide examples from the acute and chronic case studies showing positive screens to be more likely, i.e., likely false positives in smaller sample sizes. 
                  I'll skip that one. 
                  Two other concerns, which should be investigated further are the correlation of the log bias factor and the log target quantity identified in our earlier study and the impact of moderate to high level censoring on bias factoring calculations. 
                  So here is the summary slide to emphasizes some of these points.  And I'll just mention the first of these.  The terminology, protective estimate, confounds point and interval estimation and is not necessary for the interpretation of bias factor-adjusted values, i.e. interval estimates, or their use for screening.  And furthermore, the terminology of bias factor is somewhat unfortunate.  It arises, I think, from the downward bias of the standard estimators which is say, for example, linear interpolation is bias downwards, and as a result, there is a desire to correct for that bias.  So quantity was created which is the bias factor, and that corrects for the bias.  But it goes even further to provide you with a confidence interval. 
                  And it would be much clearer, maybe to refer to something like this, than say, an interval factor, which would help for the use and interpretation since it does provide an interval estimate and that's the end of the time.  Thanks.  
                  DR. ROBERT CHAPIN:  Excellent.  Thank you very much. 
                  All right.  Dr. Aldworth, you have five minutes. 
                  DR. JEREMY ALDWORTH:  Thank you.  I'm just trying to see how this thing work.  Okay.  Oh, how does this thing work. 
                  DR. ROBERT CHAPIN:  It's still coming up.  The magic needs to happen up here and then you'll have it.  
                  DR. JEREMY ALDWORTH:  Sure.  
                  Okay.  So I'm going to comment on the SEAWAVE-QEX model, which I found to be a very interesting model and in fact, I think it gets the modeling fundamentals pretty much right, at least tested on the atrazine data that we have looked at.  
                  For example, it applies a log transformation to your chemical concentrations and to atrazine data that works because for models like this, they really work better if you have normal data, so some sort of transformation that does that is good. 
                  It assumes that noise is zero and we have actually investigated that at atrazine data, and we have found that the noise is approximately zero.  We did this looking at a semivariogram analysis.  And the nuggets is just about zero.  And we also did an independent measurement or analysis of replicated data in the lab.  And that kind of bears that out. 
                  And having zero noise is potentially very good for prediction.  And by prediction I mean infilling missing days in the time series.  And the third thing is that we want to see -- serial correlation is always present in atrazine data at any rate.  And if you include that in the model, that's all, it's going to prove your prediction. 
                  So those modeling fundamentals with sort CA QH (phonetic), I see those as strengths.  I see those as strengths. 
                  Now, getting to the fixed trends, there may be a couple of questions like the linear trend term, which is always a useful term but what if the trend is nonlinear over several years? 
                  The White Paper then suggests that you split the analysis into multiple time frames.  But that might be unnecessarily restrictive.  What if that renders those time frames just not analyzable and it's also less efficient?  There are other ways to do -- to work with nonlinear trends. 
                  And so are some of the seasonal wave term.  What if that pattern varies across the years?  Or what if chemicals are used sporadically?  In fact, in one of the case studies, if you read the document, it says the usage is sporadic.  So there are a couple of questions about some of those fixed terms.  
                  And the one that might cause a bit of concern is streamflow.  There are actually two terms.  There's median term flow anomaly and short term flow anomaly.  
                  Now, in usage periods, flow and concentration may be positively correlated and that's a good thing for the model.  But in non-usage periods, they may be negatively correlated.  So how would you square that in the same model?  That could be an issue. 
                  We have also found that short term flow anomaly is right skewed and that may cause overpredictions of extrema, which is what you are interested in.  And when you're working in the log scale, those overpredictions may not be that serious.  But you want to back transform to the original scale, and it blows up those overpredictions.  We have seen that. 
                  I'm going to go on to the next slide.  Paul and I have been working on something similar.  We call it the universal kriging model.  But it's very similar to what is done in SEAWAVE.  We use the same fundamental modeling assumptions, log transformation of the concentrations, virtually zero noise and we apply serial correlation very similar to what is done in SEAWAVE. 
                  The real differences are in the fixed terms.  We have much simpler fixed terms.  We just have a linear and quadratic trend terms, and for trend and seasonality and for short term flow anomaly, the problems we saw with skewness, we saw thought there may be a better approach.  
                  We came up with the Box-Cox transformation of the flow term that basically removes the right skewness.  It makes it symmetric.  And that, we found, reduces the overpredictions of extrema.  
                  Our much simpler model can be fit to a single year, as opposed to having to have three years.  So it's more useful when data limited or if you want to focus on a particular year of interest.  Maybe there's just a year where there's a concern.
                  We can also fit it to the usage season only.  So we don't have the complications of having to fit the same model to the non-usage season as well.  
                  DR. ROBERT CHAPIN:  Okay.  So you're at -- you're past five minutes now okay.  So you're past five minutes.  So can you come to your last slide please?  
                  DR. JEREMY ALDWORTH:  Okay. 
                  DR. ROBERT CHAPIN:  Please and thank you. 
                  DR. JEREMY ALDWORTH:  I am going to come to the second last slide and end here.  This how serial correlation is determined.  Okay, in the SEAWAVE-QEX model, it's determined by correlogram analysis applied to residuals after the SEAWAVE fixed terms have been modeled, which is good. 
                  But the residuals of censored data in the model, I think are incorrectly determined.  In equation 9 in Vecchia's paper, he says the residual of censored data is the log of the censoring limit which is typically a constant, minus the fitted value.  So those censored residuals mimic the structure of the model, not the serial correlation of the data. 
                  And these residuals are then fed into -- mixed in with the uncensored residuals and they're contaminate the correlation and the correlogram analysis.  And that worsens as the censoring rate increases all the way up to 70 percent.  I would recommend that you don't use censored residuals if they're going to be calculated like that.  Thank you. 
                  DR. ROBERT CHAPIN:  Excellent.  Thank you very much. 
                  Okay.  And our last public commenter today is Dr. Basu from CropLife America.  
                  So you need to punch the little thing and get a little red light. 
                  DR. MANOJIT BASU:  Okay.  All set now.  Good afternoon and thank you for the opportunity to provide oral comments at the US EPA Scientific Advisory Panel.  I'm Manojit Basu, managing director of Science Policy at CropLife America. 
                  CropLife America is the national trade association that represents the manufacturers, formulators, and distributors of the wildlife crop protection and biotechnology products used by farmers, ranches, landowners, and home gardeners.  CropLife America encourages farming practices and supports environmental policies that are based on sound science and best practices in that respect and maintain U.S. farmers' ability to grow healthy and abundant food, feed, and fill. 
                  CropLife America appreciates the progress made by the agency to develop tier drinking water risk assessment and inclusion of drinking water quality monitoring data for risk characterization.  CropLife America has provided extensive technical comments to the charged questions for this scientific advisory panel meeting.  I would like to request the SAP to consider the following three issues as they review the SEAWAVE-QEX and public responses to the charged questions. 
                  First, developing a standard operating procedure for specifically identifying the conditions, situations, or decision points, for which modeling and monitoring refinements would be acceptable.  
                  Second, providing more clarity on the application and use of monitoring data.  Post sampling bias factors for risk assessment. 
                  And third, for clarity of communication, CropLife America recommends that bias factor, adjusted values be reported as interval estimates, together with associated point estimates. 
                  This will allow a better understanding of the uncertainty associated with using sampling bias factors.  And the estimated statistics would be communicated clearly. 
                  Once again, I'd like to thank the Agency and the SAP for your time this afternoon.  Thank you.  
                  DR. ROBERT CHAPIN:  Bless you.  All right.  Okay.  Let me congratulate the panel for staying awake and engaged and focused through a significantly challenging topic, at least for me.  Both my neurons were challenged by it. 
                  Let's see, so it is after 5.  We're not going to tackle the first question today.  We'll start off with that first thing in the morning.  Let's -- I hear moans of disappointment to my left, which I am going to resolutely ignore. 
                  Let's see, we'll start off at 9.  Feel free to engage with other respondents to the questions that you've been assigned or have to the degree that you have energy for tonight.  And let me see, we've been told that there will be pastries in the breakout room.  Do we know when those pastries show up?  First thing in the morning.  Is that 9:00?  Earlier than that, 8:00.  Some indeterminate time there might be breakfast down there.  We don't know. 
                  I think that's all.  I think that's all from here.  Okay.  You're done.  Thank you very much and thanks to our EPA presenters.  Nicely done. 
                   
                        [MEETING ADJOURNED FOR THE DAY]

                          OPENING OF MEETING - DAY 2 
                  
                  MS. TAMUE GIBSON:  Good morning.  We'd like to welcome everyone again for day two.  And thank each one of you for participating in today's meeting for the Approaches for Quantitative Use of Service Water Monitoring Data in Pesticide Drinking Water Assessments.  Thank you.  I want to, at this time, turn the meeting over to our Chair, Dr. Chapin.
                  
                            PREVIOUS DAY FOLLOW UP 
                  
                  DR. ROBERT CHAPIN:  All right.  Thank you, Ms. Gibson.  Good morning, everyone.  Thanks for showing up fully caffeinated and ready to go.  Let's see.  So, today, we're going to start to answer the questions that the EPA has posed to us.  We get to the real meat of why we're here.
                  Let me just remind you that these microphones are directional; so if you've got the mic pointed at you and you turn this way, then all of the sudden the mic's not working very well.  It's helpful, not as much for the people in the room but for the transcripts, to be able to keep talking into the -- or make sure the microphone is in front of you as you turn your head.  That's number one.  Identify yourself for the transcript.
                  Let me also make the point to everyone that if you're -- if the point that you want to make for one of your answers has been made already, you can say, yeah, I second Dr. Nowell's comment about this, and this, and this.  Just -- that's perfectly fine.  And that gives our EPA colleagues a sense of how many people are sort of holding up their hands saying, yes, I support this concept.
                  And finally, our EPA friends are going to be using this methodology or a variance thereof in the very near future.  And so, what they would be happy to have from us, what they would really like to have from us, is a sense of the stuff that's really a must-do before this rolls out and starts to be fired in anger, versus the nice to have longer term stuff.
                  If we could maybe spend less time focusing on the nice-to-haves, and more time and be clear about the things that really need to happen before they roll it up and pull the trigger on the go.  That would be helpful for them.
                  Having said all that, let me just ask the Panel, are there any questions before we get started?  Okay.  All right; then let's do this.  So, the format is that they read the question, read the lead-in to the question and then read the question.  And then we'll turn to the various discipline leads and associates.  And I think I was asked to start off the charge question responses for this first one with -- yeah?
                  DR. REBECCA KLAPER:  Are the EPA folks going to be here the whole time that we're answering questions?
                  DR. ROBERT CHAPIN:  Yes.
                  DR. REBECCA KLAPER:  Okay.  To ask -- answer questions about the question basically?
                  DR. ROBERT CHAPIN:  Yeah, that's right.
                  DR. REBECCA KLAPER:  Okay.
                  DR. ROBERT CHAPIN:  And so what'll happen is -- excuse me -- after we give our responses, I'm going to turn to them and say, did we answer your question?
                  DR. REBECCA KLAPER:  Okay.
                  DR. ROBERT CHAPIN:  And they'll -- there will be a huddled conversation.  And then they'll look at us and go either, no, it's fine, or they'll turn to someone and say, could you please go over what you said about blah-blah-blah?
                  DR. REBECCA KLAPER:  Okay.
                  DR. ROBERT CHAPIN:  So we are here to answer their questions.  We don't necessarily -- we don't have the time to engage in a discussion with them; but if there's a clarification, something that you want, they're here to answer that for you.
                  DR. REBECCA KLAPER:  Okay.
                  DR. ROBERT CHAPIN:  Does that help?
                  DR. REBECCA KLAPER:  Yeah; thanks.
                  DR. ROBERT CHAPIN:  Cool.
                  Okay.  We ready?  All right.  I'll turn it over to our EPA colleagues to lead us into question number 1.  Who gets that lucky task?  All right.
                  
                               CHARGE QUESTION 1 
                  
                  DR. ROCHELLE BOHATY:  All right.  Good morning, everyone.  Rochelle Bohaty here.  So we'll start with Charge 1.  And I will read the background.
                  Because of the sporadic nature of pesticide concentrations in surface water, monitoring programs with limited sampling frequency often do not provide a reliable estimate of the range of pesticide concentrations relevant to cancer and noncancer durations of toxic logical concern typically considered in pesticide human health risk assessments, for example, 1, 4, 21, and 365 days.
                  The Scientific Advisory Panels in 2010, 2011, and 2012 suggested that EPA look into use of a seasonal wave regression model, SEAWAVE-Q, developed by USGS to help interpret surface water monitoring data by generating daily pesticide concentration kymographs.
                  SEAWAVE-QEX, a modified version of SEAWAVE-Q is designed to estimate extreme, that is peak or daily average pesticide concentrations using streamflow as a covariate.  EPA evaluated SEAWAVE-QEX using high-frequency surface water monitoring data and streamflow data and concluded that SEAWAVE-QEX is a suitable tool for estimating pesticide concentrations for non-sample days, so that upper-end pesticide concentrations may be estimated.
                  EPA also evaluated alternatives to using streamflow, that is precipitation and stream stage, as a covariate.  This was done because daily streamflow data are not always complete or readily available and may be seasonal; that is periods of the year will not have flow record.
                  This would allow for sampling sites located in low-flow or non-flow systems where flow would not be a suitable covariate.
                  
                             CHARGE QUESTION 1(a) 
                  
                  Charge 1(a):  Please discuss the strengths and weaknesses of using SEAWAVE-QEX to estimate short-term, that is 1-, 4-, 21-day average pesticide concentrations as well as longer-term, that is 365-day, pesticide concentrations.
                  DR. ROBERT CHAPIN:  Okay.  So we'll turn to the statisticians.  And who wants to go first?  Okay.  Dr. Portier.
                  DR. KENNETH PORTIER:  Thank you.  Good morning.  I asked to go first, I don't know why, except maybe I'm prepared this morning.  
                  In answering this question, I felt I had to look under the hood.  I know in the presentation yesterday, there was a little bit of discussion, like, well, we wanted you to kind of get the gestalt of the model, rather than get into the details.  But I think if you ask car enthusiasts to a car show and don't open the hood, you're asking for trouble.
                  You have statisticians, hydrologists, and modelers and you haven't shown the model to them so that we can kind of talk in a little bit more than generality.
                  It's maybe a little thin, but I extracted this out of the Vecchia paper.  And I wanted to go through and just talk a little bit this morning about what SEAWAVE-Q and then SEAWAVE-QEX actually are doing in a little bit more detail, so that we can then talk about the different components.
                  SEAWAVE-QEX is a parametric regression model involving four explanatory variables that account for background pesticide concentrations, for a seasonality of pesticide use, for correlations with two components that together describe variation and streamflow about its mean.  And it also accounts for temporal autocorrelations.  
                  And then this is discussed in Appendix A of the -- excuse me -- of the minutes, I think of the 2012 SAP.  Fitting the model is a five-step process involving 17 tasks that are implemented as 15 functions in R.  And all of this is described in the Vecchia 2018, USDOI USGS Scientific Investigations Report.
                  So, just quickly looking at this equation you see that they're fitting Log concentration to an overall average, which is this b0 or beta zero.  You have a parameter, b1, which is a weighting function against this W(t) form.
                  And the W(t) is a seasonal wave, right?  A seasonal wave is a function.  We don't see the form of that function; we're just told that it's indexed to two parameters, h, which is a decay rate, measured in decimal months, and s, which is a phase shift, which is also measured in decimal months.
                  And together, with all these different combinations --
                  UNIDENTIFIED FEMALE:  And also m.
                  DR. KENNETH PORTIER:  Huh?
                  UNIDENTIFIED FEMALE:  There is also m.
                  DR. KENNETH PORTIER:  Oh, and then there's also m; that's right.  And so, there's three parameters.  It represents over 1,000 forms that they're looking at.
                  And when the model is fit to certain data, it looks through all these 1,000 forms to pick the one that's closest to it.  And the W(t) form is basically a concave function that asymptotes out to a certain point and then it decays kind of exponentially from that asymptote.  So, it's kind of a hump that goes down.
                  So, that's the seasonal wave component.  And then there's this -- the AMT or the midterm flow anomaly function, which is kind of a measure of long-term streamflow averaged out over about a month.  It's a 30-day rolling average, so it gives you some idea of kind of long-term average flow.  And it's weighted by these B2 terms.
                  And then you've got the short-term flow anomaly, which is kind of the residual flow that's taken away from the average long-term flow.  So there's a lot of variability in STFA.  And MTFA is kind of just a nice smooth function.  So, you've got those two functions that actually are the covariates that are used to explain Log concentration.
                  And then you've got this (t-tm) term, which it's taken me a little while to figure out, but that's kind of, I think, best described as a triangular function that goes up linear to tm and then goes down linear to tm, as long as the b4 term is positive.  And then you've got an error term.
                  And the difference between the Q and the QEX model is that they added a little bit more to the error term.  So, they looked at the residual, the e(t) term, and said, well, this is a time series, so there's probably autocorrelation that's occurring.  You know, tomorrow's residual is probably correlated with today's residual to some extent, so let's model that in.
                  But they also looked at the residuals and said, well, we get more variation when concentrations are high and less variations when concentrations are very low; so this SSD term here on the right attempts to take into account that heterogeneity that occurs with increasing or decreasing concentration.
                  And then the bottom term correlation takes into account that autocorrelation, which they're modeling as an exponential decay curve.  Okay?  So all of that put together is what they call the SEAWAVE-QEX model.  
                  So, it's got a lot of moving parts.  When you open the hood, the engine is full.  There's a lot in there.  It shines, but there's a lot in there.  And you don't want to be messing with any part because it might mess the whole thing up.
                  Next slide.  So, I said there were a bunch of data steps.  In the sense of a regression, it's not a simple regression, where you kind of throw it in, you do a least squares, and there it is.  Because you're pulling out these correlates with streamflow, you're having to kind of take it into account, compute residuals, and then go back and figure out correlation structures and apply that in.
                  So it takes this long data -- you know, five-step process where you do data preparation.  Then you do a regression and try to figure out the shape of the seasonal wave.  Then you compute the residuals and you try to correlate that with the streamflow data.  And then you put that back in and you compute some new residuals.
                  Next one.  And then you try to figure the correlation function.  And that gives you different residuals.  And then at the end of the day, you have a fitted model called SEAWAVE-QEX that's applied to a certain set of data.
                  Go back to the first -- the second slide set.  Oh, you got the -- just leave -- next one.  So, I'm going to leave this up as I continue with my discussion.
                  So, SEAWAVE-QEX is a process for imputing missing values and pesticide concentration time series.  And I have a note to myself.  It does not predict -- and we use the word yesterday, a couple of people used the word yesterday to predict.
                  And I don't think it predicts.  It's for simulating missing values.  That's what imputation does; it kind of fills in with a possible value.  And so, this model is just a fill-in model that imputes.
                  In step 4 of the fitting process, random values are generated for censored normalized residuals; and random residual values are generated for days with no observations.  These are added to the mean trend values generated in the previous steps to produce a complete time series.
                  As is typical with imputing, more than one realization for missing values are generated.  In this case 100 values are generated to create 100 equally likely conditional traces.  So, that's kind of what they're doing.  And this is very standard imputation methodology.
                  In the first use of SEAWAVE-QEX described in the White Paper, the method is used to fill-in concentrations to complete concentration time series of selected USGS and NAWQA sites.
                  In this study, we can examine how the one-day series mean can change between observed time series and the mean of equally likely conditional traces.  And we can see that it's not -- you know, the conditional traces are -- produce maximums that are not far from the maximum of the observed time series.
                  And again, this is to me a classical imputation study.  And the findings that they report are what one would expect from imputation.  And when you have most of the data and you're just imputing a few points, the maximum is going to go up a little bit, because some of your random realizations are going to jump above the previously recorded maximum, most of them are going to be below.
                  You know, when we have the roughly 100 USGS datasets that are almost complete, you don't see a whole lot of difference between the observed maximum and the long-term imputed maximum.
                  Okay.  As pointed out by some of the public commenters, there are other methods of imputing values into the near-complete USGS and NAWQA time series, for example, kriging.
                  One of the reasons SEAWAVE-QEX may be superior to other imputation methods is its explicit handling of temporal trend, incorporation of streamflow information and predicting both temporal trend via the MTFA, and day-to-day variation via the STFA.  And then the use of temporal autocorrelation in the generation of residual variation.
                  The temporal trend components constrain the infill values, following the general pattern of concentrations prior to and following the missing days.  The variation components allow for the generation of concentration that exceed those recorded on other days in the monitoring of time series, while not allowing these days to have more variation than has been observed prior to or following the missing days.
                  In the second use of SEAWAVE-QEX reported in the White Paper, the process is used to impute values in time series with large gaps of missingness.  The question being addressed is, how well does SEAWAVE-QEX perform with sparse monitoring data?
                  To answer that question, measured datasets or subsample to create smaller datasets using a sampling protocol designed to mimic typical practices employed by various monitoring programs across the country.  By generating the sparse dataset from the complete monitoring data, the study is able to compare findings from fitting SEAWAVE-QEX to sparse data to the actual series maxima.
                  The sparse data study also examines the performance of SEAWAVE-QEX process when input datasets that have more intensive sampling, for example, seven-day sets, are compared to datasets with longer subsampling windows, for example, 14-day sets.
                  Two subsampling protocols are used to generate five realizations each of monitoring time series with data recorded on 7- and 14-day sample time intervals.  Each of the synthetic monitoring time series had a minimum of 12 samples per year to meet the SEAWAVE-QEX minimum requirements.
                  Each synthetic monitoring time series has input to SEAWAVE-QEX -- is input into SEAWAVE-QEX, and model results compared to full-measured datasets.
                  This is a large study involving four NCWQR sites assessed for two pesticides, atrazine and metolachlor, two subsampling protocols, the 7- and 14-day replicated each five times, and two covariates, a total of 160 runs.
                  And then there's two AEMP sites are also assessed for one pesticide, atrazine, two subsample protocols, 7 and 14 days, replicated each five time, and two covariates for a total of 40 runs.  Together, 200 runs are examined.
                  The White Paper in Chapters 3 and 9 -- Chapter 9 is Appendix B -- reports fitting results.  A total of 100 conditional traces are generated for each of the 200 synthetic datasets.
                  For each dataset, SEAWAVE-QEX outputs were condensed to the annual maximum estimate.  That is for each site, there are 100 estimates for the maximum for daily data, 100 estimates for the maximum of a 4-day average time series, 100 estimates for 21-day and for 365-day average.
                  The maximum of these 100 values, i.e. the maximum of the maximum, was then compared to the true value computed from the original complete dataset.  In addition, summary statistics means mediums, upper, mid, and lower percentiles of the 100 distributions were examined and summarized as well.
                  As an aside, one can argue that allocation of effort in this study might have been better balanced between imputation variability, estimated by the 100 equally likely conditional traces, and sparse sample to sample variability estimated by the five replicated datasets.
                  Five replicates does not really provide the best understanding of sample to sample variability.  With 100 traces, one can examine individual percentile estimates as alternatives to the maximum.  So, I can understand why they use the 100 there.
                  In addition to examining the maximum of the 100 maximum, other percentiles of interest were examined and are discussed in Appendix E.  These include the maximum of the annual 99th percentile concentrations, the 99th percentile of the annual 99th percentile concentrations, and the 90th percentile of the -- so there were other alternate estimates for estimating the maximum.
                  The White Paper provides a written summary of findings and evaluates performance with broad trends in the estimates.  For example, the 365-day average was underestimated more often than for other averaging times.
                  The magnitude of underestimation was mentioned.  For example, the 365-day average went underestimated at estimates that were at least 50 percent of the true average, about 40 percent of the 4-day, averaging 60 percent of the 21-day, and 20 percent of the 1-day series had those -- the -- oh, I'm sorry; that's -- the 50 percent true average -- it was about a 40 percent true average for the 4-day averaging, 60 percent for 21-day, and 20 percent for the 1-day series.
                  The discussions of findings from both the imputation study, and the sparse data study could and should've, included more details and a more rigorous and logically organized presentation.
                  The White Paper presents some diagnostic plots.  And the Panel was provided access to all diagnostic plots for each imputed site and much of the sparse data study outputs.
                  When discussing the quality of model fits, the accuracy and performance target was not clearly stated.  So the discussion kind of implied that the method performed well, but it was never very clear, performed well against what criteria?
                  While the White Paper does not go into detail, the flow of SEAWAVE-QEX tool include the eight diagnostic plots, that together allow users to get a very good understanding of the model performance and the quality of the fit through a particular set of data.  And I've listed in my notes here the eight diagnostic plots, simulation and summary of the output graph, a seasonal wave summary, an adjusted concentration, regression against a midterm flow, adjusted concentration versus the short-term flow, the trend for measured concentrations over the entire simulation.
                  Each of these graphs allow you to look at specific components of the model to see how well it's going.  So, you can actually visualize, well, how well is this b2Amt(t) doing in the model?  You know, is that b2 parameter very big or is it close to zero; which means it's not adding much to the explanation of the time series.
                  And then there's a couple of residual plots, one by month and one by year that allows you to look and see whether the model has captured all of the residual pattern, so that there's not any leftover temporal between your pattern that hasn't been captured in the model.  And then there's an important graph, which is the correlation function of the normalized residuals; which depicts the fitted exponential correlation function curve, plotted to the normalized residuals versus time.
                  And on that graph, you actually see the estimated correlation time scale, the CTS term, which is important, which estimates the average time between successive observations.  And then given acceptable fit criteria for the correlation function, that the 95 percent confidence interval for empirical autocorrelations, with time steps lower than the CTS should overlap the fitted correlation function curve.
                  The White Paper in Section 9.3 attempts to present the results of this assessment.  Because the assessment covers so many different issues, it's difficult reading, and often the key findings are not immediately obvious.  In some places there's too much detail and in others too little.
                  Nevertheless, reviewing the White Paper along with the SOP document and the Vecchia paper, the following strengths and weaknesses can be determined.  At least this is my take on the strengths and weaknesses.
                  So, the strengths related to using SEAWAVE-QEX to estimate pesticide concentrations with near-complete monitoring data for both concentrations and streamflow.
                  SEAWAVE-QEX appears to logically deconstruct the concentration time series into components that describe major sources of variation.  The multistep model fitting process is a logical approach to a difficult parameter estimation problem.
                  SEAWAVE-QEX performs well in describing near-complete monitoring data -- and this, I got out of the Vecchia paper.  It seems able to adequately simulate time series scatter of pesticide concentrations in near-complete monitoring data.
                  When fit to this near-complete monitoring time series, it provided an estimate of maximum concentrations for various averaging schemas that tend not to underestimate the true maxima, and which also tends to not overestimate the maxima by more than an order of magnitude.
                  SEAWAVE-QEX performs well when applied to sparse monitoring data scenarios, when the monitoring data meet general requirements of greater than three years of data, 12 or more samples per year, spread across the application season and less than or equal to 70 percent censored observations.
                  The estimated maxima and 80 percent error bounds are less than two orders of magnitude greater than the true series maxima in most of the examples that they had.
                  As expected, the more samples per year and the more uncensored samples, the better are the SEAWAVE-QEX maximum in estimating the true maximum.
                  SEAWAVE-QEX performance does not significantly degrade when missing streamflow data are estimated with precipitation data.  SEAWAVE-QEX seems to work best when the model time period does not significantly extend beyond the min and max sampling times, and when the annual or seasons-limited time series is best described by one seasonal wave.
                  I didn't see any scenarios where multiple seasonal waves seemed to do any good.  And I see a lot of that seems to be just artifact of the fitting process rather than a real deficit of the model.
                  Weaknesses related to using SEAWAVE-QEX to estimate pesticide concentrations with near-complete monitoring data for both concentrations and streamflow.
                  SEAWAVE-QEX is a complex model and will be difficult to understand by the uninitiated.  This may be less of an issue if use of the model is limited to tier 4 assessments as suggested, and when it's supported by staff that truly understand the model and the complexities of fitting it.  So, this is not a turnkey system.  It's going to need a little bit of expertise the two or four times it's applied each year.
                  SEAWAVE-QEX is a mix of empirical and theoretical constructs.  But at its core, it's a regression.  As such, it's better at estimating mean patterns and less competent at estimating stochastic or residual patterns.  In addition, the important stochastic patterns may not be fully captured with less than 100 monitoring data spread over multiple years, and when fully 70 percent of these data points are below detection.
                  Also, because of this, it's difficult for regression models -- and I would say in most statistical models for that matter -- to predict with any accuracy extreme events such as maximum.  This was a point I made in the 2012 panel report.  
                  You know, estimating extreme owner statistics like the maximum is very difficult to do, because there's a lot of variability in the maximum value.  And it's a very skewed distribution, highly -- I have to always remember the tail right skewed, right?  It's always the tail goes way out, which means the maximum of the maximum has the potential of being a very high overestimate.  But I think in this study there's a realization that that can occur.  And there's been some attempts to avoid going way out in the tail where you're going to get overly conservative estimates.
                  SEAWAVE-QEX performance very much depends on the situation.  As mentioned on page 141 of the White Paper, "The summary statistic that performs best, differ somewhat for each pesticide combination, and depends on the definition of best.  Whether the objective is for the subsample to always overestimate the highest annual maximum, or overestimate as often as underestimate the highest annual maximum, or all annual maxima.  This means that use of the model can never be automatic.  It'll require an educated user to properly fit and use.
                  And finally, I have a quote that I started typing right as the meeting started about the long-term 365-day average maximum estimate -- estimation of it.  I think the reason the 365-day average underestimation is more frequent than we see in the other scenarios, is that it's more likely that the seasonal wave, the W(t) component in the model, is not well-captured in that annual data.  And the regression on the MTFA and the STFA don't kind of step up and take care of it.  It doesn't come in and compensate for the loss of that seasonal wave curve.
                  And that produces -- and that puts a lot of the effort in the modeling on the variability of residuals to kind of give you a high maximum.  And so the model is just not capable, in that situation, of going high enough to really capture the average annual estimate.  But I think I need to think a little bit more about that.
                  And with that, I'm done.  I'm sorry.  It's all written though, Bob.  Questions?
                  DR. RAYMOND YANG:  Ray Yang.  I thank Ken for giving me these three pages of this writeup and really clarify things.  And I want to echo that.
                  I'd want to tell the EPA colleagues that you cannot talk about modeling using 255 pages without talking about the model.  Therefore, this stuff had to go into your White Paper; okay?  In fact, you will do yourself a great favor if you attach Vecchia 2018 paper as an appendix.
                  DR ROBERT CHAPIN:  All right.  Thank you very much.  Let's see.  Dr. Berrocal, do you have stuff to add?
                  DR. VERONICA BERROCAL:  Yes.  Not very much, because he already has covered a lot.  But I wanted to start by saying that I think that this problem of estimating or imputing pesticide daily concentration data, and the response migrating data, is actually a very hard problem.  It's not very easy since there are many factors that play a role in our factors besides concentration.
                  Yesterday, I was asking lots of questions about this model.  I would actually like to commend Dr. Vecchia for tackling these complex issues and having propose a model that is in my mind elegant, robust, and as simple as possible, even though somebody believes it's not too simple.
                  I'm not going to go over the details of the model, but I think as it was said, that it captures the main model variation in daily concentration, pesticide concentration data.  I also believe that the fact that now there is introduction of these serially-autocorrelated adverse terms, that allows to account for lingering temporal correlation in the daily pesticide concentration that cannot be explained by the seasonal wave and the streamflow.
                  The introduction of these serially-correlated adverse teams with seasonal variance standard deviation has been sought to allow the model to provide a more flexible characterization of the viability in the daily pesticide concentration, that one could do with a kriging model, which typically assumes a constant variance.
                  Also, I would like to point out that without the addition of these residuals that have seasonal variance standard deviation, the SEAWAVE-QEX model can actually be thought as a specific case of universal kriging.  So, the SEAWAVE-Q is an example of universal kriging, while the SEAWAVE-QEX is a step above universal kriging, which I think it's more adapt for this situation.
                  I believe that the model has several strengths.  It's an elegant and not overly-complexed model that allows, through conditional simulation, to reproduce the statistical characteristic of the daily pesticide concentration data.  And thus, it can be used to infer upon the distribution of certain summary statistics from an incomplete time series of pesticide concentration measurements.
                  The parameters in this model have an interpretation that can actually be useful for designing future sampling strategies.  For example, the correlation temporal skill is important for one designing the sampling frequency.
                  The assessment that Vecchia performed of the SEAWAVE-QEX model in the paper show that the model has an overall good performance in real pesticide concentration data.  And the simulation experiments that Vecchia performed showed that when the assumptions of the model are satisfied, the SEAWAVE-QEX model actually performs pretty well in recovering the true daily concentration data.
                  And this is particularly true when the sampling rate is low, and the sampling frequency is lower than the correlation temporal scale.  And I really want to underline this because I think this is an important point.
                   Also, for long-term in the data evaluation that EPA performed, it seems as SEAWAVE-QEX is able to neither consistently underestimate, and not always overestimate, long-term pesticide concentration when it is characterized in terms of extremes.  So, either the maximum of the 95th or maximum of the 99th percentile.
                  In terms of weakness, I think that as it was mentioned, the SEAWAVE-QEX model might not be easy to implement for somebody that has not experience with this type of modeling or with this type of data.  Also, the evaluation of the SEAWAVE-QEX model discussed in the White Paper in appendix B, Section 9(2), showed instances where the two season waves model was selected, as the best model to fit the data, but the data did not support the presence of the two waves.  Hence, a user and assessor should be exercising care when feeding the SEAWAVE-QEX model to data and determine whether the model should be fit in the four modalities using all data or using only a partial record.  So, that's another complexity that somebody who doesn't have the expertise might not understand.
                  I was a little bit disappointed, once after I had read the Vecchia paper; in seeing that in terms of evaluation of the performance of the SEAWAVE-QEX to derive summary statistics for characterizing short-term and long-term pesticide concentration, the results were not as good as the ones presented in Vecchia's paper.  And there was not an easy generalization since the results varied by pesticide and site, as it was already mentioned.
                  Particularly, for each site pesticide combination subsample we did, a 7-day or 14-day samplings frequency were derived with pesticide concentration data.  There was subsamples that include only 12 measurement, which was the lower limit for the number of samples indicated by Vecchia.
                  So, given that 12 was the lower threshold indicated by Vecchia's number of observation, it seems here that the utility of the SEAWAVE-QEX model has been tested in the worst-case scenario.  I think it would be useful to assess whether the capacity of SEAWAVE-QEX to adequately capture pesticide concentration, was assessed when a slightly larger subsample of data was used.
                  I'm not yet ready to describe the SEAWAVE-QEX model as a model that can capture short-term concentration just because I feel that the subsampling was really put in the model in the worst-case scenario of only 12 samples.
                  Another point that was mentioned, but I think that this was not highlighted.  It was that the empirical correlation that is produced by the SEAWAVE-QEX diagnostics, seems to indicate basically that there is independence in the residuals.  When instead, the fitted exponential correlation indicated there should be a correlation because that's what the model assumes.
                  I believe that this is a result of the subsampling scheme that has been used, that has made the data less correlated.  I understand that the subsampling scheme was performed to generate data that was similar to what is observed in real datasets.
                  But I wonder if the performance of the SEAWAVE-QEX model was also affected by the fact that the data that was provided was basically independent, based on the critical correlation reality the model assumes dependence.
                  I think that for the long-term concentration, it was already mentioned that SEAWAVE-QEX has problem in characterizing long-term concentration, particularly when we're thinking about annual average, 365 days.  While instead, characterization of short-term pesticide concentration does not present the same problem of underestimation.
                  And I had one question that maybe this is more a clarifying question than a comment on the charge questions.  I don't know whether I should ask it now or --
                  Dr. ROBERT CHAPIN:  Sure, go ahead.
                  DR. VERONICA BERROCAL:  SO, in the evaluation, it was assessed whether SEAWAVE-QEX would work if instead of streamflow, precipitation was used.
                  I'm just confused as in this equation that is put up here on the screen, we have Amt and Ast that are derived from the streamflow.  So, are those -- when you're using instead precipitation, are you computing the anomalies with respect to precipitation?  It was just confusing; there was no mention explicitly in the paper.  Or it was just -- you're just throwing away the AMT or the AST.
                  DR. SARAH HAFNER:  The model uses the precipitation to do that, the same as it does for streamflow.
                  DR. VERONICA BERROCAL:  Mm-hmm.
                  DR. ROBERT CHAPIN:  All righty.  Excellent.  Thank you very much.  Let me see.  I think what we'll do is we'll go through the leads and associates for hydrology and then for modeling.  And then open it up to -- and then get Dr. Sobrian's comments.  And then open it up to anybody else who wants to make comments on this question about strengths and weaknesses.  We'll go to Dr. Baffaut.
                  DR. CLAIRE BAFFAUT:  Thank you.  My name is Claire Baffaut.  So, SEAWAVE-QEX estimates a pesticide concentration during days that no sampling was done.  So, that is a set of statistical distribution of concentration simulated by SEAWAVE-QEX, is similar to the observed distribution of pesticide concentrations.
                  The nice thing about SEAWAVE-QEX is the fact that it takes flow as a covariate, in my opinion.  Because by doing that, we take care of some of the hydrological landscape processes that take place and that affect pesticide transport.
                  So, run off and flow are an outcome of precipitation.  And run off and flow, and infiltration, drive the pesticide transport.  So, by linking the pesticide concentration to the flow, we take care of other processes that take place on the landscape, runoff generation, infiltration, evapotranspiration; that all affect that flow which is a driving motor of pesticide transport.
                  So, that's what I like about it.  And in the equation of the model, all those terms correspond in some way to some physical processes.  The wave represent the input.  The short-term anomaly represents the short-term flow, which would be what's driven by runoff and quick subsurface flow.
                  The long-term flow anomaly would be what's represented by the ground water flow.  And then the last term would be degradation, I think. So, that's -- it's nice to -- even though it's a statistical model, it's nice to bring it back to what's really happening.
                  In terms of evaluation, I don't know what the pesticide water calculator does.  But in any places in the White Paper, it's mentioned that it improves the results of what is available now.  But since I don't know what the water pesticide calculator does, I would have liked to see, for all those cases, what would have been used instead of SEAWAVE-QEX.
                  Another thing that I thought was a limitation of the evaluation is that it's evaluated at four sites.  All of those are in northeast Ohio.  Two of the sites, Rock Creek and Honey Creek, are nested within the larger Sandusky River water shed.
                  All have a dominance of agricultural land that varies between 72 and 81 percent.  And the main crops are the same in four watershed, corn, soybean, wheat, and some hay.
                  Most agricultural land in those four watersheds is artificially drained with subsurface tile.  And those tiles speed up the transport of many things, including pesticide, from the fields to the stream.  And the rate of subsurface drainage in those four watersheds is about the same; that's about 70 to 80 percent of the agricultural land, which represent roughly 50 percent of the watershed.  So, as intensive hydrologic transports, there is not a lot of difference between those watersheds.  They kind of works the same way.
                  When it comes to the simulation of hydrologic variables, including pesticide concentration, it's important to test them all under different sets of condition; different water, different -- that includes amount of rainfall and intensity of rainfall.  Different crops because it affects residues, they affect ET, they affect other things.  Different soils and different management practices.
                  The biophysical property of a region do have an impact on the vulnerability of the land to transport a contaminant.  Thus, it seems important to evaluate the tool across a range of its biophysical characteristic, rainfall amount, timing, intensity, has a strong impact on contaminant transport from agricultural areas.
                  If datasets with daily data are not available in other biophysical settings, I think maybe EPA should consider the next best option, which may be near-daily data.  And it may not be a whole dataset that has several watersheds.  There may be -- you know, one watershed with a good dataset in that region and another dataset in another region.  But I think it's very important to test this in systems that function differently in terms of hydrology.
                  SEAWAVE-QEX assumes constant pesticide rates from year to year.  This is very large assumption because crop distribution, crops choice change from year to year, and therefore pesticide application.
                  I know that in some watershed that are well-drained, for example, in Iowa, that may be more stable.  In other areas where soils are more vulnerable to excess moisture, then that excess or moisture in the weather will affect the crop choice.
                  So, even though the main crops are corn and soybean, I can guarantee you that in northeast Missouri this year, there was much less corn and much more soybean planted because it was very, very wet in the spring.  And that affects the amount of pesticides -- the choice of pesticides and the amount of pesticides that are applied.
                  I'll get back a little bit on the evaluation criteria.  And Dr. Portier raised the point, is that there is no clear criteria set in the White Paper that tells us what is good.
                  So, in the Vecchia 2018 paper there is actually some criteria.  For example, let's say that if the bias between an estimated and observed metric -- whatever it is -- was between minus 10 and 25 percent, then the bias was considered low, which is good.  If it was between 25 and 50 percent, it was moderate.  If it was greater than 50 percent, it was high.  I didn't see similar criteria in the evaluation that was run for the four sites.
                  I recommend EPA to define such criteria to evaluate SEAWAVE-QEX.  What constitutes acceptable results?  What does that mean that the peak concentration is not underestimated, or that it's not grossly overestimated?  Is it a 10 percent difference?  Is it a 25 percent difference?  Is it a 50 percent difference, or in order of magnitude or what?  But we don't know.
                  And same thing in the appendix.  When comparing all the distribution, the concentration distribution, what is an acceptable thickness of the colored band that represents 100 simulations for each subsample?  Some of those, it's like maybe -- it's definitely less than an order of magnitude, maybe half of one.  And some cases it's more than one.
                  So, what is acceptable, what is not acceptable, I think, would make it easier to decide whether this is an acceptable model.  It would also make it easier to decide what sampling strategy is to be selected.  And it would be based on something rational.
                  There is the issue of missing or zero flow data.  So, I went through a little thing about what could cause missing flow data.  So one is that stages higher than the range of the rating curve.  And so, there is no basis to extrapolate the stage value to a discharge value.  But in that case, streamflow estimate can be obtained by extension of the reading curve, by first physical principles or by correlation with another trusted stream gauge.
                  Another case, we may be missing flow data because there was an equipment failure.  Values can be interpolated, or values can be obtained by correlation with another trusted stream gauge.  The stream bed went dry for a period of time.  Note that during that time if it went dry there is no transport of pesticide, and those concentration are zero as well.  So, that's not the most critical part of what this tool is made for.  So, I think that the idea of replacing by .1 or points was fine.
                  If there is no stream gauge, streamflow value from a different watershed that functions similarly to the one on the study may be useful as a covariate.  This would be particularly useful for smaller watershed like a 12 size or less than 100 square kilometers.  It's not likely to work for larger ones because it's more difficult to find an ecohydrologically similar watershed for larger ones.
                  But I would like also to say that hydrological model have made some progress.  And now, there's -- I've seen more and more model application that show good simulation of flow values, even with minimal calibration or even sometimes without any calibration.  And same thing, the model can be calibrated for a similar watershed and then applied to the one where there is no stream gauge to get some flow values.  And that's it.  I think I'm done.
                  DR. ROBERT CHAPIN:  All right.  Thank you very much.  And we'll turn to Dr. Green.
                  DR. TIMOTHY GREEN:  Tim Green.  I'm actually associated on, I think, the next three, and I'd like to defer to, I think, 1(c) for my next comments.
                  DR. ROBERT CHAPIN:  Thank you.  All right.  And now, we'll -- the spotlight swings far west to Dr. Zhang.  Yes?  And we'll hope that your caffeine has kicked in.  Dr. Zhang, are you with us?
                  DR. XUYANG ZHANG:  Yes, I am.  I've been drinking black tea in these couple days, and it's important.  Well, I'd like to start with a big thank you to the colleagues in EPA.  I think this is no trivial work.  It takes a lot of hard work to put things together.  And excellent presentations yesterday.  That really helped me clear lots of questions that I've had.  
                  I will go straight forward to the strengths.  Dr. Portier had a great introduction on the SEAWAVE-QEX model already.  I think there are a few strengths of the model that I really like.  The first one is that it can generate daily times zero chemograph that can be used to estimate uncertainties in the margin data.
                  The second point is that it can generate daily concentration estimates that are higher than the measured maximum, and consequently reduce the chance of underestimating the short-term average pesticide concentration.
                  This is a large advantage compared to the traditional imputation methods where, you know, the maximum value can never be exceeding the maximum of this value.
                  Number three is that it provides the diagnostic plot that really helped USGS evaluate the goodness of fit of the model.  As it can be shown in the SOP and the White Paper after model simulation, the SEAWAVE-QEX generates eight different plots that really provide lots of information about a model fitting.  And that also helps the following simulation, and it helps the users to improve based on what they've seen on this diagnostic plot.
                  Another advantage is that the development of the model has been well-documented in my opinion.  In Vecchia 2008, the questions were crystal-clear.  And like Dr. Portier and Dr. Berrocal had mentioned, I really like the model being simple and elegant.
                  And then, the next strength bullet is that based on EPA's evaluation, using the daily or a near-daily monitoring data of atrazine and metolachlor, the model seems to be able to generate daily concentrations from subsample data with reasonable accuracy.  That is, as mentioned in the White Paper, there's a low percent of underpredicting as shown in Figure 9.18 and 9.19, except for the results in the 365-day average period.
                  And in the section of sampling bias factors, it was also mentioned that the resulting sampling bias factors developed using SEAWAVE-QEX, estimated a chemograph where it's very close to those developed using the full record of data.  So, that's -- it has good applicability to enter the data then used to calculate a sampling bias factor.
                  And lastly, when running the model, there are a minimum number of parameters that require user input.  And speaking based with my experience of a hydrological model, which often involves hundreds of parameters that the models and model users have to tweak.  So in comparison to that, SEAWAVE-QEX is relatively simple to use.  And this ensures, in my mind, the consistency in modeling results and minimizing human error.
                  With that, I also realize that there are some drawbacks of the model, especially when we try to use it in pesticide drinking water assessments. 
                  The first one is the data requirement of the model.  So, the minimum requirement is more than three years of data and less than 30 percent of censorship.  Which in my mind are a stringent criteria and relative to the typical monitoring data that are currently available.
                  Many monitoring datasets do not need even the lowest of requirements, especially on the sentinel rates.  So, that results in a very limited use of the model.  So again, the significant errors could occur when model assumptions are violated.  
                  So, I want to state the following two modeling assumptions.  And I think there's a big chance that the assumptions could be valid, if nothing paying close attention, in many application cases.
                  So, the first one is that the model -- when the variation in flow and concentration during the day is large.  And the model really assumes, you know, fitting the daily concentration using largely available graph samples, which is not representative of the daily peaks under systems that have big variation during the day.
                  And also, the model uses the daily average flow.  So, in these kinds of systems, the daily average flow might not also be representative of the peak flow within the day.  So, in some hydrological systems, when the variation in both flow and daily concentration is large within a day, the model may not fit very well.  
                  And secondly, the relationship between the pesticide transport and flow, according to this model, is linear.  So, the model assumes a linear relationship between pesticide transport and flow.  But in reality, in some hydrological systems, the relationship could be nonlinear.  So, when these assumptions are valid, I think we could expect large errors in model simulation.
                  The model -- it seems that the model cannot be applied beyond the years with available sampling data.  As Dr. Portier had mentioned, that the model essentially is an imputation method, it cannot be used for prediction.  So, I just want to make a point of that.
                  And also, about the time trend.  The model assumes a leaner trend over time.  And a scientist who worked in the pesticides for many years, I know, that changes in regulation and consequently the use patterns of pesticide may results in abrupt changes in pesticide concentration trends.  For example, chlorpyriphos with the cancellation of use in urban areas.
                  Around year 2000, 2002, there's a significant change in the use pattern.  So, under these conditions, the sample data should be subdivided into different periods and the SEAWAVE-QEX model should be seeded separately for each period.  I think EPA is aware of that and it's been already mentioned in the White Paper.
                  The next point is that the model assumes constant pesticide input from year to year, which may not be true in areas with saturating pesticide usage due to climate change and crop management.
                  I think Dr. Baffaut had just mentioned that also in areas of Ohio, and I know in California too, pesticide usage changes from year to year.  And the assumption of constant pesticide usage from year to year can easily be violated.
                  The next point is that for the two-season model, pesticide removal is assumed to be no longer than 60 days.  I think this may not be true, especially for persistent pesticides, which take much longer then to dissipate.
                  And so users should avoid using the two-season model, especially for persistent pesticides.  And looking from the White Paper anyway, it seems that simulation is being done by EPA, the one-season model seemed to work better on many occasions, in the two-season model anyway.
                  So, the next point is that I think the model worked pretty well for near-complete datasets.  But the accuracy of the model, when applied to (inaudible) datasets, remains uncertain in my opinion, due to the following two reasons:
                  The first reason is that in this evaluation and as documented in the White Paper, only three pesticides were used in the evaluation.  And the pesticides were actually very similar, regarding to the case of chemical property.  And the size are very close to each other.
                  This datasets, although, I think it's very valuable, but they are not representative of the wide array of pesticides that are currently being used in the nation.
                  Take Koc for example.  I think among the three pesticides that are relatively having low Koc, compared to hydrophobic pesticides, for example, pyrethroids, which may behave very differently compared to the three pesticide.
                  So, the dataset also does not represent the variability in watershed properties and pesticide use patterns that are observed across the nation.
                  Again, Dr. Baffaut said it'll be beneficial if we can continue evaluating the model in areas with different hydrological conditions than the existing four sites.
                  In EPA's evaluation, the SEAWAVE-QEX generated results using the subsample data and would compare it to the log integrated (inaudible).  So this comparison, I think it's a valid approach to evaluate a model performance.  But this -- it did not provide an independent validation.  
                  This is perhaps a very high requirement of the evaluation -- of the person conducting evaluation.  But in an ideal world, to evaluate the performance of the model, it'll be best to be conducted if there's an independent dataset.
                  But I understand that atrazine is probably the best data available.  And obtaining a completely separate, independent dataset might be not feasible.
                  But given that, I think a cross-validation approach could be used to include the evaluation.  In which a subsampled data could be set aside, excluded from model fitting.  And the set aside dataset can then be used for model validation.  Instead of using all the subsample dataset for fitting the model, we can keep part of it for validation.  So, I think with this approach, we could have more confidence in the results regarding the performance of the model.
                  And I second Dr. Portier's opinion, that the model  --  in terms of evaluation criteria.  I was also looking for that in the White Paper and I didn't see it.  I recommend that EPA put that in the White Paper and calculate statistics that can be used to evaluate the model performance.
                  And another point about the weaknesses of the model is that to some users it may have a steep learning curve.  And the model also has limited use in nonflowing water bodies; and flow is not really correlated with concentration.
                  That's all for me.  Thank you.
                  DR. ROBERT CHAPIN:  Okay.  Thank you very much, Dr. Zhan.  All right.  Dr. Miglino.
                  DR. ANDREW MIGLINO:  Hi.  So, I'm going to keep this brief because I think a lot of what I had written down has already been said.  But I'm going to stress a couple things in here.
                  So, the strengths have been well-established here.  I think the model actually is quite elegant for how nice and tidy it is.  It captures what you need to capture.  It gives you relevant data out of it that you can make regulatory decisions from.
                  And as long as the site meets the data criteria, it seems to perform well.  But that, I think, is one of the major points that has been brought up a couple times, is how many sites really do meet these daily criteria because they are quite stringent; especially since you need both concentrations, and you need some flow characteristics or some covariate.
                  And that's where I started to kind of think about difficulties in applying this model in the future to newer chemicals that don't have nearly as much data as the few that you've shown here, or maybe are a little stranger.
                  So, you had simazine -- I think that's the name of it.  Simazine in there, which did not perform very well because it was more of an intermittent use.  And I don't exactly know how you can capture an intermittent-use pesticide with this model.  But that's certainly another limitation on top of all the data limitations you already have.
                  There's also, I think, something that I'm going to echo in a couple other comments as we move forward, which is there needs to be some bright lines for when to switch covariates or when to switch your analysis procedures.  Because in some of the test cases, you have a flow covariate performs very well and is excellent.  But the same covariate for a different chemical no longer performs as reasonably as precipitation.  
                  And it didn't seem to make sense why that would occur.  Because they -- as it was just said -- they are pretty similar chemicals overall, at least the ones that you could use.  So that's something, I think, to look into, maybe just rules for proceeding down that or chemical properties that define what covariate.  I don't know, something to that effect.
                  Something that was brought up also yesterday by, I believe, Ian, was repeatability.  And I know it's probably not actually an issue of, is the number going to change by an order of magnitude.  But it is probably an issue of, hey, how come my number is different than that guy's number?  And so again, kind of defining that rule set to say this is how I chose my number, it's the same number every time, or extremely close to that same number every time, would be advised.
                  And I think that's probably everything I had that wasn't already said.  Yeah.  I'll stop there.
                  DR. ROBERT CHAPIN:  Brevity is the soul of delight sometimes.  Dr. Sobrian?
                  DR. SONYA SOBRIAN:  I'm going to be even briefer.  At this point, all the concerns that I wrote and some of the strengths have already been covered.
                  DR. ROBERT CHAPIN:  Okay.  Anybody else who hasn't been assigned?  Tom.
                  DR. TOM POTTER:  I'll try to be -- I will be brief, but I wanted to echo a few things that some of the previous discussants had highlighted.  And that includes identifying model weakness relative to flow and/or the absence of it.  I think that whole area needs further development.
                  And I'm still wondering what the actual equation is that exists in the White Paper, that shows the form in which precipitation data is used in the model.  Now, maybe it exists and perhaps I missed it, but if I did, I would appreciate your input on that.  So again, I see that as a major weakness, and we'll discuss that in some detail later on.
                  I also want to highlight the issue of potentially incorporating pesticide use data into the model.  I think several of the prior discussants have highlighted this in different ways; for example, indicating that cropping patterns change, there are new products on the market, et cetera, et cetera.
                  There's also the problem of intermittent use.  And those can be dealt with to some degree by incorporating a flow -- a covariate in the model that actually addresses the actual pesticide use.
                  And there's certainly data out there that could be used for that purpose.  The State of California is a leader in this area.  I think we heard that from Dr. Zhang in terms of compiling use data on a year to year basis.  And certainly, there's a rich source of information there.  And I think we heard yesterday mention of the USGS program that uses sales and NASS data to basically estimate pesticide use on a national scale by county.
                  Now, I do recognize that county data can't be directly imported into making an estimate on a watershed scale.  But again, I think there have been efforts applied in that area.  There are indeed publications, which I will provide as citations.  And with advancements in our GIS and our -- also our landcover databases, I think tremendous progress has been made, where I think actual pesticide use can be used in the model.  And to me, that's critical in terms of having confidence in the result.  So I offer that as a thought.
                  With regard to applications of the model, I realize, you know, we're in the early stages here.  But we didn't see any that represented cases where pesticide degradates that would be of concern or evaluated.  Certainly, that needs to be perhaps a next step in the evaluation process.  It may not be critical; depends again on the active ingredient you're looking at and whether or not you have difference of concern.
                  I will note that there was a paper -- again, I'll provide the citation -- that was published, I believe, in 2011 or '12 by the USGS team to look (inaudible), herbicide degradates in Iowa rivers.  And actually, the model seemed to fit the data quite well.  So, I thought that was really encouraging.
                  I also would like to see some more effort in looking at pesticide scenarios where there are multiple applications during the season.  There are lots of active ingredients that fall into this category, especially fungicides.  We didn't see a example on fungicides, and I realize fungicide surface waterData is pretty limited, so it becomes a challenge.  But I think that's a class of chemicals that definitely needs more work in SEAWAVE-QEX; in particular, looking at how wave forms respond to situations where you have multiple applications for a growing season.
                  I worked for many years with peanut growers, so I'm quite sensitive to that.  They have their tractors in the field every two weeks, literally spraying their crops with the same or different active ingredients, again, to grow plant diseases.  So you have season-long applications rather than a single application such as the case of a Post-Emergent herbicide.  So, absolutely critical that we look at some scenarios that deal with that.
                  Finally, I would like to, you know -- I'm not -- I wasn't quite sure where to shoehorn this in, but I think it's important to say, so I'll put it in here in my comment.  And that has to do with the constraint that SEAWAVE-QEX handles only daily data and can't deal with composite sampling data.  So, in essence, we can't use the valuable sampling approach of compositing if the intention is, down the stream, to use something like SEAWAVE-QEX.
                  I think there needs to be some thought put into that, particular with regard to the design and construct of monitoring progress.  And I greatly appreciated the talk by our colleague from Washington State yesterday, who was using the tool to evaluate and assess future monitoring programs.  
                  Because obviously, that's something that we need to take away from this meeting.  Not necessarily the use of the tool today, but where we're headed down the road in the future in terms of the data collection process.  I certainly commend you for doing that and I think there needs to be more work in that area.
                  I will note that regard to this issue of a grab and composite sample, I saw a really interesting paper that was published maybe a couple months ago, in Environmental Science and Technology by a Swedish research group that monitors pesticides in surface waters there.  And they had this neat little tool where they were basically collecting weekly composite samples in rivers by using a microscale solid phase extraction system deployed in the river.  I think they called it a TIMPHY (phonetic) sampler.
                  But what was interesting about the results, and that seemed to validate the sampler quite well, is that they showed that they got almost twice as many pesticide hits when they composited as when they grab sampled, and they did side-by-side sampling.
                  So, compositing certainly has some potential for -- because again, because of the concentration effect in this case, to identify more active ingredients in the surface water samples above detection thresholds.  So, that's important, as we know, in terms of our dealing with other issues of censoring.
                  And the other thing they noted was that grab sampling appeared to underestimate pesticide peaks; which again, I thought was interesting and certainly worthy of consideration as we go forward in designing monitoring programs that mesh well with the use of this tool.
                  DR. ROBERT CHAPIN:  Excellent.  Thank you.  So Ken, do you have a --
                  DR. KENNETH PORTIER:  Yeah.  Two paragraphs.  And I promise after this not to talk that long or to bring up this slide.  I want to bring it up one more time.
                  While the Panel has not asked to comment on the specifics SEAWAVE-QEX model, there is one issue with the model that, I think, should be addressed as it represents a potential weakness.  
                  In the model the t-tM, the tM term in the model is specified as the midpoint of the time period being analyzed.  So, somewhere in the fitting process, the tM term is set.  And it's not estimated, and it's clearly a function of how the season is specified.
                  And in addition, it's clear that there should be a relationship between this tM term and the time at which W(t) maximizes; right?
                  So you get the tM term, obviously -- let me see what I said -- okay.  If the beta 4 term is positive, the t-tM regression component is designed to add linear increasing trend from the beginning of the season up to time tM, and then subtract linear trend to the end of the season.  
                  If the season is specified wrong and we start too early, or we end too late, the tM term has shifted.  Right?  So they end up getting some kind of added peak before the W(t) or some after the W(t).  And I think this may be kind of one source of situations where the model doesn't seem to work.
                  I'm really concerned with that particular term and exactly what it's trying to capture.  It's almost like capturing a quadratic added peak that doesn't show up in the seasonal wave function.
                  The next step of the procedure development, or development of the model, you might want to look into estimating tM.  So, rather than just specifying it as a function of the middle of the season, actually let the model try to set it and see how that relates to the middle of the season.  You might find that that tM term is close to the time at which W(t) is maximized; in which case that tells me something about the wave function may not be capturing exactly what's happening in the seasonal pattern.
                  Or the tM term may be far from that time, in which case you've kind of got two peaks, two potential peaks within the season, again, that the wave form is not capturing.  That's one of those things that came up.
                  And the other one was the issue of --
                  DR. ROBERT CHAPIN:  Ken, let me just interrupt for clarification.  Do you think this is a got to do right now or --
                  DR. KENNETH PORTIER:  No, no, I think, this is the next step of the development.
                  DR. ROBERT CHAPIN:  Okay.  Thank you.
                  DR. KENNETH PORTIER:  You know, everything we see in the model fits well.
                  DR. ROBERT CHAPIN:  Go ahead.  That's fine.  Go ahead.
                  DR. KENNETH PORTIER:  You know, one of the things they asked us to do is, where's the weakness?  And to me, this represents a weakness in the model; this is not completely tied down.
                  And then the second issue was the use of the model for intermediate use chemicals that Dr. Miglino talked about.  But I think potentially, this can be seen as a failure of the process to properly define season for the chemicals of interest.  As currently used, season is broadly defined as the potential window of use, but also the potential window of monitoring data.
                  For intermediate use chemicals, there may be multiple windows of use, high removal and high removal of the chemical between use.  Which this is not going to pick up.  This model is not going to -- if you define season widely, it's just not going to pick that up.  And it ties in with this tM and max W(t) term, trying to accommodate that kind of thing and not doing a good job.  So, this may require different monitoring programs to really address these kinds of uses.
                  The nice thing is, though, you have complete daily data for some of these sites.  You can actually experiment.  Well, what if I monitored closer to the season of use, higher.  You know, what does the model do with multiple seasons in the year, but monitoring that's targeted to the intermediate use chemical rather than target it to the whole season?
                  And one of the benefits of this model is that you can explore that kind of thing.  They've got it all in hand; it's just a little side experiment.  Thank you.
                  DR. ROBERT CHAPIN:  Ray?
                  DR. RAYMOND YANG:  Ray Yang.
                  DR. ROBERT CHAPIN:  Strengths and weaknesses?
                  DR. RAYMOND YANG:  What was that?
                  DR. ROBERT CHAPIN:  Strengths and weaknesses is what we're addressing now.
                  DR. RAYMOND YANG:  Yeah, yeah, yeah, I understand.  First of all, I want to compliment EPA colleagues for utilizing computational technology in your scientific and regulatory process, okay.  This is a really, in many ways, advancement of the TOX 21 initiative and so on.
                  And also, I want to compliment the effort you put in to put this White Paper together.  So not all criticism, okay?  As long as you put the model in your paper front center, like the Vecchia 2018 paper.
                  Now, I have one recommendation.  An imminent statistician, George Box made this statement.  All models are wrong, some are useful; and therefore, from that perspective, your model has weaknesses, and you need to improve it.  And from the history, since 2007, you have eight peer review -- and this is the ninth one -- is indication of that sort of process, of slowly improving.
                  And this is very, very much in the spirit and essence of Bayesian approach.  This is one of my recommendation, okay.  For EPA to consider Bayesian approach and coupling with technology such as Markov chain Monte Carlo, okay.  
                  Because I don't know how these parameters are set up, whether they have probability distribution or not and so on.  But one thing is certain, there's uncertainty, there's variability.  Okay.  If you need to deal with these issues, Bayesian approach, coupled with Markov chain Monte Carlo, is one very important aspect to consider.  And thank you.
                  DR. ROBERT CHAPIN:  Dr. Green?
                  DR. TIMOTHY GREEN:  Tim Green.  My comment -- oh, good, Ken's coming back in -- is a follow up to Ken Portier's comment.  It appears to me that SEAWAVE-QEX may be over-parameterized.  So, I wonder with the statisticians in the room, why you wouldn't throw out this term that Ken discussed; or just a thought even from my EPA colleagues.
                  DR. XUYANG ZHANG:  I actually wanted to be on record to say that I don't necessarily agree with the overestimating tM, because I don't think that if tM now becomes something that is estimated, the model is even identifiable, since you have b4 times tM, that is to estimate.  You also have an intercept term, so how can you distinguish what is the intercept and what is this b4 times tM.  So, I'm not sure that the overestimating tM is possible.  
                  I think that in some ways there was discussion about this issue of how to determine the time period.  The data that needs to be fed to SEAWAVE-QEX in the White Paper; when there was a discussion about the fact that for certain sites and pesticides, if you fed the whole data to SEAWAVE-QEX, then this wave was estimated to have two peaks.  When in reality, the data showed that there should be only one peak.  And you could remedy that by just specifying maybe I'm only fitting to SEAWAVE-QEX, the four months or three months when the pesticide was applied, rather than use the whole year of data.
                  I would just not embark into this exercise of doing model fitting, adding more complication to this model.  The other comment I wanted to make --
                  DR. ROBERT CHAPIN:  You're suggesting removing a parameter, right?  Right?
                  DR. TIMOTHY GREEN:  Tim.  Yes, that's what I'm suggesting, but I'm asking my colleagues for their opinion.  To me, you have redundant trend parameters.
                  DR. XUYANG ZHANG:  I also want to say that when the model is fit, it's all together, one.  It's not that you're estimating b1, and then you go a second time and you estimate b2.  You estimate everything together.  So, there is going to be some balance that is going to be reached between the seasonal wave and the linear trend to best fit the data.
                  So, I wouldn't be playing with the model as it is.  I think that the expression for the mean or the expected daily concentration is following what we expect from the hydrological processes that govern daily concentration, being as simple as possible.  Of course, you can add more parameters, but that's making your model overly parameterized.
                  DR. ROBERT CHAPIN:  I'm sorry, George, did you -- were you waiving your hand?  Rebecca?
                  DR. REBECCA KLAPER:  Rebecca Klaper.  Just a short comment, because a lot of the things that I was thinking about have been mentioned.  
                  But one, going along with what I had brought up yesterday and Dr. Zhang brought up again today, about evaluating this with compounds that are not of similar chemical characteristics.  The four compounds are all relatively insoluble in water and have very similar chemical characteristics.  
                  So, it's unclear -- although, I think this is a really interesting way of solving the problem and trying to estimate pesticides, it isn't clear from the evaluations that you did that this would be applicable to a water-soluble compound.  And I understand that there are a lot of pesticides that are not water-soluble; and you're more interested in the pots because they have the greater potential for cancer, et cetera, over a long period of time.  
                  But it would be very important to eventually have an evaluation of how well this works for a more water-soluble pesticide from a risk assessment perspective.  Just to demonstrate that this also works, in that case, since you're trying to tie it to 1-, 4-, 21-, and 365-day exposure periods.  
                  And that was just my comment from that.
                  DR. LISA NOWELL:  Lisa Nowell.  I wanted to go back to the trends term for a moment, because it's  my understanding that that's what that term is.  And this originally was the SEAWAVE-Q model.  And I mean I believe that was what this was designed to do, was to assess trends over time.
                  And with respect to something that Dr. Baffaut said, I actually would like to solicit more input on this.  Do you think this model actually does assume constant use?  Because I thought one of the functions of that trends term was to address this in an empirical way.
                  My tendency is to agree with Veronica that it's dangerous to start pulling pieces out of the model.  And I believe Dr. Portier also said that this was -- the model is complex and if you start messing with pieces of it, it's not going to work as well as it does necessarily.
                  And I also just want to mention one more thing.  I believe there is the -- of the four test compounds, there are some water-soluble ones.  Fipronil is the least so.  It's probably more different from the others -- well, I guess chlorpyrifos actually also has a fairly high Koc.  But there is some variability among the four, maybe not as extreme as if we had one of the pyrethroids included.
                  DR. REBECCA KLAPER:  It's not variable enough for --
                  DR. LISA NOWELL:  Yeah.  And it would be great to have a fungicide in there too.
                  DR. REBECCA KLAPER:  Yeah.
                  DR. LISA NOWELL:  I did notice that azoxystrobin was listed in one of the tables as having sufficient sites and data to do SEAWAVE calculations, so that would be an interesting one to take on.  However, I believe you probably used the datasets that were in the Vecchia paper, so; I don't blame you for that.
                  DR. KENNETH PORTIER:  With regard to Dr. Green's question.  In 2012, the Panel spent a lot of time talking about this term.  And Dr. Vecchia had a very good reason for putting it in the model, and I can't think of that reason right now.  So, I don't really want to recommend getting rid of the term.
                  But when you look at that tM component, you can estimate it, but it becomes a nonlinear regression.  It ups the fitting process at least in order of magnitude.  You know, you got to go to something like an (EM) algorithm or something like that to estimate it.  And I haven't had enough time to think about whether it's non-estimable.  I have to think about that.  I think it is.
                  DR. VERONICA BERROCAL:  No.
                  DR. KENNETH PORTIER:  I think it is.  So we agree to disagree.  Our first disagreement on the first question.  I think it is with the full data.
                  But I'm worried as to what it's estimating, you know, what it's capturing.  And I'd have to go back to Dr. Vecchia's paper to try to understand exactly what aspect of the curve.
                  The issue about use, I think this is more about release.  I think the idea is that use in this spatial is attenuated over time.  Not every farmer is putting it out on April 1st.  And so, application is attenuated.  And so, it's presence in this stream is kind of long-term.  
                  And I think that's what this model is just trying to catch, the fact that it's used over a period of time.  It finds its way into the stream, over time, concentration builds up and then it decays out.  And that's all it's trying to capture here, is that time pattern.  Not get to too much details of mechanism.  I mean that's my impression.  
                  This is a regression model.  And you know, that's what regression models do, regress to the mean.  So it's mostly focusing on mean pattern rather than on the details.
                  The unfortunate part is we're really interested in the maximum.  So, the stochastic component becomes as important here as well.  And that's where the difficulty comes.
                  DR. ROBERT CHAPIN:  Okay.  I'm sensing that we're starting to eat our own tail here.  So let me turn to our EPA colleagues and say, do you have any questions for us?  Do you have any questions for clarification that you want to go back to various parts and pieces and get clarification?
                  DR. ANNA LOWIT:  We've heard a lot on, a good hour-and-a-half on the first question.  I was hoping maybe we could take a five-minute bio break and check with my team and see if they have anything specific.  If that's okay.
                  DR. ROBERT CHAPIN:  Something quick?
                  DR. CLIFFORD WEISEL:  Well, this is a question to Dr. Berrocal.  If I heard you correctly, you said that you thought the data -- the examples in the Vecchia project paper was very good, less so here.  And you said something about maybe 12 is not the right number.  Did I hear that correctly, or was I missing something?
                  DR. VERONICA BERROCAL:  No, I didn't say that 12 is not the right number.  I said that Dr. Vecchia had in the paper different simulation experiments; where he played with how much observation do you need to have and what is the frequency, and things like that.  And basically arrived to the conclusion that if you had at least one observation, you were able to do a good job at interpolating the pesticide data concentration.
                  But 12 is the lower bound.  So, if you have more than 12 it would be even better.  And so, my comment was that I guess maybe this was a wave to test how good the SEAWAVE model was; that when they tested they use the lowest number of observation possible.
                  DR. CLIFFORD WEISEL:  And do you think they should be going -- I don't know if -- do you think they should be probably pushing it a little higher for what they're trying to use?  Or are you dissatisfied with what they have?
                  DR. VERONICA BERROCAL:  I'm hearing that maybe this is the next question; so maybe we should wait.
                  DR. ROBERT CHAPIN:  My gavel.  Bam.  Look at your watch; we'll be back in 10 minutes.
                  [BREAK].
                  
                  DR. ROBERT CHAPIN:  So should I wait for the rest of the EPA chairs to fill in, or are you ready to go ahead and do this?
                  DR. ROCHELLE BOHATY:  Yeah.  This is Rochelle.  I think we can go ahead and go.  We were satisfied with 1(a).
                  DR. ROBERT CHAPIN:  Excellent.  All right.  Go for it.
                  DR. KENNETH PORTIER:  I was just going to say, we're no longer in disagreement.  In 30 seconds she showed me why I was wrong.  The tM term just centers the fitting of the model, so that the competence intervals on the estimates are as tight as they can be.  
                  It's like in linear regression, if you set the set point at zero and you're trying to estimate the slope, you don't get the best term.  But if you set it zero at the middle, you get -- you know what I'm saying?  Okay.
                  DR. ROBERT CHAPIN:  It reminds me so much of a pre-calc teacher I had one time.  We started off January 2nd, he picked up his chalk and said, now, as I was saying, and just kept on going.
                  MS. JESSICA JOYCE:  I just want to clarify that you're saying that we shouldn't be changing the way the tM is handled.  Thank you.
                  DR. ROBERT CHAPIN:  Okay.  So, we're done with question 1(a).  Whew.  Okay.  Question 1(b).  And we'll let Dr. Bohaty read the charge question to us.
                             CHARGE QUESTION 1(b) 
                  
                  DR. ROCHELLE BOHATY:  Okay, 1(b).  EPA subsampled daily or near-daily measured concentration data to generate SEAWAVE-QEX input data that are more reflective of available non-targeted surface water monitoring data.  These subsample data were used as inputs in SEAWAVE-QEX, and the results were compared to the original source data.  Discuss the soundness of the approach for evaluating the use of SEAWAVE-QEX to develop reliable daily kymographs.
                  DR. ROBERT CHAPIN:  Okay, excellent.  I haven't been told to change the order of who talks when, so we'll just start off and work our way down.  And so, Dr. Berrocal?
                  DR. VERONICA BERROCAL:  Yes.  As it was mentioned, daily pesticide concentration of atrazine and metolachlor measured the monitors within the NCWQR were subsample by EPA to perform auto-sample evaluation of SEAWAVE-QEX.
                  The sampling frequency used to create a daily subsample was either 1 in 7 days or 1 in 14 days.  At first reading of the White Paper, I was not entirely clear what type of procedure was used to derive the subsample.  Probably also because of the fact that the White Paper described another subsampling procedure at another point in the document when describing the sampling bias factors.
                  At a second reading and during yesterday's EPA's presentation, it appeared that the strategy adopted was to use a systematic sampling approach with only the start day chosen randomly within a range of possible dates.  Between the first sampling date and the midpoint of the pesticides monitoring period.  In both situations, once the start date was selected, subsequent pesticide concentration data was sampled every 7 or 14 days, as long as 12 daily measurements were subsampled.
                  The charge of the sampling procedure was selected to obtain pesticide concentration data that are more representative of a typical sampling schedule for pesticide monitoring sites.  Five datasets of daily pesticide concentration sampled every 7 and 14 days were used as input to SEAWAVE-QEX.  
                  While I understand the sampling interval of 7 or 14 days is representative of real pesticide concentration data, I believe that the inter-frequency, so sampling every 7 or every 14 days, might be disruptive of the dependent structure.
                  Daily pesticide concentration data is not an independent series of observation.  And subsampling observation should be done in a way that it preserves the original dependent structure in the data.
                  In the 2018 paper, Vecchia showed in the simulation experiments the bias and 80 percent upper-bounds were smaller when the pesticide concentration data was sampled with an inter-measurement time interval that is smaller than the concentration time scale.  And it also showed that the concentration timescale for many USGS sites was estimated to be below 5 days as it's shown in Figure 16 in Vecchia's 2018 paper.
                  I do believe that the idea of subsampling is a sound idea.  Vecchia himself, in the 2018 paper, suggested to (inaudible) the original time series of daily pesticide concentration data to avoid the SEAWAVE-QEX model is overfit.  However, I believe that Vecchia (inaudible) the data so the data was subsampled every 3 days, again, making sure that the sampling frequencies were smaller than the estimated CTS.
                  Another issue with the subsampling procedure was that the subsample data typically has lower daily pesticide concentration values as the original daily data, as we see in Figure 9.2 in the White Paper.
                  Given the smaller magnitude and concentration of the subsample pesticide data, there's no surprise that the interpolated daily pesticide concentration obtained from applying SEAWAVE-QEX might be underestimating, particularly when looking at the 365-day average concentration.  I already made a comment on this sampling frequency for charge question 1(a).
                  It was also not clear to me why, for one side in the AMP program, random daily sampling was allowed.  However, it was not performed for the NCWQS sites.
                  Finally, I believe that EPA should investigate whether the results obtained in terms of characterization of long- and short-term pesticide concentration using the SEAWAVE-QEX, just depend on the fact that only five subsamples were generated.  Or is this really a presentation of the actual performance of the SEAWAVE-QEX?  So, it was just chance, or it is something that you would see if you were doing more than five subsamples per site on pesticide?
                  DR. ROBERT CHAPIN:  Okay.  Dr. Portier?
                  DR. KENNETH PORTIER:  Thank you.  There's a little bit of overlap between -- I'll just go ahead and read my comments.
                  DR. ROBERT CHAPIN:  I hoped there would be some.
                  DR. KENNETH PORTIER:  The soundness of the approach depends on fully understanding the stratified sampling process.  Unfortunately, the sampling process is poorly described in the White Paper; with some key details glossed over, making a firm assessment of the soundness of the approach difficult.
                  Page 111 of the White Paper describes the process of creating a stratified subsample from the NCWQR data.  And I quote:
                  "To meet the suggested SEAWAVE-QEX minimum number of samples, each dataset contain 12 samples per year.  The first concentration for each subsample dataset was from a random date, between the first sampling date of the year and the midpoint of the sampling period (the start date limited to Spring) and samples were taken every 7 or 14 days, until 12 samples were subsampled for the year".
                  And to me, the statement wasn't clear, although clarified yesterday in the presentation by the EPA staff before the Panel.  The goal seems to be to create a sparse subsample that exactly captures the original observed concentration timeseries, at its selected dates that occur exactly 7 or 14 days after the initial randomly selected sample date.  These are in effect systematic samples.
                  The study might be what is called a "what if" study; designed to explore questions, such as, what if we only had weekly grab samples that started at a random day during the first halves of the season?  How does the estimated maximum concentration change if one only has biweekly grab samples?  Oh, and don't forget that one has the complete flow time series to help recreate the concentration time series.
                  This last part is critical to the effective use of SEAWAVE-QEX, since both the MTFA and the STFA are very important in constructing trend and adding variability information to the simulated traces.
                  Also important is the definition of the characteristics of the minimum sparse dataset.  Three years of data, 12 data points a year, no more than 70 percent no detects.  This means that the smallest usable dataset might only have 36 sample points, of which 30 percent or 4 or 5 datapoints are actual detects.
                  The SEAWAVE-QEX model has about seven parameters that need estimating.  While some of the parameters do get support from the flow time series, the covariates, it's still asking a lot of 4 or 5 real datapoints to estimate this much model.
                  Note that for the AEMP data, a random sampling strategy is also examined.  The White Paper, on page 14, states that the random process selects 13 values from each year but fails to indicate how this is done.  It's assumed this means selecting 13 dates, from each year at random and retaining associated concentrations; since the White Paper talks about random samples, including neighboring dates, which requires dropping some samples because of the minimum three-day spacing rule.
                  An alternative could be selecting 13 concentrations from the statistical distribution of concentrations, developed from the full dataset for that year, and then assigning those concentrations to randomly selected dates.  The two methods have the potential of generating quite different results.  And I think they should be further clarified in the White Paper.
                  The ultimate goal of the subsampling is to create synthetic sparse monitoring datasets from a complete dataset where the truth of maximum concentration is known.  Using five subsamples allows us to see how limited monitoring, combined with SEAWAVE-QEX modeling, produces variability in maximum concentration estimates and allows us to look at some confidence intervals.
                  For each subsample, one can assess the closeness of maximum concentration estimates to the true maximum, but more importantly assess the extent to which estimated confidence intervals include the true maximum.  Looking across the five subsamples gives one an idea of how far off the point estimates can be from the true values, and how often the estimated confidence bands do not include the true value.
                  Using only five subsamples limits how well this sample-to-sample variability in the estimates can be determined.  It would've been better to examine 20 or 30 subsamples and perform a more rigorous analysis of within and between variability, in maximum concentrations values generated, and report that in the White Paper.
                  The assessment of point estimates is performed by looking through the figures presented in Chapter 12, Appendix E of the White Paper.  The dot plots allow assessment of how different point estimates of maximum concentration, compare to measured maximum or true values for different pesticides, locations, and years.  The summaries are provided in Section 9.3.1.2, and Figures 918 to 920.
                  It's difficult to understand the take-all message from this study.  The summary concentrates on the percent of time the SEAWAVE-QEX estimates from the limited monitoring subsamples are lower than the true measured maximum concentration.
                  The White Paper offers that this occurs approximately 15 percent of the time, which does not sound very bad.  But Figure 920 also suggests that when the SEAWAVE-QEX estimates from the limited monitoring exceed the true maximum, it's typically less than twice the true maximum, but can be as high as 19 times the true maximum.
                  No effort seems to have been applied to exploring which site or sample characteristics correlate to better or poor estimates.  Such information might further inform when SEAWAVE-QEX could be used or could be effective.
                  And we pointed out during the break that -- a couple of us were talking -- that in the White Paper, there's no real definition of what's the target for this estimate.  Is 15 percent underestimate an acceptable level?  Are you targeting 5 percent?  You know, do you want overestimates to be below 2 times the true, or is 19 times acceptable?
                  So, there's no specification of what the target for the estimation is.  And that kind of makes it a little harder for us to be able to come back and say, you did a good job.  You know, because we don't know what you were trying to accomplish at that point.
                  The difficulty of applying SEAWAVE-QEX to sparse monitoring data is tied to the general difficulty of properly estimating the seasonal standard deviation, or the SSD term, in associated autocorrelation coefficient or the correlation time lag from so few data.  And I think Dr. Berrocal mentioned this.  Getting these statistics estimated well is key to producing maximum estimates that have good performance.  The results summary suggest that when the minimum data requirements are met, the maximum estimates are within some acceptable limits of over and underestimation.  But it's not clear what happens when this doesn't occur.  You know?
                  And I suspect -- and I think Dr. Berrocal probably would agree -- that there's something wrong with the autocorrelation term.  In the way that particular sample replicate is set up, you're just not able to capture that autocorrelation term.  Which means you're either underestimating the variability, or you're grossly overestimating the variability.
                  I can't tell from this where that's happening.  And I'll leave it at that.
                  DR. ROBERT CHAPIN:  I'm trying to crystalize out a do it sooner rather than do it later thing from your collective comments.  And maybe a useful one might be to crystalize out the performance characteristics that they'd be happy with and do that sooner; right, the criteria for being a good estimation.
                  DR. KENNETH PORTIER:  So, they asked to address soundness; and we feel like it's a sound approach.
                  DR. ROBERT CHAPIN:  Okay.
                  DR. KENNETH PORTIER:  The next step is probably to understand, what's the characteristic of that small sparse subsample that caused that 19-fold maximum estimate?  There's going to be some -- something in the characteristics of that sample that's going to flag why -- and they haven't gone to that step.
                  And that's going to require generating more than five replicates.  You're going to have to generate a lot more replicates to understand where is this failing?  And that's going to then go back to inform which monitoring datasets look good that's going to work and which ones don't look good.  They don't have that information yet.  So it's going to inform the SOP --
                  DR. ROBERT CHAPIN:  And so, my sense is that might be a medium-term deliverable?
                  DR. KENNETH PORTIER:  I think that's the next step.  Yeah.
                  DR. ROBERT CHAPIN:  Okay.  Dr. Berrocal?
                  DR. VERONICA BERROCAL:  Yeah.  I wanted to add that I think, since you guys have the daily data for some of the sample -- at least almost the daily data -- I would look also into subsampling that was done not just every 7 days or 14 days, but even a shorter time interval just to see whether the performance improves when you're looking at shorter time intervals.  That might be informative for how monitoring should be done for certain pesticides and in certain sites.
                  DR. ROBERT CHAPIN:  All right.  Working our way back down the list of folks.  And we just move down the table; Dr. B?
                  DR. CLAIRE BAFFAUT:  All right.  This is Claire Baffaut.  I don't have that much to add to what has been said.
                  I would like to bring to the attention of EPA that the American Society of Agricultural and Biological Engineers, ASABE, really had a long thought process about model performance.  And we were focusing on process-based model.
                  But I think some aspect of a discussion, and the papers that came out of it, applied to statistical models as well.  There is a model calibration and validation guideline; it's called EP-621 -- and I can send it to you if you want.  And then there is a few peer-reviewed papers that go with it.
                  And in it they advise to set criteria.  And they advise to set performance criteria as a function of the model's objectives, how the model is going to be used.  And with the idea that if the model is used for regulatory or house protection purposes, then the criteria should be more stringent than if the model is used for planning purposes, for example.  I would recommend looking at those discussions and then reemphasize a need to set this criteria for this case.
                  I would like also to emphasize the need to look at the overestimation as much as the underestimation.  I understand the underestimations are needed for protecting health.  But I think the overestimations are going to be needed to get by in the model itself, for whatever it's going to be used for.  And that's important as well.  And it'll make your life much easier, I think.
                  And finally, for the purpose of comparing all those sampling strategies, personally, I would like to have the model parameters that come out of each sampling strategies, and sample all that so that I can see the difference in those model parameters.  And I can start seeing, you know, do those differences make sense or not?  Is there some rationale behind it?  And that's it.
                  DR. ROBERT CHAPIN:  All right.  Thank you.  Dr. Green?
                  DR. TIMOTHY GREEN:  Tim Green.  Just one small comment on sampling frequency.  And I certainly appreciate and acknowledge all the work that my colleagues in EPA have put into synthesizing a lot of information in the report and in your oral presentations yesterday.  Thank you.
                  This is about kind of the educational nature of your function.  What was shown in sampling frequency was, I believe, a weekly sampling that missed all of the peaks.  And that can actually provide kind of misinformation or a lack of confidence from savvy members of the public who would say, well, you chose the period where you didn't hit any peaks.
                  So that you can have a large sampling bias, instead, I would recommend that you shift your time, your start time, by a day, with the same frequency, and illustrate what happens over a range of start times.  And then you would have a more reasonable example, I think.  Thank you.
                  DR. ROBERT CHAPIN:  All right.  Dr. Zhang?
                  DR. XUYANG ZHANG:  Yes.  I -- so first of all, I'd like to acknowledge that the sampling, the current subsampling approach as representative of the typical monitoring programs.  And I think overall, the evaluation is sound.
                  In the current EPA evaluation, EPA used the two types of results in model evaluation, the summary statistics and then the full distribution.  I think these two types of model output are appropriate and representative of the exposure parameters typically used in drinking water assessments.
                  However, there are a few aspects of the approach, I think, that could be further improved.  The current approach, which compared the SEAWAVE-QEX generated results using subsamples to the logs integrated full dataset.  It's across model evaluation methods, but I think there is a risk of overfitting of the model for mainly two reasons.
                  First, the SEAWAVE-QEX always tried to generate the best fit to the data, to the particular dataset of data that the users provide.  And the second is that the step sample dataset, they're essentially from the same population, which is the full dataset.  And this does not speak to the performance of the model when applied to datasets that have different seasonal waves and trends from the current dataset.
                  In another word, there's a lot of independent validation.  I think all subsample of the datasets were used for model fitting, as I mentioned in my comments in the previous question.
                  So, I'm seeing some independent validation dataset is likely not available, I propose a cross-validation approach that could be used to include the evaluation in which a subset of the subsample dataset of set-a-side excluded from model fitting and then the set-a-side dataset are then used for model validation.  
                  So the cross-validation approach, it seems the way to mitigate the (inaudible) risk, considering a more definite to the evaluation, because the datasets may not be feasible.
                  My second point is that regarding to the starting date of the sampling.  So, the current sampling approach with 7-day and 14-day intervals, they -- it'll force the starting date in the Spring.  I think that would result in a lack of data from the winter months.
                  As I've mentioned yesterday, I did ask a question about, if we force the starting date always in the springtime, the latest date that we can sample, probably around August or September time.  So the samples in October or in the winter months are not really fully represented.
                  Now, this may result in a better model fitting as the majority of sample did occur during the use season.  It doesn't reflect pesticide concentration in winter months.  And in some situations, the winter storms, it's actually the first flush after the dry period, it could often generate peaks in pesticide concentration.
                  And I think the current subsampling method may result in a lack of testing of the model for this kind of scenario.  And this is further confirmed by the artifacts of results that, as mentioned in the White Paper, that the 14-day sampling interval -- that result on the 14-day sampling interval outperforms the 7-day interval.
                  This doesn't mean that the 14-day interval samples are better than the 7-day sampling interval.  It's purely because the 14-day sampling approach covers a longer period of the year than the 7-day.  So, I think that's one of the drawbacks and that should be improved if we do not force the beginning day in the springtime.
                  In addition to the stratified sampling strategy, this is sort of related to the previous point, and also the point that Dr. Green just made.  I think a random sampling approach, which allowed flexible starting date, should be considered in the future work.
                  That's all my comments to these question.  And I'd like to get back to Dr. Nowell, her comment about the assumption of constant pesticide use.  But I'm not sure if I should address it now or later.
                  DR. ROBERT CHAPIN:  Well, you've got the floor, why don't you do it now?
                  DR. XUYANG ZHANG:  Okay.  All righty.  Yeah.  Well, Dr. Nowell mentioned that we should not assume constant usage in the model.
                  And I agree.  I didn't mean to say that the model assumes constant usage.  It's just that the use trend from year to year, when fitting the model across multiple years, because the model assumes the same parameter for that long-term trend, then it means that the model assumes the year-to-year valuation of the pesticide use follows a linear trend.  So when things happen in the pesticide use from year to year, that trend became now linear, then that would become an issue.
                  And I also wanted to acknowledge that the SEAWAVE-Q model is a very elegant and simple model.  And I really appreciate Dr. Vecchia's effort in developing the model.  My pointing out of the drawbacks, hopefully trying to bring to attention that the model should be used in situations where the assumptions can be met.  And using the models out of the context of the model assumptions could be very dangerous.
                  That's it for now.  Thank you.
                  DR. ROBERT CHAPIN:  Excellent.  Okay; thank you very much, Dr. Zhang.  Dr. Miglino?
                  DR. ANDREW MIGLINO:  Again I'll be quite short, because I think pretty much everything has been said.
                  The one thing that I'm curious about, and that I think I'd like to see investigated in the future though, is the repeatability from these graphs that you create.  Does the -- so you have five replicates in this case, right?  Can we compare them each other to say we're getting similar numbers, because we make similar trends, we make similar graphs?
                  I think that would be at least useful to determine whether it really is highly dependent on when I start this sampling frequency.  Or if I sampled well enough, I get the same numbers.  If that makes sense?
                  DR. ROBERT CHAPIN:  Is that a short-term requirement, or a midterm, or a longer --
                  DR. ANDREW MIGLINO:  I'd say midterm.  I mean, I don't think it's necessary to say, well, if the model works; I do think it is necessary to validate that the model is acceptable in the future.
                  DR. ROBERT CHAPIN:  Excellent.  All right.  And that brings us to Dr. Weisel.
                  DR. CLIFFORD WEISEL:  So again, I agree with what was said.  I just want to reemphasize one or two small things.  
                  One is Dr. Zhang's point about your sampling window that you're doing is not the whole year.  And for most of the compounds, you're probably okay.  But a couple places, you talk about multiple seasons for some compound, some pesticides.  If you actually have one, then you're going to have to expand that frame.  Now, the question is, is 12 the right number?  Which I asked before.  I don't know, but that may be something to look at.
                  The other thing I found confusing is, on page 112, you talked about this; you indicated that the 14-day sample can go for 3 months, and the 7-day sample can go for 6 weeks.  And I don't understand those  numbers.  I think the 7-day sample goes for 3 months, and the 14-day sample goes for 6 months, because when I added up 12 times -- yeah.
                  So, take a look at things like that and make sure that there isn't something that I'm missing there.  But I think the approach is right.  
                  And the randomized, I have less concerns, but that's probably about whether you do it every 7 days or you randomize that as was suggested because of autocorrelation.  Because I don't know enough about the autocorrelation in the water system.  But from a sampling perspective, I know more about air sampling; and what we do in air sampling is actually every 6 days.  Because we're worried about weekends and utilization.
                  I don't know if that's a problem here; whether the 7 days may always put you at the same day of the week and whether that matters in this or not.  But you should take those into consideration.
                  DR. ROBERT CHAPIN:  Okay.  Dr. Portier?
                  DR. KENNETH PORTIER:  Hi.  Ken Portier.  It's wonderful on these panels, where you think you've said everything; and then somebody mentions something, and synapses go off.
                  Dr. Baffaut mentioned this issue of estimated model parameters.  And as one looks through the diagnostic plots, the impression is that often some of the regression parameters are not significant.  And at the same time, the estimated values are quite small.  In regular regression, we drop these nonsignificant components and we refit the model.  And that small bit of variability that was explained gets dumped into the residual.  
                  And as I said in the last question, that SSD term is really important in estimating the maximum.  So, when you've refit by dropping a component, you've added variability, which means you increase the probability of seeing a higher maximum.
                  So I think that's something, in the short-term, that they should look at.  Is what happens when you drop -- and in particular, some of them, the nonsignificant term is the STFA.  Which means the STFA adds a lot of daily variability to the model, and now you're taking that out of a regression component, a weighted value, and you're throwing all of that variability into the residual term.  I think you will significantly impact the maximum.  You don't think so?
                  Dr. Berrocal?
                  DR. VERONICA BERROCAL:  Yes.  I would not drop anything.  The model is fit, has a specific expression.  And when you're fitting it, it's true some of the parameters are not significant, but they're not significant maybe because for that particular dataset, the regression coefficient was not significantly different from zero.  But if you remove it, you're changing the expression of what is the expected daily concentration.
                  I would just leave the model as it is and not change the model.  And do this ad hoc where the model changes depending what pesticide and site you are considering.
                  DR. KENNETH PORTIER:  Isn't it fun when statisticians start talking?  I would argue not changing -- I'm changing the model for that site chemical, not the model in general.
                  I'm saying, in that situation, STFA doesn't add to the explanation of the temporal pattern over those years.  So, what is the use of a statistical test on the parameters if you're not going to act on that test?
                  You're still explaining some of the variability with a parameter estimate that statistically you've told yourself is not important.  So, why wouldn't you act on that, drop that term, and add that variability to the estimate?
                  DR. VERONICA BERROCAL:  No.  So, statistical models can be designed for two purposes.  One is to study association, and one is to predict.
                  So, when you're doing a predictive model, as I think the SEAWAVE was designed, because it was designed to fill in whether daily sampling data is not available, you don't just throw away pieces because they don't turn out to be significant.
                  You look at significant terms when you are interested in association.  But here, we're not interested in association in determining are these particular terms in the model significantly affecting daily pesticide concentration.
                  We're trying to figure out with these terms in the model, how should I pull up one value or pull down another value so that I get the best prediction possible?  So, if you are changing the structure of the model, you are changing the prediction.  And that's the game that we're playing here.
                  DR. ROBERT CHAPIN:  Okay.  I'm going to break in and say that I'll get you two a table for lunch.  This is totally -- reminds me of my thesis committee.  Where the prayer was that they would get in an argument among themselves and just run out the clock.
                  All right.  Other people's comments on the soundness of this approach, this subsampling approach?  Anybody else?
                  All right.  Hearing none, I'll turn to my EPA colleagues and ask if you guys are happy or if you have questions for clarification from us?  Or if you think you understand everything you've heard, statistics aside.
                  DR. ROCHELLE BOHATY:  We do.  Thank you.  That was enough.
                  DR. ROBERT CHAPIN:  Life is good.  All right.  Okay.  Shall we move on to the next question?  Let's do it.  Let's move on to the charge question 1(c).
                             CHARGE QUESTION 1(c) 
                  
                  MR. DANA SPATZ:  Please comment on the use of precipitation and stream stage data as inputs into SEAWAVE-QEX, to provide reasonable estimates of pesticide concentrations.
                  Also, please discuss the suitability and limits of using methods for in-filling missing streamflow data.  For example, waterData R package, Ryberg and Vecchia, 2017.
                  DR. ROBERT CHAPIN:  And this time, we get to start off with a hydrologist.  Dr. Baffaut?
                  DR. CLAIRE BAFFAUT:  About the using stage as a covariate, the relationship between the flow and the stages of power function, which goes well with the Log linear model.  So, mathematically, it looks pretty good.  
                  The power of the relationship between flow and stage is usually not the same over the whole range of stages.  But it shouldn't matter too much because we are interested in the high concentrations, which are usually associated with high flow.
                  Note that stage will not work wherever flow doesn't work; meaning, where there is backflow effect, where there's tidal flows and so on.  In addition, it is difficult to use stage for reservoir -- and that's noted in the White Paper -- because stages may change minimally in response to flows coming in.
                  In addition, a reservoir stage is often managed and operated on by water releases from the reservoir, which have nothing to do with hydrological processes.  So, yes for flowing streams, no if there is back water effects, and probably not in reservoirs.
                  Using precipitation as a covariate.  So my problem with that is I said what I liked -- at the beginning I said what I liked about it; is that flow is an outcome of precipitation.  And as such, it integrates a lot of the processes that take place in a landscape.  Where if we use precipitation, we lose that big advantage.
                  So for example, if we use precipitation, there is nothing that differentiates precipitation on wet soils from the same amount of precipitation with the same intensity on very dry soils.  And yet, one may cause a lot of runoff in pesticide transport and the other one may cause no transport at all.  So that's the problem.
                  I realize it's a test run.  And the four Ohio watersheds show that precipitation could be used as a covariate.  However, again, on cultural land, in those four watersheds, is tile-drained, for the most part.
                  This makes a transport of pesticides from field to streams rapid.  The tiles kind of shortcut a lot of processes.  This may give an advantage to using precipitation as a covariate.
                  So, I would like to see use of precipitation tested in areas without artificial drainage and with high infiltration, where pesticides with high solubility and low absorptions have a chance to reach the aquifer and get back to the streams with the ground water.  And we would have totally different transport processes.  They would be slower.  And then I'm not sure what precipitation as a covariate is going to do in that case.
                  And for each of those tests, I would like to see the parameters obtained with precipitation stage and flow, so that we can assess the changes and whether they make sense or not.
                  And that's it.  That's all I have to say about this.
                  DR. ROBERT CHAPIN:  Excellent.  Dr. Green?
                  DR. TIMOTHY GREEN:  Yes.  Brief comment.  I retract my comment on 1(a).  I understand it's a long-term trend, so two different things.
                  On 1(c), I'm again acknowledging the work put into the White Paper, written for a broad audience.  It was a little overwhelming to process in a short time, so caveat is that I haven't read all of the supplementary information.  
                  But that was useful, so I do welcome your feedback.  And what I'm focusing on here is results not really the method.  So I'll just read this.
                  Covariates, daily streamflow and precipitation, noting not irrigation -- at this point, I understand, which in Washington also may be important -- are used to both interpolate and extrapolate measured pesticide concentration to daily time series.
                  Importantly, the values of interest to estimating extremes, maximum of daily concentrations are driven by covariate maximum used to -- I'm calling this "extrapolate" -- concentration values.
                  In the White Paper, on page 123, Figure 9.10, which is an apparent worse case, this illustrates an issue of estimating extreme values.
                  In this case, the maximum measured concentration, visually from the figure, is less than 10 micrograms per liter.  One realization is shown in the figure, with a maximum of 100 micrograms per liter, which is not near the medium.  And then the estimated annual maximum concentration from all 100 realizations is approximately 10,000 micrograms per liter.
                  So, all these were estimated visually from the figure, but this illustrates kind of the issue of estimating extremes.  Extremes from covariate relationships, in particularly, that impact these Log normal distributions, where the 80 percent confidence interval spans the full order of magnitude of estimated maximum daily concentrations.
                  As noted, Figure 9.10 is likely a worst-case scenario, which is affected by data sampling and censoring noted in the document.
                  Even so, a much better, maybe a best case, is shown in Figure 9.5 on page 118.  And that illustrates the same issue, but to a lesser degree.  That is the maximum measured concentration always falls below or maybe at the lower limit of the 80 percent confidence interval for annual maximum concentrations estimated using SEAWAVE-QEX.  Which, again, is driven by the covariate data streamflow, I believe in that case.
                  Likewise, all of the plots, which I reviewed in the supplemental files -- thank you again for those, and I can give you the folder where those are -- display these features, and its measured concentration maximum in each year fall below the 80 percent confidence interval of estimated annual maximum concentrations.
                  So, these comments are made in the context that estimation of extreme values is a very difficult statistical problem.  Other regression-based methods for extrapolating extreme values beyond measured values of concentration will likely face similar issues.
                  These comments are not intended to exclude use of streamflow as a covariate, but to caution -- and note that caution is warranted -- when none of the measured concentrations fall within the annual competency intervals; which seems to be a common case.
                  So those are concerns about, I guess, short-term interpretation.  Longer-term advances will likely demand more biophysically-based flow -- which Dr. Baffaut has referred to -- and transport models, which would only be feasible maybe in tier 4, or if you invented a new tier 5 approach to some data-rich watersheds as a proof of concept to improve -- again, it's all about confidence in the limits of estimating maximum concentration.  That's all I have.
                  DR. ROBERT CHAPIN:  Okay.  Excellent.  Dr. Kennedy?
                  DR. IAN KENNEDY:  I tried to review some of the things.  And I found it not as clear as I was hoping in this case.
                  One thing is that because precipitation on probably most days in most regions of the country is zero, the Log transformed that is done by SEAWAVE-QEX doesn't work.  And as a result, you changed it, basically, into a cubed root.  I'd kind of like to see some sort of explanation about why that was done over say, just putting in a default value; which I guess SEAWAVE-QEX has built in of .1 -- was it?  I don't remember exactly.
                  I didn't really understand why that was done over some other method.  And why a cubed root was chosen instead of something else.
                  I guess my other comment is that a lot of these were done on the NCWQR sites.  And those, as Dr. Baffaut mentioned earlier, are all in the same general area.  They're all going to have the same precipitation characteristics, similar flow.  I'm not sure how well the precipitation would work, in a more arid environment, where you have even more of those zeros in the precipitation.
                  If you use the dayshift concept and kind of move the precipitation along more slowly, to generate a kind of pseudo-flow, then that's going to remove some of those.  But it probably will leave some zeros still there.  And that's going to depend on where you are.  Also, areas that have more sudden precipitation might behave differently.
                  Finally, you asked about the infilling some of the data, which was missing, using the waterData R package.  And that package has a function called fill/miss, that will fill missing data in the time series.  And it seems to do it mostly just by linear -- something close to linear interpolation.
                  Maybe someone else can answer this better than me.  But it seems that maybe for a few missing values, that would work.  But the default block of missing values that it will handle is set for 30, which seems like it's way too much for that sort of interpolation.
                  And it think that's all I have to say right now.
                  DR. ROBERT CHAPIN:  Okay.  Dr. Miglino?
                  DR. ANDREW MIGLINO:  You're probably used to it; this will be brief.  I agree with what's been said so far.
                  Dr. Baffaut and I seem, I think, to have a brainwave thing going on here because she basically said exactly what I had written down.  And that complexity of you've analyzed only really one set of sole characteristics and drainage characteristics in one small area of the country.  That's a real potential issue down the road, right?
                  If we had a large dataset we could really determine what is going on.  And maybe it performs very adequately everywhere you apply it.  It's just not known to us.
                  And beyond just arid and Ohio and Idaho, some mixed-use watershed kind of data, right, the PCA, Percent Cropped Area, crops up in all your previous tier work, that may be applicable here as well; to say that there is some level of urbanization in the watershed, which would have vastly different characteristics than cropland.
                  So, the only other thing I'll say is about the inflow filling -- or infilling missing streamflow data.  I agree with what Dr. Portier has said.  It seems fairly reasonable over shorter timespans than 30 days; although, perhaps there's some justification out there for waterData to be applicable to those large timespans of a month.
                  I'd be a little more hesitant to kind of reach out to those extremes.  But it seems reasonable over shorter terms for filling things in there.  And I think a lot of this is going to carry over into the next discussion, the next question, where maybe things are a little more sparse.
                  DR. ROBERT CHAPIN:  All righty.  Excellent.  Dr. Potter?
                  DR. THOMAS POTTER:  Okay.  Well, again, thanks to the colleagues, the discussants who have spoken before me because they've obviously covered a lot of ground.  And I think they've emphasized the critical points regarding the variability of rainfall runoff relationships.  And those certainly need to be taken into account in evaluating the potential for precipitation for use as a surrogate.
                  There appears to be some cases where it worked quite well.  And so, further analysis of why it worked would appear appropriate.  It may have to do with the shape and size of the watershed, et cetera, a whole series of other factors, including tile flow.  So, certainly, again, a detailed assessment there would seem appropriate.
                  With all of that said, I will say that the use of precipitation as a surrogate for flow appears to reduce confidence in results.  So, I guess I would ask the question, is it appropriate to use?
                  Your confidence is the lowest among -- lower than, for example, stage and certainly less than flow.  So, I think it presents a communication problem, among other things, that you really need to take into account as you go forward with this.
                  DR. ROBERT CHAPIN:  Let me just interrupt and say, what they are looking for from us are recommendations, not questions.  So, if you've got a recommendation not to use it, that would be more useful --
                  DR. THOMAS POTTER:  That's my recommendation.  Yeah.  Yeah.  I think I stated that right up front when I said it reduces the confidence in results, and therefore you need to question why you're using it.
                  Okay?  And that's my position on it.  We can get into all the reasons why.
                  DR. ROBERT CHAPIN:  That's helpful.
                  DR. THOMAS POTTER:  Yeah.  And I will say that there are other regression models out there.  And I don't know if we've discussed them at all during this meeting.  But USGS has a WARP model, a watershed area regression model.  
                  And interestingly, in that model, they found that Dunne overland flow, which is a percent of rain discharge as runoff, was very informative when describing atrazine losses; also, the presence of the sole restrictive layer, and a whole bunch of other factors.
                  I would encourage you to take a look at the experiences of -- that were gained by that modeling effort, because I think it further informs the use of precipitation in this case in the model, and ultimately may guide decisions regarding the function that's used to describe precipitation.
                  And I understand at this point that it's a cube root.  I'll make a note that regarding use of precipitation, you may want to consult with a SEAWAVE model, with Dr. Vecchia and others who have worked on this project.
                  I read in his report and he states -- I can give the exact page -- but he states that if daily streamflow are not available, surrogate value variables, computed using estimated precipitation from the watershed, may be considered in place of streamflow.  Yeah.  I mean, that sounds reasonable.
                  And he goes on to cite Johnson et al.  So I actually went back and dug up the paper -- again, I'll provide the citation.  And in this paper, SEAWAVE-Q was used to describe pesticide monitoring data in a number of sites in the western US, in particular California.  
                  And in the model, transformed precipitation was used, but was used in conjunction with flow.  And there was a parameter that basically -- describing flow that took into account the 30-day antecedent precipitation, divided by the 5-day antecedent precipitation.
                  To me, that makes sense because it gets back at the question that Dr. Baffaut -- or point that Dr. Baffaut made regarding whether or not you have rain on wet or dry soils.  And antecedent precipitation certainly plays an important role there.
                  I suggest you -- again, maybe you can go back, and there's some further insight that would be gained by talking to the SEAWAVE developers, in particular, Dr. Vecchia, and also, Johnson et al. and that research group in California that used an alternate transform for precipitation.  It seemed to work quite effectively, again, combined with flow, not used separately.
                  We've brought up irrigation several times.  You know, again, I think it's an important factor for a lot of different reasons; including the fact that irrigation can have a strong impact on runoff in subsequent storms.  Again, rain on wet soil, et cetera.  So, I think that needs to be factored in, in some way.  Perhaps there can be a -- I hate to say this, but another parameter that can be added or it could be lumped in some way with either flow or precipitation to gain some insight in there.
                  I will say that well-timed irrigation incorporates pesticides into soil and reduces the amounts available for runoff.  And we led research effort in that area for a number of years.  And certainly -- it's actually a good conservation practice.  So, it's something to consider and contemplate.
                  What else do I have to say here.  Oh.  I will say that, you know, if we're, again, looking at this we ought to focus efforts on static water systems.  So, we're going to get into that into the next section.  But, you know, whether or not it's appropriate.  Either stage, which we've heard is probably a challenge to use in a static water system because of the small variation in stage.  Or the precip, indeed, is appropriate to use in a static water system.
                  I think priority needs to be given to that area.  And we'll talk about that, I think, in the next question.  I agree that the infilling approach is useful.  And there are a lot of ways we can attack it, but I think you've -- you certainly have a technically defensible approach there and it seems reasonable.
                  You know, one of the things we have to remember is that there's a lot of data out there that you may want to use.  Whether it's flow data or what not, that's provisional.  And so you're going to have to make a decision about, you know, whether or not provisional data is used.  The geological survey has a very nice disclaimer, which they say, all data is provisional until it's not.  So, I think that's reasonable to take into account.  
                  And I think I'll stop there.
                  DR. ROBERT CHAPIN:  Wonderful.  Please do.
                  DR. REBECCA KLAPER: Tom, since one of your points is that the precipitation doesn't necessarily -- it's just not good in the model, probably for a variety of reasons, right, it goes through all sorts of different landscapes before it gets to that river.  Wouldn't irrigation, then, also be bad to add to that model?  Because essentially, it's false precipitation, it doesn't necessarily add -- it's streamflow and it would capture what kind of irrigation ended up in that stream.
                  DR. THOMAS POTTER:  Yeah, I admit I was going out on a limb on that one.  I'm not suggesting that we add to the model.  I'm just saying that somehow, if we indeed, we're going to consider precipitation in the model, we have to consider other water inputs on the landscape.
                  And that's irrigation.  So, certainly that needs to be taken into account in some form.
                  DR. ROBERT CHAPIN:  Like my professor said, it's complicated.  Okay.  Mr. Councell?  Go ahead.
                  DR. LISA NOWELL:  Hi.  It's Lisa Nowell.  Without disagreeing with all of the things that you said, all of the complicated real factors, I'm just sort of being devil's advocate here, I think.
                  If we don't have flow, you can't use it?  Is that absolute?  I mean that's basically the question, because we don't have flow most places.  So, if they can't use it with precipitation, then it'll make a big difference in how usable the model is.  Just from a practical point of view.
                  And in terms of the work model, I just want to add one qualifier; which is that that generates a 4- or 21-day moving average.  So, it's not going to get those high estimated concentrations, which is what this whole effort is kind of pushing at.
                  I know that they used their work model or have used it before.  So, it's a good suggestion, but it's also got limitations.  They all do.
                  MR. TERRY COUNCELL:  All right.  And Dr. Nowell stole part of my thunder.  And that was what I was going to say, is back when I was with USGS, the number of sites where they were measuring flow and discharge was shrinking.  And I understand with frustration and further budget cuts, that those number of sites are continuing to reduce and get smaller and smaller.  
                  So therefore, the usability of your data, and being able to use this model and have that flow data, is going to keep getting smaller, and smaller, and smaller.  And so, you have this nice neat model, and you're going to be restricted even further with what you can use it on.  So having something to fall back, maybe precipitation or something else, is good.
                  One of the challenges that you have is that our country spans an entire continent; and we have all these ecoregions and climate regions, and it's a huge mess.  And it's very difficult to deal with.
                  And so when you did your examples, you did it in the Ohio watershed.  And we have so many other climates, you know, arid, tropical rain forest out there in the Pacific Northwest.  I think you need to move it around and show that the precipitation or some of these other models work in some of these other climate zones so that you can show that.
                  I did appreciate that you had a hierarchy, and you recognize that having the flow as your preferential use of the data.  And then the stage, and then the precipitation data.  So, you all have been playing with the model, and you know well some of the caveats of it, so I appreciate that.  Maybe having a good hydrologist on your staff would be a good thing.
                  The other thing I would say is, is that each of these situations that you're going to with all these different climate zones, document what you do very well for transparency.  Because, you know, there's going to be so many little nuances; you might have to do something here, but over here you can't do that.
                  And you know, there's no perfect model, as, I think, all the caveats that we've shown.  So, the best you can do is document what you do for the future and move on.  Thank you.
                  DR. ROBERT CHAPIN:  All right.  Ken?
                  DR. KENNETH PORTIER:  Ken Portier.  I wanted to follow up on something Dr. Nowell said.  What we're talking about for the near-complete data is an imputation model for the MTFA and the STFA time series; right?
                  And they proposed using precipitation and stage data to kind of inform that imputation.  But for the near-complete time series, the MTFA changes very little.  So, if the gap of data is pretty small, linear amputation probably works as well as anything else you're going to do.  The bigger issue is the STFA, the Short-Term Flow Anomaly, and capturing that variability and putting that back into the model.  
                  I would kind of recommend that you go look at simply looking at the variability in the STFA and imputing just with random numbers, if the period is small enough, and see what happens.  Okay.  You can do an imputation within an imputation, if you like.
                  And I think you're going to find, that may work as well, is using precipitation and streamflow for a lot of the reasons that we've heard.  It becomes a bigger issue when we're doing the sparse data; right?
                  Because if you've got -- you've only got streamflow and you're missing a lot of the streamflow.  Now, it's really important to get the STFA pattern right, and to kind of capture that trend that the MTFA is adding to the model.
                  And as I was listening to the discussion, I'm hearing, well, precipitation doesn't quite have the same trend as the waterflow MTFA.  So, the beta parameter is going to change, and its importance in explaining variability is going to change and likely go down.  And then the precipitation variability, the short-term flow anomaly estimate, is going to be much more variable because precipitation's likely to be more variable.  So, you're adding more variability there and I don't know what that does.
                  I think the point is, you're kind of looking at what you're trying to do with these covariates.  And I would have liked to have seen what the MTFAs and STFAs look like for some of the streamflow data, and then what you replace them with for the precipitation or stage flow data.  So, we could've seen how the different covariates make to that component in the model.  I think that would help the White Paper to explain why it does or does not work.
                  Because I think in some situations, the precipitation data or the stage data really mimics those patterns well.  And in a lot of situations, it doesn't mimic them well at all.
                  DR. ROBERT CHAPIN:  Tom?  Other comments on the use of precipitation in the stream stage data?
                  DR. THOMAS POTTER:  I had a follow-up with something I forgot to say, and I was looking back in my notes here.  
                  And I'm going to say that if precipitation is used, again, you know, hopefully, as you go forward we'll get a more defensible position on that that will add confidence to the model output.  
                  One of the great challenges with precipitation is deciding what data to use.  I think we all know and appreciate that precipitation can be widely variable across the landscape.  It depends a lot on what part of the country you are.  I'll say that the part of the country that I worked for many years, precipitation is wildly variable, especially during the growing season.
                  There's a very nice paper that was done by one of my former colleagues, David Bosch, who analyzed all of the precipitator and the Little River Experimental Watershed where I worked for many years.  And I think it would be insightful to look at that, particularly if you're going to use precipitator in the southeast.  
                  But you really need to explore further, providing guidance into how measured precipitation data is going to be used.  I don't think it's a simple answer and one that you need to spend some time on.
                  And with that said, perhaps it may be useful to look at alternate estimates of precipitation that would operate on a watershed scale or larger scale, such as NEXRAD Radar Data.  I recall that it was suggested that that be used in the Spatial Aquatic Model, SAM.  And I certainly would support that.  I think there's been additional research effort in that area.  I'll provide a citation on that.
                  But I think, again, the whole issue of if you use precipitation, you're going to have to decide on which data to use.  And that matters.
                  DR. ROBERT CHAPIN:  Okay.  And other comments on the --
                  DR. XUYANG ZHANG:  Xuyang Zhang here.  I have two quick points that I want to add.
                  DR. ROBERT CHAPIN:  Go for it.
                  DR. XUYANG ZHANG:  Okay.  In the occasion that the flow data is moving, I think Dr. Baffaut had probably mentioned this.  But I recommend that we could possibly looking to using hydrological models to predict the flow, when the flow data is moving.
                  That sort of takes care of the drawbacks using solely precipitation data, because the landscape processes and the soil process we're taking into consideration, if we use hydrological models.
                  My second point is that if we deem to use precipitation data, I think we should consider irrigation in arid and semi-arid regions.  Coming from California, I know that irrigation is a big driver, and pesticide runoff, especially in arid and semi-arid regions; especially when the use site or the crop is commonly irrigated, using flood and the sprinkler irrigation.
                  You know, case by case scenarios, you know that the use site is an irrigated crop.  I think special attention should be paid to the irrigation water use and perhaps either add that to the precipitation data or incorporate it in some other way.
                  That's it.  Thank you.
                  DR. ROBERT CHAPIN:  Okay.  Thank you.  Other comments about the use of precipitation in stream stage data?
                  Okay.  Let me turn to my EPA colleagues and ask if you have any questions for us for clarification, if you want to go back to some comment that we made.
                  DR. ANNA LOWIT:  I'll ask one was, while the other members of my team huddle down there.
                  Dr. Potter, I've heard you say a couple of things that weren't exactly identical.  In your first round of comments, you had very strong recommendation to not use precipitation.  And then as the conversation went further, with Dr. Portier's suggestion of analysis and some of the other Panelists who were a little bit more tempered in their response, that you also tempered your response with some really nice comments about the weaknesses and some other things to do, and that sort of thing.
                  So I'd just like some clarity on the strength of your response.  Is it the latter that's more in line with the others that's -- we don't really like precipitation, but if that's all you got, just be careful with it?  Which is sort of what I heard from the others; as opposed to, no, don't do it at all.
                  DR. THOMAS POTTER:  I guess I would agree that there is potential for using precipitation data as a surrogate for streamflow.  But there needs to be further work to demonstrate that it's an effective covariate.
                  There seems to be some data described in the White Paper that supports that.  But the assessment is very limited.  And I just wanted to make the point in the outside of my comments that -- you know, and I've heard it here in the room during some of the presentations yesterday -- that there's less confidence in the outcome when precipitation is used for a surrogate side.
                  I make the point that, you know, you need to kind of think about that and what that represents in the overall outcome of the assessment.
                  DR. ANNA LOWIT:  Thank you.
                  DR. ROCHELLE BOHATY:  I think the only point of clarification might be -- I think we only heard one person talk about stage.  I just want to make sure that there was no other comments.  A lot of the discussion was on precipitation.
                  DR. ROBERT CHAPIN:  Anybody want to dip their oar into stage as it were?  No?  Well -- oh, Lisa.
                  DR. LISA NOWELL:  Lisa Nowell.  I agree with everything that Dr. Baffaut said about stage.
                  DR. ROBERT CHAPIN:  You happy?
                  DR. ROCHELLE BOHATY:  Thank you.
                  DR. ROBERT CHAPIN:  Okay.
                  DR. THOMAS POTTER:  This is Tom Potter.  I want to say something about stage.  And that is that in areas of the country where you have flooding, you're rating curve kind of falls apart at that point.
                  And so again, this relates to extreme events which are of concern here.  I think there's some nexus there where you need to look at your data and see whether you have a scenario where you have extreme flooding, the rivers are leaving their banks, and what that looks like in terms of your model outcome.
                  DR. ROBERT CHAPIN:  Okay.  All right.  Excellent.  I think we've had a productive morning.  I have a little after 12:00, so we get almost an hour for lunch.  So we'll be back here at 1:00.
                  Yeah.  Thank you all.  And we'll see you in 55 minutes.
                  
                                    [LUNCH]

                              CHARGE QUESTION 1(d)
                  
                  DR. ROBERT CHAPIN:  All right.  I've got 1:00.  We're going to start, and our EPA friends are here.  All right.  So who gets to read the question? 
                  DR. ROCHELLE BOHATY:  Hi, Rochelle here again to read 1(d).  Considering that SEAWAVE-QEX was developed from data from flowing systems and that community drinking water systems use drinking water from a variety of surface water sources, including low- or non-flowing systems, please comment on the utility of using SEAWAVE-QEX for low- or non-flowing systems.  Is the panel aware of a better tool for infilling monitoring data for low- or non-flowing systems? 
                  DR. ROBERT CHAPIN:  Excellent.  Sort of the obverse of some of the stuff we've just been discussing.  All right.  And I've been told that Dr. Potter's going to lead off on this response.  Okay.  Just a second.  There'll be a momentary -- everybody take a breath.  And while you're doing that, I'll remind the group that we've been requested to sort of couch our needs and recommendations as to immediate need, mid-term, and long-term.  Okay.  The more we can do that the better, more useful our recommendations will be for our EPA friends.  Dr. Potter, over to you. 
                  DR. THOMAS POTTER:  Okay.  Thank you.  Well, I guess I'm responding to this one with a risk assessment hat on, so I'm going to start off with a few comments in that area.  And essentially, they identify something you probably already know that maybe needs to be priority placed on evaluating concentration dynamics in static water systems.  I found a paper by Mossgan (phonetic) et al. that looked at the upper centiles of atrazine concentrations, comparing static and flowing systems.  And indeed, for raw water, the upper centiles for atrazine were always higher.  So there seems to be a potential for peak concentrations being higher in static systems.  In addition, the duration of sustained concentrations is likely much longer.  So that's also a problem in terms of exposure in many ways.  
                  I'll say this was indicated in SEAWAVE-QEX assessments with ways for static systems that were analyzed that had -- the waves had very low amplitude and long duration.  I also think that was somewhat problematic in terms of the application of the model.  I think there would be much greater uncertainty associated with that wave shape rather than one that has a much higher amplitude that comes to a peak and follows a decay.  
                  I did spend some time, probably too much time, digging around trying to find out just how many community water systems out there are static and flowing.  I did find a presentation that was made at the environmental working group meeting some years ago.  And in that presentation, the authors had looked at the water systems as part of the atrazine monitoring program, and 70 percent of the systems were static.  
                  And again, maybe you guys already know all of this.  But that certainly, again, was another indication that, from a risk perspective, these systems should take priority.  And what I'm going to do here, because there are several other folks here with hats on, they're going to talk about the model and hydrology.  So I'll stop here and defer to my colleagues. 
                  DR. ROBERT CHAPIN:  All right.  Excellent.  And then, Dr. Miglino? 
                  DR. ANDREW MIGLINO:  Okay.  So unlike the previous ones, I have a lot more to say here.  And the first thing is I kind of want to define these terms low- and non-flowing systems because they're not clearly defined in the white paper, and to me, that means there's a lot of variability in what we could be talking about.  So when I'm going to be talking about a low-flow system, I mean it never dries.  It just has a very low flow and may be very sensitive to changes in flow.  Non-flowing I'm going to consider maybe locked, and it may experience an overflow.  But it's a rare event, so it's typically dry.  Okay.  
                  So for a low-flow system -- I'll talk about that one first.  I don't really think the model is inappropriate.  I know that was a weird way of saying that, so sorry if I scared you.  It should probably work.  You just have to be very sensitive to those events where flow would suddenly jump.  And I believe in one of the Vecchia papers, or maybe it's in the white paper, there's some discussion that one of those anomaly terms can handle that kind of event.  But regardless, there still needs to be some care in application to those systems because those are short-term, potentially very high concentration events compared to the normal standard scenario.  
                  Those events are also probably never going to be sampled or rarely sampled.  So there's even less confidence that the model is treating them appropriately or getting close to those estimates.  So again, care, not that it's inappropriate, just take care.  
                  The non-flowing system poses, in my mind, a bunch of additional questions and thoughts.  And I'm not certain that the white paper really addresses whether SEAWAVE is appropriate for those kind of systems to the detail necessary to really make an assessment.  You discuss the anomaly terms and whether their p values are sufficient to show statistical proof that it's reasonable but not really what they are doing, if that makes sense.  If you look into the white paper and Vecchia and kind of try and parse out what these terms are trying to do in a mechanistic way rather than a statistical way, they're described as positive values, meaning that flow is important.  Negative values mean dilution is important over some timespan.  
                  Well, for a static system, flow is rarely important.  The input flow is supposed to be captured by the wave phenomenon, and the degradation is also supposed to be captured by the wave phenomenon.  So I'm not really sure what that short-term parameter is doing, other than making it fit.  It's not, again, to say that it's not appropriate to keep in there because we don't want to pull apart the model.  But it's just hard from a mechanistic modeling perspective to determine what's going on.  
                  The longer-term flow I think makes at least more sense because in filtration it's these other processes that take years, months, whatever the long-term kind of pieces.  The one real issue I have with the non-flowing system is these terms are now aggregating a lot of other terms that wouldn't be evident in a flowing system.  And what I mean by that are all the longer-term fate processes like slow degradation, absorption, burial, all of those are getting buried into these other terms that are called flow terms.  And those do have an impact in a non-flowing system that you may not exhibit in a flowing system.  Clearly, they make a difference in a flowing system, as well.  But they're going to have much more of an effect where the flow doesn't matter.  Okay.  
                  I had written in here that maybe we could explore removing the two anomaly terms and seeing what that's like.  But after this morning's discussion, I'll pull that back.  What I had been thinking is, because that wave term encapsulates so much of the input and mass loss of the pesticide in these systems, restricting it to two waves a year for static waterbody may not be the best restriction.  You may have multiple events much closer to a large runoff event that would be encapsulated by that kind of term.  So that's the discussion on non-flowing systems.  I just think more work needs to be there, and I think that a short-term, near-term issue is it really needs to be shown in more than I have p values that are under my level of significance.  
                  As for better tools, I have -- I am not aware of a better tool.  I will say that there are alternative tools that probably cause large headaches, other than things like kriging or things like that, which may be appropriate and could be used.  There are certainly hydrodynamic models that could be used to generate flow values on some kind of scale.  They are certainly going to be a headache to implement.  So it's kind of an effort versus reward there.  And they're also going to be prone to probably significant error.  So with all that said, I don't know of a way forward to get a non-flowing system enough data to simulate properly in SEAWAVE with flow values.  That's all I have. 
                  DR. ROBERT CHAPIN:  Okay.  Excellent.  Thank you.  And Dr. Green? 
                  DR. TIMOTHY GREEN:  So short-term responses affirming the consensus that no -- the answer is no, we don't actually have an alternative estimation tool to recommend at this time, unfortunately, and yet recognize the importance of surface water reservoirs and short-term recommend further more analysis of more reservoirs and water supply storage systems to guide either different sampling schemes and/or potentially a whole different model.  So this will address your question "Is the SEAWAVE model functional form relevant and maybe optimal for such waterbodies?"  
                  Longer-term, I'd like to address something that's a little bit ancillary maybe, but it is a scale issue.  And this actually related to item 2(c).  So this will all be here instead of there.  So it raises this issue that has not been addressed, to my knowledge, in the current evaluations, and it's the issue of watershed scale.  I have -- I think Don has a quick set of slides I want to show you on a reference to Augustus and et al in Environmental Sciences and Technology back in 2004 using monitoring data from the Heidelberg Water Quality Laboratory and the USGS dataset.  
                  And those analyses concluded that maximum daily concentrations generally decrease with watershed scale.  This occurs despite the fact that lower percentiles of concentration don't display a similar scaling behavior.  So the scaling behavior with watershed scale is particular to those maximum concentrations, maximum daily.  And the concentration estimated at the edge of field from those extrapolations match the pesticide root zone model by Carcel.  That model was by Carcel in 1998 that matches as well.  
                  So this just shows the map, the next slide -- and you've seen this.  This is just to show you that you obviously know these watersheds because you've used these data.  This is a 1996 time series.  Next slide, please.  We can skip this.  This shows a range of chemical attributes, but I meant to take that out.  You're aware of these.  Next slide, please.  
                  And then this is the scaling with watershed areas.  So this is actually streamflow, and this is per unit area.  And this just shows that, if you look at that P99.7, that's your daily maximum value.  And you see a clear with log/log relationship again.  The log is important.  So those look flat, but they're not so flat.  Next slide, please.  
                  And then this shows a few chemicals: atrazine, metolachlor, metribuzin.  And again, we see that the maximum, or the 99.7 percentile, has a clear scaling relationship.  Lower quantiles don't.  So if we're looking at average values of a year, we may not expect this.  So we need to be aware of watershed scale with all of these estimations, whether it's from a correlation model or, in this case, from data and be aware that your drinking water supply will need to scale according to watershed area.  I think that was the last slide.  
                  Oh, there's one more.  This is a normalized concentration.  And again, we see this log/log linear relationship with watershed scale.  So just wanted to point out that I wasn't aware of anything in the white paper supplement.  I might have missed it, but this is very important to your estimation.  
                  So based on this knowledge, peak concentrations will likely be lower in rivers of large watersheds than in upstream monitoring locations and contributions from large watersheds to drinking water supply storages.  And reservoirs are expected to follow this kind of input behavior as was talked about.  The cycling within those water bodies will affect the results.  And then again, this does get, as pointed out by your presentation, confounded by multiple contributing sources to these water bodies.  And that can cause complications in time series.  Thank you. 
                  DR. ROBERT CHAPIN:  Excellent.  Thank you very much. 
                  DR. TIMOTHY GREEN:  That's a longer-term need pointed out.  Let me just comment that that scaling relationship can be included in something shorter term.  I don't know how you would couch that as medium term.  But it's something that doesn't have to involve a very complex model.  Thank you. 
                  DR. ROBERT CHAPIN:  That's very helpful.  Thank you.  Okay.  And finally, the last listed person for this is Dr. Klaper. 
                  DR. REBECCA KLAPER:  Hi.  My answer I think is the same as what I'm hearing -- is that, no, I don't think you can use this model for doing reservoirs.  I think maybe low --
                  DR. ROBERT CHAPIN:  Sorry, Rebecca.  Could you move the microphone back?  Thank you. 
                  DR. REBECCA KLAPER:  Just in our experience in trying to model chemical contaminants in a semi-enclosed system, the flow models don't work.  To make the contaminant estimation within basically a reservoir, you have to do different hydrodynamic modelling in 3D space, and it's not the same as the stream almost point source moving down in one direction.  Modelling that is done in this kind of an application where -- and the inputs from the stream may be modeled as far as what's going into the reservoir at any period of time using maybe this existing example.  But then what happens in the residence time, et cetera?  The breakdown, the movement within that reservoir wouldn't be estimated within this type of model.  
                  The characteristics of the reservoir, including the underlying geology that determines the reservoir, the hydrological inputs such as streams, rivers, whatever's going in there, the bathymetry are necessary to determine the potential movement of a plume and what that plug of water is doing once it goes into that system and then how it ends up averaging over a period of time.  Surface water monitoring stations could potentially be used as estimation inputs into the model.  However, linear model estimating source and discharge downstream wouldn't be appropriate.  As much as a fore-creaking could estimate the distribution of a plume across the waterbody, if done, multiple measurements have been made over space, which, again, is not necessarily common for a lot of waterbodies.  The draw down rate due to the outflow from the drinking water intake would also have to be considered, as well as any other kind of outputs from that system, as it would affect contaminant migration in the system from the source.  And the source load would have to be calculated as an input rather than strictly the point measurements taken in this pesticide model.  
                  The EPA already uses several hydrodynamic models it seems from looking at your webpages.  There's the environmental fluid dynamics code, which looks at a bunch of different characteristics in lakes and estuaries and another one that is the -- you have inputs from the NARS estimations, inputs on lakes from lake surveys, which could inform parameters going into such a hydrodynamic model of a watershed into a reservoir.  There are also other publications out there and groups that have estimated things in estuaries and in reservoir systems, the CE-QUAL-W2 model originally by Huntinger and Buchak but now is most used -- I found a paper by Jesnak (phonetic) et al, which I can put the reference in our documents, from 2016 where they used it to look at modelling over several different watersheds.  And they have a website talking about their model use, which is going across many different watersheds.  So it might not answer, necessarily, the predictability of -- not the predictability but trying to estimate unknown when you don't have the data.  But it seems that maybe estimating one watershed from some of the characteristics from another would be better in this type of a model, instead of using that linear model.  
                  I'll also note that in the absence of the bathymetry data there are several people that have used other types of estimations, so they kind of figure out the characteristics of that watershed.  Hollister and Milstead have used GIS to estimate the lake volume, the maximum depth, and using GIS layers of the shoreline in order to figure out what the lake looks like in the absence of that kind of information.  So it can be used within the hydrodynamic models of the lakes in order to give you some information in the absence of that data.  
                  For low-flowing systems, I do believe that it's possible, if you're talking about a stream or a wide river that's not moving very quickly, that the SEAWAVE could potentially be useful.  But in those reservoir systems, my personal opinion is that it cannot. 
                  DR. ROBERT CHAPIN:  And I was just going to look over here to see if Dr. Portier had his hand/card up.  Dr. Ken? 
                  DR. KENNETH PORTIER:  Sorry.  This is Ken Portier.  I wrote two paragraphs, four lines.  So on the issue of a jump, I think the STFA might catch that because you'd get negative residuals and positive residuals.  So you'd see that.  Clearly, you'd see it in the diagnostics that that's happening.  
                  The other thing, it's not clear what happens to the model in particular to the estimated model parameters when there's no flow in the middle of the season.  So you've got a stream that has intermittent flow, and it just stops.  So the seasonal WAVE says it's supposed to be up here, concentrations up here, but you have no data and no flow to predict concentrations -- and in particular what happens when flow starts again, and you get a pulse.  So you've been storing up this pesticide, and now it's released.  Absolutely the model won't pick that up.  But again, the diagnostics are going to show you that the model's not fitting for that.  It's going to be obvious.  
                  I would think a short-term -- I put down an intermediate-term task for EPA could be to identify some of these complete monitoring sites that have this kind of no-flow problem and just document what happens to the model when you apply it to that scenario, the full dataset, because I really can't see in my mind the impact of three weeks of no flow in the middle of the season on the fit of this model.  You're going to put in zeros.  The MTFA's going to do something.  STFA's going to have no variability.  The trend is going to be affected, so I think it's a nice little task that could be done in a couple of days and documented. 
                  DR. REBECCA KLAPER:  Can I just ask a clarifying question?  If the system isn't flowing, that means there's no water going in.  So how would the pesticide concentration increase? 
                  DR. KENNETH PORTIER:  I'm saying it won't be until there's flow.  But I'm saying pesticides sitting there.  My worry is the pulse that occurs when it starts to flow. 
                  DR. REBECCA KLAPER:  But then that would be captured in their flow, right?  So the stream volume would increase, and then it would pulse. 
                  DR. KENNETH PORTIER:  Yeah.  So you might --
                  DR. REBECCA KLAPER:  Which gets captured in some of their -- you see the pulses going on in their system. 
                  DR. KENNETH PORTIER:  Right.  But I don't think, for example, the short-term flow anomaly would really capture all of that.  You'd be missing that big peak.  And then that big peak really should be affecting the maximum, and you might not catch it.  
But again, I'm not sure.  There's a lot of uncertainty here, and I think what they need to do is demonstrate it for some examples.  
But the way, in one of the meetings, the 2011 panel or the 2012, there was a whole day talking about low flow streams.  I remember Nebraska, during the middle of the pesticide season and all these little streams with no water in it.  And what do you do with it?  And I don't think they came to a conclusion either, what to do with it. 
                  DR. ROBERT CHAPIN:  Other comments about how to handle low-flow systems or other tools for dealing with that?  All right.  I'll turn back to our EPA colleagues and ask if there are any clarifying questions you have of the panel to clarify answers that we've given. 
                  MR. CHARLES PECK:  Hi, this is Chuck Peck.  I just wanted to clarify, with the hydrodynamic models that you talk about, are you suggesting that we use that information to infill monitoring data?  Or are you suggesting we use it to predict what's going on in a particular system? 
                  DR. REBECCA KLAPER:  That's a good question.  I don't know if I know the answer.  They're designed to be more of a predictive tool than to definitively identify a hole in the data where they were going to put a different number.  But since it is a predictive tool and what you're trying to do is predict exposure, it seems like it would be relevant -- more relevant than a model that doesn't fit that type of hydrodynamic system.  Did you have something else you wanted to say? 
                  MR. CHARLES PECK:  Thank you.
                  DR. ROBERT CHAPIN:  Other questions from our EPA friends? 
                  DR. ROCHELLE BOHATY:  We're good.  Thank you. 
                  DR. ROBERT CHAPIN:  All right.  Question 1(e).  Bingo. 
CHARGE QUESTION 1(e)
                  DR. ROCHELLE BOHATY:  1(e), in Section 6.6. of the white paper describes the utility of SEAWAVE-QEX in the context of designing a surface water monitoring program.  Please comment on EPA's conclusions regarding how SEAWAVE-QEX can be used to optimize a surface water monitoring program design in order to use monitoring data in pesticide drinking water assessments. 
                  DR. ROBERT CHAPIN:  Mr. Councell? 
                  MR. TERRY COUNCELL:  All right.  Thank you.  All right.  So our colleague from Washington State did an awesome job in presentation on using your SEAWAVE-QEX model and how he modified his monitoring program, so you have an excellent source right there who's working with you already.  So I wouldn't hesitate to reach out to him for advice.  But you have this model, and you have some data requirements for it.  Yes, use those three years of data.  You want a USG staging station nearby that measures flow.  You want all the past use data.  Put all those requirements in.  
                  The one thing I would say to EPA is put all this down and go do some outreach to a lot of these programs that are doing monitoring.  If they can slightly modify their programs and get you data as well, you have a win-win situation.  And I think they would.  The last I looked, the cost of doing pesticide analysis was about $800 to collect and analyze a sample.  So it's a fair amount of investment.  So if a program can get dual use out of it, they're going to.  So I would do that.  I would outreach to all these state programs and universities that are doing pesticide monitoring.  
                  The other thing I would suggest you do is somewhat what FDA did with their ELEXNET.  They took all the FERN, the Food Emergency Response Network labs, and they created this database called ELEXNET.  And they asked them to put all their microbial testing data and chemical data into that.  So maybe EPA could create a database of water monitoring pesticide data in which these various state and federal and local agencies could input that data in.  If we do that sooner than later, you might have some data by the time you implement your SEAWAVE-QEX model and using it in your Riggs assessment.  So that would be my suggestions.  EPA would have to do a significant amount of QCing of all that data but look what source you have for minimal input and cost.  That's all I have.
                  DR. ROBERT CHAPIN:  Thanks very much.  Lisa, you're up next. 
                  DR. LISA NOWELL:  That Section 6.6. does have a lot of useful information of features of a surface water monitoring program, so I'm just going to focus on those conclusions relative to how the SEAWAVE-QEX would obtain to that model.  And as Dr. Councell said, there are basic data requirements for SEAWAVE-QEX, so it makes sense if you can incorporate those into a monitoring program.  I had a little bit of -- there was a statement in your conclusion, I think.  
                  One concluding sentence is that, if surface water monitoring data are insufficient for use in drinking water assessments and modeling indicates estimated drinking water concentrations exceed the drinking water level of concern, then development of a well-designed, targeted surface water monitoring program may be an option to refine uncertainties in the drinking water assessment for those areas that are identified to potentially have concentration.  So I was wondering does this mean that you have the ability to request that a monitoring program be set up?  Is that what we're talking about, or are you talking about sort of incidental use of existing data and the kind of tweaks that we were hearing about?  
                  MR. DANA SPATZ:  We do have the authority to ask registrants for pesticide monitoring data. 
                  DR. LISA NOWELL:  Okay.  That obviously would make a difference.  So the data requirements were pretty clear-cut.  So I think it makes sense to follow the number of years of sampling and the per sample-able censorship.  Of course, that you don't know until you get your data back, but you can make some estimations.  And presumably, if you're talking about a designed or targeted monitoring program, you're talking about a chemical that is frequently detected, at least in the area of concern.  For a targeted program like that, it's very reasonable to develop a sampling frequency that would be designed specifically for that pesticide, so that makes a lot of sense.  And all the arguments are spelled out in this section very nicely about why that might be necessary.  
                  In terms of making use of USGS data or any other external surface water monitoring program data, I just want to point out that, usually, we're using broad spectrum methods.  So we're analyzing not for one compound but a lot of them.  So we're going to make any decisions on sampling frequency.  It's all or nothing.  So we would either have to go with the most demanding chemical or, if we were trying to meet that specific objective, we might not have as much flexibility, I guess.  
                  You have a table in there, 6.1 Components for Surface Water Monitoring Program, which has a lot of information in it.  I found the format a little odd in that there was a section on considerations and then one on program elements and one on additional.  And I thought there was some redundancy.  I just thought that might be more useful if things were separated a little bit and the elements of monitoring program, say that particular table was just listing the elements of monitoring program.  But I thought your discussion following those was excellent.  
                  Oh, and yeah.  A very important point that you make is that the analytical methods have to be sufficiently sensitive to detect and quantify pesticide concentrations in water that you have to reach biologically significant thresholds and also that the QC has to be adequate.  And I'm sure that's one of your stumbling blocks in terms of putting data together from various sources.
                  DR. ROBERT CHAPIN:  Alrighty.  Dr. Kennedy?  Make sure you're close enough to the microphone, please, sir.  Thank you. 
                  DR. IAN KENNEDY:  I'll just make a few more comments, particularly on section 6.6.
                  DR. ROBERT CHAPIN:  So let me just remind you that we need volume for the transcribers so that they can capture it later.  So that's right.  Pull that sucker in there. 
                  DR. IAN KENNEDY:  On page 99 it says, "If monitoring data are collected for use in a model such as SEAWAVE-QEX, daily sampling is not necessary as the tool can provide estimates of these values."  And of course, that's probably true.  But you don't necessarily want to be designing your sampling program too closely to the requirements of SEAWAVE-QEX because something's going to go wrong.  At some point, you have to account for Murphy's Law.  And in generally, I think it's always better to have more samples rather than fewer.  
                  Another thing is, of course, that you only evaluated this on five chemicals, one of which didn't work, and a limited number of sites.  So if you can ever have a sampling program that would let you expand that evaluation, go for it, especially if you can do it in different regions with different chemical properties and such.  And related, the SEAWAVE-QEX requires three years of data.  But as there was some discussion yesterday, sometimes years can be very different from one another, and it might be that you have one particularly anomalous year.  
                  So I would suggest at least having one extra year just to account for that sort of thing.  And because it wasn't really mentioned, although it was mentioned earlier in the comments, of course you want to do it at a site that has your flow measurement.  
                  DR. ROBERT CHAPIN:  Thank you.  And now we'll return to Dr. Nowell as she switches hats. 
                  DR. LISA NOWELL:  Not sure which hat I was wearing.  Was that water quality I was addressing first, supposedly? 
                  DR. ROBERT CHAPIN:  Ostensibly, and now it's environmental risk assessor. 
                  DR. LISA NOWELL:  I can't really separate them.  Sorry. 
                  DR. ROBERT CHAPIN:  Okay.  That's fine. 
                  DR. LISA NOWELL:  I did think of one other thing to say, though, that a lot of times we're really particularly interested in those compounds that are intermittent, sporadically used, not often high detection frequencies, or at least not necessarily high detection frequencies, but biologically very toxic both to humans and aquatic life.  So we're particularly interested in tracking those.  And it does seem, although you did a good job -- the model did a good job with chlorpyrifos, which would be in that category I would say, and carbaryl.  It does raise an issue that a lot of the compounds of interest we would most want to predict the at maximum concentrations might be the ones we might have the most difficulty fitting the model.  I don't have a solution for that. 
                  DR. ROBERT CHAPIN:  Okay.  Cliff? 
                  DR. CLIFFORD WEISEL:  I also thought this is a good section.  I also, as I mentioned the other day, I liked what was in that table because it's good guidance for some of the thoughts.  And I appreciated what was done from the group in Washington.  But when I read this, one of the things it's saying is can you currently use the SEAWAVE to help optimize any number of monitoring programs, is what I'm reading this charge question.  
                  If I'm interpreting it correctly, we have to remember what we said was the limitations we currently have with this system.  Limitations are we have it only for limited compounds, which was mentioned.  So if you're going to new compounds, I don't know that you can do that yet until you can really demonstrate that the model and the parameters that you set are relative to that compounds.  The same thing if you're going to other watersheds that have different systems.  We heard climate differences that may have that.  Again, you can't use what you currently have to predict those until you actually have enough data to say that that works in those.  And that's limitations.  
                  So the action is you need to define what you currently have and what you conditions you can currently do it under.  Take a look where you're meeting that.  And if you're not, what is it that you can do to do it?  And maybe you can do more water ranges and therefore give not exactly what the people do in a water monitoring zone but say, "Okay.  We think you're in this range.  Maybe double it just to get us enough data to see are things going to work."  So you'd get that.  
                  In a system that's much friendlier to what you have, I think you then can go ahead and do a lot better and give them finer systems.  And I certainly agree with the comment that this is a current evaluation -- like any time I use modeling that gives me measurements, I say, "Okay.  That's my minimum."  That's not what I do.  I do plus a little extra just to make sure because modeling and measurements go from one to the other and build upon each other.  So keep that in mind. 
                  DR. VERONICA BERROCAL:  Can I add something?  This is Dr. Berrocal, and I wanted to breakdown your comments, Dr. Weisel.  So actually, you raise a good point in terms of we have discussed in the previous hours about the fact that, when SEAWAVE was used on some sample data, the performance, at least for the long-term concentration was not what we were hoping.  And there were also some limitation in terms of the short-term exposure.  So I think that all those comments that we made at that point could actually be used to design a monitoring sampling scheme for assessment so that, when you guys would be implementing SEAWAVE-QEX on this monitoring data, you actually will get the similar performance that Dr. Vecchia got in his paper. 
                  DR. KENNETH PORTIER:  I'm kind of responding to our chair saying we should be thinking about what EPA might be able to do in the intermediate- or long-term.  Somebody said more samples are good.  Well, while more samples are good, a design concern is what's the value of a new sample?  So the white paper reports on a small study that looked at increasing sample size and concluded that diminishing returns are greater than ten samples per year.  
                  In the intermediate term, EPA could perform -- what happened here?  EPA could perform a simulation add-on to the sparse data study to estimate the value of getting on additional fill-in data point, say mid-period add-in, added to the middle of the season to answer the question does this added $800 point -- does this added data point substantially reduce the width of the 80 percent confidence interval?  Does it significantly decrease the percent underestimation in situations where underestimation is an issue, for example, in the 365-day long-term maximum?  
                  I think a little study like this adds a way of arguing with people who do monitoring, like state what's the value of having you go out in the middle of the season and get one more sample?  And you can come back and say, "Wow, a confidence interval is reduced 30 percent with that one sample."  To me, that's a good return on investing that effort of improving, not asking for five-day sampling for the whole year.  You might be able to get one more sample if you can convince them of that.  And you need this simulation study to do that. 
                  DR. ROBERT CHAPIN:  All right.  Other comments about the use of -- yup, Tom. 
                  DR. THOMAS POTTER:  Dr. Portier got the wheels rolling in my brain here, so thank you, Ken.  Hopefully it's going to be positive -- the outcome anyway.  But this motivates what Dr. Portier was saying about value of sampling.  I think always we have to ask the question what about extreme hydrologic events and whether or not we need to bear to invest the time, energy, and treasure necessary to capture them.  They don't necessarily fit well into the SEAWAVE-QEX framework.  But certainly, I think they are potentially, again, depending upon the time of year that they happen, very high value samples.  
                  DR. ROBERT CHAPIN:  All right.  Other comments on the use of --? 
                  DR. REBECCA KLAPER:  From what I'm reading and remembering from the document, it seems that --
                  DR. ROBERT CHAPIN:  Make sure you're close enough to the --
                  DR. REBECCA KLAPER:  The monitoring that you're talking about here would be monitoring after you decide that the SEAWAVE model actually says that there's a concern.  So then you can go back and get other samples where maybe the model is giving you questionable results.  It's not like you're talking about designing a whole new monitoring program based on the fact that you have now this model that you want to use for lots of different pesticides.  Is that correct? 
                  MR. DANA SPATZ:  I think we're talking about both. 
                  DR. REBECCA KLAPER:  Okay.  Then I'll just say that it seems most valuable for the first one of those where you found something where you think it might be potentially harmful.  You've gone all the way to this part of the process, and then you say, "Well, we don't really have enough samples in these particular kinds of watersheds, et cetera," but not necessarily for developing a wholescale watershed monitoring program for everything. 
                  DR. CLIFFORD WEISEL:  I think the utilization for the second actually is potentially very strong.  I don't think you're there yet.  If you had something that allowed you, I think that would be a big bonus because then that saves a lot of time, money, and measurement, which is what we're always trying to do with it, too.  It's probably what I'd say mid- to long-term; whereas, certainly what you currently have is the short-term for the first one. 
                  DR. ROBERT CHAPIN:  All right. 
                  DR. VERONICA BERROCAL:  So I think actually I read this question as, given that the goal long-term would be to try to use SEAWAVE-QEX to characterize short-term and long-term exposure based on the fact that we do not have access to daily pesticide concentration data, what would be the best monitoring design that we could have that would allow us to use SEAWAVE-QEX and get these estimates of short-term and long-term concentration that we actually can predict?  Right now, it seems what they have shown in the white paper with the subsampling every seven days, every 14 days is that using SEAWAVE-QEX when you only have samples every seven days or every 14 days, you were not able to actually characterize that long-term concentration well.  And with the short-term, it's hit and miss depending on where you are and which pesticide you're looking at.  
                  So I think the question here is how can we incorporate what we have learned from the actual study that we have done on SEAWAVE-QEX to design a better monitoring program?  So I think actually what Dr. Ken -- can I call you just Ken?  What Ken said is that you could actually do an evaluation, as you've done in the white paper, where you change the sampling frequency, instead of seven days every three days, see what happens when you implement SEAWAVE-QEX to characterize long-term and short-term concentrations, see if it improves over what you have with the seven day and 14 day sampling frequency, and then use that information to maybe push for a different monitoring design. 
                  DR. ROBERT CHAPIN:  Better the sound of loon than crickets I suppose.  Let's see.  So other comments about this question from the panel?  All right.  Hearing none, I'll turn to our friends at the EPA and say do you have any clarifying questions for us?  Did we answer this question for you clearly enough? 
                  MR. DANA SPATZ:  Thank you. 
                  DR. ROBERT CHAPIN:  I'll take that as a yes.  Okay.  My agenda says lunch.  Let's go to -- sorry.  1(f), forward on the --
                  DR. ROCHELLE BOHATY:  I promise I wasn't ignoring you on the last one.  I was reading over my notes to make sure we had covered everything. 
                  DR. ROBERT CHAPIN:  We want to capture everything we can at the right time, so take your time.  It's all good. 
CHARGE QUESTION 1(f)
                  DR. ROCHELLE BOHATY:  Okay.  So I(f), during EPA's evaluation of SEAWAVE-QEX, we used the following criteria in running the model: 3 years of data, 12 samples per year, and greater than 30 percent of the samples were detections.  However, Vecchia (2018) notes that there is flexibility around the data requirements for input into SEAWAVE-QEX, provided that the diagnostic plots indicate that the model assumptions are fulfilled.  Please comment on any data characteristics, such as sample frequency and timing within and across years, that should be considered when exercising flexibility in the data requirements. 
                  DR. ROBERT CHAPIN: Data flexibility, do we think we can allow them, all right, and how do they know?  Dr. Berrocal? 
                  DR. VERONICA BERROCAL:  Yes.  Simulation results by Dr. Vecchia indicated that, if the modeling assumptions underlying SEAWAVE-QEX are satisfied, then bias of 80 percent of bounds are smaller when centering rate is 30 percent compared to 70 percent.  Additionally, greater sampling frequency is also associated with the smaller bias and smaller 80 percent outer bounds, particularly in relationship to the correlation in the data.  In other words, if the pesticide concentration data as residual temporal correlation, once the effect of stream flow or precipitation, since anatomy and fluctuation of the streamflow both at the monthly time scale and at the daily time scale, then it is important that the pesticide concentration is sampled frequently enough to capture these residual temporal correlation.  Instead, all the other correlation in the water-pesticide concentration data is already explained by these factor, seasonality, streamflow, and anomalies, then frequent samples are not necessary.  
                  Even though the SEAWAVE-QEX model does incorporate seasonal correlated errors, it also meets the possibility of a very small, if not mild, concentration temporal scale.  Subject matter, hydrological, toxicology, et cetera, knowledge may be useful in understanding whether it is likely that, for a given site, pesticide concentration data will still be correlated even after having already accounted for seasonality, streamflow, and medium to short range streamflow anomalies.  That informing a user or assessor whether pesticide data with a higher sampling frequency is needed to reliably generate daily chemograph.   So I am not one of these people.  I don't have that expertise, so I just think that you should talk with somebody that has that expertise.  
                  To modeling assumption of which SEAWAVE-QEX model relies on, the assumption anomaly distributed residuals, and there is no measurement error.  If the data have been collected over several years and all the less reliable instruments were used in the first part of the data series, it might be preferable that older years were not analyzed jointly with more recent pesticide concentration data.  Histograms of log daily pesticide concentration data potentially even stratified by season, since the SEAWAVE-QEX model assumes different level of variability in the residuals and in the data by season, should be generated prior to feeding a SEAWAVE-QEX model to determine whether the normative assumption is appropriate for the data.  If the assumption of normality or log daily pesticide concentration data is not stratified, a SEAWAVE-QEX model might not be appropriate for the data.  Particularly, the SEAWAVE-QEX derived chemograph might fail to capture the highly daily pesticide concentration if the histogram in the log concentration data appears to be very skewed.  
                  And then I had another note about how in the white paper it was shown that it is possible the pesticide concentration data over multiple years might lead to plus two WAVE model feat during and that's supported by the data.  Hence, when pesticide concentration data is measured over a few number of seasons over multiple years, the SEAWAVE-QEX model might be only applied to partial records provided via the recommendation in terms of data frequency until the size are satisfied. 
                  DR. KENNETH PORTIER:  My comments fall into two categories.  The first category is just the suggestion on how to improve the diagnostic plots to determine adequacy of fit.  And then I have a few more suggestions that basically are around the theme what can go wrong, even if the minimum dataset conditions are met.  So on the first one, SEAWAVE-QEX assumes that the normalized residuals are approximately normally distributed with mean zero and variance equal to one.  The SOP suggests examining plots of the normalized residuals with time of year to assess adequacy of this assumption.  With these plots, the advice is to look for obvious seasonality remaining in either the mean trend or changes in spread over time.  
                  I also suggest that this plot -- it also suggests that this plot will show obvious non-normality, so for example skewedness, outliers, and so forth.  A better diagnostic for these latter conditions is to examine non-normality through a normal quantile plot.  And I'll provide a reference to that.  Normal quantal quantile plots would show clearly the percent of censoring in the data and how much of the censored residuals have been imputed and how well the imputation is doing.  
                  And on the recommended plot of normalized residuals over time, I'd suggest maybe adding a low S smooth curve to give the viewer an indication of what kind of trend are in the residuals over time.  So it'd be a lot clear to see any trends, which shouldn't be there, but a smooth would at least give you an indication of what's happening.  And there's a capability to do a similar kind of plot on variability, so you can kind of do a smooth variance plot at the bottom that would show how much the variance may be changing over time.  And those are minor additions that are easily add in all.  
                  So what can go wrong, even if the minimum dataset conditions are met?  So under the minimum data requirements, one would expect to see at least 36 monitoring data points with at least 11 data points not below detection.  So what happens if one of the three years has no uncensored values?  So all the non-detects occur in a year.  Well, now you only really have two years' worth of data.  You may have 36 data points, but one whole year is useless.  So if this happens in the first or the last year, the year can be dropped from consideration.  But at that point, the data falls below minimum data requirements.  
                  But what happens if this is in the middle year?  So you've got data at the beginning and data at the end and nothing in the middle.  And if you think that's weird, you can go look.  There are some datasets in there that have whole years' worth of no detects in the middle of a period of three to five to seven years.  And the model happily goes through and pitstop.  
                  My belief is that this likely also indicates a site with data below minimums.  And note that to ignore this situation and fit SEAWAVE-QEX model is to later find residual by time plots showing heterogeneity of residuals over time.  So it's going to show up in the diagnostics.  This suggests that the minimum 30 percent detection rate kind of should apply to every year being modeled, not just the whole kind of period of record.  And some of this is discussed in Section 6.2.1 of the SOP.  So there is some recognition within EPA that this can happen, and it should be looked at carefully.  
                  Notice that the SEAWAVE-QEX SOP suggests a more stringent minimal dataset.  In Section 6.1 of the SOP, "Providing Advice on Handling Poorly Fitted Data," one finds the following: "If more than three years of data are being analyzed, do at least the first and/or last year of data individually fulfill the SEAWAVE-QEX requirements of minimum samples and detection rates?  If the first and last year of data being analyzed does not fulfill the SEAWAVE-QEX requirements, then remove any non-compliant first or last years from the analysis.  The first and last year of the datasets are anchor years for the model, so it's important for them to meet the minimum requirements, if not having higher detection frequencies, while maintaining a full dataset that meets minimal requirement."  I like that.  I fully think that's a great recommendation.  
                  Another issue not addressed in the minimal dataset conditions is that detections should offer some indication of seasonal pattern.  In Section 6.1 of the SOP, the question is asked "Do the chemical detections appear to be random with little or no seasonal pattern?"  Clearly where the detections are found along the timeline is important to the success of the model timing.  So two non-detects or detects, three non-detects and detect, two non-detects is not going to be as useful as three non-detects, four detects, and four non-detects in a 12 dataset.  So the timing is something that needs to be looked at.  I don't know if any of the diagnostics really help you look at that, except maybe the first one, the summary of the fit.  
                  The white paper and the SOP both mention alternating years of analysis and the benefit of using in season monitoring data for sites.  But it's not clear how a partial sampling season is to be defined.  The SOP mentions a partial sampling season as one having less than or equal to six months and suggests using best judgement in the designation of the season.  It would seem important to the selection of the best seasonal wave form for there to be non-detects at the beginning and at the end of the sampling season.  But how many should there be on either side I think needs to be determined.  
                  With 12 data points per season and only three to four detects, say, located in the middle of the season, that leaves about four non-detects at the beginning and four at the end, if it's balanced.  Well, what happens if you only have one non-detect at the beginning and all the rest of the non-detects are at the end or the opposite?  I think that may have an impact on how the models fit.  But in my mind, I don't see how that's going to work.  I think you're going to have to play around with that to see what that means.  
                  The point is that it seems clear that the spread of monitoring data across the use season within a year, and across all years, and the location of detects within and among years is critical to obtaining acceptable results from SEAWAVE-QEX model.  And I think this comment also ties into the last comment I made about the extra datapoint.  Sometimes putting one more detect in the middle of the season is going to make a big difference in how well it estimates the maximum or tightens up on the confidence interval.  That's my comment. 
                  DR. ROBERT CHAPIN:  I'm going to break with tradition and ask do you want to ask any questions about that?  I'm not sure I followed it all. 
                  DR. KENNETH PORTIER:  I'm looking at Dr. Hartless.  She's my anchor.  And when she's kind of like doing this, I'm like okay.  I've got it.  If she does this, I have to try over. 
                  DR. ROBERT CHAPIN:  If Dr. Hartless is happy, I'm happy.  All right.  Dr. Miglino? 
                  DR. ROCHELLE BOHATY:  Just one clarifying question.  I thought I heard in there that you thought that we should have these minimum requirements for each year of data.  But then you read the quote from the paper that said just in anchor years, so some clarification there would be helpful. 
                  DR. KENNETH PORTIER:  That quote was from your SOP, and kind of the SOP is giving some advice, and I think it's very good advice, to look at your data and say, if you don't have an adequate year at the beginning and a year at the end that anchors your fit, you're going to have problems.  You don't even have to look too hard.  You're going to have to adjust your period of record, maybe eliminating that first year if it only has one or two detects and they're at the end of the season.  It's not going to add to the model fit, and it may actually detract from the model fit.  But that's not my rec.  I was just quoting from your SOP that that works.  
                  I like the idea of having it every year, but I think it may be unreasonable to hold to that tightly.  That's maybe a stretch goal.  I don't think it's a requirement.  Because I think the model works, even when you have a little bit of data in the middle.  I get a little nervous when there's a whole year of no-detects in the middle of five years, and you wonder what went wrong and is that real data or are those really indication of something.  And I don't know what happens if you pull that year out and just stick the years back together and assume that year never happened.  2009, nobody wants to think about that.  
                  DR. ROCHELLE BOHATY:  Thank you.  That was helpful. 
                  DR. ROBERT CHAPIN:  All right.  Now. 
                  DR. ANDREW MIGLINO:  So I'm going to take this from the perspective of a modeler who's happy that he has two data points to work with, and clearly we have a lot more here.  So I think you've done a very good job in the white paper of outlining minimum data requirements.  Those are more stringent then what's in Vecchia 2018, which just says six points with 30 percent detect.  So again, I think much better perspective.  There's one quote in Vecchia, though, that I think should be kept in mind.  Let me make sure I have it.  "Model verification can be especially difficult when a large percentage of data, more than 50 percent, are censored," which is clearly a higher bar.  
                  I'm just thinking of myself fitting this model, looking at those seasonal wave patterns and seeing 50 percent of the data as non-detects with made up concentrations that I'm now saying, "Oh, they fit in there."  And of course they do because they're supposed to fit in there when I've made up the data to fit.  So from a modeling perspective, I'm not saying we need to go up to 50 percent.  I'm just saying there needs to be some kind of bright line in this SOP that says, "Here's what you do when you have 30 percent detect.  Here's how you handle this.  Here's the important features to look at in these more extreme cases where data is low."  
                  I think I had one other thing in here.  The other thing that I had been thinking of -- and I don't know if it's possible.  I don't think it is.  I suspect it isn't -- is everything we've been talking about is tied to calendar years.  And if we can adjust those to maybe incorporate the very beginning of a season through  a year from that season, you may be able to include more data.  I don't know.  I haven't looked at the data that deeply.  But you may be able to kind of throw out the first four months of the year that had nothing, and then suddenly start picking up data rich years in future years.  And that may be an opportunity.  Again, I don't know if the way that the model is coded and the way that it's developed ties it to a true calendar year.  But water years, farm years, whatever you want to call it may be more fruitful for larger data sets.  I think that's all I had. 
                  DR. ROBERT CHAPIN:  Mr. Councell. 
                  MR. TERRY COUNCELL:  I really don't have anything to add to the discussion other than to reiterate what Dr. Nowell said earlier -- that for many of the compounds, finding 30 percent detects that are going to make it to that tier four assessment, you might find trouble finding that many detects in there.  That's just part of the monitoring and how frequently these pesticides of concern show up.  Other than that, everybody else -- I agree with their comments. 
                  DR. ROBERT CHAPIN:  Great.  Dr. Yang? 
                  DR. RAYMOND YANG:  My comments are for mid- to long-term for your benefit.  May we have Dr. Ken Portier's first slide this morning with the model?  I need that because I need to explain what are my thoughts. 
                  DR. ROBERT CHAPIN:  The answer is yeah, in a minute.  Sure, go ahead.  Ray, we're going to get a comment from Cliff. 
                  DR. CLIFFORD WEISEL:  Two things, one is the SOP I thought was very, very good.  I found it very helpful.  It helped me understand really what's going on.  That said that SOP was put together with probably very clear set of examples that you could use that SOP.  I've heard several times that the most difficult thing about using this is interpreting those plots, diagnostic plot.  So my advice is, if you're going to do this, set up a training program for who you think might be using it because my feeling is, once this gets out there, there will be people that don't necessarily have the background that you do that are going to attempt it.  And it could end up getting into a point that it's not being used properly, and that will bring the program down a step. 
                  DR. RAYMOND YANG:  Ready for me?  My explanation also will tell you why I'm so obsessed with model, that you have to have the model.  These are the models.  What I will do, I will ask you to look at the right panel, that particular equation, but also consult your left panel, the explanation of the terms.  I'm going to talk this way, looking at -- Tim gave me three pages of notes and so on.  
                  So this is the SEAWAVE-QEX model.  If you look at the model, look at the left-hand side.  Forget about log.  For my purpose, don't worry about log.  C as a function of T, that is the concentration of pesticide of interest.  What is the concentration?  
                  Now, that concentration is dependent on the righthand side.  Now, let's look at the righthand side.  B0, B1, and so on and so forth, all this beta, forget about that because, at the risk of offending statisticians here, these are all flux factors.  That's the reason why component statistician or mathematician could fit any curve.  So forget about all the beta.  Forget about first term and last term.  Last term is the arrow.  The arrow is dependent on all the rest of the terms.  
                  So now you have four terms.  So w as a function of t, that is seasonal wave.  And what is A sub mt?  A sub mt is dependent on MTFA, which is mid-term anomaly.  So it's a variable.  And AST is short-term anomaly.  So these are the variables.  And T minus tM is also a variable because what are they?  t is the time reported --
                  DR. ROBERT CHAPIN:  Ray, excuse me.  Can you get to your suggestion? 
                  DR. RAYMOND YANG:  Yeah.  I have to do this explanation.  Otherwise, you don't know what I'm doing.  T is the time recorded in decimal years.  And Tn is the midpoint of the time interval being analyzed.  So these four terms will determine your concentration.  It's very important, okay?  
                  And when I look at this, because all of these things are variables, therefore, over the years, you're going to have a probability distribution.  And therefore, this morning I mentioned that you need to consider Bayesian approach and multichain Monte Carlo because Bayesian approach has to take care of all of your available data, including some of the data right now you rejected.  And there was one public comment -- written comment, Crop Life America specifically concerned about large unused data.  These unused data could be used in the Bayesian approach as posterior distribution formulation.  
                  So what you know right now is prior.  When you have new information added, you formulate a posterior distribution for your parameters.  Now, where does multichain Monte Carlo come in?  It assesses all the probability distribution, taking into consideration variability and uncertainty at the same time.  And it also takes into consideration the covariate of interactions among parameters.  If you use those two in combination, then you don't have to worry about I only have three years of data -- I only have 12 samples in a year and so on because you can scientifically, validly, using Monte Carlo simulation to sample from the probability distribution to reach 1,000 sets of data if you want to, a million set.  So this is very important for you to consider.  Thank you. 
                  DR. ROBERT CHAPIN:  Thank you, Dr. Yang.   Other comments on -- go for it. 
                  DR. VERONICA BERROCAL:  So I want to respond to Dr. Yang.  I am actually a Bayesian statistician, and I do appreciate you sticking out for Bayesian statistics, typically doesn't happen in more -- I don't know -- outside of academia.  Typically, it's hard to convince people of using Bayesian methods.  I do agree with you that there is a lot of uncertainty in some of the data sources that go into that regression equation.  But I do want to acknowledge that EPA has been doing some sort of Monte Carlo simulations when they generated those concentration -- those chemograph.  So the ideas are there.  Maybe they're not doing a fully Bayesian modeling approach.  But they are simulating daily chemograph by changing the values of the residual ones.  So you could increase the uncertainty by also simulating the parameters from the posterior distribution, which would be accounting for all sorts of uncertainty.  But I don't know if this is going to be too much of a big ask, given that we already mentioned that maybe SEAWAVE-QEX is already hard to implement as it is for people that are not experts with the model.  So in principle, I agree with you.  I just don't know whether it's feasible in practice. 
                  DR. RAYMOND YANG:  Well, we use this particular model as a basis for this discussion.  This has been going on since 2007.  At the beginning, it's always difficult.  I'll tell you right now.  If you get used to what I'm saying in terms of applying Bayesian approach and using multichain Monte Carlo, you're going to get used to it, just like you get used to a SEAWAVE-QEX. 
                  DR. ROBERT CHAPIN:  Okay.  All right.  Anyone -- any other comments from the panel? 
                  DR. THOMAS POTTER:  I just wanted to come back to the idea -- or bring forth the idea that actual use patterns of active ingredients can inform a lot in terms of what your data looks like and what you're phased with, whether you have sporadic use or there's consistent use from year to year.  And I think from a risk assessment perspective, you'll always want to have that in mind as you go in and start exploring the data.  And that's going to guide, I think, a lot of your thinking. 
                  DR. ROBERT CHAPIN:  Comments from any other panel members?  Okay.  And we'll turn to our EPA colleagues and ask if you have clarifying questions for us. 
                  DR. ROCHELLE BOHATY:  We're good.  Thank you. 
                  DR. ROBERT CHAPIN:  Life is good.  All right.  So it's 2:15.  What I'd like to do is we'll keep working up until 2:30, and then, wherever we are in responding to the next charge question, we'll stop and take a ten-minute break and then come back.  Okay.  So question two, 2(a).
                  
CHARGE QUESTION 2
                  
                  DR. ROCHELLE BOHATY:  Okay.  Question two, while SEAWAVE-QEX provides a way to estimate daily pesticide concentrations from non-daily surface water monitoring data, for many pesticides there are not sufficient monitoring data to use SEAWAVE-QEX.  This is because the data are too highly censored or there are not enough samples per year or across years.  Therefore, EPA would like to have an alternative approach to estimate drinking water concentrations from non-daily pesticide surface water monitoring data.  
                  Previous Scientific Advisory Panels supported the development of sampling bias factors to derive bounds on pesticide concentrations in surface water.  EPA used SEAWAVE-QEX to develop daily pesticide chemographs from infrequent surface water monitoring data.  These chemographs were used to generate short-term sampling bias factors for acute exposure durations of concern, utilizing the methods supported by past Scientific Advisory Panels for sites with varying attributes across the contiguous United States.  
                  Building upon that previous work, EPA evaluated different sampling strategies and imputation techniques to develop short-term sampling bias factors and concluded that using a random sampling strategy with a log-linear imputation is suitable for deriving short-term sampling bias factors.  In addition, EPA developed a new method for developing long-term sampling bias factors for chronic and cancer exposure durations of concern.  EPA concluded that for pesticides with chronic and/or cancer endpoints, that as few as four samples per year can be used to estimate a 365-day average concentration for use in drinking water assessments.  
                  
CHARGE QUESTION 2 (a)
                  
                  2(a), please comment on the use of SEAWAVE-QEX estimated daily pesticide concentrations, including the underlying data requirements, to expand the data available to derive pesticide-specific sampling bias factors.  Please comment on EPA's optimization of the short-term sampling bias factors, derived using a random sampling strategy with log-linear imputation, to estimate the range of potential concentrations not measured between sampling events.  Please also comment on the long-term sampling bias factor approach.  
                  DR. ROBERT CHAPIN:  Now both short-term and long-term comments focus here.  Dr. Berrocal?
                  DR. VERONICA BERROCAL:  Yes.  So I'm going to anticipate that this is going to be a long response.  I will not talk so much on 2(b), but you have to listen to my voice for this one.  So first off, I would like to state that EPA's decision to derive sampling bias factor for both short-term and long-term is responsive to recommendations made in previous SAP meetings, which invited EPA to make more use of migrant data rather than model data for drinking water assessment.  So in light of that, I support EPA's effort in developing short- and long-term bias sampling factor.  
                  But both short- and long-term bias sampling factor are derived by computing a ratio where the numerator is a summary statistics of the daily pesticide concentration data representing a short-term or a long-term pesticide concentration obtained by using the SEAWAVE-QEX model.  And the denominator is the fifth percentile of a summary statistics obtained using a different declaration function, not the one in the SEAWAVE-QEX model, which is used to fill in missing data in a subsample series.  Given definition of the sampling bias factors, there are many components that affect the magnitude and the distribution of the sampling bias factor.  
                  Looking to each of them, first, I would like to note that, from the white paper, it was not clear how the bootstrap data was sampled using the varying sampling strategy.  The white paper seemed to indicate a procedure that would include first sampling the first date for the daily concentration data and then perform a sequential sampling of all the other data with constraints to fall within a certain sampling interval, either seven, 14, 28 days and so forth.  However, yesterday's presentation seemed to indicate that instead the sampling procedure used was more of a systematic sample where just the first day was selected at random from a set of possible dates.  And then all the other dates were derived systematically every seven, 14, 21 days and so forth.  So this was not clear to me.  I am still very confused of what strategy did you use.  
                  Although it might seem very pedantic to the extent where sampling procedure was used, the devil is in the details here.  And the type of sampling scheme used in conjunction with the form of interpolation function used to in fill the bootstrap data will influence the magnitude of the summary statistics that go in the denominator for the sampling bias factor calculation.  Specifically, my main concern initially regarded mostly the fact that the bootstrap data was interpolated with a function that was either a step or a lean or a log leaner that did not allow for the imputed daily pesticide concentration to be higher than any of the sample data.  
                  Again, even though this might seem quite irrelevant, it does matter since, with the current choices, the daily interpolated pesticide concentration would lead to summary statistics that, one, will always be smaller than the maximum volume average computed just using the seven day and the 14 day bootstrap data in the case where the sampling frequency is -- in the case of a 20 day volume average.  And two, it will always be smaller or equal to the summary statistic derived using the bootstrap data if the step-wise function is used to infill the missing data.  So the denominator, the choice of sampling, the choice of interpolation function that you're using is going to affect the magnitude of the summary statistic that goes in the denominator of the sampling bias factor.  
                  The repetition of the bootstrapping procedure multiple times does allow for variability in the summary statistics that gets generated from the interpolated data.  However, as a result of the method used for imputation, I believe that the short-term sampling bias factor might tend to be larger more over than it would be if other interpolation functions were used, in particular, interpolation functions that would allow for the imputed data to be larger than what is sampled.  In light of that, the tendency of short, small sampling bias factor to have smaller, medium, and more concentrated distribution as the average in time increases or as the sampling frequency decreases is expected to me.  Similarly, it is expected that the histogram of the short sampling bias factor did show an extremely wide skewed distribution.  
                  Since the sampling bias factors are used to derive an upper level prediction limit, the fact that the sampling bias factor might tend to be larger than what they should actually be -- it would just translate in higher upper level prediction limits, which might not be an issue as we will be averaging on the protective side.  The fact that there was no substantial difference between a random sampling strategy versus a stratified sampling strategy is no surprise to me mostly because of a point I already raise, which is that the imputed, interpolated data will never be larger than the bootstrap data using the chosen interpolating functions.  Additionally, given that the stratified sample strategy is a minimal of every seven or every14 days, the sample pesticide concentration varies would very likely be independent.  The fact that the log leaning interpolation works better than the linear function or stabilized function is also surprising, given that the log linear function seems to be closer to what is expected behavior of daily pesticide concentration, at least according to the SEAWAVE-QEX model.  Okay.  We're getting close to the end.  
                  For the long-term sampling bias factor approach, no biometric form was used to infill the bootstrap data.  I could not find information in the white paper about the distribution of the long-term sampling bias factor.  My only concern is, with the use of the formula for the sum of error of the mean used to compute the average -- for the sum of error of the mean and raw pesticide concentration.  While it is true that the sample data is very likely to be independent since you're sampling four, six, eight, or ten points at maximum, the true daily pesticide concentration data is correlated.  This means that the standard error used to compute the upper limit of the 90 percent prediction interval for the mean is likely too small, and it does not capture the sampling variability in the estimated annual mean pesticide concentration.  As a result, this means that the upper limit of the 90 percent prediction intervals are smaller than they should be, making the long-term sampling bias factor larger than they should be.  
                  Again, as for the short-term sampling bias factor, larger long-term sampling bias factor will add on the protective side.  That means there will be a higher titer error or higher false positive, as it was mentioned yesterday during a public comment.  However, personally, I don't find this to be an issue given that the sampling bias factors are usually mostly for screening and moving from Tier 3 to Tier 4.  
                  Finally, as I mentioned, the sampling bias factor is made of denominator and numerator.  For the use of SEAWAVE-QEX for the numerator in the sampling bias factor, both in the short-term and long-term, I do believe that, given that the SEAWAVE-QEX performs quite well in characterizing the distribution of both short-term and long-term pesticide exposure when the data is quite dense and it satisfies the criteria, at least in the Vecchia paper, I don't see any problem in using SEAWAVE-QEX to compute the numerator term in the bias sampling factor. 
                  DR. KENNETH PORTIER:  I wish I had this better written.  Now, you're going to see where I start to fall apart here because these are kind of a bunch of thoughts, and I'm not sure I have them all properly organized.  So on the optimization of the short-term SBS, Section 4.4. describes a simulation study which examined the impact of four factors: exposure duration, sampling frequency, sampling strategy, and interpolation methods.  On the ability of SEAWAVE-QEX chemographs, that is time series of daily chemical concentrations to produce short-term one day, four-day, 21 day rolling average maxima SBFs with acceptable properties.  
                  So just right off the bat, there's a lot going on in this simulation.  You've got four factors.  Plus you've got three rolling actions.  This is why it kind of gets complicated to make a coherent statement on stuff.  
                  So acceptable is operationalized by looking for combinations of these factors that minimized the fiftieth percentile and ninetieth percentile of the root mean square errors from the simulations.  So a general linear mixed model assuming log normal error was used to estimate these data across sites and years and in particular to identify the best interpolation method and sampling methodology to be used in further development of SBFs.  And this is all in the supplemental file, USGS sampling bias factor evaluation, short-term SBF analysis.  
                  While it's not clear that a log normal distribution is appropriate for the distribution of the median RMSE, I wasn't quite sure that that's the right distribution.  It's probably acceptable for the ninetieth percentile of the RMSE.  One's an extreme value, and one's a central location parameter.  The central location parameter is more likely to have a normal distribution or something like a normal or T distribution; whereas, an upper extreme, like a ninetieth percentile, is likely to have a skewed log normal distribution.  
                  So instead of -- in addition, instead of separating out year and site by year effects, it might have been better to think in terms of a year within site covariance term.  And I'm going to talk about why this is.  And I made a note to myself.  Boy, if I had time, I would go into the CSV file that they gave us and run some QQ plots to see if the distributions did look lognormal.  It's one of those things that you just run out of time to do.  
                  So what should the analysis model be for this simulation experiment?  With 15 realizations per condition for every effect -- for this condition, of course every effect will be significant.  This is an experiment with a lot of residual sample size.  So statistically testing for effects, everything's going to be P of 0.0001.  So you don't kind of want to be looking at statistical tech -- you want to be looking at the size of the estimates of the effect and see where there's big effects versus small effects.  
                  There are 108 sites, three sampling protocols, three interpolation methods, and two sampling intervals.  Analysis was done by sampling interval.  So you only have 108 sites, three sampling protocols, and three interpolation methods.  By the way, another aside, the SASS code has no comment, nada.  This is a no-no.  You've got to at least add comments if you want us to look at this stuff.  
                  So the residual is the variation among realizations within site year by sampling by interpolation methods for each sampling interval.  One can think of year within site as a random effect, and site as a random effect.  So the random statement for site -- I would have thought the random statement for site should have an identity structure.  But the random statement for year within site have something like a correlation structure, like an AR1 or a CS structure.  
                  So your analysis model is a little too simple for the experiment.  The model assumes random variation among realizations within site year is constant across the site.  But you don't show any statistics to convince me that that's the right assumption to make.  So I think having a slightly more flexible correlation structure would have helped.  And doing the analysis this way give you some idea of how important is site to site variability and how important is year within site variability and how variability is year within site variability is it for these situations, like interpolation.  
                  So the white paper concludes on page 66 the results suggested the SBFs developed with estimated daily concentrations from SEAWAVE-QEX compared well to SBFs developed from measure daily data.  In addition, the analysis lead to the decision to use the log linear interpolation method to develop site specific SBFs.  Both random sampling and stratified sampling methodologies could be used because there was, in a sense, very little effect difference between those two things.  
                  Another aside, there's an error in the short-term SBF, SBFA sites Chem1.CSV file.  You get halfway down, and it's summary data instead of the individual data.  I was looking at that file.  So when one scrolls down the 14-day sample interval, one sees the summary statistics from SBFA Summary Chem1.CSV instead of the by year summaries seen in the seven-day sample interval.  So you gave us good data to look at variability in the seven-day sampling interval, but somebody copied the wrong data to the 14-day interval.  So I can't comment on that one.  
                  I'm sorry this is very technical.  But the white paper also included on page 66 "In order to summarize the SBFs for each site, EPA employed a two-step summarization process that calculated the median SBF across chemographs for each year and then calculated the median SBFs across years."  One can see the need for this two-step process by viewing the wide range of estimates produced across the hundred simulated samples within each year.  And that's where you need to go to that summary table that has problems with the 14 days.  When you look through it, you see, oh, yeah.  You need to look at medians by medians.  
                  In that supplemental file, you see that for most of this two-stage estimate demonstrates little year to year variability in medians.  But for some sites, for example CSIMS0242354750 -- and I'll put that in the report -- there still remains much year to year variability with a range of 100-fold for the one-day SBFs, fivefold for the P50 -- I'm sorry, the SBF P50, and 5X for the 90-day SBF P50.  So there's still sites that have a lot of variability in here.  
                  The approach was also applied to static systems in the monitoring program.  The white paper indicated that each static fit produced issues, for example, broad wave systems, positive and negative correlations with MTFA, empirical correlograms lower than fitted correlogram functions, et cetera, that were captured during examination of the diagnostics plot.  So this experiment didn't work for the static systems as well.  
                  So each of these sites have a lot of observations where the high fraction that detects for most of these, the estimated max is very close to the observed max, as should happen.  I think Dr. Berrocal mentioned that.  It's also clear from the diagnostic plots provided in the supplemental files that fitting and appropriate seasonal wave model is at the heart of the estimation issues.  And part of the difficulty is that these static systems have active management and results that active management -- and the results show that active management is evident in the diagnostics plots.  
                  So I concluded, looking at this, that the study and the associated analysis looks to be properly done, and conclusions are reasonable, given the results of the analysis.  And it's clear from the comments that each and every fitting of the SEAWAVE-QEX needs to be examined for proper fit.  This is clearly not an apply-it-and-move-on process.  
                  So on the long-term sampling bias factors, which is the second part of the question, right?  So in Section 4.2, the white paper states that the creation of long-term SBFs is not that different from the short-term SBFs and that SBF approach is likely to produce better results when applied to sparse monitoring data because, one, the influence of missing data -- and this is a quote.  "The influence of missing data is believed to have less of an impact on longer term concentrations that shorter-term concentrations." And two, "Interpolation between sampling concentrations has minimal impact on estimating a long-term average."  
                  While this may be true, it's likely that when monitoring samples are taken during the use season -- when monitoring samples are taken during the use season can have a large impact on the average, which in turn will have a large impact on an SBF estimated long-term average annual concentration.  So we're back to the same question of where are the detects in the year, and that has a big impact, especially on the long-term estimation.  
                  So on the experiment described in Section 4.5 where it produced sparse monitoring datasets with four, six, eight, or ten samples per set, using random or stratified protocols.  And they looked at the fifth percentile of the 10,000, ninetieth percentile upper confidence limit for the mean.  EPA concluded that results suggested SBFs developed with estimated daily chemo -- daily concentrations derived from (ifible) compared well to SBFs developed from measured data.  
                  So I wasn't -- in the writeup it was confusing.  You had to keep reading it to say, well, where was SEAWAVE used, and why did you make this statement?  Because you've got monitoring data which has a little bit of missing values.  And you use SEAWAVE to fill that in, and then you went back and looked at -- compared those two again.  
                  Bob, do you want to take a break at this point and let me finish summarizing or move on and I'll come back at the end? 
                  DR. ROBERT CHAPIN:  I was thinking that taking a break might be good.  Being able to summarize --
                  DR. KENNETH PORTIER:  That's what I'm trying to do. 
                  DR. ROBERT CHAPIN:  Summarizing your comments and suggests rather than summarizing what they did will be good.  So let's take a ten minutes.  Look at your watch.  Ten minutes from now we'll be back.  Thank you. 
                  
                                             [BREAK]
                                                
                  DR. ROBERT CHAPIN:  All right.  Here we go.  Okay.  Let me see.  Ken, shall we go to somebody else? 
                  DR. KENNETH PORTIER:  I'm going to summarize real quickly.  I wrote three summary statements. 
                  DR. ROBERT CHAPIN:  Awesome.  I love it when you talk like that.  Let me just make sure that our EPA friends are ready to receive the words of wisdom. 
                  DR. KENNETH PORTIER:  They're looking at the error file that I suggested they look at.  So in summary, I think the study was performed properly, but a slightly different statistical analysis model should be considered.  Secondly, the conclusions from the analysis are probably correct, but they need a better, read clearer, discuss and summarization.  I think part of my problem is I'm trying to understand what they actually concluded.  And most of the time, I'm saying, okay, if I read this correctly, they're right.  But I'm not always sure I'm reading it correctly.  And the third thing is there does not seem to be that much difference in the findings between the short-term SBFs and long-term SBF properties and performance.  So I think that's kind of my conclusion on this. 
                  DR. ROBERT CHAPIN:  That's helpful.  Thank you.  Okay.  All right.  We're done with the statistical end of the table.  We'll move down to Dr. Potter. 
                  DR. THOMAS POTTER:  I have found my risk assessment hat.  So let me get closer to the mic here.  As noted, just some statements of the obvious here, but sample bias factor has been a topic of discussion for a long time.  I feel like somewhat of an innocent stepping into this at this point since I haven't been part of that dialogue.  I will say I think from reading the tea leaves here that there is a consensus that they may be a valuable tool for risk assessment.  And some samples were shown, particularly in Tier 3, which seem perfectly logical and I think would be a valuable next step, if it's not already being implemented.  
                  With that said, a number of folks have raised concerns about SBF favoring false positives over false negatives.  From a health perspective, or a health protective perspective, maybe that bias towards false positives seems reasonable in a sense of misclassifying or missing a health exposure that would be a real concern.  So again, that will be an ongoing concern I think by some.  But perhaps it's not an exceptionally of a concern.  
                  And then it was observed, and has been observed, and I think sort of the motivation of this question here today is that sampling bias factors can be really uncertain, particularly when you have not much data.  So I look at SEAWAVE-QEX and, to some degree, that can solve the problem.  And indeed, looking at the way the model was applied, it all seems reasonable to me.  I think from a risk perspective this is a really neat way and elegant way to get at this particular parameter, the sampling bias factor.  
                  Ultimately, I believe its best use as a tool for risk assessment is screening, and I'm getting the feeling that that's the perspective on the Agency.  We'll perhaps get to that later on in our discussion of, I think, it's Question 4 when we talk a little bit more about the whole risk assessment process.  But as a screening tool, it appears to be quite valuable.  
                  My concern in the analysis that's gone on here, kind of again, some of it's going to be discussed later on.  But if we have enough data to calculate -- to use SEAWAVE-QEX to calculate the sample bias factor, I'm not sure what the game is, other than that you may be able to screen out selected sites for further analysis in Tier 4.  Ultimately, the real challenge, I think, with SBF is that we have problems that they don't regress well.  So it's hard to extrapolate across chemicals.  And I'm not sure there's a good solution for that.  We'll talk a little bit about that later on.  
                  So I'll simply say, from a risk assessment perspective, I think sampling bias factors are a very powerful and useful tool.  You can screen your data, particularly at Tier 3.  And that can serve to minimize efforts or at least direct efforts in areas where you have a real water quality concern in terms of exposure.  And that'll optimize resources and hopefully get the job done. 
                  DR. ROBERT CHAPIN:  Dr. Klaper? 
                  DR. REBECCA KLAPER:  I think overall I was really impressed with how well the models ended up fitting the data and anticipated peak concentrations, which seems to be the goal -- is to figure out high concentrations in particular areas, not necessarily daily concentration.  So that the bioconcentration factor put that boundary on where you thought that the top limit would be and helped to give a fudge factor to the data that I think is necessary based on  concurring with comments from other panels that have happened -- so overall, I think it's a really good tool and that the sampling factor bias calculations are really useful and seem good.  The one problem that I think that I voiced before is that I think that it would be good to keep in mind about how climate change is causing extreme variability in systems and that when you go to calculate the upper end of pesticide concentrations and variability over time that we haven't necessarily seen the variability evaluated as far as estimating how that will impact the high concentration and this bias factor analysis.  But otherwise, I was very impressed with how well the model stuck to the data. 
                  DR. ROBERT CHAPIN:  All right.  And Dr. Sadd? 
                  DR. JAMES SADD:  I agree with the panel that you've demonstrated the SBFs have value in some cases for screening at Tier 3.  And I also agree with Dr. Berrocal's explanation in the optimization analysis that your decision to use a deterministic interpolation method limits the ability of the model to identify extreme values.  I know that you recognize that, and you also recognize that the range of values for SBFs are wide.  And you're concerned about that and the additional uncertainty in the data.  
                  That would be particularly true if you were going to apply it to non-flowing systems or small rivers or those with rapid fluctuations of discharge.  So if additional testing analysis and new datasets as possible, which it seems like it might be, then it might help resolve some of these reasons and explain the wide range of SBF values in some cases and maybe offer the opportunity to conduct testing with different pesticides.  I don't have anything else to add regarding the long-term SBFs. 
                  DR. ROBERT CHAPIN:  Excellent.  Thank you.  Comments from other members of the panel on these SBFs for short-term and long-term approaches?  All right.  I turn to my EPA colleagues and ask if you want to ask clarifying questions of the panel. 
                  DR. ROCHELLE BOHATY:  I think we're good.  Thank you. 
                  DR. ROBERT CHAPIN:  You can have another third 20 seconds to discuss among yourselves if you need to.  We have a point of clarification. 
                  DR. CHRISTINE HARTLESS:  Thank you.  I wanted to address some of the earlier comments made by Dr. Berrocal about whether or not the use of the 14 day and the seven-day sampling intervals and clarify that that pieces of that was for just evaluation of SEAWAVE itself.  We used a different sampling methodology for evaluation of the sampling bias factors.  So obviously in our white paper and/or the presentations, the distinction between those wasn't quite made clear enough.  So we will have to work to make sure that we can do a better job of clarifying those differences -- clarifying what we've done. 
                  DR. ROBERT CHAPIN:  Great. 
                  DR. XUYANG ZHANG:  Hi.  This is Xuyang Zhang here, I have a brief comment. 
                  DR. ROBERT CHAPIN:  Go for it. 
                  DR. XUYANG ZHANG:  I just want to second Dr. Portier's comment on a better statistical analysis on the results, especially on the site to site variability.  I was trying to explore optional reasons why the regression models did not work well.  And I think a better understanding of the site to site variability and the magnitude and the factors that are contributing to the differences on the outside would be instrumental in performing the regression analysis.  So I just think that, in the future, if EPA could do a little bit more in that regard, that might be helpful.  That's it. 
                  DR. ROBERT CHAPIN:  All right.  Good.  Thank you very much.  So we're good around the table.  Anything else from the EPA folks on this question?  You're good?  Shall we move on to 2(b)? 
CHARGE QUESTION 2(b)
                  DR. ROCHELLE BOHATY:  2(b), please discuss the strengths and weaknesses of developing site-specific sampling bias factors using a percentile, for example, the median across realizations and the median across years, from SEAWAVE-QEX estimated daily pesticide concentrations.  Describe the utility of this approach for use in pesticide drinking water assessments as highlighted in the attached drinking water assessment case studies. 
                  DR. ROBERT CHAPIN:  Just because we haven't given him enough work to do already, we're going to start off with Dr. Portier.  All right.  So we're going to do strengths and weaknesses of site-specific SBFs. 
                  DR. KENNETH PORTIER:  Pass.  No, actually I don't have a lot to say on this.  Using the medians of the medians makes practical sense, and I think it makes statistical sense.  Because of the way SBFs are conceptualized as a ratio of the true value divided by a statistic derived from subsampled monitoring realization, any statistic could be used in the denominator, and any statistic could be used to describe the kind of average effect across all the realizations.  And the median makes a lot of sense, because of all the percentile estimates, the median has some of the better statistical properties.  It's more likely to be symmetrically distributed and skewed.  It's easier to estimate its P50 value.  So that's one of the strengths of what you're doing.  
                  And I was sitting there saying, "A weakness of the use of the median?"  And I'm thinking I can't think any reasons why I'd not use the median for this kind of thing.  Again, the whole point is, to a certain extent, any statistic would work.  You want a statistic that has good properties, and I think the median's there.  
                  And I may have been one of those that recommended that in 2012, say "Why don't you use the median?"  I didn't go back and look at it, but he may have been using an average.  I said, "Oh, no.  That's too --"  For example, the average is very susceptible to the extreme values.  So if you get one bad value, the average could bounce all around, but the median doesn't care.  
                  DR. ROBERT CHAPIN:  There's statistics that don't care? 
                  DR. KENNETH PORTIER:  So I kind of say, in the end, the best justification is that it works when tested with best available data, which is what's presented in the white paper.  So one of the argues for why it works is because it works.  It has good properties.  
                  "Describe the utility of this approach for use in pesticide drinking water assessments as highlighted in the drinking water assessment case studies."  And I think I'm going to punt on that one and let the risk assessors and others address that.  And Veronica here's going to be really upset with me because she really wanted me to talk for 35 minutes, and now she's got to say something. 
                  DR. VERONICA BERROCAL:  That is true.  I didn't have much to say on this Question 2(b).  So that's why I asked that he goes first, and I was hoping he would talk forever.  But once I hope then he doesn't do it.  So yes, I don't have anything else to add besides what Dr. Portier already said. 
                  DR. ROBERT CHAPIN:  I love it when you talk like that.  Let's see.  So that leads us up to Dr. Sadd. 
                  DR. JAMES SADD:  So the site-specific sample bias factor can be valuable in some cases to inform an existing sampling program as it goes forward to make it more effective or in deciding a sampling plan at sites where they are sufficiently comparable to the one where the SBF was developed.  EPA recognizes that SBFs at a given site can vary substantially from year to year.  And they made the choice to select the median as the SBF specific to that site.  That's a reasonable decision to address a difficult problem.  But I can see that EPA also recognizes that that's not the optimal long-term solution.  
                  Another potential negative aspect to site-specific SBFs is the reasonable expectation that in some places there will be environmental changes associated with changes in land use within the watershed or perhaps even changing climate conditions that might add to the year to year variation and eventually render a site-specific SBF less and less effective over time.  And I recommend the EPA consider a further analysis to better understand the year to year variability and the reasons for it. 
                  DR. ROBERT CHAPIN:  You think that ought to be short-term, mid-term, or long-term reanalysis? 
                  DR. JAMES SADD:  Well, I think mid- and long-term probably because that's where the changes really would be most pressing or most effective. 
                  DR. ROBERT CHAPIN:  Good.  Excellent.  Thank you.  All right.  And Dr. P? 
                  DR. THOMAS POTTER:  I guess that's me.  Thomas Potter.  Well, I'll echo much of what's been said already.  I think that it's a reasonable approach.  It's actually a very conservative approach compared to the alternatives, which are applying across sites and in other factors.  We certainly saw the need for it in some of the analysis that was done, which showed there are large variations between some of the sites that were evaluated with the SEAWAVE-QEX data.  So yeah.  
                  It seems reasonable.  I think your data analysis was excellent, and it's a tool that can be valuable.  Although, again, I'm concerned about how limited it might be if indeed you can't extrapolate between chemicals.  So again, we're going to get to that.  But I think that's really the discouraging part.  And again, maybe there's a path forward and we'll talk about that. 
                  DR. ROBERT CHAPIN:  Okay.  Thank you very much.  Comments from other panel members on Question 2(b)?  Okay.  Seeing none, I'll look to my EPA colleagues and ask if you want any clarifying questions of us? 
                  DR. ROCHELLE BOHATY:  I think we're good.  Thank you. 
                  DR. ROBERT CHAPIN:  Cool.  Tick.  Okay.  Turn the page over.  Question 2(c). 
                  
CHARGE QUESTION 2(c)
                  
                  DR. ROCHELLE BOHATY:  2(c), please comment on the utility of using the maximum short-term and the median long-term sampling bias factor for the four pesticides, that is atrazine, carbaryl, chlorpyrifos, and fipronil, to estimate upper-bound concentrations of other pesticides in surface water for which either sampling bias factors cannot be or have not been derived.  In addition to using these values as a screen, EPA proposes an option to select sampling bias factors for an individual pesticide, either atrazine, carbaryl, chlorpyrifos, or fipronil, based on other defining attributes, such as environmental fate properties, use profile, flow rate, basin size, waterbody type, and/or land use.  Please comment on what factors EPA should consider when selecting an alternative sampling bias factor for estimating upper-bound pesticide concentrations on a national or regional scale.  
                  DR. ROBERT CHAPIN:  Another simple question.  Dr. Potter? 
                  DR. THOMAS POTTER:  Yeah.  That's a tough one.  We've alluded to this several times, and of course you folks made, I think, a valuable presentation yesterday on this topic.  The challenge, again, sampling -- if you could extrapolate between chemicals, sampling bias factors would be an extremely powerful tool, rather than a moderately effective tool, from my perspective.  So I'm not sure there's an effective path forward on this.  But one thing I've been kind of thinking about, and perhaps there's some way to dig into this, is to look at the shapes of the waves because I think, to some degree, they could be governed by chemical properties -- sort of relates to the idea of field dissipation processes that are governing that wave shape in addition to hydrology.  
                  So if there's an agreement in wave shape, you could look at some parameter associated with wave shape, such as amplitude or width if it's a relatively symmetric one -- width at half-height.  It's sort of analogous to what one does in a chromatographic analysis in chemistry.  You're looking at, from a quality perspective, analyzing peak shape to see if it fits the criteria.  And again, you can compare similarity across things that can be tailing factors that could be built in.  
                  But perhaps that's a path forward is to really actually go back to the model and look at what the waves are and do some kind of comparison between peak shape to see if there's some uniformity there between certain types of chemicals.  And then therefore, there's seems to be some potential for extension of the SBF factors calculated for the four chemicals to others that would be of interest.  So I offer that as a thought.  
                  What else do I have to say on this?  I had one other point.  No, I'll stay right there.  I think again, peak shape -- oh, I know what I wanted to say.  This probably is really going to dig things out.  But let me ask this question because it comes back to yesterday.  I thought I heard you say yesterday that you did a regression with terrestrial field dissipation half-lives.  Is that correct?  And I thought, wow, that's a great idea.  
                  So I still think it's a great idea.  I believe you saw some relationship.  And it would stand to reason that terrestrial field dissipation might be an interesting parameter because, again, it's integrating a lot of processes that are going out there in the landscape.  I was thinking, yeah, finally, there's some real, possible, valuable use for TFD other than as a screening tool.  So I commend you on that and recommend that you look into that in more detail.  
                  DR. ROBERT CHAPIN:  Excellent.  Thank you very much.  Dr. Sadd? 
                  DR. JAMES SADD:  Using SBFs in this way would provide EPA with a potentially valuable tool, I agree, if you can pull it off.  But applying them to other pesticides introduces more uncertainty and potential bias because of differences in the properties of other pesticides and their behavior and fate and the environment as compared to the four pesticides for which SBFs have been carefully evaluated.  Using the other attributes as mentioned in the question to inform -- such as a substitution of -- has a potential to introduce even greater error and greater bias.  And their likely effects probably vary substantially, perhaps more than pesticide to pesticide properties variation.  And if they were going to be -- and some of them are going to be not quantitative, such as waterbody type and possibly some land use characteristics.  
                  So this would require subjectivity in applying them in this context.  If EPA plans to use them in this way, I would just suggest that clear rules for how they would be used to inform a decision on SBF use should be developed and justified, preferably through testing.  
                  DR. ROBERT CHAPIN:  Let's see.  Dr. Baffaut? 
                  DR. CLAIRE BAFFAUT:  So I'm going to address the second part of this question.  I think here in a sense EPA is asking how the vulnerability of land could be introduced in the analysis and in the selection of the SBF.  And I carefully looked at the list of attributes listed in Table 11.2.  I couldn't find any that should not be there, so keep those.  
                  But in addition to those, I could think of a few:  for example, the dominate hydrologic soil group of the land likely to be treated with a pesticide of interest, the similarities between land cover to which pesticide is applied -- that would be the percentage of area in the watershed, as well as the type and management of the land covered.  Try to be more specific in ag land, perhaps land and the crops that are relevant to the pesticides of interest -- the slope of the land likely to be treated with that pesticide.  The biochemical properties, we've talked about it -- mean annual temperature, just because it may affect product degradation; rainfall amount or intensity; shape of the watershed --  I think you had some of those variables that reflect that -- the presence of subsurface artificial drainage.  You do have the presence of a restrictive layer in that list, but the presence of subsurface artificial drainage, or any kind of drainage, even if it's surface drainage, would be important.  
                  Then I would like to add that this question of land vulnerability -- there's been a lot of research I'd say in the last ten years, and several groups have come up with different indices.  So I'm going to list three that may be interesting.  The Cornell group, and that's Tamue Stenuis (phonetic), came up with a topographic index.  And they have different versions of it.  But it was directed at land likely to be saturated when it drains.  And it was in a setting that was characterized by permeable land or permeable soils on top of a restrictive layer.  
                  In the Midwest, my group, we have come up with a conductivity claypan index.  When you compare with a topographic index, it's actually very similar.  And then NRCS came up recently with something called the Soil Vulnerability Index.  The interesting thing is all those three index use very similar soil properties.  And the other thing is those -- in the equations that you present in Section 11, the factors are additive.  All those indices that I'm talking about, the primers are -- or the indices are multiplicative.  So the different variables are multiplied by each other or divided.  But it's multiplication over fractions, not an addition.  And it has proven to be effective at predicting vulnerability.  And that's -- I've said everything I wanted to say. 
                  DR. ROBERT CHAPIN:  All right then.  Dr. Green? 
                  DR. TIMOTHY GREEN:  Thank you.  Those are great comments.  I'll be brief.  First, to reiterate what I said in 1(d), I think you can use scaling information of watershed area with the sample bias factors.  Tomorrow, I think in 3B we'll look at regarding quantitative watershed characteristics, which may support some of Dr. Baffaut's ideas in terms of how you would use watershed characteristics as inputs.  And then I didn't think of this, but thanks Dr. Potter for ideas about leveraging chemograph characteristics, so essentially using the response as another way of characterizing the different chemicals.  That's all I have. 
                  DR. ROBERT CHAPIN:  You can just shuffle that microphone boom right over next to Dr. Nowell. 
                  DR. LISA NOWELL:  I agree with the comments previously stated.  And I was thinking about the multiple regression.  I guess this is going to come up with a later question in three.  So EPA concluded at the end of the regression experiment that additional research was needed before the SBFs could be confidently predicted using watershed characteristics, and that seemed quite clear.  But it seemed like it was promising and that it'd be worth continuing to work on this.  So if you can't explain the variability of a same compound in different watersheds, it's going to be harder still to explain across different compounds.  
                  So I was think that, as a goal, we could use the SBFs to try to understand more about the sources of variability.  And I think the regressions so far were oriented -- they were predictive models.  You were trying to use basic GIS data that you could get for any site.  So a lot of the factors that we know are going to affect those concentrations, like flow and sources use within the watershed, would be much harder or even impossible to get generically.  But there might be -- I think you could rationalize a need to develop -- to work on an explanatory model rather than a predictive model, just to try to get a handle on the factors that are affecting the variability and then use that maybe as a guide to try again with a predictive model.  And some of these more data intensive variables, like pesticide use or flow, to the extent -- or surrogates for these would be worth trying, and scale, of course, watershed scale.  
                  Again, we've only got four chemicals.  Although, really I think you could count metolachlor as five.  I don't know if you have others.  But you used metolachlor in this as a way of evaluating your SBFs.  But it looks like you have data for metolachlor, as well as the other four, for both the NCWQR sites and presumably the USGS sites as well.  I don't know if you've done that yet.  But anyway, that would be five.  But that's still not really enough to develop a regression model among compounds.  So I'm hoping that you'll be able to continue the effort, expand the number of compounds that you can work with.  And then maybe a ration might be feasible. 
                  DR. ROBERT CHAPIN:  All right.  And for this question, we get to end as we began.  Dr. Potter is down here as the last with his aquatic modeler hat on.  Would you like to add anything else to your earlier comments? 
                  DR. THOMAS POTTER:  Well, I'm thinking that there's only one other thing that I would add, and it's not necessarily connected to my earlier comments.  But it was when I was listening to Dr. Baffaut talking about watershed properties.  One thing that would be very useful to look at is the relative distribution of base flow and storm flow.  I think those are key parameters related to chemical transport.  And there may be some watershed where you can get that information readily available.  There's a very nice paper that was out, again by -- I keep on promoting a former colleague -- Dave Bosch, and I will give you the citation on that.  And they talk quite a bit about the importance of base flow and contaminate transport, stormflow, and how to separate those out.  I think that might be something that would be very useful as a predictive tool. 
                  DR. ROBERT CHAPIN:  Comments from any untapped panel -- go ahead? 
                  DR. LISA NOWELL:  I just forgot I just wanted to make one further comment.  I was thinking about the univariate relation between -- was it the terrestrial field dissipation?  And it looked to me like fipronil might be driving that because it has a notably lower SBF, and it's also by far the most persistent.  So again, more compounds with  more varied properties will help with that. 
                  DR. CLIFFORD WEISEL:  So I agree completely with everyone saying you need more compounds.  The question is how you get that data.  And I'm looking to my colleagues to see if this is at all something to think about when you talk about environmental fate.  There's a lot of work done -- we're not looking at the watershed but rather absorption of the pesticides into soils and how long they last, which is related, I guess, to the terrestrial -- that each company has to do it to understand it.  I wonder if that might be a source of data just to look at how different pesticides might go around and then combine that solubility information just to see whether that may extend some of the underlying principles because if you don't have enough data you can't do the complete statistical analysis.  So you have to meld the mechanistic with the statistical.  And then they'd just be another source of thinking. 
                  DR. REBECCA KLAPER:  I'll just say that the compounds that do bind to soil more heavily still end up in the water column when they wash up on land.  But they still could behave differently as far as transport goes and such.  So it's apples and oranges.  It's very useful information.  It tells you the chemicals are different and can behave different in an aquatic environment.  But it's not easy to translate the octanol water coefficient to some behavior in a water system, unfortunately.  It would be nice if it did. 
                  DR. ROBERT CHAPIN:  Other untapped, unheard comments from the panel on this question?  All right.  None yet?  Okay.  Anything that we can clarify for our EPA friends? 
                  DR. ROCHELLE BOHATY:  Yes.  So I just want to make sure that I think I heard consensus on using the four chemicals as a screen, but there was some reluctance on identifying one of those individually for application to another pesticide.  Is that correct?  And if not, can you please clarify? 
                  DR. ROBERT CHAPIN:  Where did that come from?  Jim, is that --? 
                  DR. JAMES SADD:  I agree with what you just said. 
                  DR. ROCHELLE BOHATY:  Is that the consensus?  Is there any --? 
                  DR. REBECCA KLAPER:  I would agree with that, too, but it's very difficult to extrapolate from one to the other at this point.  It's very difficult to, right now, with these four chemicals and given the limitations of the variability in their chemical primers, to say that you could extrapolate from one of these to everything or everything with that category.  But it is good for the screening purposes and for the examples that you've given to demonstrate that they were given enough data. 
                  DR. ROCHELLE BOHATY:  Great.  Thank you. 
                  DR. ROBERT CHAPIN:  Okay.  Check.  D, 2(d).  Here we go. 
                  
CHARGE QUESTION 2(d)
                  
                  DR. ROCHELLE BOHATY:  All right.  2(d), Rochelle here again.  In Chapter 4.6, EPA concludes that sampling bias factors are a reasonable tool for increasing the amount of available monitoring data that can be used as a quantitative measure of exposure in pesticide drinking water assessments beyond those meeting the SEAWAVE criteria.  Considering the answers to questions 2(a) through 2(c), please comment on this conclusion.  
                  DR. ROBERT CHAPIN:  Okay.  So we have Dr. Sadd. 
                  DR. JAMES SADD:  I'll just say briefly that I agree with this conclusion, and I agree with it with consideration of the limitations and the issues of concern that have been discussed earlier in the panel's answers to Questions 2(b) and 2(c). 
                  DR. ROBERT CHAPIN:  Short and sweet.  Dr. Potter? 
                  DR. THOMAS POTTER:  Yes. 
                  DR. ROBERT CHAPIN:  Shorter and sweeter.  They're clinking glasses down at the end of the table.  Did we have any clarifying questions from the EPA? 
                  DR. ROCHELLE BOHATY:  No, that was very clear.  Thank you. 
                  DR. ROBERT CHAPIN:  So let me just see if any other panelists have comments on this question.  All right.  I think we've finally hit everybody's tolerance limit for long, involved answers.  All right.  So are we done with 2(d)?  No questions from you guys?  Do you have anything to follow up? 
                  DR. ROCHELLE BOHATY:  No questions.  Thank you. 
                  DR. ROBERT CHAPIN:  2(e). 
                  
                  
CHARGE QUESTION 2(e)
                  
                  DR. ROCHELLE BOHATY:  2(e), Section 6.6 of the white paper describes the utility of sampling bias factors in the context of designing a surface water monitoring program.  Please comment on EPA's conclusions regarding how sampling bias factors can be used to further optimize a surface water monitoring program design for the greatest utility of monitoring data in a pesticide drinking water assessment.  
                  DR. ROBERT CHAPIN:  All right.  Mr. Councell? 
                  MR. TERRY COUNCELL:  So your Section and your Table 6.1 is very comprehensive.  It's very similar -- I think my response would be look at the previous response on the monitoring program.  It's all --
                  DR. ROBERT CHAPIN:  Terry, if you could get just a little closer to the mic, that would help the transcribers.  Please and thank you. 
                  MR. TERRY COUNCELL:  Use all the data you can, the SBF factors, whatever to design your monitoring program.  And then, as I said before, put that out so the other monitoring programs can somewhat mimic it and potentially provide you with a good source of data. 
                  DR. ROBERT CHAPIN:  All right.  Dr. Sadd? 
                  DR. JAMES SADD:  SBFs would be of significant value in designing new and improving existing water monitoring programs, primarily in specifying sampling intervals and schedules for flowing water sources.  Using SBFs to reduce required sampling for pesticides of concern for long-term exposure and chronic health impacts seems the most appropriate situation for applying SBFs for this purpose.  
                  DR. ROBERT CHAPIN:  I love it.  Dr. Sobrian? 
                  DR. SONYA SOBRIAN:  I can't say I just agree because I do have a couple of points I'd like to at least ask you to consider.  We've talked a lot about which covariates used.  Your white paper is somewhat bipolar in that it does list, like on Page 44 -- it says, "In exploring alternative covariates to steam flow that may be available for a given pesticide occurrence dataset when steam flow is not available and not appropriate, such as periods of back flow.  Daily stage data was found to be an ideal alternative when available for typical flowing streams.  Moreover, measured precipitation was determined to improve model fit over model precipitation."  
                  And then on Page 152, it states that steam -- this is in the white paper.  "Steam stage data performed very similarly to flow with the WSDA sites since flow data are typically derived from stage data.  As such, stage data is likely a viable alternative covariate for sites that have typical positive flow patterns if the daily stage data is available and stream flow is not."  
                  I just wanted to say that in trying to -- there's been a lot said about the pluses and minuses of when to use various covariates.  And I think yesterday the gentleman from Washington said that flow is better than stage is better than precipitation.  It's just that when you read through the white paper you get different aspects, and maybe that just needs to be cleared up.  
                  The second point I want to make is something that Dr. Portier alluded to in an earlier question.  And that's when sampling durations of three years have been chosen for use with SEAWAVE-QEX, even when the advantages and programs three to five years in duration are listed in the white paper.  Vecchia indicates -- 2018 -- indicates that three years are sufficient to produce estimations of the annual maximum daily concentration if the assumptions to SEAWAVE-QEX model are met.  He also states, however, "Short sampling records, sparse sampling frequency or more frequently highly censored data make verification of the assumptions more difficult."  Bias was found to be lower for record length of six years when compared to three.  So I guess my question is sort of why three years when it's been proven that six is better?  
                  And the other issue is a three-year time period does not have to be consecutive.  As Dr. Portier pointed out, the first and the last year, the anchor years have to fit the assumptions, but the middle years do not.  I didn't find anywhere in the SOP or in Vecchia 2018 any indication of how you can choose which years to skip and any kind of criteria for that.  I thought the SOP, as was said earlier, was excellent.  But I didn't see that in there.  
                  The other question I want to ask is about the chronic and cancer risk assessments.  As few as four samples per year require applied sampling bias factors to estimate upper and long-term and average concentrations, even though the SBF adjusted 365-day average actually is protective.  Do you think it will be so over 30 years, given I think as the gentleman from Washington yesterday pointed out that you have changes in environmental conditions and pattern of use?  How reasonable is it to consider using a 365-day daily average over a minimum amount of time to estimate a chronic cancer? 
                  DR. ROBERT CHAPIN:  So we don't want to ask them questions.  We want to give them recommendations. 
                  DR. SONYA SOBRIAN:  Okay.  The recommendations would be to at least address those questions, especially with respect to the SOP because it's excellent.  It really is.  Reading that, I should have read that before I read the white paper, but I didn't.  But it's excellent. 
                  DR. ROBERT CHAPIN:  All right.  Excellent.  Thank you.  Dr. Yang? 
                  DR. RAYMOND YANG:  I don't have anything to add.
                  DR. ROBERT CHAPIN:  Okay.  Somebody please check Dr. Yang's pulse.  Let's see.  Okay.  Comments from other members of the panel that would like to add something to this question?  Oh, we have pain.  We have pain. 
                  DR. CLIFFORD WEISEL:  I thought that the SBFs were for Tier 4, and they were single number values.  Am I wrong on that? 
                  DR. ROCHELLE BOHATY:  So what we put forth was that you would use the full distribution of sampling bias factors available for the four pesticides derived from the USGS data at Tier 3 as a screen.  And then at Tier 4, you would use -- you'd develop pesticide specific sampling bias factors. 
                  DR. CLIFFORD WEISEL:  Right.  So this is -- this question's related to I guess Tier 4, I thought.  Or is this earlier?  Because if it's Tier 4, even though it's specific to a pesticide, it's a single number.  I'm not sure how that guides your sampling program.  I'm sorry.  
                  DR. ROCHELLE BOHATY:  Hang on.  We can look. 
                  MR. CHARLES PECK:  My interpretation of this is sort of like we've gotten to Tier 4.  We've seen that there may be a problem.  And so now, we'd like to use the sampling bias factors to sort of help in the development either in a pesticide specific surface water monitoring program or in the even that a state agency or somebody like that comes up and wants to help look at developing a monitoring program or improve their monitoring program.  How could the SBFs be used for that as well? 
                  DR. CLIFFORD WEISEL:  So think based on that watershed and various factors?  That's why I'm confused.  It sounded to me like this was a single number that gives you towards the maximum in Tier 4.  So I'm not sure how that really derives and helps you get to a full monitoring program.  
                  DR. ROBERT CHAPIN:  Yes, please. 
                  DR. KENNETH PORTIER:  I was thinking about this specific question, too.  How useful are SBFs in designing a monitoring --?  To me, the only thing the SBFs might help you do is identify poor monitoring sites -- monitoring sites with poor data that you might want to buff up the monitoring so that you could really determine if that SBF estimated maximum is real or just an artifact of poor data.  And to me, that's about all you can get out of these SBFs in terms of a design.  
                  We had a lot of other discussion about how SEAWAVE-QEX gives us a lot of information on how to buff up a nice monitoring site and/or what we're looking for in a new monitoring site.  But in terms of going to the state and saying, "You know at level 3,  level 4, these sites seem to have an estimate that exceeds your DLLOC.  What we want you -- but we're not fully confident that you've got to do something now.  I think you need to raise your monitoring so we can do a better determination."  I think the SBFs help you with that, and that's about it. 
                  DR. VERONICA BERROCAL:  I think also on the thread that -- the discussion we had before about the magnitude of this SBFs, that many of them are going to be large.  So even though you can use them to help you figure out where you might want to put your effort in terms of monitoring, I think you also have to maybe concentration just on the ones that are extremely large, since many of them would be very large. 
                  DR. ROBERT CHAPIN:  All right.  So let me come back to Cliff and make sure that -- do you want to save this for later kind of discussions? 
                  DR. CLIFFORD WEISEL:  Yeah.  It seems to have some limited value, and I was trying to figure out what that was because this seemed to be much broader than the question. 
                  DR. ROBERT CHAPIN:  I think taking that offline would be -- getting educated about that would be the best thing to do.  All right.  Other comments to respond to this question?  Yes. 
                  DR. TIMOTHY GREEN:  Just a quick comment that the issue of stream stage versus discharge or stream flow has come up.  I think this is common knowledge, but it's a non-linear rating curve.  So replacing stage with discharge will give you different results.  And also the rating curve's developed over a limited range of stages.  So uncertainty also changes with stage or discharge.  That's just a comment because we've mentioned the use of both, and I think that it's clear that those uncertainties exist.  
                  DR. ROBERT CHAPIN:  Excellent.  Thank you.  Other comments from panelists?  Nope.  Okay.  We good?  All right.  So I think the panel's exhausted itself on this question.  Clarifying questions from our EPA friends?  You guys good? 
                  DR. ROCHELLE BOHATY:  No clarifying questions.  Thanks.  
                  DR. ROBERT CHAPIN:  So three, here we go.  Over to you. 
                  
CHARGE QUESTION 3
                  
                  DR. ROCHELLE BOHATY:  Question 3, to meet EPA's drinking water protection goals, monitoring data used in drinking water assessments should be relevant to drinking water intakes.  EPA developed two methods to investigate the spatial relevancy of monitoring data and sampling bias factors.  EPA developed sampling bias factors for four pesticides with a range of use profiles, physical-chemical properties, and environmental fate properties, using occurrence data from sampling sites across the contiguous United States.  SEAWAVE-QEX was used to develop sampling bias factor development for these sites, which have different watershed and catchment attributes.  The sampling bias factors were related to watershed and pesticide attributes to develop regression equations to estimate sampling bias factors for watersheds where there are not enough data available to develop these factors.  Additionally, a weight-of-evidence approach was developed to assess how relevant sampling bias factors or monitoring data concentrations are to drinking water intakes.  
                  
CHARGE QUESTION 3(a)
                  
                  3(a), please discuss the suitability of the underlying data, methods, and parameters  used to develop watershed regression equations for estimating sampling bias factors for systems with limited data.  Please comment on EPA's conclusions that the short-term sampling bias factor regression equations for the four pesticides evaluated, as well as the environmental fate properties regression analysis, provide minimal predictive ability for estimation of sampling bias factors for other sites.  Discuss approaches and the value in continuing to investigate quantitative relationships of sampling bias factors with watershed and pesticide characteristics.
                  DR. ROBERT CHAPIN:  Okay.  So before we dive into this, I'll point out that -- I'll point out to the panel that the last sentence says they're looking for guidance on the value of continuing to investigate these relationships.  So if we can be specific about answering that question, that would help them.  All right.  So we have please comment on their conclusions and then discuss the approaches and the value.  All right.  We're going to start off with Dr. Portier. 
                  DR. KENNETH PORTIER:  Thank you for that question, something I can answer.  This last one was a disaster, but I'm ready here on this one.  So performing the regression analysis makes sense because a key aspect of making the SBFs useful is applying SBFs generated at data rich sites in years and poor data sites or years or weak data sites and year, if you like.  Utilizing known site and year characteristics in a regression context would seem a logical way of making this what I think of as a spatial interpolation, or it could be an extrapolation.  I'm not sure.  But whatever it is, making that interpolation work.  The risk assessors and the exposure scientists can weigh in on the appropriateness of the individual predictors, the site and year characteristics included in the analysis, but I'm going to talk more about the regression approach itself.  
                  For the 102 unique USGS monitor sites, 60 potential predictors were initially proposed.  And those are given in Table 11.1.  102 unique sites of the SBF reg input data.CSV file has 189 observations.  So I didn't quite figure out why you talk about 102.  And I think it's the chemical and site combinations together.  So initially, you're doing a regression with 189 observations, so that's the bottom line.  
                  So the white paper reports a traditional multiple regression analysis approach.  Some pre-processing was performed to reduce the number of predictors through looking at correlations, contingency tables, and principle components analysis.  While variables were dropped for lack of variability, some were dropped due to high correlation to other predictors.  So lam covered classes were combined.  It also looks like some initial set-wise regressions were used to support the variable reduction.  Still, at the end of this, there were 36 variables available to be used in the final step-wise regression analysis.  
                  Regression modeling was only done for the one-day SBFs generated under 14-day sampling intervals.  Final models resulted in one to two predictors being used and R square values of 0.1 to 0.26.  These are not confidence inspiring results.  And I think that was the conclusion the EPA came to, that with such low R squared values, these regressions were not particularly useful.  
                  So I started thinking, well, what might I use instead of step-wise regression.  So stepwise regression is a traditional choice for building a predictive model.  But there are more recent approaches.  And in particular, I wanted to look at the use of regression treatment methodology for this data.  Now, this is what we would call a small to moderate sized dataset.  Usually, people who do tree regression talk in terms of thousands of observations.  But the methodology works, and it would be interesting to see what the results are.  
                  So from the white paper, the final regression for the atrazine one day SBFs for 14-day sampling interval is given by an equation that's coming up here, where Y is just a linear function of precipitation in the catchment area and egg 2000 value.  Yeah.  Go ahead and bring it up because it's got animation, and it probably doesn't work, right? 
                  DR. ROBERT CHAPIN:  My brain's going to melt.  
                  DR. KENNETH PORTIER:  I like to bring things in.  It keeps some suspense, right?  So this model has an R square of 0.26.  So on the next one, I ran a simple complexity proven regression tree model for the atrazine --
                  DR. ROBERT CHAPIN:  There will be a quiz after class on these. 
                  DR. KENNETH PORTIER:  And this model is a heck of a lot simpler than the just two variable simple regression.  And I actually got a significant increase in the R square simply by using two parameters in the same way but two different parameters.  So here, Y is simply a function of whether your AgKfact categorical variable -- I'm sorry, which now I'm trying to remember what that variable is.  It's some catchment area F factor, right?  Some ag factor that says, well, if it's less than 0.145, then we're just going to estimate the SBF at 4.19.  If it's greater than 1.49, then use the percent of the hay in 2006 in the catchment area.  And if that percent is less than 1.1, we're going to assign a high value of 17.12.  And if it's greater, 7.56.  Click.  
                  And this just kind of shows you how the model is fit.  And this table, in addition to the means, I've given you the standard error.  So you can see not only does the mean SBF value go up, but you know that also the standard error of that goes up as we get up higher values.  So I'm now seeing that when I'm using higher SBFs under a certain condition I have less certainty that that's the right SBF.  Some of them are much better.  Next slide.  
                  Just to kind of describe this, if you look at that AgKfact  cat variable against the SBF values, the area to the -- what would that be? -- left of the red line is the first split.  And you see this 419.  That's our estimate.  You can get an idea of the split there.  The curve that I've drawn in is a low S curve.  So it tells me that this, really even against this variable, is not a linear regression.  It's kind of a flat in one section.  Then it goes up and is kind of flat in the other section.  
                  And then the picture on the right is the second split that has percent hay at the bottom.  And it basically says, well, when there's low percent hay, not only is there a high SBF, but there's a lot of variability in it.  Click.  You get your estimate of 7.12 for just like one, two, three -- there's like nine variables, seven data points there.  And then the next click shows you 7.56 for the remaining group.  So a simple model, easy to utilize.  And I think it ties to variables that have -- watershed variables that have a little bit more kind of realistic utility than the one that dealt with precipitation May/June.  We already have precipitation.  And I didn't do anything.  I threw everything in and just let it go, and here's the model result.  Next one.  
                  I did the same thing for the carbaryl data.  The equation at the top shows that you did a regression on curve number in the catchment area.  I don't know what curve number is, and you didn't really tell me.  But okay.  And it's got an R square of 0.13.  
                  When I did the classification tree, it also used curve number, but now it just tells me, look, there's a value of ten when that number is less than 65 and a value of 30.  When it's greater than 65, then I've got an R square that's seven points higher.  So I've got a slightly better R square.  It's still a poor model.  R squares of 0.2, I don't get excited, although I've worked in the social sciences where they really get excited about a 0.2.  In the hard sciences, that doesn't do anything.  
                  By the way, if I don't have curve number, what these regression trees will do is give me a surrogate number and say, well, maybe instead you might want to use percent watershed wet areas or whatever percent, WD wet 2006 WS is.  But if you use that decision row, you get exactly the same result.  So it tells me that I can come at this a couple of different ways.  
                  Next slide is the chlorpyrifos.  You used the precipitation May/June.  The classification model used this combined percent ag 2006 and the watershed estimate of plus or minus on either side of 45.  And you can see a big difference.  If it's less than 45, 4.8.  If it's greater than 45, it's 18.  So that tells you something about which sites have high SBFs and which sites have low.  And again, you can use percent crop in the watershed as a predictor, and you get almost the same result.  
                  And then the last one is for fipronil.  And that one uses this SATOF48 in the catchment area.  I'm not quite sure what that one was -- with an R square of 0.21.  I can double that R square by using the runoff in the catchment area and just splitting it on 289.5.  A four value versus a two value, it's not a lot, but it's a significant difference in the SBFs and an R square 0.41.  
                  For this particular model, there's no good surrogate to that.  My next best available alternate is the SATOF48 in the catchment area, but it produces a much poorer fit.  So actually, this is a better -- so this just gives you an idea that there's still some analysis that you guys can do and the regression that kind of improves it.  But even taking a different approach here doesn't really improve the fit to the point where it becomes a very useful predictive model.  For me, if I were looking for a 0.7, I'd get excited.  A 0.2 doesn't do anything.  
                  There's some alternate approaches.  You did apply principle components analysis in the exploratory point.  But you could actually do -- generate the first couple of principle components and put them in the regression model instead of the watershed characteristics.  So you're putting linear combinations in watershed characteristics.  Or you can do exploratory factor analysis which would kind of polish the data a little bit and put the first and second factors into the regression model.  And that might improve things a little bit.  It gets a little more complicated to interpret, but it's an alternative that you can easily look at in something like R.  
                  Another approach -- and this one I haven't thought all the way through.  But in this regression, you're trying to predict a ratio.  And ratios have bad statistical properties.  They're not normal.  They've usually got some kind of extreme distribution.  And one of the things we sometimes do is go ahead and take the denominator and put it as a predictor.  So instead of assuming the ratio, you try to predict the maximum concentration as a function of the estimated concentration, see if the beta is one.  That would tell you that that ratio means something.  
                  But it could also tell you how kind of correlated -- how related is that estimated max SBF -- not SBF -- estimated average maximum is to the true average maximum after you've adjusted for some of these watershed characteristics.  That would help you get a better idea of whether your estimated SBFs really have some logic to them behind just being correlated with one.  I haven't thought this all the way through, but it's another way to look at it.  And if you did something like that, you could do things like plot the estimated against the true, put a low S through it to see whether that follows roughly a slope of line one.
                  DR. ROBERT CHAPIN:  Somebody remind me never to ask Ken a question when he's fresh. 
                  DR. KENNETH PORTIER:  Hey, it's 4:00 in the afternoon.  I've only had six cups of coffee.  Finally, I think in the discussion of Question 2(c), Dr. Potter mentioned looking at how the estimated seasonal waveform might correlate with attributes like environmental fate properties or some of these watershed attributes.  And this fired off some neurons saying, well, wait.  Is it really the SBF that we're looking like?  Maybe we can be looking at some of how the coefficients of the SEAWAVE models might be functionally related to watershed characteristics.  
                  And that might give us a little better idea of how to compute a better estimate for the maximum that would produce better SBFs.  Now, we're at a two or a three-step process, but it's just something that, when Tom said that, I thought yeah.  You could easily look at that as well, and that might be very interesting.  Is beta two a function of percent hay?  Something like that.  And I'm done. 
                  DR. ROBERT CHAPIN:  Don, can we put up the charge question again, please?  I just want to make sure that that performance that we just got actually addresses the -- so can you help me with sort of a summary sentence about sort of answering their questions. 
                  DR. KENNETH PORTIER:  I think I answered their question.  "Comment on the conclusion that short-term sampling bias factor regression equations for the four pesticides evaluated, as well as the environmental fate properties regressions provide minimal predictive ability for estimation."  The answer is yes, I agree with that.  The other one is I provided them three different approaches -- alternate approaches for looking at how to investigate those relationships, different kinds of regression models, different kinds of predictor variables that were put into it.  And I've written all this in. 
                  DR. ROBERT CHAPIN:  Awesome.  Thank you.  Just amazing.  So Dr. Berrocal gets -- of course not.  Go ahead. 
                  DR. CHRISTINE HARTLESS:  While this is fresh in our minds, I'd like to ask a clarifying question on Dr. Portier's work there.  This is Christine Hartless.  I appreciate all the work that you did, and you did a lot of leg work in terms of looking at some of those additional regression techniques and playing with the data.  My question then back, and maybe to the rest of the panel as you all are deliberating these other approaches that he presented, do any of them kind of show enough promise at this point that we should kind of continue along in those veins in terms of thinking?  Are they going to provide us adequate predictive ability, or do we really need to kind of look at other work outside of the realm of regression in watersheds? 
                  DR. ROBERT CHAPIN:  I'm going to use what little power the chair has and say that I'm not going to ask the panel to evaluate what he did.  I can barely understand it, and it was in English.  So without having had a chance to study it and stuff, I feel poorly about asking people's considered opinions about whether the EPA should sort of put more weight on these other analyses.
                  DR. KENNETH PORTIER:  So you will notice I put no intermediate or long-term -- I really do think you haven't even shown enough yet to say that this is going to be a productive approach.  I looked at alternatives to see if, well, if we push it a little bit harder, do we get something much better?  The answer is you don't.  So I'm not sure this kind of broad regression approach is going to bear fruit, even on the intermediate term.  Now, I'm going to leave it to the rest of the panel to discuss maybe alternatives that they've seen that are not regression approaches that may help you do that.  I'm hoping Tom might have a really bright idea that's going to set something off again, but I didn't see that.  And that's how I would answer your question, Dr. Hartless. 
                  DR. CHRISTINE HARTLESS:  Thank you.  That does clarify the question, and I'll withdraw the part about asking the panel. 
                  DR. ROBERT CHAPIN:  All right.  Dr. Berrocal? 
                  DR. VERONICA BERROCAL:  So I admit that my neurons are not as active as Dr. Portier this time.  But I do believe that I wouldn't leave this unexplored.  I don't know if it is becomes I'm a special statistician that I always think that you do have this SBF -- this sampling bias factor at different locations.  And we want to predict where we don't have data.  This is what special statisticians do.  So I wouldn't just drop the ball and not try to see whether you can continue investigating whether the relationship between the sampling bias factor and other characteristics maybe did not have the right predictors.  
                  I am also, as I said -- I'm not completely -- my brain is really dying.  But I do think that I have this doubt in my mind whether the regression is really the right modeling tool for the sampling bias factor just because of what the sampling bias factor are.  They are more of the extremes in a distribution.  So maybe when you're using a regression approach, you're trying to model the center of the distribution.  But here, what you're trying to model are the extremes, the highest end of distribution.  So I hope that other colleagues in this panel will not hate me if I let my brain reset and come back tomorrow morning with suggestion of what else you can investigate. 
                  DR. ROBERT CHAPIN:  You can do that.  It's okay.  It's all right.  Okay.  Let's see.  Who's up next?  Dr. Rogers? 
                  DR. JOHN ROGERS, JR.:  So I'll not even wear on you with the statement that I'll be brief.  But I'll try to be brief.  I thank Dr. Portier for tossing the ball in our court.  This is good.  I had to go to sort of fundamental knowledge, and I'm going to share with you I've been through this exercise.  And this is where I ended up.  So I'll share this with you.  
                  I'm not going to look under the light for my keys.  I didn't lose my keys there.  You've heard that joke, right?  Why are you looking under the light for your keys?  Well, there's light here.  Where'd you lose them?  Well, they were over there.  We just looked under the light for this higher correlation coefficient.  We looked hard under that light, and I think we came to a conclusion that we need to go to where we lost the keys.  So I'm going to try to go to where we lost the keys.  
                  First thing I would look at -- and you've done some of this, I think -- the proximity of the sampling site to the location of the drinking water intake, preferably the drinking water intake.  If you're trying to predict the maximum concentrations that's going to end up in drinking water, what I've found is you need to be there.  The further you withdraw from that, the more variants you bring to the matter and the less likely you are to predict that very accurately.  
                  Number two, this may surprise you.  Shoreline development index, that old basic menology that you had where the shoreline development index says how much shoreline do I actually have.  Because in mobilizing a lot of these pesticides that we're talking about, if the shoreline is right there and people are plowing with what I call one wheel in the water, or nearly so, without a lot of buffer, you mobilize pesticides.  They're in the water, and next thing you know they're over by the drinking water intake.  
                  By personal example -- the example I'm using is a 57,000-acre reservoir in my back yard.  My wife drinks tea.  She said, "Why are you wandering around the country studying in California and all these other places issues related to water and pesticides?  We've got issues right here.  They're in my tea, so how about stay how and fix it."  And after 50 years of marital bliss, if you want to maintain that, you stay home and work on this kind of stuff.  
                  Similarity of crops in the crop areas, in other words, I'm looking for convergence here.  The idea is what are the factors that may cause convergence, in other words, the same sort of factors that may cause these data to converge, i.e. correlate, i.e. give you strong regression factors?  If you don't remember what shoreline develop index is, it's the ratio of the shoreline linked to the ratio of a circle that encompasses that same area.  And that's an interesting one because a lot of these watersheds have dendritic reservoirs, and they feed right to drinking water.  So that's what we've found is I can sit up here all day long and look at the inflow to this 57,000-acre reservoir, the major inflows, and I would never predict what's going into the drinking water that supplies 200,000 people in my neighborhood.  
                  The number and types of applications, that may not resonant.  But with the same pesticides, there are options in terms of application: application rates, application tools, frequency of application, and so on.  That turned out to be a pretty important factor in controlling how much got to drinking water.  The similarity of the soil properties, even within a watershed, we can have soil properties walking around.  So we have to pay attention to those in terms of not the traditional -- the USD soil analyses in terms of what kind of soil do I have but particle size analysis, cation exchange capacities, stuff like that that's fairly readily available -- controls either partitioning or release of these materials.  
                  Rainfall events especially important.  Dr. Potter brought this up, but I'll follow it up with it's the frequency and the pattern of rainfall events.  In other words, what we found if we hadn't had a rainfall event for a while, like a couple of months -- and actually went to three months -- and then you get a rainfall event, that's a total different loading to the system and loading to the drinking water than a fairly frequent rainfall event.  So it's that inter interval that's more important than the actual size of the rainfall event.  The size of the rainfall event and stream flow and all that didn't have anything to do directly, predictably with the amount of pesticide that got to drinking water.  
                  It was -- in other words, and I'll probably repeat this briefly later.  The smaller rainfall events actually did not mobilize anything off the land.  The larger rainfall events mobilized so much but diluted so much.  If we're talking about concentrations and predicting concentrations and we're not talking about mass loading, that wasn't predicted because the larger the rainfall event the lower the concentration.  So that blew up.  And then it was those intermediate events that were large enough to mobilize the materials but not sufficiently large to dilute that mobilized pesticide.  
                  And then from there, we took a hard look again at what part of the actual watershed was feeding drinking water.  And it turns out it's not the watershed per se.  There's a sub-watershed that's feeding our drinking water in this particular reservoir and lots of the others that we worked on.  
                  So I will go to my brief comments here real quick.  I agree with your conclusion that short-term sampling bias factor regression equations for the four pesticides evaluated, as well as the environmental fate processes regression analysis, provided minimal predictive ability for estimating sample bias factors for other sites.  These are the kinds of things that I just talked about that allowed us to provide estimates for other sites in terms of what was actually getting to drinking water.  
                  Previous studies have indicated that the lack of relationship between streamflow, rainfall intensity, volume, and concentrations of materials, such as pesticides in trained in rainfall.  So there's not a strong relation, so I would not worry too much about that.  I would be delighted if we could find one, but I would not worry too much if you don't.  Small rainfall events, again, often fail to mobilize those materials from agricultural fields, while large rainfall events may mobilize relatively large mass.  But again, your concentration if you're sitting in the Midwest and your house has water up through the first story, you're not talking about a high concentration of pesticides.  So the materials are essentially diluted.  Rainfall events producing the highest concentrations are often intermediate with a relatively intensive initially pulse.  And that's usually after that field has been treated several times over a season and has sat there and not been irrigated, and it's not had any rainfall.  
                  So I guess simplifying my suggestions, it would be let's look at some other things.  Just from the database that I looked at, I did not subject it to the analysis like my colleague, Dr. Portier, did, but I said I don't think I would find in there what I was looking for if I were looking to predict the concentration of pesticide getting to drinking water.  So I go on to say here briefly that there's some other things you could look at.  I've got those in my report, and I'll capture these also.  So that's my thought. 
                  DR. ROBERT CHAPIN:  Do you guys want to hear the other methods, or are you willing to wait?  John, you've got other suggested methods in your text there.  Is that right? 
                  DR. JOHN ROGERS, JR.:  I do, and I also took a look at the standard things we were looking at, like streamflow, rainfall patterns.  The other thing I thought was real important was soil character, not some soil scientist name for the soil but the sand, silt, clay particulate matter, the organic matter content, things like cation exchange capacity and so on.  So I think the other few comments I have will be clear. 
                  DR. ROBERT CHAPIN:  Okay.  Do you guys want to hear any more specifics from -- it sounds like he's got more stuff written down.  Do we want to hear that verbally, or are you good with a written answer? 
                  MR. DANA SPATZ:  I think as long as it's in the written report we're good. 
                  DR. ROBERT CHAPIN:  Good.  Okay.  All right.  Excellent.  Thank you very much. 
                  DR. ROCHELLE BOHATY:  Just one point, can you make sure in the written portion of that, if there's specific databases you pulled the data from, can you make sure that you include those?  Thank you. 
                  DR. JOHN ROGERS, JR.:  Yes.  I haven't done that to this point, but I certainly will. 
                  DR. ROBERT CHAPIN:  All right.  Dr. Zhang? 
                  DR. XUYANG ZHANG:  So just to address the question, based on the performance of the regression model, I agree with EPA's conclusion that the short-term sampling bias factor regression equation for four pesticides evaluated provides minimal predictive ability and should not be used for drinking water evaluation without further improvement.  And I'm glad to hear Dr. Portier's comments that the regression model could be improved, so my comments will mostly focus on the additional predictors that should be considered in addition to the existing list of parameters that might be helpful in the future improvement of the regression analysis.  
                  So the first factor that I think we should consider is the pesticide usage data.  As I've mentioned yesterday, pesticide usage is perhaps the most important variable in determining pesticide concentration in surface water.  I understand that obtaining the usage data for areas outside of California might be difficult.  I think these variables might improve the regression analysis.  EPA could potentially use, as Dr. Nowell mentioned yesterday, the USDF accumulative county level usage data.  And in addition, there's an ongoing effort within EPA to generate additional nationwide pesticide usage data.  In EPA's environmental modeling in public meeting, EMCM meeting, earlier this year in October, I believe, 2019, the meeting featured discussions on how to better obtain pesticide usage data for the entire nation.  So there's some promise there.  
                  The second point is that precipitation is another important factor, as Dr. Rogers had just indicated in his comments.  And I agree with EPA that further analysis should be used -- the annual precipitation data instead of 30-year average.  The 30-year average data really doesn't consider the year to year variability.  And as you all know, due to the climate change, precipitation really varies a lot from year to year.  
                  So in addition to perception, in arid and semi-arid regions, irrigation should also be considered.  One potential approach is to add the accumulated irrigation water to precipitation as total water increases.  Sorry.  My Airpod just ran out of power.  Can you guys here my right now? 
                  DR. ROBERT CHAPIN:  Variably.  Try again. 
                  DR. XUYANG ZHANG:  This is better.  Okay.  And my third comment is that I think understanding the nature of sampling bias factor is really important to the regression analysis.  EPA's approach of separating the sampling bias factor as a ratio of the maximum from the full dataset and over fifth percentile of the component bootstrap -- this concept is kind of new to me.  But I'm trying to comprehend to see what that really means in the physical sense.  In my opinion -- and correct me if I'm wrong.  But in my opinion, the sampling bias reflects the monitoring data's ability in capturing the peak and the variabilities of pesticide concentration within a year.  
                  So therefore, in theory, sampling bias factor likely is related to the variability of the two-pesticide concentration within a year for a particular site.  So in other words, the more variable the pesticide concentration is, the more difficult to capture the peak.  As shown in the white paper, flashing systems, such as the Honey Creek and Rock Creek, tend to have higher sampling bias factor.  So this is also in line with the white paper's finding that terrestrial field dissipation half-life is the most significant factor explaining parameters along with all pesticide properties that have been explored.  
                  So following this idea, I think we could focus on exploring the parameters that really impact the variability of pesticide confrontation within a watershed, such as the timing of use, whether there's a constant inflow over the year or it's a sporadic use and also the flow rate, which really determines the dynamic of the system, and the variability within precipitation within a year, like Dr. Rogers had mentioned.  Like the rainfall pattern and intensity should be considered, and the types of small rainfall, intermediate, or large rainfall event could potentially be an important factor.  And also pesticide degradation should also be considered.  
                  So my last comment is that I agree with EPA that the timing between the independent predicting variables and the sampling bias factors should be in alignment.  So for future work, you should try to make sure that the timeframe and the predicting variables should align as best as possible with the sampling bias factors that you're calculating.  And with that, that's all I have for now.  Thank you. 
                  DR. ROBERT CHAPIN:  Do you guys have any questions you want to ask of Dr. Zhang, given the communication challenges?  Okay.  All right.  Thank you very much.  All right.  Looking around the room to other panelists.  Dr. Portier?  No new models after 2:00, right? 
                  DR. KENNETH PORTIER:  No, but the coffee's still working.  And I can't allow EPA to get away from a panel without suggesting a research project.  So long-term -- what I put in here, a long-term, expensive, time consuming, low probability of satisfactory return project.  And I was thinking back to the 2012 presentation to the panel by Syngenta of the GIS analysis that they did of watersheds across the Midwest and atrazine concentrations.  Some of you remember that.  I think it's 2012.  It might have been '11, '12.  But you remember that massive study they did.  
                  And I'm sitting there thinking one issue that the geospatial aspect of the relationship of land use to these monitoring sites is something we'd really like to address.  And Dr. Rogers tried to get at that.  And I was sitting there thinking almost all of these characteristics come out of a GIS database.  So being able to, in a sense, look at the high concentration -- high intensity monitoring sites and kind of slowly build your characteristics, your attributes out from that site to see if things closer in correlate more with the SBFs.  
                  And at what point do you get the watershed level in the R squares of 0.2?  It may be that when you're looking more closely your R squares might be much, much higher, 0.7, 0.8.  You get excited.  That kind of tells you that SBFs are much locally impacted rather than globally impacted.  But again, I say this is expensive, time consuming, with low probability of satisfactory return.  So I put it in the long-term research category, which I realize this part of EPA doesn't do.  But you have friends in other places that you can suggest.  Or you can punt it back to Syngenta and say, "Why don't you do this?  It's your data anyway." 
                  DR. ROBERT CHAPIN:  All right.  Other comments, answers to this question from the panel?  All right.  I'm smelling the charred smoke of fried neurons.  Let me talk to my EPA colleagues and ask if there are any clarifying questions you have of us. 
                  DR. ROCHELLE BOHATY:  Nope.  We're good.  Thank you. 
                  DR. ROBERT CHAPIN:  Okay.  I'm going to declare victory for today.  And we will do you guys a favor and let everybody who's got to wrestle with the metro start on that a little early.  So we will close off the meeting for today, and it's going to be done.  The panel, we need to have some discussions with Dr. Gibson -- with Ms. Gibson about changing travel plans, et cetera.  So don't everybody go anywhere.  But I think our EPA friends are free at last to go deal with other stuff.  So we will see you guys back here at 9:00 in the morning.  
                  
                        (MEETING ADJOURNED FOR THE DAY)

                           OPENING OF MEETING - DAY 3
                  
                  MS. TAMUE GIBSON:  All right.  Good morning, everyone.  Once again, my name is Tamue Gibson.  I am the DFO for this session, which is the Quantitative Use of Surface Water Monitoring Data in Pesticide Drinking Water Assessments.  
                  I would like to thank members of the public as well as the panel for their participation for this session.  And at this time, I would like to turn it over to our chair, Dr. Chapin.
                  
                             PREVIOUS DAY FOLLOW-UP
                  
                  DR. ROBERT CHAPIN:  Good morning, everyone.  And welcome to our last day.  I'm looking forward to a productive and expeditious morning, but we want to make sure we focus on quality of response of questions.  
                  And we'll remember that our hosts are looking for our advice on how they can improve things and also some sort of prioritization.  so when we suggest you ought to do this or this and this, it would be helpful if we can give them short-term, immediate got-to-dos, mid-term, and then longer-term sort of nice-to-haves.  
                  So, we're going to start off the morning by coming back to Question 3(a).  And for better or worse, we've given Dr. Berrocal the whole evening to come up with a War and Peace.  And so, we'll start with her just after we hear from Dr. Baffaut.  Dr. Baffaut, do you have a question?
                  DR. CLAIRE BAFFAUT:  Well, I have something to correct on one, like one little detail of what I said yesterday.
                  DR. ROBERT CHAPIN:  Okay, so, she's got something to correct from yesterday.
                  DR. CLAIRE BAFFAUT:  Um-hmm.  Yeah.
                  DR. ROBERT CHAPIN:  So, yeah, what?  
                  DR. CLAIRE BAFFAUT:  This is fine.  She can go.
                  DR. ROBERT CHAPIN:  This is fine?  Okay.  All right.  So, after you, then you.  Yes?  Is that okay?  Okay.  Dr. Berrocal.
                  DR. VERONICA BERROCAL:  Okay.  So I am fresh, and I'm also inspired by my neighbor, Ken, so I have prepared some slides.  It's three slides, on which two of you guys from EPA have prepared, so I stole your slides.  
                  But I've been trying to think about this Question 3(a) and regression.  So, I'm going to start my mark by going over a little bit, trying to think what are you trying to do with this regression approach?  What is it that you are regressing?  What is the goal of linear regression, to give you, then, some suggestion.
                  Okay.  I want to start by saying that I believe that evaluating whether there are site specific and potentially time specific variables that explain SBFs, which are Sampling Bias Factors, and allow to predict the value of sampling bias factors of sites that don't have enough data, is a work endeavor; and something that I personally would suggest EPA continues to pursue despite the non-favorable results that have been reported in the White Paper.
                  Secondly, I think that, in order to make a definitive conclusion as for whether the effort performed so far provide negative results, it is important to evaluate whether the methods used were appropriate or not.  
                  First, I would like to remember, what is the goal of a linear regression and what are the necessary assumption?  This is a slide that shows the linear regression.  Linear regression and statistical modeling approach to describe how the expected value of a random variable changes linearly as a function of the predictor.  
                  So, the expected value would be the center of the distribution of the random variable.  And the underlying assumption is that the outcome variable follows a normal distribution, so it's like a sematic distribution.  And regardless of the value of the covariate, the spread of that distribution remains constant.  
                  So, now that we have arrived at the necessary assumption for linear regression, I think that we need to tie this to what is the problem that EPA is trying to address with the sampling bias factor?
                  Can you go to the next slide?  So in the next slide -- oh, no, you can move without the -- sorry.  This is from when I teach and try to keep students engaged.  Okay.  
                  So, this is the slide from your presentation, and this shows the sampling bias factor for three sites over time.  As you can see, basically you have different sampling bias factor, 50 of them for each site, for each pesticide and for each year.  And you summarize the sampling bias factor by taking the median.
                  So, now we have -- if you look at each year of the distribution, it's not really normal anymore.  And you are trying to look at the median of the skewed distribution.  You will have as many SBFs for every year, and then you take the median of all these SBFs.  And you are trying to regress the annual median of the SBFs on site-specific covariates.   
                  Because of all this skewness in the distribution, I think that summarizing all of this variability over year and the skewness of the distribution by just taking one value, one SBF for site, might not be the best approach.  I think this was also written somewhere in the White Paper; that maybe it would be more appropriate to develop a regression model for each year, individual SBFs separately, rather than fitting a regression to the median of all these medians.  
                  And so, because the goal then would be, for each year to basically try to establish a relationship between the horizontal line -- which is the median of the box plot -- for each year, then, since we are trying to predict the median instead of a linear regression that is trying to predict the center of a sematic distribution, it would be more appropriate to use a median regression type of approach.    
                  So, now, I've made this point that I think we should use SBFs for each single year, median regression other than linear regression.  And then the next point I want to make is, what is it that EPA is trying to understand as outcome variables?  So, what is the Y variable in your regression model?  Can you go back to the first slide with the linear regression?
                  When we perform linear regression, what we're trying to understand is how the center of the distribution is related to certain covariates.  Now, what is the SBF factor?  Is it a center of the distribution?  It is a multiplying factor that is used to understand where would be the upper prediction limits for summary statistics of the pesticide concentration data.  
                  So, if you're assuming for now that here Y is the pesticide concentration data, what you're trying really to understand is, what is this factor that you need to apply to your summary statistic to get the upper limit of the 90 percent prediction interval?  
                  So, I am not sure that -- because we are not trying to understand the expected concentration data, but we're trying to understand a factor that characterizes the variability in the pesticide concentration data, the question that EPA should answer is, what characteristics of this particular sampling site, for this particular year, can explain the variability in the pesticide concentration data?  And this is a different question than asking, what is affecting the pesticide concentration data?  
                  So, that question is what the SEAWAVE-QEX model is addressing by putting a streamflow -- a normalizing streamflow.  But here, you're trying to predict a factor, a multiplicative factor.  So, I am not sure that the value of the water flow or accumulated total, or the average precipitation between May and September, the right exploratory variable to put in the regression -- whether it's the median or the linear regression -- for the SBF.  
                  I think that subject matter expert in this field can help you identify what are the variables that are necessary for that factor.  But I don't believe that they're the same variables that you're thinking affect pesticide concentration data.  
                  I also want to point out something that our colleague in California mentioned.  I don't know if this was registered by everybody.  She made a good point, that I'm not sure the climate variables that are averaged over 30 years should be included in the list of potential variable because these are constant over the course of the time.  And these SBFs vary over time, so I'm not sure that they would be useful to predict the median SBF for a given pesticide in any year.
                  So, to me, this is inherently a question of space and prediction that could be addressed by doing the analyses separately for every year and tackling the problem as the problem of predicting the median SBF over space.  
                  So, I would look into spatial and statistical modeling approaches.  So, spatial regressions where the predictors are some of the predictors that can influence the degree of viability in pesticide concentration.  And these predictors could be identified through some of the methods that you've used, like stepwise regression or any other method.  Or you could also use, as you mentioned in the White report, PCA.  Although, as Dr. Portier said yesterday, using PCAs might come at the loss of interpretability.  
                  As for the type of spatial analyses to perform, there are many routes that can be taken.  And I can provide references for example of spatial analysis.  When I think about data that is collected over space, as a spatial statistician, I always think, why don't we make a map?  
                  So, I think it would be nice to make a map of these SBFs for every year and see whether the -- we can already visual -- by eyes -- understand whether there is spatial variability.  From an exploratory point of view, you could make diagrams to see if there is, in fact, a degree of spatial correlation in the SBFs.  
                  And finally, my last comment is for the fate properties.  While it was reassuring to see that for different values of the fate properties there are different average SBFs, I did not understand why EPA performed the analyses the way they did.  From what I can tell, fate properties are properties of a pesticide that do not vary in time.  
                  So, the fact that the analysis was done so that sites were restricted -- the analysis was only restricted to sites that has SBF available for overlapping years; it's just very hard for me to understand, conceptually, given that the fate properties don't change over time.  So, I would just suggest that the analysis was performed using all of the SBFs for all the years.  
                  So, in conclusion, I've given a lot of suggestions, so I hope that you can understand which ones are most immediate or more intermediate.  I do think that this is an effort worth pursuing; and the lack of good results so far should not demotivate EPA from continuing to pursue this direction of understanding what site-specific -- time-specific variable, and also what's the pesticide-specific fate properties that can be used to predict yearly SBFs aside for the insufficient data?
                  DR. ROBERT CHAPIN:  Okay.  This is Bob Chapin.  I'm going to jump in and say, could you please specify which ones you want them to do first?
                  DR. VERONICA BERROCAL:  So, the easiest one would be first to just do a map; like this one can be done easily.  The second one would be to maybe change some of the predictors that you have right now -- your variable and your linear regression.  And just think about what variables -- explain the variability, what variables explain pesticide concentration data.  And I would do the analysis separately by years.  So, these are the most immediate ones.
                  DR. ROBERT CHAPIN:  Questions from our friends?  Let's see.  Dr. Hartless is writing furiously.  I'll give you a moment or two to capture that stuff and then figure out what kind of questions you want to ask for clarification.
                  DR. CHRISTINE HARTLESS:  This is Christine Hartless.  I don't think I have any further clarifications at this point.  You've given me a very good path forward.  I'll be happy when we get the notes so that I'll be able to follow in all the details.  Thank you very much.
                  DR. ROBERT CHAPIN:  Nicely done.  Okay.  Dr. Baffaut?
                  DR. CLAIRE BAFFAUT:  This is Claire Baffaut.  I just wanted to correct something I said yesterday.  It was in response to Question 2(c), and I listed a bunch of variables that I think might be useful.  And then, at the end, I talked about some vulnerability index that have proven useful in the literature.  And I said that the variable needed to be considered as a multiplicative form instead of additive; and that's not exactly right, I meant together.  
                  Because, actually, one of the vulnerability index that I gave as an example, the soil vulnerability index, was developed using the same methods that Dr. Portier described yesterday, the decision tree.  And so, it's not a multiplication; it's just factors considered together.  That's all.
                  DR. ROBERT CHAPIN:  Okay.  Any other loose ends from yesterday?  Okay.  All right.
                  DR. XUYANG ZHANG:  Hi.  This is Xuyang Zhang.
                  DR. ROBERT CHAPIN:  Go ahead.
                  DR. XUYANG ZHANG:  Hello.  Good morning.  I have just a comment regarding Dr. Berrocal's comment.  So, first of all, I agreed with her understanding that the sampling bias factor really captures the variability in pesticide concentration.  So, we should really focus on the predictors that impact the variability in pesticide concentration, rather than the factors that are affecting pesticide concentration.  
                  I think I also made that point in my comment yesterday.  But I disagree that we should completely disregard the pesticide properties, specifically, the field dissipation half-life.  I think the half-life of pesticides does -- although in theory, doesn't change in partial -- because we used one -- probably just one value in the equation; but in reality, it does impact the pesticide dissipation over time.  So, I think that's an important variable and we should not toss that out in the equation.
                  DR. VERONICA BERROCAL:  This is Dr. Berrocal.  I did not say that that has to be tossed out.  My remark was about the fact that the analysis for the fate properties was done by restricting the dataset to just look at the sites that had SBFs in the same time period.  And I didn't understand why that was done, given that the fate properties of a pesticide remain constant.  
                  It's an inherent property of the pesticide.  So, you don't need to have data that is on the same years if you think that that's important to explain the SBFs.  So, I was not at all saying you should remove that from the list of equations -- from the list of covariates in the equation.  I just wanted to make sure that the message was clearly received.
                  DR. XUYANG ZHANG:  Yeah.  Cool.
                  DR. CHRISTINE HARTLESS:  This is Christine Hartless.  I did want to make one clarification with regards to the regression analysis that was done using fate properties.  That was the -- we did restrict the number of chemicals or the number of sites that were used in that.  And it was restricted to those sites that had multiple chemicals measured on them.  It wasn't restricted strictly just by year.  But I think your point is still well taken, and it is an applicable point.
                  DR. ROBERT CHAPIN:  Dr. Portier?
                  DR. KENNETH PORTIER:  I just wanted to add something to my comments on 2(b) where we talked about the strengths and weaknesses of using the median of the median, the median across realizations, and the median across years to estimate the sampling bias factors.  
                  Estimating across years means we're losing year-to-year variability.  And when she threw up that graph and I looked at it, I realized there's probably a lot of information in year-to-year variability; not just in your estimated SBFs, but in your site characteristics that you haven't really looked at.  
                  I'm going to just make a little note here that kind of as an intermediate term task.  It might be useful to look at, at least, first, how much the site characteristics -- some of those site characteristics change from year to year.  And how much information you have on year-to-year variability and site characteristics.  
                  You know?  I looked through the variable set and I thought, well, some of these -- and I remember yesterday, somebody mentioned -- some of these may change significantly from year to year.  Crop type, crop coverage. 
                  
                  I remember talking with Tom yesterday, and he pointed out that not only is crop type changed, but the chemicals used on the crop will change.  So, while you're using atrazine on corn one year, if they don't plant corn the next year, you're not likely to see atrazine on that site, which could impact a lot of that year-to-year variability.  
                  So, it's going to be just a small comment with a recommendation for an intermediate term.  At least, in the White Paper, say, we looked at it, we can't do it.  
                  DR. ROBERT CHAPIN:  Okay.  We ready to move to Thursday?  
                  DR. THOMAS POTTER:  Are we opening the floor for comments by non-listed discussants?  Is that where we're at?
                  DR. ROBERT CHAPIN:  No.  I thought we did that yesterday for 3(a).  Do have some more comments on 3(a)?
                  DR. THOMAS POTTER:  I have a comment on 3(a).
                  DR. ROBERT CHAPIN:  Go for it.
                  DR. THOMAS POTTER:  Okay.  I want to say, perhaps we're asking too much of the data in the regression analysis to provide insight.  We only have four chemicals, so there's a very limited universe of information to work with.  Certainly, I think we're all in agreement to that.  
                  I would also like to, at least here, emphasize that used characteristics, i.e. the type of the compound, when and where it's applied probably have as much, or even sometimes more, impact on measured concentrations than the actual fate properties.  So, you may have a similar fate property, but you have a pre-emergent herbicide that's applied to bare soil.  And you have an insecticide like carbaryl, which is applied to crop canopies.  
                  So, there's wide differences in the timing of application, where the chemicals are applied.  And that can dramatically affect how much leaves the field.  
                  So, again, I think we're maybe asking too much of the data here in terms of expecting the regressions to provide the ultimate answer.  I would suggest that, rather than lumping the SBFs that are computed in Tier 3 -- for Tier 3 analysis -- it may be better to just do it on a basis of compound class.
                  So, if you're looking at a pre-emergent herbicide, okay, let's look at the pre-emergent herbicide data that we have and use that.  The same would hold with the insecticides, et cetera.
                  We've got two insecticides, and they may be used similarly, so perhaps some lumping can be done there.  But I really think it's important not to lose sight of -- you know, how the chemicals are used is critically important here.  
                  DR. ROBERT CHAPIN:  Ian.
                  DR. IAN KENNEDY:  This is Ian Kennedy.  I just wanted to quickly state that I think the chemical properties themselves are not strictly properties of the chemical, but of the chemical and the environment.  And as such, they vary substantially over space.
                  DR. ROBERT CHAPIN:  All right.  Anything else to finish off 3(a)?  Are we good?  Responses or questions from our EPA friends?
                  DR. ROCHELLE BOHATY:  I think we're good.  Thank you.
                  DR. ROBERT CHAPIN:  Let's start our Thursday morning agenda.  We'll move to Question 3(b) and ask Dr. Bohaty to read the question for us.
                  
                              CHARGE QUESTION 3(b)
                  
                  DR. ROCHELLE BOHATY:  All right.  Good morning.  Rochelle Bohaty here.  Question 3(b): please comment on the weight-of-evidence approach to determine a spatial relevancy of monitoring sites to source drinking water.  Are there additional factors EPA should consider in this approach?  Please discuss the relative importance of these factors considering data availability and quality.
                  DR. ROBERT CHAPIN:  And our first discussant is Dr. Sadd.
                  DR. JAMES SADD:  Jim Sadd.  EPA promoted use of a weight-of-evidence approach is a creative and potentially useful approach to addressing the question of what they term spatial relevancy, providing relevant monitoring data or SBFs to areas lacking them.  Searching for appropriate surrogates and evaluation of the possible utility of nearby monitoring detects and values a concern.  A GPS would be central to this approach for risk assessors, providing both the necessary spatial tools and also the access to a variety of data types.  
                  The White Paper states that the more similar the properties are, the more confidence the assessor has in applying the SBFs; which is why they call it weight of evidence.  In the short term, EPA needs to define these terms, as well as be specific about what this requirement for a site that is spatially relevant is, beyond what it's currently in the White Paper.  
                  The reasonableness or utility of this requirement should be demonstrated either by analysis or a clear rationale.  This determination can be operationalized in a number of ways, such as clear guidance through a decision tree or using -- and I'm assuming that you use ArcGIS for your -- GIS?  Or an ArcGIS builder model that could be used to compare a given watershed to acceptable, SBF watersheds making the process simpler, more standardized and less subject to error from an unwise subjective decision.
                  As an intermediate goal, I suggest EPA ease implementation of this approach by arranging for geospatial data, that risk assessors and others might use, to be easily obtained.  My suggestions include seeing if it might be included in the national atlas; or you could arrange for a download site to ensure that the data is available, updated appropriately, and that assessors nationwide are using the same data.  
                  In addition to the data used in the White Paper example in this section, 5.3, geospatial data that would be useful in this process and is widely available include -- I have a whole list of them.  Do you want me to go through them or can I just -- does it have to be in the spoken record?  I'll just abbreviate them.  
                  Digital elevation models, preferably ones that have been hydrologically corrected.  Watershed boundaries and stream reach information.  The national hydrologic dataset would be good there, not only for conductivity, but calculating stream flow distance.  
                  If you want, you could also do this online using the opentopography.org website; which gives you the data and the computational routines from the San Diego Supercomputer Center.  And that it could just be done as you need it for the area you need it.  Agricultural use and crop data, some states might vary in terms of the quality and completeness of that information.  
                  Something that would be very useful, I think, would be land cover from the National Land Cover Dataset.  It's the definitive nationally consistent dataset for land cover, and it also covers land cover change.  It goes from 2001 to 2016.  It's updated every five years, kind of; but it gives you an opportunity to look at land cover change for the period covered by the data.  
                  Certainly, precipitation and accumulated precipitation, which would probably be raster layers, which might complicate things a little bit for people that don't use ArcGIS a lot.  
                  There's soils information.  SSURGO and STATSGO are nationally consistent.  I'm not sure if they have the appropriate descriptive and physical characteristics, that John Rodgers mentioned are important, if you're looking at them in terms of absorption and infiltration.  
                  Pesticide application is available for some states.  And census block level information for estimating potentially impacted populations, community water system boundaries is available from state to state.  And also the location of groundwater drinking wells, which is available, I believe, from the USGS.  And this could be used to estimate populations that might not be exposed to surface waters but might be in the area.  
                  Some of these data vary among states and among watersheds in terms of variety and data quality and completeness, spatial resolution, and other factors.  But EPA should not place national consistency above providing the best data available wherever it is available.  This is a weakness of EPA's current screening tool for air pollution, EJSCREEN.  Although EJSCREEN development team might be a useful partner in this effort, because they've done a lot of this already.  
                  Finally, back to the term weight.  Using a GIS allows one to easily apply weights to various factors as appropriate, and that is something that needs to be worked out by you all.  And I know you're thinking about that because of the wording of the charge question.
                  DR. ROBERT CHAPIN:  Great response.  Okay.  Dr. Zhang?
                  DR. XUYANG ZHANG:  Thank you, Chairman.  Yeah, I agree with EPA that the weight-of-evidence approach is a very useful and important aspect to determine the spatial relevancy of monitoring sites that stores drinking water.  I think it's a viable approach, especially when the quantitative relationship between sampling bias factors and watershed characteristics have not been established.  
                  The lines of evidence that EPA is currently considering -- and I have a whole list of it, which are all listed in the White Paper.  I think they all make sense to me.  And I agree with EPA's inclusion of all the listed factors.  Specifically, I have a few suggestions or comments regarding this method.  
                  The first one is, in addition to considering pesticide usage in the watershed where the community drinking water system is located, I think we should also consider the usage, the upstream watershed, which could contribute to the flow in the particular watershed that is adjoined to the drinking water intake point.  So, yeah, instead of looking at immediate area, also consider the upstream watershed.  
                  The second point is that the location of drinking water intakes, being relative to the monitoring site, are very important in the weight-of-evidence approach.  So, I echo Dr. Sadd's comment about the quality of the GIS data.  Having a reliable GIS file, specifically on drinking water intake point and the contributing watershed, are very essential.
                  In one of my previous projects, on chlorpyrifos detections and their relevance to drinking water sources, I tried to obtain a good GIS dataset on drinking water intake.  Of course, it's in California.  
                  But I found that the government dataset that I was able to find tended to have large uncertainties.  And eventually, I was able to find a good dataset from the nature conservancy in which they made improvements based on the available dataset from this data in federal agency.  So, I just wanted to bring your attention to the quality of the GIS data on drinking water intake sites; as this is the most important dataset in the weight-of-evidence analysis.  
                  My third point is that, although it's not a quantitative analysis, but I think at least we can use some of the lines of evidence more quantitatively.  For example, the distance of the sample size in relative to the drinking water intake point.  You can further quantify the criteria.  
                  It doesn't have to be -- like, for example, the ratio of the travel time in relative to the half-life of the pesticide, we can define for example, one or two if it's above.  If it's above three, for example, you don't need to consider this factor.  So, I'd recommend, for the -- quantify that using a criteria rather than just say if it's smaller than that or greater than that.
                  My last comment is that, regarding the National Hydrography Dataset, watershed boundary may not be accurately captured, like, in areas of low release.  For example, the Central Valley of California.  And also areas with lots of engineering that really impacted the hydrology of the watershed.  
                  In those areas, I recommend that -- this is quite common in agricultural areas.  So I recommend that you would take a closer look at the actual watershed boundary and examine the accuracy of the delineation by the National Hydrography Dataset.  That's it.  Thank you.
                  DR. ROBERT CHAPIN:  Wonderful.  Thank you.  All right, Dr. Rogers?
                  DR. JOHN ROGERS:  Good morning.  I'd like to begin by thanking you for what you're doing in terms of protecting drinking water.  It's very important, and I don't know that our appreciation has been expressed.  That may be a policy statement, but it's not meant as one.  Interesting thing came to mind while folks were talking this morning -- do you have any idea where the water on this table comes from?  
                  I got involved in that not terribly long ago.  And you won't find it on GIS and you won't find it on databases.  It's an interesting situation.  You won't find it on maps.  You won't find the drinking water intake pinned down either.  So, it's a security question.  And I learned a lot by getting involved in that particular situation.  
                  But to my points, Dr. Sadd and Dr. Zhang have done a great job already.  I don't need to repeat things they've said, so I'm going right to the point.  
                  The weight-of-evidence approach could be very useful for ranking the available information and monitoring data.  And then, the approach needs to be carefully considered and guided by a well-documented protocol.  And I would consider that a fairly urgent need so that all stakeholders can be clear regarding how the assessment will progress, given the weight of evidence.  
                  And the data and the modeling results will be evaluated, and that will be well known to folks in terms of how they would be weighted and utilized.  I mentioned the other day that I would consider measurements of concentrations of pesticides in drinking water from a consumer's tap as, perhaps, the most persuasive evidence.  
                  Then, if we're weighting evidence, I think what you will find is that the information regarding the proximity of the monitoring stations to the water intake are going to provide the most useful and accurate information.  So, this approach can provide a mechanism by which important decision-making criteria for risk assessment can be formalized and communicated to the public.  That's my comments.  
                  DR. ROBERT CHAPIN:  Thank you very much.  Mr. Councell.
                  MR. TERRY COUNCELL:  Dr. Sadd gave a much more comprehensive list than I had.  I agree with EPA's list, but please take Dr. Sadd's list as well.  I think he did a really good job of coming up with those and gave sources of data as well.  
                  I like on page 86 of the White Paper, it says, the assessor will need to use judgment to weigh these factors.  And again, I'd like to say that, because our country spans a whole continent, there are things in the Northeast that aren't going to apply, or weighted, or affect pesticides nearly as much as in the Southwest.  It's a difficult job to put weighting factors on these, so you guys have your work cut out for you on that.  
                  So, at the moment, you're using it qualitatively, I completely understand that.  It's nice to work towards getting that quantitatively.  I wish you the best of luck in that.  Thank you.
                  DR. ROBERT CHAPIN:  And Dr. Green.
                  DR. TIMOTHY GREEN:  Yes, thank you to my colleagues who just presented a lot of good details.  The question was posed in two parts, one very general and one more specific.  So, I'll respond with quite a bit of overlap.  
                  The weight of evidence appears to be a smart way of incorporating process understanding and spatial context, which is not seen in the rest of the regression methods.  The results -- well, potential downsides are complexity and subjectivity of the analyses.  So, results may not always be repeatable or transferrable.  
                  Despite those issues, I'm not saying it's not worth doing; I think it's, as I said, a smart way to go.  I'll repeat, in the White Paper, Section 5.3.4, pulling it all together.  
                  It states, the assessor will need to use their judgment to weight the factors that give them confidence that the monitoring sites represent concentrations that may be anticipated at the drinking water intake, and those factors that make this conclusion -- it says less certain.  I'm wondering if you meant less uncertain or more certain.  So, just maybe check that statement.  I think the goal is to be more certain about what our understanding is from the data.  
                  The short-term, I guess, immediate concern is that these criteria and SOPs need to be clarified because of this subjectivity in the approach.  So, that's the broader weight-of-evidence.  Jim Sadd gave a nice list of available resources, so the next part I'm speaking to is more specific based on my biased experience.  
                  So, yes, auxiliary spatial data are becoming more broadly available and accessible.  In particular, web services are currently available to provide high resolution land use, irrigated area -- which I think is neglected -- field level crop rotation and associated management operations for user-defined areas or polygons, if you wish.  
                  On the land use and agricultural management practices web service -- I'll give a reference; we call it LAMPS.  It was published by Kipka et al in 2016.  It uses annual crop type derived from landsat images.  These are provided by USDA NASS in the Cropland Data Layer.  The difference being that this tool automates the retrieval.  
                  Also provides irrigated occurrence.  Yes/no, whether it's irrigated or not at different time slices from the USGS irrigation map, which is derived from MODIS data, which the raw data is 250 meters.  A little note on that, they are not keeping on their 5-year update cycle.  So, maybe the demand for it would help.  But I think that's an important product.  I think it was last updated in 2012.  
                  And then, along with those things, LAMPS produces management operations, dates of planting, harvest and tillage with associated depth and intensity, from the USDA NRCS Land Management Operations Database, or called LMOD.  This was published in a proceedings paper by David et al in 2014, and I'll provide references.  
                  So, LMOD information is relevant at the field scale if appropriate field boundaries are provided using representative management for the spatial Conservation Management Zone, CMZ.  So, one can automate quantification in time and space of management.  And the new component may relate these data -- this would be future work, probably long-term -- to the space-time distribution of pesticide application for an area of interest.  
                  A course of resolution.  So, going now from field -- if you have field scale, you can do crop rotations and come up with more management details, defaults of those CMZs.  LAMPS may be used to determine annual series of the fraction of land area within each area of interest.  
                  For example -- another reference -- at the county level, a recent study answered the question, where is the USA corn belt and how is it changing?  Meaning, changing from year to year.  
                  So, in the short term, LAMPS could be explored as an option to enhance the weight-of-evidence approach.  In the midterm, a custom web interface and new analysis tools could be developed.  A LAMPS source code is available in an open-source repository.  You can see Kipka et al for that.  
                  And the developers offer a Jupyter -- it should be with a y, mine spelt auto-corrected -- Jupyter notebook just to demo its deployment.  Because it is a web service but not a web interface, if that makes sense.  
                  So, that's an example.  There will be probably other tools similar to this that could be deployed to better characterize the space-time distribution of management that's relevant to the problem.  Thank you.
                  DR. ROBERT CHAPIN:  Both my neurons are just kind of spinning with all these online databases.  I'm thinking, man, somebody's going to apply AI to this stuff.  It's going to happen.  It's going to be so cool.  Sorry.  
                  Do other panelists have comments about the weight-of-evidence approach and the relative importance of the factors?  Dr. Sadd.
                  DR. JAMES SADD:  This is Jim Sadd.  I just wanted to follow up on Terry Councell's point about weighting, which I think is a good one.  That's a really difficult part of this job.  And Terry's right that for different areas, and for different parts of the country, you might want to weight differently or even discount some of those variables.  So, this is a problem that I've seen in other efforts of this type.  
                  A couple of things to remember.  And I think it's obvious that if you don't weight at all, you're implicitly weighting everything at the same level, which may not be what you want to do.  
                  And an intermediate step, as you're thinking about it, might be to rank things.  And then, that way you're giving some greater weight and lesser weight to certain factors, or groups of factors, without implying greater quantitative meaning than you have.  Because, in reality, no one really knows the weights of these factors.
                  DR. ROBERT CHAPIN:  Other comments?  Discussions?  Inputs?
                  DR. XUYANG ZHANG:  Yes, I second Dr. Sadd's advice on ranking things.  So, I think, maybe, for illustration purpose and for documentation, it'd be good to develop a decision tree graph to both help streamline the process and also the transparency of the decision.  Thank you.
                  DR. ROBERT CHAPIN:  Cool.  Thank you.  Other comments?  Discussions?  Yup.
                  DR. LISA NOWELL:  Lisa Nowell.  I just wanted to mention one more database.  This is a USGS database that came out in 2015, by James Falcone; and it's called NWALT, abbreviated NWALT.  It's A Conterminous Wall-to-Wall Anthropogenic Land Use Trends.  That's a mouth full.  
                  But one thing it does that's handy, if it's useful, is its goal is to try to bring information together over the last 20 years on land use and make it consistent.  So, it's based heavily on NLCD from 2001 to 2011.  It only goes through 2012.  
                  But it attempts to treat the different years equivalently, because things change over time.  The methods have changed over time in these databases.  It's also consistent with -- incorporates consensus derived information and I think the focus of the project is to capture trends.  But if that's useful, it's kind of unique.
                  DR. ROBERT CHAPIN:  Anything else?  Fun stuff.  Okay.  I think we're done with Question 3(b).  Comments, clarifications from our friends at the agency?
                  DR. ROCHELLE BOHATY:  We're good.  Thank you.
                  DR. ROBERT CHAPIN:  All right.  Let's move to Question 4.
                  
                               CHARGE QUESTION 4
                                        
                  
                  DR. ROCHELLE BOHATY:  Okay.  Question 4:  In general, EPA relies primarily on model-estimated pesticide concentrations for drinking water assessments with limited use of surface water monitoring data for most pesticides.  To improve transparency and support communication with stakeholders, the Office of Pesticide Programs developed a drinking water framework that describes the Office of Pesticide Program's longstanding peer-reviewed tiered approach to drinking water assessments.  
                  The Drinking Water Framework describes the continuum of approaches from highly conservative and simple to highly refined, complex temporal and spatial assessments.  EPA applied the Drinking Water Framework to two case studies that represent pesticides with available surface water monitoring data, toxicity endpoints typically evaluated by the Office of Pesticide Programs (that is, a 1-day acute and a 365-day average for chronic and a 30-year average for cancer), different use patterns and different environmental fate and transport properties.
                  
                              CHARGE QUESTION 4(a)
                    
                  4(a): Please comment on the clarity and organization of drinking water framework.
                  DR. ROBERT CHAPIN:  Single short sentence.  Please comment on the clarity -- okay.  Dr. Kennedy.
                  DR. IAN KENNEDY:  Okay.  One thing is that the White Paper often discusses 4 and 21-day averaging periods; and so it's unclear what those are used for and why you would need to develop processes to deal with them.  Overall, I found the framework generally very clear and well organized.  But of course, I have a few comments on things. 
                  One is that the Tier 3 and 4 assessments used typical use rates for modeling.  And of course, any monitoring data is going to include the use rate that was used during the period of monitoring.  And if you use a lower than labeled rate, it's not really a refinement as much as sort of a relaxation.  It's probably good for saying that, well, no one is being harmed by this.  But you're not really making things easier in the sense that, well, you're just using one number in place of another number.  
                  And by evaluating a lower than labeled rate, it weakens the claim that the pesticide is safe when used as directed, because used as directed is higher than what you evaluated it at.  And it's not clear from the framework if there's any adjustment to compensate for that; for example, lowering the rate on the label to match what is actually used, which would then strengthen that claim again.  
                  The modeling, of course, also uses only a single value for each fate characteristic and these can vary substantially.  I know that they are generally chosen to be fairly conservative values.  And that's probably one reason that the modeling often is substantially higher than the monitoring.  So, there's probably some refinements that can be made to all sorts of things regarding modeling, I guess.  
                  I wrote down a little more here, and maybe I won't go over it all.  But the process that is used, uses a very slow rate from a rate that generally declines with time during the study.  The study is not really capable of explaining why it is doing that, so that's why we use the more conservative value there.
                  But again, it's probably a very conservative value.  And if there's some way of looking at refining that, I think that EPA should do that; although that is not necessarily a short-term goal at this point.  
                  So, if you could refine the modeling by getting more accurate fate data, or other means, it would -- first of all, it would apply to PWC use as well as SAM use.  Second of all, it might mean that you don't necessarily have to go to Tier 3 or 4 as often; and would lessen the need to use things like SEAWAVE-QEX.  
                  Just on one specific thing, there's mention made in the framework on adjusting for temperature; but there's no kind of reference for how that's -- I mean, there's a reference at the back, and the references is for how it's done, but there's no link to that reference from the place where it's needed.  
                  And finally, the SOP for SEAWAVE-QEX that's included in our package is not referenced in the framework.  So, that probably should be in there.  Yeah, that's it.
                  DR. ROBERT CHAPIN:  All right.  Excellent.  Thank you.  Dr. Potter.
                  DR. THOMAS POTTER:  Okay.  Well, Dr. Kennedy hit exactly on my first comment about the framework.  But first, I want to say I thought it was clear.  It's appropriately tiered with most of the conservative assessments being conducted at Tier 1 and Tier 2.  I think it's in alignment with work that's done internationally, whether in the EU or elsewhere in the world.  So, again, I think it's a powerful tool and I think the agency is to be commended on developing a logical pathway to assess risk and, of course, exposure.  
                  I had a few thoughts that might require further amplification or discussion, so I'll just go through with my list here.  
                  The top of the list is, if, when and where are probabilistic assessments conducted?  It's not clear to me in the framework what decision process guides that decision.  And, again, what the tools are that's used.  
                  I think Dr. Kennedy hit upon an important question, and that is as assessment tiers -- across the assessment tiers, how are use estimates refined?  There appears to be refinement, but again, there needs to be some logical basis for that.  And I think it would be very valuable to discuss that.  If not here, in some other documentation that allows people to gain insight in that area.  
                  The use of terrestrial field dissipation data is not specified that I can see.  It's my understanding that it's largely used to inform rather than any quantitative sense at this point; although we did hear some potential value for using it in the context of SEAWAVE and SBF assessments.  So, perhaps there's a future -- an opportunity for using that data.  
                  And we'll make note that -- you know, for example, in EU, TFD data are now used in assessments.  And I'm assuming you guys are looking at that and scratching your head over it and seeing if it might be something that could be rolled into the framework.  So, I say that would be something important.  
                  I heard Dr. Kennedy say several times the word label; so I think that that's something that, again, needs further reamplification as to, you know, how do you get to the label, for one thing.  And perhaps, it doesn't necessarily belong here, but again, I think there needs to be some discussion as to what role, for example, label restrictions play, and when and where and how they might be developed.  
                  Are they conducted during the risk assessment?  Or are they risk management decisions that are made post-analysis?  But again, I think it will be helpful to put that in the framework, and at least state explicitly where those restrictions may come from.
                  I already made this comment earlier about the Tier 3 and the application of the SBFs for the four compounds that were studied using SEAWAVE-QEX.  Obviously, there's some limitations there.  I would strongly support some kind of weight of -- or judgement on how the chemicals are used, being factored into choosing which values are appropriate to apply to a certain chemical that's being evaluated.  i.e., whether it's an herbicide or an insecticide, et cetera.  I think that has more rational basis to it, from my perspective, on the exposure side than simply lumping and averaging the parameter together.  
                  Again, I'm not clear as to quite how you propose to do that, whether you take an average or a mean across the SBFs.  And again, an explicit statement of that would be very helpful.  
                  Also, with regard to the use of SBFs and PWCs data, how are the outcomes of those two analyses balanced?  I mean, does one take precedence over the other?  In some ways I think it might be less confusing to simply say one works -- this is the path we're taking, this PWC or SBF, and go with it.  And that gets you to Tier 3 rather than adding this additional factor that somehow it needs to be balanced in determining the decision-making process.  Again, I'm not clear on that; perhaps it's embedded somewhere, but it didn't leap out at me.  
                  Regarding Tier 4, I thought that was really powerful.  And I think, you know, of course from a science side and somebody who's done a lot of field investigation over the years, I'd say, yeah, let's just go right to Tier 4; you know, that's the fun part.  But I recognize that's very costly in time for the agency as well as registrants; so it's obviously not something to wish for.  
                  But I will say, about Tier 4, again, I think the Spatial Aquatic Model would encourage you to speed development.  I think it's a very powerful tool.  If not SAM, then some other model that can be used spatially would be appropriate.  And again, there are several out there including the USDA models, in particular, SWAT.  And I'm sure you're deeply familiar with that, perhaps more so than I.  And I think that there really needs to be a ramped-up effort in the modeling area at Tier 4.  
                  I'll say one other final thought along the lines going back to the label assessment and stuff.  Is that -- and again, perhaps this model is applied in Tier 4.  But the APEX model, is a USDA model, which is really a powerful tool for looking at what's going on in the landscape and how conservation practices impact fate transport and ultimately exposure.  There are just tons of examples of how it can be usefully applied.  
                  One thing that's really interesting about APEX is how it's used in the Conservation Effects Assessment Program.  Perhaps you're familiar with it, but it was a combined USDA NRCS effort that is still ongoing.  It started back in the early 2000s.
                  But there are a number of products from that effort basically looking at pesticide fate and transport and how conservation practices can affect that HUC-O2 scale almost across the country, with the exception of the desert areas and whatnot.  
                  All of those assessments have been published online.  I suspect, if you wanted to get base data that went into them in terms of model parametrization or whatever, that you could probably find some relatively easy way to get that through either NRCS or ARS or other partners that were involved.  
                  And did I say one of the really interesting things that I liked about what was done there is that when APEX was applied, they stripped away all of the conservation practices.  And so, the base case is doing farming the worst way.  
                  But again, that may be good from an exposure risk assessment perspective.  And so, again, I think you would gain some useful insight by taking a look at that.
                  DR. ROBERT CHAPIN:  Thank you very much.  Dr. Nowell?
                  DR. LISA NOWELL:  Lisa Nowell.  I thought the document was very straight-forward, clear, and logical.  You started with the regulatory context and explained the conceptual model and the very key concept of the risk cup and the associated drinking water level, the concern, very concisely.  But those are really important pieces of information in terms of communicating to other people what you're doing.  Very fundamental, and I thought that was really well-done.
                  You also addressed problem formulation and scoping.  And then you marched through the tiers successively with a section on each one.  So, I thought it was -- and then closed with some future direction, so I thought it was very well-organized and clear.  And I also liked that you added in the timeline and regulatory time constraints, including the number of assessments that you have to do every year.
                  So, I do think that your document's accomplishing your stated goals, that it will forward consistency and transparency, in deriving estimated pesticide concentrations and also foster cross-division coordination.
                  I guess I have a few suggestions for clarification but nothing too substantive.  I do agree with Dr. Kennedy.  I think it would be good to add the rationale for your exposure durations, which I probably read in the White Paper.  I know I've read them somewhere.  But I think that would be a good place for them.
                  On page 2, there's a reference to SOPs for drinking water assessment.  And you might want to consider attaching those in an appendix.
                  Okay.  On page 23, there's a statement, when multiple residues of concern are identified, alternative model input parameters are considered.  And that's a little bald.  You might want to just add what you mean by multiple residues of concerns, probably degradates principally.  But then clarify what you mean by alternative input parameters.
                  In Section 4, 4.2.1.2.1, called Percent Cropped Area, this comes up a couple of times in the different tiers.  But I'd really like to see you address what you do, if pesticide has significant nonagricultural uses.  I know this is a weakness of the whole setup because there's no quantitative use data.
                  In case study number 1, chem1 had residential use as well as several crop uses.  And in the section in the percent cropped area, you -- or regarding the percent cropped area, in case study number 1, there was a statement, "PCA adjustment factors of 1 were applied to estimated concentrations, because chem1 is used in both residential and agricultural areas.  And this is one way in which the residential use patterns are accounted for in modeling and assumes the entire watershed can be treated with chem1 at the modeled application rate."
                  This was in one of the case studies; but I thought it might be relevant to include some kind of general statement in the framework itself on how you deal with non-ag use.
                  In case study number 2, in the same section, it said, "Chem2 is used on multiple crops not including turf or nonagricultural uses.  So, a national agricultural percent cropped area adjustment factor of 1 was used to account for the percentage of the watershed that may be treated with chem2."
                  So, the PCA was one in both of these cases; although, I think they were defined differently, one was probably total, and one was ag, I'm not sure.  But this just suggests that there's room for some -- I think you're operating according to principles, and it might be helpful to put some guidance in that framework to indicate what they are.
                  Okay.  There's a section on -- in Tier 2, on page 30, where you say, "The order and the refinements considered is pesticide-specific, and generally determined based on the assessor's best professional judgment considering physical, chemical, environmental fate and transport properties of the pesticide under evaluation, the nature of the data that is available to carry out the refinement, as well as the uses and the difference between the pesticide concentration and the DWLOC.
                  This means the assessor will determine which refinement is the quickest and most effective to implement and will conduct that first and repeat this process until the estimated concentration is below the DWLOC.  I get that this means to be general, but maybe an example might help clarify.
                  And I'm also interested in how this differs from the relaxed exposure assumptions that are part of Tier 3, which also were mentioned as a usage data environmental process, half-lives and environmental condition.
                  In several places, obviously, the framework needs to be updated when you get your White Paper finalized.  Just mention that generally; having to do with SEAWAVE-QEX and elaborating on the weight of evidence approach, and also, I guess the SBFs.
                  And then finally, just one more comment.  Appendix B is a table that summarizes each tier of the drinking water assessment, which I found to be really helpful.  And I wondered if there's a place you could address refinements in each tier in that table.  That's it.
                  DR. ROBERT CHAPIN:  All right.  Thank you.  Next up, Dr. Rogers.
                  DR. JOHN ROGERS:  Good morning again.  I'll be brief because most of what I had to say has been said.
                  We were asked to comment on the clarity and organization, and it was noted that this is a longstanding peer reviewed tiered approach which is sort of inherent in the agency.  This is a cross agency, so that's neat.
                  But it is ever evolving.  You're constantly refining it and it's getting better.  And I just was going to suggest maybe one thing in addition to what others have suggested.
                  That as you refine Tier 3 and Tier 4, and you're looking at SEAWAVE-QEX, I think we agree that it has -- currently, that model has limited use in so-called nonflowing waters that are actually flowing.  We just don't know which way.  And it changes day to day.
                  And unless you've got some ground study, some serious ground study, you don't know which way these so-called static or nonflowing waters are flowing.  At least that's been my education, as I sat there and watched them, and watched my boat moving.  And put dye in the water and actually realized where a drinking water plant's pulling in several million gallons a day, and the water's not coming from where I thought it was coming.  So it's sometimes useful to have that level of information, depending on the tier you're in.
                  So we're talking about a regression model, SEAWAVE-QEX, that's designed to focus on means, medians, middle data, to tell us about the middle.  And we're using SEAWAVE-QEX for infilling and to predict peak and maximum concentrations.
                  I just thought as the framework document evolves, you may need to let people know, you know, where this is fitting in and how it's -- what the plan to use it is.  And then once we've got that maximum concentration, we're going to do essentially a risk assessment, comparing that maximum concentration to a drinking water LOC, if my understanding is okay.  And I've already commented before; is the drinking water LOC also undergoing similar refinement and so on?
                  So, with those parts changing and becoming refined, I think this has to be a living document; it has to keep updating.  And that was my comment.
                  DR. ROBERT CHAPIN:  Dr. Corcoran, any comments on the clarity and the organization of the framework?
                  DR. GEORGE B. CORCORAN:  Thank you, Dr. Chapin.  Allow me to open by noting that the comments that follow are developed through a lens of a toxicologist, who is an engaged stakeholder.  And the comments will not specifically address modeling or water science per say.  So, this is going up to 30,000 to 50,000 feet.
                  Dr. Chapin, for the recommendations that are deemed appropriate by the agency, they should be considered for immediate action, due to their high level of importance.  Okay.  Actually, due to their nominal cost of implementation.
                  I've taken a look at the document from a distance and asked some much broader questions, because those around this table have eminent expertise, modeling, tiering and the logic of approaching drinking water assessment.  So, these are going to be a different set of comments.
                  I asked first of all, after reading the document, about -- let's see if I got it right here -- the voice of the document -- did I -- the -- I'm sorry.  When one gives a presentation, the first question you want to ask is, who is my audience, and how do I communicate with them?  And it wasn't clear to me that when this drinking water assessment document was created, who is the audience?  
                  The intended target audience of this framework document is not defined and appears not to be fixed.  Much communication appears to be directed towards agency personnel.  However, some passages address stakeholders and others; yet other passages address what may be considered members of the general public.
                  For audiences outside of the agency, some communication relies on terms and concepts that require insider knowledge and training and might actually be termed, agency speak.  This limits communication effectiveness for nonagency readers.  Two examples of these terms include ground truth for model refinement and the risk cup which is referred to numerous times.  But for those who aren't trained agency professionals, it sends up a question mark.
                  Another foundational consideration at writing is voice.  And there are recurring instances in which human characteristics are imputed in -- excuse me -- in animate objects and organizations.  This use of anthropomorphism can be distracting and may even imply to some that the framework is not a fully refined document.
                  Introduction.  The introductory orientation of this document is strategic, important, and effective.  It is clearly a strength of the framework in confirming the consistency, transparency and cross division coordination as founding principles.  And is excellent in preparing the reader to place in context, assimilate and understand each subsequent passage in part of this framework.
                  One then encounters Figure 1.1.  The strategy beginning with presentation of an illustration, that attempts to convey the multi-tier approach used by the agency for drinking water assessments, is logical and valuable.  However, the content of Figure 1 is overwhelming, intimidating, and difficult to digest.  Thought may want to be given to revising Figure 1 into more digestible portions or different illustration.
                  The pesticide water calculator comes up early in tier descriptions for lower tier assessments.  Due to the brevity of the framework, there was not an opportunity for it to be clearly presented.  Because it is not expressly defined and exemplar as presented, there is some concern that it may not be as systematic as desired; and it may not be as reproducible across agency assessments as desired.  I know that is not the case because it's a well-developed tool and it's been used for many years.  It's very finely grounded in theory and application.  But it doesn't come across that way in the document.
                  This illustrates the fact that I'm not a water scientist when I talk here about water regions.  A nonagency reader notes that there are 18 HUC-02 regions, on page 28, and 21 different water resources on page 10.
                  These different systems should be explained in how they play different roles in drinking water assessments.  So, you can see, I'm just not a water scientist.  And if you want someone reading this document who is concerned, engaged and wants to know how the agency is protecting their drinking water, that was an issue for me.
                  DWLOC, this abbreviation is defined as the drinking water level comparison in the framework document; then in other documents it is defined as the drinking level water of concern.  This discrepancy should be explained and addressed as appropriate.
                  Acknowledge of space and conceptual constraints of the framework document; but additional explanation should be added to place DWLOC, in the broader context of other parameters employed by the agency in the space of risk assessment.
                  Assessment tiers.  Presentation of modeling methods, modeling parameters, monitoring and exposure characterizations and the next steps, while very parsimonious, are clear and they are fit for purpose for this document.
                  Scenarios.  There are no existing, as I understand it from reading the document and additional preparation materials, there are no existing model scenarios for California and some other regions.
                  This will be unexpected for some readers.  This may be an error on my part in reading the document.  But I believe that's what it said, and I was not expecting to read that.  It would be helpful to discuss why and if any plans are in place to develop scenarios for this region.
                  I've got a list of a few minor technical items that follow.  I only talk about one of them.  And that's the use of the term assessor, used in the context of a specific issue or accession.  And it almost appears that a single assessor is responsible for issue resolution in the final decision.  So, this is just a semantics English issue.
                  I'm certain that's not the case.  I'm certain that there's not one person who's going to drive an assessment and make all the calls in areas which are up to discretion.  So, perhaps this could be addressed through rewording those accessions where the term assessor is used.
                  In final comment, although I've brought up a number of points and issues, the framework document itself is a tour de force.  It's an outstanding document.  I agree with Dr. Rogers that it's a living document.  It should have a version and a date on it that is continually changed as the document is strengthened and incorporates new information.  And I want to commend the agency on a fine job.
                  DR. ROBERT CHAPIN:  Thank you very much.  Cliff, comments on the clarity and organization of the framework?
                  DR. CLIFF WEISEL:  This is Cliff Weisel.  And I want to echo also, I think the document's clear, done well, organized well.  Most of what I was going to say has already been said.  Just a couple of very short issues.
                  There are some things brought up here that are not part of what we've been talking about the last couple of days, and probably because this is going further.   
                  One has to do with the -- there's a section on drinking water treatment, how that might affect -- now, that's not really going to affect your model, but it's something to keep in the back of your mind about when you're looking at your compounds and understanding.  And maybe it's the people that do the LOCs that are really in charge of that.
                  Same thing as far as route of exposure.  To send a registration review every 15 years for the pesticides discussed here, that would seem to be important because that may impact how pesticides are actually being used.  And we're starting to -- one of the things, I think, that came to you is utilization is very critical.  So, keeping in mind about that review.
                  I also had a problem with, "based on assessor's best professional judgment."  That's written three or four times.  You need to give more guideposts.  Otherwise, you're not going to necessarily have a consistent decision process.  And maybe inputting so you can have that consistency; so when it changes, a different group comes in, not a whole new set of ideas come forward.
                  Similarly, there's a whole discussion on new pesticides versus new use of pesticides.  I think that was done very well here.  I don't know if that should be implemented and considered in what we've been talking about for the last few days because I didn't hear that discussion.
                  All right.  And then on the end of page 31, it says, if monitored concentrations exceed the model's estimate, the appropriateness of the modeling should be revisited, and any applied refinements reexamined.  That's certainly true.  Being a measurement person, you also have to do the same thing to the monitoring data.  Make sure, first of all, check off the QAQC and make sure that's not what's doing it.
                  But even more important, find out why that might be happening.  There may be an unusual event that's causing that higher monitoring data than you expected.  And that may be an underlying issue that you may be missing in your model.  So, not just look at the model but look at it in context of why that monitoring number may be high, because that may give you some insight.
                  And then lastly, which was sort of -- when I looked at that future direction section, I thought it was very good.  I think that's important, but you need this to be a living document.  And you need to say, maybe have an -- I don't know if you want to put it in the framework, but there has to be a time period that this is rereviewed to see have any of those future directions now come to a point that they can be incorporated as not future directions of the part of the process.
                  Some of these documents sort of lay on a shelf.  And until you come to a new SAP, you don't do anything with it.  This, I think, is too important to leave there.  And I think it's done too well to have that case.
                  DR. ROBERT CHAPIN:  Comments from other members of the Panel on the clarity and organization of the framework?
                  MR. TERRY COUNCELL: This is a direct follow up to Dr. Potter's comment, I think.  Which is it's important that there are available process-based biophysical models that are out there for -- I guess Tier 4 would be the appropriate level.
                  And my comment is that those are often deemed infeasible because they are so complex.  But when we start to look at weight of evidence and the complexity of what you're managing on your data use for those analyses, I would say that it's no more complex if some of the model is put in the background.  And that's becoming more and more common.
                  So, that's a very long-term thing, I suppose, recommendation; but not to disregard the models that are provided elsewhere.  And that Dr. Potter listed some of those.  Thank you.
                  DR. ROBERT CHAPIN:  Thank you.  Other comments?  Yeah; Dr. Baffaut?
                  DR. CLAIRE BAFFAUT:  This is Claire Baffaut, and I would like to add something else about those models that could potentially be used for Tier 4 assessments.  And I'm thinking SWAT, APEX, HSPF, NPSP, whatever.  All those models are very performance when it comes to flow estimation; when it comes to transport of pollutants, it's much more difficult to get good performance.  It is possible.
                  But the other thing is that they all route pollutants from the fields through each stream reach on a mass basis.  And that complicates things when the variable of interest is a concentration.
                  And there has been some -- there's a study that comes out of the TMDL Group at Virginia Tech.  And it was developed for bacteria specifically.
                  But it showed that all the low-flow days where assimilated was a very high concentration, and it's purely a computational issue.  It's because it's routed on a mass basis, and then to have the concentration you divide by the flow.  And on low-flow days that flow is slow, and you get a very high concentration.
                  So, there is a note of caution here when you're using those types of models that route on a mass basis.  And to use them for concentration estimation would require additional work and probably truncation of all those low-flow days results.
                  DR. ROBERT CHAPIN:  Other comments on the framework?  Comments from -- Ian?
                  DR. IAN KENNEDY:  I just wanted to add one -- this is Ian Kennedy.  I just wanted to add one quick thing about fate studies.  And I think that they are constraining the fate parameter evaluations; and probably the study design will need to be revisited for a substantial improvement in what we're doing now.  Thanks.
                  DR. ROBERT CHAPIN:  Other general comments?  No.  Okay.  I'll turn to our EPA colleagues and ask if you'd like to respond or get clarification on things that you heard that were maybe less than perfectly clear.
                  DR. ROCHELLE BOHATY:  That was great.  Thank you.
                  DR. ROBERT CHAPIN:  Okay.  We are right at 10:30.  We're going to take a break.  And since we're -- since it's going to be -- it's evident that we have just -- we're going to deal with each case study next, and then we will be done.  
                  So, it's looking like we're going to be finished by lunch.  Which means that we may have some travel modifications that could happen if you wanted.  I'm going to turn them over to Ms. Gibson who will educate us about what we should do if we want to modify our travel plans.
                  MS. TAMUE GIBSON:  Certainly.  Thank you.  If you want any revisions you may go out to our contractor, who is directly outside the door, and you can make your modifications.  And he would need your name, your contact information, and he will look for your availability.  And this could be done after the -- after the culmination of the meeting.
                  DR. ROBERT CHAPIN:  Can it be done after the -- can we not do it now?
                  MS. TAMUE GIBSON:  He won't have time, no.  Yeah, after we complete our meeting.
                  DR. ROBERT CHAPIN:  Okay.  All right.  So, let's break for 15 minutes.  We'll be back at quarter of.
                  
                  [BREAK]
                  
                      CHARGE QUESTION 4(b) - CASE STUDY 1
                  
                  DR. ROBERT CHAPIN:  Okay; here we go.  Let's talk about some case studies.  Okay.  Perfect.  4(b).  One, two, three, four; we have the four -- five permanent members.  We're rocking and rolling.  Okay; Dr. Bohaty?
                  DR. ROCHELLE BOHATY:  Hello.  Rochelle Bohaty, 4(b).  In case study 1, EPA demonstrates the implementation of the drinking water framework and specifically the use of SEAWAVE-QEX and bias factors, to analyze monitoring data for a pesticide with acute risk concerns.
                  Please comment on the SEAWAVE-QEX analysis used to assess sites that had a potential drinking water level of comparison exceedances.  Please comment on EPA's use of chem1 short-term sampling bias factors, to adjust chem1 concentrations for comparison to potential drinking water level of comparison.
                  DR. ROBERT CHAPIN:  Talk about the examples.  Dr. Kennedy?  There will be a brief pause.
                  DR. IAN KENNEDY:  Okay.  Overall, I think I found the case studies useful in understanding how SEAWAVE-QEX and sampling bias factors fit into the evaluation framework.  And I think the document shows the processes in the framework and moving through them from screening up through Tier 4.
                  I think I would've liked to see a little more detail about the SEAWAVE-QEX fits and how they were evaluated.  The short-term sampling bias factors were calculated for 31 sites; two of them were discussed briefly.
                  But I think that because the evaluation of the results of the SEAWAVE fits, as we've discussed here yesterday, is a very important part, and maybe the most difficult part, a little more discussion, especially kind of how the weaker fits, why they were acceptable or why some were not acceptable, would, I think, really help.  And would help in anyone learning about how to evaluate and deal with SEAWAVE-QEX results.
                  And I'm not very experienced at looking at these, obviously.  But it seemed to me that many of the SEAWAVE-QEX, especially the second plot on the diagnostic plot showing the wave, showed a very weak amplitude, often with a very long period.  So, it was unclear if there was really much of a wave in some of those.
                  Also, there was one site that was split into two parts, each of about 10 years.  And the diagnostic plots, again, for the wave showed very different shapes between those two sets.  So, it wasn't really clear how the two were combined, because one sampling bias factor was provided for the site.  Although, that site also showed generally decreasing concentrations over the whole 20 years or so of the sampling period.  So, the lower concentrations in the second half may have been one of the reasons for the difference.
                  Another interesting thing about that is that the highest detect recorded was recorded one month before the start of flow data was available.  And so it couldn't be used in the SEAWAVE-QEX fits.  And it would be interesting to know how that sort of situation impacts the evaluation.
                  On page 42, it stated that the sampling bias factors were developed by running the Python Program, which was included with the package or with the supplemental files.
                  In spyder, and just as a suggestion, I think that doing that in a Jupyter notebook would be useful because it would allow for commentary and discussion along with the actual evaluation; and would form a better record of how the calculations were done.
                  And then the same might really apply to the SEAWAVE-QEX runs themselves, which could be done in either a Jupyter notebook or an R Notebook using RStudio.  Which allow the same sort of mixture of text, commentary and calculations.
                  Another question I had.  Because there -- the majority of the sites that were sampled did not have enough datapoints to apply the sampling bias factors.  And therefore, they're used more qualitatively.
                  But what do you do in a case where you have one or more sites, where if you had applied the sampling bias factor, you would have a DWLOC exceedance.  Especially, this would be true for this case and not the chem2 case for the yearly.  But for a peak, even if you had all the other -- even if you had daily sampling, all the missing days were sampled as non-detects, you might still have a sampling bias factor that would -- you know, if you had more datapoints, even if they were non-detects, the sampling bias factor would still give you that DWLOC exceedance.
                  So, I think there's something there to think about how you would deal with that.  And I think that's it for now.
                  DR. ROBERT CHAPIN:  Thank you.  Dr. Nowell?
                  DR. LISA NOWELL:  Lisa Nowell.  I enjoyed reading the case studies.  I thought it was a fun opportunity to see how the tools work and illustrated the level of the complexity of the assessment that you have to do, the logic and the balancing of the factors that are needed for a pesticide-specific document.  So overall, I thought that this case study was effective at illustrating this.  And also, as Dr. Kennedy said, how the SEAWAVE-QEX and the SPFs fit into the drinking water assessment.
                  I know I keep harping on agricultural use, but I'm going to bring it up again.  Chem1 had high urban use.  At one point, on page 6 in Section 3, there was a statement that residential use patterns were not modeled, as the agriculture modeling is expected to be protective of residential use patterns based on lower application rates.  And residential use patterns are expected to be more spread out over time and space as compared to agricultural use patterns.
                  But later, you mentioned that -- on page 31 -- that residential urban use was 3.8 million pounds in contrast to 700,000 pounds applied in agriculture.  So, that raised a bit of a flag for me.  It might work out that that first statement and the expectation are still valid, but it definitely raised a bit of a flag for me.
                  And also, depending on the study areas, what are the chances that you have vulnerable areas that are receiving both agricultural and urban inputs?  Would concentrations from urban use still be negligible compared to what you're seeing in agriculture?  Or should the results be added?
                  And I will note, regarding nonagricultural use, that throughout the case study you do mention urban use several times; noting that the detection frequencies were higher in urban areas, noting the lack of information on urban use and that this is adding uncertainty.  And also noting potential for drinking water level of concern exceedances in urban areas.
                  And I encourage you to address residential use in the usage section and also the monitoring and conclusion section.  Right now they don't say anything about that.  But I think it would be good to address whether you would consider monitoring at urban sites as well as the citrus, apple, and soybean areas, which are mentioned in the monitoring section -- or conclusions, I'm not sure which, but one of the last few sections.
                  In Section 4.3.1, page 20, this is Tier 3, investigation of the SBF.  And the equation at the bottom of the page defined a term that shows up in Figure 4.3 or axis label is minimum SBF at DWLOC.
                  And it defines it as the DWLOC divided by the maximum measured concentration.  And I found this a really confusing way of expressing this relationship.
                  I think fundamentally, it's easy to understand it as, you know, the SBF is a multiplicative factor.  You multiply it times your measured concentration to estimate the maximum concentration.  And mathematically or arithmetically, you can derive your minimum SBF at DWLOC from that basic relationship.  It's obviously correct, but it's not at all intuitive.
                  And basically, when you readjust the original definition of, say, the SBF is equal to the concentration -- the true maximum concentration divided by the maximum measured concentration, then we don't want that true maximum concentration to exceed the drinking water level of concern.
                  So, we're looking for situations where basically the true maximum concentration is greater than that DWLOC.  And that does actually convert to an equation.  If you divide both sides by the measured maximum concentration, and basically reverse a couple of equations, you end up with an exceedance will occur if the SBF is greater than the drinking water level of concern, divided by the measured maximum.
                  So, this is basically consistent with what you say.  The lower that minimum SBF to DWLOC is, that means the higher the potential that concentrations will exceed the DWLOC.  And it does work out mathematically, but it's convoluted, nonintuitive.
                  I do see why you set it up that way.  Because in Figure 4.3, you're trying -- I think -- this is what I was inferring at least.  That you're trying to avoid selecting a single SBF value to use in computing an SBF adjusted concentration.
                  You've got a distribution of SBF values for each of the four referenced pesticides, and you've got a distribution of sites that have measured their concentration measured maximum.  So, you're trying to avoid choosing an individual value.
                  So I do get why you had to do that.  But I wonder if there's a different way you can name it something that would indicate that this is kind of a safety margin to the DWLOC; which would at least intuitively let people know that smaller -- that closer the -- the closer you are to the DL- -- the smaller that number is, the worse it is, because it's counterintuitive.
                  Anyway, I had to -- during that section, I had to stop, backtrack and recalculate what you were working with; which I think is a drawback in the case study just because maybe not everybody will bother to do that.  And if you don't do that, you don't really understand Figure 4.3.
                  I thought your follow-up explanation to Figure 4.3 was well-done; and how you handled regions with studies having less than 13 samples per year were regions that had insufficient monitoring data.  At least this analysis resulted in bringing three additional regions into Tier 4.  So, you weren't ignoring those data.
                  On page 29, Section 5.2, I think, there's a sentence that reads, SEAWAVE-QEX was used to estimate pesticide concentrations for WQP sites that met the following criteria; as few as 12 samples per year, for three years of data and censorship on non-detections up to 75 percent.  These criteria were used to simplify the case study and are not intended to reflect specific criteria to screen data as the model may be suitable for use with data that do not meet these strict criteria.
                  So, I'm wondering about the second sentence.  They seemed like maybe not fixed criteria, but they did seem like criteria to me.  And I wasn't sure that that was necessary to bring up in a case study.  But that minor point.  But I think you should mention the flow requirement here, because that's -- we don't -- that's one of the big things that leaves data -- leaves sites out, is because there's no flow.  And that is -- at least some kind of covariate is essential.
                  In general, I think you did a great job illustrating and interpreting the results of the SEAWAVE-QEX analysis.  It made things much clearer.  It was nice to see it fit in the way it did.
                  I didn't think the explanation for the SBF analysis was as clear as the SEAWAVE-Q.  I think that it might help if you started with what I -- by the time I got to the end, I was thinking that these are the criteria that you're using.  You're defining an exceedance as having a detected concentration greater than the method reporting level and an SBF adjusted concentration greater than the drinking water level of concern; and the site location is relevant to DWI intake.
                  And there was a lot of references to concentrations greater than 1 microgram per liter.  And if you didn't catch that that was the reporting level, you were going, why is 1 microgram per liter important?  At least that was what I was doing until I found that spot.  So once -- if you set up your criteria at the beginning, and then go through the analytical steps, I think it would be easier to follow, if we know where you're heading, as we start, to watch you get there.
                  It became confused in the text that accompanied Figures 5.15 and 5.16, but I think there's a figure numbering error; so that might be part of the problem.  The description of color coding of the sites and the texts doesn't seem to match either one of the figures.  So, maybe check that.
                  And in the figures which show sites and regions 3 and 17, along with information on SBFs, I would love to see you distinguish in these figures, any sites that were predicted to exceed the drinking water level of concern and are relevant to the drinking water intake.
                  So, these are like the final -- I think there were three sites in one region; I can't remember how many in the other, but I couldn't tell which ones they were.  And I wasn't quite sure which figure I should be looking in.  So, that's just a, you know, a housekeeping.
                  So again, in the monitoring conclusions, I would definitely address whether you think there should be monitoring in urban areas.
                  You recommended a more robust service water monitoring program in Florida, Washington, and Oregon.  I'm not sure whether that was meant to encompass urban areas or not.  And also, do the conclusions about urban areas extend beyond the two regions of concern?  Or is that more of an agricultural association, regions 3 and 17?  I wasn't sure about that.
                  And it did seem that if the use of the SBFs is going to be restricted to screening in Tier 3, then the analysis -- I guess the question in the analysis that we read in Tier 4 wouldn't be there for the SBFs.  Is that right?
                  MS. JESSICA JOYCE:  The analysis in Tier 3, we used the SBFs from the White Paper calculated for four other chemicals.  And the SBFs in Tier 4 were calculated based on the SEAWAVE-QEX runs for chem1.  So, they were chemical-specific.
                  DR. LISA NOWELL:  Okay, perfect.  Great.  And I really like Figure 1.1.  It was complicated, but I found out that I was going -- once I noticed it, I went back and checked it three or four times as I read.  And I found it very helpful.  And that's it.
                  DR. ROBERT CHAPIN:  All righty.  Mr. Councell?
                  MR. TERRY COUNCELL:  All right.  So this is the fun part, where you tie everything together and show us if this thing works.  And so I think it illustrated a lot of really good points.
                  You had 19,000 sites that you had monitoring data for.  And when you narrowed it down to what you could use, you know, the criteria for the SEAWAVE-QEX, that was 39.  And then when you actually ran it to make sure you had all the -- it met the model criteria, you ended with 32.
                  And so that really shows how narrow the data that you have to work with is.  And out of that, you were kind of interested in HUC-2, Region 3 and 17.  And so, I thought there were six sites in each of those.  And both of those were -- or all of those were streams, I think you mentioned.  And so, that really shows that you really have limited data to work with.
                  That did bring up a point is, you know, of the sites that you were able to use the SEAWAVE-QEX, how representative is that of that.  Because a lot of the streams are so small and variable flow that they're not used as community water system intakes.  And so, can you translate those to the larger intakes where that would be?  They're obviously contributors, but something to consider.  But you kept moving on, which was good; and I like that.
                  So, you then went over and you calculated the SBFs.  And they -- you showed that they had -- you know, they were large and had a lot of uncertainty, but you were still able to look at the monitoring data.  And then you looked kind of at your evidence, and kind of saw that one of your sites that you did the SEAWAVE-QEX was a drain.  And so, you eliminated that one.
                  I liked it that you went through and you checked every time that you went through each stage and kind of, is the data reasonable.  You know, you went and did the extra steps; and I think that was good that you did that.
                  I like the diagnostic plots that you put in there; those were very good.  And then on page 55, you kind of gave some conclusions that doing the modeling you didn't see any drinking water level of concern for these sites.  Yet this monitoring data, when you were able to use that, you were able to pick up some.
                  So I think it illustrates the utility of using this model.  And you know, it's not perfect; I think there's problems.  It would be nice if we had more samples for the SEAWAVE-QEX.  That would be helpful.  Maybe if we can find another covariate to use other than flow, that would be helpful.  There's a big long dream list.  
                  But in reality, you guys are resource limited.  You only have so many resources, so many people, so much time.  You have 100 to 150 of these cases a year to work on, so you have limited time to put to this.  
                  And so, I think even taken all those limitations in, it showed a lot of promise.  And I applaud you all for putting all the work into this so far.  And I hope that very soon that you can use this tool.  Thank you.
                  DR. ROBERT CHAPIN:  Excellent.  Thank you.  Dr. Kennedy, you're listed as a -- with your GIS hat on for this one.  Do you have any additional comments?
                  DR. IAN KENNEDY:  I don't think I have any additional comments.
                  DR. ROBERT CHAPIN:  Excellent.  And Dr. Corcoran, comments on chem1?
                  DR. GEORGE B. CORCORAN:  No additional comments.  Thank you.
                  DR. ROBERT CHAPIN:  All right.  Let me turn to our EPA colleagues and ask if you have -- I see a vertical.  
                  DR. KENNETH PORTIER:  Dr. Portier has woken up.  Now I just wanted to comment on something Dr. Nowell said.
                  You know, when you look at this Figure 4.3, you see the word exceedance, and you're expecting to see a threshold and looking for points above a threshold.  And instead, you have a threshold and you're looking for points below the threshold.  So exceedance is the issue here, the word exceedance.
                  You know, there's nothing wrong -- and this reminds me of MOE kinds of analysis that you do in risk -- well, I understand why you do this.  But it's the word, brings up a vision and the vision doesn't match with the graph.  And that's a no-no in imaging.  So, I would suggest you change the title not the graph.
                  DR. ROBERT CHAPIN:  Other comments from non-designated -- anybody else in the Panel have comments on the chem1 example?
                  Okay.  Well, let me turn to our colleagues and find out if they have any requests for clarification, or comments, or feedback or whatever.
                  DR. ROCHELLE BOHATY:  We're good.  Thank you.
                  DR. ROBERT CHAPIN:  Let us move -- 
                  DR. ROCHELLE BOHATY:  All right.
                  DR. ROBERT CHAPIN:  -- said Poo, to the last question.
                  
                      CHARGE QUESTION 4(c) - CASE STUDY 2
                  
                  DR. ROCHELLE BOHATY:  So, the last question.  This is Rochelle Bohaty, 4(c).  In case study two, EPA demonstrates the implementation of the drinking water framework, and specifically the use of SEAWAVE-QEX and bias factors to analyze monitoring data for a pesticide with chronic and cancer risk concerns.
                  Please comment on SEAWAVE-QEX analysis used to assess sites that had potential drinking water level of comparison exceedances.  Please comment on EPA's use of chem2 long-term sampling bias factors to adjust chem2 concentrations for comparison to the potential drinking water level of comparison.
                  DR. ROBERT CHAPIN:  And we just got done with a short-term chemical of concern, and now we're dealing with the longer-term stuff.  Dr. Kennedy?
                  DR. IAN KENNEDY:  Okay.  I think that there's a lot of overlap between the two case studies.  But I will say that the comment about the evaluation of SEAWAVE-QEX applies to Page 2 of course.
                  In Section 5.1 for case 2, it's not clear to me how the EDWCs were calculated.  It says, the maximum PCA within each HUC-2 region was multiplied by every relevant scenario EDWC; but I'm not clear what a relevant scenario EDWC was.  And how many of them there were; because the figures have a lot of datapoints on them.  So, some clarification there would be helpful.
                  Since there were questions about using precipitation or stream stage where flow was not available, it would've been nice if the site -- especially the sites that were thrown out for lack of streamflow, this would have been tried at least for illustrative purposes.  Because as it is, the case study covers the streamflow case, but we don't get that information on the case, especially using precipitation as a covariate.
                  Some additional discussion of how the 10 sites with unacceptable, SEAWAVE-QEX fits were rejected, would be helpful to understanding the evaluation process of SEAWAVE-QEX.
                  And finally, a minor note.  In Table 4.5, the half-life of zero is given to denote stable.  And strictly speaking, that's kind of the opposite of stable.
                  So I know that internally, that's often used.  But for something like this, it would probably be best to just use the word stable in those cases.  Thanks.
                  DR. ROBERT CHAPIN:  Do you have any comments wearing your GIS hat?
                  DR. IAN KENNEDY:  No, I don't.
                  DR. ROBERT CHAPIN:  Okay.  Let's see.  Dr. Potter?
                  DR. THOMAS POTTER:  Good morning.  Tom Potter here.  Again, I agree with Dr. Kennedy.  There's a substantial amount of overlap between comments that would be appropriate on chem1.
                  So, all of the above, yes, we should take those into account in terms of the discussion of chem2 and the long-term sampling bias factor application.
                  I'll say one thing that would've really been helpful to me, maybe it just depends on whether you're a visual person or not, but a graphical representation of the decision process that went on.  You know, having the opportunity to refer back to that regularly, as you go through the text.  The go and no-go junction points would be much easier to identify and ultimately get to the end of the assessment.
                  So, really encourage you to put some sort of graph figure that helps people go through the step by step decision process.
                  A few comments.  Perhaps they're probably best addressed, or may have already been addressed, because I need to ask this question for clarity.  The application of SEAWAVE-QEX was based upon what is in the SOP.  Is that correct?
                  MR. DANA SPATZ:  Yes, that's correct.
                  DR. THOMAS POTTER:  Yeah.  And it was -- you know, it may be a question of the obvious, but I just wanted to be sure that that was the case.  And so, some of what was done here would've been helpful to refer back to the SOP, where you know, this is discussed and why it was discussed.
                  And I'll just use -- you know, one example would be, you'll look at the monitoring table in Table 3.4, and the results are characterized in several different ways, dissolve, total recoverable, et cetera.  And so, it would've been helpful to say, refer back to the SOP in terms of how we decided whether the dissolved and the recoverable, or whatever, and what the equivalencies were, and what was retained and what was rejected.
                  Obviously, you don't need to go into all that detail here in the writeup.  But again, alerting the reader that that exists out there, and there are a whole series of decisions that were made regarding data use that are embedded in this, would certainly help the document be more comprehensive.
                  Regarding Tier 3, I guess the only way SEAWAVE-QEX comes into the dialogue is for the values that were computed for the four chemicals that were part of the USGS data.  And we've talked a lot about that in terms of how it may be best -- those data may be best applied.
                  So, I believe the discussion was clear in the sense you followed your game plan and applied those.  And you know, we've made some recommendations on maybe some alternate techniques or an alternate track.  So, I'll just note that those are out there.
                  A question that I would have, and it wasn't inherently clear in the Tier 3 analysis is, were the PWC and the SBF assessments in agreement?  It just seemed that you did both.  And I was kind of left a little confused as to what the decision process is there.  And whether there's more weight given to one analysis, i.e., the PWC or the SBF.  Perhaps it's in there.  And if -- you're looking at me with a question, so -- 
                  MS. JESSICA JOYCE:  I can answer --
                  DR. THOMAS POTTER:  Yeah.
                  MS. JESSICA JOYCE:  But I think we would have to consider the data in what we are looking at and how we would weight different things.
                  I think one of the key things everyone has to understand is that the pesticide and water calculator modeling, we are simulating applications to watershed and index reservoir that we move around the country.  And so, what we estimate exposure from the PWC modeling is not expected to be the same as where we would measure it across the United States, where we have different flowing systems, different watershed sizes, something very different than what we're modeling.
                  So, we don't necessarily expect them to be the same.  But we would hope that the PWC modeling would be higher than what we're seeing in drinking water relevant areas.
                  And so then, whether you weight the modeling or the monitoring, depends on how confident you are in either of those results.  When you did your simulation and your PWC modeling, is it in the area of where you have high usage that would be representative of where you expect the chemical to be used?  Does it have the similar precipitation and all of that?
                  When you're looking at the monitoring, you consider the same -- like how confident are you that that monitoring would be capturing what you might be concerned about?  Does that answer?
                  DR. THOMAS POTTER:  It does answer the question.  And as I say, I didn't really get the flavor of that in the document.  So my comment -- and I'm -- I was -- I'm confident, knowing the staff here, that you guys have gone through and applied the best scientific judgment here.  But it's not clear as to how that process flowed.  And so it made it difficult to kind of -- I would ask the question, you know, is there increased value in doing one assessment over the other PWC, SBF?  And I realize probably we could spend all day on that.
                  But you know, I think that's sort of part of the questioning process.  And ultimately, much of it is embedded into the judgment of the risk assessor.  But a comment there.  
                  Again, moving on to Tier 4.  We at Tier 3 said, yeah, we need to go to Tier 4.  I think how SEAWAVE-QEX was applied, it was consistent with, I believe, what was in the SOP.  I believe that there's some language in there that would've made it kind of hard to figure out just how many sites were applied, in what way, with regard to the use of the flow as a variate and possibly the use of alternate parameters as covariates.
                  And I'll provide a, you know, citation or text on that in the -- I don't have it on my -- right up here, but I will identify the confusing paragraph.  But it starts off, however -- you know, there were 67 sites that met sampling criteria.  And then it goes on to kind of break those down.  Again, a graph or a table would've been helpful, rather than having to kind of -- trying to sort that out and do the arithmetic to the text.
                  And again, some of this relates back to SOP.  And so you can say -- ah, according to the -- on the basis of our SOP, this is what we did.
                  And I will also say, it just seems, again, unresolved.  Perhaps, again, a table or graph would help.  You know, were nonflowing systems is included in the analysis?  I don't believe they were, but I was not entirely confident that I was making that correct assessment.
                  Overall, I thought, as was the case of the example for chem1, I thought this was a very good and illustrative use of the tools that we've been discussing here over the last few days.  The process was logical, and I commend you for doing that.
                  I believe that, perhaps, most of the fixes that I would recommend are editorial in nature rather than technical or scientific; again, to provide clarity and transparency as to what the decision process is.  I'll stop there.
                  DR. ROBERT CHAPIN:  Thank you.  Mr. Councell?
                  MR. TERRY COUNCELL:  Okay.  Again, many of the comments are similar to what they were for the first example.  It really didn't show -- you know, when you finally got to the number of sites that you could use, the SEAWAVE-QEX was .51 percent of the available sites.  Again, that really illustrates that you're very limited in the data that you can use.
                  It does make me concerned about, you know, how representative is that?  Especially, when you talk about Region 3, that none of the sites were in a citrus-growing region in Florida.  That was your main area of use, you know, the chemicals applied to citrus.  So, it makes you wonder, okay, how representative is that?
                  In Region 18, there was one site that was collocated on a potential use area.  So, again, how representative are the sites that you did the SEAWAVE-QEX to the areas of interest?  So that's a little concerning.
                  But you went ahead, and you did -- I thought you appropriately used the long-term SBFs and calculated them.  You used your weight of evidence.  You found one site that did exceed the DWLOC when it was a retention pond.  So, you removed it, because nobody's going to use that as a drinking water intake.
                  So, I'm glad that you guys go through there and you actually look, does this make sense.  And you all are, you're very good professionals.  And we have lots of confidence in you and appreciate your efforts on that.
                  So the data that you use, this case example was showed and illustrated very well, real-world examples that you're going to, and problems that you're going to encounter as you're using this model.  But I think what you did was fairly reasonable.  And again, given the resources it would take to do further analysis, it would be nice, as Dr. Kennedy said, to maybe on this case have used precipitation as a covariate and see what you came up with.  But it would be a lot of extra effort.  So I'm not saying go back and redo it.  
                  And so again, it showed the utility of the model.  And I think it's a good case.  And it was a neat writeup.  And good work on that.  I appreciate that, so thank you.
                  DR. ROBERT CHAPIN:  Excellent.  Thank you.  Comments from other members of the Panel on chem2?  Ken, anything to say?  Dr. Yang?  Okay.  All right.  
                  So questions or comments for clarification from our EPA colleagues?
                  DR. ROCHELLE BOHATY:  No, we are good.  Thank you.  Oh, sorry.  Yeah.  Thank you.  We're done.
                  And I just want to thank everybody for your feedback over the course of several days.  It's been very helpful.  And it will come in handy as we move forward with the implementation of these tools.  Thank you.
                  DR. ROBERT CHAPIN:  Excellent.  So, we finished the second trimester of our gestation period here.  And we're entering the third.  And Ms. Gibson will talk to us about that third trimester.
                  MS. TAMUE GIBSON:  All right, everyone.  I would like to thank members of the public and the Panel for your participation.  Panel members, thank you so much for a robust discussion during this session.  And my colleagues from EPA, thank you again.  You all did an exceptional job, and we greatly appreciate the dialogue here.
                  I would also like to basically state that this will close the public portion of the meeting.  And with that, we are officially adjourned.  And I would like to speak to the panel members for a brief second downstairs in the breakroom.
                  DR. ROBERT CHAPIN:  Okay.
                  MS. TAMUE GIBSON:  Thank you.
                  DR. ROBERT CHAPIN:  Excellent.
                  
                  (MEETING ADJOURNED)
