Wednesday, December 16, 2009

EPA-HSRB-09-03

Kevin P. Teichman, PhD

Acting Science Advisor

Office of the Science Advisor

1200 Pennsylvania Avenue, NW

Washington, DC 20460 

Subject: October 20-21, 2009 EPA Human Studies Review Board Meeting
Report

Dear Dr. Teichman,

	The United States Environmental Protection Agency (EPA or Agency)
requested the Human Studies Review Board (HSRB) to review two completed
intentional studies examining the effects of intentional exposure of
human subjects to pesticides containing pyrethrins/ pyrethroids. The
Agency proposes to rely on these two  studies, conducted prior to
publication of the EPA’s expanded final rule for protection of
subjects in human research (40 CFR 26) on February 6, 2006 (71 Federal
Register 24, 6137), for regulatory actions under the pesticide laws. The
Agency asked the HSRB to advise the Agency on a range of scientific and
ethics issues regarding how the studies should be assessed against the
provisions in 40 CFR sections 26.1701 – 26.1704 of the final human
studies rule.

	The Agency also requested the HSRB to provide scientific and ethics
reviews of two proposed human studies: a proposed Carroll-Loye
Biological Research, Inc. (CLBR) insect repellent efficacy study
(LNX-003); and a proposed Antimicrobial Exposure Assessment Task Force
II (AEATF II) aerosol spray application protocol (AEA04).

	The enclosed report provides the Board’s response to EPA charge
questions presented at the October 20-21, 2009 meeting.

Assessment of Completed Research Study: Newton, J., Breslin, A. (1983)
Asthmatic reactions to a commonly used aerosol insect killer. Medical
Journal of Australia 1:378-380. 

Science

Because of the use of a complex mixture represented by an insecticide
product rather than a specific chemical in this study, the Board
concluded that the Newton and Breslin study was not relevant when
considering the asthmatic or allergic respiratory response for
pyrethrins/pyrethroids as a class. The Newton and Breslin study was
relevant when considering the asthmatic or allergic respiratory response
only for the specific insecticide product that was tested.

Considering the substantial limitations of the study, the Board
concluded that the study was of limited utility. The Board recommended
that the Agency be cautious in its use of these data, limiting it to a
qualitative contribution to the overall weight of evidence analysis. 

Ethics

Given the limited information available about this study, the Board
concluded that there neither was clear and convincing evidence that the
study was fundamentally unethical, nor clear and convincing evidence
that the study was significantly deficient relative to the ethical
standards prevailing at the time the research was conducted.

Assessment of Completed Research Study: Lisi, P. (1992) Short
Communication: Sensitization risk of pyrethroid insecticides. Contact
Dermatitis 26:349-350. 

Science

The Board concluded that the Lisi study was likely scientifically sound,
but the brevity of the report and limited details as to how the study
was conducted prevent an accurate assessment of its reliability.  

If carried out according to accepted guidelines and criteria, the Lisi
study likely provides hazard assessment data that are relevant to
assessing whether exposures to pyrethrins/pyrethroids may be associated
with allergic contact dermatitis or sensitization responses.

Unanswered questions about participant history and selection, study
procedures, and outcome definitions should be taken into account when
considering these data in the overall analysis.

Ethics

Given the limited information available about this study, the Board
concluded that there neither was clear and convincing evidence that the
study was fundamentally unethical, nor clear and convincing evidence
that the study was significantly deficient relative to the ethical
standards prevailing at the time the research was conducted.

Assessment of Proposed Carroll-Loye Biological Research Study LNX-003:
Efficacy Test of KBR 3023 (Picaridin; Icaridin) - Based Personal Insect
Repellents (20% Cream and 20% Spray) with Ticks Under Laboratory
Conditions.

Science

The Board concluded that the protocol submitted for review, if modified
in accordance with Agency recommendations and conducted accordingly,
will likely yield scientifically valid results on the efficacy of these
two picaridin-based insect repellent formulations against ticks.

Ethics

The Board concluded that the protocol submitted for review, if modified
in accordance with Agency recommendations and conducted accordingly, is
likely to meet the applicable requirements of 40 CFR 26, subparts K and
L.

Assessment of Proposed AEATF II Scenario and Protocol AEA04: Research on
Exposure of Janitorial Works Applying Pesticides Formulated as Aerosol
Sprays.

Science

The Board concluded that the protocol submitted for review, if modified
in accordance with Agency and HSRB recommendations, will likely generate
scientifically reliable data, useful for assessing the exposure of
handlers who apply antimicrobial pesticides formulated as aerosol
sprays.

The Board provided a number of recommendations for how the study could
be improved, including: clarifying the criteria for exclusion of 
subjects who deviate grossly from the protocol; acknowledging that
systematic differences in the air sampling results are likely to occur
between the two methods used; and, considering carefully other variables
that may influence exposure measurements.

Ethics

The Board concluded that the protocol submitted for review, if modified
in accordance with Agency and HSRB recommendations and conducted
accordingly, is likely to meet the applicable requirements of 40 CFR 26,
subparts K and L.

The Board recommended that the protocol be revised to address certain
concerns, including: ensuring that documents in Spanish are reviewed by
someone familiar with the dialect written and spoken in the target
community; design and posting of community notification flyers
appropriate for guests staying at the motels where study procedures are
likely to take place; and, exclusion of volunteers who may be at greater
risk of product-related harm due to immunodeficiency or other underlying
conditions.

Sincerely,

 

Sean Philpott, PhD, MSBioethics

	Chair

	EPA Human Studies Review Board

NOTICE

This report has been written as part of the activities of the EPA Human
Studies Review Board, a Federal advisory committee providing advice,
information and recommendations on issues related to scientific and
ethical aspects of human subjects research.  This report has not been
reviewed for approval by the Agency and, hence, the contents of this
report do not necessarily represent the view and policies of the
Environmental Protection Agency, nor of other agencies in the Executive
Branch of the Federal government, nor does the mention of trade names or
commercial products constitute a recommendation for use.  You may obtain
further information about the EPA Human Studies Review Board from its
website at   HYPERLINK "http://www.epa.gov/osa/hsrb" 
http://www.epa.gov/osa/hsrb .  You may also contact the HSRB Designated
Federal Officer, via e-mail at   HYPERLINK "mailto:phre@epa.gov" 
phre@epa.gov 

	In preparing this document, the Board carefully considered all
information provided and presented by the Agency presenters, as well as
information presented by public commenters.  This document addresses the
information provided and presented within the structure of the charge by
the Agency.

US ENVIRONMENTAL PROTECTION AGENCY

HUMAN STUDIES REVIEW BOARD

Chair

Sean Philpott, PhD, MSBioethics, Director for Research Ethics, Assistant
Professor of Bioethics, The Bioethics Program of Union Graduate College
and the Mount Sinai School of Medicine, Schenectady, NY

Vice Chair

Janice Chambers, PhD, DABT, William L. Giles Distinguished Professor,
Director, Center for Environmental Health Sciences, College of
Veterinary Medicine, Mississippi State University, Mississippi State, MS

Members

Suzanne C. Fitzpatrick, PhD, DABT, Senior Science Policy Analyst, Office
of the Commissioner, Office of Science and Health Coordination, US Food
and Drug Administration, Rockville, MD

Vanessa Northington Gamble, MD, PhD, University Professor of Medical
Humanities, Professor of Health Policy and American Studies, The George
Washington University, Washington, DC

Sidney Green, Jr., PhD, Fellow of the ATS, Professor, Department of
Pharmacology, Howard University College of Medicine, Washington, DC

Dallas E. Johnson, PhD, Professor Emeritus, Department of Statistics,
Kansas State University, Manhattan, KS

Michael D. Lebowitz, PhD, FCCP, Retired Professor of Public Health &
Medicine, University of Arizona, Tucson, AZ 

Lois D. Lehman-Mckeeman, PhD, Distinguished Research Fellow, Discovery
Toxicology, Bristol-Myers Squibb Company, Princeton, NJ 

Jerry A. Menikoff, MD, JD, Director, Office of Human Subjects Research,
Office of the Director, National Institutes of Health, Bethesda, MD

Rebecca Parkin, PhD, MPH, Associate Dean for Research and Public Health
Practice, School of Public Health and Human Services, The George
Washington University, Washington, DC

William Popendorf, PhD, MPH, Professor, Department of Biology, Utah
State University, Logan, UT 

*Ernest D. Prentice, PhD, Associate Vice Chancellor for Academic
Affairs, University of Nebraska Medical Center, Omaha, NE

Linda J. Young, PhD, Professor, Department of Statistics, Institute of
Food and Agricultural Sciences, University of Florida, Gainesville, FL 

Human Studies Review Board Staff

**Paul I. Lewis, PhD, Executive Director, Human Studies Review Board
Staff, Office of the Science Advisor, United States Environmental
Protection Agency, Washington, DC 

Not in attendance at October 20-21, 2009 Public Meeting

**Service ending date was November 21, 2009. As of November 22, 2009,
the DFO is Jim Downing, Office of the Science Advisor, United States
Environmental Protection Agency, Washington, DC

INTRODUCTION 

From October 20-21, 2009, the United States Environmental Protection
Agency’s (EPA or Agency) Human Studies Review Board (HSRB) met to
address scientific and ethical issues concerning: two completed human
toxicity studies involving one class of pesticide active ingredients –
pyrethrins/pyrethroids– conducted prior to publication of the EPA’s
expanded final rule for protection of subjects in human research. In
accordance with 40 CFR 26.1602, EPA sought HSRB review of these
completed studies. Each of these completed studies is discussed more
fully below.

In addition, the Agency submitted two protocols for conducting new
research involving human subjects: one study measuring the efficacy of
two registered insect repellents containing picaridin against ticks
under laboratory conditions; and one study measuring levels of exposure
received by janitorial workers when applying a commercially-available
antimicrobial pesticide formulated as an aerosol spray. In accordance
with 40 CFR 26.1601, EPA sought HSRB review of these two proposed
protocols. Each of these protocols is discussed more fully below.

REVIEW PROCESS

On October 20-21, 2009, the Board conducted a public face-to-face
meeting in Arlington, Virginia. Advance notice of the meeting was
published in the Federal Register as “Human Studies Review Board;
Notice of Public Meeting” (74 Federal Register 106, 26861).

Following welcoming remarks from Agency officials, the Board heard
presentations from EPA on the following topics: two completed studies
involving intentional human exposure to pyrethrin/pyrethroid pesticides
(Newton and Breslin [1983], and Lisi [1992]); a proposed Carroll-Loye
Biological Research, Inc. (CLBR) insect repellent efficacy study
(LNX-003); and a proposed Antimicrobial Exposure Assessment Task Force
II (AEATF II) aerosol spray application protocol (AEA04).

The Board also asked clarifying questions of several study sponsors
and/or research investigators, including:

Dr. Scott Carroll, Principal, Carroll-Loye Biological Research

Dr. Jeffery Driver, Principal, infoscientific.com, Inc.

Mr. Shawn King, Director of Operations, Carroll-Loye Biological Research

Dr. Bryce Landenberger, Dow AgroSciences

Dr. Sami Selim, President, Golden Pacific Laboratories

Oral comments were provided by: 

Dr. Scott Carroll, Principal, Carroll-Loye Biological Research

Mr. Stephen McFadden, Independent Advocate

Dr. Lawrence Plumlee, Independent Advocate

No written public comments were provided.

For their deliberations, the Board considered the materials presented at
the meeting, oral comments, and Agency background documents (e.g.,
published literature, sponsor and investigator research reports, study
protocols, data evaluation records, and Agency science and ethics
reviews of proposed protocols and completed studies). A comprehensive
list of background documents is available online at
http://www.regulations.gov. 

CHARGE TO THE BOARD AND BOARD RESPONSE

Assessment of Completed Research Study: Newton, J., Breslin, A. (1983)
Asthmatic reactions to a commonly used aerosol insect killer. Medical
Journal of Australia 1:378-380.

Overview of the Study

	In the Newton and Breslin (1982) study, seven subjects diagnosed with
asthma and a history of chest tightness were evaluated for airway
narrowing and chest tightening before and after a controlled exposure to
aerosol insecticide sprays. Subjects were exposed to a
commercially-available aerosol insecticide (Mortein Pressure Pak insect
killer containing 3.0 g/kg pyrethrins, 0.9 g/kg tetramethrin, 15 g/kg
piperonyl butoxide, 7.5 g/kg N-octyl-bicycloheptene dicarbons,
propellants [chlorofluorocarbons and hydrocarbons] and solvents
[non-water based]). The aim of the study was three-fold: to study the
response of asthmatic  to exposure to insecticides containing pyrethrins
and tetramethrin; to study the time-course of exacerbation of asthma
following exposure to the insecticide; and to evaluate bronchial
reactivity to histamine after exposure to the insecticide. Data were
reported as simple counts or proportions across exposure groups.

Science

Charge to the Board

	Is the Newton and Breslin (1983) study scientifically sound, providing
reliable data?

Board Response to the Charge

  In light of the substantial limitations of the study, the Board
concluded that the study was of limited utility. The Board recommended
that the Agency be cautious in its use of these data, limiting it to a
qualitative contribution to the overall weight of evidence. The use of
these data should also be limited to assessments of the specific
insecticide product tested rather than pyrethrins/pyrethroids as a class
of pesticides.

Charge to the Board

	Is the Newton and Breslin (1983) study relevant to an assessment of the
proposition that exposures to pyrethrins/pyrethroids may be associated
with asthmatic or allergic respiratory responses?

Board Response to the Charge

 		

	The Board concluded that the Newton and Breslin study was relevant only
when considering the asthmatic or allergic respiratory response for the
specific insecticide product that was tested. It was not relevant when
considering the asthmatic or allergic respiratory response for
pyrethrins/pyrethroids as a class because of the use of a complex
mixture represented by an insecticide product rather than a specific
chemical.

Charge to the Board

	What limitations of the Newton and Breslin (1983) study should be taken
into account by EPA in assessing the proposition that exposures to
pyrethrins/pyrethroids may be associated with asthmatic or allergic
respiratory responses?

Board Response to the Charge

	The Board cautioned against using the Newton and Breslin study for
assessing the proposition that exposures to pyrethrins/pyrethroids may
be associated with asthmatic or allergic respiratory responses. The use
of a complex mixture represented by an insecticide product rather than a
specific chemical was of considerable concern, as was the more robust
response for respiratory irritation rather than asthma or allergy
reported in the study. 

HSRB Detailed Recommendations and Rationale

	The Newton and Breslin (1982) article provided an incomplete record of
the conduct of the study and analysis of the data, such that a number of
important data gaps were identified. 

	One primary concern was that the tested material contained a mixture of
ingredients, some of those ingredients at higher concentrations than the
pyrethrins. One such ingredient, piperonyl butoxide, has been
identified previously as a potential respiratory allergen (c.f. FDA
warnings on pediculicide products, 21 CFR § 358.601). Accordingly, the
Board concluded that the data obtained by Newton and Breslin are
directly applicable only to the product tested, and are not broadly
applicable to pyrethrins/pyrethroids as a pesticide class.  

	Other major issues and limitations noted during the Board’s
discussion included: 

1.	Limited and likely inaccurate exposure data.

a. 	Exposures to the test material were not directly measured, but were
calculated based on the assessment of the amount of product delivered
per unit time. Study subjects were likely to have been exposed to the
experimental product, but the lack of data on the actual exposure rate
or concentration precludes any quantitative evaluation.

 

b.	The Newton and Breslin article provides a description that is
inadequate to assess whether the facility was appropriate for this type
of testing. The test chamber was described as a 7-m3 room with large
windows on one side for observation purposes and an exhaust fan that
“removed fumes between sprays.” There is no indication, however,
that there was a standardized protocol that governed spraying procedures
or cleaning requirements between exposures. There also was no
indication whether the air exchange rate of the room or cleaning
procedures between exposures were sufficient to ensure that the exposure
levels intended by the investigators were delivered. Thus, there may be
a high degree of error in the estimated exposures, as compared to the
actual exposures.

c.	There was no mention of factors that may have affected actual
exposures, such as the distance between participant and spray nozzle,
spray droplet size, or any odorants or deodorants used to “mask” the
product spray from the placebo.

 

2.	Small sample size.

a.	Although study subjects were noted to have a predisposition for
asthma, and thus likely to be potentially susceptible to showing an
effect, only seven subjects were evaluated.

b.	There is insufficient descriptive information for the seven subjects
chosen, such as parameters of respiratory function and disease state. It
was also noted that the two oldest subjects were both male, but no
consideration whether age, gender or other participant characteristics
may have affected the study results.

3.	Incomplete results and data analysis. 

a.	Two subjects did not complete the pretest histamine challenge, and
three subjects did not complete the post-exposure challenge. Maximum
mid-expiratory flow rate (MMEFR) was not measured for one subject. More
notable is the fact that the only subject with a significant decrease in
forced expiratory volume in one second (FEV1) did not complete either
histamine challenge evaluation. No explanation for subjects’ failure
to complete these study procedures and tests was provided.

b.	There are limited details on how FEV1 and MMEFR values were reported.
  It was noted during the Board’s discussion that standard clinical
practice is to report the maximum values, but it is unclear if this was
done in the Newton and Breslin study.  

c.	There was no indication that baseline FEV1 or MMEFR values were
within the normal ranges for these subjects, nor any additional detail
on participant characteristics that could affect these values, such as
age, gender, body size, or underlying disease.

d.	There was no specific designation of the symptoms that each patient
experienced.   It is not clear which patient was the person not counted
among the “6 of 7” who experienced chest tightness with other
symptoms.

e.	There was no specific indication whether and how the observation of
respiratory irritation correlated with changes in FEV or MMEFR values.

f.	It is unclear which subjects were tested at which levels of exposure
(e.g. 10 versus 20 seconds), how many times each person was exposed at
each exposure duration, or which volunteers received the placebo on day
2.

 

4.	Potential information bias.

a.	 Subjects were not blinded to treatment, and there was no description
of how symptoms were self-diagnosed and reported by volunteers. Without
the use of standardized definitions as a reference, there is no way to
know whether each subject had the same interpretation and reporting
practices for self-diagnosed respiratory symptoms.

Ethics

Charge to the Board 

Is there clear and convincing evidence that the conduct of the Newton
and Breslin (1983) study was fundamentally unethical, or significantly
deficient relative to the standards of ethical research prevailing when
it was conducted?

Board Response to the Charge 

	Given the limited information available about this study, the Board
concurred with the Agency’s assessment (Sherman 2009a) that there was
neither clear and convincing evidence that the study was fundamentally
unethical, nor clear and convincing evidence that the study was
significantly deficient relative to the ethical standards prevailing at
the time the research was conducted.

HSRB Detailed Recommendations and Rationale 

The Newton and Breslin study was conducted in Australia in 1981 or 1982.
The Board concurred with the Agency’s assessment that most widely
recognized ethical standard at that time was the 1975 Declaration of
Helsinki, but also noted the importance of placing ethical standards
within a specific national context. The Australian National Health and
Medical Research Council (NHMRC) -- the largest public supporter of
research in Australia -- issued guidelines as early as 1966 that
required all NHMRC-funded research to be reviewed by an independent
ethics committee. A 1976 supplementary note to these guidelines,
prompted by and explicitly referencing the Declaration of Helsinki,
called upon all Australian research facilities to establish committees
to review all research involving human , NHMRC-funded or otherwise. 

Limited information is available about how the Newton and Breslin study
was conducted. The Agency attempted to obtain additional information
about the conduct of this study, but was unable to do so. The only
available information is from the published article, which states that
the written informed consent of all subjects was obtained. The article
also indicates that all subjects were adults, and that none of the
female volunteers were pregnant. Although the article does not describe
the type of ethical oversight of this research that took place, it is
assumed that independent ethics review occurred in accordance with NHMRC
guidelines.

Considering the information available, the Board concurred with the
conclusions and factual observations of the ethical strengths and
weaknesses of the study, as detailed in the Agency’s Ethics Review
(Sherman 2009a). The Board also concluded that this study met all
applicable ethical requirements for such research involving humans, as
required by the Agency’s expanded human studies rule, according to the
following criteria: 

a. 	Not fundamentally unethical. When determining whether or not a study
is fundamentally unethical, the Board’s standard approach is to decide
if the research was intended to seriously harm human subjects, if it
failed to obtain informed consent, or if it was fundamentally unethical
for other reasons. In this case, there was no evidence that the study
was intended to seriously harm subjects. The published article also
failed to provide evidence supporting a conclusion that the informed
consent of subjects was not obtained. Given lack of clear and convincing
evidence that for any other reasons it might have been fundamentally
unethical, the Board concluded that it was not fundamentally unethical.

b. 	Not significantly deficient. With regard to determining whether a
study was significantly deficient relative to the ethical standards
prevailing at the time the research was conducted, the Board’s
standard is to determine whether or not any ethical deficiencies
identified could have resulted in serious harm to human subjects, based
on knowledge available to researchers at the time the study was
conducted, or whether the information provided to study subjects could
seriously impair informed consent. As noted above, there is limited
information available regarding this study. Only the published article
provides any information, and the information contained therein fails to
support a conclusion that the study was significantly deficient relative
to the standards at the time.

Assessment of Completed Research Study: Lisi, P. (1992) Short
Communication: Sensitization risk of pyrethroid insecticides. Contact
Dermatitis 26:349-350. 

Overview of the Study

	In the Lisi (1992) study, 230 volunteers were patch tested with each of
seven pyrethroids: allethrin, cypermethrin, deltamethrin, fenothrin
(assumed phenothrin), fenvalerate, permethrin, resmethrin. The goal was
to establish the irritation and sensitization potential of these widely
used pyrethroids. Of the volunteers who participated, 162 were men and
68 were women.  Subjects ranged from 19 to 78 years in age.
Approximately 35% of the subjects were agricultural workers, 12% had
previously worked on a farm, and the remaining 53% were engaged in
nonagricultural activities. Fifty-four subjects had irritant or allergic
contact dermatitis of the hands, 18 of which were correlated with
agricultural activities. Further, 176 subjects had non-allergic skin
disorders, and 16 subjects were also atopic. Simple counts and
proportions were used to compare irritation and sensitization effects
across the three occupational groups. 

Science

Charge to the Board

	Is the Lisi (1992) study scientifically sound, 	providing reliable
data?

Board Response to the Charge

	The Board concluded that the Lisi study was likely sound, but the
brevity of the report and limited details as to how the study was
conducted prevent an accurate assessment of its reliability.  

Charge to the Board

	Is the Lisi (1992) study relevant to an assessment of the proposition
that exposures to pyrethrins/pyrethroids may be associated with allergic
contact dermatitis or sensitization responses?

Board Response to the Charge

	If the Lisi study was carried out according to accepted guidelines and
criteria, it likely provides hazard assessment data that are relevant to
assessing whether exposures to pyrethrins/pyrethroids may be associated
with allergic contact dermatitis or sensitization responses.

	

Charge to the Board

	What limitations of the Lisi (1992) study should be taken into account
by EPA in assessing the proposition that exposures to
pyrethrins/pyrethroids may be associated with allergic contact
dermatitis or sensitization responses?

Board Response to the Charge

	A lack of experimental detail and subject history represent two major
limitations of the Lisi study. The Board also noted that the
allergic/irritant response rate was extremely low in the study. The low
response rate, along with concerns about participant history and
selection, study procedures, and outcome definitions, should be taken
into account when considering these data in the overall weight of
evidence analysis.

HSRB Detailed Recommendations and Rationale

The extremely brief nature of the report raised a number of questions
about how the study was conducted. Because the study was reportedly
performed according to standards and criteria established by the
International Contact Dermatitis Research Group (ICDRG), the Board
assumed that standard testing procedures were used.  

	Major issues and limitations noted during the Board’s discussion
included:  

1.	Poorly defined participant population.  

a.	It was unclear how the 230 were chosen for study.  

b.	Subjects were identified as “agricultural workers,”
“ex-agricultural workers,” or “other,” but there was no
explanation for creating and using these categories.

c.	Participant compliance with study procedures was not reported. There
was no indication as whether any withdrew or were otherwise not
observed, nor did the report provide any detail whether all test patches
were evaluated for all on days 2 and 3.

2.	Lack of protocol detail.

a.	ICDRG protocols may have been used, but the report provides few
details on how investigators distinguished allergic versus irritant
responses or any information on how they graded response severity. 

3.	Low response rate.

a.	The low response rate (less than 2% of all study) suggests that the
seven compounds tested pose little risk of allergic contact dermatitis
or skin sensitization. However, these data could also be indicative of
problems in study design and conduct that invalidate the results.
Because the appropriate positive and negative controls were not included
in the study design, the Board felt that it could not rule out this
alternative explanation for the low response rate.

Ethics

Charge to the Board

Is there clear and convincing evidence that the conduct of the Lisi
(1992) study was fundamentally unethical, or significantly deficient
relative to the standards of ethical research prevailing when it was
conducted?

Board Response to the Charge 

Given the extremely limited information available about this study, the
Board concurred with the Agency’s assessment (Sherman 2009b) that
there was neither clear and convincing evidence that the study was
fundamentally unethical, nor clear and convincing evidence that the
study was significantly deficient relative to the ethical standards
prevailing at the time the research was conducted.

HSRB Detailed Recommendations and Rationale 

The Lisi study was conducted in Italy sometime in 1990 or 1991. The
Board concurred with the Agency’s assessment that the most widely
recognized ethical standard at that time was the 1989 Declaration of
Helsinki, but again noted the importance of placing ethical standards
within a specific national context. Italy had established a National
Bioethics Committee in 1990. That Committee may have developed
directives applicable to this study. Nevertheless, as explained below,
the potential existence of those national standards does not alter the
Board’s analysis. 

Extremely limited information is available about how the Lisi study was
conducted. The EPA attempted to obtain additional information about the
design and conduct of the study, but was unable to do so. The only
available information is from the published article, which does not
state whether written informed consent of the subjects was obtained. The
published article also does not address what type of ethical oversight
took place (e.g. review by an independent ethics committee). Study
subjects were reported to be adults, and there is no indication that any
of the subjects were pregnant or nursing. Although the EPA ethics review
states that “there is no evidence suggesting that any subjects came
from an especially vulnerable group,” several of were current and
former agricultural workers. Such may be socioeconomically or
educationally disadvantaged, and thus may be vulnerable. The Board
recommended that the Agency more clearly articulate or standardize its
definition of vulnerability.

Considering the information available, the Board concurred with the
conclusions and factual observations of the ethical strengths and
weaknesses of the study, as detailed in the Agency’s Ethics Review
(Sherman 2009b). The Board also concluded that this study met all
applicable ethical requirements for such research involving human, as
required by the Agency’s expanded human studies rule, according to the
following criteria: 

a. 	Not fundamentally unethical. When determining whether or not a study
is fundamentally unethical, the Board’s standard approach is to decide
if the research was intended to seriously harm, if it failed to obtain
informed consent, or if it was fundamentally unethical for other
reasons.  In this case, there was no evidence that the study was
intended to seriously harm. The published article also failed to provide
evidence supporting a conclusion that the informed consent of was not
obtained. Given the lack of clear and convincing evidence that for any
other reasons it might have been fundamentally unethical, the Board
concluded that it was not fundamentally unethical.

b. 	Not significantly deficient. With regard to determining whether a
study was significantly deficient relative to the ethical standards
prevailing at the time the research was conducted, the Board’s
standard is to determine whether or not any ethical deficiencies
identified could have resulted in serious harm to subjects, based on
knowledge available to researchers at the time the study was conducted,
or whether the information provided to study subjects could seriously
impair informed consent. Only the published article provides any
information, and the information contained therein fails to support a
conclusion that the study was significantly deficient relative to the
standards at the time.

Assessment of Proposed Carroll-Loye Biological Research Study LNX-003:
Efficacy Test of KBR 3023 (Picaridin; Icaridin) - Based Personal Insect
Repellents (20% Cream and 20% Spray) with Ticks Under Laboratory
Conditions.

Overview of the Study

The protocol describes a study to measure the effectiveness of picaridin
as a tick repellent when used in one of two compound formulations (20%
picardin KBR 3032 All-Family Insect Repellent Cream and 20% picaridin
KBR 3023 All-Family Insect Repellent Spray). Dosimetry data accumulated
in previous Carroll-Loye studies (LNX-001 and LNX-002) would be used for
dose selection. The efficacy of picaridin as a tick repellent will be
determined in a controlled laboratory setting by placing both Western
black-legged ticks (Ixodes pacificus) and American dog ticks
(Dermacentor variabilis) on picaridin-treated and untreated forearms and
measuring the speed and distance that moving ticks would penetrate into
the treated area at 15-minute intervals. A total of 20 subjects will be
enrolled.

Science

Charge to the Board

	If the proposed laboratory tick repellency study protocol LNX-003 is
revised as suggested in EPA’s review and if the research is performed
as described, is the research likely to generate scientifically reliable
data, useful for assessing the efficacy of the tested materials in
repelling ticks? 

Board Response to the Charge

	

	The Board concluded that the protocol submitted for review, if modified
in accordance with Agency recommendations and conducted accordingly,
will likely yield scientifically valid results on the efficacy of these
two picaridin-based insect repellent formulations against ticks.

HSRB Detailed Recommendations and Rationale 

	

	Protocol LNX-003 from Carroll-Loye Biological Research (Carroll 2009a,
2009b) will be conducted using methods similar to those presented to and
commented on by the Board in the past. Although the study protocol was
overly long and includes redundant or unnecessary text, it was
relatively clear and addressed adequately a number of key scientific
issues, including: scientific justification, objectives, and data
collection and compilation methods.

	The proposed methods largely follow EPA’s guidelines, with the one
notable exception being the use of ten volunteers per study aim, rather
than the Agency’s existing recommendation of six volunteers per study
aim. The greater number of study subjects should yield more useful
information than might otherwise be obtained. The protocol also
incorporated the use of dosimetry-generated data, which will likely
generate data representative of real-world use by consumers.

	As has been pointed out previously in Board reviews of other repellency
protocols, the proposed statistical approach fails to account for
censoring of data and the calculation of complete protection time is not
the best end-use of the study data. Calculating the proportion of
individuals protected for a given time may be a better way to report
this type of data and should be considered by the Agency.

Ethics

Charge to the Board 

	If the proposed laboratory tick repellency study protocol LNX-003 is
revised as suggested in EPA’s review and if the research is performed
as described, is the research likely to meet the applicable requirements
of 40 CFR part 26, subparts K and L? 

Board Response to the Charge

	The Board concluded that the proposed laboratory tick repellency study
protocol LNX-003, if modified in accordance with EPA (Sherman and
Sweeney 2009) recommendations, and performed as described, will likely
meet the applicable requirements of 40 CFR 26, subparts K and L. 

HSRB Detailed Recommendations and Rationale 

	The submitted documents assert that the study will be conducted in
accordance with the ethical and regulatory standards of 40 CFR 26,
Subparts K and L, as well as the requirements of US EPA’s GLP
Standards described at 40 CFR 160, and the California Department of
Pesticide Regulation study monitoring (California Code of Regulations
Title 3, Section 6710) (Carroll 2009a, 2009b). Requirements of FIFRA
§12(a)(2)(P) also apply. The protocol was reviewed and approved by an
independent human subjects review committee, Independent Investigational
Review Board, Inc. (IIRB, Inc.), of Plantation, FL, prior to submission.
Minutes of IIRB, Inc. meetings and a copy of IIRB, Inc. policies and
procedures were provided to the EPA as a separate document (IIRB, Inc.
2009). These documents indicate that IIRB, Inc. reviewed this protocol
pursuant to the standards of the Common Rule (45 CFR Part 46, Subpart
A).

1. 	The Board concurred with the conclusions and factual observations of
the ethical strengths and weaknesses of the study, as detailed in the
EPA’s Ethics Review (Sherman and Sweeney 2009). The proposed study is
likely to meet the applicable ethical requirements for research
involving human subjects, in accordance with the following criteria:

a.	Acceptable risk-benefit ratio. The risks as noted in the study
protocol are fivefold: 1) allergic reaction to test materials
themselves; 2) exposure to biting arthropods; 3) possible exposure to
arthropod-borne diseases; 4) physical stress from the test conditions;
and 5) psychological stress and/or breach of confidentiality for
pregnancy test results. These risks are minimized appropriately and are
justified by the potential societal benefits, particularly data on the
efficacy of these new formulations as personal tick repellents.

Based on toxicological data currently available for picaridin, coupled
with appropriate exclusion criteria, study subjects are unlikely to be
at risk of adverse side effects with exposure.

The risk of bites is negligible and minimized by the study design; tick
questing and biting behavior is slow, and study subjects are trained to
remove ticks from their forearms prior to biting. Study subjects will be
trained in proper tick observation and handling techniques.

The ticks used for the study are bred and raised in a laboratory
environment and are considered to be pathogen-free, minimizing the risk
of vector-borne disease. Tick colonies and their rabbit hosts are also
screened regularly for known tick-borne diseases, including the
rickettsial illness Rocky Mountain Spotted Fever that has been observed
in the past to be transmitted within laboratory tick colonies through a
trans-ovarian mechanism.

The potential risks to subjects from physical stress are minimized.
Although the 12-hour duration of the study protocol raises some concerns
about physical stress and exhaustion, the study investigators attest
that similar protocols of equivalent length have never been seen as
unduly stressful by study subjects. Appropriate stopping rules and
medical management procedures are in place.  Subjects are also given
frequent breaks and can withdraw from the study at any time should the
investigational procedures prove too strenuous.

Minors and pregnant or lactating women are excluded from participation,
with pregnancy either confirmed by over-the-counter pregnancy testing on
the day of study or by opt-out. The potential stigma resulting from
study exclusion due to pregnancy is also appropriately minimized.

b.	Voluntary and informed consent of all 

The study protocol includes several mechanisms designed to minimize
coercive recruitment and enrollment. For example, although many of the
research subjects will be recruited from the University of California at
Davis student population, where Dr. Carroll holds an adjunct
appointment, student and employees of the Study Director are excluded
from participation. Additional mechanisms designed to minimize coercive
recruitment, developed in response to earlier HSRB concerns and
recommendations (c.f. EPA HSRB 2006a; 2006b) are also in place.

Monetary compensation is not so high as to unduly influence study
subjects.

c.	Equitable selection of study 

The majority of research subjects will be recruited from the University
of California at Davis student population. Study subjects are likely to
reflect the ethnic and racial diversity of individuals in the City of
Davis, but the use of this convenience sample may limit the broad
applicability of the study results to the general population. The
investigators have noted this fact in the protocol.

Assessment of Proposed AEATF II Scenario and Protocol AEA04: Research on
Exposure of Janitorial Workers Applying Pesticides Formulated as Aerosol
Sprays.

Overview of the Study

	AEATF II aerosol spraying scenario is designed to measure a typical
occupational handler’s daily exposure to an antimicrobial spray
packaged in a pressurized aerosol spray can. These data will be used
generically to estimate dermal and inhalation exposures and risks for
other antimicrobial ingredients where the product is packaged in a
pressurized aerosol spray can. The Task Force expects these data to be
useful for estimating exposures for various types of aerosol spray
products, different types of surfaces, different room configurations,
different types of buildings, different handlers, and various
antimicrobial active ingredients. Eighteen professional janitors will be
enrolled in the study, and asked to apply (spray, but not wipe) the
product at one of three motels in the Fresno, CA area. Study subjects
will be randomized to apply different amounts of product, from 1 to 4
cans of product in 1/2-can increments (i.e., 1 to 1.5 cans, 1.5 to 2
cans, up to 3.5 to 4 cans). Dermal and inhalation exposure will be
measured using whole-body dosimeters (inner and outer), hand and face
washes, and personal air monitors.

Science

Charge to the Board

	If the proposed AEATF II aerosol application scenario and field study
protocol AEA04 is revised as suggested in EPA’s review and if the
research is performed as described, is the research likely to generate
scientifically reliable data, useful for assessing the exposure of
handlers who apply antimicrobial pesticides formulated as aerosol
sprays? 

Board Response to the Charge  

The Board concluded that this protocol, if modified in accordance with
EPA (Leighton, Walls and Sherman 2009) and HSRB recommendations will
likely generate scientifically reliable data, useful for assessing the
exposure of handlers who apply antimicrobial pesticides formulated as
aerosol sprays.

The Board also recommended a number of protocol modifications, as listed
below. Additional Board review of the protocol is not required prior to
study implementation.

HSRB Detailed Recommendations and Rationale

The Board concluded that the protocol (AEATF II 2009a, 2009b) and
supplemental SOPs (AEATF II 2009b) address adequately a number of key
scientific issues, including: scientific objective, quantification of
the test materials, data collection and compilation methods and summary
of test results, justification for selection of the test substances, and
QA/QC requirements. The process used to select the product to be tested
seems rational. The AEATF II has described in detail their sampling
design for the aerosol scenario, and has incorporated random elements
where feasible. The aerosol scenario is well defined, and the diversity
of daily exposures under the aerosol scenario as defined in this
proposal is likely to describe adequately a typical occupational
handler’s daily exposure to this antimicrobial application. The
environmental monitoring, exposure monitoring and analytic techniques
appear to be adequate.   The variation in exposure should be sufficient
to determine a distribution of exposures.

	The Board did raise a number of concerns about perceived inadequacies
in the study design, as summarized below:

Use of the results: EPA plans to use the data generated from the
proposed aerosol study generically to estimate dermal and inhalation
exposures and risks for other antimicrobial ingredients where the
product is packaged in a pressurized aerosol spray can. However, other
variables can affect rates of exposure, including different nozzle
sizes, spray and ejection rates, the size of the particles generated,
and the generation of nonvolatile active ingredients. Inhalation versus
dermal exposure may also vary with these variables.

It is also unclear if occupational handler exposure can be used to
estimate exposures of nonprofessionals using similar consumer products.
However, for risk assessment purposes it was felt that exposure among
occupational users is likely to exceed that of consumers; the higher
frequency of exposures among occupational users is likely to exceed a
plausibly higher individual but less frequent exposure among
nonprofessional users. 

2. 	Sample Size & Analysis: Raw and descriptive data will be provided,
but no statistical analyses have been planned. Sample size adequacy
cannot be judged without a statistical analysis plan. Placing 18
subjects into three clusters of six each requires consideration of
cluster effects, and may complicate analysis of the results. Assumed
constants related to exposure to the active ingredient may also prove to
be incorrect, and additional subjects may need to be enrolled.

3.	Application of spray: The target number of cans sprayed in each
location by each worker on each day could be specified better within the
protocol. For example, does a worker need to continue spraying a room if
the target application is met before completion of that room? Additional
protocol clarifications needed include: addressing the issue of the
interval between sampling at different sites (currently, it appears that
no two monitoring events can occur within the same building within one
week); clarifying the issue of using empty apartments rather than motel
rooms; and clarifying the effect of exhaust vent effects on exposure,
which could be different than indicated in the protocol. 

	Quality assurance and control (QA/QC): QA/QC can be improved by: 

a. 	Determining likely differences in the air sampling results depending
on the method used (e.g. depending on the air sampling equipment used,
differences in the orifice diameters and air flow sampling rates could
affect aerosol collection efficiencies and effects).

b. 	Setting the minimal spike to two to four times that of the limit of
quantitation (LOQ; cf. AEATF II 2009b: 42, 46).

c. 	Considering other variables that may influence the measurement of
exposure, including whether the surface being treated is dry or wet at
the time of the next spray application, and whether the applicant
accidentally wiped the surface after spraying. Data accumulated during
gross deviations from the protocol should also be excluded. Finally, the
protocol should indicate that there will be a maximum of two workers on
any given day in the same location. 

Ethics

Charge to the Board

	If the proposed AEATF II aerosol application scenario and field study
protocol AEA04 is revised as suggested in EPA’s review and if the
research is performed as described: Is the research likely to meet the
applicable requirements of 40 CFR part 26, subparts K and L?

Board Response to the Charge

	The Board concluded that the protocol submitted for review, if modified
in accordance with EPA (Leighton, Walls and Sherman 2009) and HSRB
recommendations, is likely to meet the applicable requirements of 40 CFR
26, subparts K and L.

HSRB Detailed Recommendations and Rationale

	This is a protocol for third-party research involving intentional
exposure of human subjects to a pesticide, with the intention of
submitting the resulting data to EPA under the pesticide laws (AEATF II
2009a, 2009b, 2009d).  Thus, it is subject to the ethical and regulatory
standards of 40 CFR 26, Subparts K and L. The requirements of FIFRA
§12(a)(2)(P) relating to informed, voluntary consent and those of the
California Department of Pesticide Regulation study monitoring
(California Code of Regulations Title 3, Section 6710) also apply. The
protocol was reviewed and approved by an independent human subjects
review committee, Independent Investigational Review Board, Inc. of
Plantation, Florida, prior to submission (AEATF 2009b, 2009c).

1. 	Except as noted below, the Board concurred with the conclusions and
factual observations of the ethical strengths and weaknesses of the
study, as detailed in the EPA’s Ethics Review (Leighton, Walls and
Sherman 2009). The proposed study is likely to meet the applicable
ethical requirements for research involving human subjects, in
accordance with the following criteria:

a.	Acceptable risk-benefit ratio. The risks from participation in the
study include: 1) discomfort and heat-related illness from wearing two
layers of clothing; 2) discomfort from wearing an air sampling device;
3) reactions from exposure to the test material (including accidental
exposure) or to solvents used to obtain residues from hands, face and
neck; 4) embarrassment from disrobing in the presence of a research
technician; and 5) psychological stress and/or breach of confidentiality
for pregnancy test results. These risks are minimized appropriately and
are justified by the potential societal benefits, particularly data for
new exposure assessments for antimicrobial products applied with
pressurized aerosol cans.

The surrogate materials consist of a commonly used commercial
antimicrobial product, which is known to have low risks. The subjects
will only be exposed to concentrations of the surrogate compound at
accepted exposure thresholds. Candidates who are known to be sensitive
to that compound, or who are in poor health, or who have broken skin on
their hands, face, or neck will be excluded.

Risk of heat-related illness is minimized appropriately. Testing will
take place in a temperature-controlled environment, and the heat index
will be monitored with associated stopping rule. There will be a limited
time of exposure with rest periods at 30-minute intervals, or more
frequently if requested. There also will be close observation of
subjects, with subjects alerted to the signs and symptoms of heat
stress.

Subjects will be reminded about safe handling practices and procedures,
wear appropriate personal protective equipment (PPE), and will be
monitored for any accidental or unintended product exposure.

 Minors and pregnant or lactating women are excluded from participation,
with pregnancy either confirmed by over-the-counter pregnancy testing on
the day of study or by opt-out. The potential stigma resulting from
study exclusion due to pregnancy is appropriately minimized.

The purpose of the study is to develop more accurate information on
worker exposures to liquid antimicrobial products applied using
pressured aerosol cans for indoor surface disinfecting. Although there
is no clear benefit to study participation, Subjects may benefit by
requesting their results and using these results to change their working
practices. Benefits and risks of participation are clearly articulated
in the protocol and informed consent documents.

Possible subjects for this study may be undocumented immigrants.
Recruitment materials should state that government-issued identification
is necessary for enrollment in this study.

A valid government-issued form of identification is necessary for
enrollment. Volunteers who lack proper identification are excluded from
study participation but no other action is taken, minimizing any social
risks to undocumented workers from disclosure of their status to
law-enforcement or immigration authorities.

b.	Voluntary and informed consent of all 

The informed consent documents clearly state that study subject can
withdraw at any time without penalty.

The Board disagrees with the EPA comment that “no potential subjects
are from a vulnerable population” (Leighton, Walls and Sherman 2009:
11). In fact, subjects in this study might represent particularly
vulnerable populations and thus might be susceptible to coercion or
undue influence. Some subjects may be educationally or economically
disadvantaged, or may not be native speakers of English. However, the
risk of coercion or undue influence is appropriately minimized as
follows:

Monetary compensation is roughly equivalent to the average daily wage of
janitorial workers in Fresno, California, and so is not so high as to
unduly influence the subjects.

Spanish translations of all informed consent documents, informational
packets, and recruitment flyers are available. If necessary, the
informed consent process will be conducted in Spanish by a member of the
research team fluent in that language.

The subjects communicate directly with the researchers about their
interest in enrolling in the study. Recruitment is not conducted through
employers.

c.	Equitable selection of study 

The study is designed to recruit an appropriately diverse population of
subjects who represent professional janitorial workers in the Fresno
County area. The researchers plan to place recruitment notices in three
newspapers that cater to the general population, the African-American
population, and the Latino population.

2.	The Board recommended that the study protocol be modified to address
the few concerns noted in the EPA’s Ethics Review (Leighton, Walls and
Sherman 2009). The Board also made the following additional
recommendations:

The Board recommended that the consent form be reviewed to make sure
that it is at an appropriate reading level for the proposed participant
population. In some instances, there appear to be word choices that
could be simplified (e.g., wording such as “non-porous surfaces”,
“pressurized aerosol can” instead of “spray can”, and assuming
that the volunteer knows what a Material Safety Data Sheet is).

The Board agrees with the EPA ethics review that the informed consent
form should read, “We will pay for needed medical treatment that is
not paid for by your own insurance or the insurance of a third party
under which you are covered.”

As some possible subjects for this study may be undocumented immigrants,
recruitment materials should more explicitly state that a valid
government-issued form of identification is necessary for enrollment. 

The Board recommended that the Task Force ensure that documents in
Spanish are reviewed by someone familiar with the dialects written and
spoken in the target community. The National Standards for Culturally
and Linguistically Appropriate Services in Health developed by the
Department of Health and Human Services, the Office of Minority Health
has stressed the importance of treating medical patients in a
“culturally and linguistically appropriate manner” (Office of
Minority Health 2001, 3). The same practice should be extended to
subjects in EPA- or third party-conducted research, with materials such
as informed consent forms and recruitment flyers written in ways that
reflect “the dialectic and cultural nuances as well as the
acculturation, educational, and literacy levels of the local
target population” (Office of Minority Health 2001, 80).

The protocol provides for use of a community notification flyer.
However, one neglected community includes persons who might be staying
in the hotels where the study is conducted. The flyer should be revised
so that it communicates the goals of the study and risks to that group.
These flyers, in both English and Spanish, should also be posted in
locations so that hotel guests are likely to see it. Alternatively, the
researchers should consider conducting the research in areas away from
hotel guests.

The exclusion criteria should be revised to eliminate some groups that
might be at higher risk of physical harm but are not presently excluded.
This might include subjects who might be immunosuppressed for a variety
of reasons, those with severe diabetes, and those with other conditions
that pose a health risk.

REFERENCES

Antimicrobial Exposure Assessment Task Force II (AEATF II). 2009a. AEATF
II Aerosol Study Vol. 1 Primary Documentation: Aerosol Application
Scenario Design. Dated August 4, 2009. Unpublished protocol prepared by
the AEATF II.

Antimicrobial Exposure Assessment Task Force II (AEATF II). 2009b. AEATF
II Aerosol Study Vol. 2 Primary Documentation: Study Protocol and IIRB
Approval and Documentation. Dated August 4, 2009. Unpublished protocol
and documentation prepared by the AEATF II.

Antimicrobial Exposure Assessment Task Force II (AEATF II). 2009c. AEATF
II Aerosol Study Vol. 3 Primary Documentation: IIRB Communications.
Dated August 4, 2009. Unpublished documentation prepared by the AEATF
II.

Antimicrobial Exposure Assessment Task Force II (AEATF II). 2009d. AEATF
II Aerosol Study Vol. 4 Standard Operating Procedures for a Multi-Year
Antimicrobial Chemical Exposure Monitoring Program. Dated August 4,
2009. Unpublished documentation prepared by the AEATF II.

Carroll, S. 2009a. Efficacy Test of KBR3023 (Picaridin; Icaridin)-Based
Personal

Insect Repellents (20% Cream and 20% Spray) with Ticks under Laboratory
Conditions. Dated July 30, 2009. Unpublished protocol prepared by
Carroll-Loye Biological Research. MRID 47836801.

Carroll, S. 2009b. Carroll-Loye Study LNX-003 Proposed Amended Language.
Dated September 18, 2009. Unpublished protocol amendment prepared by
Carroll-Loye Biological Research. 

EPA HSRB. 2006a. June 27-30, 2006 EPA Human Studies Review Board Meeting
Report.

EPA Human Studies Review Board. 2006b. October 18-19, 2006 Human Studies
Review Board Meeting Report.

Independent Institutional Review Board, Inc. (IIRB, Inc.). 2009.
Supplemental IIRB, Inc., Documentation Package. Unpublished materials
prepared by IIRB.

Leighton, T., C. Walls and K. Sherman. 2009. Science and Ethics Review
of AEATF II Aerosol Scenario Design and Protocol for Exposure
Monitoring. Dated September 21, 2009.  Unpublished document prepared by
Office of Pesticide Programs, United States Environmental Protection
Agency.

Lisi, P. 1992. Sensitization Risk of Pyrethroid Insecticides. Contact
Dermatitis 26:349-350. MRID 47852801.

Newton, J.G. and A.B. Breslin. 1983. Asthmatic Reactions to a Commonly
Used Aerosol Insect Killer. Medical Journal of Australia 1:378-380. MRID
47686201.

Office of Minority Health, Department of Health and Human Services.
2001. National Standards for Culturally and Linguistically Appropriate
Services in Health Care: Final Report. Washington, DC: U.S. Department
of Health and Human Services.

Sherman, K. 2009a. Ethics Review of Pre-Rule Intentional Exposure Human
Toxicity Study (Newton and Breslin 1983). Dated September 16, 2009. 
Unpublished document prepared by Office of Pesticide Programs, United
States Environmental Protection Agency.

Sherman, K. 2009b. Ethics Review of Pre-Rule Intentional Exposure Human
Toxicity Study (Lisi 1992). Dated September 16, 2009.  Unpublished
document prepared by Office of Pesticide Programs, United States
Environmental Protection Agency.

Sherman, K., and K. Sweeney. 2009. Science and Ethics Review of Protocol
for Human Study of Tick Repellent Performance. Dated September 21, 2009.
 Unpublished document prepared by Office of Pesticide Programs, United
States Environmental Protection Agency.

Page   PAGE  26  of   NUMPAGES  26 

Page   PAGE  1  of   NUMPAGES  26 

