                    UNITED STATES ENVIRONMENTAL PROTECTION AGENCY
                         WASHINGTON, D.C. 20460      

                                                	OFFICE OF CHEMICAL SAFETY AND 
                                                           POLLUTION PREVENTION
	



MEMORANDUM 

DATE: 	January 6, 2021

SUBJECT:	Supplemental Materials for The Product Performance Test Guidelines OCSPP 810.3300: The Efficacy of Topically Applied Pet Products Against Certain Invertebrate Pests

		
PC Code: NA
   DP Barcode: NA
Decision No.: 
   Registration No.: NA
Petition No.: NA
   Regulatory Action: NA
Risk Assessment Type: NA
   Case No.: NA
TXR No.: NA
   CAS No.: NA
MRID No.: NA
   40 CFR: NA
		 									Ver.Apr.08
	 	
FROM:	James Nguyen, Mathematical Statistician
      Chemistry and Exposure Branch                                                           
		Health Effect Division (7509P)	

THROUGH:	David J. Miller, Chief  
		Chemistry and Exposure Branch
		Health Effects Division (7509P)
			
TO: 		Autumn Metzger, Biologist
		Invertebrate &Vertebrate Branch 1 
		Registration Division 
			

Please find attached per request supplemental materials for the Product Performance Test Guidelines OCSPP 810.3300 which relate to the efficacy of topically applied pet products against certain invertebrate pests. 
Briefly, the attached supplemental materials were produced following the June 11-14, 2019 FIFRA SAP meeting peer review of EPA's proposed guideline for efficacy testing of topically applied pesticides used against certain ectoparasitic pests on pets. The SAP review included various comments about the sample size determination which had been presented to the SAP in the EPA document "Sample Size for Pet Product Studies". The attached Supplemental Materials address the issues, concerns, and questions raised by the SAP during this June 2019 FIFRA SAP meeting as well as their requests for additional information/analyses.    
Supplemental Materials for The Product Performance Test Guidelines OCSPP 810.3300: The Efficacy of Topically Applied Pet Products Against Certain Invertebrate Pests
                                       
                               Table of Contents


    Executive Summary......................................................................................3

    Simulations to determine sample sizes for pet product efficacy studies...........................4
   
    Conclusions and suggested sample sizes...............................................................9

    An example of data analysis of a pet product efficacy study......................................11

    Appendices...............................................................................................14

   Appendix I: 	Results of power vs. sample size simulations.......................................14
   
   Appendix II:	Mathematical derivation of the estimated efficacy using a binomial model 
                  with logit link function....................................................................18
                  
   Appendix III:	True blood-fed proportion vs. proportions of animals or datasets meet the required blood-fed proportion..........................................................20
   
   Appendix IV: 	Values of SD_sub and SD_day used in the simulations..............................24
   
   Appendix V: 	ICF's report of Evaluations of the applications of additional statistical 
                  methods for pet product studies .........................................................25
                  
   Appendix VI: 	SAS codes....................................................................................37
   
			SAS code of simulation to determine the lowest blood-fed proportions............37

            SAS code of power vs. sample size simulations.......................................39

            SAS code of an example data analysis...................................................46
   


    Executive Summary
   
This document is directly related to the guideline for Product Performance Test Guidelines OCSPP 810.3300: The Efficacy of Topically Applied Pet Products Against Certain Invertebrate Pests.  This supplemental document will (1) present more detail about how the sample sizes provided in the guideline were calculated and (2) provide an example (with SAS code) of data analysis for pet product efficacy studies using the statistical methods recommended in the guideline.
      
This document provides statistical simulations based on sections (j)  -  (l) of the draft guideline, which illustrate the potential power of several different experimental designs to detect efficacy of 90% or greater based on the behavior of different types of arthropods. Adequate power is typically considered 80% power to achieve an estimated effect at a given level of precision. For the studies in the guideline, EPA determined the sample size which results in 80% power to achieve an estimated efficacy of >=90% with a precision of <= 5%, given that the true efficacy of the product is 92.5%.  Specifically: 

 For studies assessing efficacy against ticks (study design 1), 11 animal subjects (if using 50 ticks per animal) would be appropriate per treatment.  
 For studies assessing efficacy against fleas, 7 animal subjects (with 100 fleas per animal) per treatment would be appropriate for tests on cats (study design 2), and 6 animal subjects (with 100 fleas per animal) per treatment would be appropriate for tests on dogs (study design 5). 
 For studies assessing efficacy against mosquitoes, 6 animal subjects (with 100 mosquitoes per animal) per treatment would be appropriate if testing against one species (study design 5), and 8 animal subjects (with 50 mosquitoes of each species per animal) per treatment if testing against three mosquito species simultaneously (study design 3).  
 For studies assessing efficacy against stable flies, horse flies, and deer flies, 8 animal subjects (with 50 pests per animal) per treatment (study design 3) or 12 animal subjects (with 25 pests per animal) per treatment (study design 4) would be appropriate. 
 For studies assessing efficacy against sand flies, black flies, or biting midges, 6 animal subjects (with 100 pests per animal) per treatment would be appropriate (study design 5).

Note that the suggested sample sizes above are based on the simulations where the delta approach is used in which an asymptotic (only appropriate for large sample size) standard error was used to obtain the 95% confidence interval of the estimated efficacy.  While the bootstrapping technique is more appropriate to obtain the 95% CI of the estimated efficacy in the small sample size studies, it is not practical to use bootstrapping technique in simulations to evaluate sample size and power given the required computing power and the time required to run all the simulations.  Therefore, the bootstrapping technique was used only for one scenario with small numbers of datasets to compare the results of power and sample sizes between simulations using the delta approach and simulations using bootstrapping technique.  The results show that the power from simulations using bootstrapping technique was slightly higher than that from the simulation using delta approach; however, there was no evidence that the sample sizes would substantially different  (Appendix V).

To demonstrate how data from a pet product efficacy study are analyzed, this document also provides an example of data analysis of a synthetic dataset, using the statistical methods suggested in the guidelines.  SAS codes of this data analysis are also provided.


    Simulations to determine sample sizes for pet product efficacy studies

Background

Purpose

Historically, the Agency has recommended using a minimum of 6 animals per study group for efficacy testing of topically applied products on pets, however studies have been submitted with 2-12 animals per group. Previously no statistical simulation or calculation had been conducted to evaluate the statistical power associated with the sample size of these experimental designs in detecting a treatment-related effect. Therefore, these simulations have been conducted to determine the sample sizes and associated power of the study designs in the proposed guideline. 

Study design

    2 groups (1 control and 1 treatment) with X animals per group
    Data are collected at 4 different days over the study duration
    Each day, Y arthropods (+-10%) per animal
Objective of study

    Evaluate the efficacy or effectiveness of the treatment where efficacy or effectiveness of the treatment is defined as Abbott's formula:
           (retained or blood-fed proportion of control group  -  retained or blood-fed proportion of treated group)/retained or blood-fed proportion of control group for repellency evaluation. 
           (Survival proportion of control group  -  survival proportion of treated group)/survival proportion of control group for mortality evaluation.
Objective of Simulations

Determine the sample size and the power to obtain an observed efficacy >= 90% with a given (suitable) pre-stated level of precision in studies to evaluate repellency of pet products.


Assumptions 
   
    The data from pet product efficacy studies are in the form of n events/N trials (e.g., n blood-fed pests/N pests used) and are reasonably assumed binomial distributions.
         
    In addition to the variation of the arthropods (described by the variation of binomial distribution), the difference between animals is also a factor influencing the variation in the data.  There may be similarities between measurements of the same animal compared to measurements of different animals.  This factor of variation in the data is called the "random animal effect".

    Also, there may be similarity between measurements from the different animals on the same testing day compared to measurements from different animals on different testing days.  The similarity between measurements on the same testing days is called the "random day effect".  Because the pet product studies are conducted in laboratorial environments, the random day effect is believed to be very minimal and is assumed to be zero in order to simplify the simulations.
         
    Another factor that may often occur in binomial distribution data is over- dispersion.  Over dispersion in binomial distribution data generally affects confidence intervals of the estimates.  Since the precision of the estimates is one of two criteria that significantly affects the power of a study and the regulatory decision-making  surrounding a product, it is important that the  sample size is adequate to  account for over-dispersion in the data.  Synthetic or artificial over-dispersion in the data was created by adding small additional variation (the parameter SD_day) to the variation of the binomial data of each animal on each day in the step of data generation. 
         
    The random effects of animal and SD_day are assumed to follow normal distributions.  

    The simulations are used to determine the sample size based on the efficacy of repellency (i.e., the simulated data are the counts of pests used and counts of blood-fed pests on the animals). However, for application in the guideline, the words "mortality" and "repellency" can be used interchangeably for each model when determining sample size. 

Parameters in simulations
   
The following are parameters included in the simulations: 

 NminSub = the smallest number of animals per group; 
 NmaxSub = the largest number of animals per group; 
 NPDay = number of arthropods per animal per day; 
 RNdiff = allowable proportion of deviation in number arthropods/animal; 
 CP = true blood-fed proportion of control group; 
 TE = true efficacy of treatment; 
 ReqTE = required observed efficacy of treatment; 
 ReqPrecision = required maximum precision value; 
 SubVar or SD_sub = degree of variation between animals; 
 DayVar or SD_day = degree of additional variation introduced to create the exhibiting of over dispersion in the binomial data; 
 Nsim = number of simulated datasets per scenario (set to be 2000); and 
 seed = seed to regenerate the results of simulations.
   
As the power of a study design is defined as the proportion of datasets that have the estimated efficacy >= 0.90 and the precision of the estimated efficacy <= ReqPrecision, the power is impacted by the values of parameters: 
   
    The required precision (ReqPrecision) of the estimates: a smaller value of required precision would result a smaller number of datasets that have the precision of the estimated efficacy less than the required precision. This means the study design would have lower power to achieve a lower uncertainty in estimated efficacy from the accepted studies.  
         
    Number of animals per group (ranges from NminSub to NmaxSub): larger number of animals in each group would result in better precision for the estimated efficacy and a larger proportion of datasets that have the precision of the estimated efficacy smaller than the required precision (i.e., higher power).

    Number of pests per animal (NPDay): larger number of pests per animal would result in better precision for the estimated efficacy and a larger proportion of datasets that have the precision of the estimated efficacy smaller than the required precision (i.e., higher power).  However, practical and humane reasons generally determine how large the number of pests per animal could be. 

    The proportion of blood-fed in the control population (CP): the variance of a binomial distribution becomes greater as the value of true proportion or rate approaches 0.5.  However, the proportions of blood-fed pests are determined by natural pest behaviors.  Based on the required level of the lowest observed proportion of blood-fed pests on an animal in order for the animal to be qualified to participate in the study, the proportion of blood-fed pests on the (qualified) animal population is estimated such that a very high proportion of the animals would be qualified to participate in the study and a low probability that a study would be rejected due to the control animals failing to meet the required lowest proportion of blood-fed pests.  (The true proportion of blood-fed pests are estimated using the SAS file "percent animals or datasets have at least a certain percent insect non-blood-fed.sas", Appendix III).

    The required observed efficacy of treatment (ReqTE): a smaller value of required observed efficacy would result a larger number of datasets that have the estimated efficacy greater than the required observed efficacy (i.e., higher power).
         
    True efficacy (TE) of the product: If the study can be replicated thousands of times, we would have a distribution of estimated efficacy values and expect half of the estimated efficacy values to be less than the true efficacy TE and half of the estimated efficacy values to be greater than the true efficacy TE.  When the true efficacy TE is exactly equal the required observed efficacy ReqTE, half of the estimated efficacy values will be >= the true efficacy value TE =ReqTE, and therefore the power of the study design will be <= 50%.  A greater difference between the true efficacy TE and the required observed ReqTE would result in a greater proportion of datasets with observed efficacy values >= the required observed ReqTE, and therefore a study with greater power.  This is why the selected true efficacy TE of a product has to be greater than the required observed efficacy ReqTE in order to have acceptable power for the studies.  
Data generation
   
Step 1: calculate the proportion of blood-fed in the treatment group (TP) 
   
From Abbott's formula:

    - 		TE = (CP - TP)/CP,

where TE is true efficacy and CP and TP are the proportions of blood-fed pests in the control group and treated group, respectively. 

Then the proportion of blood-fed in the treated group is calculated as:
      	
      TP =CP x (1  -  TE)

Step 2: generate number of pests used and number of blood-fed pests on each animal

Assume P is the true blood-fed proportion of a group on a specific day (P = CP for the control group and P=TP for the treated group).  The logit of the group is calculated as:

      Q = log(P/(1-P))
        
Then, the logit of the of animal s is generated as:

      Qs= Q + rands,
       
where rands is randomly generated from a normal distribution with mean = 0 and SD = 0.33.
   
Similarly, the logit of the animal s on a specific day i is generated as:

      Qsi = Qs + randi 
            = Q + rands + randi,
       
where randi is randomly generated from a normal distribution with mean=0 and SD=0.166.
     
The blood-fed proportion of pests on the animal s on the specific day i is 

      Psi = exp(Qsi)/(1+exp(Qsi)).
      
The number of arthropods used on animal s on day i is randomly generated from uniform distribution 

      Nsi = U [ NPDayx(1  -  RNdiff), NPDayx (1  -  Rndiff) ],
      
where NPDay is the target number of pests used on each testing day, and RNdiff is the allowed factor for the number of pests used on each animal to be deviated from the target number of pests
   
The number of blood-fed arthropods on animal s on day i is randomly generated from binomial distribution of Nsi trials and success proportion = Psi.
   
Step 3: generate 2000 datasets
   
For each dataset, step 2 above is repeatedly used to generate the numbers of pests used and numbers blood-fed on each animal for each of four testing days for n animals of each control and treated groups.
   
The process above is repeated to create 2000 datasets.
   
Analysis of simulated datasets
   
Generalized linear mixed effects models (SAS PROC GLIMMIX), with logit link function for binomial distribution data is used to analyze each of simulated datasets.  In the models, the animal is experimental subject (RANDOM Day/subject = animal type = cs;).  Over dispersion in the data is accounted in the model (RANDOM _residual_;).  For those datasets that have a computational issue (non-convergence) in the generalized linear mixed effects model (GLMM), a generalized linear model is used to separately analyze the data by each Day.  
   
The ratio of blood-fed proportions between treated and control groups and its 95% CI are estimated for each Day, using the delta approach (Appendix II).  The efficacy of the treatment group = 1  -  (blood-fed proportion of treated group/blood-fed proportion of control group).  The precision of the estimated efficacy = estimated efficacy  -  lower limit of 95% CI.
   
The power of a study design is the proportion of datasets that have observed treatment efficacy >= ReqTE = 0.90 and observed precision of the estimated efficacy <= ReqPrecision.

Results of power vs. sample size simulations

Results of simulations are presented in Appendix I.

    Conclusions and suggested sample sizes
   
Conclusions

Given the results of sample sizes, a required precision ReqPrecision = 0.05 is selected since a smaller value of 0.03 or 0.04 for the required precision parameter would require much larger sample sizes in the studies in order to obtain a sufficient power.  

A true efficacy value of 0.925 is selected since it is the lowest value such that the studies would still have acceptable power with reasonable sample sizes (not too large).  When the true efficacy of a product is 0.85 (i.e., less than the required efficacy of 0.90), there is a very low probability that the product would be incorrectly accepted (i.e., low probability that the estimated efficacy from a study of this product >= 0.90 and the precision <=0.05). When the true efficacy of a product is 0.925 (slightly greater than the required efficacy of 0.90), there is a high probability that the product would be correctly accepted (i.e., there is at least 0.80 power that the study would result an estimated efficacy >= 0.90 and a precision <=0.05) for reasonable sample size studies.

Note that there was a high proportion of datasets for which the GLMM model did not converge, and the generalized linear model is subsequently used to analyze those datasets for each separate Day.  Simulations using a GLMM model (with link function for binomial data) without overdispersion are performed for one scenario (PC = 0.60 and NPDay = 100).  The results from the simulations of this scenario indicate GLMM model without overdispersion significantly alleviates the non-convergence issue (only 1.59% of the datasets that do not converge).  The power results are quite similar to those from GLMM model with the option of overdispersion; therefore, the simulations with GLMM without overdispersion has not been replicated for all scenarios or presented in this document.  (Appendix V)

Suggested sample sizes

Below is the summary table including simulation results of all the scenarios.  EPA determined that the sample size which results in at least 80% power to detect the true efficacy of >=90% with a precision of <= 5% using the simulations where random effects were randomly generated from normal distributions and the data were analyzed using GLMM for binomial distributions for the studies in sections (k)-(m) of the guideline. Specifically:

 For studies assessing efficacy against ticks (study design 1), 11 animal subjects (with 50 ticks per animal) would be appropriate per treatment. 
 For studies assessing efficacy against fleas, 7 animal subjects (with 100 fleas per animal) per treatment would be appropriate for tests on cats (study design 2), and 6 animal subjects (with 100 fleas per animal) are appropriate for tests on dogs (study design 5). 
 For studies assessing efficacy against mosquitoes, 6 animal subjects (with 100 mosquitoes per animal) per treatment would be appropriate if testing against one species (study design 5), and 8 animal subjects (with 50 mosquitoes of each mosquito species per animal) per treatment if testing against three mosquito species simultaneously (study design 3).  
 For studies assessing efficacy against stable flies, horse flies, and deer flies, 8 animal subjects (with 50 pests per animal) per treatment (study design 3) or 12 animal subjects (with 25 pests per animal) per treatment (study design 4) would be appropriate. 
 For studies assessing efficacy against sand flies, black flies, or biting midges, 6 animal subjects (with 100 pests per animal) would be appropriate (study design 5). 

                                       
                       GLMM for Binomial with logit link
                                       
                                 Study design
     Required lowest observed blood-fed proportion on each control animal
                  True blood-fed proportion in control group
                         No. pests per animal per day
                                    True TE
                         Required Sample Size (power)
 Ticks on cats or dogs, (k)(2)
                                       1
                                     0.25
                                     0.45
                                     50+-5
                                     0.925
                                      11
                                    (0.806)
 Fleas on cats, (j)(2)
 
                                       2
                                     0.40
                                     0.60
                                    100+-10
                                     0.925
                                       7
                                    (0.858)
 Mosquitoes or flies on cats or dogs using 50 pests, (l)(2)
                                       3
                                     0.60
                                     0.80
                                     50+-5
                                     0.925
                                       8
                                    (0.822)
 Biting flies on animal subjects using 25 pests, (l)(2)
                                       4
                                     0.60
                                     0.80
                                     25+-2
                                     0.925
                                      12
                                    (0.815)
 
                                       
                                       
                                     0.85
                                     25+-2
                                     0.925
                                      12
                                    (0.820)
 Fleas on dogs, (j)(2) 
 Mosquitoes or flies for cats and dogs using 100 pests, (l)(2)
                                       5
                                     0.60
                                     0.75
                                    100+-10
                                     0.925
                                       6
                                    (0.825)

   
    An example of data analysis of a pet product efficacy study
   
Data Example1

This dataset Example1 is a synthetic dataset which mimics the data of a pet product study.  There are 5 variables (columns) in the dataset: 

 Treatment: values = 0 (for control group) and 1 (for treated group)
 Animal: 12 different animal IDs per group in this study
 Day: values = 1, 2, 3, and 4 in this dataset
 Ntotal: each animal's total number of insects used in the data analysis
 Nblood: number of blood-fed insects (this can be the number of retained insects or number of survival insects)



                                                                      Treatment
                                                                         ANIMAL
                                                                            Day
                                                                         Ntotal
                                                                         Nblood
                                                                              0
                                                                            0-1
                                                                              1
                                                                             24
                                                                             21
                                                                              0
                                                                            0-2
                                                                              1
                                                                             23
                                                                             19
                                                                              0
                                                                            0-3
                                                                              1
                                                                             27
                                                                             26
                                                                              0
                                                                            0-4
                                                                              1
                                                                             27
                                                                             22
                                                                              0
                                                                            0-5
                                                                              1
                                                                             26
                                                                             20
                                                                              0
                                                                            0-6
                                                                              1
                                                                             23
                                                                             15
                                                                              0
                                                                            0-7
                                                                              1
                                                                             24
                                                                             20
                                                                              0
                                                                            0-8
                                                                              1
                                                                             23
                                                                             22
                                                                              0
                                                                            0-9
                                                                              1
                                                                             23
                                                                             18
                                                                              0
                                                                           0-10
                                                                              1
                                                                             25
                                                                             22
                                                                              0
                                                                           0-11
                                                                              1
                                                                             26
                                                                             26
                                                                              0
                                                                           0-12
                                                                              1
                                                                             25
                                                                             13
                                                                              1
                                                                            1-1
                                                                              1
                                                                             27
                                                                              0
                                                                              1
                                                                            1-2
                                                                              1
                                                                             26
                                                                              1
                                                                              1
                                                                            1-3
                                                                              1
                                                                             25
                                                                              0
                                                                              1
                                                                            1-4
                                                                              1
                                                                             26
                                                                              0
                                                                              1
                                                                            1-5
                                                                              1
                                                                             27
                                                                              2
                                                                              1
                                                                            1-6
                                                                              1
                                                                             26
                                                                              1
                                                                              1
                                                                            1-7
                                                                              1
                                                                             25
                                                                              1
                                                                              1
                                                                            1-8
                                                                              1
                                                                             27
                                                                              1
                                                                              1
                                                                            1-9
                                                                              1
                                                                             27
                                                                              6
                                                                              1
                                                                           1-10
                                                                              1
                                                                             26
                                                                              2
                                                                              1
                                                                           1-11
                                                                              1
                                                                             27
                                                                              0
                                                                              1
                                                                           1-12
                                                                              1
                                                                             23
                                                                              2
                                                                              0
                                                                            0-1
                                                                              2
                                                                             24
                                                                             23
                                                                              0
                                                                            0-2
                                                                              2
                                                                             27
                                                                             24
                                                                              0
                                                                            0-3
                                                                              2
                                                                             23
                                                                             22
                                                                              0
                                                                            0-4
                                                                              2
                                                                             26
                                                                             25
                                                                              0
                                                                            0-5
                                                                              2
                                                                             25
                                                                             20
                                                                              0
                                                                            0-6
                                                                              2
                                                                             26
                                                                             21
                                                                              0
                                                                            0-7
                                                                              2
                                                                             25
                                                                             22
                                                                              0
                                                                            0-8
                                                                              2
                                                                             25
                                                                             22
                                                                              0
                                                                            0-9
                                                                              2
                                                                             23
                                                                             22
                                                                              0
                                                                           0-10
                                                                              2
                                                                             26
                                                                             20
                                                                              0
                                                                           0-11
                                                                              2
                                                                             26
                                                                             20
                                                                              0
                                                                           0-12
                                                                              2
                                                                             26
                                                                             21
                                                                              1
                                                                            1-1
                                                                              2
                                                                             27
                                                                              0
                                                                              1
                                                                            1-2
                                                                              2
                                                                             27
                                                                              1
                                                                              1
                                                                            1-3
                                                                              2
                                                                             24
                                                                              4
                                                                              1
                                                                            1-4
                                                                              2
                                                                             24
                                                                              0
                                                                              1
                                                                            1-5
                                                                              2
                                                                             23
                                                                              2
                                                                              1
                                                                            1-6
                                                                              2
                                                                             23
                                                                              0
                                                                              1
                                                                            1-7
                                                                              2
                                                                             24
                                                                              3
                                                                              1
                                                                            1-8
                                                                              2
                                                                             26
                                                                              1
                                                                              1
                                                                            1-9
                                                                              2
                                                                             26
                                                                              2
                                                                              1
                                                                           1-10
                                                                              2
                                                                             27
                                                                              2
                                                                              1
                                                                           1-11
                                                                              2
                                                                             27
                                                                              2
                                                                              1
                                                                           1-12
                                                                              2
                                                                             23
                                                                              2
                                                                              0
                                                                            0-1
                                                                              3
                                                                             24
                                                                             20
                                                                              0
                                                                            0-2
                                                                              3
                                                                             26
                                                                             23
                                                                              0
                                                                            0-3
                                                                              3
                                                                             24
                                                                             23
                                                                              0
                                                                            0-4
                                                                              3
                                                                             25
                                                                             24
                                                                              0
                                                                            0-5
                                                                              3
                                                                             27
                                                                             21
                                                                              0
                                                                            0-6
                                                                              3
                                                                             27
                                                                             18
                                                                              0
                                                                            0-7
                                                                              3
                                                                             27
                                                                             23
                                                                              0
                                                                            0-8
                                                                              3
                                                                             27
                                                                             21
                                                                              0
                                                                            0-9
                                                                              3
                                                                             24
                                                                             22
                                                                              0
                                                                           0-10
                                                                              3
                                                                             24
                                                                             18
                                                                              0
                                                                           0-11
                                                                              3
                                                                             25
                                                                             20
                                                                              0
                                                                           0-12
                                                                              3
                                                                             25
                                                                             21
                                                                              1
                                                                            1-1
                                                                              3
                                                                             23
                                                                              1
                                                                              1
                                                                            1-2
                                                                              3
                                                                             24
                                                                              0
                                                                              1
                                                                            1-3
                                                                              3
                                                                             27
                                                                              2
                                                                              1
                                                                            1-4
                                                                              3
                                                                             26
                                                                              1
                                                                              1
                                                                            1-5
                                                                              3
                                                                             23
                                                                              4
                                                                              1
                                                                            1-6
                                                                              3
                                                                             27
                                                                              4
                                                                              1
                                                                            1-7
                                                                              3
                                                                             25
                                                                              1
                                                                              1
                                                                            1-8
                                                                              3
                                                                             24
                                                                              3
                                                                              1
                                                                            1-9
                                                                              3
                                                                             27
                                                                              4
                                                                              1
                                                                           1-10
                                                                              3
                                                                             24
                                                                              1
                                                                              1
                                                                           1-11
                                                                              3
                                                                             26
                                                                              3
                                                                              1
                                                                           1-12
                                                                              3
                                                                             24
                                                                              1
                                                                              0
                                                                            0-1
                                                                              4
                                                                             25
                                                                             23
                                                                              0
                                                                            0-2
                                                                              4
                                                                             25
                                                                             19
                                                                              0
                                                                            0-3
                                                                              4
                                                                             23
                                                                             22
                                                                              0
                                                                            0-4
                                                                              4
                                                                             25
                                                                             24
                                                                              0
                                                                            0-5
                                                                              4
                                                                             26
                                                                             22
                                                                              0
                                                                            0-6
                                                                              4
                                                                             27
                                                                             24
                                                                              0
                                                                            0-7
                                                                              4
                                                                             26
                                                                             21
                                                                              0
                                                                            0-8
                                                                              4
                                                                             23
                                                                             22
                                                                              0
                                                                            0-9
                                                                              4
                                                                             27
                                                                             19
                                                                              0
                                                                           0-10
                                                                              4
                                                                             25
                                                                             19
                                                                              0
                                                                           0-11
                                                                              4
                                                                             27
                                                                             19
                                                                              0
                                                                           0-12
                                                                              4
                                                                             24
                                                                             20
                                                                              1
                                                                            1-1
                                                                              4
                                                                             27
                                                                              2
                                                                              1
                                                                            1-2
                                                                              4
                                                                             27
                                                                              4
                                                                              1
                                                                            1-3
                                                                              4
                                                                             26
                                                                              5
                                                                              1
                                                                            1-4
                                                                              4
                                                                             23
                                                                              1
                                                                              1
                                                                            1-5
                                                                              4
                                                                             25
                                                                              5
                                                                              1
                                                                            1-6
                                                                              4
                                                                             24
                                                                              0
                                                                              1
                                                                            1-7
                                                                              4
                                                                             25
                                                                              3
                                                                              1
                                                                            1-8
                                                                              4
                                                                             24
                                                                              1
                                                                              1
                                                                            1-9
                                                                              4
                                                                             26
                                                                              8
                                                                              1
                                                                           1-10
                                                                              4
                                                                             25
                                                                              3
                                                                              1
                                                                           1-11
                                                                              4
                                                                             24
                                                                              3
                                                                              1
                                                                           1-12
                                                                              4
                                                                             26
                                                                              0




Statistical methods

Generalized linear mixed models (GLMMs) with logit link function for binomial data are used to analyze the data using the following procedure:

 A GLMM model with over/under-dispersion (model 1) is first used to analyze the data.  If model 1 converges, the point estimated efficacy of the treatment is calculated using the proportions of blood-fed insects (of the groups) estimated from this model 1. 
 If model 1 fails to converge, the data are then analyzed using a GLMM model without the over/under-dispersion option (model 2). If model 2 converges, the point estimated efficacy of the treatment is calculated using the proportions of blood-fed insects (of the groups) estimated from this model.  
 If model 2 fails to converge, a GLMM with Laplace estimation method and without the over/under-dispersion option (model 3) is used to analyze the data.  If model 3 converges, the point estimated efficacy of the treatment is calculated using the proportions of blood-fed insects (of the groups) estimated from this model.
         Notes: 
 The probability of all three models 1, 2, and 3 above failing to converge for a dataset is very small (probably less than 0.5%).   
 Since only the point estimates (not the 95% CIs of the estimates) from the models 1, 2, and 3 are used, a compound symmetric (CS) variance-covariance matrix is used in each of models to increase the possibility of convergence of the models.  
               
 If model 3 fails to converge, a generalized linear model (model 4) is used to analyze the data on each day separately. The point estimated efficacy of the treatment is calculated using the proportions of blood-fed insects (of the groups) estimated from this model.  
 95% CI of the point estimated efficacy is obtained using bootstrapping technique.  1000 bootstrap datasets are created using resampling technique with replacement.  Each of bootstrap dataset is analyzed as the procedure (models 1, 2, 3, and 4) above.  The 95% CI of the estimated efficacy is the 2.5th and 97.5th percentile of the efficacy estimates obtained from 1000 bootstrap datasets.
   
SAS code

The SAS code of the data analysis of dataset Example1 are included in the Appendix VI.


Results of data analysis of dataset Example1

                                                   Estimated_    lower     upper
                  day    probtreat    probcontr     Efficacy     95% CI    95% CI

                   1       0.047        0.832         0.943       0.898     0.978
                   2       0.060        0.875         0.931       0.897     0.966
                   3       0.079        0.840         0.906       0.870     0.944
                   4       0.109        0.846         0.872       0.809     0.928


    Appendices

Appendix I: 	Results of power vs. sample size simulations

      Table A1.1: Study Design 1
      
         Binomial-Normal: PC=0.45, NPDay=50, SD_sub=0.33, SD_Day=0.166
                                       N
                                      day
                                    true_TE
                                     ReqTE
                                     Power
                                       
                                       
                                       
                                       
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.110
                                                                          0.266
                                                                          0.673
                                                                          0.901
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.010
                                                                          0.046
                                                                          0.285
                                                                          0.626
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.001
                                                                          0.006
                                                                          0.090
                                                                          0.320
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.005
                                                                          0.031
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.124
                                                                          0.330
                                                                          0.774
                                                                          0.953
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.020
                                                                          0.073
                                                                          0.372
                                                                          0.720
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.000
                                                                          0.007
                                                                          0.114
                                                                          0.355
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.006
                                                                          0.035
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.166
                                                                          0.414
                                                                          0.843
                                                                          0.975
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.018
                                                                          0.076
                                                                          0.441
                                                                          0.773
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.000
                                                                          0.013
                                                                          0.171
                                                                          0.424
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.005
                                                                          0.029
                                                                             11
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.198
                                                                          0.479
                                                                          0.881
                                                                          0.978
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.029
                                                                          0.111
                                                                          0.555
                                                                          0.806
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.003
                                                                          0.018
                                                                          0.196
                                                                          0.424
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.006
                                                                          0.026
                                                                             12
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.261
                                                                          0.562
                                                                          0.940
                                                                          0.984
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.035
                                                                          0.138
                                                                          0.644
                                                                          0.840
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.003
                                                                          0.017
                                                                          0.256
                                                                          0.438
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.012
                                                                          0.024


      Table A1.2: Study Design 2

        Binomial-Normal: PC=0.60, NPDay=100, SD_sub=0.33, SD_Day=0.166
                                       N
                                      day
                                    true_TE
                                     ReqTE
                                     Power
                                       
                                       
                                       
                                       
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              5
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.212
                                                                          0.442
                                                                          0.798
                                                                          0.938
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.035
                                                                          0.118
                                                                          0.444
                                                                          0.738
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.007
                                                                          0.031
                                                                          0.182
                                                                          0.391
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.009
                                                                          0.026
                                                                              6
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.309
                                                                          0.564
                                                                          0.886
                                                                          0.956
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.050
                                                                          0.170
                                                                          0.581
                                                                          0.789
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.005
                                                                          0.035
                                                                          0.231
                                                                          0.412
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.008
                                                                          0.023
                                                                              7
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.394
                                                                          0.699
                                                                          0.935
                                                                          0.977
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.067
                                                                          0.238
                                                                          0.700
                                                                          0.858
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.008
                                                                          0.045
                                                                          0.313
                                                                          0.435
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.013
                                                                          0.017
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.526
                                                                          0.795
                                                                          0.958
                                                                          0.983
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.101
                                                                          0.332
                                                                          0.787
                                                                          0.874
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.013
                                                                          0.078
                                                                          0.381
                                                                          0.445
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.008
                                                                          0.010
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.646
                                                                          0.885
                                                                          0.978
                                                                          0.996
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.168
                                                                          0.487
                                                                          0.860
                                                                          0.909
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.021
                                                                          0.126
                                                                          0.433
                                                                          0.460
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.008
                                                                          0.009
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.738
                                                                          0.928
                                                                          0.988
                                                                          0.998
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.214
                                                                          0.594
                                                                          0.899
                                                                          0.926
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.027
                                                                          0.164
                                                                          0.441
                                                                          0.458
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.005
                                                                          0.005

      
      Table A1.3: Study Design 3

         Binomial-Normal: PC=0.80, NPDay=50, SD_sub=0.33, SD_Day=0.166
                                       N
                                      day
                                    true_TE
                                     ReqTE
                                     Power
                                       
                                       
                                       
                                       
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              5
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.121
                                                                          0.268
                                                                          0.636
                                                                          0.858
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.016
                                                                          0.063
                                                                          0.280
                                                                          0.581
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.006
                                                                          0.016
                                                                          0.108
                                                                          0.287
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.001
                                                                          0.001
                                                                          0.004
                                                                          0.023
                                                                              6
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.166
                                                                          0.361
                                                                          0.762
                                                                          0.920
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.022
                                                                          0.085
                                                                          0.383
                                                                          0.694
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.003
                                                                          0.017
                                                                          0.142
                                                                          0.350
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.005
                                                                          0.025
                                                                              7
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.208
                                                                          0.448
                                                                          0.833
                                                                          0.953
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.025
                                                                          0.110
                                                                          0.514
                                                                          0.777
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.004
                                                                          0.019
                                                                          0.196
                                                                          0.393
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.010
                                                                          0.024
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.308
                                                                          0.582
                                                                          0.900
                                                                          0.980
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.051
                                                                          0.174
                                                                          0.635
                                                                          0.822
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.005
                                                                          0.036
                                                                          0.273
                                                                          0.419
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.007
                                                                          0.012
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.380
                                                                          0.685
                                                                          0.941
                                                                          0.992
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.067
                                                                          0.258
                                                                          0.728
                                                                          0.860
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.006
                                                                          0.053
                                                                          0.339
                                                                          0.434
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.008
                                                                          0.012
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.454
                                                                          0.761
                                                                          0.967
                                                                          0.995
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.074
                                                                          0.344
                                                                          0.801
                                                                          0.896
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.005
                                                                          0.071
                                                                          0.372
                                                                          0.438
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.007
                                                                          0.008



      Table A1.4: Study Design 4
      
         Binomial-Normal: PC=0.80, NPDay=25, SD_sub=0.33, SD_Day=0.166
                                       N
                                      day
                                    true_TE
                                     ReqTE
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.112
                                                                          0.242
                                                                          0.633
                                                                          0.889
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.014
                                                                          0.039
                                                                          0.261
                                                                          0.627
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.003
                                                                          0.008
                                                                          0.079
                                                                          0.292
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.001
                                                                          0.001
                                                                          0.004
                                                                          0.022
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.106
                                                                          0.254
                                                                          0.676
                                                                          0.900
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.014
                                                                          0.056
                                                                          0.324
                                                                          0.648
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.009
                                                                          0.094
                                                                          0.321
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.001
                                                                          0.001
                                                                          0.004
                                                                          0.023
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.140
                                                                          0.378
                                                                          0.843
                                                                          0.973
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.017
                                                                          0.068
                                                                          0.462
                                                                          0.809
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.004
                                                                          0.009
                                                                          0.160
                                                                          0.435
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.008
                                                                          0.026
                                                                             11
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.150
                                                                          0.376
                                                                          0.833
                                                                          0.968
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.013
                                                                          0.069
                                                                          0.462
                                                                          0.779
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.004
                                                                          0.012
                                                                          0.159
                                                                          0.391
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.001
                                                                          0.001
                                                                          0.006
                                                                          0.019
                                                                             12
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.195
                                                                          0.454
                                                                          0.874
                                                                          0.981
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.018
                                                                          0.114
                                                                          0.543
                                                                          0.815
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.001
                                                                          0.017
                                                                          0.211
                                                                          0.395
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.006
                                                                          0.016
                                                                             13
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.227
                                                                          0.513
                                                                          0.911
                                                                          0.989
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.021
                                                                          0.131
                                                                          0.628
                                                                          0.827
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.014
                                                                          0.253
                                                                          0.419
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.009
                                                                          0.013


      Table A1.5: Study Design 4

         Binomial-Normal: PC=0.85, NPDay=25, SD_sub=0.33, SD_Day=0.166
                                       N
                                      day
                                    true_TE
                                     ReqTE
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.087
                                                                          0.227
                                                                          0.621
                                                                          0.869
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.011
                                                                          0.040
                                                                          0.249
                                                                          0.589
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.003
                                                                          0.010
                                                                          0.087
                                                                          0.284
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.003
                                                                          0.023
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.092
                                                                          0.251
                                                                          0.716
                                                                          0.929
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.015
                                                                          0.054
                                                                          0.342
                                                                          0.677
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.011
                                                                          0.109
                                                                          0.325
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.001
                                                                          0.002
                                                                          0.005
                                                                          0.019
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.134
                                                                          0.345
                                                                          0.803
                                                                          0.957
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.019
                                                                          0.065
                                                                          0.421
                                                                          0.726
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.008
                                                                          0.140
                                                                          0.355
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.005
                                                                          0.020
                                                                             11
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.159
                                                                          0.411
                                                                          0.844
                                                                          0.977
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.021
                                                                          0.099
                                                                          0.523
                                                                          0.786
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.001
                                                                          0.014
                                                                          0.181
                                                                          0.375
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.007
                                                                          0.017
                                                                             12
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.198
                                                                          0.467
                                                                          0.889
                                                                          0.985
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.020
                                                                          0.120
                                                                          0.603
                                                                          0.820
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.017
                                                                          0.220
                                                                          0.384
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.006
                                                                          0.013
                                                                             13
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.169
                                                                          0.458
                                                                          0.915
                                                                          0.992
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.019
                                                                          0.114
                                                                          0.574
                                                                          0.829
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.006
                                                                          0.024
                                                                          0.233
                                                                          0.387
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.006
                                                                          0.011



      Table A1.6: Study Design 5

        Binomial-Normal: PC=0.75, NPDay=100, SD_sub=0.33, SD_Day=0.166
                                       N
                                      day
                                    true_TE
                                     ReqTE
                                     Power
                                       
                                       
                                       
                                       
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              5
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.297
                                                                          0.538
                                                                          0.851
                                                                          0.953
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.062
                                                                          0.180
                                                                          0.561
                                                                          0.780
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.010
                                                                          0.050
                                                                          0.239
                                                                          0.413
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.012
                                                                          0.019
                                                                              6
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.408
                                                                          0.679
                                                                          0.909
                                                                          0.972
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.085
                                                                          0.259
                                                                          0.682
                                                                          0.825
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.011
                                                                          0.059
                                                                          0.309
                                                                          0.426
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.007
                                                                          0.013
                                                                              7
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.513
                                                                          0.780
                                                                          0.939
                                                                          0.986
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.115
                                                                          0.363
                                                                          0.790
                                                                          0.877
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.015
                                                                          0.083
                                                                          0.360
                                                                          0.432
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.008
                                                                          0.010
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.655
                                                                          0.846
                                                                          0.961
                                                                          0.993
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.179
                                                                          0.491
                                                                          0.834
                                                                          0.895
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.024
                                                                          0.141
                                                                          0.394
                                                                          0.434
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.006
                                                                          0.006
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.749
                                                                          0.888
                                                                          0.987
                                                                          0.999
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.275
                                                                          0.603
                                                                          0.868
                                                                          0.918
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.045
                                                                          0.198
                                                                          0.410
                                                                          0.437
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.001
                                                                          0.002
                                                                          0.006
                                                                          0.006
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.803
                                                                          0.912
                                                                          0.993
                                                                          0.999
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.377
                                                                          0.720
                                                                          0.907
                                                                          0.945
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.066
                                                                          0.262
                                                                          0.425
                                                                          0.447
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.003
                                                                          0.003






Appendix II: 	Mathematical derivation of the estimated efficacy using a binomial model with logit link function

Using the binomial model with the canonical logit link, the probability that an insect is blood-fed is given by the equations:
Logit Pbloodfed=logPbloodfedPnot bloodfed=μ+groupg+dayi+group*daygi
P is Probability. In this model, μ is the intercept, groupg is the effect for group g (either the treatment or the control group), dayi is the effect for day  i (1, 2, 3, or 4), and group*daygi is the group*day interaction term. The logarithms are natural logarithms. The control group is group 0 and the treatment group is group 1.

For day i, define the logits for the control group and treatment group by
      α=μ+group0+dayi+group*day0i,
      β=μ+group1+dayi+group*day1i.

For the SAS coding we also define the total
      τ=α+β
so that the covariance between the estimates of α and β can easily be calculated.
It follows that on day i,
      
      Pbloodfed for a control animal=11+e-α ,
      Pbloodfed for a treated animal=11+e-β ,
      Efficacy=1-Pbloodfed for a treated animalPbloodfed for a control animal=e-β-e-α1+e-β=f(α,β)

Let α,β, and τ be the estimated values of α,β, and τ, and let their standard errors be seα,seβ, and se τ . The estimated efficacy is f(α,β). To approximate the standard error of the efficacy we can apply the delta method, a first order Taylor series expansion using the partial derivatives of f (see Klein, 1953 or Oehlert, 1992, for example):
      
      Estimated efficacy= fα,β≈fα,β+(α-α)dfdα(α,β) +(β-β)dfdβ(α,β).
This gives:
      Estimated efficacy ≈fα,β+(α-α)e-α1+e-β +(β-β)-e-β1+e-α1+e-β2 

Therefore, the variance of the estimated efficacy is approximately given by:

Variance Estimated efficacy≈Varαe-α1+e-β2 +Varβ-e-β1+e-α1+e-β22+2 Cov(α,β)e-α1+e-β-e-β1+e-α1+e-β2
Taking the square root and replacing the unknown true values of the parameters by their estimates gives

Standard Error Estimated efficacy≈Varαe-α1+e-β2 +Varβ-e-β1+e-α1+e-β22+2 Cov(α,β)e-α1+e-β-e-β1+e-α1+e-β2

In the last two equations, Var denotes the variance and Cov denotes the covariance of the estimated parameters. These can be estimated as follows:

      Varα=se(α)2
      Varβ=se(β)2
      Varτ=se(τ)2
      Varα+β=Varτ=Varα+Varβ+2 Cov(α,β)
      2 Covα,β=Varτ-Varα-Varβ

If the model with day effects and a group*day interaction fails to converge, then for each day, separate models are fitted of the form:
      Logit Pbloodfed=logPbloodfedPnot bloodfed=μ+groupg

For each day the logits for the control group and treatment group are defined by
      α=μ+group0,
      β=μ+group1 




Appendix III: 	True blood-fed proportion vs. proportions of animals or datasets meet the required blood-fed proportion

The true proportion of blood-fed pests (or the true proportion of retained pests; for simplicity, only "blood-fed" terminology is used in this section) is one of the important factors that impact the power of the study design.  In reality, the proportion of blood-fed pests is determined by the natural behaviors of each specific pest species, and we don't have sufficient available data to reliably estimate these values yet.  One of the criteria for an animal to be selected for a study is that the observed blood-fed proportion of the animal from the screening test is greater than the required lowest observed proportion of blood-fed pests value to ensure the attractiveness of the animal to the arthropods.  It also requires that the observed blood-fed proportion of each control animal is greater than the required value during the study period.  This simulation (percent animals or datasets have at least a certain percent insect non-blood-fed.sas) is simply to determine the lowest blood-fed proportion such that a low probability for an animal to have an observed blood-fed proportion below a given required blood-fed proportion and a low probability for a dataset (single measurement/animal) to have at least 1 animal with observed blood-fed proportion below the required blood-fed proportion.  We determine the lowest true blood-fed for simulations because the power of the study will increase if the true blood-fed proportion is greater. It is worth mentioning that the lowest true blood-fed proportions obtained from these simulations are used in the simulations of determining power and sample size and used for the purpose of helping experts to determine whether the required blood-fed proportions are reasonable or not.  

Two Tables A3.1 and A3.2 below present the results of simulations.  From these simulations, we conclude that the lowest blood-fed proportion are assumed to be:

    When the required blood-fed proportion for an animal is 0.25 and the number of arthropods/animal = 50, the lowest true value for blood-fed-proportion that can be used is 0.45
         
    When the required blood-fed proportion for an animal is 0.40 and the number of arthropods/animal = 100, the lowest true value for blood-fed- proportion that can be used is 0.60
         
    When the required blood-fed proportion for an animal is 0.60 and the number of arthropods/animal = 50, the lowest true value for blood-fed- proportion that can be used is 0.80

    When the required blood-fed proportion for an animal is 0.60 and the number of arthropods/animal = 25, the lowest true value for blood-fed- proportion that can be used is 0.85
         
    When the required blood-fed proportion for an animal is 0.60 and the number of arthropods/animal = 100, the lowest true value for blood-fed- proportion that can be used is 0.75
Note that if the true blood-fed proportion is greater than the assumed value in the simulations, the true power of the study design is greater than the power observed from the simulations.

Table A3.1: Proportion of animals with observed blood-fed proportion < require blood-fed proportion

                                          True             blood-fed proportion
                                         Required          blood-fed proportion
                                                   Number arthropods per animal
                                                                 Total animals 
Proportion of animals with observed blood-fed proportion < require blood-fed proportion
                                                                           0.75
                                                                            0.6
                                                                             50
                                                                         460000
                                                                        0.01065
                                                                            0.8
                                                                            0.6
                                                                             50
                                                                         460000
                                                                        0.00073
                                                                            0.8
                                                                            0.6
                                                                             25
                                                                         550000
                                                                        0.01133
                                                                           0.85
                                                                            0.6
                                                                             25
                                                                         550000
                                                                        0.00129
                                                                            0.7
                                                                            0.6
                                                                            100
                                                                         460000
                                                                        0.01759
                                                                           0.75
                                                                            0.6
                                                                            100
                                                                         460000
                                                                        0.00060
                                                                           0.55
                                                                            0.4
                                                                            100
                                                                         460000
                                                                        0.00152
                                                                            0.6
                                                                            0.4
                                                                            100
                                                                         460000
                                                                        0.00005
                                                                            0.4
                                                                           0.25
                                                                             50
                                                                         460000
                                                                        0.01486
                                                                           0.45
                                                                           0.25
                                                                             50
                                                                         460000
                                                                        0.00209

Table A3.2: Proportion of datasets having at least one animal with observed blood-fed proportion < require blood-fed proportion

                                          True             blood-fed proportion
                                         Required          blood-fed proportion
                                                   Number arthropods per animal
                                                       Number animals per group
                                                             Number of datasets
Proportion of datasets having at least one animal with observed blood-fed proportion < require blood-fed proportion 
                                                                           0.75
                                                                            0.6
                                                                             50
                                                                              8
                                                                           5000
                                                                        0.08320
                                                                               
                                                                               
                                                                               
                                                                              9
                                                                           5000
                                                                        0.08700
                                                                               
                                                                               
                                                                               
                                                                             10
                                                                           5000
                                                                        0.09940
                                                                               
                                                                               
                                                                               
                                                                             11
                                                                           5000
                                                                        0.11240
                                                                               
                                                                               
                                                                               
                                                                             12
                                                                           5000
                                                                        0.12300
                                                                               
                                                                               
                                                                               
                                                                             13
                                                                           5000
                                                                        0.12660
                                                                               
                                                                               
                                                                               
                                                                             14
                                                                           5000
                                                                        0.13780
                                                                               
                                                                               
                                                                               
                                                                             15
                                                                           5000
                                                                        0.15900
                                                                            0.8
                                                                            0.6
                                                                             50
                                                                              8
                                                                           5000
                                                                        0.00600
                                                                               
                                                                               
                                                                               
                                                                              9
                                                                           5000
                                                                        0.00560
                                                                               
                                                                               
                                                                               
                                                                             10
                                                                           5000
                                                                        0.00560
                                                                               
                                                                               
                                                                               
                                                                             11
                                                                           5000
                                                                        0.00820
                                                                               
                                                                               
                                                                               
                                                                             12
                                                                           5000
                                                                        0.00860
                                                                               
                                                                               
                                                                               
                                                                             13
                                                                           5000
                                                                        0.00960
                                                                               
                                                                               
                                                                               
                                                                             14
                                                                           5000
                                                                        0.01000
                                                                               
                                                                               
                                                                               
                                                                             15
                                                                           5000
                                                                        0.01280
                                                                           0.80
                                                                            0.6
                                                                             25
                                                                              5
                                                                           5000
                                                                        0.05780
                                                                               
                                                                               
                                                                               
                                                                              6
                                                                           5000
                                                                        0.06560
                                                                               
                                                                               
                                                                               
                                                                              7
                                                                           5000
                                                                        0.07540
                                                                               
                                                                               
                                                                               
                                                                              8
                                                                           5000
                                                                        0.08240
                                                                               
                                                                               
                                                                               
                                                                              9
                                                                           5000
                                                                        0.10040
                                                                               
                                                                               
                                                                               
                                                                             10
                                                                           5000
                                                                        0.11460
                                                                               
                                                                               
                                                                               
                                                                             11
                                                                           5000
                                                                        0.11160
                                                                               
                                                                               
                                                                               
                                                                             12
                                                                           5000
                                                                        0.12900
                                                                               
                                                                               
                                                                               
                                                                             13
                                                                           5000
                                                                        0.13980
                                                                               
                                                                               
                                                                               
                                                                             14
                                                                           5000
                                                                        0.14940
                                                                               
                                                                               
                                                                               
                                                                             15
                                                                           5000
                                                                        0.15320
                                                                           0.85
                                                                           0.60
                                                                             25
                                                                              5
                                                                           5000
                                                                        0.00760
                                                                               
                                                                               
                                                                               
                                                                              6
                                                                           5000
                                                                        0.00640
                                                                               
                                                                               
                                                                               
                                                                              7
                                                                           5000
                                                                        0.00880
                                                                               
                                                                               
                                                                               
                                                                              8
                                                                           5000
                                                                        0.00900
                                                                               
                                                                               
                                                                               
                                                                              9
                                                                           5000
                                                                        0.01040
                                                                               
                                                                               
                                                                               
                                                                             10
                                                                           5000
                                                                        0.01220
                                                                               
                                                                               
                                                                               
                                                                             11
                                                                           5000
                                                                        0.01500
                                                                               
                                                                               
                                                                               
                                                                             12
                                                                           5000
                                                                        0.01500
                                                                               
                                                                               
                                                                               
                                                                             13
                                                                           5000
                                                                        0.01700
                                                                               
                                                                               
                                                                               
                                                                             14
                                                                           5000
                                                                        0.02280
                                                                               
                                                                               
                                                                               
                                                                             15
                                                                           5000
                                                                        0.01660
                                                                            0.7
                                                                            0.6
                                                                            100
                                                                              8
                                                                           5000
                                                                        0.13320
                                                                               
                                                                               
                                                                               
                                                                              9
                                                                           5000
                                                                        0.14680
                                                                               
                                                                               
                                                                               
                                                                             10
                                                                           5000
                                                                        0.15900
                                                                               
                                                                               
                                                                               
                                                                             11
                                                                           5000
                                                                        0.18220
                                                                               
                                                                               
                                                                               
                                                                             12
                                                                           5000
                                                                        0.19080
                                                                               
                                                                               
                                                                               
                                                                             13
                                                                           5000
                                                                        0.20140
                                                                               
                                                                               
                                                                               
                                                                             14
                                                                           5000
                                                                        0.21840
                                                                               
                                                                               
                                                                               
                                                                             15
                                                                           5000
                                                                        0.23560
                                                                           0.75
                                                                            0.6
                                                                            100
                                                                              8
                                                                           5000
                                                                        0.00540
                                                                               
                                                                               
                                                                               
                                                                              9
                                                                           5000
                                                                        0.00420
                                                                               
                                                                               
                                                                               
                                                                             10
                                                                           5000
                                                                        0.00600
                                                                               
                                                                               
                                                                               
                                                                             11
                                                                           5000
                                                                        0.00660
                                                                               
                                                                               
                                                                               
                                                                             12
                                                                           5000
                                                                        0.00680
                                                                               
                                                                               
                                                                               
                                                                             13
                                                                           5000
                                                                        0.00780
                                                                               
                                                                               
                                                                               
                                                                             14
                                                                           5000
                                                                        0.00860
                                                                               
                                                                               
                                                                               
                                                                             15
                                                                           5000
                                                                        0.00980
                                                                           0.55
                                                                            0.4
                                                                            100
                                                                              8
                                                                           5000
                                                                        0.01260
                                                                               
                                                                               
                                                                               
                                                                              9
                                                                           5000
                                                                        0.01260
                                                                               
                                                                               
                                                                               
                                                                             10
                                                                           5000
                                                                        0.01340
                                                                               
                                                                               
                                                                               
                                                                             11
                                                                           5000
                                                                        0.01780
                                                                               
                                                                               
                                                                               
                                                                             12
                                                                           5000
                                                                        0.01680
                                                                               
                                                                               
                                                                               
                                                                             13
                                                                           5000
                                                                        0.01880
                                                                               
                                                                               
                                                                               
                                                                             14
                                                                           5000
                                                                        0.02160
                                                                               
                                                                               
                                                                               
                                                                             15
                                                                           5000
                                                                        0.02540
                                                                            0.6
                                                                            0.4
                                                                            100
                                                                              8
                                                                           5000
                                                                        0.00000
                                                                               
                                                                               
                                                                               
                                                                              9
                                                                           5000
                                                                        0.00020
                                                                               
                                                                               
                                                                               
                                                                             10
                                                                           5000
                                                                        0.00080
                                                                               
                                                                               
                                                                               
                                                                             11
                                                                           5000
                                                                        0.00040
                                                                               
                                                                               
                                                                               
                                                                             12
                                                                           5000
                                                                        0.00060
                                                                               
                                                                               
                                                                               
                                                                             13
                                                                           5000
                                                                        0.00080
                                                                               
                                                                               
                                                                               
                                                                             14
                                                                           5000
                                                                        0.00060
                                                                               
                                                                               
                                                                               
                                                                             15
                                                                           5000
                                                                        0.00120
                                                                            0.4
                                                                           0.25
                                                                             50
                                                                              8
                                                                           5000
                                                                        0.11520
                                                                               
                                                                               
                                                                               
                                                                              9
                                                                           5000
                                                                        0.12620
                                                                               
                                                                               
                                                                               
                                                                             10
                                                                           5000
                                                                        0.14180
                                                                               
                                                                               
                                                                               
                                                                             11
                                                                           5000
                                                                        0.14640
                                                                               
                                                                               
                                                                               
                                                                             12
                                                                           5000
                                                                        0.16220
                                                                               
                                                                               
                                                                               
                                                                             13
                                                                           5000
                                                                        0.17880
                                                                               
                                                                               
                                                                               
                                                                             14
                                                                           5000
                                                                        0.18600
                                                                               
                                                                               
                                                                               
                                                                             15
                                                                           5000
                                                                        0.19780
                                                                           0.45
                                                                           0.25
                                                                             50
                                                                              8
                                                                           5000
                                                                        0.01580
                                                                               
                                                                               
                                                                               
                                                                              9
                                                                           5000
                                                                        0.02000
                                                                               
                                                                               
                                                                               
                                                                             10
                                                                           5000
                                                                        0.02060
                                                                               
                                                                               
                                                                               
                                                                             11
                                                                           5000
                                                                        0.02380
                                                                               
                                                                               
                                                                               
                                                                             12
                                                                           5000
                                                                        0.02280
                                                                               
                                                                               
                                                                               
                                                                             13
                                                                           5000
                                                                        0.02720
                                                                               
                                                                               
                                                                               
                                                                             14
                                                                           5000
                                                                        0.02840
                                                                               
                                                                               
                                                                               
                                                                             15
                                                                           5000
                                                                        0.03140




Appendix IV: Values of SD_sub and SD_day used in the simulations

In the simulations, the random animal effects and the additional variation parameter were assumed to be normally distributed. Because of unavailable data from actual pet product efficacy studies to assess the values for parameters SD_sub of the variation between subjects or SD_day of the exhibiting of over dispersion of binomial data in pet product efficacy studies, SD_sub = 0.33 and SD_day = 0.166 were used in the simulations based on theoretical effects of these values on the variation of distributions of blood-fed proportion Ps.  The below table presents the 95% coverage of the distributions of the blood-fed pests between animals or between days when SD_sub = 0.33 and SD_day = 0.166.  For example, if the average proportion of proportions of blood-fed pests of the animals is 0.45, the 95% coverage of the distribution of true proportions of blood-fed pests of the animals is from 0.297 to 0.613.  If the proportion of blood-fed pests of all animals on a day (without exhibiting of over dispersion in the data) is 0.45, due to the additional effect associated with a SD_day = 0.166 (due to exhibiting of over dispersion in the data), 95% confidence limits of the proportions of blood-fed pests of the animals on the day would be from 0.370 to 0.533.

                                 Distributions
                                 population P
                                     logit
                                      SD
                                  logit - 2SD
                                  logit + 2SD
                                 95% CL of Ps
                                    Animals
                                     0.45
                                    -0.2007
                                     0.33
                                    -0.8607
                                    0.4593
                                     0.297
                                     0.613
                                       
                                      0.6
                                    0.4055
                                     0.33
                                    -0.2545
                                    1.0655
                                     0.437
                                     0.744
                                       
                                     0.75
                                    1.0986
                                     0.33
                                    0.4386
                                    1.7586
                                     0.608
                                     0.853
                                       
                                      0.8
                                    1.3863
                                     0.33
                                    0.7263
                                    2.0463
                                     0.674
                                     0.886
                                     Days
                                     0.45
                                    -0.2007
                                     0.166
                                    -0.5327
                                    0.1313
                                     0.370
                                     0.533
                                       
                                      0.6
                                    0.4055
                                     0.166
                                    0.0735
                                    0.7375
                                     0.518
                                     0.676
                                       
                                     0.75
                                    1.0986
                                     0.166
                                    0.7666
                                    1.4306
                                     0.683
                                     0.807
                                       
                                      0.8
                                    1.3863
                                     0.166
                                    1.0543
                                    1.7183
                                     0.742
                                     0.848



Appendix V: 	ICF's report of Evaluations of the applications of additional statistical methods for pet product studies


 
 MEMORANDUM

To:
David Miller, EPA
From:
Jonathan Cohen, ICF
Cc:
Kathryn Korthauer, EPA; Lori Brunsman, EPA; James Nguyen, EPA; Philip Villanueva, EPA; Matthew Crowley, EPA; Dave Burch, ICF
Date:
September 23, 2020
Re:
Estimation of Sample Sizes for Pet Product Studies

 Summary
In 2019, the Federal Insecticide, Fungicide, and Rodenticide Act (FIFRA) Scientific Advisory Panel (SAP) completed their peer review of EPA's proposed guideline for efficacy testing of topically applied pesticides used against certain ectoparasitic pests on pets. The review was completed at the June 11-14, 2019 FIFRA SAP meeting and the meeting minutes and final report were submitted to the docket. That review included various comments about the sample size determination which had been presented in the EPA document "Sample Size for Pet Product Studies." This memorandum presents the results of additional statistical analyses conducted by ICF for EPA in response to some of the FIFRA SAP comments.
The initial set of analyses repeated the original analysis in the EPA document "Sample Size for Pet Product Studies" for all 2000 simulations of one of the cases using data with normally distributed random effects. This provided the simulated data for all the analyses described in this memorandum as well as the power calculations for the probability of achieving a sufficiently precise efficacy estimate using the default method.
The next three sets of analyses evaluated the performance and power of the statistical models using the alternative Laplace and Quad estimation methods. Since the Laplace and Quad methods do not allow models with over- or under-dispersion, we first refitted the original model without the dispersion parameter. This approach reduced the percentage of cases that failed to converge from 10.01% in the original analysis to 1.59%. Replacing the RSPL estimation method by the Laplace and Quad methods further reduced the percentage of cases that failed to converge to 0.04% for the Laplace method and 0.43% for the Quad method.  These three approaches all gave very similar estimates of the power to the original set of analyses.
As another alternative approach we evaluated the Rao-Scott approach recommended by the FIFRA SAP. The Rao-Scott approach does power calculations for clustered binary data using a survey-adjusted variant of the standard chi-square test for homogeneity. Unfortunately this method does not provide comparable power estimates to the original statistical approach because the power for the Rao-Scott approach is defined as the probability that the blood-fed rates are statistically significantly different between the control and treatment groups and the statistical model assumes that the efficacy does not vary from day to day. In contrast, the power for the original approach is defined as the probability that the efficacy estimate is within a specified level of precision, that varies daily. The Rao-Scott approach produced power estimates that were uniformly equal to 1.
The final approach investigated in this memorandum was to use a bootstrap method to estimate the confidence intervals for the efficacy instead of the original delta method (a first order Taylor series approximation valid for sufficiently large samples). The bootstrap method resamples K sets of data from the fitted model multiple times and computes the efficacy estimates for each of the K samples. The K samples are called bootstrap samples. The 95% confidence interval for the efficacy is the interval from the 2.5[th] to the 97.5[th] percentile of the K efficacy estimates. Since the code takes several hours to run, the bootstrap confidence intervals were computed for the first 500 simulations only and used K = 250 bootstrap samples for each fitted model. As expected from the fact that the delta method is a large sample approximation, the power estimates from the original and bootstrap modeling differed most for small values of N, such as 5 and 6. The differences were also largest for days 1 and 2 and for the lowest desired precision (e.g. <= 0.025). For example, for N = 9 the power was within 0.1 of the original calculation in all cases except for day 2 with precision <= 0.025, day 2 with precision <= 0.030, and day 3 with precision <= 0.030. For N = 10 the power was within 0.1 of the original calculation in all cases except for day 2 with precision <= 0.025.   
  Original Model
The original analysis in the EPA document "Sample Size for Pet Product Studies" can be summarized as follows. For more details see the EPA document. The analysis began by simulating 2000 sets of data for N animals each in the control and treatment groups where each animal was tested for 4 days. The simulated data for the treatment group assumed true adjusted survival rates of 0.95, 0.925, 0.9, and 0.85 for the four days and also had various required adjusted survival rates. The simulated data for the control group assumed certain true survival rates for each day. The numbers of insects that infest each animal on each day were drawn from a uniform distribution.  The simulated data used either normal or Weibull distributions for the random animal and day effects. The SAS generalized linear mixed model procedure GLIMMIX was used to estimate the statistical power of detecting a treatment effect.
The selected GLIMMIX model was fitted using the default RSPL estimation method. The selected GLIMMIX model assumes that the number of blood-fed insects has a binomial distribution where for each animal and day, the number of trials is the number of insects infesting an animal, with a probability of success p (i.e., the probability that an insect is blood-fed) of the form logit(p) = a + b*group + c*day + d*group*day + e*randomday. The logit function is logit(p) = log(p/1 - p)). "Group" is the fixed treatment effect (control or treated), "day" is the fixed day effect (days 1, 2 ,3 and 4), and "randomday" is the random day effect, which is a multivariate normally distributed four-component vector for each animal. The four components are for the four days, allowing for possible correlations between blood-fed rates for the same animal on different days. The model allows for overdispersion or underdispersion so that the variance of the number of successes is φnp(1-p), where n is the number of trials, p is the probability of success, and the dispersion parameter is  φ. For the standard binomial distribution,  φ equals 1. The binomial distribution is overdispersed if φ is greater than 1 and underdispersed if φ is lower than 1.; these overdispersed or underdispersed models do not fully define a valid probability distribution. The estimated probability of success is p(group, day) = invlogit(a + b*group + c*day + d*group*day), where invlogit(x) = 1/(1 + exp(-x)). The efficacy is then defined as 1  -  p(treated, day)/p(control, day). The power is defined as the probability that the lower bound of the 95% confidence interval for the efficacy is at least the required adjusted survival rate and the precision (estimate minus lower bound) is at most the required precision level.  If the GLIMMIX model fails to converge, then in the second step a simpler model with separate parameters for each day and no random animal effects was fitted. 
The analyses described in this memorandum are for the example set of cases where N is between 5 and 10, the mean number of insects per animal per day is 100, the true survival rate of the control group is 0.6 for each day, and the required adjusted survival rates are 0.9 for each day. Daily required precision levels are 0.025, 0.030, 0.040, and 0.050, respectively. The same SAS programs and methods can easily be applied to other cases of interest. For these analyses, the simulated data were for the normal distribution and not the Weibull distribution, based on the FIFRA SAP's response to Charge Question 9 related to the fact that the SAS GLIMMIX procedure does not allow random effects that are not normally distributed.
The first set of analyses described in this memorandum were to simulate the data and refit the original set of models, using 2000 simulations. The same simulated data set was used for all the alternative analyses described in this memorandum. Note that 4,804 (10.01%) of the 48,000 fitted models failed to converge in the first step where all four days were modeled together. (There were actually 12,000 models fitted at the first step since there are 6 values of N, each model gives estimates for all 4 days, and each data set is simulated 2,000 times.) All the models fitted in the second step with separate models for each day converged. Also note that only 57.56% of the models that converged in the first step had overdispersion (where φ is greater than 1). The primary SAS program "Sample size for Pet Product Studies (normal)_Binomial.original.2000.sas" is attached. Also attached are the SAS programs used to count the numbers of models with overdispersion: "Sample size for Pet Product Studies (normal)_Binomial_original_2000.overdispersionparam.sas" and "overdisp.sas". The power calculations are shown in the following Table 1.

                Table 1. Power Calculations for Original Model
        Binomial-Normal: PC=0.60, NPDay=100, SD_sub=0.33, SD_Day=0.166

                                       N
                                      day
                                    true_TE
                                     ReqTE
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              5
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.212
                                                                          0.442
                                                                          0.798
                                                                          0.938
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.035
                                                                          0.118
                                                                          0.444
                                                                          0.738
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.007
                                                                          0.031
                                                                          0.182
                                                                          0.391
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.009
                                                                          0.026
                                                                              6
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.309
                                                                          0.564
                                                                          0.886
                                                                          0.956
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.050
                                                                          0.170
                                                                          0.581
                                                                          0.789
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.005
                                                                          0.035
                                                                          0.231
                                                                          0.412
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.008
                                                                          0.023
                                                                              7
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.394
                                                                          0.699
                                                                          0.935
                                                                          0.977
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.067
                                                                          0.238
                                                                          0.700
                                                                          0.858
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.008
                                                                          0.045
                                                                          0.313
                                                                          0.435
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.013
                                                                          0.017
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.526
                                                                          0.795
                                                                          0.958
                                                                          0.983
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.101
                                                                          0.332
                                                                          0.787
                                                                          0.874
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.013
                                                                          0.078
                                                                          0.381
                                                                          0.445
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.008
                                                                          0.010
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.646
                                                                          0.885
                                                                          0.978
                                                                          0.996
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.168
                                                                          0.487
                                                                          0.860
                                                                          0.909
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.021
                                                                          0.126
                                                                          0.433
                                                                          0.460
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.008
                                                                          0.009
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.738
                                                                          0.928
                                                                          0.988
                                                                          0.998
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.214
                                                                          0.594
                                                                          0.899
                                                                          0.926
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.027
                                                                          0.164
                                                                          0.441
                                                                          0.458
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.005
                                                                          0.005


 Original Model Without Dispersion
The second set of analyses refit the original models without allowing for overdispersion or underdispersion. This step was necessary for comparison purposes since the alternative analyses for evaluating the Laplace and Quad integration methods do not allow the fitting of models with over- or under-dispersion because the marginal models need to be well-defined. The refitted models in this second set also used the RSPL estimation method. 764 (1.59%) of the 48,000 fitted models failed to converge in the first step. The SAS program "Sample size for Pet Product Studies (normal)_Binomial_nodispersion_2000.sas" is attached. The power calculations are shown in the following Table 2. The power estimates are quite similar to those shown in Table 1 for the original model.

       Table 2. Power Calculations for Original Model Without Dispersion
        Binomial-Normal: PC=0.60, NPDay=100, SD_sub=0.33, SD_Day=0.166

                                       N
                                      day
                                    true_TE
                                     ReqTE
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              5
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.146
                                                                          0.401
                                                                          0.854
                                                                          0.983
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.012
                                                                          0.072
                                                                          0.449
                                                                          0.769
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.001
                                                                          0.008
                                                                          0.164
                                                                          0.411
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.005
                                                                          0.023
                                                                              6
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.254
                                                                          0.592
                                                                          0.956
                                                                          0.996
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.017
                                                                          0.126
                                                                          0.626
                                                                          0.845
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.000
                                                                          0.017
                                                                          0.246
                                                                          0.438
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.006
                                                                          0.022
                                                                              7
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.381
                                                                          0.750
                                                                          0.990
                                                                          0.998
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.035
                                                                          0.208
                                                                          0.772
                                                                          0.899
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.032
                                                                          0.327
                                                                          0.456
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.013
                                                                          0.020
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.559
                                                                          0.871
                                                                          0.997
                                                                          0.998
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.073
                                                                          0.326
                                                                          0.857
                                                                          0.902
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.006
                                                                          0.053
                                                                          0.403
                                                                          0.461
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.009
                                                                          0.012
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.690
                                                                          0.948
                                                                          1.000
                                                                          1.000
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.124
                                                                          0.479
                                                                          0.907
                                                                          0.922
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.010
                                                                          0.105
                                                                          0.443
                                                                          0.465
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.008
                                                                          0.009
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.800
                                                                          0.981
                                                                          1.000
                                                                          1.000
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.190
                                                                          0.616
                                                                          0.931
                                                                          0.933
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.011
                                                                          0.148
                                                                          0.462
                                                                          0.466
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.005
                                                                          0.005

 Laplace and Quad Methods for Models Without Dispersion
The third and fourth sets of analyses evaluate the Laplace and Quad estimation methods. In those cases, models with over- or under-dispersion are not permitted in SAS because marginal distributions of the data need to be available, hence "R-side" covariance structures and overdispersion are not available for those two methods. 
For the Laplace method, using the GLIMMIX model without over- or under-dispersion, 0.04% of the 48,000 fitted models failed to converge in the first step. For the Quad method, using the GLIMMIX model without over- or under-dispersion, 0.43% of the 48,000 fitted models failed to converge in the first step. Also note that a concern for the Laplace and Quad methods is that frequently the error message "At least one element of the gradient is greater than 1e-3" was reported even though the convergence criterion was satisfied. These results show that the Laplace and Quad estimation methods substantially reduce the problem of non-convergence but at the expense of being unable to model over- or under-dispersion. For the Laplace method, the SAS program "Sample size for Pet Product Studies (normal)_Binomial_laplace_2000.sas" is attached. The corresponding power calculations are shown in the following Table 3. For the Quad method, the SAS program "Sample size for Pet Product Studies (normal)_Binomial_quad_2000.sas" is attached. The corresponding power calculations are shown in the following Table 4. The power estimates are quite similar to those shown in Table 1 for the original model.

                      Table 3.  Power for Laplace Method.
        Binomial-Normal: PC=0.60, NPDay=100, SD_sub=0.33, SD_Day=0.166

                                       N
                                      day
                                    true_TE
                                     ReqTE
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              5
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.197
                                                                          0.504
                                                                          0.906
                                                                          0.993
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.019
                                                                          0.103
                                                                          0.556
                                                                          0.833
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.019
                                                                          0.241
                                                                          0.473
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.010
                                                                          0.029
                                                                              6
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.326
                                                                          0.683
                                                                          0.973
                                                                          0.997
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.028
                                                                          0.183
                                                                          0.714
                                                                          0.871
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.001
                                                                          0.028
                                                                          0.319
                                                                          0.476
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.013
                                                                          0.026
                                                                              7
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.455
                                                                          0.831
                                                                          0.995
                                                                          0.998
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.063
                                                                          0.294
                                                                          0.845
                                                                          0.913
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.052
                                                                          0.403
                                                                          0.484
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.016
                                                                          0.022
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.645
                                                                          0.914
                                                                          0.998
                                                                          0.998
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.109
                                                                          0.422
                                                                          0.893
                                                                          0.912
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.009
                                                                          0.090
                                                                          0.464
                                                                          0.496
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.013
                                                                          0.014
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.780
                                                                          0.977
                                                                          1.000
                                                                          1.000
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.173
                                                                          0.589
                                                                          0.926
                                                                          0.933
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.019
                                                                          0.156
                                                                          0.493
                                                                          0.502
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.012
                                                                          0.012
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.865
                                                                          0.990
                                                                          1.000
                                                                          1.000
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.263
                                                                          0.717
                                                                          0.946
                                                                          0.946
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.028
                                                                          0.218
                                                                          0.503
                                                                          0.503
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.006
                                                                          0.006


                       Table 4.  Power for Quad Method.
         Binomial-Normal: PC=0.60, NPDay=100, SD_sub=0.33, SD_Day=0.166

                                       N
                                      day
                                    true_TE
                                     ReqTE
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              5
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.196
                                                                          0.499
                                                                          0.905
                                                                          0.992
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.018
                                                                          0.102
                                                                          0.553
                                                                          0.829
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.019
                                                                          0.241
                                                                          0.469
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.010
                                                                          0.028
                                                                              6
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.324
                                                                          0.678
                                                                          0.972
                                                                          0.997
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.028
                                                                          0.179
                                                                          0.713
                                                                          0.869
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.001
                                                                          0.027
                                                                          0.316
                                                                          0.472
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.013
                                                                          0.026
                                                                              7
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.455
                                                                          0.826
                                                                          0.995
                                                                          0.998
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.061
                                                                          0.289
                                                                          0.843
                                                                          0.912
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.002
                                                                          0.052
                                                                          0.399
                                                                          0.483
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.015
                                                                          0.021
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.643
                                                                          0.912
                                                                          0.998
                                                                          0.998
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.112
                                                                          0.420
                                                                          0.890
                                                                          0.911
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.009
                                                                          0.090
                                                                          0.462
                                                                          0.493
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.001
                                                                          0.013
                                                                          0.014
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.780
                                                                          0.977
                                                                          1.000
                                                                          1.000
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.174
                                                                          0.587
                                                                          0.925
                                                                          0.932
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.018
                                                                          0.154
                                                                          0.489
                                                                          0.498
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.011
                                                                          0.011
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.864
                                                                          0.989
                                                                          1.000
                                                                          1.000
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.265
                                                                          0.716
                                                                          0.945
                                                                          0.945
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.026
                                                                          0.216
                                                                          0.499
                                                                          0.500
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.002
                                                                          0.006
                                                                          0.006

 Rao-Scott Approach
For another alternative statistical approach, we evaluated the Rao-Scott approach recommended by the FIFRA SAP. The Rao-Scott approach is described in the paper "A Simple Method for the Analysis of Clustered Binary Data" by J. N. K. Rao and A. J. Scott (Biometrics, 1992, Vol 48, No 2, pages 577-585). 
The Rao-Scott approach does power calculations for clustered binary data using a survey-adjusted variant of the standard test chi-square test for homogeneity. Unfortunately this method does not provide comparable power  estimates to the original statistical approach because the power for the Rao-Scott approach is the probability that the blood-fed rates are statistically significantly different between the control and treatment groups and the statistical model assumes that the efficacy does not vary from day to day. The power for the original approach is the estimated probability that the efficacy estimate for a given day is at least the required adjusted survival rate for that day and is within a specified level of precision for that day.
To implement the Rao-Scott approach in SAS, a new data set was created so that for each simulated animal and day with ntotal insects of which nblood were blood-fed, ntotal binary observations were created where the first nblood observations had blood-fed = "Yes" and the remaining  ntotal - nblood observations had blood-fed = "No". The Rao-Scott approach was then implemented using the SURVEYFREQ procedure where the cluster variable is animal. A two-way table of treatment group against the blood-fed indicator was created and the chisq option was used to implement the Rao-Scott test. To improve the accuracy, the more accurate second-order calculation was used. Each binary observation is for a specific insect and all the insects that infest a given animal are in the same cluster, regardless of the day. (Otherwise the data for the same animal on different days would be incorrectly treated as independent.) 
For each value of N, and each of the 2000 simulations, the p-value for the Rao-Scott test of equal blood-fed rates was at or below 0.05 and so the estimated power for each N was 1. The SAS program for these calculations "Sample size for Pet Product Studies (normal)_Binomial_raoscott.sas" is attached.
 Bootstrap Method for Confidence Intervals
The final approach investigated in this memorandum was to use a bootstrap method to estimate the confidence intervals for the efficacy instead of the original delta method (a first order Taylor series approximation valid for sufficiently large samples). The bootstrap method resamples K sets of data from the fitted model multiple times and computes the efficacy estimates for each of the K samples. The K samples are called bootstrap samples. The 95% confidence interval for the efficacy is the interval from the 2.5[th] to the 97.5[th] percentile of the K efficacy estimates. In some statistical situations, a parametric bootstrap method is used, meaning that the bootstrap samples are drawn from the fitted model. However, in this situation a parametric bootstrap could not be used because the binomial model with over- or under-dispersion is not a specific probability model. An example of a valid probability model would be to replace the binomial model by a beta-binomial model, which is a binomial distribution where the success probability is a random variable drawn from a beta distribution. However, the beta-binomial distribution has overdispersion and not underdispersion, and, as noted above, only about 58 percent of the original fitted models were overdispersed (among the models that converged in the first step). There is no well-known complete probability model for a binomial distribution with underdispersion. For these analyses we used a non-parametric bootstrap approach under which we resampled the animals at random with replacement from each treatment group in the original data set. For example, for simulation 1 when N = 5 in the control group, the five simulated animals were named 0-1, 0-2, 0-3, 0-4, and 0-5, and the data for animal 0-1 had  92 insects and 71 blood-fed insects on day 1, 108 insects and 61 blood-fed insects on day 2, 96 insects and 75 blood-fed insects on day 3, and 107 insects and 75 blood-fed insects on day 4. For the first bootstrap sample for simulation 1 and N = 5, the five simulated animals in the control group were 0-1, 0-1, 0-2, 0-3, and 0-5, and the data for both simulated animals 0-1 were the same as in the original data set:  92 insects and 71 blood-fed insects on day 1, 108 insects and 61 blood-fed insects on day 2, 96 insects and 75 blood-fed insects on day 3, and 107 insects and 75 blood-fed insects on day 4. By keeping all the daily data for all four days the same for each resampled animal, the blood-fed rates and correlation structure within animals are preserved. 
Since the program takes a long time to run, we chose to do these analyses only for the first 500 simulations (instead of 2000) and to create 250 bootstrap samples for each of those simulations. The SAS program "Sample size for Pet Product Studies (normal)_Binomial_bootstrap_500.sas" is attached. The program took almost 19 hours to run on my computer.  (The program for the original analysis using the delta method to estimate standard errors took only 15 minutes to run.) 
In the first step, the original model was fitted to 12,000 of the original datasets (i.e., 500 simulations, 6 values of N, and 4 days, although this actually created 3,000 models since in the first step each fitted model provides estimates for all four days). Out of these 12,000 models, 10,836 (90.3%) converged, leading to 10,836 efficacy values. To calculate the corresponding confidence intervals, 250 bootstrap samples were created for each model, corresponding to 3,000,000 fitted models. Out of these models fitted to the bootstrap samples, 87.2% converged in the first step, providing a total of 2,614,552 efficacy estimates. The convergence of the models fitted to the bootstrap samples is less crucial as long as the 2.5[th] percentile of the available efficacy estimates for each N, simulation, and day can be calculated. In the second step, separate models were fitted for each N, simulation, and day for the remaining 1,164 models (i.e., 12,000  -  10,836). For the second step, all those 1,164 models converged, as did the corresponding 291,000 models fitted to the bootstrap samples (i.e., 1,164 x 250 = 291,000). Ideally, the number of bootstrap samples should be large enough that a good estimate of the 2.5[th] percentile of the efficacy values can be calculated from the models that successfully converged. This is a compromise because increasing the number of bootstrap samples increases the execution time. 
The results are shown in the following Table 5.  As expected from the fact that the delta method is a large sample approximation, the power estimates from the original and bootstrap modeling shown in Tables 1 and 5 differ most for small values of N, such as 5 and 6. The differences were also largest for days 1 and 2 and for the lowest desired precision (e.g. <= 0.025). For example, for N = 9 the power was within 0.1 of the original calculation in all cases except for day 2 with precision <= 0.025, day 2 with precision <= 0.030, and day 3 with precision <= 0.030. For N = 10 the power was within 0.1 of the original calculation in all cases except for day 2 with precision <= 0.025.



                     Table 5.  Power for Bootstrap Method.
        Binomial-Normal: PC=0.60, NPDay=100, SD_sub=0.33, SD_Day=0.166

                                       N
                                      day
                                    true_TE
                                     ReqTE
                             Precision <= 0.025
                             Precision <= 0.030
                             Precision <= 0.040
                             Precision <= 0.050
                                                                              5
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.442
                                                                          0.626
                                                                          0.840
                                                                          0.944
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.242
                                                                          0.374
                                                                          0.618
                                                                          0.782
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.112
                                                                          0.196
                                                                          0.356
                                                                          0.448
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.004
                                                                          0.004
                                                                          0.014
                                                                          0.020
                                                                              6
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.534
                                                                          0.710
                                                                          0.898
                                                                          0.972
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.244
                                                                          0.392
                                                                          0.650
                                                                          0.790
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.092
                                                                          0.178
                                                                          0.330
                                                                          0.400
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.004
                                                                          0.006
                                                                          0.016
                                                                          0.024
                                                                              7
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.548
                                                                          0.740
                                                                          0.930
                                                                          0.982
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.234
                                                                          0.430
                                                                          0.706
                                                                          0.848
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.076
                                                                          0.156
                                                                          0.318
                                                                          0.404
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.008
                                                                          0.018
                                                                          0.026
                                                                              8
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.636
                                                                          0.802
                                                                          0.964
                                                                          0.996
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.308
                                                                          0.498
                                                                          0.758
                                                                          0.866
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.100
                                                                          0.202
                                                                          0.388
                                                                          0.472
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.004
                                                                          0.008
                                                                          0.012
                                                                          0.014
                                                                              9
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.688
                                                                          0.862
                                                                          0.974
                                                                          0.996
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.356
                                                                          0.594
                                                                          0.848
                                                                          0.936
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.114
                                                                          0.236
                                                                          0.402
                                                                          0.468
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.004
                                                                          0.004
                                                                          0.010
                                                                          0.014
                                                                             10
                                                                              1
                                                                           0.95
                                                                            0.9
                                                                          0.758
                                                                          0.896
                                                                          0.988
                                                                          0.998
                                       
                                                                              2
                                                                          0.925
                                                                            0.9
                                                                          0.366
                                                                          0.600
                                                                          0.882
                                                                          0.930
                                       
                                                                              3
                                                                            0.9
                                                                            0.9
                                                                          0.104
                                                                          0.206
                                                                          0.376
                                                                          0.408
                                       
                                                                              4
                                                                           0.85
                                                                            0.9
                                                                          0.000
                                                                          0.000
                                                                          0.004
                                                                          0.006



Appendix VI: SAS codes

          




SAS code of simulation to determine the lowest blood-fed proportions
(percent animals or datasets have at least a certain percent insect non-blood-fed.sas)

      *=========================================================================*
      * Programmer: James Nguyen, US EPA                                        *
      *                                                                         *
      * Purposes:                                                               *
      *        - estimate proportion of animals or datasets that have at least  *
      *          PF percent of mosquitoes with blood-fed, given a true percent  *
      *          P of blood-fed                                                 *
      *                                                                         *
      * Assumptions for data generation:                                        *
      *        - Control Survival Rate:  0 < CP < 1                             *
      *        - Adjusted Survival Rate  0 < ATP < 1                            *
      *        - In addition to the variation of binomial distribution (or      *
      *          Poisson distribution), the data also have two more sources of  *
      *          variation:                                                     *
      *              + variation due to animal                                  *
      *              + variation due to day                                     *
      *                                                                         *
      * Date: 6/25/2018                                                         *
      *========================================================================*;
      options formdlim="=" nonumber nodate ls=100 ps=100;
      
      
      * PF is required observed blood-fed for the animal;
      * for example, animal is excluded if the proportion blood-fed insects X/NPSub <= PF=60% 
        (i.e., at least 60% of insects do not take blood meal);
      
      %Macro PercentFail(TrueP_bloodfed=,Nmin=,Nmax=,NPSub=,RNdiff=,ReqP_bloodfed=, seed=);
      
      	Proc datasets nolist; save sasmacr; quit;
      
      	Data Simmer;
      		call streaminit(&seed);
      		do N = &Nmin to &Nmax;
      			do Sim =  1 to 5000;
      				do sub = 1 to N;
      					N_insect = &NPSub;
      					TrueP_bloodfed = &TrueP_bloodfed;
      					NPSub = round((1-&RNdiff)*&NPSub + (2*&RNdiff)*&NPSub*rand("uniform"), 1);  
      					X = rand("binomial", &TrueP_bloodfed, NPSub);
      					output;
      				end; *sub;
      			end; *sim;
      		end; *N;
      	run;
      
      	Data Simmer;
      		set Simmer;
      		ReqP_bloodfed = &ReqP_bloodfed;
      		Fail = (X/NPSub <= ReqP_bloodfed);
      		*==> the above code should have been "Fail = (X/NPSub < ReqP_bloodfed)".  We did not
      			 made this correction and rerun the program since we believe this change would not 
      			 alter "the results of suggested proportion of blood-fed in control group" obtained 
      			 from this SAS program;
      	run;
      
      	title "proportion of animals with P_bloodfed <= ReqP_bloodfed";
      	Proc means data = Simmer mean maxdec=5;
      		class TrueP_bloodfed ReqP_bloodfed N_insect ;
      		var fail;
      	run;
      
      	Proc SQL;
      		create table Simmer1 as
      		select  TrueP_bloodfed, N_insect, N, Sim, sum(Fail) as NFail
      		from Simmer
      		group by TrueP_bloodfed, N_insect, N, Sim
      		order by TrueP_bloodfed, N_insect, N, Sim;
      	quit;
      
      	Data Simmer1;
      		set simmer1;
      		ReqP_bloodfed = &ReqP_bloodfed;
      		NPDFail = (Nfail > 0);
      	run;
      
      	title "proportion of datasets that have at least 1 animal with P_bloodfed <= ReqP_bloodfed";
      	Proc means data = Simmer1 mean maxdec = 5;
      		class TrueP_bloodfed ReqP_bloodfed N_insect N ;
      		var NPDFail;
      	run;
      
      %mend;
      ods rtf file = "C:\Users\JNguyen\Desktop\All\percent.rtf" startpage=no bodytitle;
      %PercentFail(TrueP_bloodfed=0.75, Nmin=8, Nmax=15, NPSub=50,  RNdiff=0.1, ReqP_bloodfed=0.6, seed=165468);
      %PercentFail(TrueP_bloodfed=0.80, Nmin=8, Nmax=15, NPSub=50,  RNdiff=0.1, ReqP_bloodfed=0.6, seed=165468);
      
      %PercentFail(TrueP_bloodfed=0.80, Nmin=5, Nmax=15, NPSub=25,  RNdiff=0.1, ReqP_bloodfed=0.6, seed=165468);
      %PercentFail(TrueP_bloodfed=0.85, Nmin=5, Nmax=15, NPSub=25,  RNdiff=0.1, ReqP_bloodfed=0.6, seed=165468);
      
      %PercentFail(TrueP_bloodfed=0.70, Nmin=8, Nmax=15, NPSub=100, RNdiff=0.1, ReqP_bloodfed=0.6, seed=165468);
      %PercentFail(TrueP_bloodfed=0.75, Nmin=8, Nmax=15, NPSub=100, RNdiff=0.1, ReqP_bloodfed=0.6, seed=165468);
      
      %PercentFail(TrueP_bloodfed=0.55, Nmin=8, Nmax=15, NPSub=100, RNdiff=0.1, ReqP_bloodfed=0.4, seed=165468);
      %PercentFail(TrueP_bloodfed=0.60, Nmin=8, Nmax=15, NPSub=100, RNdiff=0.1, ReqP_bloodfed=0.4, seed=165468);
      
      %PercentFail(TrueP_bloodfed=0.40, Nmin=8, Nmax=15, NPSub=50, RNdiff=0.1, ReqP_bloodfed=0.25, seed=165468);
      %PercentFail(TrueP_bloodfed=0.45, Nmin=8, Nmax=15, NPSub=50, RNdiff=0.1, ReqP_bloodfed=0.25, seed=165468);
      ods rtf close;


SAS code of power vs. sample size simulations
(Sample size for Pet Product Studies.sas)

      *=========================================================================*
      * Programmers: James Nguyen, USEPA and Jonathon Cohen, ICF                *
      *                                                                         *
      * Purposes:                                                               *
      *        - Perform simulation for Power vs. Sample Size in the Guideline  *
      *          of Pet Product Studies) where the random effects of ANIMAL and *
      *          DAY are generated from normal distributions                    *
      *                                                                         *
      * Assumptions for data generation:                                        *
      *        - Control Survival Rate:  0 < CP < 1                             *
      *        - Adjusted Killing Rate:  0 < TE < 1                             *
      *        - In addition to the variation of insects (binomial distribution *
      *          of Poisson distribution), the data also have two more sources  *
      *          of variation:                                                  *
      *              + variation due to animal                                  *
      *              + variation due to day                                     *
      *                                                                         *
      * Note: Show summary tables only                                          *
      *                                                                         *
      * Date: 12/02/2020                                                        *
      *========================================================================*;
      options formdlim="=" nonumber nodate ls=100 ps=100;
      
      
      %let PETSTDY=C:\Users\JNguyen\OneDrive - Environmental Protection Agency (EPA)\Insect Repellency\Sample Size of Repellency Studies\Sample size of Pet Product studies\Final;
      
      libname opath "C:\Users\JNguyen\OneDrive - Environmental Protection Agency (EPA)\Insect Repellency\Sample Size of Repellency Studies\Sample size of Pet Product studies\Final";
      
      %macro exist(dsn);
      %global exist;
      %local libname;
      %let dsn=%upcase(&dsn);
      %if %index(&dsn,.) %then %do;
      	%let libname=%scan(&dsn,1);
      	%let dsn=%scan(&dsn,2);
      %end;
      %else %let libname=WORK;
      proc contents data=&libname.._all_ memtype=data noprint out=temp(keep=memname);
      run;
      
      data _null_;
      set temp end=last;
      if upcase(memname)="&dsn" then do;
      	call symput('exist','yes');
      	stop;
      end;
      else if last then call symput('exist','no');
      proc datasets library=work nofs nolist;
      	delete temp;
      quit;
      
      %mend exist;
      
      ods noptitle;
      
      *===> Parameters in Macro:
                          NSub = number of replicates/group
                         NPDay = number of insects/replicate
                            CP = true survival rate of control group
                            TE = true adjusted survival rate of treatment group
              SubVar or SD_sub = degree of variation between ANIMAL
              DayVar or SD_Day = degree of variation between DAY
                          Nsim = number of simulated datasets per scenario
                          seed = seed to regenerate the results of simulations;
      
      %Macro PetProduct_NORMAL(NminSub=,NmaxSub=,NPDay=,RNdiff=,CP=,TE=,ReqTE=,ReqPrecision=,SD_sub=,SD_Day=,Nsim=,seed=);
      	dm log 'clear';title;
      	Proc datasets nolist; save sasmacr; quit;
      
      	*===> count number of precisions;
      	%let NPrecision = 1;
      	%let Precision&NPrecision = %nrbquote(%scan(&ReqPrecision,&NPrecision, %str( )));
      	%do %while (&&Precision&NPrecision ^=);
      		%let NPrecision=%eval(&NPrecision+1);
      		%let Precision&NPrecision = %nrbquote(%scan(&ReqPrecision,&NPrecision, %str( )));
      	%end;
      	%let NPrecision=%eval(&NPrecision-1);
      
      
      	*===> count number of time points;
      	%let ND=1;
      	%let CP&ND = %nrbquote(%scan(&CP,&ND, %str( )));
      	%let TE&ND = %nrbquote(%scan(&TE,&ND, %str( )));
      	%let ReqTE&ND = %nrbquote(%scan(&ReqTE,&ND,%str( )));
      	%do %while (&&TE&ND ^=);
      		%let ND=%eval(&ND+1);
      		%let CP&ND = %nrbquote(%scan(&CP,&ND, %str( )));
      		%let TE&ND = %nrbquote(%scan(&TE,&ND, %str( )));
      		%let ReqTE&ND = %nrbquote(%scan(&ReqTE,&ND,%str( )));
      	%end;
      	%let ND=%eval(&ND-1);
      
      
      
      	Data opath.Simmer2000;
      		call streaminit(&seed);
      
      		SD_sub=&SD_sub;
      		SD_Day=&SD_Day;
      
      		do N = &NminSub to &NmaxSub;
      			do sim = 1 to &Nsim;
      				do Group = 0 to 1; *0 = Control, 1 = Treatment;	
      
      					%do i = 1 %to &ND;
      						if Group = 0 then P&i = &&CP&i;
      						if Group = 1 then P&i = &&CP&i*(1-&&TE&i);
      					%end;
      
      					array lstCP(*) P1-P&ND;
      
      					do Sub = 1 to N;
      
      						ANIMAL = compress(group||"-"||sub);
      
      						*===> create a random animal effect on logit from a NORMAL distribution;
      						if SD_sub = 0 then rand_sub=0;
      						if SD_sub ^= 0 then rand_sub = rand('normal', 0, SD_sub);
      
      						do Day = 1 to &ND;
      
      							*===> create a random day effect from a NORMAL distribution;
      							if SD_Day = 0  then  rand_day = 0; 
      							if SD_Day ^= 0 then  rand_day = rand('normal', 0, SD_Day); 
      
      							PG = lstCP[Day]; 
      							logit = log(PG/(1-PG)) + rand_sub + rand_day;
      
      							P = exp(logit)/(1+exp(logit));
      
      							*==> uniform, NTotal +- 10%;
      							Ntotal = round((1-&RNdiff)*&NPDay + 2*&RNdiff*&NPDay*rand("uniform"), 1); 
      
      							Nblood = rand('binomial', P, Ntotal);
      
      							logNT = log(NTotal);
      							output;
      						end; *day;
      					end; *Sub;
      				end; *Group;
      			end; *sim;
      		end; *N;
      		keep N Sim Group ANIMAL Day NTotal Nblood logNT;	
      	run;
      
      	ods listing close; 
      	ods select none;
      
      
      /**** GLIMMIX START ******/ 
      
      
      	data est1; set _NULL_; run;
      
      	%do i = &NminSub %to &NmaxSub;
      		dm log 'clear';
      		ods output estimates=est1_&i;
      		Proc glimmix data = opath.simmer2000(where=(N=&i)) method=rspl;
      			by N Sim;
      			class Group(ref="0") ANIMAL Day;
      			model Nblood/Ntotal = Group Day Group*Day/dist=binomial dfm=bw solution;
      			random day/subject=ANIMAL type = cs g gi;
      			random _residual_;
      			estimate 'treat1' intercept 1 group 1 day 1 group*day 1;
      			estimate 'treat2' intercept 1 group 1 day 0 1 group*day 0 1;
      			estimate 'treat3' intercept 1 group 1 day 0 0 1 group*day 0 0 1;
      			estimate 'treat4' intercept 1 group 1 day 0 0 0 1 group*day 0 0 0 1;
      			estimate 'contr1' intercept 1 group 0 1 day 1 group*day 0 0 0 0 1;
      			estimate 'contr2' intercept 1 group 0 1 day 0 1 group*day 0 0 0 0 0 1;
      			estimate 'contr3' intercept 1 group 0 1 day 0 0 1 group*day 0 0 0 0 0 0 1;
      			estimate 'contr4' intercept 1 group 0 1 day 0 0 0 1 group*day 0 0 0 0 0 0 0 1;
      			estimate 'total1' intercept 2 group 1 1 day 2 group*day 1 0 0 0 1;
      			estimate 'total2' intercept 2 group 1 1 day 0 2 group*day 0 1 0 0 0 1;
      			estimate 'total3' intercept 2 group 1 1 day 0 0 2 group*day 0 0 1 0 0 0 1;
      			estimate 'total4' intercept 2 group 1 1 day 0 0 0 2 group*day 0 0 0 1 0 0 0 1;
      		run;
      
      	/* Only include estimate files that exist (i.e. some simulations converged) */
      
      	%exist(est1_&i);
      
      	%if &exist=yes %then %do;
      
      		data est1;
      			set est1 est1_&i;
      		run;
      	%end;
      
      
      	%end;
      
      	/* Compute efficacy and precision by day */
      
      	data est1;
      	length var $ 5;
      	set est1;
      	keep estimate label stderr n sim day var df;
      	day=substr(label,6)+0.;
      	var=substr(label,1,5);
      	run;
      
      	proc sort data=est1;
      	by n sim day var;
      	run; 
      
      	proc transpose data=est1 out=estimate prefix=est;
      	by n sim day;
      	var estimate;
      	id var;
      	run;
      
      	proc transpose data=est1 out=stderr prefix=se;
      	by n sim day;
      	var stderr;
      	id var;
      	run;
      
      	proc transpose data=est1 out=df prefix=df;
      	by n sim day;
      	var df;
      	id var;
      	run;
      
      	data converged;
      	merge estimate stderr df;
      	by n sim day;
      	probtreat=1/(1+exp(-esttreat));
      	probcontr=1/(1+exp(-estcontr));
      	obs_te=1-probtreat/probcontr; /* efficacy */
      	vartreat=setreat*setreat;
      	varcontr=secontr*secontr;
      	covar2=setotal*setotal-vartreat-varcontr; /* Twice the covariance */
      	derivtreat=-exp(-esttreat)*probtreat*probtreat/probcontr; /* partial deriv of efficacy wrt esttreat */
      	derivcontr=exp(-estcontr)*probtreat; /* partial deriv of efficacy wrt estcontr */
      	varte=vartreat*derivtreat*derivtreat + varcontr*derivcontr*derivcontr + covar2*derivtreat*derivcontr;
      	/* delta or Taylor series method: efficacy = (approximately) constant + esttreat*derivtreat + estcontr*derivcontr */
      	obs_te_lower=obs_te - tinv(0.975,dftreat)*sqrt(varte); /* normal approximation */
      	run; 
      
      	*===> select datasets that were not convergenced;
      
      	Proc SQL;
      		create table Analy as
      		select distinct N, Sim
      		from est1
      		order by N, Sim;
      	quit;
      
      	Data opath.Simmer2000_1;
      		merge opath.simmer2000 analy(in=s1);
      		by N Sim;
      		if s1=0;
      	run;
      	Proc sort data = opath.Simmer2000_1;	by N Sim Day; run;
      
      	data est2; set _NULL_; run;
      
      	%do i = &NminSub %to &NmaxSub;
      		dm log 'clear';
      		ods output estimates=est2_&i;
      		Proc glimmix data = opath.simmer2000_1(where=(N = &i)) method=rspl;
      			by N Sim day;
      			class Group(ref="0") ANIMAL Day;
      			model Nblood/Ntotal = Group/dist=binomial dfm=bw solution;
      			random _residual_;
      			estimate 'treat' intercept 1 group 1;
      			estimate 'contr' intercept 1 group 0 1;
      			estimate 'total' intercept 2 group 1 1;
      		run;
      
      
      	/* Only include estimate files that exist (i.e. some simulations converged) */
      
      	%exist(est2_&i);
      
      	%if &exist=yes %then %do;
      
      		data est2;
      			set est2 est2_&i;
      		run;
      	%end;
      
      	%end;
      
      	/* Compute efficacy and precision by day */
      
      	data est2;
      	length var $ 5;
      	set est2;
      	keep estimate label stderr n sim day var df;
      	var=label;
      	run;
      
      	proc sort data=est2;
      	by n sim day var;
      	run; 
      
      	proc transpose data=est2 out=estimate2 prefix=est;
      	by n sim day;
      	var estimate;
      	id var;
      	run;
      
      	proc transpose data=est2 out=stderr2 prefix=se;
      	by n sim day;
      	var stderr;
      	id var;
      	run;
      
      	proc transpose data=est2 out=df2 prefix=df;
      	by n sim day;
      	var df;
      	id var;
      	run;
      
      	data notconverged;
      	merge estimate2 stderr2 df2;
      	by n sim day;
      	probtreat=1/(1+exp(-esttreat));
      	probcontr=1/(1+exp(-estcontr));
      	obs_te=1-probtreat/probcontr; /* efficacy */
      	vartreat=setreat*setreat;
      	varcontr=secontr*secontr;
      	covar2=setotal*setotal-vartreat-varcontr; /* Twice the covariance */
      	derivtreat=-exp(-esttreat)*probtreat*probtreat/probcontr; /* partial deriv of efficacy wrt esttreat */
      	derivcontr=exp(-estcontr)*probtreat; /* partial deriv of efficacy wrt estcontr */
      	varte=vartreat*derivtreat*derivtreat + varcontr*derivcontr*derivcontr + covar2*derivtreat*derivcontr;
      	/* delta or Taylor series method: efficacy = (approximately) constant + esttreat*derivtreat + estcontr*derivcontr */
      	obs_te_lower=obs_te - tinv(0.975,dftreat)*sqrt(varte); /* normal approximation */
      	run; 
      	
      	ods	listing; 
      	ods select default;
      
      
      	data all;
      	length Converged $ 3;
      	set converged(in=inconv) notconverged;
      	Precision = obs_TE - obs_TE_Lower;
      
      		%do j = 1 %to &ND;
      			if Day = &j then true_TE = &&TE&j;
      			if Day = &j then ReqTE = &&ReqTE&j;
      		%end;
      
      		%do k = 1 %to &NPrecision;
      			Precision&k = &&Precision&k;
      			success&k = (obs_TE >= ReqTE and Precision <= Precision&k);
      		%end;
      	if inconv then Converged="Yes";
      	else Converged="No";
      	run;
      
      	%let tvar1= %sysevalf(&CP1*100);
      	%let tvar2= %sysevalf(&SD_sub*100);
      	%let tvar3= %sysevalf(&SD_Day*1000);
      
      	Proc SQL;
      		create table opath.original_BN_P&tvar1._N&NPDay._S&tvar2._D&tvar3 as
      		select N, Day, true_TE, reqTE
      			%do k = 1 %to &NPrecision;
      				, avg(success&k) as Power&k label = "Precision <= &&Precision&k" format 6.3
      			%end;
      		from all
      		group by N, Day, true_TE, reqTE
      		order by N, Day, true_TE, reqTE;
      	quit;run;
      
      	title "Binomial-Normal: PC=&CP1, NPDay=&NPDay, SD_sub=&SD_sub, SD_Day=&SD_Day";
      	Proc Report data = opath.original_BN_P&tvar1._N&NPDay._S&tvar2._D&tvar3;
      		column N Day true_TE reqTE 
      			%do k = 1 %to &NPrecision;
      				Power&k
      			%end;;
      		define N/order;
      		define Day/display;
      		define true_TE/display;
      		define reqTE/display;
      			%do j = 1 %to &NPrecision;
      				define Power&j/display;
      			%end;
      	run;
      
      	proc freq data=all;
      	tables converged;
      	run;
      
      /**** GLIMMIX END ******/
      
      %Mend;
      
      ods rtf file = "&PETSTDY\CP=45 NPDay=50.RTF" startpage=no bodytitle;
      
      *==> study design 1;
      %PetProduct_NORMAL(NminSub=8,NmaxSub=12,NPDay=50,RNdiff=0.1, CP=0.45 0.45  0.45 0.45,
      															 TE=0.95 0.925 0.90 0.85,
      														  ReqTE=0.90 0.90  0.90 0.90,
      		ReqPrecision=0.025 0.030 0.040 0.050, SD_sub=0.33, SD_Day=0.166, Nsim=2000, seed=241354);
      
      
      
      *==> study design 2;
      ods rtf file = "&PETSTDY\CP=60 NPDay=100.RTF" startpage=no bodytitle;
      %PetProduct_NORMAL(NminSub=5,NmaxSub=10,NPDay=100,RNdiff=0.1, CP=0.60 0.60  0.60 0.60,
      															 TE=0.95 0.925 0.90 0.85,
      														  ReqTE=0.90 0.90  0.90 0.90,
      		ReqPrecision=0.025 0.030 0.040 0.050, SD_sub=0.33, SD_Day=0.166, Nsim=2000, seed=241354);
      ods rtf close;
      
      
      
      *==> study design 3;
      ods rtf file = "&PETSTDY\CP=80 NPDay=50.RTF" startpage=no bodytitle;
      %PetProduct_NORMAL(NminSub=5,NmaxSub=10,NPDay=50,RNdiff=0.1, CP=0.80 0.80  0.80 0.80,
      															 TE=0.95 0.925 0.90 0.85,
      														  ReqTE=0.90 0.90  0.90 0.90,
      		ReqPrecision=0.025 0.030 0.040 0.050, SD_sub=0.33, SD_Day=0.166, Nsim=2000, seed=241354);
      ods rtf close;
      
      
      *==> study design 4;
      ods rtf file = "&PETSTDY\CP=80 NPDay=25.RTF" startpage=no bodytitle;
      %PetProduct_NORMAL(NminSub=5,NmaxSub=25,NPDay=25,RNdiff=0.1, CP=0.80 0.80  0.80 0.80,
      															 TE=0.95 0.925 0.90 0.85,
      														  ReqTE=0.90 0.90  0.90 0.90,
      		ReqPrecision=0.025 0.030 0.040 0.050, SD_sub=0.33, SD_Day=0.166, Nsim=2000, seed=241354);
      ods rtf close;
      
      *==> study design 4;
      ods rtf file = "&PETSTDY\CP=85 NPDay=25.RTF" startpage=no bodytitle;
      %PetProduct_NORMAL(NminSub=8,NmaxSub=13,NPDay=25,RNdiff=0.1, CP=0.85 0.85  0.85 0.85,
      															 TE=0.95 0.925 0.90 0.85,
      														  ReqTE=0.90 0.90  0.90 0.90,
      		ReqPrecision=0.025 0.030 0.040 0.050, SD_sub=0.33, SD_Day=0.166, Nsim=2000, seed=241354);
      ods rtf close;
      
      
      *==> study design 5;
      ods rtf file = "&PETSTDY\CP=75 NPDay=100.RTF" startpage=no bodytitle;
      %PetProduct_NORMAL(NminSub=5,NmaxSub=10,NPDay=100,RNdiff=0.1, CP=0.75 0.75  0.75 0.75,
      															 TE=0.95 0.925 0.90 0.85,
      														  ReqTE=0.90 0.90  0.90 0.90,
      		ReqPrecision=0.025 0.030 0.040 0.050, SD_sub=0.33, SD_Day=0.166, Nsim=2000, seed=241354);
      ods rtf close;



SAS code of an example data analysis
(An example of data analysis of pet product studies.sas)

      *=================================================================================*
      * Programmer: James Nguyen, USEPA                                                 *
      *                                                                                 *
      * Purpose: To analyze data of Pet Product Efficacy Studies                        *
      *                                                                                 *
      * Methods: Generalized Linear Mixed Effects Models (GLMM) with logit link         *
      *          function for binomial distribution data are used to analyze the data.  *
      *                                                                                 *
      *        - A GLMM model with over/under-dispersion (model 1) is first used to     *
      *          analyze the data.  If model 1 converges, the point estimated efficacy  *
      *          of the treatment is calculated using the proportions of blood-fed      *
      *          insects (of the groups) estimated from this model 1.                   *
      *        - If model 1 fails to converge, the data are then analyzed using a GLMM  *
      *          model without the over/under-dispersion option (model 2). If model 2   *
      *          converges, the point estimated efficacy of the treatment is calculated *
      *          using the proportions of blood-fed insects (of the groups) estimated   *
      *          from this model.                                                       *
      *        - If model 2 fails to converge, a GLMM with Laplace estimation method    *
      *          and without the over/under-dispersion option (model 3) is used to      *
      *          analyze the data.  If model 3 is converged, the point estimated        *
      *          efficacy of the treatment is calculated using the proportions of       *
      *          blood-fed insects (of the groups) estimated from this model.           *
      *        - If model 3 fails to converge, a generalized linear model (model 4) is  *
      *          used to analyze the data on each day separately. The point estimated   *
      *          efficacy of the treatment is calculated using the proportions of       *
      *          blood-fed insects (of the groups) estimated from this model.  (Note    *
      *          that the probability for all three models 1, 2, and 3 above fail to    *
      *          converge for a dataset is very small (probably less than 0.5%)).       *
      *        - 95% CI of the point estimated efficacy is obtained using bootstrapping *
      *          technique.  1000 bootstrap datasets are created using resampling       *
      *          technique with replacement.  Each of bootstrap dataset is analyzed as  *
      *          the procedure (models 1, 2, 3, and 4) above.  The 95% CI of the        *
      *          estimated efficacy is the 2.5th and 97.5th percentile of the efficacy  *
      *          estimates obtained from 1000 bootstrap datasets.                       *
      *                                                                                 *
      * Date: 11/30/2020                                                                *
      *=================================================================================*;
      option formdlim="=" ls=100 ps=100;
      
      libname opath "C:\Users\JNguyen\OneDrive - Environmental Protection Agency (EPA)\Insect Repellency\Sample Size of Repellency Studies\Sample size of Pet Product studies\Final";
      
      Data example1;
      	set opath.Simmer2000_1;
      	if n = 12;
      	if sim = 1;
      	rename group = Treatment;
      	drop sim n logNT;
      run;
      
      ods rtf;
      Proc print data = example1 noobs; run;
      ods rtf close;
      
      %Macro est;
      	estimate 'treat1' intercept 1 Treatment 1 day 1 Treatment*day 1;
      	estimate 'treat2' intercept 1 Treatment 1 day 0 1 Treatment*day 0 1;
      	estimate 'treat3' intercept 1 Treatment 1 day 0 0 1 Treatment*day 0 0 1;
      	estimate 'treat4' intercept 1 Treatment 1 day 0 0 0 1 Treatment*day 0 0 0 1;
      	estimate 'contr1' intercept 1 Treatment 0 1 day 1 Treatment*day 0 0 0 0 1;
      	estimate 'contr2' intercept 1 Treatment 0 1 day 0 1 Treatment*day 0 0 0 0 0 1;
      	estimate 'contr3' intercept 1 Treatment 0 1 day 0 0 1 Treatment*day 0 0 0 0 0 0 1;
      	estimate 'contr4' intercept 1 Treatment 0 1 day 0 0 0 1 Treatment*day 0 0 0 0 0 0 0 1;
      %Mend;
      
      %Macro Analysis(datain=, 	/* name dataset */
                      varN=, 	/* name variable of interest: number blood-fed, retained, or survival insects */
                      boots=, 	/* number of bootstrap datasets used to obtain the 95% CI */
                      seed=);	/* random seed number */
      
      	Proc sort data=&datain;
      		by Treatment animal;
      	run;
      
      	Proc sort data=&datain out=animals(keep=Treatment animal) nodupkey;
      		by Treatment animal;
      	run;
      
      	Proc SQL;
      		create table sampsize as
      			select Treatment, count(*) as samplesize
      			from animals
      			group by Treatment;
      	quit;
      
      	*==> for each bootstrap replicate, select a random sample of n animals with 
      		 replacement for each Treatment group;
      
      	Proc surveyselect out=animals2 data=animals method=urs outhits reps=&boots 
      					sampsize=sampsize seed=&seed;  
      		strata Treatment;
      	run;
      
      	Proc sort data = animals2;
      		by replicate Treatment;
      	run;
      
      	Data animals3;
      		length newanimal $10.;
      		set animals2;
      		by replicate Treatment;
      		retain count 0;
      
      		if first.Treatment then count=0;
      		count+1;
      		newanimal=compress(animal || "-" || put(count,6.));
      	run;
      
      	Proc SQL;
      		create table animals4 as
      			select a.Treatment, a.animal, a.replicate, a.newanimal, b.day, b.&varn, b.ntotal
      			from animals3 a, &datain b
      			where a.Treatment=b.Treatment and a.animal=b.animal;
      	quit; run; 
      
      
      	Data animals5(rename=(newanimal=animal));
      		set &datain(in=base) animals4;
      		if base then do; 
      			replicate=0; 
      			newanimal=animal; 
      		end;
      		drop animal;
      	run;
      
      	proc sort data=animals5;
      		by replicate Treatment animal day;
      	run;
      
      /*	dm log 'clear';*/
      
      
      
      	ods select none;
      
      	*===> model 1;
      
      	ods output estimates=est1;
      	Proc glimmix data = animals5 method=rspl; 
      		by replicate;
      		class Treatment(ref="0") ANIMAL Day;
      		model &varn/Ntotal = Treatment Day Treatment*Day/dist=binomial link=logit dfm=bw;
      		random day/subject=ANIMAL type = cs g gi;
      		random _residual_;
      		%est;
      	run;
      
      	data est1;
      		length var $5;
      		set est1;
      		keep estimate label replicate day var;
      		day=substr(label,6)+0.;
      		var=substr(label,1,5);
      	run;
      
      	proc sort data=est1;
      		by replicate day var;
      	run; 
      
      	proc transpose data=est1 out=estimate1 prefix=est;
      		by replicate day;
      		var estimate;
      		id var;
      	run;
      
      	data estimate1;
      		set estimate1;
      		probtreat=1/(1+exp(-esttreat));
      		probcontr=1/(1+exp(-estcontr));
      		obs_te=1-probtreat/probcontr; /* efficacy */
      		converged = (obs_te ne .);
      	run;
      
      	Data Estimates;
      		set Estimate1;
      	run;
      
      	*==> select datasets (replicates) that converged;
      
      	proc means data=estimate1 ;
      		by replicate;
      		var converged;
      		output out=converged1 sum=sumconverged;
      	run;
      
      	data analy1;
      		set converged1;
      		where sumconverged =4;
      		keep replicate;
      	run; 
      
      	*==> select replicates failed to converge;
      
      	Data Animals6;
      		merge animals5 analy1(in=s1);
      		by replicate;
      		if s1=0;
      	run;
      
      	Proc SQL noprint;
      		select count(*) into : nrow6
      		from Animals6;
      	quit;
      	%let nrow6=&nrow6;
      
      	*==> Analyze the unconverged replicates without using over/under-dispersion;
      
      	%if &nrow6 > 0 %then %do;
      
      		*===> model 2;
      
      		ods output estimates=est2;
      		Proc glimmix data = animals6 method=rspl; 
      			by replicate;
      			class Treatment(ref="0") ANIMAL Day;
      			model &varn/Ntotal = Treatment Day Treatment*Day/dist=binomial link=logit dfm=bw;
      			random day/subject=ANIMAL type = cs g gi;
      			%est;
      		run;
      
      		data est2;
      			length var $5;
      			set est2;
      			keep estimate label replicate day var;
      			day=substr(label,6)+0.;
      			var=substr(label,1,5);
      		run;
      
      		proc sort data=est2;
      			by replicate day var;
      		run; 
      
      		proc transpose data=est2 out=estimate2 prefix=est;
      			by replicate day;
      			var estimate;
      			id var;
      		run;
      
      		data estimate2;
      			set estimate2;
      			probtreat=1/(1+exp(-esttreat));
      			probcontr=1/(1+exp(-estcontr));
      			obs_te=1-probtreat/probcontr; /* efficacy */
      			converged = (obs_te ne .);
      		run;
      
      		Data Estimates;
      			set Estimates Estimate2;
      		run;
      
      		*==> select datasets (replicates) that converged;
      
      		proc means data=estimate2 ;
      			by replicate;
      			var converged;
      			output out=converged2 sum=sumconverged;
      		run;
      
      		data analy2;
      			set converged2;
      			where sumconverged =4;
      			keep replicate;
      		run; 
      
      		*==> select replicates failed to converge;
      
      		Data Animals7;
      			merge animals6 analy2(in=s1);
      			by replicate;
      			if s1=0;
      		run;
      
      		Proc SQL noprint;
      			select count(*) into : nrow7
      			from Animals7;
      		quit;
      		%let nrow7=&nrow7;
      
      
      		%if &nrow7 > 0 %then %do;
      
      			*===> model 3;
      
      			ods output estimates=est3;
      			Proc glimmix data = Animals7 method=laplace; 
      				by replicate;
      				class Treatment(ref="0") ANIMAL Day;
      				model &varn/Ntotal = Treatment Day Treatment*Day/dist=binomial link=logit dfm=bw ;
      				random day/subject=ANIMAL type = cs g gi;
      				%est;
      			run;
      
      			data est3;
      				length var $5;
      				set est3;
      				keep estimate label replicate day var;
      				day=substr(label,6)+0.;
      				var=substr(label,1,5);
      			run;
      
      			proc sort data=est3;
      				by replicate day var;
      			run; 
      
      			proc transpose data=est3 out=estimate3 prefix=est;
      				by replicate day;
      				var estimate;
      				id var;
      			run;
      
      			data estimate3;
      				set estimate3;
      				probtreat=1/(1+exp(-esttreat));
      				probcontr=1/(1+exp(-estcontr));
      				obs_te=1-probtreat/probcontr; /* efficacy */
      				converged = (obs_te ne .);
      			run;
      
      			Data Estimates;
      				set Estimates Estimate3;
      			run;
      
      			*==> select datasets (replicates) that converged;
      
      			proc means data=estimate3 ;
      				by replicate;
      				var converged;
      				output out=converged3 sum=sumconverged;
      			run;
      
      			data analy3;
      				set converged3;
      				where sumconverged =4;
      				keep replicate;
      			run; 
      
      			*==> select replicates failed to converge;
      
      			Data Animals8;
      				merge animals7 analy3(in=s1);
      				by replicate;
      				if s1=0;
      			run;
      
      			Proc SQL noprint;
      				select count(*) into : nrow8
      				from Animals8;
      			quit;
      			%let nrow8=&nrow8;
      
      			%if &nrow8 > 0 %then %do;
      
      				*===> model 4;
      
      				ods output estimates=est4;
      				Proc glimmix data = Animals8 method=rspl;
      					by replicate day;
      					class Treatment(ref="0") ANIMAL Day;
      					model &varn/Ntotal = Treatment/dist=binomial link=logit dfm=bw ;
      					estimate 'treat' intercept 1 group 1;
      					estimate 'contr' intercept 1 group 0 1;
      				run;
      
      				data est4;
      					length var $ 5;
      					set est4;
      					keep estimate label replicate day var;
      					var=substr(label,1,5);
      				run;
      
      				proc sort data=est4;
      					by replicate day var;
      				run;
      
      				proc transpose data=est4 out=estimate4 prefix=est;
      					by replicate day;
      					var estimate;
      					id var;
      				run;
      
      				data estimate4;
      					set estimate4;
      					probtreat=1/(1+exp(-esttreat));
      					probcontr=1/(1+exp(-estcontr));
      					obs_te=1-probtreat/probcontr; /* efficacy */
      				run;
      
      				Data Estimates;
      					set Estimates Estimate4;
      				run;
      
      			%end; *nrow8;
      
      		%end; *nrow7;
      
      	%end; *nrow6;
      
      	Proc sort data = Estimates;
      		by Day;
      	run;
      
      	Data Estimate0 Estimateboots;
      		set Estimates;
      		if replicate = 0 then output Estimate0;
      		else output Estimateboots;
      	run;
      
      	Proc univariate data = Estimateboots noprint;
      		by Day;
      		var obs_te;
      		output out=pctls(keep=Day Est_lower Est_upper) pctlpts=2.5 97.5 
      				pctlpre=Est_ pctlname= lower upper;
      	run;
      
      	Data Est_Efficacy;
      		merge Estimate0(keep= Day probtreat probcontr obs_te) pctls;
      		by Day;
      		rename obs_te = Estimated_Efficacy;
      		label Est_lower = "lower 95% CI" Est_upper = "upper 95% CI";
      	run;
      
      	ods select default;
      	Proc print data = Est_Efficacy noobs label; 
      		format probtreat probcontr Estimated_Efficacy Est_lower Est_upper 6.3;
      	run;
      
      %Mend;
      
      
      %Analysis(datain=Example1, 	/* name dataset */
                varN=Nblood, 	/* name variable of interest: number blood-fed, retained, or survival insects */
                boots=1000, 		/* number of bootstrap datasets used to obtain the 95% CI */
                seed=23648);		/* random seed number*/
      


