DEVELOPING AND USING INFORMATION COLLECTION INSTRUMENTS FOR ASSESSING
THE EFFECTIVENESS OF TRAINING, EDUCATION, AND OUTREACH ACTIVITIES UNDER
EPA’S ASSISTANCE AGREEMENTS

FEBRUARY 25, 2008TABLE OF CONTENTS

  TOC \h \z \t "Heading 1,2,Heading 2,3,Title,1"    HYPERLINK \l
"_Toc190442905"  Preface	  PAGEREF _Toc190442905 \h  1  

  HYPERLINK \l "_Toc190442906"  Section 1: Overview Of Questionnaire
Development And Information Collection Process	  PAGEREF _Toc190442906
\h  2  

  HYPERLINK \l "_Toc190442907"   Why a Special Process?	  PAGEREF
_Toc190442907 \h  2  

  HYPERLINK \l "_Toc190442908"  How Does the Process Work?	  PAGEREF
_Toc190442908 \h  2  

  HYPERLINK \l "_Toc190442909"  Special Requirements	  PAGEREF
_Toc190442909 \h  3  

  HYPERLINK \l "_Toc190442910"  Program Office Role	  PAGEREF
_Toc190442910 \h  3  

  HYPERLINK \l "_Toc190442911"  Section 2: Questionnaire Development
Process	  PAGEREF _Toc190442911 \h  5  

  HYPERLINK \l "_Toc190442912"  Step 1: Identify Type of Assessment	 
PAGEREF _Toc190442912 \h  5  

  HYPERLINK \l "_Toc190442913"  Step 2: Select Appropriate Questionnaire
  PAGEREF _Toc190442913 \h  6  

  HYPERLINK \l "_Toc190442914"  Step 3: Include Optional Questions	 
PAGEREF _Toc190442914 \h  6  

  HYPERLINK \l "_Toc190442915"  Step 4: Include Program Office Questions
  PAGEREF _Toc190442915 \h  7  

  HYPERLINK \l "_Toc190442916"  Step 5: Tailor Questions and Response
Options	  PAGEREF _Toc190442916 \h  7  

  HYPERLINK \l "_Toc190442917"  Tailoring Questions	  PAGEREF
_Toc190442917 \h  7  

  HYPERLINK \l "_Toc190442918"  Tailoring Response Options	  PAGEREF
_Toc190442918 \h  8  

  HYPERLINK \l "_Toc190442919"  Section 3: Required And Optional
Questions For Post-Training Questionnaires	  PAGEREF _Toc190442919 \h  9
 

  HYPERLINK \l "_Toc190442920"  Required Questions for Post-Training
Questionnaires	  PAGEREF _Toc190442920 \h  9  

  HYPERLINK \l "_Toc190442921"  Optional Questions For Post-Training
Questionnaires	  PAGEREF _Toc190442921 \h  10  

  HYPERLINK \l "_Toc190442922"  Section 4: Required And Optional
Questions For Follow-Up Questionnaires	  PAGEREF _Toc190442922 \h  14  

  HYPERLINK \l "_Toc190442923"  Required Questions for Follow-Up
Questionnaires	  PAGEREF _Toc190442923 \h  15  

  HYPERLINK \l "_Toc190442924"  Optional Questions for Follow-Up
Questionnaires	  PAGEREF _Toc190442924 \h  16  

  HYPERLINK \l "_Toc190442925"  Section 5: Response Options For
Anticipated Or Actual Behavior	  PAGEREF _Toc190442925 \h  18  

  HYPERLINK \l "_Toc190442926"  General Response Options	  PAGEREF
_Toc190442926 \h  21  

  HYPERLINK \l "_Toc190442927"  Detailed Response Options	  PAGEREF
_Toc190442927 \h  22  

  HYPERLINK \l "_Toc190442928"  Program-Specific Response Options	 
PAGEREF _Toc190442928 \h  26  

  HYPERLINK \l "_Toc190442929"  Section 6: Preparing Questionnaires	 
PAGEREF _Toc190442929 \h  30  

  HYPERLINK \l "_Toc190442930"  Required Questions	  PAGEREF
_Toc190442930 \h  30  

  HYPERLINK \l "_Toc190442931"  Optional Questions	  PAGEREF
_Toc190442931 \h  30  

  HYPERLINK \l "_Toc190442932"  Section 7: Questionnaire Administration
And Assistance Agreement Recipient Reporting	  PAGEREF _Toc190442932 \h 
34  

  HYPERLINK \l "_Toc190442933"  Questionnaire Administration	  PAGEREF
_Toc190442933 \h  34  

  HYPERLINK \l "_Toc190442934"  Reporting	  PAGEREF _Toc190442934 \h  34
 

  HYPERLINK \l "_Toc190442935"  Attachment A: Questionnaire
Administration Decisions	A-  PAGEREF _Toc190442935 \h  1  

  HYPERLINK \l "_Toc190442936"  Planning the Survey Administration
Process	A-  PAGEREF _Toc190442936 \h  2  

  HYPERLINK \l "_Toc190442937"  Sampling Frame	A-  PAGEREF _Toc190442937
\h  2  

  HYPERLINK \l "_Toc190442938"  Sample Selection	A-  PAGEREF
_Toc190442938 \h  3  

  HYPERLINK \l "_Toc190442939"  Data Collection Method	A-  PAGEREF
_Toc190442939 \h  5  

  HYPERLINK \l "_Toc190442940"  Follow-Up to Enhance Response Rate	A- 
PAGEREF _Toc190442940 \h  6  

  HYPERLINK \l "_Toc190442941"  Developing a Plan	A-  PAGEREF
_Toc190442941 \h  8  

  HYPERLINK \l "_Toc190442942"  Attachment B : Guidance For OMB
Clearance Process	B-  PAGEREF _Toc190442942 \h  1  

  HYPERLINK \l "_Toc190442943"  I. Guidance for Preparing Part B
Document	B-  PAGEREF _Toc190442943 \h  3  

  HYPERLINK \l "_Toc190442944"  1. Survey Objectives, Key Variables, And
Other Preliminaries	B-  PAGEREF _Toc190442944 \h  3  

  HYPERLINK \l "_Toc190442945"  2. Survey Design	B-  PAGEREF
_Toc190442945 \h  4  

  HYPERLINK \l "_Toc190442946"  3. Pretests And Pilot Tests	B-  PAGEREF
_Toc190442946 \h  6  

  HYPERLINK \l "_Toc190442947"  4. Collection Methods And Follow-up	B- 
PAGEREF _Toc190442947 \h  7  

  HYPERLINK \l "_Toc190442948"  5. Analyzing And Reporting Survey
Results	B-  PAGEREF _Toc190442948 \h  7  

  HYPERLINK \l "_Toc190442949"  Glossary of Terms	B-  PAGEREF
_Toc190442949 \h  7  

  HYPERLINK \l "_Toc190442950"  II. Guidance for Preparing Burden
Worksheet	B-  PAGEREF _Toc190442950 \h  8  

  HYPERLINK \l "_Toc190442951"  1. Survey Objectives, Key Variables, And
Other Preliminaries	B-  PAGEREF _Toc190442951 \h  14  

  HYPERLINK \l "_Toc190442952"  2. Survey Design	B-  PAGEREF
_Toc190442952 \h  15  

  HYPERLINK \l "_Toc190442953"  3. Pretests And Pilot Tests	B-  PAGEREF
_Toc190442953 \h  16  

  HYPERLINK \l "_Toc190442954"  4. Collection Methods And Follow-up	B- 
PAGEREF _Toc190442954 \h  16  

  HYPERLINK \l "_Toc190442955"  5. Analyzing And Reporting Survey
Results	B-  PAGEREF _Toc190442955 \h  17  

  HYPERLINK \l "_Toc190442956"  Attachment C: Reporting Forms	C- 
PAGEREF _Toc190442956 \h  1  

  HYPERLINK \l "_Toc190442957"  Attachment D: Questionnaire Formats	D- 
PAGEREF _Toc190442957 \h  1  

 

Table of Exhibits

  TOC \h \z \t "Heading 3" \c    HYPERLINK \l "_Toc190441915"  Exhibit
1: Questionnaire Development and Information Collection Process	 
PAGEREF _Toc190441915 \h  4  

  HYPERLINK \l "_Toc190441916"  Exhibit 2: Identifying Appropriate
Questionnaire	  PAGEREF _Toc190441916 \h  6  

  HYPERLINK \l "_Toc190441917"  Exhibit 3: Tailoring Questions Through
Completion of Bracketed Phrases	  PAGEREF _Toc190441917 \h  8  

  HYPERLINK \l "_Toc190441918"  Exhibit 4: Tailoring Questions Through
Deletion of Unnecessary Terms	  PAGEREF _Toc190441918 \h  8  

  HYPERLINK \l "_Toc190441919"  Exhibit 5: Required Post-Training
Questions	  PAGEREF _Toc190441919 \h  10  

  HYPERLINK \l "_Toc190441920"  Exhibit 6: “Menu” of Optional
Post-Training Questions	  PAGEREF _Toc190441920 \h  11  

  HYPERLINK \l "_Toc190441921"  Exhibit 7:  Required Follow-up Questions
  PAGEREF _Toc190441921 \h  15  

  HYPERLINK \l "_Toc190441922"  Exhibit 8:   “Menu” of Optional
Follow-up Questions	  PAGEREF _Toc190441922 \h  17  

  HYPERLINK \l "_Toc190441923"  Exhibit 9:  Text of Anticipated and
Actual Behavior Questions	  PAGEREF _Toc190441923 \h  18  

  HYPERLINK \l "_Toc190441924"  Exhibit 10: Anticipated and Actual
Actions: Types of Response Options	  PAGEREF _Toc190441924 \h  20  

  HYPERLINK \l "_Toc190441925"  Exhibit 11: Adding Optional Questions to
a Questionnaire	  PAGEREF _Toc190441925 \h  32  

  HYPERLINK \l "_Toc190441926"  Exhibit 12: Tailoring Phrases and Adding
Responses Options to a Questionnaire	  PAGEREF _Toc190441926 \h  33  

  HYPERLINK \l "_Toc190441928"  Exhibit A-1: Questionnaire
Administration Issues	A-  PAGEREF _Toc190441928 \h  2  

  HYPERLINK \l "_Toc190441929"  Exhibit A-2: Calculation of Required
Sample Size	A-  PAGEREF _Toc190441929 \h  4  

  HYPERLINK \l "_Toc190441930"  Exhibit A-3: Follow-Up Method to Enhance
Response Rate	A-  PAGEREF _Toc190441930 \h  7  

  HYPERLINK \l "_Toc190441931"  Exhibit B-1: Overview of Expedited OMB
Clearance Process	B-  PAGEREF _Toc190441931 \h  2  

  HYPERLINK \l "_Toc190441932"  Exhibit B-2: Burden Calculation
Worksheet	B-  PAGEREF _Toc190441932 \h  10  

  HYPERLINK \l "_Toc190441933"  Exhibit B-3: Checklist for Reviewing
Clearance Request Prior to Submittal	B-  PAGEREF _Toc190441933 \h  11  

  HYPERLINK \l "_Toc190441934"  Exhibit B-4: Sample OMB Clearance
Package Documents	B-  PAGEREF _Toc190441934 \h  12  

  HYPERLINK \l "_Toc190441935"  Exhibit C-1: Reporting Forms Required by
Type of Activity	C-  PAGEREF _Toc190441935 \h  1  

 



Preface



This document describes a streamlined process for developing and using
pre-approved information collection instruments to assess the
effectiveness of training and outreach activities conducted under
EPA’s assistance agreements (including grants and cooperative
agreements). It is designed to make it more efficient for Assistance
Agreement Recipients to get the approval needed from the Office of
Management and Budget (OMB) before collecting questionnaire data from
more than nine members of the public. Don’t be daunted by the length
of this document. Developing a questionnaire and getting it approved is
easy. In a nutshell, the process looks like this:

Select a questionnaire format appropriate to your program – it will
contain one or more required questions.

Add to that format any questions you wish to select from a “Menu” of
pre-approved optional questions. 

Decide how you will make sure that the questionnaire goes to the right
people, and that those people actually return it.

Summarize your plan in a document for expedited OMB approval and submit
it to the EPA Project Officer. 

After receiving OMB approval, 

Administer the questionnaire. 

Enter the data received in response to the questionnaire into the Excel
document Assistance Agreement Recipient Reporting Form.

Send the completed Assistance Agreement Recipient Reporting Form in to
your EPA Project Officer.

Use the results to assess your activity’s success, and decide what
kinds of improvements you’d like to make in the future. 

In the pages that follow, you’ll find clarification about each step in
the process, as well as formats, examples, and tips. If you have any
questions, don’t hesitate to contact your EPA Project Officer for
assistance. 

Section 1: Overview Of Questionnaire Development And Information
Collection Process 

Why a Special Process?

Assistance Agreement Recipients involved in training, education, and
outreach need to be able to assess whether their programs are effective.
A survey is one way to collect such information. However, developing
appropriate questions and methods requires survey research skills that
Assistance Agreement Recipient staff may not have. In addition,
questionnaires that have been developed by or in conjunction with the
Federal government, and that collect information from more than nine
people, must be approved by OMB. 

To give Assistance Agreement Recipients an efficient way to collect
information, EPA’s Office of Grants and Debarment (OGD) has worked
with OMB to obtain pre-approval of questions that may be applicable for
multiple Assistance Agreement Recipients. Assistance Agreement
Recipients must still receive OMB approval before administering any
survey. But using pre-approved questions and data collection methods
will minimize the burden on Assistance Agreement Recipients and allow
for an expedited OMB review process.

How Does the Process Work?

Assistance Agreement Recipients and EPA Project Officers work together
to develop an appropriate questionnaire and collect the needed
information. Steps in the process are illustrated in the flowchart
provided in  HYPERLINK  \l "Exhibit1" Exhibit 1 . They include the
following:

Identify Appropriate Questionnaire Format and Question Types. Somewhat
different pre-approved questionnaire formats and questions have been
developed for use immediately after a training session, immediately
after an outreach effort, and for follow-up 3-6 months following a
training or outreach effort. The Assistance Agreement Recipient must
decide which are appropriate for the activity to be evaluated. 

Develop Questionnaire. Assistance Agreement Recipients develop a
questionnaire based on a list of pre-approved questions and response
options. Some of those questions are required by OGD. Others may be
required by particular program offices. Still others are optional, and
can be selected from a “Menu” of options at the Assistance Agreement
Recipient’s discretion. The “Menu” of questions and questionnaire
formats are provided in   HYPERLINK  \l "SectionPostTraining"  Section 3
 (Post-Training) and   HYPERLINK  \l "SectionFollowUp"  Section 4 
(Follow-Up).

Develop Strategy. The Assistance Agreement Recipient prepares an
approach for administering the survey in accordance with OMB
requirements and  other best practices.  Guidance is provided in  
HYPERLINK  \l "AttachmentA"  Attachment A  on survey approaches.

Get EPA Project Officer Review. The Assistance Agreement Recipient
submits an expedited clearance request to the EPA Project Officer that
includes its proposed questionnaire and related information on its
collection approach.  The Project Officer will review them for
compliance and completeness. EPA then submits the request to OMB for
approval. The Project Officer will submit the reviewed document through
the EPA Gatekeeper and EPA’s Office of Environmental Information
(OEI), after which it will be submitted to OMB for approval. 
Information about how to prepare the material needed for OMB is provided
in   HYPERLINK  \l "AttachmentB"  Attachment B .

Administer Survey. After receiving OMB approval the Assistance Agreement
Recipient distributes the questionnaires and collects responses from
program participants.

Report Results. The Assistance Agreement Recipient reports the results
to the EPA Project Officer. The reporting form is provided in the form
of an Excel spreadsheet to eliminate the need for manual calculations.
This companion document is discussed in  HYPERLINK  \l "AttachmentC"
Attachment C .

Special Requirements

In order to take advantage of expedited OMB clearance, Assistance
Agreement Recipients must carefully adhere to the following
requirements.

CAUTION!

Keep the wording of all pre-approved questions exactly as it is
presented unless the instructions specifically authorize you to make
changes.

Use only questions from the approved list, without adding any new
questions.

Always include the question(s) required of all Assistance Agreement
Recipients.

Always include the question(s) required by your EPA Project Officer (if
any).

More details about these requirements are provided as part of the
questionnaire development process described in   HYPERLINK  \l
"SectionPostTraining"  Section 3  (Post-Training) and   HYPERLINK  \l
"SectionFollowUp"  Section 4  (Follow-Up).

Program Office Role

In some situations, Program Offices may choose to develop a
questionnaire and survey administration procedures that may be used by
multiple grantees. In this situation, the Program Office would follow
the instructions in this document for developing a questionnaire for OMB
approval. The Program Office also would estimate the collective burden
for all Assistance Agreement Recipients that are expected to use the
approved instrument, as described in   HYPERLINK  \l "AttachmentB" 
Attachment B . If an Assistance Agreement Recipient has obtained OMB
approval for a questionnaire and supporting materials, the Program
Office may use those materials as the basis for obtaining this type of
collective approval from OMB.  

If a Program Office has obtained OMB approval for use of a
questionnaire, the Assistance Agreement Recipient can simply use the
approved questionnaire as-is, and is not required to seek additional OMB
approval. In this situation the Assistance Agreement Recipient would not
need to follow the instructions in this document, but would instead work
closely with the Program Office to ensure compliance with all OMB
requirements. 

Exhibit 1:

Questionnaire Development and Information Collection Process

 

Section 2: Questionnaire Development Process

This section describes how to develop a questionnaire to assess the
effectiveness of a training, education, or outreach activity funded
under an EPA assistance agreement using pre-approved questions.
Developing a questionnaire with pre-approved questions is not difficult.
However, certain steps must be followed in order to ensure compliance
with OMB requirements. 

Step 1: Identify type of assessment (training or outreach; immediate or
longer-term)

Step 2: Select appropriate questionnaire for the chosen question type,
including required questions. 

Step 3: Include optional questions in the questionnaire (if desired)

Step 4: Include questions required by the Program Office in the
questionnaire (if applicable)

Step 5: Tailor questions (if applicable). (Only limited types of
tailoring are permitted. Please read instructions carefully before
tailoring any question.)

Each of these steps is summarized below. The rest of the document
provides detailed information about how to implement each step in the
process.

If you have questions at any point in the process, contact the EPA
Project Officer or EPA’s Office of Grants and Debarment (GAD) for
assistance.





Step 1: Identify Type of Assessment

Knowing the type of assessment needed will guide Assistance Agreement
Recipients in selecting appropriate questions. To determine the type of
assessment consider: 

Kind of activity. This refers to whether the activity involves training
or outreach.

Time frame. This refers to whether the Assistance Agreement Recipient
wants to: 

Investigate the likely effectiveness of an activity immediately after
the activity has taken place (Prospective), or 

Determine what the longer-term effect of the activity has been
(Retrospective). 

Step 2: Select Appropriate Questionnaire

Questionnaires are available to investigate the effectiveness of
different types of assessments. 

The post-training questionnaire asks about changes in the
participant’s awareness, understanding, and anticipated behavior
stemming from the training. It also asks about perceptions of the
training itself (e.g., skills of the presenter, quality of the
materials, comfort of facilities). It is designed for use immediately
after a training session.

The follow-up questionnaires focus on anticipated or actual changes in
behavior. Changes in verb tense (past or future) allow these questions
to be applied either prospectively or retrospectively. These
questionnaires can be used to determine what action a respondent has
taken, or plans to take. 

  HYPERLINK  \l "Exhibit2IDAppropriateQuestionnaire"  Exhibit 2 
illustrates the situations in which each questionnaire should be used. 

Exhibit 2:

Identifying Appropriate Questionnaire

	Training	Outreach

Prospective (Immediately after the activity 	Post-Training Questionnaire
Follow-Up Questionnaire:

Prospective

Retrospective (Longer-term effect)	Follow-Up Questionnaire:

Retrospective	Follow-Up Questionnaire:

Retrospective



Each questionnaire includes one or more required questions. Because
these questions must be appropriate for a wide range of Assistance
Agreement Recipients, they investigate key indicators of success using
general enough wording to be applicable to any Assistance Agreement
Recipient’s work. EPA can then use data from the required questions to
assess the effectiveness of assistance agreement activities involving
training, education, and outreach nationwide. For this reason, these
questions must be included in the survey and must not be edited or
changed unless the instructions specifically authorize you to do so.

Step 3: Include Optional Questions

Some Assistance Agreement Recipients may wish to obtain more detailed
information for their own use. Such information would not be useful to
EPA for purposes of a nation-wide comparison. However, it can be
extremely valuable for purposes of program improvement and future
planning within the Assistance Agreement Recipient organization.

To address this Assistance Agreement Recipient-level need, optional
questions have been developed and pre-approved by OMB. The optional
questions remain at a fairly general level in order to accommodate the
needs of a wide variety of programs. However, they are more specific and
detailed than the required questions. 

Optional questions are provided as “Menu” items that the Assistance
Agreement Recipient may select and add to the list of required
questions. Some optional items are quite similar to each other.
Different versions are included because the nuances in wording are
expected to accommodate Assistance Agreement Recipients with different
program focuses. 

Assistance Agreement Recipients are permitted to add to the
questionnaire any optional questions that they believe will provide
useful data. However, only optional questions listed on the “Menu”
of choices are permitted. Assistance Agreement Recipients are not
permitted to  add questions unless they are on the “Menu.”
Assistance Agreement Recipients are not required to include any optional
questions.

In addition, Assistance Agreement Recipients should be mindful of the
time and effort that may be required of respondents to complete the
questionnaire. Assistance Agreement Recipients should aim to keep
respondents’ burden as low as reasonably practicable by including only
those optional questions that will result in data that will be used in a
meaningful way.

Step 4: Include Program Office Questions

Some Program Offices may wish to compare outcomes across Assistance
Agreement Recipients with respect to a question that is available on the
“Menu” of choices, but that is not required of all Assistance
Agreement Recipients. In order to allow consistent comparison, Program
Offices may ask that Assistance Agreement Recipients include specific
optional questions. EPA Project Officers will notify their Assistance
Agreement Recipients if inclusion of such questions is requested. 

Step 5: Tailor Questions and Response Options 

As a general rule, both required and optional questions selected from
the “Menu” of choices must be included on the questionnaire as
written. OMB approval has been obtained only for the questions as
listed; new or revised questions would require separate OMB approval. In
limited circumstances, however, Assistance Agreement Recipients may
tailor certain elements of a pre-approved question, or certain response
options, to better coincide with their particular activities. 

Tailoring Questions

In a few cases Assistance Agreement Recipients are permitted to tailor
an optional question’s wording by inserting a phrase appropriate to
the Assistance Agreement Recipient’s particular situation, as
illustrated in   HYPERLINK  \l "Exhibit3BracketedPhrase"  Exhibit 3 .
Assistance Agreement Recipients are permitted to insert whatever word or
phrase best describes their activity.

Exhibit 3:

Tailoring Questions Through Completion of Bracketed Phrases

Question with [bracketed phrase]:	Question as tailored by Assistance
Agreement Recipient:

Because of [the activity] I took actions that I would not have taken
otherwise.

	Because of the training I took actions that I would not have taken
otherwise.

Because of the brochure I took actions that I would not have taken
otherwise.

Note that only the bracketed phrase has been altered. The rest of the
sentence remained unchanged. 

Tailoring Response Options

For some response options a series of underlined words is provided, and
the Assistance Agreement Recipient must delete those words that are not
relevant.   HYPERLINK  \l "Exhibit4DeletionofTerms"  Exhibit 4 
illustrates how an Assistance Agreement Recipient may retain a single
word or several words that apply to a project. 

Exhibit 4:

Tailoring Questions Through Deletion of Unnecessary Terms

Question with word selections/choices to make	Question as tailored by
Assistance Agreement Recipient:

…Follow recommended procedures for preventing contamination/ hazards/
emissions/ pollution	…Follow recommended procedures for preventing
pollution.

…Follow recommended procedures for preventing contamination and
hazards.

Note that no new words have been added (except for the conjunction
“and”).  The Assistance Agreement Recipient only deletes terms that
are not applicable and ensures that the sentence reads grammatically. 

Section 3: Required And Optional Questions For Post-Training
Questionnaires



This section presents the pre-approved questions that Assistance
Agreement Recipients can use immediately after a training event. It
discusses:

Required questions for Post-Training Questionnaires

Optional questions for Post-Training Questionnaires

All required questions, as well as many optional questions, use a
5-point response scale. Response options include:

1 – Completely Disagree

2 – Generally Disagree

3 – Neither Agree Nor Disagree

4 – Generally Agree

5 – Completely Agree

NA – Not Applicable

Most of the optional questions are to be answered using the same
agree/disagree 1-5 rating scale as the required questions. However,
three other types of responses are used:

2-Part Answers (questions 52-55) require selection of a follow-up
response for cases in which the respondent is not completely satisfied
(e.g., if the level of detail was not excellent, was there too much or
too little?). 

General Comments (questions 56-59) require open-ended responses.

Anticipated Behavior questions require selection of appropriate response
options. Response options concerning anticipated and actual actions
apply to both training and outreach activities, and are discussed
separately in   HYPERLINK  \l "SectionResponseOptions"  Section 5 .

For instructions on preparing a questionnaire once questions have been
selected in accordance with this section, see   HYPERLINK  \l
"SectionPreparingQuestionnaires"  Section 6 . 

Required Questions for Post-Training Questionnaires

Assistance Agreement Recipients that wish to collect post-training data
are required to ask seven questions. Those seven questions are presented
in   HYPERLINK  \l "Exhibit5RequiredPostTrainingQs"  Exhibit 5 . A
format that can be copied and used for data collection is provided in 
HYPERLINK  \l "AttachmentD" Attachment D . 

Exhibit 5:

Required Post-Training Questions

Please indicate how strongly you agree or disagree with each statement
by circling a number between 1 and 5. 1 means you disagree completely. 5
means you agree completely. If the question is not applicable, select
NA.

	Completely Disagree	Generally Disagree	Neither Agree nor Disagree
Generally Agree	Completely Agree	Not Applicable

The training significantly increased my awareness of the problems/issues
addressed in the training.	1	2	3	4	5	NA

The training significantly increased my understanding of the material
covered in the training.	1	2	3	4	5	NA

The training addressed all of my major questions regarding the training
topic.	1	2	3	4	5	NA

The training significantly improved my ability to carry out my
activities (e.g., activities at my job, in my community).	1	2	3	4	5	NA

It is extremely important for me to implement what I learned in this
training.	1	2	3	4	5	NA

I will definitely use what I learned in the training within the next six
months.	1	2	3	4	5	NA

There are no major barriers that will prevent me from carrying out what
I learned in the training (e.g., institutional, resource, or other
constraints).	1	2	3	4	5	NA

Optional Questions For Post-Training Questionnaires

The “Menu” of optional questions presented in  HYPERLINK  \l
"Exhibit6" Exhibit 6  provides opportunities for Assistance Agreement
Recipients to gather additional information about their training
programs. Assistance Agreement Recipients may “mix and match,”
choosing any combination of optional questions from the “Menu” to
supplement the seven required questions. 

The following guidelines should inform selection of optional questions:

Select questions carefully. Some of the questions are quite similar to
each other, with only slight differences in wording. Assistance
Agreement Recipients should choose questions with the phrasing that best
matches the nature of their training programs. 

Ask only for data that will be used. Careful thought should be given to
how the information, once collected, will be used. Assistance Agreement
Recipients should ask only questions for which there is a clear need for
the data. 

Ask a maximum of 22 optional questions. The entire questionnaire should
not exceed 30 questions. 



Exhibit 6:

“Menu” of Optional Post-Training Questions

Part I. Changes in Awareness

The training significantly increased my awareness of the implications of
failing to implement what I learned in the training.

The training significantly increased my awareness of the benefits of
implementing what I learned in the training.

Part II. Changes in Understanding

The training provided me with highly effective ways to implement what I
learned (e.g., strategies, tools, best practices).

The training provided me with highly effective strategies to implement
what I learned.

The training provided me with highly effective procedures to implement
what I learned.

The training provided me with highly effective processes to implement
what I learned.

The training provided me with highly effective techniques to implement
what I learned.

The training provided me with highly effective tools to implement what I
learned.

The training provided me with highly effective best practices to
implement what I learned.

The training showed me how to get the resources I need to implement what
I learned (e.g., helpful reference documents, web sites, or contact
information of key personnel).

The training gave me the reference documents I need to implement what I
learned.

The training gave me the web sites I need to implement what I learned.

The training gave me the contact information for key personnel that I
need to implement what I learned.

Part III. Anticipated Changes in Behavior

If I use what I learned, I will see improved results.

The training significantly improved my ability to get better results.

The training gave me information that I could use to improve health in
my home, my community, or my workplace

The training gave me information that I could use to improve the
environment in or around my home, my community, or my workplace

The training gave me information that could significantly improve the
way I perform my activities (e.g., in my job, my home, or my community).


Implementing what I learned will lead to better results in my home, my
community, or my workplace.

Implementing what I learned will lead to improved compliance (e.g., with
the regulations, procedures or guidance).

Implementing what I learned will lead to improved efficiency.

Implementing what I learned will lead to improved effectiveness.

There are no major barriers that will delay me in carrying out what I
learned in this training. 

Part IV. Perceptions of Training

The information provided before the course was excellent

I had the prerequisite knowledge and skills for this course

The course objectives were clearly articulated.

I needed to learn the material covered in this course.

The training introduced information that is relevant to me.

I was in need of some or all of the information presented.

The course objectives were completely met.

The course content was completely up to date.

No important information was left out of this course. 

The instructional methods were excellent for helping me master this
subject.

The exercises/activities were excellent for helping me master this
subject.

The trainer was a highly engaging presenter.

The trainer was completely prepared

The trainer’s presentations were completely clear.

The written materials/handouts were excellent.

The visual aids/slides were excellent.

The tools, best practices, and/or case studies provided were excellent.

The course organization was excellent.

I was able to get all of my questions answered.

The tests used in the course gauged my understanding of the material
extremely effectively.

The number of participants was excellent.

The location of the training facility was excellent.

The overall quality of the training facility was excellent.

The classroom/auditorium had all of the tools needed for the training
(e.g., projector, whiteboards.).

The classroom/auditorium layout was appropriate for the course format
(e.g., enough room for group work).

The size of the room was appropriate for the number of participants.

The classroom/auditorium was free of distractions.

The course hours (e.g., start times, stop times, breaks) were excellent.

The amount of material covered was excellent.

If the amount of material covered was not excellent would you:

(	Reduce content covered in course?

or

(	Increase content covered in course?

The difficulty of the material covered was excellent.

If the difficulty of the material covered was not excellent would you:

(	Make the course less difficult? 

or 

(	Make the course more difficult?

The pace of the course was excellent.

If the pace of the course was not excellent would you:

(	Slow down the pace of the course? 

or 

(	Speed up the pace of the course?

The length of the course was excellent.

If the length of the course was not excellent would you:

(	Shorten the course? 

or 

(	Lengthen the course?

General Comments (open-ended responses)

What improvements would you recommend in this course?

What was least valuable about this course?

What was most valuable about this course?

Do you have any additional comments about the training? 

For any open ended question the following text may be added: If you need
more space, please continue on the back of the page.

Anticipated Behavior

Because of the training I plan to take the following actions that I
would not have taken otherwise (select all that apply). 

(Possible response options for this question are presented in  HYPERLINK
 \l "SectionResponseOptions" Section 5 ).

Section 4: Required And Optional Questions For Follow-Up Questionnaires



This section presents the pre-approved questions that Assistance
Agreement Recipients can use to follow up after a training or outreach
activity. It discusses:

Required questions for Follow-Up Questionnaires

Optional questions for Follow-Up Questionnaires

Follow-up questions can be used to:

Assess what actions a respondent has actually taken some period of time
(3-6 months) after a training program. 

Assess what actions a respondent plans to take during some period of
time (3-6 months) immediately after an outreach effort.

Assess what actions a respondent has actually taken some period of time
(3-6 months) after an outreach effort. 

Follow-up questions are presented in both past and future tenses. This
will allow questionnaires to be distributed either before or after
respondents have an opportunity to act on what they have learned. 

Prospective (Future Tense): Select future tense to assess outreach
activities at the time the outreach effort takes place, before the
respondent has had an opportunity to take action (e.g., survey of
attendees at a conference presentation)

Retrospective (Past Tense): Select past tense to assess either training
or outreach activities after the respondent has had an opportunity to
take action (e.g., 3-6 months after the training or outreach effort).

In general, follow-up questionnaires must be extremely brief in order to
elicit participation. For this reason, only one pre-approved follow-up
question is required. Several optional questions also are provided. 

Some of the questions are to be answered using a 5-point response scale.
Response options include:

1 – Completely Disagree

2 – Generally Disagree

3 – Neither Agree Nor Disagree

4 – Generally Agree

5 – Completely Agree

NA – Not Applicable

Two other types of responses also are used:

Multiple Selection questions (question 3) permit respondents to
elaborate on the reasons for their actions (or anticipated actions).

Anticipated Behavior questions (question 4) require selection of
appropriate response options, as described in   HYPERLINK  \l
"SectionResponseOptions"  Section 5 . 

For instructions on preparing a questionnaire once questions have been
selected in accordance with this section, see   HYPERLINK  \l
"SectionPreparingQuestionnaires"  Section 6 . 

Required Questions for Follow-Up Questionnaires

There is one question that all Assistance Agreement Recipients are
required to ask. That question, which can be asked in either a
prospective or retrospective context, can apply to either training or
outreach activities, and focuses on the actions that the respondent has
taken or anticipates taking because of the activity.    HYPERLINK  \l
"Exhibit7RequiredFollowupQs"  Exhibit 7  presents the question as
phrased prospectively and retrospectively.

Exhibit 7: 

Required Follow-up Questions

Please indicate how strongly you agree or disagree with each statement
by circling a number between 1 and 5. 1 means you disagree completely. 5
means you agree completely. If the question is not applicable, select
NA.

	Completely Disagree	Generally Disagree	Neither Agree nor Disagree
Generally Agree	Completely Agree	Not Applicable

Prospective	Because of [the activity] I plan to take actions that I
would not have taken otherwise.	1	2	3	4	5	NA

Retrospective	Because of [the activity] I took actions that I would not
have taken otherwise.	1	2	3	4	5	NA



The retrospective and prospective formats that can be copied and used
for data collection are provided in   HYPERLINK  \l "AttachmentD" 
Attachment D . 

  HYPERLINK  \l "FollowupProspective"  Follow-Up Questionnaire:
Prospective  

  HYPERLINK  \l "FollowupRetrospective"  Follow-Up Questionnaire:
Retrospective 

Optional Questions for Follow-Up Questionnaires

The “Menu” of optional questions presented in   HYPERLINK  \l
"Exhibit8OptionalFollowupQs"  Exhibit 8  provides opportunities for
Assistance Agreement Recipients to gather additional follow-up
information about their outreach and training programs. Assistance
Agreement Recipients may “mix and match,” choosing any combination
of optional questions from the “Menu”. The number of optional
questions is quite restricted, focusing only on gathering information
about the actions respondents have taken or intend to take. 

For outreach, it sometimes may be necessary to screen potential
respondents to ensure that they actually were involved in the activity.
Question 1 allows the Assistance Agreement Recipient to ask such a
screening question. 

For instructions on Preparing a Questionnaire, see   HYPERLINK  \l
"SectionPreparingQuestionnaires"  Section 6 . 

Exhibit 8: 

 “Menu” of Optional Follow-up Questions

1. Participation in Activity 

I participated in/received [the activity].

(	Yes (please respond to the remaining questions)

(	No or not sure (please skip the remaining questions)

2. Effect on Outcomes (uses 1-5 response scale)

Future Tense	I expect using what I learned from [the activity] to lead
to better outcomes.

Past Tense	Using what I learned from [the activity] led to better
outcomes.

3. Reasons for Action or Inaction (uses multiple choice response scale)

Future Tense	Think about what you learned from [the activity]. Check the
one item that best describes what you think will happen in the next 6
months: 

(	I will take the actions suggested, but would have done so even without
[the activity].

(	I will take the actions suggested at least in part because of [the
activity].

(	I would like to take the actions suggested by [the activity], but will
not be able to (e.g., lacking time, lacking resources).

(	I would like to take the actions suggested by [the activity], but I am
not well enough prepared.

I will take the actions suggested by [the activity] but not within the
next six months.

(	I am not interested in taking the actions suggested by [the activity].

(	I will not have an occasion to take the actions suggested by [the
activity].

Past Tense	Think about what you learned from [the activity]. Check the
one item that best describes your experiences since then: 

(	I took the actions suggested, but would have done so even without [the
activity].

(	I took the actions suggested at least in part because of [the
activity].

(	I would have liked to take the actions suggested by [the activity],
but could not (e.g., lacking time, lacking resources).

(	I would have liked to take the actions suggested by [the activity],
but I was not well enough prepared.

I plan to take the actions suggested by [the activity] but have not done
so yet.

(	I was not interested in taking the actions suggested by [the
activity].

(	I did not have an occasion to take the actions suggested by [the
activity].

4. Anticipated or Actual Behavior (Possible response options for this
question are presented in   HYPERLINK  \l "SectionResponseOptions" 
Section 5 )

Future Tense 

	Because of [the activity] I plan to take the following actions that I
would not have taken otherwise (select all that apply).

Past Tense 

	Because of [the activity] I took the following actions that I would not
have taken otherwise (select all that apply).

Section 5: Response Options For Anticipated Or Actual Behavior

Note: The response options discussed in this section apply to 

questions in both Section 3 and Section 4.

As indicated in the lists of optional questions provided in Sections 3
and 4, both post-training and follow-up questionnaires offer the
opprtunity to ask an optional question about the particular actions that
participants have taken, or expect to take, as a result of the training
or outreach activity. 

Question 60 for post-training questionnaires (Section 3)

Question 4 for follow-up questionnaires (Section 4)

The text of these questions is provided in   HYPERLINK  \l
"Exhibit9TextofBehaviorQs"  Exhibit 9  for easy reference. The response
options for these questions are the same, regardless of which question
is being asked. 

Exhibit 9: 

Text of Anticipated and Actual Behavior Questions

From questions 60 (post-training) and 4 (follow-up)

Future Tense 

Because of [the activity] I plan to take the following actions that I
would not have taken otherwise (select all that apply).

(	

(	

(	

Past Tense 

Because of [the activity] I took the following actions that I would not
have taken otherwise (select all that apply).

(	

(	

(	





Three levels of specificity are available for the response options for
these questions, as illustrated in   HYPERLINK  \l
"Exhibit10TypesfoResponseOptions"  Exhibit 10 . 

  HYPERLINK  \l "GeneralResponseOptions"  General.  These response
options are generic enough to fit virtually any activity. Assistance
Agreement Recipients can select this suite of response options without
doing any tailoring for an individual project.

  HYPERLINK  \l "DetailedResponseOptions"  Detailed.  These response
options are considerably more specific than the generic options.
Assistance Agreement Recipients would need to select only those that are
appropriate to their activities. However, they are still general enough
to be applicable to a broad range of programs. If no program-specific
response options have been approved, Assistance Agreement Recipients may
turn to this list to find the response options that best fit their
program activities.

  HYPERLINK  \l "ProgramSpecificResponsesOptions"  Program-Specific .
These options are provided by Program Offices and are specific to the
goals of the program. When the Program Office has provided an approved
list of response options, the Assistance Agreement Recipient will
normally want to select the applicable items from the program-specific
list. This option is not available for all programs.

Program Offices that wish to submit program-specific response options
may do so by contacting the Office of Grants and Debarment (OGD). Only
Program Offices are permitted to establish program-specific responses.
Assistance Agreement Recipients are not permitted to establish their own
response options.



Editing Responses

When multiple words in a response option are underlined and separated by
slashes (e.g., “contamination/ hazards/ emissions/ pollution”),
those terms that do not apply should be deleted. Connecting words such
as “and” may be added to create a grammatically correct sentence. 

Terms in brackets should be replaced by an appropriate term for the
activity (e.g., “Because of [the activity] …” might become
“Because of reading the pamphlet.” 



Exhibit 10:

Anticipated and Actual Actions: Types of Response Options

Because of [the activity] I took the following actions that I would not
have taken otherwise (select all that apply).

 

 

 

Other (specify) ____________________

I do not plan to take any actions as a result of [the activity].



The possible responses for this question must be selected from the
following:



  HYPERLINK  \l "GeneralResponseOptions"  General  

RESPONSES

These possible responses are the most generic and can apply to almost
any activity.

	  HYPERLINK  \l "DetailedResponseOptions"  Detailed  

RESPONSES

These possible responses are less generic and cover a wide range of
possible topics

  HYPERLINK  \l "Preventingreducinghazards"  Preventing/Reducing Hazards


  HYPERLINK  \l "EquipmentProductsServices"  Equipment, Products and
Services

   HYPERLINK  \l "SafetySecurity"  Safety and Security

   HYPERLINK  \l "ProgramManagement"  Program Management

   HYPERLINK  \l "ProgramAdministration"  Program Administration  

   HYPERLINK  \l "PermittingLicensingCertification"  Permitting,
Licensing, and Certification

   HYPERLINK  \l "CommunityCapacityBuilding"  Community Capacity
Building 

  HYPERLINK  \l "ReportingRecordkeeping"  Reporting and Recordkeeping

   HYPERLINK  \l "EducationOutreach"  Education and Outreach

 

	  HYPERLINK  \l "ProgramSpecificResponsesOptions"  Program-Specific  

RESPONSES

These possible responses are supplied by Program Offices and are
provided for the following topics:

  HYPERLINK  \l "Mold"  Mold 

 HYPERLINK  \l "SecondhandSomkePublic" Second-hand Smoke 

 HYPERLINK  \l "SecondhandSomkePublic" Public

  HYPERLINK  \l "SecondhandSomkeHealthCareProviders" Health Care
Provider

  HYPERLINK  \l "BuildingDesignConstruction" Building
Design/Construction

   HYPERLINK  \l "AsthmaPublic"  Asthma

   HYPERLINK  \l "AsthmaPublic"  Public

   HYPERLINK  \l "AsthmaHealthCareProvider"  Health Care Provider

  HYPERLINK  \l "SchoolIndoorAirQuality" School Indoor Air Quality

  HYPERLINK  \l "RadonPublic" Radon

  HYPERLINK  \l "RadonPublic" Public

   HYPERLINK  \l "RadonHomebuilders"  Homebuilder

  HYPERLINK  \l "RadonOfficials" Official 



General Response Options

These response options are generic enough to fit virtually any activity.
Assistance Agreement Recipients can select as many of the response
options as pertain to their training or outreach activities. 

General Response Options

Because of [the activity] I took/plan to take the following actions that
I would not have taken otherwise (select all that apply).

Follow/followed recommended procedures for ___________________. 

Preventing hazards 

Reducing hazards

Cleaning up hazards

Complying with EPA requirements

Following recommended procedures

Teaching others about this topic

Other (specify) ____________________

I did not/do not plan to take any actions as a result of [the activity].






General Response Options Example

The following response options might be selected to assess anticipated
actions based on a brochure aimed at helping homeowners prevent, reduce,
and clean up dust from lead-based paint. 

Note: Bold and underlined text is provided to highlight how the general
response options were tailored in the example. All text would be
presented in normal font in an actual questionnaire.

Because of reading the brochure I plan to take the following actions
that I would not have taken otherwise (select all that apply).

Follow recommended procedures for preventing hazards 

Follow recommended procedures for reducing hazards

Follow recommended procedures for cleaning up hazards

Other (specify) ____________________

I do not plan to take any actions as a result of reading the brochure. 

Detailed Response Options

Detailed response options are provided in this section to give
Assistance Agreement Recipients flexibility in obtaining information
about how their training and outreach efforts affect individual
behavior. Detailed responses are available for the following topics:

Preventing/Reducing Hazards

   HYPERLINK  \l "EquipmentProductsServices"  Equipment, Products and
Services

   HYPERLINK  \l "SafetySecurity"  Safety and Security

   HYPERLINK  \l "ProgramManagement"  Program Management

   HYPERLINK  \l "ProgramAdministration"  Program Administration  

   HYPERLINK  \l "PermittingLicensingCertification"  Permitting,
Licensing, and Certification

   HYPERLINK  \l "CommunityCapacityBuilding"  Community Capacity
Building 

  HYPERLINK  \l "ReportingRecordkeeping"  Reporting and Recordkeeping 

  HYPERLINK  \l "EducationOutreach"  Education and Outreach 

  HYPERLINK  \l "ProgramSpecificResponsesOptions"  Program-specific
response options  are available for activities associated with some
Program Offices.

Detailed Response Option Example

The following response options might be selected to assess actual
actions based on a training aimed at helping remodelers identify,
prevent, and clean up dust from lead-based paint. 

Note: Bold and underlined text is provided to highlight how the general
response options were tailored in the example. All text would be
presented in normal font in an actual questionnaire.

Because of attending the training I took the following actions that I
would not have taken otherwise (select all that apply).

(	Followed recommended procedures for finding out about contamination
around my workplace

(	Followed recommended procedures for preventing hazards 

(	Followed recommended procedures for cleaning up hazards

(	Other (specify) ____________________

(	I did not take any actions as a result of attending the training. 





Detailed Response Options

Because of [the activity] I took/plan to take the following actions that
I would not have taken otherwise (select all that apply).

  HYPERLINK  \l "DetailedResponseOptions"  Preventing/Reducing Hazards 

(	Follow/followed recommended procedures for ___________________. 

preventing contamination/ hazards/ emissions/ pollution

reducing contamination/ hazards/ emissions/ pollution

remediating contamination/ hazards/ emissions/ pollution 

cleaning up contamination/ hazards/ emissions/ pollution

controlling releases

obtaining insurance

avoiding exposure to hazards/ contamination/ pollutants

acting to minimize exposure to hazards/ contamination/ pollutants

finding out about hazards/ contamination/ pollutants around me/ my
family/ my workplace

making changes to my household/ workplace environment to reduce exposure
to hazards/ contamination/ pollutants

minimizing waste or pollution

using resources more effectively

  HYPERLINK  \l "DetailedResponseOptions"  Equipment, Products and
Services 

(	Follow/followed recommended procedures for ___________________. 

installing equipment

training workers to use equipment

operating equipment

inspecting equipment

maintaining equipment

repairing equipment

decommissioning equipment

making design decisions to control hazards 

selecting less damaging/ hazardous products

considering this information when making decisions related to my work

improving the way I provide client services related to this issue

considering this information when making decisions about or
recommendations to clients.

  HYPERLINK  \l "DetailedResponseOptions"  Safety and Security 

(	Follow/followed recommended procedures for ___________________. 

improving contingency planning 

improving emergency preparedness

improving emergency prevention

improving emergency response procedures

improving worker safety 

  HYPERLINK  \l "DetailedResponseOptions"  Program Management 

(	Follow/followed recommended procedures for ___________________. 

improving administrative policies and procedures 

improving financial policies and procedures

improving human resources policies and procedures

improving program policies and procedures

improving grant compliance

improving grant administration

designing effective programs or program components

improving strategic planning

improving program development procedures

improving implementation procedures

improving enforcement procedures

improving program management 

  HYPERLINK  \l "DetailedResponseOptions"  Program Administration  

(	Follow/followed recommended procedures for ___________________. 

improving testing 

improving inspections 

improving compliance monitoring 

improving notification 

improving compliance assistance

improving enforcement

adhering to building and design standards

ensuring compliance with EPA regulations

obtaining EPA approval	

reducing the number of violations 

following up on complaints

  HYPERLINK  \l "DetailedResponseOptions"  Permitting, Licensing, and
Certification 

(	Follow/followed recommended procedures for ___________________. 

obtaining a permit

obtaining a license

obtaining a certification

submitting an application

  HYPERLINK  \l "DetailedResponseOptions"  Community Capacity Building

 (	Follow/followed recommended procedures for ___________________. 

running the organization or department or program more effectively

achieving organization goals more effectively

managing resources/ personnel more effectively

obtaining resources/ personnel

understanding/ complying with regulations/ standards

interacting with governments

record-keeping and reporting

recruiting partners

developing partnerships

involving stakeholders

assembling or taking part in a group or team to address the problem

  HYPERLINK  \l "DetailedResponseOptions"  Reporting and Recordkeeping 

(	Follow/followed recommended procedures for ___________________. 

reporting 

record-keeping 

  HYPERLINK  \l "DetailedResponseOptions"  Education and Outreach 

(	Follow/followed recommended procedures for ___________________. 

educating or training others

training trainers

educating the public about this topic

educating my peers about this topic

conducting workplace training



Program-Specific Response Options

Some program offices have provided program-specific response options
that are tailored to the needs of their Assistance Agreement Recipients.
Program-specific responses are available for the following topics:

  HYPERLINK  \l "Mold"  Mold

  HYPERLINK  \l "SecondhandSomkePublic"  Second-hand Smoke

  HYPERLINK  \l "SecondhandSomkePublic" Public

  HYPERLINK  \l "SecondhandSomkeHealthCareProviders" Health Care
Provider

  HYPERLINK  \l "BuildingDesignConstruction" Building
Design/Construction

  HYPERLINK  \l "AsthmaPublic" Asthma

   HYPERLINK  \l "AsthmaPublic"  Public

   HYPERLINK  \l "AsthmaHealthCareProvider"  Health Care Provider

  HYPERLINK  \l "SchoolIndoorAirQuality" School Indoor Air Quality

  HYPERLINK  \l "RadonPublic" Radon

  HYPERLINK  \l "RadonPublic" Public

   HYPERLINK  \l "RadonHomebuilders"  Homebuilder

   HYPERLINK  \l "RadonOfficials"  Official

 

 If program-specific response options have not been provided by the
Program Office, the   HYPERLINK  \l "DetailedResponseOptions"  detailed
response options  above may be used.

Program-Specific Response Option Example

The following response options might be selected to assess actual
actions based on a presentation at an industry meeting aimed at
encouraging property managers to perform preventive maintenance to avoid
mold problems, and to educate their maintenance staff about required
actions.

Note: Bold and underlined text is provided to highlight how the general
response options were tailored in the example. All text would be
presented in normal font in an actual questionnaire.

Because of the presentation I took the following actions that I would
not have taken otherwise (select all that apply):

Followed recommended procedures for following EPA mold
cleanup/remediation guidance.

Followed recommended procedures for educating others about mold and
moisture control.

Other (specify) ____________________

I did not take any actions as a result of the presentation. 



Program-Specific Response Options

  HYPERLINK  \l "ProgramSpecificResponsesOptions"  Mold 

(	Follow/followed recommended procedures for ___________________. 

following EPA mold cleanup/remediation guidance.

hiring a professional who follows EPA mold cleanup/remediation guidance.

educating others about mold and moisture control.

  HYPERLINK  \l "ProgramSpecificResponsesOptions"  Secondhand Smoke
(Public)  

(	Follow/followed recommended procedures for ___________________. 

making my home and car smoke-free. 

making sure that when family and friends are in my home or car they do
not smoke around my children.

telling others about the dangers of secondhand smoke exposure to
children.

talking to my children’s doctor or my doctor about secondhand smoke
and how it affects my children.

  HYPERLINK  \l "ProgramSpecificResponsesOptions"  Secondhand Smoke
(Health Care Providers) 

(	Follow/followed recommended procedures for ___________________. 

incorporating evaluation of children’s potential secondhand smoke
exposure in medical histories and work-ups.

suggesting strategies to parents and/or caregivers for reducing
children’s exposure.

  HYPERLINK  \l "ProgramSpecificResponsesOptions"  Building
Design/Construction 

(	Follow/followed recommended procedures for ___________________. 

establishing Indoor Air Quality (IAQ) design goals for new building
projects to minimize pollutant exposure.

implementing building designs and strategies to control moisture in
buildings.

establishing and implementing an IAQ construction management plan. 

implementing a commissioning process for new buildings.

establishing and implementing operations and maintenance plans with
clear IAQ and moisture control targets.

  HYPERLINK  \l "ProgramSpecificResponsesOptions"  Asthma (Public) 

(	Follow/followed recommended procedures for ___________________. 

working with a health care provider to identify my asthma triggers.

working with a health care provider to develop an asthma action plan.

using an asthma action plan to manage asthma at home and work or school.

making my home and car smoke-free.

taking steps to control the triggers that make my asthma worse.

  HYPERLINK  \l "ProgramSpecificResponsesOptions"  Asthma (Health Care
Providers) 

(	Follow/followed recommended procedures for ___________________. 

helping patients find ways to identify their personal triggers (either
in-house or through referrals).

developing asthma action plans together with my patients.

helping patients identify community resources to help them manage their
asthma.

including education about asthma triggers, including secondhand smoke,
at every asthma visit.

  HYPERLINK  \l "ProgramSpecificResponsesOptions"  School Indoor Air
Quality  

(	Follow/followed recommended procedures for ___________________. 

identifying an IAQ coordinator to lead and track IAQ activities.

engaging senior management, such as the superintendent, on the
importance of comprehensive IAQ management in schools.

developing a systematic, sustainable process for identifying, resolving
and preventing IAQ issues in my school(s). 

developing prioritized list(s) of the short-term and long-term actions
to be taken to resolve or prevent IAQ problems.

communicating goals, policies, planned actions, timelines, findings and
next steps to interested parties such as system administrators,
teachers, staff, and parents.

regularly evaluating the effectiveness of my school’s indoor air
quality (IAQ) management program.

  HYPERLINK  \l "ProgramSpecificResponsesOptions"  Radon (Public) 

(	Follow/followed recommended procedures for ___________________. 

having my home tested for radon.

mitigating my home if I learn that my home test result is above the EPA
radon action level of 4 pCi/L.

mitigating my home if I learn that my home test result is between 2
pCi/L and EPA’s action level of 4 pCi/L.

  HYPERLINK  \l "ProgramSpecificResponsesOptions"  Radon (Homebuilders) 

(	Follow/followed recommended procedures for ___________________. 

including radon resistant features in my new homes in areas with high
radon potential, in my role as a homebuilder.

including radon resistant features in all my new homes, in my role as a
homebuilder.

  HYPERLINK  \l "ProgramSpecificResponsesOptions"  Radon (Officials) 

(	Follow/followed recommended procedures for ___________________. 

advocating adoption of radon-resistant features in my code, in my role
as a state/local building code official.

advocating for radon risk reduction, in my role as a state/local
building code official.





Section 6: Preparing Questionnaires

Required Questions

If the Assistance Agreement Recipient wishes to ask only the required
questions, little or no preparation is required. 

  HYPERLINK  \l "TrainingFormat"  Training Questionnaire Format  can be
printed and distributed as-is; no modifications are required.

  HYPERLINK  \l "FollowupProspective"  Follow-Up Questionnaire Format
(Prospective ) can be used as-is after replacing the [bracketed text] to
state what activity is being evaluated.

  HYPERLINK  \l "FollowupRetrospective"  Follow-Up Questionnaire Format
(Retrospective)  can be used as-is after replacing the [bracketed text]
to state what activity is being evaluated.

The formats are provided in Attachment D. 

CAUTION!

Be sure to:

Use all required questions. 

Keep the wording of all required questions exactly as it is presented
except for substituting appropriate words for terms in [brackets]. 

Optional Questions

To add optional questions to the required questions in the questionnaire
format, take the following steps:

Copy and paste the required questionnaire format from   HYPERLINK  \l
"AttachmentD"  Attachment D  into a new document. 

From the “Menu” of optional questions provided in   HYPERLINK  \l
"SectionPostTraining"  Section 3  or   HYPERLINK  \l "SectionFollowUp" 
Section 4 , decide on the questions to add. 

Choose only questions from the “Menu” of optional questions. No
other questions may be added to the questionnaire.

Include any question that the EPA Project Officer indicates should be
added.

Add optional questions only if you know what you will do with the
information. 

Ask no more than 30 questions all together, including the required
questions already included on the questionnaire format. 

Add the optional questions to the document created in Step 1. How you do
this depends on the kind of question being added. 

Rating Scale. For each optional question to be answered on a scale of 1
to 5, add a row to the table. Copy and paste the question and the
response options (1-5 and NA) into the appropriate cells. 

Two Parts. For a question with two parts, include both parts of the
question in the same cell in the table. 

Multiple Responses. For questions that will be answered by selecting
from a list of choices, copy and paste the question below the existing
table. Then insert the appropriate list of response options.

CAUTION!

Be sure to:

Keep the wording of all optional questions exactly as-is except for
substituting appropriate words for terms in [brackets] or deletion of
underlined terms that do not apply to the activity.

Add only questions that are on the approved “Menu” of optional
questions.





  HYPERLINK  \l "Exhibit11AddingQuestions"  Exhibit 11  uses the
Training Questionnaire Format as an example to illustrate the tailoring
process for adding a line, or for adding a question with two parts.  
HYPERLINK  \l "Exhibit12TailoringQuestionnaires"  Exhibit 12 
illustrates the tailoring process for adding a question with multiple
responses for follow-up questionnaires. An in-depth discussion of
selecting response options was provided in   HYPERLINK  \l
"SectionResponseOptions"  Section 5 . 

Note that the questionnaire formats in   HYPERLINK  \l "AttachmentD" 
Attachment D  include an OMB burden statement, control number, and
expiration date. These items must appear on any OMB-approved
questionnaire. 

Exhibit 11:

Please indicate how strongly you agree or disagree with each statement
by circling a number between 1 and 5. 1 means you disagree completely. 5
means you agree completely. If the question is not applicable, select
NA.

The trainer was a highly engaging presenter.	1	2	3	4	5	NA

The amount of material covered was excellent

If the amount of material covered was not excellent, would you: 

(	Reduce content covered in course?

or



Exhibit 12:

Because of the training I plan to take the following actions that I
would not have taken otherwise (select all that apply).

Follow recommended procedures for preventing hazards.

Follow recommended procedures for reducing hazards.

Follow recommended procedures for cleaning up hazards.

Other (specify) ____________________.

I do not plan to take any actions as a result of the training.





Section 7: Questionnaire Administration And Assistance Agreement
Recipient Reporting

This section describes the processes that are to be followed once the
questionnaire has been developed. These include developing and
implementing a plan for administering the questionnaire effectively,
obtaining OMB approval, and reporting the results. 

Questionnaire Administration

After developing the questionnaire as described in Sections 1-6, the
Assistance Agreement Recipient must obtain OMB approval and administer
the survey. This requires the following steps.

Develop Plan to Collect Information. The Assistance Agreement Recipient
must develop a plan to administer the questionnaire and collect
information in accordance with OMB requirements and best practices.
Suggested approaches are described in  HYPERLINK  \l "AttachmentA"
Attachment A .  

Obtain Required Reviews and Approval. Once developed, the questionnaire
and plan to collect information must be submitted to the EPA Project
Officer, and then to the EPA Gatekeeper for review. After ensuring that
the materials comply with OMB specifications the Gatekeeper, in
conjunction with Office of Environmental Information (OEI), will submit
the materials to OMB for approval, as described in  HYPERLINK  \l
"AttachmentB" Attachment B . Because the required and optional questions
are pre-approved, OMB approval can be expedited. 

Administer Questionnaire. Finally, once OMB approval has been obtained
the Assistance Agreement Recipient administers the questionnaire, taking
care to follow procedures to maximize response rate. 

Reporting 

Once an Assistance Agreement Recipient administers the questionnaire and
collects the responses, the data must be tabulated and reported.
Assistance Agreement Recipient Reporting Form.xls is an Excel file that
allows Assistance Agreement Recipients to report the results of their
survey efforts.  HYPERLINK  \l "AttachmentC" Attachment C  discusses the
use of that file.

Raw data for each response must be entered into the form. 

The responses for each question are automatically calculated as
percentages for ease of analysis. 

Typically, data will be reported immediately after the data are
collected, or as prescribed in the assistance agreement work plan or
terms and conditions. An Assistance Agreement Recipient that develops an
on-going data collection effort would develop a periodic reporting
schedule in consultation with their EPA Project Officer.

EPA Program Offices may aggregate data for all required questions in
order to obtain a picture of the participating programs’ overall
effectiveness with respect to the training, education, and outreach
activities being assessed by these instruments. 

Data from optional questions can be aggregated at the Program Office
level if all Assistance Agreement Recipients associated with that
program collect parallel data.

Assistance Agreement Recipients may report their results without using
an electronic spreadsheet if they prefer to tabulate the results
manually. Such recipients should work with their EPA Project Officers to
obtain hard-copy print-outs of the reporting form.  

Assistance Agreement Recipients can use the data to understand the
strengths and weaknesses of their programs. Depending on when in the
assistance agreement cycle the data are obtained, this can allow for
mid-course corrections, or for program improvements prior to applying
for assistance in the future. 

Attachment A: Questionnaire Administration Decisions





Selecting questions for a survey is only one step in the process of
developing and administering a questionnaire. In addition to deciding
what type of survey to conduct (post-training or follow-up; prospective
or retrospective) Assistance Agreement Recipients must decide how to
administer their questionnaires, how to select the participants who will
be asked to provide responses, and how to minimize bias in the responses
obtained.

This attachment provides some best practices on how to prepare an
approach for administering the questionnaires discussed in this
document.  Assistance Agreement recipients must describe their proposed
approach in a document called the “Part B Document, ” which is
submitted to the EPA Project Officer as part of the expedited request
package.  The Part B Document is described in Attachment B, along with
the other materials that must be submitted.  

Questionnaire Administration Does Not Have To Be Complicated!

This Attachment discusses a range of questionnaire administration
decisions and provides more information than most Assistance Agreement
Recipients need. Read it through to decide what you do and don’t need
to know. But keep in mind that questionnaire administration is normally
an extremely simple 2-step process:

Ensure that the survey is distributed to a representative group. Give
the survey to all participants, or to a random sample of participants.

Ensure that there is a high response rate. Collect the questionnaires,
making sure to follow up to get responses from as many participants as
possible.

This attachment provides details to help Assistance Agreement Recipients
implement these steps effectively. In particular, look for 

Best Practice Tips to help avoid common pitfalls. 

Planning and implementation suggestions to help you integrate
information collection smoothly into your training or outreach activity.


If the more advanced techniques (e.g., stratified random sampling,
calculation of required sample size) don’t apply to you, just move on
to the next topic. 





Planning the Survey Administration Process

This attachment discusses four key elements of planning the survey
administration process to ensure that the data collected are
representative of the population being surveyed.   HYPERLINK  \l
"ExhibitA1AdminIssues"  Exhibit A-1  summarizes the issues that must be
considered for each type of questionnaire. These issues are discussed in
more detail in the sections that follow. 

Exhibit A-1:

Questionnaire Administration Issues

	Post-Training Questionnaire	Follow-Up Questionnaire (training or
outreach)

  HYPERLINK  \l "SamplingFrame"  Sampling frame 	All training
participants 	All training participants 

All individuals exposed to the outreach activity

  HYPERLINK  \l "SampleSelection"  Sample Selection 	100% of trainees
for small groups

Random sample for larger groups	100% of trainees for small groups

Random sample for larger groups

  HYPERLINK  \l "DataCollectionMethod"  Data Collection Method 
Typically paper-and-pencil survey conducted at the end of the training
session	Any of a variety of data collection methods (e.g.,
paper-and-pencil, email, web, in-person, telephone)

  HYPERLINK  \l "FollowUp"  Follow-Up to Enhance Response Rate 

	Activities to ensure that trainees submit their questionnaires prior to
leaving the training site. 	Follow-up activities to remind people to
submit their materials from a remote location 

Capturing information through telephone interviews.

Sampling Frame

Selecting the participants to complete a survey begins with identifying
all of the people who could potentially provide the desired information.
This group – called the sampling frame – includes everyone who has
received training or been exposed to the activity.

For training programs, the sampling frame includes all individuals who
participated in a training program.

For an outreach program, the sampling frame includes all individuals who
were exposed to the outreach effort.

When deciding to administer a survey, the Assistance Agreement Recipient
must carefully consider how all individuals who might be asked to
complete the survey will be identified. This should be done prior to the
activity, in order to allow the Assistance Agreement Recipient to
initiate a data collection method (e.g., sign-in sheets that include
e-mail addresses; verification of contact information at training
registration).

Best Practice Tip: Sampling Frame

Particularly for follow-up surveys, it is essential to have complete and
accurate contact information. Collection of contact information should
be built into the training or outreach effort to ensure that complete
information is available when needed.

Sample Selection

Sometimes Assistance Agreement Recipients will seek responses from
everyone who was touched by an outreach or training activity. Other
times, particularly when a large number of participants are involved, it
will be more effective to gather information from a sample of those
individuals affected. 

If the number of program participants is relatively small, collect data
from everyone. (E.g., in a training program for 30 people, collect
questionnaires from each participant.) In this case, no sample selection
is required. 

If the number of program participants is larger, select a sample from
which to gather data.

In most cases, the sample will be selected randomly.

In some situations, Assistance Agreement Recipients may wish to stratify
their sample (that is, intentionally collect information from certain
subgroups). For example, if a training audience mixes homeowners and
contractors, the Assistance Agreement Recipient might like to know
whether there are differences in effectiveness for these two groups. 

Any design other than a purely random sample of all participants would
need to be carefully thought through and documented.

The number of participants to include in the sample depends on the total
number of people affected by the activity and the kind of analysis that
will be done.   HYPERLINK  \l "ExhibitA2SampleSize"  Exhibit A-2 
provides guidelines for the number of completed questionnaires that must
be returned in order to be 95% confident that the estimated percentages
are accurate within +/- 5 percentage points. The guidelines assume that
the analysis will consider the number of respondents that score at or
above a pre-determined benchmark. For other types of analysis, the
required sample size would need to be carefully thought through and
documented.



Exhibit A-2:

Calculation of Required Sample Size

This exhibit illustrates how to determine the number of completed
surveys needed to analyze the percentage of responses reaching a given
benchmark score with a given level of confidence. First, find the row
containing the number of participants, M. Either use the maximum number
of participants shown in the second column, or use the formula shown in
the third column to get a lower, and more accurate sample size number
(based on linear interpolation). 

Example: Suppose a series of training sessions has 300 participants.
This occurs in the row for 101-500, so the maximum number to be surveyed
is 220. The more accurate answer is given by the formula ((M-100) (
0.350) + 80, which gives the number to be surveyed as ((300-100) (
0.350) + 80 = 150. 

Total Number of Participants

M

Maximum Number of Participants to be Surveyed

Number of Participants to be Surveyed

1-10

10

M (or all)

11-100

 80

 ((M-10) ( 0.778) + 10

101-500

 220

((M-100) ( 0.350) + 80

501-1,000

 290

 ((M-500) ( 0.140) + 220

1,001-2,000

330

((M-1000) ( 0.040) + 290

2,001-3,000

350

 ((M-2000) ( 0.020) + 330

3,001-4,000

360

 ((M-3000) ( 0.010) + 350

4,001 and above

370

370



The calculations in   HYPERLINK  \l "ExhibitA2SampleSize"  Exhibit A-2 
indicate the number of completed surveys required in various
circumstances. However, not all individuals who receive a questionnaire
will return it. To account for this, the Assistance Agreement Recipient
must determine what response rate is expected and increase the number of
surveys distributed accordingly. 

Example: If 100 responses are needed to achieve the desired level of
confidence and an 80% response rate is anticipated, the Assistance
Agreement Recipient would need to distribute 125 surveys (125 x .80 =
100).

 

Best Practice Tip: Sample Selection

Never select respondents based on convenience. Collecting surveys from
the first group of people to leave the room, the group of people sitting
in the front row, or the group of people for whom mailing addresses are
readily available all risk biasing the results. Always select
respondents randomly.

Always remember that despite rigorous follow-up efforts, some people
will not return their questionnaires. The sample selected must be
sufficiently large to ensure that even with non-respondents, the desired
number of useable responses will be obtained. 





Data Collection Method

For a survey administered immediately after a training program, the data
collection method is often simple: pass out hard copies of the
questions, give participants time to fill them out, and collect the
completed surveys. For surveys that involve following up after a
training event, or that involve contacting people who have been touched
by an outreach effort, however, other methods may be required.
Assistance Agreement Recipients are encouraged to think creatively about
effective ways to collect information from program participants. 

Depending on the activity and the type of questionnaire, Assistance
Agreement Recipients might consider methods such as:

Handing in hard copy at the event

Faxing hard copy

Mailing hard copy 

E-mailing electronic copy

Completing a web-based survey

Conducting a telephone survey

Conducting an interviewer-administered in-person survey 

Assistance Agreement Recipients should consider the best ways to collect
information before beginning their education or outreach efforts. This
will allow structures to be put in place, if needed, to ensure that data
can be collected as anticipated. (E.g., e-mail addresses may be needed
to allow cost-effective follow-up for a survey form sent out 3 months
after a training).

Best Practice Tip: Data Collection

Whenever interviewers are used (whether in person or on the telephone)
careful training is essential in order to avoid interviewer-induced
bias.

Questions must always be asked in exactly the same way.

The interviewer must not indicate satisfaction or dissatisfaction with
responses.

The interviewers themselves must be ‘neutral people’ (e.g., do not
have the person who conducted the training conduct a follow-up
interview).



Follow-Up to Enhance Response Rate

To make the results of a survey trustworthy, Assistance Agreement
Recipients must work diligently to ensure that a high percentage of
those to whom a survey is distributed actually complete and return the
survey. If only a small percentage return the survey, there is a high
chance that the results won’t be representative of the group as a
whole. This is because people who fail to respond to a survey can differ
systematically from those who do respond. In order to avoid this kind of
bias, it is essential to ensure a high response rate.

Assistance Agreement Recipients should work to attain the highest
possible response rate feasible for the type of data collection being
conducted.

For on-site questionnaire collection, staff should develop a system for
following up with participants. This might include:

Providing a specific time and place for completing the questionnaire

Reminding everyone who is selected as part of the sample to participate

Emphasizing the importance of participating and explain how the data
will be used (to improve future training programs)

For follow-up questionnaire collection, staff should:

Ensure that complete contact information is available for all
participants

Notify participants that a questionnaire may be sent to them

Emphasize the importance of participating and explain how the data will
be used (to improve future training programs)

Assistance Agreement Recipients are expected to follow well-established
techniques for enhancing response rates for follow-up questionnaires, as
laid out in   HYPERLINK  \l "ExhibitA3ResponseRate"  Exhibit A-3 . 



Exhibit A-3:

Follow-Up Method to Enhance Response Rate

 (May be done by e-mail or by mail)

1. Distribute pre-notification letters. Send a letter indicating that a
survey will be arriving, explaining the purpose of the survey, and
urging participation. 

2. Distribute the surveys. One week later, send the actual survey with a
cover letter again explaining the purpose and urging participation. 

3. Distribute the first reminder. Two weeks after the survey is sent,
send a reminder note or postcard to those individuals who have not
responded. 

4. Distribute the surveys (second mailing). Two weeks after the first
reminder, send the survey a second time to those who have not responded.


5. Distribute the second post card reminder. Two weeks after the second
survey is sent, send a reminder note or postcard to those individuals
who still have not responded.

NOTE: Time frames are approximate and may be shortened for electronic
communications.

Creative approaches to enhancing response rate also may be used, as long
as participants are not coerced into participating, and the approach
does not affect participants’ responses. For example, Assistance
Agreement Recipients could:

Recognize individuals who complete a questionnaire with a certificate

Provide a modest 'reward' (e.g., pens, refrigerator magnets, or T-shirts
with the organization's logo are provided upon receipt of a completed
questionnaire)

Offer a raffle-style prize (e.g., a donated gift is offered for raffle,
and a raffle ticket is provided upon receipt of a completed
questionnaire)

Assuring respondents that their privacy will be protected is another key
to enhancing response rates. In some cases responses can be truly
anonymous. In other situations anonymity is impractical (e.g., when
responses are submitted via e-mail). In these cases, Assistance
Agreement Recipients can assure respondents that all responses will be
aggregated to make it unlikely anyone would be able to identify any
particular respondent. 

 

Best Practice Tip: Follow-Up

Information from a relatively small number of people can provide very
good information about the effectiveness of a training or outreach
activity if the sample is randomly selected, and if a high percentage of
those selected randomly respond to the survey. Always set aside
resources (e.g., staff time, mailing costs) to ensure that rigorous
follow-up efforts are made.



Developing a Plan

Once the Assistance Agreement Recipient has thought through all of these
issues, a plan must be laid out systematically, and included as part of
the package submitted to the EPA Project Officer for review, and passed
on to OMB for approval.   HYPERLINK  \l "AttachmentB"  Attachment B 
describes how to prepare a statement about the decisions made in each of
these four areas.

Attachment B : Guidance For OMB Clearance Process



Under the 1995 Paperwork Reduction Act (PRA), the U.S. Office of
Management and Budget (OMB) must review and approve all collections of
information that are conducted or sponsored by a Federal agency. In
general, a “collection of information” includes any request for ten
or more members of the public to submit information or retain records.
The PRA requires a Federal agency to submit an Information Collection
Request (ICR) to OMB for approval before the collection can be carried
out. The ICR explains and justifies the proposed collection and
estimates the burden to respondents and government. 

On XXXX, 2008, OMB approved a generic ICR for EPA Assistance Agreement
programs involving training, education, and outreach activities. This
approval allows EPA to modify the questionnaires in this document within
certain parameters and re-submit them and supporting material for
expedited OMB approval. Hence, before a questionnaire can be distributed
to the public, EPA must obtain OMB approval.

  HYPERLINK  \l "ExhibitB1ExpeditedProcess"  Exhibit B-1  lays out the
basic steps in the expedited OMB approval process. It shows that it is
the Assistance Agreement Recipient’s responsibility to provide a draft
clearance request package to EPA for its review. EPA will review these
materials for compliance and completeness and use them as the basis for
its official request to OMB. OMB will take up to [ ] working days to
respond to the request. Its response may be an approval or a request for
revisions or additional information. EPA may need to contact the
Assistance Agreement Recipient for assistance in responding to OMB’s
request. After OMB approval is obtained, EPA will notify the Assistance
Agreement Recipient that the proposed information collection can
proceed. Assistance Agreement Recipients must not distribute the
questionnaire until EPA provides this go-ahead.

An Assistance Agreement Recipient should submit the following
information to the appropriate EPA Project Officer unless otherwise
specified:

A cover letter or memorandum addressed to the appropriate EPA Project
Officer that includes the following information:

A brief overview of the Assistance Agreement (e.g., grant or cooperative
agreement name, objectives, general activities to be performed,
duration);

A description of the particular training and/or outreach activit(ies)
that will be evaluated (e.g., content, expected outcomes, method of
delivery of information, number/size of events);

Identification of the type of questionnaires to be used; and

A brief overview of the survey approach (e.g., will you conduct a census
or sample? Will you send questionnaires by fax, postal mail, etc.? What
is the overall timeframe for your collection?). 

Note: The Part B Document, described below, may address some of this
information. Insofar as this is the case, the cover letter can reference
the relevant sections of the Part B Document to avoid duplication.

A “Part B Document” that provides a fuller description of the
proposed approach for using the questionnaire, including statistical
methods, if any. If two or more different questionnaires will be used
and the survey approaches for the two differ substantively, separate
Part B Documents may be required for each; 

A “Burden Worksheet” that estimates the total burden to the
Assistance Agreement Recipient and other members of the public in using
the questionnaire; and 

The questionnaire(s) that the Assistance Agreement Recipient is
proposing to use, which has been prepared in accordance with the
instructions in this guidance.

Exhibit B-1:

 





The following sections provide guidance for preparing the Part B
Document and the Burden Worksheet. After this, an example of a cover
memorandum, Part B Document, and Burden Worksheet is presented for
clarification purposes.  Following this guidance, as well as the
guidance in Attachment A, will help to maximize the likelihood that OMB
will approve the proposed approach and not have questions or concerns,
which could delay approval.  Questions about these materials should be
addressed to the appropriate EPA program.

I. Guidance for Preparing Part B Document

Each clearance request should include a Part B Document. Note that there
is not a standardized “Part B form”; rather, each Assistance
Agreement Recipient needs to develop its own document (e.g., in
Microsoft Word or other software as requested by EPA) according to the
format and information needs laid out below. The length and complexity
of the Part B Document will depend on the proposed survey approach. If a
sampling approach will be used instead of a census, for example, more
information is generally required because of the statistical issues
involved. If particular information is not applicable, the Part B
Document only needs to indicate briefly why the section does not apply.
If a section cannot be written because the necessary information is not
readily available, this should be indicated briefly.

1. Survey Objectives, Key Variables, And Other Preliminaries

(a)  Survey Objectives

The Part B Document should describe the survey objectives. The
objectives should be stated in the form of issues or questions for
resolution. In general, survey objectives should be limited to a
reasonable number, based on the Assistance Agreement Recipient and
EPA’s information needs. If too many objectives are listed, it may be
wise to focus on broader objectives. Only objectives that can be
reliably accomplished by the survey effort should be listed. A best
practice is to keep the survey objectives simple (e.g., “Are
homeowners taking recommended steps to prevent mold growth in their
homes?”).

(b)  Key Variables

A key variable is a characteristic of the individuals being surveyed
that could determine how they respond to the survey questions. For
example, in a survey measuring public support for EPA’s Aging
Initiative (e.g., an initiative to reduce hazards to older citizens in
their communities), a key variable may be “the age of the individuals
being surveyed.” One might assume that older individuals, as a group,
will be more supportive of the initiative. The description should try to
give the reviewer a sense for how the variables might influence the
survey results. However, if the key variables are not known, this should
be explained briefly (e.g., “insufficient information exists on
population characteristics”). This section should briefly describe the
key variables if known.

(c)  Sampling Approach

Some Assistance Agreement Recipients may opt to survey a limited number
of individuals rather than all individuals that could possibly be
surveyed. Assistance Agreement Recipients opting to do so would conduct
a “sampling approach.” For example, if a brochure is being mailed to
1,000 homes, the Assistance Agreement Recipient may not want to survey
all of these homeowners about the brochure. Rather, it may decide to
survey only a subset of them because of time or resource constraints
under the assistance agreement.

If a sampling approach is proposed, this should be stated here. In
addition, a statistically valid sampling approach should be presented in
accordance with the procedures laid out below. 

If, on the other hand, the Assistance Agreement Recipient decides to
survey all 1,000 homes, this would be considered a “census.” In this
case, a sampling approach would not be performed. If a census is
proposed, it should be stated here.

(d)  Feasibility

The feasibility of the proposed survey effort should be described
briefly. This entails a brief discussion of likely problems and how they
will be addressed.

2. Survey Design

This section describes the mechanics of the survey design - from the
respondent pool is selected to what type of questions will be asked in
the survey. If a sampling approach is being proposed, all four sections
below (i.e., Sections 2.a-2.d) need to be addressed. If a census is
proposed, only Section 2.d needs to be addressed.

(a)  Target Population and Coverage

This section should briefly describe the overall group of individuals or
organizations that are the focus of the survey (i.e., the “target
population”). For example, if a brochure is being mailed to 1,000
homes and the Assistance Agreement Recipient would like to gather
feedback from this entire group, then the entire group is the target
population. On the other hand, if the Assistance Agreement Recipient
would like to gather feedback from homeowners only in specific a state,
then the homeowners in this state would be the target population.

This section also should briefly describe if any key subgroups within
the target population will not be surveyed. For example, if all 1,000
homes are the target population and a sample (i.e., subset) of them will
be surveyed, the Assistance Agreement Recipient needs to indicate if the
selected sample omits any key subgroups. If, for example, the sample
does not include homeowners in a certain region of the country (e.g.,
because another organization is already surveying them on a similar
topic and the Assistance Agreement Recipient does not want to duplicate
that survey), this should be explained.

(b)  Sample Design

The following issues should be addressed with regard to the sampling
approach:

Indicate the sampling frame. See   HYPERLINK  \l "AttachmentA" 
Attachment A  for additional information on sampling frame. 

Indicate the sample size, including how it was derived. See   HYPERLINK 
\l "AttachmentA"  Attachment A  for additional information on sample
sizes. 

Briefly describe the sampling method. The sampling method is the set of
rules or procedures for selecting the individuals, or “sample," for
the survey from the population of interest, or "sampling frame.” The
recommended approach is a random, or probability, sampling. If, on the
other hand, a nonrandom method is selected, the Assistance Agreement
Recipient should discuss why a random sample is not being proposed,
including an explanation why the proposed approach will yield results
that are representative of the population sampled and will meet the
objectives of the survey.

Briefly describe multi-stage sampling, if applicable. Multi-stage
sampling is a method for selecting a sample of respondents that employs
more than one sample frame and sampling procedure. For example, in
planning a survey on the effectiveness of a training that had been
delivered to the employees of 1,000 businesses, the Assistance Agreement
Recipient could conduct the sampling in two stages: 1) first, it could
randomly select a subset of, say, 400, businesses to survey; and then 2)
it could randomly select a subset of employees within these 400
businesses that would be asked to complete a survey. It is doubtful that
most Assistance Agreement Recipients will use multi-stage sampling for
the questionnaires in this guidance. If multi-stage sampling will not be
used, this should be indicated briefly. On the other hand, if the
Assistance Agreement Recipient’s survey involves multi-stage sampling,
it should describe the sample design at each sampling stage, including
sample size, stratification variables, and sampling method.

Briefly describe stratification variables, if applicable, including how
they were selected and how they will improve the survey estimates.
Stratification is the method used to segment a population into
homogeneous groups, i.e., groups that are expected to give similar
responses. For example, in a survey measuring public support for EPA’s
Aging Initiative, it could be expected that responses will be strongly
related to age of the respondent. In this case, an Assistance Agreement
Recipient’s stratification variable could be “age of the
respondent.” If so, the Assistance Agreement Recipient could divide
the population into relevant age groups and separately select random
samples from each group. The ultimate goal of stratification is to
reduce the sampling error or, looked at another way, to increase the
precision of estimates derived from sample data. It is doubtful that
most Assistance Agreement Recipients will use stratification for the
questionnaires in this guidance. If stratification will not be used,
this should be indicated briefly. 

(c)  Precision Requirements

This section should describe the precision targets and sampling error.
The term, “precision,” means the measure of how closely estimates
made from the sample data are expected to approximate the
characteristics of the population of interest. If the sample size has
been established using   HYPERLINK  \l "ExhibitA2SampleSize"  Exhibit
A-2  in Attachment A, this should be indicated. The equations in  
HYPERLINK  \l "ExhibitA2SampleSize"  Exhibit A-2  will give at least 95%
confidence that the estimated percentages will be accurate within +/- 5
percentage points. 

The Assistance Agreement Recipient needs to discuss the potential
sources of nonsampling error, the steps it will take to minimize its
impact on sample estimates, and what effect it may have on the value of
the survey as a decision-making tool. 

Errors of estimates occur whenever there is a difference between the
distribution of responses from the sample and the true distribution of
responses from the target population. Statisticians often define two
types of error, sampling error and nonsampling error. Sampling error is
due solely to random variation when the number in the sample is smaller
than the population. Sampling error would disappear if the sample size
equaled the population size. 

Nonsampling error is all other errors of estimates. The main types of
nonsampling error for surveys of human populations are:

Noncoverage error. Failure to represent the target population because
the sampling frame from which the samples are selected differs from the
target population. For example, if the survey will be sent by postal
mail, but a number of the addresses are obsolete (e.g., some of the
homeowners have moved to another home), this can lead to a nonresponse
error.

Nonresponse error. Failure to obtain responses from all members of the
sample. For example, this occurs if the survey questionnaire is not
delivered to some members of the selected sample or if some members who
received the survey refuse to answer one or more questions.

Measurement error. Failure to obtain the true responses either because
the surveyed person does not exactly know their own true response or
because their response was biased. For example, a respondent might
deliberately not give a true answer to a question; perhaps trying to
instead give the answer that he or she thinks is the desired one.

Coding errors. Failure to properly code or tabulate the answers. This
includes computer processing errors and reporting errors. 

Certain nonsampling errors lower the accuracy of the resulting estimates
and directly affect survey precision. Of particular concern may be
biases. For example, a particularly serious form of bias results from
nonresponse, because the potential exists that those who do not respond
are different - in the worst case, all similarly different - from those
who do. Thus, if the very fact that a sampled person did not answer a
question is related to their true, but unknown answers, then there will
be bias due to the nonresponse. For example, if people who did not
benefit from a training are less likely to respond than those who
benefited, then the benefit will be overestimated.

(d)  Description of Questionnaire

The questions included in the questionnaire should be described briefly,
including a general indication that they will achieve the survey
objectives in Section 1.a. In addition, the Assistance Agreement
Recipient should describe the steps taken to ensure the reliability and
usefulness of the design. 

3. Pretests And Pilot Tests

In this section, simply include the following boilerplate text:

“EPA conducted a pilot test of the generic questions and questionnaire
formats to identify and resolve potential problems in regard to
effectiveness, clarity, feasibility, and burden/usability. The pilot
test lasted about two months and all of the questionnaire formats were
tested, as well as the Assistance Agreement Recipient Reporting Form.
EPA solicited feedback from Assistance Agreement Recipients as well as
those individuals who completed a questionnaire (e.g., training
participants), and addressed their feedback in finalizing the forms. See
Section 7(g) in the supporting statement of ICR 2255.01, “Performance
Measurement Reporting for Training and Education/Outreach,” for
additional information on the pilot tests. In no case were identical
questions asked of more than nine persons.”

4. Collection Methods And Follow-up

This section should describe methods to be used for collecting data,
including any follow-up activities. 

(a)  Collection Methods

This section should identify which collection method(s) have been
selected and why. If an interview method has been selected, for example,
the document should briefly describe the interviewer training program,
any special qualifications the interviewers must have, and the interview
procedures they will follow. As another example, if the Post-Training
Questionnaire will be used in a classroom setting with the intent of
surveying all trainees (i.e., a census), then the document should
describe how full participation will be ensured (e.g., by collecting a
completed questionnaire from each participant before adjourning for the
day).

(b)  Survey Response And Follow-up

The Part B Document should state the target response rate and describe
how the actual response rate will be measured. See Appendix A for best
practices on increasing response rates and conducting follow-up. 

5. Analyzing And Reporting Survey Results

Proposed data entry procedures and quality control measures should be
described. This should include a description of the procedures to be
used for accurately counting the data and entering it on the data
reporting forms provided by EPA. If additional analysis will be
conducted, the anticipated procedures also should be described. 

Glossary of Terms

Census – the process of obtaining information about every member of a
population.

Confidence interval - an interval which has a known and controlled
probability (generally 95% or 99%) to contain the true value.

Coverage – the extent to which a survey includes respondents from the
target population.

Key variable - a characteristic of the individuals being surveyed that
could determine how they respond to the survey questions. 

Nonsampling error - a statistical error that causes differences between
the responses measured on the sample and the responses of the target
population that are not due to random variation. It can occur from
noncoverage or similar errors in the sampling frame, nonresponse,
measurement errors, coding errors, computer processing errors, reporting
errors, and other errors.

Random sampling - a method of selection of a sample in which each
individual in the target population has an equal chance of being
selected. Also called “simple random sampling” or “simple random
sampling without replacement” (assuming that the same person cannot be
selected more than once).

Sampling - the process of selecting and surveying a subset of
individuals from a target population.

Sampling error – the difference between an estimate derived from a
sample survey and the true value that would result if a census of the
whole target population was taken. This difference is entirely due to
random sampling variation.

Sampling frame – a list of all members in the population from which
the sample will be selected. Ideally this should be the same as the
target population.

Stratification – a method of dividing the population into subsets
(called strata) within each of which an independent sample is selected.

Stratification variable – the variable that forms the basis for
dividing the population into different strata.

Target population - the population to be reached through the survey.

II. Guidance for Preparing Burden Worksheet

An estimate of the total hour burden to respondents in connection with
the proposed information collection must be provided. The attached
worksheet is intended to help estimate burden. (Note: The worksheet is a
recommended format; however, another format may be used if it is more
convenient and/or appropriate). For this submission, the term
“respondents” includes the Assistance Agreement Recipient’s time
invested in developing the questionnaire and the Part B Document, as
well as any individuals that are asked to complete a questionnaire. It
does not include time spent by EPA or other Federal agency staff.

  HYPERLINK  \l "ExhibitB2"  Exhibit B-2  reflects the three primary
activities that respondents will perform for which burden must be
estimated:

An Assistance Agreement Recipient develops and submits the request
package (proposed questionnaire, Part B Document, etc.) to EPA for
review for compliance and completeness as part of the expedited OMB
clearance process.

Individuals complete the questionnaire distributed by an Assistance
Agreement Recipient subsequent to OMB approval. 

Assistance Agreement Recipient completes/submits the Assistance
Agreement Recipient Reporting Form.

For each activity in the table, the hour burden associated with the
specific collection that is being undertaken should be estimated as
follows: 

In Column B, estimate the total number of responses for each activity.
For example, under “Completion of Questionnaire Forms,” estimate the
total number of individuals who will be asked to complete each type of
questionnaire being used. If, for example, a Post-Training Questionnaire
will be distributed to 100 trainees at five separate training courses,
then “500” responses should be entered. As another example, if an
Assistance Agreement Recipient Reporting Form will be submitted to EPA
quarterly, then “4” responses would be listed for that activity in
the table. Note: A default estimate of 1 response has been included. If
this is not accurate, a revised estimate should be provided.

In Column C, estimate the total burden per response for each activity.
Default estimates are currently included (e.g., “0.08 hr” = 5
minutes and “0.17 hr” = 10 minutes). The default estimates are based
on feedback from Assistance Agreement Recipients during EPA’s pilot
tests. If they are reasonably accurate for this effort, they may be
used. If not, a revised estimate should be provided. 

In Column D, multiply the total number of responses by the burden per
response for each activity, in order to estimate the total hour burden
for each activity.

After completing Columns B through D as described above, sum the numbers
in Columns B and D in order to derive total the total number of
responses and the total hour burden. Enter each total into the shaded
cell in its column. EPA will submit these estimates to OMB so it is
essential that they be provided in this worksheet.

  HYPERLINK  \l "ExhibitB3Checklist"  Exhibit B-3  provides a checklist
that the EPA gatekeeper will use to determine whether the Assistance
Agreement Recipient’s materials are ready for submission to OMB.
Assistance Agreement Recipients can use this same checklist to ensure
that their materials are complete before submission to the Gatekeeper.

  HYPERLINK  \l "ExhibitB4Samples"  Exhibit B-4  provides three sample
documents to help Assistance Agreement Recipients prepare their
submissions. These include: 

Sample Cover Memorandum To EPA

Sample Part B Document 

Sample Burden Worksheet

Assistance Agreement Recipients are invited to use the language
contained in these documents in their submissions, altering it as
appropriate for their individual circumstances. 

Exhibit B-2:

Burden Calculation Worksheet

A	B	C	D

Activity	Total Number of Responses	Burden Hours per Response	Total Hour
Burden

1. Development of Proposed Questionnaire and Related Materials

Assistance Agreement Recipient prepares/submits Request Package to EPA	

1	

2 hrs	

2 hrs

2. Completion of Questionnaire Form(s)

Individuals complete Post-Training Questionnaire



0.17 hr

	Individuals complete Follow-up Questionnaire Prospective



0.08 hr

	Individuals complete Follow-up Questionnaire Retrospective



0.08 hr

	3. Completion of Assistance Agreement Recipient Reporting Form

Assistance Agreement Recipient prepares/submits Assistance Agreement
Recipient Reporting Form



1 hr

	Total



	

 

Exhibit B-3:

Checklist for Reviewing Clearance Request Prior to Submittal

Review of Clearance Request Package	  (

General

1	Does the Clearance Request include all of the elements described on
pages B-1 and B-2 of this attachment (e.g., cover letter, proposed
questionnaire, etc.)?  

	Cover Letter or Memorandum

2	Does the cover letter or memorandum include all of the elements
requested on page B-1 of this attachment?

	Part B Document

3	Is the Part B Document prepared in accordance with Section I of this
attachment (e.g., have all requested elements been addressed)?

	Burden Worksheet

4	Has the worksheet been completed in accordance with Section II of this
attachment?

	5	Has the grant recipient summed the number of responses and annual
burden in Columns B and D and presented the totals in the shaded cells
of the worksheet?

	6	Is the worksheet free of computational errors?

	Proposed Questionnaire

7	Does the questionnaire include only the questions taken from the
“Menu” of questions in this document (i.e., no other questions may
be added to the questionnaire)?

	8	Are the questions taken verbatim from the “Menu” except for
questions that may be tailored in accordance with the instructions in
this document?

	9	Does the questionnaire include all questions that are required of all
grantees?

	10	If the Project Officer asked for the inclusion of program-specific
questions, are they included?

	11	For post-training questionnaire: Are there not more than 30
questions on the questionnaire in total?

	12	Does the questionnaire include an OMB Control Number and burden
statement?

	

Exhibit B-4:

Sample OMB Clearance Package Documents

SAMPLE COVER MEMORANDUM TO EPA

SAMPLE PART B DOCUMENT 

SAMPLE BURDEN WORKSHEET SAMPLE COVER MEMORANDUM TO EPA

MEMORANDUM

DATE: 	March 3, 2008

SUBJECT: 	Request for Federal Approval to Use the EPA Form, “Follow-Up
Questionnaire – Prospective” (OMB Control Number 2030-0020) under
OSW Training and Outreach Grant

FROM:	Joe Smith, Professor of Environmental Sciences, Aspen Hill
University

TO:	Betty Jones, U.S. Environmental Protection Agency (EPA), Office of
Solid Waste

The purpose of this memorandum is to request Federal approval for the
Aspen Hill University to collect information under the above-referenced
grant using the EPA Form, “Follow-Up Questionnaire – Prospective.”
The Aspen Hill University was awarded a two-year grant by EPA’s Office
of Solid Waste (OSW) to provide training and outreach to small
businesses that generate hazardous waste. The objective of the grant is
to increase small business compliance under the Federal program by
raising their awareness of their regulatory obligations and
familiarizing them with best practices for compliance.

Under this grant, we intend to develop a DVD entitled, “Small Business
Compliance under RCRA Subtitle C Regulations.” The DVD will recommend
best practices for improving their regulatory compliance under the
Federal regulations (e.g., proper container inspection, notification
procedures). Aspen Hill University would like to use the Follow-Up
Questionnaire – Prospective, in order to evaluate the effectiveness of
the DVD. Specifically, we would like to learn if small businesses expect
to adopt the recommended compliance activities presented in the DVD.
Their feedback also will help us identify needed improvements to the
DVD. The questionnaire will lay out 15 recommended compliance activities
and ask respondents to identify which ones they expect to adopt, if any.


We will mail the DVD to approximately 500 small businesses in the U.S.
that we believe may be generating hazardous waste. We would like to
include the questionnaire in a random sample of the mail-outs. Refer to
the attached Part B Document for a discussion of our sampling approach.
We would like to gather responses and report the results to EPA.

Please do not hesitate to contact me with questions or comments.

Attachments:

Part B Document

Burden Worksheet

Proposed Questionnaire

SAMPLE PART B DOCUMENT

1. Survey Objectives, Key Variables, And Other Preliminaries

(a)  Survey Objectives

Aspen Hill University will distribute a DVD to approximately 500 small
businesses that will recommend compliance activities. We will use the
Follow-Up Questionnaire – Prospective in order to evaluate whether
small businesses that have watched the DVD expect to perform the
recommended activities. The questionnaire will lay out 15 compliance
activities and ask if any will be performed Because of what the
respondent learned from the DVD. Following are the key objectives the
survey is designed to answer:

How many small businesses expect to adopt the recommended compliance
activities because of the DVD?

Which of the compliance activities do small businesses expect to
undertake because of the DVD?

This information will help Aspen Hill University evaluate if our DVD is
having the desired effect of increasing awareness and compliance of
regulated entities under the hazardous waste program. This information
will help in evaluating if Aspen Hill University is achieving our
overall grant objective of increasing small business compliance.

(b)  Key Variables

The key variable is the size of the small business as measured by the
number of employees. It is standard practice for very small businesses
(e.g., those with few on-site employees) to use contractors to handle
all of their regulatory activities. It can be expected that small
businesses relying on contractors will be less interested in the DVD and
in responding to the survey.

(c)  Sampling Approach

A sampling approach is proposed because the sampling frame is too large
for a census. There are limited resources under this grant, and we have
determined that a sampling approach would provide adequate results.

(d)  Feasibility

There are three critical challenges/problems associated with our
proposed use of the Follow-Up Questionnaire:

A primary problem is the fact that respondents that do not intend to
undertake the activities may be less likely to respond. This will bias
the results. We intend to address this problem by including a cover
letter assuring respondents that we will not share individual
questionnaire results with EPA for compliance purposes. We intend to
maximize responses by using proven methods for follow-up. Refer to
Sections 2.c and 4 of this document for further information.

Another critical challenge is that the person filling out the
questionnaire may not have the requisite authority or knowledge to say
reliably whether or not the organization will adopt the recommended
activities. We recognize that there may be a hierarchy or approval chain
in businesses through which such decisions must be made. Our cover
letter will emphasize the importance of making a good faith effort to
reflect their organizations’ likely actions.

A third challenge is that a positive response (i.e., a check mark) for a
given activity may not necessarily indicate that the DVD has had the
desired effect. It is possible that a respondent has already adopted the
activity or would have done so regardless of the DVD. The questionnaire
asks respondents specifically if they will perform the activities
because of the DVD, but it is not possible to ensure that respondents
will always discriminate between actions taken because of the DVD and
those actions that would have been taken anyway.

2. Survey Design

(a)  Target Population and Coverage

The target population consists of the 500 small businesses that were
sent a DVD. For purposes of this survey, a “small business” is
defined as an entity that has notified EPA or authorized State as being
a small quantity generator under 40 CFR 262.34(d), i.e., a generator of
less than 100 kilograms of hazardous waste per month. 

(b)  Sample Design

The sampling frame consists of 500 small businesses that were randomly
selected from EPA’s online database, RCRAInfo and were sent a DVD. The
sample size will consist of 367 small businesses that will be selected
randomly from the sampling frame. We will use a simple randomization
tool in Microsoft Access to select 367 out of the 500 businesses. The
sample size for completed surveys was derived based on the formula:
((M-100) ( 0.350) + 80, where “M” is the number of businesses in the
sampling frame, 500. This formula applies for M between 101 and 500, and
gives the minimum sample size for completed surveys such that we can be
at least 95% confident that the percentage of positive responses to a
question is within 5 percentage points of the population percentage. The
formula was calculated using the hypergeometric probability distribution
for several selected population sizes and then applying linear
interpolation to obtain sample sizes for population sizes, “M,” in
between the selected values. Applying this formula gives a sample size
of 220 completed surveys. We also conservatively assumed a response rate
of 60%, so that the sample size for the total number of businesses to be
surveyed is 220/0.6 = 367 businesses. We are not proposing to use
stratification or multi-stage sampling.

(c)  Precision Requirements

The proposed sample size will provide at least 95% confidence that the
results for the percentages of positive responses are accurate within
+/- 5 percentage points. 

There is a potential for nonsampling error, which is defined a
statistical error that causes differences between the responses measured
on the sample and the responses of the target population that are not
due to random variation.. It can occur from noncoverage, nonresponse,
measurement error, coding errors, computer processing errors, errors in
the sampling frame, reporting errors, and other errors. Nonsampling
error could give a biased characterization of the effects of our DVD.
For example, some respondents may not want to respond to the
questionnaire if they do not intend to adopt its recommendations. We
will reduce nonresponses by enclosing pre-addressed, postage paid return
envelopes, using follow-up methods (e.g., multiple phone calls to
nonresponders), and conveying confidentiality assurances in a cover
letter that the results will not be used for compliance
monitoring/enforcement. We also will provide multiple options for
submitting responses. 

In addition, we will minimize human error by training our data entry
staff on the proper compilation and tabulation of results and closely
monitor them to detect and correct clerical errors.

(d)  Description of Questionnaire 

The questionnaire lays out 15 recommended compliance-related activities
and asks small businesses if they will be performed. The activities were
selected because they describe the types of activities recommended in
the DVD and are most important in achieving compliance. These questions
will enable us to quantify the number of small businesses expecting to
adopt the recommended compliance activities as a result of viewing the
DVD, as well as the particular compliance activities they expect to
undertake. We believe the questionnaire will provide a clear picture of
whether the DVD has a beneficial effect on small business compliance.

3. Pretests And Pilot Tests

EPA conducted a pilot test of the generic questions and questionnaire
formats to identify and resolve potential problems in regard to
effectiveness, clarity, feasibility, and burden/usability. The pilot
test lasted about two months and all of the questionnaire formats were
tested, as well as the Assistance Agreement Recipient Reporting Form.
EPA solicited feedback from Assistance Agreement Recipients as well as
those individuals who completed a questionnaire (e.g., training
participants), and addressed their feedback in finalizing the forms. See
Section 7(g) in the supporting statement of ICR 2255.01, “Performance
Measurement Reporting for Training and Education/Outreach,” for
additional information on the pilot tests. In no case were identical
questions asked of more than nine persons.

4. Collection Methods And Follow-up

(a)  Collection Methods

We propose to mail the Follow-up Questionnaire and a cover letter, to
the sample of small businesses that receive the DVD. The cover letter
will describe, among other things, the purpose of the questionnaire,
timeframe for their response, and options for submitting the completed
questionnaire, including fax, regular mail, and email. We also will
include our phone number and email address if assistance is needed.

(b)	Survey Response And Follow-up

We will establish a target response rate of 80% and a minimum response
rate of 60%. We will perform the follow-up activities, below, after the
DVD and questionnaire are mailed out, in order to increase response
rates:

Distribute a first reminder two weeks after they are sent. The reminder
will be sent to those who have not responded.

Two weeks after the first reminder, send the questionnaire a second time
to those who have not responded.

Two weeks after the second reminder, send a postcard or reminder note to
those who have not responded

5. Analyzing And Reporting Survey Results

We will take the following steps to compile, tabulate and report the
data to EPA:

When a questionnaire is received from a respondent, our data entry staff
will review it initially for completeness and quality. If there are any
problems (e.g., an empty form or unclear responses), they will contact
the respondent for resolution before processing the data.

Questionnaire data will be quantified and entered into the Assistance
Agreement Recipient Reporting Form. 

If we do not receive at least a 60% response rate, we will notify EPA in
the Assistance Agreement Recipient Reporting Form.

SAMPLE BURDEN WORKSHEET

A	B	C	D

Activity	Total Number of Responses	Burden Hours per Response	Total Hour
Burden

1. Development of Proposed Questionnaire and Related Materials

Assistance Agreement Recipient prepares/submits Request Package to EPA	

1	

2 hrs	

2 hrs

2. Completion of Questionnaire Form(s)

Individuals complete Post-Training Questionnaire	

0	

0.17 hr	

0

Individuals complete Follow-up Questionnaire Prospective	

367	

0.08 hr	

 29 hrs

Individuals complete Follow-up Questionnaire Retrospective	

0	

0.08 hr	

0

3. Completion of Assistance Agreement Recipient Reporting Form

Assistance Agreement Recipient prepares/submits Assistance Agreement
Recipient Reporting Form	

1	

1 hr	

1 hr

Total	369

32 hrs

 Attachment C: Reporting Forms



The reporting forms that Assistance Agreement Recipients complete for
submission to the EPA Project Officer are provided in an Excel
spreadsheet – Assistance Agreement Recipient Reporting Form.xls –
which is a companion document to these instructions. The spreadsheet
requires data input only. All calculations have been automated. 

The Excel file contains 6 worksheets. The first sheet gives instructions
for completing the others. As illustrated in   HYPERLINK  \l
"ExhibitC1RequiredWorksheets"  Exhibit C-1 , Assistance Agreement
Recipients will never need to complete more than three worksheets. 

Assistance Agreement Recipient Information. Assistance Agreement
Recipient information is required for all reports. 

Required Questions. Results for required questions are required for all
reports. Different reporting worksheets are used depending on whether
the Assistance Agreement Recipient was involved in a post-training or a
follow-up activity. 

Optional Questions. Results for optional questions are to be completed
only if the Assistance Agreement Recipient has chosen to ask optional
questions. 

Exhibit C-1:

Reporting Forms Required by Type of Activity

Excel Worksheet Name	Post-Training	Follow-Up

Instructions



Assistance Agreement Recipient Information	X	X

Required Questions –Training	X

	Optional Questions – Training	X (if applicable)

	Required Questions – Follow-Up

X

Optional Questions – Follow-Up

X (if applicable)



 Attachment D: Questionnaire Formats

For all of the questionnaire formats, the Assistance Agreement Recipient
should provide the name of the activity or training as well as the date
of delivery. This will help respondents focus on the particular activity
that is being investigated. 

Three formats are provided. Each contains the required question(s) for
that type of questionnaire. Assistance Agreement Recipients may add
optional questions from the “Menu” of choices if desired. 

CAUTION!

Keep the wording of all pre-approved questions exactly as it is
presented unless the instructions specifically authorize change through
[bracketed phrases] or underlined terms.

Use only questions from the approved list, without adding any new
questions.

Always include the question(s) required of all Assistance Agreement
Recipients.

Always include the question(s) required by your EPA Project Officer (if
any).



The questions may be asked using different media. For example, follow-up
questions might be sent on a postcard, asked in a telephone survey, or
provided electronically. The format that follows simply provides the
required text and response options. 

Note that each format includes a statement required by OMB concerning
the public reporting and record-keeping burden. This statement must be
included on each questionnaire, and the OMB control number and the
approval expiration date must be accurately completed. 

The three formats provided include the following.

The   HYPERLINK  \l "TrainingFormat"  Post-Training Questionnaire Format
 can be printed and distributed as-is. No modifications are required
except for completing the OMB control number and approval expiration
date.

Two versions of the Follow-Up Questionnaire Format are provided. Each
can be used as-is after replacing the [bracketed text] to state what
activity is being evaluated, and completing the OMB control number and
approval expiration date.

The   HYPERLINK  \l "FollowupProspective"  Follow-Up Questionnaire:
Prospective Format  uses future tense, and is appropriate for
prospective evaluations (asking respondents what they expect to do in
the future).

The   HYPERLINK  \l "FollowupRetrospective"  Follow-Up Questionnaire:
Retrospective Format  uses past tense, and is appropriate for
retrospective evaluations (asking respondents what actions they actually
have taken).

POST-TRAINING QUESTIONNAIRE

Name of Training: _________________________________ Training
Date:_______________

Please indicate how strongly you agree or disagree with each statement
by circling a number between 1 and 5. 1 means you disagree completely. 5
means you agree completely. If the question is not applicable, select
NA.

	Completely Disagree	Generally Disagree	Neither Agree nor Disagree
Generally Agree	Completely Agree	Not Applicable

The training significantly increased my awareness of the problems/issues
addressed in the training.	1	2	3	4	5	NA

The training significantly increased my understanding of the material
covered in the training.	1	2	3	4	5	NA

The training addressed all of my major questions regarding the training
topic.	1	2	3	4	5	NA

The training significantly improved my ability to carry out my
activities (e.g., activities at my job, in my community).	1	2	3	4	5	NA

It is extremely important for me to implement what I learned in this
training.	1	2	3	4	5	NA

I will definitely use what I learned in the training within the next six
months.	1	2	3	4	5	NA

There are no major barriers that will prevent me from carrying out what
I learned in the training (e.g., institutional, resource, or other
constraints).	1	2	3	4	5	NA



The public reporting and recordkeeping burden for this collection of
information is estimated to average 10 minutes per response. Send
comments on the Agency's need for this information, the accuracy of the
provided burden estimate, and any suggested methods for minimizing
respondent burden, including through the use of automated collection
techniques to the Director, Collection Strategies Division, U.S.
Environmental Protection Agency (2822T), 1200 Pennsylvania Ave., NW,
Washington, D.C. 20460. Include the OMB control number in any
correspondence. Do not send the completed form to this address.

FOLLOW-UP QUESTIONNAIRE: PROSPECTIVE

Name of Activity: ___________________________ Date of Activity:
_________________

Please indicate how strongly you agree or disagree with each statement
by circling a number between 1 and 5. 1 means you disagree completely. 5
means you agree completely. If the question is not applicable, select
NA.

	Completely Disagree	Generally Disagree	Neither Agree nor Disagree
Generally Agree	Completely Agree	Not Applicable

Because of [the activity] I expect to take actions that I would not have
taken otherwise.	1	2	3	4	5	NA



The public reporting and recordkeeping burden for this collection of
information is estimated to average 5 minutes per response. Send
comments on the Agency's need for this information, the accuracy of the
provided burden estimate, and any suggested methods for minimizing
respondent burden, including through the use of automated collection
techniques to the Director, Collection Strategies Division, U.S.
Environmental Protection Agency (2822T), 1200 Pennsylvania Ave., NW,
Washington, D.C. 20460. Include the OMB control number in any
correspondence. Do not send the completed form to this address.

FOLLOW-UP QUESTIONNAIRE: RETROSPECTIVE

Name of Activity: ___________________________ Date of Activity:
_________________

Please indicate how strongly you agree or disagree with each statement
by circling a number between 1 and 5. 1 means you disagree completely. 5
means you agree completely. If the question is not applicable, select
NA.

	Completely Disagree	Generally Disagree	Neither Agree nor Disagree
Generally Agree	Completely Agree	Not Applicable

Because of [the activity] I took actions that I would not have taken
otherwise.	1	2	3	4	5	NA



The public reporting and recordkeeping burden for this collection of
information is estimated to average 5 minutes per response. Send
comments on the Agency's need for this information, the accuracy of the
provided burden estimate, and any suggested methods for minimizing
respondent burden, including through the use of automated collection
techniques to the Director, Collection Strategies Division, U.S.
Environmental Protection Agency (2822T), 1200 Pennsylvania Ave., NW,
Washington, D.C. 20460. Include the OMB control number in any
correspondence. Do not send the completed form to this address.

 This table was derived as follows: First, assume the population has M
members of which K give a Yes response, and that a sample of m is
randomly selected without replacement. Use the hypergeometric
distribution to calculate the probability that the sample proportion of
Yes responses is within 0.05 of the true proportion, K/M. Second, for
each M, calculate the minimum sample size n such that this probability
is at least 0.95 for all possible sample sizes, m, greater than or equal
to n, and all possible values of K. This gives the first two columns in
the table, where M is the highest value shown in column 1, e.g., M=100
for the row “11-100” (use M = 5000 for the group “4,001 and
above”). Finally, use linear interpolation to obtain the formulae in
the third column.

 Statisticians often express the target "level of precision" as a
confidence interval estimate of one of the key variables. 

 PAGE   

Response Options: Program-Specific

  PAGE  iii 

  PAGE  35 

Optional Questions – Follow-Up

Response Options: Detailed

Response Options: Program-Specific

A-  PAGE  8 

B-  PAGE  18 

C-  PAGE  1 

D-  PAGE  1 

OMB Control No. 2030-XXXX

Approval Expires xx/xx/xx

EPA Form 6600-03

EPA Form 6600-04

EPA Form 6600-02

Official request submitted to OMB

OMB approves request

Package submitted to EPA

OMB conducts expedited review. It may provide comments to EPA Program
prior to approval

Required questions will appear here.

Tip: To navigate quickly to the response options relevant to a
particular type of activity, hold the control key and click on the
desired topic.

For optional questions that have two parts, include the additional
material within the same line in the table, like this. 

Add optional questions by adding lines to the table

Tip: To navigate quickly to the response options relevant to a
particular activity, hold the control key and click on the desired
topic.

“General” response options were selected.

The bracketed phrase “[the activity]” has been replaced.

Revise as needed

Report Results

Improve Program

Administer Survey

Submit for Project Officer and EPA Gatekeeper/OEI Review

Get OMB Approval

Develop Questionnaire

Develop Strategy

3-6 Month Follow-Up

Immediate Post-Outreach

Immediate Post-Training

EPA reviews submittal for compliance and completeness. EPA may provide
comments to Grant Recipient based on its review. It also may convey
comments from OMB resulting from the OMB review (see below).

Select Question Type

Grant Recipient prepares Proposed Questionnaire(s), Cover Letter, Bart B
Document and Burden Worksheet

Future tense was selected.

