SUPPORTING
STATEMENT
INFORMATION
COLLECTION
REQUEST
for
Public
Involvement
Activities
Feedback
Surveys
of
The
United
States
Environmental
Protection
Agency
January
2004
ii
Table
of
Contents
Title
Page
1.
Identification
of
the
Information
Collection
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
1
1(
a)
Title
of
the
Information
Collection
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
1
1(
b)
Short
Characterization/
Abstract
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
1
2.
Need
for
and
Use
of
the
Collection
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
2
2(
a)
Need/
Authority
for
the
Collection
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
2
2(
b)
Practical
Utility/
Users
of
the
Data
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
4
3.
Non­
duplication,
Consultations,
and
Other
Collection
Criteria
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
6
3(
a)
Non­
duplication
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
6
3(
b)
Public
Notice
Required
Prior
to
ICR
Submission
to
OMB
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
6
3(
c)
Consultations
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
6
3(
d)
Effects
of
Less
Frequent
Collection
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
7
3(
e)
General
Guidelines
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
7
3(
f)
Confidentiality
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
7
3(
g)
Sensitive
Questions
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
7
4.
The
Respondents
and
The
Information
Requested
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
7
4(
a)
Respondents/
SIC
Codes
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
7
4(
b)
Information
Requested
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
8
(
I)
Data
items,
including
record
keeping
requirements
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
8
(
II)
Respondent
Activities
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
8
5.
The
Information
Collected­­
Agency
Activities,
Collection
Methodology,
and
Information
Management
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
8
5(
a)
Agency
Activities
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
8
5(
b)
Collection
Methodology
and
Management
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
9
5(
c)
Small
Entity
Flexibility
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
9
5(
d)
Collection
Schedule
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
9
6.
Estimating
the
Burden
and
Cost
of
the
Collection
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
12
6(
a)
Estimating
Respondent
Burden
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
12
6(
b)
Estimating
Respondent
Costs
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
16
(
I)
Labor
Costs
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
16
(
II)
Capital
and
Operations
and
Maintenance
Costs
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
18
(
III)
Capital/
Start­
up
vs.
Operations
and
Maintenance
(
O&
M)
Costs
.
.
.
.
.
.
.
.
.
.
18
(
IV)
Annualizing
Capital
Costs
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
18
6(
c)
Estimating
Agency
Burden
and
Cost
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
18
6(
d)
Estimating
the
Respondent
Universe
and
Total
Burden
and
Costs
.
.
.
.
.
.
.
.
.
.
22
6(
e)
Bottom
Line
Burden
Hours
and
Cost
Tables
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
22
(
I)
Respondent
Tally
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
22
(
II)
The
Agency
Tally
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
22
(
III)
Variations
in
the
Annual
Bottom
Line
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
23
(
IV)
Reasons
for
Change
in
Burden
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
24
(
V)
Burden
Statement
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
24
iii
List
of
Tables
Table
5­
1
Planned
Use
of
Questionnaires
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
11
Table
6­
1
(
a)
Respondent
Burden
2004
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
13
Table
6­
1
(
b)
Respondent
Burden
2005
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
14
Table
6­
1
(
c)
Respondent
Burden
2006
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
15
Table
6­
2
Respondent
Universe,
Total
Burden
and
Costs
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
17
Table
6­
3
Agency
Burden/
Cost
for
CAG,
Public
Meeting,
Public
Hearing,
Listening
Session,
Small
Group
Discussion
and
FACA
Meeting
Effectiveness
Surveys
(
6
surveys)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
19
Table
6­
4
Agency
Burden/
Cost
for
CAG
and
FACA
Process
Effectiveness
Surveys
(
2
surveys)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
19
Table
6­
5
Agency
Burden/
Cost
for
Public
Meeting,
Public
Hearing
and
Listening
Session
Follow­
up
Surveys
(
3
surveys)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
20
Table
6­
6
Agency
Burden/
Cost
for
CAG,
FACA
and
Small
Group
Discussion
Staff/
Contractor
Assessments
and
CAG
and
FACA
Participant
Process
Assessments
(
5
surveys)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
20
Table
6­
7
Agency
Burden/
Cost
for
Negotiations
Feedback
Surveys
(
2
surveys)
.
.
.
.
.
.
.
.
.
21
Table
6­
8
Aggregate
Agency
Burden
and
Cost
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
23
Table
6­
9
Aggregate
Agency
and
Respondents
Burden/
Cost
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
24
Exhibits
­
EPA
Survey
Instruments
and
Associated
Materials
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
25
1.
Community
Advisory
Groups
­
User's
Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
29
a)
Background
Information
for
EPA
Staff
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
30
b)
Checklist
for
Administering
Questionnaires
and
Using
Results
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
33
c)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
CAG
Meeting
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
35
d)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
CAG
Meeting
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
37
e)
Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
CAG
Process
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
39
f)
Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
CAG
Process
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
41
2.
Public
Meetings
­
User's
Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
42
a)
Background
Information
for
EPA
Staff
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
43
b)
Checklist
for
Administering
Questionnaires
and
Using
Results
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
46
c)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Public
Meeting
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
48
d)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Public
Meeting
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
52
e)
Questionnaire
for
Feedback
on
the
Effectiveness
of
Public
Meeting
Follow­
Up
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
54
3.
Public
Hearings
­
User's
Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
56
a)
Background
Information
for
EPA
Staff
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
57
b)
Checklist
for
Administering
Questionnaires
and
Using
Results
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
60
c)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Public
Hearing
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
62
iv
d)
Questionnaire
for
Feedback
on
the
Effectiveness
of
Public
Hearing
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
65
e)
Questionnaire
for
Feedback
on
the
Effectiveness
of
Public
Hearing
Follow­
up
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
67
4.
Listening
Sessions
­
User's
Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
69
a)
Background
Information
for
EPA
Staff
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
70
b)
Checklist
for
Administering
Questionnaires
and
Using
Results
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
73
c)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Listening
Session
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
75
d)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Listening
Session
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
79
e)
Questionnaire
for
Effectiveness
of
Listening
Session
Follow­
Up
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
81
5.
Small
Discussion
Group
Session
­
User's
Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
83
a)
Background
Information
for
EPA
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
84
b)
Checklist
for
Administering
Questionnaires
and
Using
Results
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
87
c)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Small
Discussion
Group
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
89
d)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Small
Discussion
Group
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
92
e)
Questionnaire
for
Effectiveness
of
Small
Discussion
Group
Follow­
Up
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
94
6.
Stakeholder
Negotiation
­
User's
Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
96
a)
Background
Information
for
EPA
Staff
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
97
b)
Checklist
for
Administering
Questionnaires
and
Using
Results
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
101
c)
Questionnaire
for
Feedback
on
Public
Involvement
in
EPA's
Stakeholder
Negotiations
(
stakeholder
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
103
d)
Questionnaire
for
Feedback
on
Public
Involvement
in
EPA's
Stakeholder
Negotiations
(
public's
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
106
7.
Federal
Advisory
Committee
Act
Committees
­
User's
Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
108
a)
Background
Information
for
EPA
Staff
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
109
b)
Checklist
for
Administering
Questionnaires
and
Using
Results
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
112
c)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
FACA
Meeting
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
114
d)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
FACA
Meeting
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
116
e)
Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
FACA
Process
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
117
f)
Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
FACA
Process
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
119
g)
Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
FACA
Committee
(
DFO
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
122
1
U.
S.
ENVIRONMENTAL
PROTECTION
AGENCY
VOLUNTARY
FEEDBACK
SURVEYS
TO
SUPPORT
IMPROVEMENT
OF
EPA's
PUBLIC
INVOLVEMENT
ACTIVITIES
1.
Identification
of
the
Information
Collection
1(
a)
Title
of
the
Information
Collection:
Obtaining
Feedback
on
Public
Involvement
Activities
and
Processes
1(
b)
Short
Characterization/
Abstract
On
June
6,
2003,
EPA
issued
its
new
Public
Involvement
Policy.
Evaluation
is
one
of
the
key
elements
in
the
Agency's
"
Framework
for
Implementing
EPA's
Public
Involvement
Policy."
Critical
to
overall
evaluation
is
feedback
from
participants
involved
in
the
public
involvement
activities.
A
well
designed
evaluation
plan
will
make
it
possible
for
EPA
to
better
understand:
(
1)
if
the
Agency
is
taking
the
necessary
steps
to
gather
and
consider
public
input;
(
2)
the
quality
of
the
Agency's
public
involvement
processes;
(
3)
how
to
consistently
and
systematically
learn
and
improve,
and
(
4)
how
the
Agency
can
be
more
accountable
to
the
public.

The
Environmental
Protection
Agency
is
seeking
from
the
Office
of
Management
and
Budget
(
OMB)
a
generic
clearance
for
public
involvement
activities
surveys
for
a
period
of
three
years.
The
clearance
will
be
used
to
conduct
surveys
about
public
involvement
activities
and
processes.
The
surveys
will
help
to
identify
participants'
perceptions
of
their
treatment
during
the
activity,
as
well
as
the
quality
of
pre­
activity
information,
the
activity
and
follow­
up.
All
surveys
will
solicit
opinions
from
participants
in
EPA
public
involvement
activities
on
a
voluntary
basis,
and
will
not
involve
"
fact­
finding"
for
the
purposes
of
regulatory
development
or
enforcement.

By
seeking
a
generic
clearance
for
a
set
of
questions
and
questionnaires
for
each
type
of
activity,
EPA
will
have
the
flexibility
to
gather
the
views
of
participants
and
better
determine
the
extent
to
which
our
public
involvement
activities
meet
their
needs
or
need
to
be
improved.
EPA
will
not
deviate
from
the
approved
questions
and
will
report
annually
on
the
actual
use
of
the
series
of
surveys.
The
generic
clearance
will
enable
regional
and
program
offices'
staff
to
regularly
use
approved
questionnaires
and
apply
the
resulting
data
to
improve
the
way
the
Agency
conducts
its
public
involvement
processes.

EPA's
sponsoring
organizations
that
use
any
of
the
various
questionnaires
will
report
their
use
of
the
survey
instruments
to
the
public
involvement
staff
in
the
Office
of
Policy,
Economics
and
Innovation
within
the
Office
of
the
Administrator.
That
staff
has
developed
a
Lotus
Notes
spreadsheet
program
that
enables
the
survey
users
to
enter
data
from
the
returned
questionnaires
and
quickly
process
the
results,
report
the
findings
and
take
appropriate
action
to
improve
their
next
similar
sessions.

EPA
estimates
that
users
of
the
public
involvement
questionnaires
(
public
meetings,
public
hearings,
community
advisory
groups,
Federal
Advisory
Committee
Act
groups,
listening
2
sessions,
small
discussion
group
sessions,
stakeholder
negotiations
)
will
request
voluntary
overall.

2.
Need
for
and
Use
of
the
Collection
2a.
Need/
Authority
for
the
Collection
EPA's
Public
Involvement
Policy
states
that
the
Assistant,
Regional
and
Associate
Administrators
"
Provide
leadership
and
direction
for
their
managers
and
staff
by,"
among
other
responsibilities,
"
Evaluating
the
effectiveness
of
public
involvement
processes
and
taking
action
to
improve
these
processes."
To
fulfill
their
responsibilities,
the
Agency's
leadership
and
staff
need
a
framework
and
tools
to
evaluate
and
measure
how
well
EPA
is
implementing
the
policy's
seven
basic
steps
for
conducting
effective
public
involvement
in
both
regulatory
and
non­
regulatory
processes:
1.
Plan
and
budget
for
public
involvement
activities
2.
Identify
the
interested
and
affected
public
3.
Consider
providing
technical
or
financial
assistance
to
the
public
to
facilitate
involvement
4.
Provide
information
and
outreach
to
the
public
5.
Conduct
public
consultation
and
involvement
activities
6.
Review
and
use
input,
and
provide
feedback
to
the
public
7.
Evaluate
public
involvement
activities
This
ICR
will
be
the
cornerstone
of
a
coordinated
effort
to
track
and
improve
the
quality,
effectiveness
and
consistency
of
EPA's
public
involvement
activities
and
processes.
Though
individual
programs
and
projects
occasionally
have
conducted
evaluations
of
their
public
involvement
activities
directly
or
through
contractors,
EPA
has
not
encouraged
a
sustained
agency­
wide
effort
to
determine
the
extent
and
quality
of
public
involvement
processes
and
activities
and
to
improve
their
effectiveness
for
both
participants
and
EPA.
One
of
the
findings
of
a
2003
survey
of
staff
from
a
cross
section
of
programs
and
regions
from
39
projects
was
that
"
absent
a
conscientious
evaluation
effort,
project
staff
are
unlikely
to
learn
from
their
past
mistakes."

The
ICR
will
be
part
of
a
suite
of
tools
being
developed
by
public
involvement
staff
and
practitioners
across
the
Agency
that
staff
(
and
other
partners)
can
use
to
evaluate
and
measure
public
involvement
activities
on
a
consistent
basis.
That
suite
of
tools
will
include:


Commonly
used
performance
measures
to
help
staff
and
managers
better
identify
whether
public
involvement
practices/
activities
are
performed
and
tools
used
appropriately,
and
whether
they
achieve
the
intended
effects

Training
in
the
use
of
the
tools,
measures
and
techniques
°
Survey/
feedback
templates
that
EPA
staff
can
use
to
obtain
input
from
the
participating
public
for
evaluating
processes,
activities
and
events
°
A
database/
spreadsheet
program
enabling
staff
to
enter
data
from
returned
questionnaires,
easily
determine
results
and
determine
what
actions
to
take
to
improve
events,
activities
or
processes
3
°
A
central
collection
of
evaluation
instruments
that
EPA
programs
and
regions
have
used
to
review
their
public
involvement
efforts,
and

An
OMB­
approved
Information
Collection
Request
(
ICR)
to
support
gathering
public
involvement
feedback
based
on
a
series
of
questionnaires
that
EPA
staff
can
use
to
obtain
input
from
the
participating
public
for
assessing
processes,
activities
and
events
The
Government
Performance
and
Results
Act
of
1993
requires
that
agencies
gather
and
use
feedback.
President
Bush's
Management
Agenda
underscores
the
need
for
citizencentered
service
delivery,
increased
satisfaction
with
government
services
and
the
ability
to
prove
government
is
doing
a
better
job
through
measuring
outcomes.
These
questionnaires
and
the
related
tools
will
enable
EPA
to
improve
its
public
involvement
processes,
activities
and
events,
enhance
public
involvement
opportunities
in
ways
the
public
will
value
and
ultimately
will
improve
EPA
decision
making
by
more
consistently
taking
into
account
the
values
and
views
of
the
citizens
we
serve.

The
key
to
building
trust
in
a
public
involvement
process
is
to
pro­
actively
gauge
participants'
perceptions
and
aggressively
act
on
what
they
say
went
well
and
what
needed
correction.
The
techniques
for
obtaining
such
feedback
do
not
have
to
be
complicated
or
difficult
to
administer;
however,
they
do
have
to
be
timely
and
the
changes
made
because
of
public
input
must
be
visible
and
effective.

Some
of
the
feedback
needs
in
public
involvement
parallel
those
in
customer
satisfaction.
Therefore,
staff
developing
this
Public
Involvement
ICR
used
OMB's
Resource
Manual
for
Customer
Surveys
(
dated
October
1993),
which
provides
guidance
on
obtaining
quality
survey
results.
They
also
used
EPA's
"
Hearing
the
Voice
of
the
Customer:
Guidelines
for
Customer
Feedback
and
Customer
Satisfaction
Measurement
"
(
November
1998).

EPA
developed
and
used
the
"
Guidelines"
document
to
ensure
proper
design
of
EPA
customer
feedback
and
customer
satisfaction
measurement
activities,
increase
the
use
and
application
of
customer
feedback,
and
build
internal
capacity
to
carry
out
those
activities.
The
Guidelines
use
a
five­
stage
model
for
feedback:
Plan,
Construct,
Conduct,
Analyze
and
Act.
A
long
series
of
detailed
questions
supplement
the
Guidelines
to
further
assist
those
planning
on
obtaining
feedback.
Portions
of
the
Guidelines
(
available
on
the
Internet
@
http://
www.
epa.
gov
customerservice/
guide.
htm)
will
be
included
in
public
involvement
feedback
training.

The
successful
operation
of
the
generic
clearance
for
customer
satisfaction
surveys
and
the
applications
for
1997,
1999
and
2003
generic
information
collection
requests
served
as
the
model
for
developing
this
application.
Public
involvement
staff
in
the
National
Center
for
Environmental
Innovation
(
NCEI)
will
develop,
distribute
and
post
on
the
http://
www.
epa.
gov
/
publicinvolvement
web
site
a
fact
sheet
reflecting
OMB's
"
Terms
of
Clearance"
for
the
ICR,
strongly
stating
restrictions
on
the
use
of
this
clearance.
Burden
will
be
defined
on
the
surveys.
The
agency
will
display
the
OMB
control
number
and
clearance
expiration
date
on
every
survey
and
will
inform
respondents
of
its
legal
significance
(
see
5
CFR
1320.5(
b)).
In
addition,
every
questionnaire
will
have
introductory
information
for
staff
clarifying
its
purpose
and
any
use
restrictions.
The
spreadsheet/
database
will
also
have
instructions
for
staff
use
and
the
collection
of
results.
4
2(
b)
Practical
Utility/
Uses
of
the
Data
Information
gathered
from
these
surveys
will
continuously
help
EPA
to
build,
validate
and
improve
public
involvement
measurement
systems.
Survey
results
may
be
used
to
identify:
1)
Needs
and
expectations
of
public
involvement
participants
2)
Strengths
and
weaknesses
of
EPA
involvement
activities
3)
Ideas
or
suggestions
for
improvement
of
EPA
involvement
activities
from
people
who
participate
in
them
4)
Barriers
to
achieving
improved
public
involvement
processes
and
activities
5)
Changes
in
what
and
how
we
measure
the
success
of
public
involvement
activities
and
processes
6)
Common
performance
measures
By
using
these
surveys,
EPA
personnel
will
be
able
to
better
define
desired
outcomes
of
public
involvement
processes
and
activities
and
determine
whether
the
goals
are
met.
For
example,
if
achieving
broad,
inclusive
involvement
is
a
goal,
there
will
be
standards
and
questions
to
measure
whether
those
involved
believed
that
most
(
if
not
all)
potential
interests
participated
or
had
the
opportunity
to
do
so.

While
the
information
will
not
be
used
for
regulatory
development,
the
results
of
public
involvement
surveys
could
lead
to
changes
in
those
activities
and
in
certain
Agency
processes
and
policies,
and
to
development
of
additional
guidance
related
to
public
involvement.
Ultimately,
these
changes
could
result
in
citizen­
driven
improvements
in
the
public
involvement
activities
the
Agency
provides
and
the
methods
EPA
uses
to
gather
and
consider
public
input.

In
researching
current
literature
for
measures
of
success
used
to
evaluate
decision
making
processes
and
public
involvement
activities,
EPA
found
many
goals
against
which
to
measure
attributes
of
processes,
collaboration,
decision
quality,
social
and
environmental
outcomes.
EPA
recognizes
that
only
retrospectively,
after
making
a
decision,
can
the
Agency
ask
participants
to
evaluate
the
quality
of
the
decision,
the
fairness
of
the
process
overall,
or
the
degree
to
which
their
ideas
are
reflected
in
that
decision.
Measuring
outcomes
in
the
environment
is
much
longer
term
and
not
in
any
way
part
of
the
work
to
be
done
under
this
ICR.
The
Agency
is
determined
to
focus
on
gathering
data
that
will
enable
us
to
improve
activities
that
are
component
parts
of
larger
decision­
making
processes.
If
we
can
use
the
information
gathered
to
improve
components
of
public
involvement
processes,
people
who
participate
in
those
events
will
have
more
meaningful
and
productive
opportunities
to
provide
input
to
the
Agency.

Most
of
EPA's
public
involvement
activities
have
similar
components:
finding
the
appropriate
people
to
involve,
ensuring
they
receive
adequate
notice
of
the
opportunity
to
participate,
providing
information
for
their
review,
conducting
the
activity
and
doing
follow­
up
after
the
event.
In
most
cases
asking
those
who
participated
how
well
EPA
did
[
insert
action
]
5
on
a
scale
from
1
to
6
(
with
7,
don't
know,
as
the
last
alternate)
will
provide
good
information
about
what
the
Agency
needs
to
improve
before
holding
the
next
similar
event.

The
questionnaires
are
structured
primarily
to
obtain
quantitative
data.
Most
questions
are
close­
ended
and
use
the
Lichert
Scale.
EPA
will
also
solicit
some
qualitative
data
using
a
small
number
of
open­
ended
questions.
Most
of
the
qualitative
questions
ask
for
suggestions
on
how
to
improve
an
activity
or
process.

Achieving
a
performance
standard
of
75%
positive
responses
(
5
and
above
on
a
6­
point
Lichert
Scale)
will
be
the
first
goal
EPA
uses
as
a
measure
of
our
success
at
engaging
the
public
in
Agency
decision­
making
processes
and
related
involvement
activities.
This
goal
will
provide
the
necessary
framework
for
management
to
use
survey
results.

Process
questions
can
address
the
clarity
of
the
stated
goals
and
roles
of
the
public
and
EPA,
and
focus
on
the
overall
process
transparency.
Questions
address
attributes
of
the
actions
leading
to
the
event
(
publicity,
information
provision,
access,
etc.),
the
logistics
(
site
convenience
and
comfort,
time
of
day,
equipment,
provision
for
people
with
disabilities,
etc)
conduct
of
the
event
(
courtesy,
respect,
equality,
etc.)
and
ask
for
suggestions
on
how
to
improve
the
particular
event
or
process.
More
areas
covered
in
some
of
the
of
questionnaires
follow.

Process

clear
goals

clear
process

clear
roles
for
public
and
agency

process
transparency
to
participants
Appropriate
people

diversity
of
views

balance
(
on
advisory
groups)

Information

accessibility
of
information
and
technical
information

clarity
(
understandable)


quality

timely
provision
of

responsiveness
to
requests
for
Conduct
of
event

courtesy

respect

equal
treatment
of
all

Logistics

convenience

timing

length
6

provision
for
people
with
disabilities
Effectiveness

each
event

overall
process

of
staff
or
other
support
(
FACA
group,
meeting
facilitation)


of
technical
assistance
provided
(
FACA,
CAG)


capacity
building
among
participants

new
alternatives
from
participants

of
personal
participation

otherwise
unavailable
information/
innovative
ideas/
holistic
views
provided
from
participants
Completed
process
or
decision

fair

based
on
good
science/
good
technical
information

economically
and/
or
socially
viable

environmentally
beneficial
outcomes

learning
extended
beyond
participants
to
the
general
public
(
community
capacity
building)


resolved
conflict

built
trust

considered
public
values
3.
Non­
duplication,
Consultations,
and
Other
Collection
Criteria
3(
a)
Non­
duplication
EPA
staff
who
conduct
public
involvement
activities
will
use
the
approved
questions
to
learn
how
participants
in
their
specific
activities
view
various
aspects
of
those
activities.
Therefore,
the
information
collected
will
not
overlap
with
other
programmatic
or
satisfaction
surveys.
Every
effort
will
be
made
to
channel
all
public
involvement
related
surveys
through
this
ICR
and
to
prevent
misuse
of
this
ICR
for
other
purposes.

3(
b)
Public
Notice
Required
Prior
to
ICR
Submission
to
OMB
EPA
will
conform
to
the
requirement
for
public
notice
by
publishing
a
preliminary
and
final
Federal
Register
Notice
concerning
our
intent
under
this
ICR
and
requesting
comment.

3(
c)
Consultations
EPA's
National
Center
for
Environmental
Innovation
(
NCEI)
public
involvement
staff
requested
input
to
this
ICR
application
from
practitioners
across
the
Agency.
This
ICR
was
prepared
in
consultation
with
the
Public
Involvement
Evaluation
and
Accountability
Task
Group
of
the
Public
Involvement
Improvement
Council
and
representatives
from
all
organizations
that
7
responded
to
the
opportunity
to
outline
their
possible
uses
of
the
public
involvement
questionnaires
through
the
year
2006.
This
feedback
was
used
to
develop
the
estimates
of
staff
and
respondent
burden.
EPA
contacted
professional
practitioners
through
their
networks
and
organizations
to
request
their
comments
during
the
period
of
review
for
the
draft
ICR.

3(
d)
Effects
of
Less
Frequent
Collection
This
information
collection
could
not
be
conducted
less
frequently.
EPA
will
learn
from
participant
reactions
to
and
perceptions
of
public
involvement
activities
that
the
Agency
now
provides
in
order
to
improve
them.
Programs
will
not
survey
all
participants
in
every
activity
provided.
There
will
be
sufficient
time
between
surveys
to
allow
the
actions
taken
in
response
to
customer
comments
to
show
results.
There
are
no
technical
or
legal
obstacles
to
reducing
the
burden.

3(
e)
General
Guidelines
This
ICR
complies
with
OMB's
general
guidelines
for
the
collection
of
information.

3(
f)
Confidentiality
Not
applicable
3(
g)
Sensitive
Questions
Feedback
questionnaires
in
this
ICR
application
will
not
be
used
to
collect
any
sensitive
data.

4.
The
Respondents
and
the
Information
Requested
4(
a)
Respondents/
SIC
Codes
The
EPA,
by
the
very
nature
of
its
mandate,
serves
very
large
and
diverse
groups
that
participate
in
our
decision­
making
processes
or
could
in
some
way
be
affected
by
EPA
decisions.
Participants
in
public
involvement
activities
range
from
well
known
scientific
experts
to
representatives
of
various
interest
groups,
to
individual
business
or
property
owners,
local
officials,
interested
young
students
and
residents
of
environmental
justice
neighborhoods
and
tribal
members.
The
term
"
the
public"
is
used
in
the
Public
Involvement
Policy
in
the
broadest
sense
to
include
anyone,
including
both
individuals
and
organizations,
who
may
have
an
interest
in
an
Agency
decision.
Appendix
2
of
the
Policy
provides
further
definition:

"
In
addition
to
private
individuals,
"
the
public"
includes,
but
is
not
limited
to,
representatives
of
consumer,
environmental
and
other
advocacy
groups;
environmental
justice
groups;
indigenous
peoples;
minority
and
ethnic
groups;
business
and
industrial
interests,
including
small
businesses;
elected
and
appointed
public
officials;
the
news
media;
trade,
industrial,
agricultural
and
labor
organizations;
public
health,
scientific
and
professional
representatives
and
societies;
civic
and
community
associations;
faith­
based
organizations;
and
research,
university,
education
and
governmental
8
organizations
and
associations."

4(
b)
Information
Requested
(
I)
Data
items,
including
record
keeping
requirements
The
Agency
will
maintain
records
of
the
surveys'
use.
Offices
sponsoring
the
surveys
will
retain
files
of
the
surveys,
responses
and
analysis.
Since
these
surveys
will
seek
to
measure
public
opinions
on
Agency
public
involvement
activities,
the
surveys
have
not
and
will
not
involve
respondents
in
extensive
searching
of
existing
sources,
or
reformatting
information
to
submit
to
the
Agency.
The
Agency
does
not
anticipate
any
record
keeping
activities
on
the
part
of
the
public
under
this
ICR.

(
II)
Respondent
Activities
EPA
public
involvement
surveys
will
focus
on
the
quality
of
the
pre­
event
work,
the
conduct
of
the
public
participation
opportunity
provided
and
the
speed
and
completeness
of
follow­
up
requested.
All
questionnaires
will
solicit
qualitative
feedback.
Most
questionnaires
will
be
completed
immediately
following
the
public
involvement
opportunity
before
participants
leave
the
event
location.
Several
will
be
mailed
or
e­
mailed
to
participants
as
follow­
up
to
a
single
event
or
upon
completion
of
a
decision­
making
process.
All
will
involve
reading
instructions
and
completing
questionnaires;
some
will
also
require
mailing
questionnaires.
The
questionnaires
will
require
ten
minutes
or
less
to
complete.

5.
The
Information
Collected
Agency
Activities,
Collection
Methodology,
and
Information
Management
5(
a)
Agency
Activities.

Agency
activities
associated
with
the
collection
of
information
include:

National
Center
for
Environmental
Innovation
staff:


Internal
EPA
review
and
approval
of
questionnaires
(
completed
in
2003)


Pretesting
questionnaires
submitted
in
this
ICR
application
both
internally
and
with
less
than
9
non­
federal
persons
(
completed
2003)

Headquarters
Office/
Region
staff:


Selecting
public
involvement
event
type
and
the
appropriate
pre­
approved
questionnaire

Assembling
data
sources
(
mailing
lists,
etc.)
if
necessary

Providing
copies
of
the
questionnaire
at
events,
or
disseminating
questionnaire
to
potential
respondents
after
an
event
or
process

Gathering
information
from
respondents

Answering
respondent
questions,
follow­
up

Reviewing
data

Recording
submissions
and
analyzing
results

Preparing
findings
9

Storing
and
maintaining
results

Reporting
overall
results
to
NCEI
to
share
internally
(
and
possible
post
on
the
http://
www.
epa.
gov/
publicinvolvement
Internet
page)


Taking
actions
to
improve
activities,
events
and
processes
5(
b)
Collection
Methodology
and
Information
Management
The
following
process
will
be
established:


EPA's
sponsoring
organizations
report
to
PI
staff
in
the
NCEI
their
planned
use
of
the
questionnaires;
report
changes
in
those
plans.


The
database
and
spreadsheet
program
record
the
sponsoring
office,
point
of
contact,
date,
location,
type
of
event
and
number
of
respondents.


In
all
cases,
the
purpose
of
the
surveys
is
to
obtain
feedback
from
participants
in
EPA
public
involvement
activities
and
processes
so
EPA
can
consider
and
use
their
ideas
to
improve
pubic
involvement.


Costs
and
burden
to
the
Agency
are
estimated
based
on
approximate
time
necessary
for
data
entry
and
consideration
of
results
for
each
questionnaire
type

Costs
and
burden
to
the
respondents
will
be
tracked
using
the
database
program
to
count
uses
of
and
respondents
to
each
questionnaire.


Complex
statistical
methods
will
not
be
used
for
the
collections
covered
under
this
generic
clearance.
The
spreadsheet/
database
will
perform
simple
calculations
for
those
who
enter
respondent
data.


Sponsoring
organizations
within
EPA
will
maintain
records
of
all
the
occasions
they
use
the
questionnaires.
The
database
program
and
spreadsheet
will
streamline
this
record
keeping
burden.
In
general,
survey
results
will
be
maintained
for
three
years
or
until
after
follow­
up
activities
have
been
completed.


EPA
organizations
will
notify
the
public
involvement
staff
in
EPA's
NCEI
of
their
use
of
questionnaires,
enter
response
data
in
the
database
program
and
periodically
report
follow­
up
actions
taken
based
on
survey
results.


The
public
involvement
staff
will
share
results,
lessons
learned
and
success
stories
with
other
offices
and
provide
feedback
on
overall
survey
results.
This
base
of
experiences/
lessons­
learned
will
be
useful
to
state
and
federal
partners
that
carry
out
public
involvement
activities
and
are
seeking
models
for
tracking
and
measuring
success.

5
(
c)
Small
Entity
Flexibility.
Not
applicable.

5
(
d)
Collection
Schedule.

The
collection
schedule
needs
to
be
flexible.
Public
involvement
event
schedules
often
slip
and
can
drastically
change
if
a
decision
process
becomes
more
complex
or
controversial
than
anticipated.
Such
changes
are
not
predictable,
so
estimating
with
precision
is
difficult.
Table
5­
1
is
a
draft
plan
for
the
use
of
questionnaires
for
2004/
5
­
2006/
7
(
depending
on
approval
date).

This
application
is
based
solely
on
good
faith
estimates
that
follow.
The
burden
estimate
is
not
precise
since
EPA
has
never
before
attempted
to
coordinate
obtaining
feedback
from
participants
in
involvement
events
across
all
programs
and
regions.
The
10
number
of
surveys
and
the
actual
burden
could
be
quite
small
during
the
first
year
and
build
as
offices
and
regions
see
the
usefulness
of
the
data
collected.

Community
Advisory
Groups
(
CAGs)
are
not
managed
by
EPA;
however,
the
Superfund
community
involvement
coordinators
work
to
ensure
that
such
groups
are
effective.
Many
of
the
groups
would
welcome
help
in
improving
their
meetings
and
may
make
use
of
the
meeting
feedback
form.
There
are
approximately
70
active
CAGs
nationwide,
with
an
average
of
15
people.

Some
of
the
CAGs
meet
as
often
as
monthly;
others
meet
only
once
a
quarter.
Were
about
30%
of
the
groups
to
use
the
forms
every
other
meeting
(
6
or
2
times
annually),
there
could
be
up
to
21
groups
using
the
surveys
(
30%
of
70
=
21).
Of
the
21
groups,
eleven
may
use
the
survey
6
times;
ten
may
use
it
two
times
equaling
86
uses
(
11
x
6
+
10
x
2
=
86
)
of
the
meeting
survey.
Were
30%
of
the
groups
to
use
the
process
effectiveness
survey
once
a
year,
there
could
be
21
uses
annually.
Based
on
those
projected
uses,
respondent
estimates
would
equal
1605
respondents
(
86
x15
(
1290)
+
21
X
15
(
315)).
We
project
slight
increases
in
use
as
CAG
managers
make
meeting
and
process
improvements
and
more
groups
choose
to
use
the
questionnaires.

EPA
issues
approximately
150
rules
each
year.
Publication
of
a
Federal
Register
Notice
requesting
public
comment
is
required;
sometimes
public
hearings
are
required;
sometimes
they
are
requested
(
RCRA).
For
the
majority
of
rules
and
regulations,
public
comment
is
the
only
involvement
activity.
If
one
hearing
were
held
for
30
of
the
rules
(
20%),
and
the
feedback
surveys
were
distributed
at
50%
of
the
events
held,
there
would
be
approximately
15
uses
of
the
public
hearings
survey
the
first
year.
There
would
be
many
fewer
uses
of
the
follow­
up
survey,
perhaps
as
few
as
5.
Annually
we
project
a
steady,
small
increase
in
use
for
both
instruments.

EPA's
Center
for
Conflict
Resolution
and
Prevention
(
CCRP)
has
a
contract
vehicle
for
agency­
wide
use
that
enables
organizations
to
request
services
for
convening
and
facilitating
public
involvement
activities.
The
activities
can
be
meetings,
hearings,
small
group
workshops/
discussions,
listening
sessions,
FACA
sessions,
negotiations,
etc.
The
mix
of
activities
changes
from
year
to
year.

Based
on
rough
estimates,
and
the
expectation
that
contractors
will
be
required
(
by
the
terms
of
the
contract)
to
do
formative
evaluation,
and
to
consider
using
the
approved
forms
to
obtain
ongoing
participant
feedback
to
help
gauge
their
success,
we
estimate
contractors
will
distribute
the
various
surveys
to
participants
as
many
as
275
times.
This
estimate
is
based
on
the
past
history
of
the
three
five­
year
contracts.
During
each
contract
cycle
there
is
a
steady
build
up
to
approximately
100
active
task
orders
in
year
three,
and
then
a
slow
decline
in
the
number
of
active
task
orders
until
the
end
of
year
five.
The
contract
is
about
to
be
renewed.

About
40%
of
the
100
task
orders
support
contractors
who
do
mediation
and
other
alternative
dispute
resolution
techniques;
the
remaining
60%
are
for
public
involvement
activities.
Of
the
60
task
orders,
approximately
50
focus
on
a
single
project;
the
remaining
ten
average
five
projects
each
(
50
+
10
X
5
=
100
projects).
Of
the
100
projects,
approximately
50
hold
only
one
public
involvement
event;
the
remaining
50
have
three
to
six
events.
If
all
contractors
are
asked
to
distribute
the
surveys
following
all
events,
up
to
275
uses
are
projected
(
50
+
50
X
4.5
(
events)).
1
11
EPA
does
not
believe
that
it
is
realistic
to
expect
that
every
contractor
will
distribute
the
surveys
after
every
event
to
assess
its
effectiveness
however,
and
will
not
require
that
they
do
so.
We
project
about
70
­
90
uses
per
year
of
the
various
event
surveys
and
we
believe
that
contractors
will
administer
fewer
of
the
post
event
follow­
up
surveys
under
the
100
active
task
orders
of
the
CCRP.

group
is
dissolved
or
finishes
a
long
term
project,
particularly
if
a
contractor
has
been
involved
throughout
a
multi­
year
process.
Estimates
include
a
few
such
assessments.
Without
contractor
support,
offices
and
regions
sponsor
the
same
types
of
events
under
community­
based,
watershed,
voluntary,
environmental
justice,
tribal
and
other
types
of
projects
and
programs.
Without
an
Agency­
wide
requirement
to
use
the
forms
for
a
specific
percent
of
all
public
involvement
activities,
EPA
organizations
will
not
widely
adopt
and
use
the
surveys.
However,
if
those
who
do
so
show
continuous
improvement
in
event
outcomes
and
participant
satisfaction,
staff's
voluntary
use
of
the
surveys
should
increase.
Estimates
reflect
that
expectation.
We
estimate
that
EPA/
contractor
use
of
the
surveys
will
equal
approximately
90
to
110
uses
per
EPA
has
26
FACA
advisory
groups,
each
with
a
designated
federal
official
(
DFO).
Most
FACAs
are
long
term
standing
groups
that
provide
advice
on
a
wide
variety
of
topics
to
programs
and
the
Agency.
Based
on
the
number
of
meetings
held
annually
in
fiscal
years
2000
through
2003
(
reported
in
GSA
database
www.
fido.
gov/
faca
database),
the
annual
average
is
150
meetings.
Were
60
percent
of
the
events
followed
by
use
of
the
meeting
questionnaire,
we
estimate
approximately
90
uses
of
the
FACA
meeting
effectiveness
survey
in
the
second
year,
slightly
fewer
(
86)
in
2004;
slightly
more
(
100)
in
2006.

Until
a
FACA
ends
or
a
specific
task
of
FACA
input
ends,
there
is
no
use
for
the
process
effectiveness
questionnaire.
Over
the
past
four
years
308
FACA
reports
with
recommendations
were
issued.
Since
the
number
has
been
steadily
decreasing
(
from
118
to
48),
rather
than
use
an
average,
we
project
a
continued
decline
to
not
more
than
40
reports
in
2004.
Annually,
we
expect
not
more
than
15
uses
of
the
process
effectiveness
questionnaire.

Annually,
26
DFOs
will
receive
a
request
to
assess
the
usefulness
of
their
FACA's
advice.
(
This
survey
does
not
require
OMB
approval
since
it
is
a
federal
official
responding.)

Table
5
­
1
Planned
Use
of
Questionnaires
12
Questionnaires
Number
of
Uses
2004
2005
2006
Process
effectiveness
(
participant)
15
15
15
Process
effectiveness
(
EPA/
contractor
)
2
2
2
DFO
annual
assessment**
26
26
26
**
EPA
may
also
use
the
DFO
FACA
assessment
that
OMB
approved
in
GSA's
2002
application
for
all
federal
agencies
with
FACA
groups.

6.
Estimating
the
Burden
and
the
Cost
of
the
Collection
6(
a)
Estimating
Respondent
Burden.
13
Staff
who
developed
the
questionnaires
did
so
knowing
that
respondent
burden
should
be
as
low
as
possible
in
keeping
with
the
Paperwork
Reduction
Act.
The
questionnaire
designs
are
simple,
convenient,
easy
to
respond
to
and
clear
in
content
and
purpose.
At
the
end
of
a
public
event
people
are
ready
to
start
home
and
will
not
remain
and
respond
if
the
questionnaires
are
long
or
complex.
Therefore,
questionnaires
have
limited
scope
and
require
a
short
time
to
complete.

Responding
to
the
meeting
and
process
effectiveness
questionnaires
(
FACA,
CAG,
Meetings,
Hearings,
Listening
Sessions,
Small
Group
Discussions)
averages
about
ten
minutes.
The
questionnaires
that
focus
on
follow­
up
activities
after
those
events
take
not
more
than
5
minutes
to
complete.
The
first
stakeholder
negotiation
questionnaire
takes
15
minutes
to
complete
while
the
second
requires
10
minutes.
The
estimate
of
respondent
burden
are
shown
below
in
Table
6­
1
(
a­
c).
The
estimate
of
respondent
burden
was
based
on
obtaining
responses
for
questionnaire
use
as
summarized
in
Table
5­
1.

Table
6­
1
(
a)
Respondent
Burden
2004
Minutes/
response
(
a)
Uses
(
b)
Responses/
use
(
c)
Total
responses
(
d=
b*
c)
Burden
hours
(
e=
d*
a/
60)

Community
14
15
Table
6­
1
(
c)
Respondent
Burden
2006
Minutes/
response
(
a)
Uses
(
b)
Responses/
use
(
c)
Total
responses
(
d=
b*
c)
Burden
hours
(
e=
d*
a/
60)
2
16
The
EPA
estimates
respondent
burden
as
follows:
FY
2005:
The
three
year
total
burden
hours
or
per
6(
b)
Estimating
Respondent
Costs
I
Labor
Costs.
Since
the
respondents
represent
previously
applicable
instructions
or
requirements,"
"
training
personnel
to
be
able
to
respond
to
a
collection
of
information,"
"
searching
data
sources,"
nor
a
need
for
the
respondents
to
keep
records.
Burden
activities
include
only
a
few
steps:
reviewing
instructions,
responding
and
submitting
(
handing
back
or
dropping
into
a
collection
box
on
site)
or
e­
mailing/
mailing
responses
when
the
surveys
are
not
performed
in
person.

Table
6­
2
displays
the
total
burden
estimates
for
respondents
for
each
type
of
survey
and
total
estimated
respondent
cost
17
Public
Contributors
9
0.09
3
18
STAKEHOLDER
NEGOTIATION
TOTAL
18
0.198
Federal
Advisory
Committee
Act
Groups
Meeting
effectiveness
(
participant)
276
4.14
Meeting
effectiveness
(
EPA/
contractor)
90
1.8
Process
effectiveness
(
participant)
45
0.9
Process
effectiveness
(
EPA/
contractor
)
6
0.12
DFO
annual
assessment**
­
­

FACA
TOTAL
417
6.96
QUESTIONNAIRE
TOTALS
1,819
54.571
Because
some
of
the
activities
are
regularly
held
meetings
(
CAGs
and
FACAs),
some
respondents
will
complete
the
same
questionnaire
several
times
within
the
ICR
period.
This
is
necessary
and
appropriate
since
only
they
will
be
able
to
experience
and
comment
on
the
effectiveness
of
the
changes
made
in
response
to
their
input.

II
Capital
and
Operations
and
Maintenance
Costs
Not
applicable.

III
Capital/
Start­
up
vs.
Operating
and
Maintenance
(
O
&
M)
Costs
Not
applicable.

IV
Annualizing
Capital
Costs
Not
applicable.

6
(
c)
Estimating
Agency
Burden
and
Cost.

Tables
6­
3
through
6­
7
provide
the
estimates
for
agency
burden
associated
with
each
aggregated
survey
type
for
disseminating
public
involvement
surveys
and
analyzing
the
results.
Wage
estimates
were
divided
into
three
categories
of
labor:
Management
(
GS­
15),
Technical
(
GS­
13),
and
Clerical
(
GS­
7).
Wage
estimates
were
based
on
OPM's
General
Table
6­
3.
Agency
Burden/
Cost
for
Meeting
Effectiveness
Surveys*
19
Activities
for
each
Survey
Use
Gathering
information
Inputting
data
into
database
Reviewing
data;
follow­
ups
Analyzing
results
Storing
and
maintaining
results
Preparing
survey
findings
Acting
on
results
Total
burden
by
position
(
manager,
technical,
clerical)

Total
agency
burden
and
cost
per
survey
use
*
These
include
meeting
effectiveness
surveys
for
CAGs,
Public
Meetings,
Public
Hearing,
Listening
Session,
Small
Group
Discussion
and
FACA
advisory
groups
(
six
surveys
in
total).

Table
6­
4.
Agency
Burden/
Cost
for
Process
Effectiveness
Surveys*

Activities
for
each
Survey
Use
Gathering
information
Inputting
data
into
database
Reviewing
data
Analyzing
results
Storing
and
maintaining
results
Preparing
survey
findings
Acting
on
results
Total
burden
by
position
(
manager,
technical,
clerical)

Total
agency
burden
and
cost
per
survey
use
*
These
include
process
effectiveness
surveys
for
CAGs
and
FACA
advisory
groups
(
two
surveys
in
total).

Table
6­
5.
Agency
Burden/
Cost
for
Follow­
up
Surveys*
20
Activities
for
each
Survey
Use
Gathering
information
Inputting
data
into
database
Reviewing
data
Analyzing
results
Storing
and
maintaining
results
Preparing
survey
findings
Acting
on
results
Total
burden
by
position
(
manager,
technical,
clerical)

Total
agency
burden
and
cost
per
survey
use
*
These
include
follow­
up
surveys
for
Public
Meetings,
Public
Hearings,
Listening
Sessions,
and
Small
Discussion
Group
Session
surveys
(
four
surveys
in
total).

Table
6­
6.
Agency
Burden/
Cost
for
EPA/
Contractor
Meeting
and
Process
Effectiveness
Surveys*

Activities
for
each
Survey
Use
Gathering
information
Inputting
data
into
database
Reviewing
data
Analyzing
results
Storing
and
maintaining
results
Preparing
survey
findings
Acting
on
results
Total
burden
by
position
(
manager,
technical,
clerical)

Total
agency
burden
and
cost
per
survey
use
2.1
*
These
include
meeting
and
process
effectiveness
surveys
for
CAGs,
FACA
advisory
groups,
Public
Meetings,
Public
Hearings,
Listening
Sessions,
and
Small
Group
Discussion
Session
Surveys
(
eight
surveys
in
total;
six
for
meeting
effectiveness
and
two
for
process
effectiveness.

Table
6­
7.
Agency
Burden/
Cost
for
Stakeholder
Negotiation
Surveys*
21
Activities
for
each
Survey
Use
Gathering
information
Inputting
data
into
database
Reviewing
data
Analyzing
results
Storing
and
maintaining
results
Preparing
survey
findings
Acting
on
results
Total
burden
by
position
(
manager,
technical,
clerical)

Total
agency
burden
and
cost
per
survey
use
*
These
include
stakeholder
negotiation
surveys
for
Stakeholder
Participants
and
Public
Contributors
(
two
surveys
in
total).

A
cross
Agency
staff
group
developed
all
questionnaires,
so
there
are
no
additional
costs
of
development
and
management
review
shown
for
using
the
surveys
in
any
of
the
cost
tables.
Instructional
materials
accompany
the
surveys
so
their
administration
will
take
little
staff
time.
Data
entry
and
analysis
are
also
streamlined
when
staff
use
the
database
spreadsheet
package
provided.
Overall
costs
to
the
Agency
for
use
of
the
feedback
surveys
is
low
and
can
be
kept
to
a
minimum
because
of
the
use
of
standardized
surveys
and
data
entry/
analysis.

Survey
development
costs
are
estimated
at
approximately
$
21,780
in
EPA
staff
and
intern
salaries
during
2002
and
2003.
Costs
are
based
on
an
average
of
30
hours
per
survey
surveys
X
$
33.00
an
hour.
This
total
includes
the
time
of
members
of
the
cross
Agency
Public
Involvement
Evaluation
and
Accountability
Task
Group,
interns
and
practitioners
across
EPA.

Through
a
survey
and
personal
interviews
the
Task
Group
and
supporting
staff
asked
EPA
practitioners
to
state
the
most
commonly
used
public
involvement
activities,
whether
they
might
do
more
evaluation
were
it
easier
to
do
and
the
kinds
of
participant
feedback
that
would
be
useful
to
them.
Responses
helped
to
define
the
types
of
activities
for
which
to
develop
surveys
and
the
types
of
questions
to
ask.

In
addition,
the
Task
Group
used
related
input
from
EPA's
2001
two
week
Internetbased
"
Dialogue
on
Public
Involvement
in
EPA
Decisions."
Dialogue
participants
noted
problems
with
the
timeliness
of
pre­
event
information,
logistical
problems
at
meetings,
the
quality
of
information
provided
and
the
treatment
received
before,
at
and
after
events.
They
4
22
stated
that
small
improvements
to
logistics,
personal
treatment
and
information
dissemination
aspects
of
public
involvement
activities
made
because
of
their
input
could
make
large
differences
in
participants'
satisfaction
with
events,
their
belief
in
the
process
and
their
trust
in
the
decisions.

Various
Task
Group
members
drafted
the
individual
questionnaires
which
the
full
group
reviewed
several
times.
A
sample
fact
sheet
for
staff
on
how
to
use
the
surveys
was
the
basis
for
a
series
of
Guides
on
how
to
administer
and
use
the
six
different
sets
of
feedback
templates.
Several
of
the
questionnaires
and
guides
as
well
as
the
data
entry
system
were
pre­
tested
in
the
field
(
with
fewer
than
9
non
federal
persons)
and
discussed
in
conference
presentations.
Suggested
changes
were
incorporated.

A
Task
group
member
developed
the
initial
three
models
for
the
database
spreadsheet
and
instructions
for
its
use.
Using
the
models,
an
EPA
intern
completed
the
remaining
Guides,
and
will
complete
the
database
spreadsheets
and
instructional
texts.
These
should
be
in
place
by
the
Summer
of
2004.

Based
on
past
use
of
the
Customer
Satisfaction
ICR
over
the
years,
the
Agency
is
not
likely
to
administer
each
questionnaire
as
many
times
as
now
anticipated.
Programs
may
not
be
ready
to
test
whether
the
changes
they
make
to
a
process
or
activity
issue
as
quickly
as
they
now
predict,
and
it
may
take
longer
that
they
expect
to
institute
changes.

6(
d)
Estimating
the
Respondent
Universe
and
Total
Burden
and
Costs
Burden
Table
6­
1(
a­
c)
provides
information
on
each
survey
by
instrument
type,
number
of
respondents
expected,
burden
per
response,
number
of
uses
(
if
more
than
one)
and
burden
hours
requested
per
survey.
Table
6­
2
summarizes
the
total
burden
and
costs
for
respondents
by
type
of
survey.
Activities
have
been
grouped
to
reflect
the
various
types
of
surveys
and
the
total
respondents
expected
for
each
instrument
type.
In
all
cases,
the
activities
23
CAG
and
FACA
Process
Effectiveness
Surveys
Public
Meeting,
Public
Hearing
and
Listening
Session
Follow­
up
Surveys
EPA/
Contractor
Meeting
Effectiveness
and
Overall
Process
Effectiveness
Surveys
Negotiations
Feedback
Surveys
Total
III
Variations
in
the
Annual
Bottom
Line
EPA
burden
hour
projections
are:
FY
2004:
2,343
hours
respondent
burden
from
14,819
individuals;
FY
2005:
2,923
hours
respondent
burden
from
18,382
individuals;
FY
2006:
3,350
hours
respondent
burden
from
21,370
individuals.
Development
costs
which
occurred
prior
to
the
use
of
any
questionnaires
do
not
factor
into
Agency
survey
administration
costs,
since
all
surveys
will
be
pre­
approved.

IV
Reasons
for
Change
in
Burden
EPA
expects
to
see
a
steady
growth
in
the
use
of
the
surveys
as
programs
and
regions
learn
how
to
use
them
and
the
database
supporting
them,
and
to
better
apply
the
information
they
obtain
to
improve
public
involvement
processes
and
events.
During
the
three
years
of
this
ICR,
there
should
be
small
increases
in
the
costs
for
administering
surveys,
analyzing
data
and
acting
on
the
findings
due
to
more
usage
and
salary
increases.

Table
6­
9
Aggregate
Agency
and
Respondents
Burden/
Cost*
24
2004
2005
2006
Aggregate
Respondents
Agency
*
In
addition
to
the
cost
the
agency
expects
to
incur
over
the
three­
year
period
to
administer
the
questionnaires,
EPA
incurred
a
cost
of
$
21,780
to
develop
them
(
see
page
21).

V
Burden
Statement:
The
annual
public
reporting
and
recordkeeping
burden
for
this
collection
of
information
is
estimated
to
average
0.158
hours
per
response.
Burden
means
the
total
time,
effort,
or
financial
resources
expended
by
persons
to
generate,
maintain,
retain,
or
disclose
or
provide
information
to
or
for
a
Federal
agency.
This
includes
the
time
needed
to
review
instructions;
develop,
acquire,
install,
and
utilize
technology
and
systems
for
the
purposes
of
collecting,
validating,
and
verifying
information,
processing
and
maintaining
information,
and
disclosing
and
providing
information;
adjust
the
existing
ways
to
comply
with
any
previously
applicable
instructions
and
requirements;
train
personnel
to
be
able
to
respond
to
a
collection
of
information;
search
data
sources;
complete
and
review
the
collection
of
information;
and
transmit
or
otherwise
disclose
the
information.
An
agency
may
not
conduct
or
sponsor,
and
a
person
is
not
required
to
respond
to,
a
collection
of
information
unless
it
displays
a
currently
valid
OMB
control
number.
The
OMB
control
numbers
for
EPA's
regulations
are
listed
in
40
CFR
part
9
and
48
CFR
chapter
15.

To
comment
on
the
Agency's
need
for
this
information,
the
accuracy
of
the
provided
burden
estimates,
and
any
suggested
methods
for
minimizing
respondent
burden,
including
the
use
of
automated
collection
techniques,
EPA
has
established
a
public
docket
for
this
ICR
under
Docket
ID
No.
OA­
2003­
0009,
which
is
available
for
public
viewing
at
the
Office
of
Environmental
Information
(
OEI)
Docket
in
the
EPA
Docket
Center
(
EPA/
DC),
EPA
West,
Room
B102,
1301
Constitution
Ave.,
NW,
Washington,
DC.
The
EPA
Docket
Center
Public
Reading
Room
is
open
from
8:
30
a.
m.
to
4:
30
p.
m.,
Monday
through
Friday,
excluding
legal
holidays.
The
telephone
number
for
the
Reading
Room
is
(
202)
566­
1744,
and
the
telephone
number
for
the
OEI
Docket
is
(
202)
566­
1752.
An
electronic
version
of
the
public
docket
is
available
through
EPA
Dockets
(
EDOCKET)
at
http://
www.
epa.
gov/
edocket.
Use
EDOCKET
to
submit
or
view
public
comments,
access
the
index
listing
of
the
contents
of
the
public
docket,
and
to
access
those
documents
in
the
public
docket
that
are
available
electronically.
Once
in
the
system,
select
"
search,"
then
key
in
the
docket
ID
number
identified
above.
Also,
you
can
send
comments
to
the
Office
of
Information
and
Regulatory
Affairs,
Office
of
Management
and
Budget,
725
17th
Street,
NW,
Washington,
DC
20503,
Attention:
Desk
Office
for
EPA.
Please
include
the
EPA
Docket
ID
No.
OA­
2003­
0009
in
any
correspondence.
25
EXHIBITS
Questionnaires
for
OMB
Review
All
future
surveys
under
this
ICR
are
to
include,
on
or
near
the
first
page
of
the
survey,
a
burden
statement
specific
to
that
survey
explaining
the
number
of
hours/
minutes
per
year
per
respondent
and
what
that
burden
entails
(
e.
g.:
Respondent
burden
for
this
survey
is
estimated
to
be
minutes
for
reading
and
responding
to
the
questions.)
as
well
as
the
following
paragraphs.

V
Burden
Statement:
The
annual
public
reporting
and
recordkeeping
burden
for
this
collection
of
information
is
estimated
to
hours
per
response.
Burden
means
the
total
time,
effort,
or
financial
resources
expended
by
persons
to
generate,
maintain,
retain,
or
disclose
or
provide
information
to
or
for
a
Federal
agency.
This
includes
the
time
needed
to
review
instructions;
develop,
acquire,
install,
and
utilize
technology
and
systems
for
the
purposes
of
collecting,
validating,
and
verifying
information,
processing
and
maintaining
information,
and
disclosing
and
providing
information;
adjust
the
existing
ways
to
comply
with
any
previously
applicable
instructions
and
requirements;
train
personnel
to
be
able
to
respond
to
a
collection
of
information;
search
data
sources;
complete
and
review
the
collection
of
information;
and
transmit
or
otherwise
disclose
the
information.
An
agency
may
not
conduct
or
sponsor,
and
a
person
is
not
required
to
respond
to,
a
collection
of
information
unless
it
displays
a
currently
valid
OMB
control
number.
The
OMB
control
numbers
for
EPA's
regulations
are
listed
in
40
CFR
part
9
and
48
CFR
chapter
15.

To
comment
on
the
Agency's
need
for
this
information,
the
accuracy
of
the
provided
burden
estimates,
and
any
suggested
methods
for
minimizing
respondent
burden,
including
the
use
of
automated
collection
techniques,
EPA
has
established
a
public
docket
for
this
ICR
under
Docket
ID
No.
OA­
2003­
0009,
which
is
available
for
public
viewing
at
the
Office
of
Environmental
Information
(
OEI)
Docket
in
the
EPA
Docket
Center
(
EPA/
DC),
EPA
West,
Room
B102,
1301
Constitution
Ave.,
NW,
Washington,
DC.
The
EPA
Docket
Center
Public
Reading
Room
is
open
from
8:
30
a.
m.
to
4:
30
p.
m.,
Monday
through
Friday,
excluding
legal
holidays.
The
telephone
number
for
the
Reading
Room
is
(
202)
566­
1744,
and
the
telephone
number
for
the
OEI
Docket
is
(
202)
566­
1752.
An
electronic
version
of
the
public
docket
is
available
through
EPA
Dockets
(
EDOCKET)
at
http://
www.
epa.
gov/
edocket.
Use
EDOCKET
to
submit
or
view
public
comments,
access
the
index
listing
of
the
contents
of
the
public
docket,
and
to
access
those
documents
in
the
public
docket
that
are
available
electronically.
Once
in
the
system,
select
"
search,"
then
key
in
the
docket
ID
number
identified
above.
Also,
you
can
send
comments
to
the
Office
of
Information
and
Regulatory
Affairs,
Office
of
Management
and
Budget,
725
17th
Street,
NW,
Washington,
DC
20503,
Attention:
Desk
Office
for
EPA.
Please
include
the
EPA
Docket
ID
No.
OA­
2003­
0009
and
OMB
control
number
(
2010­
XXXX)
[
insert
the
appropriate
OMB
control
number]
in
any
correspondence.
26
1.
Community
Advisory
Groups
­
User's
Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
29
a)
Background
Information
for
EPA
Staff
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
30
b)
Checklist
for
Administering
Questionnaires
and
Using
Results
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
33
c)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
CAG
Meeting
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
35
d)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
CAG
Meeting
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
37
e)
Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
CAG
Process
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
39
f)
Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
CAG
Process
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
41
2.
Public
Meetings
­
User's
Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
42
a)
Background
Information
for
EPA
Staff
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
43
b)
Checklist
for
Administering
Questionnaires
and
Using
Results
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
46
c)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Public
Meeting
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
48
d)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Public
Meeting
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
52
e)
Questionnaire
for
Feedback
on
the
Effectiveness
of
Public
Meeting
Follow­
Up
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
54
3.
Public
Hearings
­
User's
Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
56
a)
Background
Information
for
EPA
Staff
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
57
b)
Checklist
for
Administering
Questionnaires
and
Using
Results
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
60
c)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Public
Hearing
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
62
d)
Questionnaire
for
Feedback
on
the
Effectiveness
of
Public
Hearing
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
65
e)
Questionnaire
for
Feedback
on
the
Effectiveness
of
Public
Hearing
Follow­
up
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
67
4.
Listening
Sessions
­
User's
Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
69
a)
Background
Information
for
EPA
Staff
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
70
b)
Checklist
for
Administering
Questionnaires
and
Using
Results
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
73
c)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Listening
Session
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
75
d)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Listening
Session
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
79
e)
Questionnaire
for
Effectiveness
of
Listening
Session
Follow­
Up
27
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
81
5.
Small
Discussion
Group
Session
­
User's
Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
83
a)
Background
Information
for
EPA
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
84
b)
Checklist
for
Administering
Questionnaires
and
Using
Results
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
87
c)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Small
Discussion
Group
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
89
d)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Small
Discussion
Group
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
92
e)
Questionnaire
for
Effectiveness
of
Small
Discussion
Group
Follow­
Up
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
94
6.
Stakeholder
Negotiation
­
User's
Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
96
a)
Background
Information
for
EPA
Staff
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
97
b)
Checklist
for
Administering
Questionnaires
and
Using
Results
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
101
c)
Questionnaire
for
Feedback
on
Public
Involvement
in
EPA's
Stakeholder
Negotiations
(
stakeholder
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
103
d)
Questionnaire
for
Feedback
on
Public
Involvement
in
EPA's
Stakeholder
Negotiations
(
public's
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
106
7.
Federal
Advisory
Committee
Act
Committees
­
User's
Guide
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
108
a)
Background
Information
for
EPA
Staff
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
109
b)
Checklist
for
Administering
Questionnaires
and
Using
Results
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
112
c)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
FACA
Meeting
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
114
d)
Questionnaire
for
Feedback
on
the
Effectiveness
of
a
FACA
Meeting
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
116
e)
Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
FACA
Process
(
participant
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
117
f)
Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
FACA
Process
(
EPA/
contractor
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
119
g)
Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
FACA
Committee
(
DFO
assessment)
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
122
28
This
page
intentionally
left
blank
29
Assessing
EPA­
supported
Community
Advisory
Groups
A
User's
Guide
with
Questionnaires
Enclosed
are
information
and
tools
for
EPA
staff
to
use
in
generating
feedback
on
the
effectiveness
of
Community
Advisory
Groups
(
CAGs).
Specifically,
this
user
guide
contains
the
following:


Background
information
for
EPA
Staff

Checklist
for
Administering
Questionnaires
and
Using
Results

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
CAG
Meeting
(
participant
assessment)


Questionnaire
for
Feedback
on
the
Effectiveness
of
a
CAG
Meeting
(
EPA/
contractor
assessment)


Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
CAG
Process
(
participant
assessment)


Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
CAG
Process
(
EPA/
contractor
assessment)

For
questions
contact:
Patricia
Bonner
(
bonner.
patricia@
epa.
gov)/
202­
566­
2204
DRAFT:
Last
Updated
6/
28/
04
30
Assessing
EPA's
Community
Advisory
Groups
Background
Information
for
EPA
Staff
Introduction
Enclosed
are
four
questionnaires
for
EPA
staff
to
use
in
generating
feedback
on
the
effectiveness
of
EPA­
supported
(
and
other)
community
advisory
groups
(
CAGs).
The
questionnaires
are
a
component
of
the
evaluation
section
in
the
Framework
for
Implementing
EPA's
Public
Involvement
Policy.
The
questionnaires
are
for
people
who
regularly
attend
CAG
meetings
and
for
EPA
staff
responsible
for
coordinating
the
work
of
a
CAG.
The
first
and
second
questionnaires
(
one
for
CAG
participants
and
one
for
EPA/
contractor
staff)
focus
on
the
effectiveness
of
specific
CAG
meetings.
The
third
questionnaire,
which
should
be
administered
once
or
twice
a
year
as
needed,
deals
with
CAG
participant
perspectives
on
the
CAG's
success.
The
fourth
questionnaire,
which
should
be
administered
once
or
twice
a
year
as
needed,
deals
with
Agency/
contractor
perspectives
on
the
CAG's
success.

What
is
a
community
advisory
group?
A
CAG
is
made
up
of
diverse
representatives
from
a
community.
Its
purpose
is
to
provide
a
public
forum
for
discussing
the
interests
and
concerns
the
community
has
about
some
project
or
set
of
actions.
CAG
members
meet
on
a
regular
basis
to
learn
about
project
plans
and
activities
and
to
offer
ideas
and
suggestions
to
the
project
sponsor.
A
CAG
may
be
selffacilitated
or
use
an
outside
professional.
It
is
not
EPA
sponsored
and
it
is
not
subject
to
the
Federal
Advisory
Committee
Act
procedures.

How
can
the
CAG
questionnaires
be
useful
to
you?

Use
of
the
questionnaires
should
make
the
process
of
gathering
feedback
for
your
public
involvement
activity
easier
for
you
to
implement
directly,
without
spending
contract
dollars.
These
questionnaires
provide
an
easy
way
to
get
useful
feedback
from
participants
about
their
CAG
experience.
This
feedback
should
help
you
to
better
understand
whether
a
particular
CAG
is
working
out
well,
whether
you
are
missing
opportunities
for
improvement,
or
whether
you
need
to
address
certain
problems.
Once
you
have
summarized
the
data,
Agency
staff
can
assess
whether
they
met
goals,
make
modifications,
and
compare
progress
over
time.

When
should
I
use
the
questionnaires?
You
can
use
the
first
and
second
questionnaires
after
every
meeting
or
every
couple
of
meetings.
Use
the
third
and
fourth
questionnaires
once
or
twice
a
year,
depending
on
the
level
of
activity
of
the
CAG.
The
questionnaires
may
also
be
helpful
in
situations
where
a
particular
problem
hinders
the
successful
working
of
the
group.
It
is
ultimately
up
to
you,
however,
to
decide
when
to
use
them.
The
key
is
to
use
the
questionnaires
to
the
extent
that
they
provide
you
with
valuable
information
and
improve
your
ability
to
design
and
implement
more
effective
listening
sessions
in
the
future.

Are
the
questionnaires
in
compliance
with
the
Paperwork
Reduction
Act?

Not
yet.
The
questionnaires
will
be
cleared
with
the
Office
of
Management
and
Budget
(
OMB)
31
under
ICR
#
XXXXX,
OMB
Control
No:
2010­
XXXX.
This
clearance
will
allow
EPA
to
collect
information
(
via
the
questionnaires)
from
more
than
nine
non­
federal
entities.

Who
designed
these
questionnaires?
The
Evaluation
Task
Group
of
EPA's
Public
Involvement
Workgroup
designed
these
questionnaires
after
consulting
with
EPA
staff
who
are
regularly
involved
in
working
with
Community
Advisory
Groups.

What
should
Ido
when
preparing
to
administer
a
questionnaire?
See
"
Checklist
for
Administering
the
Questionnaire."

What
do
I
do
with
the
questionnaire
data
once
the
forms
are
returned?

You
can
collect,
collate
and
analyze
the
data
from
the
forms
using
a
pre­
formatted
Lotus
1­
2­
3
spreadsheet
program.
In
addition,
the
spreadsheet
program
allows
you
to
summarize
data
over
a
period
of
time,
making
it
easy
to
generate
annual
summary
reports
for
your
management.
More
information
on
how
to
use
the
program
is
available
in
the
"
Checklist
for
Administering
the
Questionnaire
and
Using
Results"
and
within
the
Lotus
1­
2­
3
spreadsheet
program.
Once
completed,
Evaluation
Task
Group
members
encourage
you
to
use
your
findings
and
send
your
completed
data
sheet
files
as
e­
mail
attachments
to:
bonner.
patricia@
epa.
gov.
If
you
do
so,
we
will
be
able
to
track
the
use
of
the
questionnaires,
and
learn
from
you
what
is
and
is
not
working
well,
and
gauge
whether
and
how
the
questionnaires
are
useful
to
your
efforts
to
improve
public
involvement
activities.
We
will
not
use
the
information
you
send
to
judge
the
strength
of
individual
public
involvement
efforts.

How
can
I
use
the
results?
The
completed
questionnaires
should
provide
you
with
information
that
can
help
you
design
improvements
to
activities
and
to
your
overall
involvement
process.
You
can
learn
what
worked
well,
what
participants
feel
is
important
and
what
should
change
to
better
accommodate
their
needs.
It
will
be
up
to
you
to
decide
how
much
data
you
need
to
prompt
changes.
While
it
is
prudent
to
be
cautious
about
making
changes
based
upon
a
small
amount
of
feedback
data,
you
may
find
a
number
of
the
respondents'
ideas
useful.
If
you
can
act
on
their
suggestions,
even
in
a
small
way,
participants
will
appreciate
that
you
listened.

How
should
I
communicate
the
results?

Use
the
performance
and
summary
reports
from
the
Lotus
1­
2­
3
spreadsheet
program
to
talk
with
managers,
staff,
and
CAG
members
about
the
effectiveness
of
CAG
meetings
and
CAGs
overall
and
where
changes
may
be
necessary.
Communicate
overall
results
to
CAG
participants
in
the
CAG
performance
report
or
other
follow­
up
materials
distributed
to
CAG
participants.
A
few
brief
sentences
describing
overall
what
participants
thought
of
the
CAG,
their
suggestions
for
improvement,
and
what
might
be
considered
to
improve
the
CAG
process
would
let
participants
know
that
you
valued
their
input.

What
is
EPA's
Public
Involvement
Policy
and
where
can
I
get
more
information?
EPA's
2003
Public
Involvement
Policy
underscores
EPA's
commitment
to
improving
its
practices
and
is
based
on
seven
steps
for
effective
public
involvement:
32
1.
Plan
and
budget
for
public
involvement
activities
2.
Identify
the
interested
and
affected
public
3.
Consider
providing
technical
or
financial
assistance
to
the
public
to
facilitate
involvement
4.
Provide
information
and
outreach
to
the
public
5.
Conduct
public
consultation
and
involvement
activities
6.
Review
and
use
input,
and
provide
feedback
to
the
public
7.
Evaluate
public
involvement
activities
The
Framework
for
Implementing
EPA's
Public
Involvement
Policy
urges
development
of
tools
to
improve
public
involvement
information
sharing,
training
and
evaluation
at
EPA.
This
questionnaire
and
the
database
program
are
examples
of
such
tools.
Cross­
agency
work
groups
developed
the
Policy
and
Framework
using
public
comments
and
ideas
gathered
through
a
two­
week,
Internet­
based
Dialogue
on
Public
Involvement
in
EPA
Decisions.
Copies
of
the
Policy
and
Framework
are
available
@
http://
www.
epa.
gov/
publicinvolvement/
public/
index.
htm.
5Note:
The
instructions
above
are
designed
primarily
for
questionnaires
that
seek
the
perspective
of
participants
about
the
effectiveness
of
a
single
event.
Depending
upon
the
type
of
questionnaire
you
wish
to
distribute
and
the
type
of
public
involvement
activity
you
wish
to
acquire
feedback
for,
you
will
need
to
tailor
the
steps
of
this
checklist
accordingly.

33
Assessing
EPA's
Public
Involvement
Activities
Checklist
for
Administering
the
Questionnaire
and
Using
Results5
Review
questionnaires.

Determine
whether
a
questionnaire
should
be
administered.

Determine
what
questionnaire(
s)
should
be
administered.

Identify
one
person
to
be
responsible
for
coordinating
the
overall
dissemination
and
collection
of
the
questionnaire
and
to
input
the
data
into
the
Lotus
1­
2­
3
data
input,
analysis,
and
reporting
program
once
the
questionnaires
are
collected.

Make
plans
to
include
the
questionnaire
feedback
process
in
the
agenda
of
your
public
involvement
event.
Specifically,
plan
to
set
aside
a
few
minutes
before
the
conclusion
of
your
public
involvement
event
and
have
participants
fill
out
their
questionnaires.
You
should
also
plan
to
make
an
announcement
at
the
beginning
of
your
public
involvement
event
explaining
the
questionnaire
feedback
process
and
how
feedback
is
needed
from
all
participants
in
order
to
develop
a
more
comprehensive
understanding
of
the
overall
effectiveness
of
the
event
and
how
best
to
make
improvements.
Feedback
is
needed
from
participants
who
leave
early
and
those
who
stay
to
the
end.
If
you
don't
build
the
questionnaire
feedback
process
directly
into
your
event,
and
instead
wait
to
the
very
end,
chances
are
few
participants
will
take
the
time
to
fill
out
their
questionnaires.
Participants
may
suggest
that
they
would
prefer
to
fill
out
their
questionnaires
at
home
or
work.
While
this
option
is
appealing
for
participants,
chances
are
only
a
few
will
take
the
time
to
actually
do
this.
The
best
chance
for
good
feedback
is
when
participants
are
actually
at
the
event.
Take
the
time
to
plan
for
this!

Estimate
the
number
of
participants
who
will
be
attending
the
public
involvement
event
for
which
questionnaires
will
be
distributed.

Make
appropriate
number
of
copies
of
the
questionnaire.

Bring
questionnaire
copies
and
extra
pens
or
pencils
to
the
public
involvement
event.

Identify
official
who
will
be
responsible
for
physically
collecting
the
questionnaires.

Clearly
identify
areas
where
participants
should
return
their
questionnaires.

Hand
out
questionnaires
with
other
public
involvement
activity
materials
either
at
the
beginning
of
the
event,
or
at
a
designated
time
during
the
event.
6Note:
Separate
Lotus
1­
2­
3
programs
have
been
developed
that
match
most
types
of
questionnaires
available
(
e.
g.,
one
for
single
event
effectiveness
for
participants,
one
for
overall
effectiveness
for
participants,
etc.).
Be
sure
to
use
the
appropriate
Lotus
1­
2­
3
program
to
input
your
data.
The
only
type
of
questionnaire
for
which
a
spreadsheet
program
has
not
been
developed
is
for
the
questionnaire
dealing
with
post­
event
follow­
up.
However,
if
you
need
a
spreadsheet
program
for
this
questionnaire,
contact
Pat
Bonner
(
bonner.
patricia@
epa.
gov).

34
Make
it
very
clear
to
participants
to
whom,
or
to
where,
they
should
return
their
questionnaire.
Urge
participants
to
fill
out
their
questionnaire
even
if
they
need
to
leave
early.

Identify
the
total
number
of
participants
attending
(
e.
g.,
the
total
number
of
persons
who
could
possibly
fill
out
questionnaires).

Collect
questionnaires.

Input
data
into
the
Lotus
1­
2­
3
data
input,
analysis,
and
reporting
program
within
one
month
of
session
and
share
results.

Detach
the
Lotus
1­
2­
3
program
onto
your
computer.
6
Open
the
Lotus
1­
2­
3
program.

Click
on
the
"
Getting
Started"
tab
and
follow
the
instructions.

Review
performance
report
and
identify
whether
certain
changes
should
be
considered
in
preparation
for
the
next
public
involvement
event.

Use
the
Lotus
1­
2­
3
performance
reports
to
talk
with
managers
and
staff
about
the
effectiveness
of
your
public
involvement
activity
and
where
changes
may
be
necessary.
Communicate
overall
results
and
what
changes
you
intend
to
make
to
participants
who
attended
your
last
public
involvement
event
in
the
public
involvement
event
summary
report
or
other
follow­
up
materials
distributed
to
participants.
A
few
brief
sentences
describing
overall
what
participants
thought
of
the
event,
their
suggestions
for
improvement,
and
what
you
will
do
to
improve
the
next
event
would
let
participants
know
that
you
valued
their
input.

Sharing
data...
EPA's
Public
Involvement
Policy
Implementation
Staff
strongly
encourages
you
to
send
your
completed
Lotus
1­
2­
3
program
files
as
e­
mail
attachments
to:
bonner.
patricia@
epa.
gov.
If
you
do
so,
Implementation
Staff
will
be
able
to
track
the
use
of
the
questionnaires,
and
learn
from
you
what
is
and
is
not
working
well,
and
gauge
whether
and
how
the
questionnaire
was
useful
to
your
efforts
to
improve
public
involvement
activities.
We
will
not
use
the
information
you
send
to
judge
the
strength
of
individual
public
involvement
efforts.

When
you
conduct
another
public
involvement
event,
repeat
the
same
steps
listed
above.
However,
after
doing
this,
be
sure
to
go
to
the
same
Lotus
1­
2­
3
spreadsheet
and
follow
the
steps
required
to
link
the
data
from
your
first
public
involvement
event
to
your
second
in
order
to
summarize
and
compare
your
data
over
a
period
of
time.
35
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Community
Advisory
Group
Meeting
Participant
Assessment
Background
This
questionnaire
is
designed
to
help
Agency
staff
better
understand
what
worked
well
and
what
improvements
to
consider
implementing
before
holding
future
Community
Advisory
Group
(
CAG)
meetings.
Once
the
data
are
summarized,
Agency
staff
can
assess
whether
goals
for
success
were
met,
make
modifications
if
necessary,
and
compare
progress
over
time.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!
36
Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
Don'
t
kno
w
Meeting
objectives
were
a)
clear
1
2
3
4
5
6
7
b)
achieved
1
2
3
4
5
6
7
Meeting
was
run
effectively.
1
2
3
4
5
6
7
Sufficient
technical
information
was
available.
1
2
3
4
5
6
7
Participants
were
able
to
influence
meeting
outcome.
1
2
3
4
5
6
7
Participation
in
meeting
was
worthwhile.
1
2
3
4
5
6
7
EPA
participation
was
useful.
1
2
3
4
5
6
7
Meeting
time
was
convenient.
1
2
3
4
5
6
7
Meeting
location
was
comfortable.
1
2
3
4
5
6
7
Meeting
length
was
appropriate.
1
2
3
4
5
6
7
Do
you
have
any
comments
or
suggestions
for
improving
the
CAG
meetings?

Please
indicate
which
group
you
represent:

neighborhood
group
local
non­
profit
organization
(
not
including
local
neighborhood
non­
profit
groups)
37
national
environmental
organization
business
or
industry
or
trade
association
state
or
tribal
or
local
government
federal
government
other
(
please
specify)

no
organizational
affiliation
(
e.
g.,
interested
local
resident)

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
38
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Community
Advisory
Group
Meeting
EPA/
Contractor
Assessment
Background
This
questionnaire
is
designed
to
obtain
an
EPA/
contractor
perspective
on
the
effectiveness
of
a
Community
Advisory
Group
(
CAG)
meeting
or
set
of
meetings.
Agency
staff
will
use
this
information
to
better
understand
what
is
working
well
and
what
improvements
could
make
future
CAG
meetings
more
successful.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
Don'
t
kno
w
Meeting
objectives
were
a)
clear
1
2
3
4
5
6
7
b)
achieved
1
2
3
4
5
6
7
Meeting
was
run
effectively.
1
2
3
4
5
6
7
Sufficient
technical
information
was
available.
1
2
3
4
5
6
7
Participants
were
able
to
influence
meeting
outcome.
1
2
3
4
5
6
7
EPA
participation
was
useful.
1
2
3
4
5
6
7
Meeting
time
was
convenient.
1
2
3
4
5
6
7
Meeting
location
was
comfortable.
1
2
3
4
5
6
7
Meeting
length
was
appropriate.
1
2
3
4
5
6
7
Please
explain
your
ranking
of
the
previous
statement
about
success:
39
Please
provide
any
comments
or
suggestions
for
improving
the
CAG
meetings:

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
39
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
Community
Advisory
Group
Process
Participant
Assessment
Background
This
questionnaire
is
designed
to
help
Agency
staff
better
understand
the
overall
effectiveness
of
a
Community
Advisory
Group
(
CAG)
process.
Agency
staff
will
use
this
information
to
better
understand
what
is
working
well
and
what
improvements
could
be
make
this
CAG
process
more
successful.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
Don'
t
kno
w
Committee
membership
is
inclusive.
1
2
3
4
5
6
7
All
committee
members
are
contributing
to
and
sharing
in
the
work.
1
2
3
4
5
6
7
Committee
is
adequately
connected
to
the
community.
1
2
3
4
5
6
7
Meeting
frequency
is
appropriate.
1
2
3
4
5
6
7
Committee
has
a
good
process
for
reaching
closure
on
issues.
1
2
3
4
5
6
7
Committee
is
fulfilling
its
chartered
mission.
1
2
3
4
5
6
7
Committee
is
producing
useful
action
agenda
and
recommendations.
1
2
3
4
5
6
7
Committee
is
influencing
EPA
decisions.
1
2
3
4
5
6
7
Personal
participation
is
worthwhile.
1
2
3
4
5
6
7
Please
provide
any
comments
or
suggestions
for
improving
this
CAG
process:
40
Please
indicate
which
group
you
represent:

neighborhood
group
local
non­
profit
organization
(
not
including
local
neighborhood
non­
profit
groups)

national
environmental
organization
business
or
industry
or
trade
association
state
or
tribal
or
local
government
federal
government
other
(
please
specify)

no
organizational
affiliation
(
e.
g.,
interested
local
resident)

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
41
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
Community
Advisory
Group
Process
EPA/
Contractor
Assessment
Background
This
questionnaire
is
designed
to
obtain
an
EPA/
contractor
perspective
on
the
overall
effectiveness
of
a
CAG
process.
Agency
staff
will
use
this
information
to
better
understand
what
is
working
well
and
what
improvements
could
be
make
this
CAG
process
more
successful.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
Don'
t
kno
w
CAG
is
inclusive.
1
2
3
4
5
6
7
CAG
is
well­
facilitated.
1
2
3
4
5
6
7
CAG
has
two
way
communication
with
community.
1
2
3
4
5
6
7
CAG
is
producing
useful
advice.
1
2
3
4
5
6
7
EPA
is
providing
adequate
technical
information
and
support.
1
2
3
4
5
6
7
EPA
is
providing
appropriate
financial
and
other
assistance.
1
2
3
4
5
6
7
Please
explain
your
ranking
of
the
previous
statement
about
success:

Please
provide
any
comments
or
suggestions
for
improving
this
CAG
process:

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
42
Assessing
EPA's
Public
Meetings
A
User's
Guide
with
Questionnaires
Enclosed
are
information
and
tools
for
EPA
staff
to
use
in
generating
feedback
on
the
effectiveness
of
public
meetings
and
the
effectiveness
of
follow­
up
after
them.
Specifically,
this
user
guide
contains
the
following:


Background
information
for
EPA
Staff

Checklist
for
Administering
Questionnaires
and
Using
Results

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Public
Meeting
(
participant
assessment)


Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Public
Meeting
(
EPA/
contractor
assessment)


Questionnaire
for
Feedback
on
the
Effectiveness
of
Public
Meeting
Follow­
Up
(
participant
assessment)

For
questions
contact:
Patricia
Bonner
(
bonner.
patricia@
epa.
gov)/
202­
566­
2204
DRAFT:
Last
Updated
6/
28/
04
Draft
Draft
Draft
43
Assessing
EPA's
Public
Meetings
Background
Information
for
EPA
Staff
Introduction
Enclosed
are
three
questionnaires
for
EPA
staff
to
use
in
generating
feedback
on
the
effectiveness
of
public
meetings.
The
questionnaires
are
a
component
of
the
evaluation
section
in
the
Framework
for
Implementing
EPA's
Public
Involvement
Policy.
The
questionnaires
are
for
people
who
attend
a
public
meeting,
for
EPA
staff/
contractors
responsible
for
coordinating
the
work
of
a
public
meeting,
and
for
public
meeting
participants
who
receive
follow­
up
information
following
the
public
meeting.
The
first
questionnaire
focuses
on
the
effectiveness
of
a
public
meeting.
The
second
questionnaire
focuses
on
EPA/
contractor
perspectives
about
the
success
of
the
same
public
meeting.
The
third
questionnaire
focuses
on
participant
perspectives
of
the
quality
of
the
follow­
up
information
received
after
the
same
public
meeting.

What
is
a
public
meeting?
A
public
meeting
is
a
forum
that
allows
communities
and
the
affected
public
to
address
EPA
officials
directly
with
their
concerns.
Public
meetings
promote
two­
way
communications
and
a
means
for
all
interested
parties
to
ask
questions
and
raise
issues
in
an
informal
setting.
The
purpose
of
a
public
meeting
is
to
share
information
and
discuss
issues,
not
to
make
decisions.
Public
meetings
should:
enable
the
public
and
the
Agency
to
share
data,
ideas,
advice
and
concerns;
allow
EPA
to
hear
from
a
wide
range
of
interested
an
affected
parties;
and
compile
a
knowledge
base
of
the
public's
various
interests,
ideas
and
needs,
helping
the
Agency
to
better
understand
and
consider
the
issues
related
to
a
particular
decision.

How
can
the
public
meeting
questionnaires
be
useful
to
you?

Use
of
the
questionnaires
should
make
the
process
of
gathering
feedback
for
your
public
involvement
activity
easier
for
you
to
implement
directly,
without
spending
contract
dollars.
The
questionnaires
provide
an
easy
way
to
get
useful
feedback
from
participants
and
EPA
about
the
public
meetings
they
participated
in.
This
feedback
should
help
you
to
better
understand
whether
a
particular
public
meeting
worked
well,
whether
there
are
opportunities
for
improvement,
or
whether
certain
problems
need
to
be
addressed.
Once
you
have
summarized
the
data,
Agency
staff
can
assess
whether
they
met
goals,
make
modifications,
and
compare
progress
over
time.

When
should
I
use
the
questionnaires?
You
can
use
the
first
questionnaire
(
for
public
meeting
participants)
towards
the
end
of
a
public
meeting.
You
can
use
the
second
questionnaire
(
for
EPA
staff
or
their
contractors
who
assisted
in
planning
or
leading
a
public
meeting)
after
the
same
public
meeting.
You
can
use
the
third
questionnaire
(
for
public
meeting
participants)
when
sending
out
follow­
up
information
to
all
public
meeting
participants.
It
is
ultimately
up
to
you,
however,
to
decide
when
to
use
them.
The
key
is
to
use
the
questionnaires
to
the
extent
that
they
provide
you
with
valuable
information
and
improve
your
ability
to
design
and
implement
more
effective
public
meetings
in
the
future.
Draft
Draft
Draft
44
Are
the
questionnaires
in
compliance
with
the
Paperwork
Reduction
Act?
Not
yet.
The
questionnaires
will
be
cleared
with
the
Office
of
Management
and
Budget
(
OMB)
under
ICR
#
XXXXX,
OMB
Control
No:
2010­
XXXX.
This
clearance
will
allow
EPA
to
collect
information
(
via
the
questionnaires)
from
more
than
nine
non­
federal
entities.
Currently,
these
questionnaires
are
undergoing
a
pre­
test.

Who
designed
these
questionnaires?

The
Evaluation
Task
Group
of
EPA's
Public
Involvement
Workgroup
designed
these
questionnaires
after
consulting
with
EPA
staff
who
are
regularly
involved
in
the
designing
and
holding
of
public
meetings.

What
should
I
do
when
preparing
to
administer
a
questionnaire?

See
"
Checklist
for
Administering
the
Questionnaire."

What
do
I
do
with
the
questionnaire
data
once
the
forms
are
returned?
You
can
collect,
collate
and
analyze
the
data
from
the
forms
using
a
pre­
formatted
Lotus
1­
2­
3
spreadsheet
program.
In
addition,
the
spreadsheet
program
allows
you
to
summarize
data
over
a
period
of
time,
making
it
easy
to
generate
annual
summary
reports
for
your
management.
More
information
on
how
to
use
the
program
is
available
in
the
"
Checklist
for
Administering
the
Questionnaire
and
Using
Results"
and
within
the
Lotus
1­
2­
3
spreadsheet
program.
Once
completed,
Evaluation
Task
Group
members
encourage
you
to
use
your
findings
and
send
your
completed
data
sheet
files
as
e­
mail
attachments
to:
bonner.
patricia@
epa.
gov.
If
you
do
so,
we
will
be
able
to
track
the
use
of
the
questionnaires,
and
learn
from
you
what
is
and
is
not
working
well,
and
gauge
whether
and
how
the
questionnaires
are
useful
to
your
efforts
to
improve
public
involvement
activities.
We
will
not
use
the
information
you
send
to
judge
the
strength
of
individual
public
involvement
efforts.

How
can
I
use
the
results?

The
completed
questionnaires
should
provide
you
with
information
that
can
help
you
design
improvements
to
activities
and
to
your
overall
involvement
process.
You
can
learn
what
worked
well,
what
participants
feel
is
important
and
what
should
change
to
better
accommodate
their
needs.
It
will
be
up
to
you
to
decide
how
much
data
you
need
to
prompt
changes.
While
it
is
prudent
to
be
cautious
about
making
changes
based
upon
a
small
amount
of
feedback
data,
you
may
find
a
number
of
the
respondents'
ideas
useful.
If
you
can
act
on
their
suggestions,
even
in
a
small
way,
participants
will
appreciate
that
you
listened.

How
should
I
communicate
the
results?
Use
the
performance
and
summary
reports
from
the
Lotus
1­
2­
3
spreadsheet
program
to
talk
with
managers
and
staff
about
the
effectiveness
of
the
Public
Meetings
and
where
changes
may
be
necessary.
Communicate
overall
results
and
what
changes
you
intend
to
make
to
public
meeting
participants
in
the
public
meeting
summary
report
or
other
follow­
up
materials
distributed
to
session
participants.
A
few
brief
sentences
describing
overall
what
participants
thought
of
the
session,
their
suggestions
for
improvement,
and
what
you
will
do
to
improve
the
next
public
meeting
would
let
participants
know
that
you
valued
their
input.
Draft
Draft
Draft
45
What
is
EPA's
Public
Involvement
Policy
and
where
can
I
get
more
information?

EPA's
2003
Public
Involvement
Policy
underscores
EPA's
commitment
to
improving
its
practices
and
is
based
on
seven
steps
for
effective
public
involvement:
1.
Plan
and
budget
for
public
involvement
activities
2.
Identify
the
interested
and
affected
public
3.
Consider
providing
technical
or
financial
assistance
to
the
public
to
facilitate
involvement
4.
Provide
information
and
outreach
to
the
public
5.
Conduct
public
consultation
and
involvement
activities
6.
Review
and
use
input,
and
provide
feedback
to
the
public
7.
Evaluate
public
involvement
activities
The
Framework
for
Implementing
EPA's
Public
Involvement
Policy
urges
development
of
tools
to
improve
public
involvement
information
sharing,
training
and
evaluation
at
EPA.
This
questionnaire
and
the
database
program
are
examples
of
such
tools.
Cross­
agency
work
groups
developed
the
Policy
and
Framework
using
public
comments
and
ideas
gathered
through
a
two­
week,
Internet­
based
Dialogue
on
Public
Involvement
in
EPA
Decisions.
Copies
of
the
Policy
and
Framework
are
available
@
http://
www.
epa.
gov/
publicinvolvement/
public/
index.
htm.
Draft
Draft
Draft
7Note:
The
instructions
above
are
designed
primarily
for
questionnaires
that
seek
the
perspective
of
participants
about
the
effectiveness
of
a
single
event.
Depending
upon
the
type
of
questionnaire
you
wish
to
distribute
and
the
type
of
public
involvement
activity
you
wish
to
acquire
feedback
for,
you
will
need
to
tailor
the
steps
of
this
checklist
accordingly.

46
Assessing
EPA's
Public
Involvement
Activities
Checklist
for
Administering
the
Questionnaire
and
Using
Results7
Review
questionnaire.

Determine
whether
questionnaire
should
be
administered.

Determine
what
questionnaire(
s)
should
be
administered.

Identify
one
person
to
be
responsible
for
coordinating
the
overall
dissemination
and
collection
of
the
questionnaire
and
to
input
the
data
into
the
Lotus
1­
2­
3
data
input,
analysis,
and
reporting
program
once
the
questionnaires
are
collected.

Make
plans
to
include
the
questionnaire
feedback
process
in
the
agenda
of
your
public
involvement
event.
Specifically,
plan
to
set
aside
a
few
minutes
before
the
conclusion
of
your
public
involvement
event
and
have
participants
fill
out
their
questionnaires.
You
should
also
plan
to
make
an
announcement
at
the
beginning
of
your
public
involvement
event
explaining
the
questionnaire
feedback
process
and
how
feedback
is
needed
from
all
participants
in
order
to
develop
a
more
comprehensive
understanding
of
the
overall
effectiveness
of
the
event
and
how
best
to
make
improvements.
Feedback
is
needed
from
participants
who
leave
early
and
those
who
stay
to
the
end.
If
you
don't
build
the
questionnaire
feedback
process
directly
into
your
event,
and
instead
wait
to
the
very
end,
chances
are
few
participants
will
take
the
time
to
fill
out
their
questionnaires.
Participants
may
suggest
that
they
would
prefer
to
fill
out
their
questionnaires
at
home
or
work.
While
this
option
is
appealing
for
participants,
chances
are
only
a
few
will
take
the
time
to
actually
do
this.
The
best
chance
for
good
feedback
is
when
participants
are
actually
at
the
event.
Take
the
time
to
plan
for
this!

Estimate
the
number
of
participants
who
will
be
attending
the
public
involvement
event
for
which
questionnaires
will
be
distributed.

Make
appropriate
number
of
copies
of
the
questionnaire.

Bring
questionnaire
copies
and
extra
pens
or
pencils
to
the
public
involvement
event.

Identify
official
who
will
be
responsible
for
physically
collecting
the
questionnaires.

Clearly
identify
areas
where
participants
should
return
their
questionnaires.

Hand
out
questionnaires
with
other
public
involvement
activity
materials
either
at
the
beginning
of
the
event,
or
at
a
designated
time
during
the
event.
Draft
Draft
Draft
8Note:
Separate
Lotus
1­
2­
3
programs
have
been
developed
that
match
most
types
of
questionnaires
available
(
e.
g.,
one
for
single
event
effectiveness
for
participants,
one
for
overall
effectiveness
for
participants,
etc.).
Be
sure
to
use
the
appropriate
Lotus
1­
2­
3
program
to
input
your
data.
The
only
type
of
questionnaire
for
which
a
spreadsheet
program
has
not
been
developed
is
for
the
questionnaire
dealing
with
post­
event
follow­
up.
However,
if
you
need
a
spreadsheet
program
for
this
questionnaire,
contact
Pat
Bonner
(
bonner.
patricia@
epa.
gov).

47
Make
it
very
clear
to
participants
to
whom,
or
to
where,
they
should
return
their
questionnaire.
Urge
participants
to
fill
out
their
questionnaire
even
if
they
need
to
leave
early.

Identify
the
total
number
of
participants
attending
(
e.
g.,
the
total
number
of
persons
who
could
possibly
fill
out
questionnaires).

Collect
questionnaires.

Input
data
into
the
Lotus
1­
2­
3
data
input,
analysis,
and
reporting
program
within
one
month
of
public
involvement
event
and
share
results.

Detach
the
Lotus
1­
2­
3
program
onto
your
computer.
8
Open
the
Lotus
1­
2­
3
program.

Click
on
the
"
Getting
Started"
tab
and
follow
the
instructions.

Review
performance
report
and
identify
whether
certain
changes
should
be
considered
in
preparation
for
the
next
public
involvement
event.

Use
the
Lotus
1­
2­
3
performance
reports
to
talk
with
managers
and
staff
about
the
effectiveness
of
your
public
involvement
activity
and
where
changes
may
be
necessary.
Communicate
overall
results
and
what
changes
you
intend
to
make
to
participants
who
attended
your
last
public
involvement
event
in
the
public
involvement
event
summary
report
or
other
follow­
up
materials
distributed
to
participants.
A
few
brief
sentences
describing
overall
what
participants
thought
of
the
event,
their
suggestions
for
improvement,
and
what
you
will
do
to
improve
the
next
event
would
let
participants
know
that
you
valued
their
input.

Sharing
data...
EPA's
Public
Involvement
Policy
Implementation
Staff
strongly
encourages
you
to
send
your
completed
Lotus
1­
2­
3
program
files
as
e­
mail
attachments
to:
bonner.
patricia@
epa.
gov.
If
you
do
so,
Implementation
Staff
will
be
able
to
track
the
use
of
the
questionnaires,
and
learn
from
you
what
is
and
is
not
working
well,
and
gauge
whether
and
how
the
questionnaire
was
useful
to
your
efforts
to
improve
public
involvement
activities.
We
will
not
use
the
information
you
send
to
judge
the
strength
of
individual
public
involvement
efforts.

When
you
conduct
another
public
involvement
event,
repeat
the
same
steps
listed
above.
However,
after
doing
this,
be
sure
to
go
to
the
same
Lotus
1­
2­
3
spreadsheet
and
follow
the
steps
required
to
link
the
data
from
your
first
public
involvement
event
to
your
second
in
order
to
summarize
and
compare
your
data
over
a
period
of
time.
Draft
Draft
Draft
48
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Public
meeting
Participant
Assessment
Background
This
questionnaire
is
designed
to
help
Agency
staff
better
understand
what
worked
well
and
what
improvements
to
consider
implementing
before
holding
future
Public
Meetings.
Once
the
data
are
summarized,
Agency
staff
can
assess
whether
goals
for
success
were
met,
make
modifications
if
necessary,
and
compare
progress
over
time.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.

1a.
Pre­
meeting
publicity
availability
Please
indicate
how
you
heard
about
the
public
meeting
(
check
all
that
apply):
radio
announcements
television
announcements
newspapers
ads
flyers
in
store
windows/
bulletin
boards
website
information
Other
(
please
specify)

Was
the
information
about
the
public
meeting
made
available
in
locally
used
languages?
Yes
No__

If
you
received
a
pre­
meeting
notice
via
regular
or
electronic
mail,
how
much
in
advance
did
the
notice
arrive?
___
a
few
days
one
week
2
weeks
2­
4
weeks
more
than
1
month___
other
(
please
specify)

Please
suggest
any
additional
ways
you
think
EPA
should
publicize
future
public
meetings:

Please
explain
what
you
learned
about
the
public
meeting
or
issue
that
made
you
decide
to
Draft
Draft
Draft
49
attend:_____________________________________________________________________

1b.
Pre­
meeting
background
information
Was
anything
missing
from
the
pre­
meeting
background
information?
Yes___
No___
If
yes,
please
specify:

Did
you
call
to
ask
for
specific
information?
Yes
No
If
yes,
what
did
you
ask
for?

2.
Meeting
logistics
Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
Str
on
gly
agr
ee
Do
n't
kno
w
Pre­
meeting
invitation
arrived
early
enough
for
me
to
plan
attending.
1
2
3
4
5
6
7
Location
was
convenient.
1
2
3
4
5
6
7
Public
transportation
was
close
by.
1
2
3
4
5
6
7
Facility
was
comfortable
for
participants.
1
2
3
4
5
6
7
Time
during
which
meeting
was
held
was
convenient.
1
2
3
4
5
6
7
The
length
of
the
meeting
was
about
right.
1
2
3
4
5
6
7
Translators
were
available
(
if
needed).
1
2
3
4
5
6
7
Signers
for
hearing
impaired
were
available
(
if
needed).
1
2
3
4
5
6
7
Provisions
were
made
for
handicap
access.
1
2
3
4
5
6
7
Facility
was
well
equipped
for
all
planned
activities
(
tables,
work
space,
break­
outs,
supplies,
etc).
1
2
3
4
5
6
7
Refreshments
are
important
for
a
successful
public
meeting.
1
2
3
4
5
6
7
3.
Conduct
of
meeting
Draft
Draft
Draft
50
Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
Str
on
gly
agr
ee
Do
n't
kno
w
Greeters
made
participants
feel
welcome.
1
2
3
4
5
6
7
The
registration
process
was
efficient.
1
2
3
4
5
6
7
I
understood
the
purpose
of
the
meeting.
1
2
3
4
5
6
7
I
understood
how
the
meeting
would
be
conducted.
1
2
3
4
5
6
7
Meeting
planners
focused
on
the
right
1
2
3
4
5
6
7
I
was
comfortable
with
the
meeting
format.
1
2
3
4
5
6
7
The
format
gave
all
ample
opportunity
to
be
1
2
3
4
5
6
7
My
ideas
were
heard.
1
2
3
4
5
6
7
There
was
good
interaction
among
participants.
1
2
3
4
5
6
7
It
was
easy
to
sign
up
for
follow­
up
information.
1
2
3
4
5
6
7
There
was
a
good
mix
of
viewpoints.
1
2
3
4
5
6
7
I
understand
the
"
next
steps"
in
the
process.
1
2
3
4
5
6
7
Action
items
were
documented
(
flip
charts).
1
2
3
4
5
6
7
Time­
frames
and
accountable
persons
were
listed
1
2
3
4
5
6
7
Participants'
input
will
make
a
difference.
1
2
3
4
5
6
7
Please
suggest
how
EPA
can
improve
the
next
public
meeting:

Please
indicate
which
group
you
represent:

neighborhood
group
local
non­
profit
organization
(
not
including
local
neighborhood
non­
profit
groups)
Draft
Draft
Draft
51
national
environmental
organization
business
or
industry
or
trade
association
state
or
tribal
or
local
government
federal
government
other
(
please
specify)

no
organizational
affiliation
(
e.
g.,
interested
local
resident)

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
52
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Public
Meeting
EPA/
Contractor
Assessment
Background
This
questionnaire
is
designed
to
obtain
an
EPA/
contractor
perspective
on
the
effectiveness
of
a
Public
Meeting.
Agency
staff
will
use
this
information
to
better
understand
what
is
working
well
and
what
improvements
could
make
future
Public
Meetings
more
successful.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
Str
on
gly
agr
ee
Do
n't
kno
w
The
registration
process
was
efficient.
1
2
3
4
5
6
7
Participants
understood
the
purpose
of
the
meeting.
1
2
3
4
5
6
7
Participants
understood
how
the
meeting
would
be
conducted.
1
2
3
4
5
6
7
The
right
topics
were
addressed
during
the
meeting.
1
2
3
4
5
6
7
The
format
gave
all
ample
opportunity
to
be
1
2
3
4
5
6
7
There
was
good
interaction
among
participants.
1
2
3
4
5
6
7
The
number
of
people
that
participated
was
appropriate.
1
2
3
4
5
6
7
There
was
a
good
mix
of
viewpoints.
1
2
3
4
5
6
7
EPA
gained
new,
or
clarifying,
information,
that
will
enable
it
to
more
effectively
develop,
revise,
or
implement
rules
or
programs.
1
2
3
4
5
6
7
This
public
meeting
was
a
success.
1
2
3
4
5
6
7
Please
explain
your
ranking
of
the
previous
statement
about
success:
Draft
Draft
Draft
53
Please
suggest
how
EPA
can
improve
its
next
public
meeting:

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
54
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
Public
Meeting
Follow­
Up
Participants'
Assessment
Background
This
questionnaire
is
designed
to
help
Agency
staff
better
understand
what
worked
well
and
what
improvements
to
consider
before
implementing
future
Public
Meetings.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
Str
on
gly
agr
ee
Do
n't
kno
w
I
received
meeting
notes.
1
2
3
4
5
6
7
Notes
were
clearly
written.
1
2
3
4
5
6
7
Action
items
I
remember
were
documented.
1
2
3
4
5
6
7
Actions,
due
dates
and
accountable
persons
were
noted.
1
2
3
4
5
6
7
Issues
discussed
and
decisions
made
were
accurately
recorded.
1
2
3
4
5
6
7
I
received
the
information
I
requested
in
a
timely
manner
(
in
_______
days).
1
2
3
4
5
6
7
The
packet
I
received
was
complete.
1
2
3
4
5
6
7
Now
that
some
time
has
passed
since
the
public
meeting
you
may
have
thought
about
what
could
have
made
it
a
better
experience.
Please
suggest
how
you
think
EPA
can
improve
public
meetings
and
follow­
up
actions.

Please
indicate
which
group
you
represent:
Draft
Draft
Draft
55
neighborhood
group
local
non­
profit
organization
(
not
including
local
neighborhood
non­
profit
groups)

national
environmental
organization
business
or
industry
or
trade
association
state
or
tribal
or
local
government
federal
government
other
(
please
specify)

no
organizational
affiliation
(
e.
g.,
interested
local
resident)

Please
return
the
completed
questionnaire
in
the
pre­
addressed,
stamped
envelope
provided.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
56
Assessing
EPA's
Public
Hearings
A
User's
Guide
with
Questionnaires
Enclosed
are
information
and
tools
for
EPA
staff
to
use
in
generating
feedback
on
the
effectiveness
of
public
hearings
and
the
effectiveness
of
follow­
up
after
them.
Specifically,
this
user
guide
contains
the
following:


Background
information
for
EPA
Staff

Checklist
for
Administering
Questionnaires
and
Using
Results

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Public
Hearing
(
participant
assessment)


Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Public
Hearing
(
EPA/
contractor
assessment)


Questionnaire
for
Feedback
on
the
Effectiveness
of
Public
Hearing
Follow­
Up
(
participant
assessment)

For
questions
contact:
Patricia
Bonner
(
bonner.
patricia@
epa.
gov)/
202­
566­
2204
DRAFT:
Last
Updated
6/
28/
04
Draft
Draft
Draft
57
Assessing
EPA's
Public
Hearings
Background
Information
for
EPA
Staff
Introduction
Enclosed
are
three
questionnaires
for
EPA
staff
to
use
in
generating
feedback
on
the
effectiveness
of
public
hearings.
The
questionnaires
are
a
component
of
the
evaluation
section
in
the
Framework
for
Implementing
EPA's
Public
Involvement
Policy.
The
questionnaires
are
for
people
who
attend
a
public
hearing,
for
EPA/
contractor
staff
responsible
for
coordinating
the
work
of
a
public
hearing,
and
for
public
hearing
participants
who
receive
follow­
up
information
following
the
public
hearing.
The
first
questionnaire
focuses
on
participant
perspectives
of
the
effectiveness
of
a
public
hearing.
The
second
questionnaire
focuses
on
EPA/
contractor
perspectives
about
the
success
of
the
same
public
hearing.
The
third
questionnaire
focuses
on
participant
perspectives
of
the
quality
of
the
follow­
up
information
received
after
the
same
public
hearing.

What
is
a
public
hearing?
A
public
hearing
is
a
forum
to
enable
stakeholders
and
the
public
to
directly
address
EPA
officials
and
formally
state
their
concerns
for
an
official
record.
EPA
does
not
respond
to
public
comments
at
the
hearing.
Public
hearings
are
held:
when
requested
by
a
member
of
the
public
during
a
public
comment
period;
during
a
public
comment
period
following
the
issuance
of
a
draft
permit,
major
permit
modification,
or
at
the
selection
of
a
proposed
corrective
measure;
and
when
the
level
of
community
concern
warrants
a
formal
record
of
communication.
Typically,
a
public
hearing
will
have
a
well­
defined
procedure
for
presenting
comments,
formal
time
limits
on
statements,
a
hearing
official
or
officials
to
hear
the
comments
and
ask
clarifying
questions,
and
the
requirement
to
make
all
records
available
to
the
public.
The
purpose
of
a
public
hearing
is
to
enable
EPA
to
obtain
input
from
a
wide
range
of
interested
and
affected
parties
so
the
Agency
can
better
understand
and
consider
the
issues
relevant
to
a
particular
decision.

How
can
the
public
hearing
questionnaires
be
useful
to
you?

Use
of
the
questionnaires
should
make
the
process
of
gathering
feedback
for
your
public
involvement
activity
easier
for
you
to
implement
directly,
without
spending
contract
dollars.
The
questionnaires
provide
an
easy
way
to
get
useful
feedback
from
participants
and
EPA
about
the
public
hearings
they
participated
in.
This
feedback
should
help
you
to
better
understand
whether
a
particular
public
hearing
worked
well,
whether
there
are
opportunities
for
improvement,
or
whether
certain
problems
need
to
be
addressed.
Once
you
have
summarized
the
data,
Agency
staff
can
assess
whether
they
met
goals,
make
modifications,
and
compare
progress
over
time.

When
should
I
use
the
questionnaires?
You
can
use
the
first
questionnaire
(
for
public
hearing
participants)
towards
the
end
of
a
public
hearing.
You
can
use
the
second
questionnaire
(
for
EPA
staff
or
their
contractors
who
assisted
in
planning
or
leading
a
public
hearing)
after
the
same
public
hearing.
You
can
use
the
third
questionnaire
(
for
public
hearing
participants)
when
sending
out
follow­
up
information
to
all
public
hearing
participants.
It
is
ultimately
up
to
you,
however,
to
decide
Draft
Draft
Draft
58
when
to
use
them.
The
key
is
to
use
the
questionnaires
to
the
extent
that
they
provide
you
with
valuable
information
and
improve
your
ability
to
design
and
implement
more
effective
public
hearings
in
the
future.

Are
the
questionnaires
in
compliance
with
the
Paperwork
Reduction
Act?
Not
yet.
The
questionnaires
will
be
cleared
with
the
Office
of
Management
and
Budget
(
OMB)
under
ICR
#
XXXXX,
OMB
Control
No:
2010­
XXXX.
This
clearance
will
allow
EPA
to
collect
information
(
via
the
questionnaires)
from
more
than
nine
non­
federal
entities.
Currently,
these
questionnaires
are
undergoing
a
pre­
test.

Who
designed
these
questionnaires?

The
Evaluation
Task
Group
of
EPA's
Public
Involvement
Workgroup
designed
these
questionnaires
after
consulting
with
EPA
staff
who
are
regularly
involved
in
the
designing
and
holding
of
public
hearings.

What
should
I
do
when
preparing
to
administer
a
questionnaire?

See
"
Checklist
for
Administering
the
Questionnaire."

What
do
I
do
with
the
questionnaire
data
once
the
forms
are
returned?
You
can
collect,
collate
and
analyze
the
data
from
the
forms
using
a
pre­
formatted
Lotus
1­
2­
3
spreadsheet
program.
In
addition,
the
spreadsheet
program
allows
you
to
summarize
data
over
a
period
of
time,
making
it
easy
to
generate
annual
summary
reports
for
your
management.
More
information
on
how
to
use
the
program
is
available
in
the
"
Checklist
for
Administering
the
Questionnaire
and
Using
Results"
and
within
the
Lotus
1­
2­
3
spreadsheet
program.
Once
completed,
Evaluation
Task
Group
members
encourage
you
to
use
your
findings
and
send
your
completed
data
sheet
files
as
e­
mail
attachments
to:
bonner.
patricia@
epa.
gov.
If
you
do
so,
we
will
be
able
to
track
the
use
of
the
templates,
and
learn
from
you
what
is
and
is
not
working
well,
and
gauge
whether
and
how
the
questionnaires
are
useful
to
your
efforts
to
improve
public
involvement
activities.
We
will
not
use
the
information
you
send
to
judge
the
strength
of
individual
public
participation
efforts.

How
can
I
use
the
results?

The
completed
questionnaires
should
provide
you
with
information
that
can
help
you
design
improvements
to
activities
and
to
your
overall
involvement
process.
You
can
learn
what
worked
well,
what
participants
feel
is
important
and
what
should
change
to
better
accommodate
their
needs.
It
will
be
up
to
you
to
decide
how
much
data
you
need
to
prompt
changes.
While
it
is
prudent
to
be
cautious
about
making
changes
based
upon
a
small
amount
of
feedback
data,
you
may
find
a
number
of
the
respondents'
ideas
useful.
If
you
can
act
on
their
suggestions,
even
in
a
small
way,
participants
will
appreciate
that
you
listened.

How
should
I
communicate
the
results?
Use
the
performance
and
summary
reports
from
the
Lotus
1­
2­
3
spreadsheet
program
to
talk
with
managers
and
staff
about
the
effectiveness
of
the
Public
Hearings
and
where
changes
may
be
necessary.
Communicate
overall
results
and
what
changes
you
intend
to
make
to
Draft
Draft
Draft
59
public
hearing
participants
in
the
public
hearing
summary
report
or
other
follow­
up
materials
distributed
to
session
participants.
A
few
brief
sentences
describing
overall
what
participants
thought
of
the
session,
their
suggestions
for
improvement,
and
what
you
will
do
to
improve
the
next
public
hearing
would
let
participants
know
that
you
valued
their
input.

What
is
EPA's
Public
Involvement
Policy
and
where
can
I
get
more
information?
EPA's
2003
Public
Involvement
Policy
underscores
EPA's
commitment
to
improving
its
practices
and
is
based
on
seven
steps
for
effective
public
involvement:
1.
Plan
and
budget
for
public
involvement
activities
2.
Identify
the
interested
and
affected
public
3.
Consider
providing
technical
or
financial
assistance
to
the
public
to
facilitate
involvement
4.
Provide
information
and
outreach
to
the
public
5.
Conduct
public
consultation
and
involvement
activities
6.
Review
and
use
input,
and
provide
feedback
to
the
public
7.
Evaluate
public
involvement
activities
The
Framework
for
Implementing
EPA's
Public
Involvement
Policy
urges
development
of
tools
to
improve
public
involvement
information
sharing,
training
and
evaluation
at
EPA.
This
questionnaire
and
the
database
program
are
examples
of
such
tools.
Cross­
agency
work
groups
developed
the
Policy
and
Framework
using
public
comments
and
ideas
gathered
through
a
two­
week,
Internet­
based
Dialogue
on
Public
Involvement
in
EPA
Decisions.
Copies
of
the
Policy
and
Framework
are
available
@
http://
www.
epa.
gov/
publicinvolvement/
public/
index.
htm.
Draft
Draft
Draft
1Note:
The
instructions
above
are
designed
primarily
for
questionnaires
that
seek
the
perspective
of
participants
about
the
effectiveness
of
a
single
event.
Depending
upon
the
type
of
questionnaire
you
wish
to
distribute
and
the
type
of
public
involvement
activity
you
wish
to
acquire
feedback
for,
you
will
need
to
tailor
the
steps
of
this
checklist
accordingly.
60
Assessing
EPA's
Public
Involvement
Activities
Checklist
for
Administering
the
Questionnaire
and
Using
Results1
Review
questionnaire.

Determine
whether
questionnaire
should
be
administered.

Determine
what
questionnaire(
s)
should
be
administered.

Identify
one
person
to
be
responsible
for
coordinating
the
overall
dissemination
and
collection
of
the
questionnaire
and
to
input
the
data
into
the
Lotus
1­
2­
3
data
input,
analysis,
and
reporting
program
once
the
questionnaires
are
collected.

Make
plans
to
include
the
questionnaire
feedback
process
in
the
agenda
of
your
public
involvement
event.
Specifically,
plan
to
set
aside
a
few
minutes
before
the
conclusion
of
your
public
involvement
event
and
have
participants
fill
out
their
questionnaires.
You
should
also
plan
to
make
an
announcement
at
the
beginning
of
your
public
involvement
event
explaining
the
questionnaire
feedback
process
and
how
feedback
is
needed
from
all
participants
in
order
to
develop
a
more
comprehensive
understanding
of
the
overall
effectiveness
of
the
event
and
how
best
to
make
improvements.
Feedback
is
needed
from
participants
who
leave
early
and
those
who
stay
to
the
end.
If
you
don't
build
the
questionnaire
feedback
process
directly
into
your
event,
and
instead
wait
to
the
very
end,
chances
are
few
participants
will
take
the
time
to
fill
out
their
questionnaires.
Participants
may
suggest
that
they
would
prefer
to
fill
out
their
questionnaires
at
home
or
work.
While
this
option
is
appealing
for
participants,
chances
are
only
a
few
will
take
the
time
to
actually
do
this.
The
best
chance
for
good
feedback
is
when
participants
are
actually
at
the
event.
Take
the
time
to
plan
for
this!

Estimate
the
number
of
participants
who
will
be
attending
the
public
involvement
event
for
which
questionnaires
will
be
distributed.

Make
appropriate
number
of
copies
of
the
questionnaire.

Bring
questionnaire
copies
and
extra
pens
or
pencils
to
the
public
involvement
event.

Identify
official
who
will
be
responsible
for
physically
collecting
the
questionnaires.

Clearly
identify
areas
where
participants
should
return
their
questionnaires.

Hand
out
questionnaires
with
other
public
involvement
activity
materials
either
at
the
beginning
of
the
event,
or
at
a
designated
time
during
the
event.

Make
it
very
clear
to
participants
to
whom,
or
to
where,
they
should
return
their
questionnaire.
Urge
participants
to
fill
out
their
questionnaire
even
if
they
need
to
leave
early.
Draft
Draft
Draft
1Note:
Separate
Lotus
1­
2­
3
programs
have
been
developed
that
match
most
types
of
questionnaires
available
(
e.
g.,
one
for
single
event
effectiveness
for
participants,
one
for
overall
effectiveness
for
participants,
etc.).
Be
sure
to
use
the
appropriate
Lotus
1­
2­
3
program
to
input
your
data.
The
only
type
of
questionnaire
for
which
a
spreadsheet
program
has
not
been
developed
is
for
the
questionnaire
dealing
with
post­
event
follow­
up.
However,
if
you
need
a
spreadsheet
program
for
this
questionnaire,
contact
Pat
Bonner
(
bonner.
patricia@
epa.
gov).
61
Identify
the
total
number
of
participants
attending
(
e.
g.,
the
total
number
of
persons
who
could
possibly
fill
out
questionnaires).

Collect
questionnaires.

Input
data
into
the
Lotus
1­
2­
3
data
input,
analysis,
and
reporting
program
within
one
month
of
public
involvement
event
and
share
results.

Detach
the
Lotus
1­
2­
3
program
onto
your
computer.
1
Open
the
Lotus
1­
2­
3
program.

Click
on
the
"
Getting
Started"
tab
and
follow
the
instructions.

Review
performance
report
and
identify
whether
certain
changes
should
be
considered
in
preparation
for
the
next
public
involvement
event.

Use
the
Lotus
1­
2­
3
performance
reports
to
talk
with
managers
and
staff
about
the
effectiveness
of
your
public
involvement
activity
and
where
changes
may
be
necessary.
Communicate
overall
results
and
what
changes
you
intend
to
make
to
participants
who
attended
your
last
public
involvement
event
in
the
public
involvement
event
summary
report
or
other
follow­
up
materials
distributed
to
participants.
A
few
brief
sentences
describing
overall
what
participants
thought
of
the
event,
their
suggestions
for
improvement,
and
what
you
will
do
to
improve
the
next
event
would
let
participants
know
that
you
valued
their
input.

Sharing
data...
EPA's
Public
Involvement
Policy
Implementation
Staff
strongly
encourages
you
to
send
your
completed
Lotus
1­
2­
3
program
files
as
e­
mail
attachments
to:
bonner.
patricia@
epa.
gov.
If
you
do
so,
Implementation
Staff
will
be
able
to
track
the
use
of
the
questionnaires,
and
learn
from
you
what
is
and
is
not
working
well,
and
gauge
whether
and
how
the
questionnaire
was
useful
to
your
efforts
to
improve
public
involvement
activities.
We
will
not
use
the
information
you
send
to
judge
the
strength
of
individual
public
involvement
efforts.

When
you
conduct
another
public
involvement
event,
repeat
the
same
steps
listed
above.
However,
after
doing
this,
be
sure
to
go
to
the
same
Lotus
1­
2­
3
spreadsheet
and
follow
the
steps
required
to
link
the
data
from
your
first
public
involvement
event
to
your
second
in
order
to
summarize
and
compare
your
data
over
a
period
of
time.
Draft
Draft
Draft
62
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Public
Hearing
Participant
Assessment
Background
This
questionnaire
is
designed
to
help
Agency
staff
better
understand
what
worked
well
and
what
improvements
to
consider
implementing
before
holding
future
Public
Hearings.
Once
the
data
are
summarized,
Agency
staff
can
assess
whether
goals
for
success
were
met,
make
modifications
if
necessary,
and
compare
progress
over
time.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.

1a.
Pre­
hearing
publicity
availability
Please
indicate
how
you
heard
about
the
public
hearing
(
check
all
that
apply):
radio
announcements
television
announcements
newspapers
ads
flyers
in
store
windows/
bulletin
boards
website
information
Other
(
please
specify)

Was
the
information
about
the
public
hearing
made
available
in
locally
used
languages?
Yes
No__

If
you
received
a
pre­
hearing
notice
via
regular
or
electronic
mail,
how
much
in
advance
did
the
notice
arrive?
a
few
days
one
week
2
weeks
2­
4
weeks
more
than
1
month__
__
other
(
please
specify)_________________________________________________

Please
suggest
any
additional
ways
you
think
EPA
should
publicize
future
public
hearings:

Please
explain
what
you
learned
about
the
public
hearing
or
issue
that
made
you
decide
to
attend:
Draft
Draft
Draft
63
1b.
Pre­
hearing
background
information
Was
anything
missing
from
the
pre­
hearing
background
information?
Yes___
No___
If
yes,
please
specify:_____________________________________________

Did
you
call
to
ask
for
specific
information?
Yes
No
If
yes,
what
did
you
ask
for?

2.
Hearing
logistics
Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
Str
on
gly
agr
ee
Do
n't
kno
w
I
became
aware
of
pre­
hearing
notice
early
enough
to
plan
attending.
1
2
3
4
5
6
7
Location
was
convenient.
1
2
3
4
5
6
7
Public
transportation
was
close
by.
1
2
3
4
5
6
7
Facility
was
comfortable
for
participants.
1
2
3
4
5
6
7
Time
during
which
hearing
was
held
was
convenient.
1
2
3
4
5
6
7
The
length
of
the
hearing
was
about
right.
1
2
3
4
5
6
7
Translators
were
available
(
if
needed).
1
2
3
4
5
6
7
Signers
for
hearing
impaired
were
available
(
if
needed).
1
2
3
4
5
6
7
Provisions
were
made
for
handicap
access.
1
2
3
4
5
6
7
3.
Conduct
of
hearing
Greeters
made
participants
feel
welcome.
1
2
3
4
5
6
7
The
registration
process
was
efficient.
1
2
3
4
5
6
7
I
understood
the
purpose
of
the
hearing.
1
2
3
4
5
6
7
I
understood
how
the
hearing
would
be
conducted.
1
2
3
4
5
6
7
Hearing
planners
focused
on
the
right
questions.
1
2
3
4
5
6
7
I
was
comfortable
with
the
hearing
format.
1
2
3
4
5
6
7
Draft
Draft
Draft
64
3.
Conduct
of
hearing
(
continued)

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
Str
on
gly
agr
ee
Do
n't
kno
w
The
format
gave
all
ample
opportunity
to
be
heard.
1
2
3
4
5
6
7
My
ideas
were
heard.
1
2
3
4
5
6
7
There
was
courteous
interaction
between
speakers
and
hearing
officials.
1
2
3
4
5
6
7
It
was
easy
to
sign
up
for
follow­
up
information.
1
2
3
4
5
6
7
There
was
a
good
mix
of
viewpoints.
1
2
3
4
5
6
7
I
understand
the
"
next
steps"
in
the
process.
1
2
3
4
5
6
7
Participants'
input
will
make
a
difference.
1
2
3
4
5
6
7
Did
you
formally
speak
at
the
hearing?
Yes__
No
__

If
yes,
please
indicate
the
degree
to
which
you
agree
or
disagree
with
the
following
statements.

I
received
respectful
treatment.
1
2
3
4
5
6
7
Follow­
up
clarification
questions
were
relevant.
1
2
3
4
5
6
7
Hearing
officials
were
courteous.
1
2
3
4
5
6
7
Please
suggest
how
EPA
can
improve
the
next
public
hearing.

Please
indicate
which
group
you
represent:
neighborhood
group
local
non­
profit
organization
(
not
including
local
neighborhood
non­
profit
groups)

national
environmental
organization
business
or
industry
or
trade
association
state
or
tribal
or
local
government
federal
government
other
(
please
specify)

no
organizational
affiliation
(
e.
g.,
interested
local
resident)

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
65
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Public
Hearing
EPA/
Contractor
Assessment
Background
This
questionnaire
is
designed
to
obtain
an
EPA/
contractor
perspective
on
the
effectiveness
of
a
Public
Hearing.
Agency
staff
will
use
this
information
to
better
understand
what
is
working
well
and
what
improvements
could
make
future
Public
Hearings
more
successful.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
Str
on
gly
agr
ee
Do
n't
kno
w
The
registration
process
was
efficient.
1
2
3
4
5
6
7
Participants
understood
the
purpose
of
the
hearing.
1
2
3
4
5
6
7
Participants
understood
how
the
hearing
would
be
conducted.
1
2
3
4
5
6
7
The
right
topics
were
addressed
during
the
hearing.
1
2
3
4
5
6
7
The
format
gave
all
ample
opportunity
to
be
heard.
1
2
3
4
5
6
7
There
was
courteous
interaction
between
speakers
and
hearing
officials.
1
2
3
4
5
6
7
There
was
a
good
mix
of
viewpoints.
1
2
3
4
5
6
7
Speakers
received
respectful
treatment.
1
2
3
4
5
6
7
EPA
gained
new,
or
clarifying,
information,
that
will
enable
it
to
more
effectively
develop,
revise,
or
implement
rules
or
programs.
1
2
3
4
5
6
7
This
public
hearing
was
a
success.
1
2
3
4
5
6
7
Please
explain
your
ranking
of
the
previous
statement
about
success:
Draft
Draft
Draft
66
Please
suggest
how
EPA
can
improve
its
next
public
hearing:

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
67
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
Public
Hearing
Follow­
Up
Participant
Assessment
Background
This
questionnaire
is
designed
to
help
Agency
staff
better
understand
what
worked
well
and
what
improvements
to
consider
before
implementing
future
Public
Hearings.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
Str
on
gly
agr
ee
Do
n't
kno
w
I
received
hearing
notes.
1
2
3
4
5
6
7
Notes
were
clearly
written.
1
2
3
4
5
6
7
Action
items
I
remember
were
documented.
1
2
3
4
5
6
7
Actions,
due
dates
and
accountable
persons
were
noted.
1
2
3
4
5
6
7
Issues
discussed
and
decisions
made
were
accurately
recorded.
1
2
3
4
5
6
7
I
received
the
information
I
requested
in
a
timely
manner
(
in
_______
days).
1
2
3
4
5
6
7
The
packet
I
received
was
complete.
1
2
3
4
5
6
7
Now
that
some
time
has
passed
since
the
public
hearing
you
may
have
thought
about
what
could
have
made
it
a
better
experience.
Please
suggest
how
you
think
EPA
can
improve
public
hearings
and
follow­
up
actions.

Please
indicate
which
group
you
represent:

neighborhood
group
Draft
Draft
Draft
68
local
non­
profit
organization
(
not
including
local
neighborhood
non­
profit
groups)

national
environmental
organization
business
or
industry
or
trade
association
state
or
tribal
or
local
government
federal
government
other
(
please
specify)

no
organizational
affiliation
(
e.
g.,
interested
local
resident)

Please
return
the
completed
questionnaire
in
the
pre­
addressed,
stamped
envelope
provided.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
69
Assessing
EPA's
Listening
Sessions
A
User's
Guide
with
Questionnaires
Enclosed
are
information
and
tools
for
EPA
staff
to
use
in
generating
feedback
on
the
effectiveness
of
public
listening
sessions
and
the
effectiveness
of
follow­
up
after
them.
Specifically,
this
user
guide
contains
the
following:


Background
information
for
EPA
Staff

Checklist
for
Administering
Questionnaires
and
Using
Results

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Listening
Session
(
participant
assessment)


Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Listening
Session
(
EPA/
contractor
assessment)


Questionnaire
for
Effectiveness
of
Listening
Session
Follow­
Up
(
participant
assessment)
For
questions
contact:
Patricia
Bonner
(
bonner.
patricia@
epa.
gov)/
202­
566­
2204
DRAFT:
Last
Updated
6/
28/
04
Draft
Draft
Draft
70
Assessing
EPA's
Listening
Sessions
Background
Information
for
EPA
Staff
Introduction
Enclosed
are
three
questionnaires
for
EPA
staff
to
use
in
generating
feedback
on
the
effectiveness
of
public
listening
sessions.
The
questionnaires
are
a
component
of
the
evaluation
section
in
the
Framework
for
Implementing
EPA's
Public
Involvement
Policy.
The
questionnaires
are
for
people
who
attend
a
listening
session
to
complete
immediately
after
the
sessions,
for
EPA
staff
responsible
for
coordinating
the
work
of
a
listening
session,
and
for
listening
session
participants
who
receive
follow­
up
information
following
the
listening
session.
The
first
questionnaire
focuses
on
the
effectiveness
of
a
listening
session.
The
second
questionnaire
focuses
on
EPA/
contractor
perspectives
about
the
success
of
the
same
listening
session.
The
third
questionnaire
focuses
on
participant
perspectives
of
on
the
quality
of
the
follow­
up
information
received
after
the
same
listening
session.

What
is
a
listening
session?
A
Listening
Session
is
an
informal
public
meeting
designed
to
provide
an
opportunity
for
EPA
officials
to
listen
and
record
the
views
of
participants
about
specific
or
general
community
environmental
issues
or
proposed
actions.
The
purpose
of
a
listening
session
is
two­
fold:
to
provide
citizens
an
opportunity
to
speak
and
EPA
officials
an
opportunity
to
listen.
Topics
are
usually
those
environmental
issues
which
are
of
most
serious
concern
to
the
participants.
While
the
sessions
are
"
open,"
people
from
a
defined
community
(
e.
g.
a
neighborhood)
usually
attend.
Often
a
neutral
facilitator
will
lead
a
listening
session
to
assist
attendees
to
clarify,
categorize,
and
sometimes
prioritize
concerns.

How
can
the
listening
session
questionnaires
be
useful
to
you?

Use
of
the
questionnaires
should
make
the
process
of
gathering
feedback
for
your
public
involvement
activity
easier
for
you
to
implement
directly,
without
spending
contract
dollars.
The
questionnaires
provide
an
easy
way
to
get
useful
feedback
from
participants
and
EPA
about
the
listening
sessions
they
participated
in.
This
feedback
should
help
you
to
better
understand
whether
a
particular
listening
session
worked
well,
whether
there
are
opportunities
for
improvement,
or
whether
certain
problems
need
to
be
addressed.
Once
you
have
summarized
the
data,
Agency
staff
can
assess
whether
they
met
goals,
make
modifications,
and
compare
progress
over
time.

When
should
I
use
the
questionnaires?
You
can
use
the
first
questionnaire
(
for
session
participants)
towards
the
end
of
a
listening
session.
You
can
use
the
second
questionnaire
(
for
EPA
staff
or
their
contractors
who
assisted
in
planning
or
leading
a
listening
session)
after
the
same
listening
session.
You
can
use
the
third
questionnaire
(
for
session
participants)
when
sending
out
follow­
up
information
to
all
listening
session
participants.
It
is
ultimately
up
to
you,
however,
to
decide
when
to
use
them.
The
key
is
to
use
the
questionnaires
to
the
extent
that
they
provide
you
with
valuable
information
and
improve
your
ability
to
design
and
implement
more
effective
listening
sessions
in
the
future.
Draft
Draft
Draft
71
Are
the
questionnaires
in
compliance
with
the
Paperwork
Reduction
Act?

Not
yet.
The
questionnaires
will
be
cleared
with
the
Office
of
Management
and
Budget
(
OMB)
under
ICR
#
XXXXX,
OMB
Control
No:
2010­
XXXX.
This
clearance
will
allow
EPA
to
collect
information
(
via
the
questionnaires)
from
more
than
nine
non­
federal
entities.
Currently,
these
questionnaires
are
undergoing
a
pre­
test.

Who
designed
these
questionnaires?
The
Evaluation
Task
Group
of
EPA's
Public
Involvement
Workgroup
designed
these
questionnaires
after
consulting
with
EPA
staff
who
are
regularly
involved
in
the
designing
and
holding
of
listening
sessions.

What
should
I
do
when
preparing
to
administer
a
questionnaire?

See
"
Checklist
for
Administering
the
Questionnaire."

What
do
I
do
with
the
questionnaire
data
once
the
forms
are
returned?
You
can
collect,
collate
and
analyze
the
data
from
the
forms
using
a
pre­
formatted
Lotus
1­
2­
3
spreadsheet
program.
In
addition,
the
spreadsheet
program
allows
you
to
summarize
data
over
a
period
of
time,
making
it
easy
to
generate
annual
summary
reports
for
your
management.
More
information
on
how
to
use
the
program
is
available
in
the
"
Checklist
for
Administering
the
Questionnaire
and
Using
Results"
and
within
the
Lotus
1­
2­
3
spreadsheet
program.
Once
completed,
Evaluation
Task
Group
members
encourage
you
to
use
your
findings
and
send
your
completed
data
sheet
files
as
e­
mail
attachments
to:
bonner.
patricia@
epa.
gov.
If
you
do
so,
we
will
be
able
to
track
the
use
of
the
templates,
and
learn
from
you
what
is
and
is
not
working
well,
and
gauge
whether
and
how
the
questionnaires
are
useful
to
your
efforts
to
improve
public
involvement
activities.
We
will
not
use
the
information
you
send
to
judge
the
strength
of
individual
public
participation
efforts.

How
can
I
use
the
results?

The
completed
questionnaires
should
provide
you
with
information
that
can
help
you
design
improvements
to
activities
and
to
your
overall
involvement
process.
You
can
learn
what
worked
well,
what
participants
feel
is
important
and
what
should
change
to
better
accommodate
their
needs.
It
will
be
up
to
you
to
decide
how
much
data
you
need
to
prompt
changes.
While
it
is
prudent
to
be
cautious
about
making
changes
based
upon
a
small
amount
of
feedback
data,
you
may
find
a
number
of
the
respondents'
ideas
useful.
If
you
can
act
on
their
suggestions,
even
in
a
small
way,
participants
will
appreciate
that
you
listened.

How
should
I
communicate
the
results?
Use
the
performance
and
summary
reports
from
the
Lotus
1­
2­
3
spreadsheet
program
to
talk
with
managers
and
staff
about
the
effectiveness
of
the
listening
sessions
and
where
changes
may
be
necessary.
Communicate
overall
results
and
what
changes
you
intend
to
make
to
listening
session
participants
in
the
listening
session
summary
report
or
other
follow­
up
materials
distributed
to
session
participants.
A
few
brief
sentences
describing
overall
what
participants
thought
of
the
session,
their
suggestions
for
improvement,
and
what
you
will
do
to
Draft
Draft
Draft
72
improve
the
next
listening
session
would
let
participants
know
that
you
valued
their
input.

What
is
EPA's
Public
Involvement
Policy
and
where
can
I
get
more
information?
EPA's
2003
Public
Involvement
Policy
underscores
EPA's
commitment
to
improving
its
practices
and
is
based
on
seven
steps
for
effective
public
involvement:
1.
Plan
and
budget
for
public
involvement
activities
2.
Identify
the
interested
and
affected
public
3.
Consider
providing
technical
or
financial
assistance
to
the
public
to
facilitate
involvement
4.
Provide
information
and
outreach
to
the
public
5.
Conduct
public
consultation
and
involvement
activities
6.
Review
and
use
input,
and
provide
feedback
to
the
public
7.
Evaluate
public
involvement
activities
The
Framework
for
Implementing
EPA's
Public
Involvement
Policy
urges
development
of
tools
to
improve
public
involvement
information
sharing,
training
and
evaluation
at
EPA.
This
questionnaire
and
the
database
program
are
examples
of
such
tools.
Cross­
agency
work
groups
developed
the
Policy
and
Framework
using
public
comments
and
ideas
gathered
through
a
two­
week,
Internet­
based
Dialogue
on
Public
Involvement
in
EPA
Decisions.
Copies
of
the
Policy
and
Framework
are
available
@
http://
www.
epa.
gov/
publicinvolvement/
public/
index.
htm.
Draft
Draft
Draft
1Note:
The
instructions
above
are
designed
primarily
for
questionnaires
that
seek
the
perspective
of
participants
about
the
effectiveness
of
a
single
event.
Depending
upon
the
type
of
questionnaire
you
wish
to
distribute
and
the
type
of
public
involvement
activity
you
wish
to
acquire
feedback
for,
you
will
need
to
tailor
the
steps
of
this
checklist
accordingly.

73
Assessing
EPA's
Public
Involvement
Activities
Checklist
for
Administering
the
Questionnaire
and
Using
Results1
Review
questionnaire.

Determine
whether
questionnaire
should
be
administered.

Determine
what
questionnaire(
s)
should
be
administered.

Identify
one
person
to
be
responsible
for
coordinating
the
overall
dissemination
and
collection
of
the
questionnaire
and
to
input
the
data
into
the
Lotus
1­
2­
3
data
input,
analysis,
and
reporting
program
once
the
questionnaires
are
collected.

Make
plans
to
include
the
questionnaire
feedback
process
in
the
agenda
of
your
public
involvement
event.
Specifically,
plan
to
set
aside
a
few
minutes
before
the
conclusion
of
your
public
involvement
event
and
have
participants
fill
out
their
questionnaires.
You
should
also
plan
to
make
an
announcement
at
the
beginning
of
your
public
involvement
event
explaining
the
questionnaire
feedback
process
and
how
feedback
is
needed
from
all
participants
in
order
to
develop
a
more
comprehensive
understanding
of
the
overall
effectiveness
of
the
event
and
how
best
to
make
improvements.
Feedback
is
needed
from
participants
who
leave
early
and
those
who
stay
to
the
end.
If
you
don't
build
the
questionnaire
feedback
process
directly
into
your
event,
and
instead
wait
to
the
very
end,
chances
are
few
participants
will
take
the
time
to
fill
out
their
questionnaires.
Participants
may
suggest
that
they
would
prefer
to
fill
out
their
questionnaires
at
home
or
work.
While
this
option
is
appealing
for
participants,
chances
are
only
a
few
will
take
the
time
to
actually
do
this.
The
best
chance
for
good
feedback
is
when
participants
are
actually
at
the
event.
Take
the
time
to
plan
for
this!

Estimate
the
number
of
participants
who
will
be
attending
the
public
involvement
event
for
which
questionnaires
will
be
distributed.

Make
appropriate
number
of
copies
of
the
questionnaire.

Bring
questionnaire
copies
and
extra
pens
or
pencils
to
the
public
involvement
event.

Identify
official
who
will
be
responsible
for
physically
collecting
the
questionnaires.

Clearly
identify
areas
where
participants
should
return
their
questionnaires.

Hand
out
questionnaires
with
other
public
involvement
activity
materials
either
at
the
beginning
of
the
event,
or
at
a
designated
time
during
the
event.
Draft
Draft
Draft
1Note:
Separate
Lotus
1­
2­
3
programs
have
been
developed
that
match
most
types
of
questionnaires
available
(
e.
g.,
one
for
single
event
effectiveness
for
participants,
one
for
overall
effectiveness
for
participants,
etc.).
Be
sure
to
use
the
appropriate
Lotus
1­
2­
3
program
to
input
your
data.
The
only
type
of
questionnaire
for
which
a
spreadsheet
program
has
not
been
developed
is
for
the
questionnaire
dealing
with
post­
event
follow­
up.
However,
if
you
need
a
spreadsheet
program
for
this
questionnaire,
contact
Pat
Bonner
(
bonner.
patricia@
epa.
gov).

74
Make
it
very
clear
to
participants
to
whom,
or
to
where,
they
should
return
their
questionnaire.
Urge
participants
to
fill
out
their
questionnaire
even
if
they
need
to
leave
early.

Identify
the
total
number
of
participants
attending
(
e.
g.,
the
total
number
of
persons
who
could
possibly
fill
out
questionnaires).

Collect
questionnaires.

Input
data
into
the
Lotus
1­
2­
3
data
input,
analysis,
and
reporting
program
within
one
month
of
session
and
share
results.

Detach
the
Lotus
1­
2­
3
program
onto
your
computer.
1
Open
the
Lotus
1­
2­
3
program.

Click
on
the
"
Getting
Started"
tab
and
follow
the
instructions.

Review
performance
report
and
identify
whether
certain
changes
should
be
considered
in
preparation
for
the
next
public
involvement
event.

Use
the
Lotus
1­
2­
3
performance
reports
to
talk
with
managers
and
staff
about
the
effectiveness
of
your
public
involvement
activity
and
where
changes
may
be
necessary.
Communicate
overall
results
and
what
changes
you
intend
to
make
to
participants
who
attended
your
last
public
involvement
event
in
the
public
involvement
event
summary
report
or
other
follow­
up
materials
distributed
to
participants.
A
few
brief
sentences
describing
overall
what
participants
thought
of
the
event,
their
suggestions
for
improvement,
and
what
you
will
do
to
improve
the
next
event
would
let
participants
know
that
you
valued
their
input.

Sharing
data...
EPA's
Public
Involvement
Policy
Implementation
Staff.
strongly
encourages
you
to
send
your
completed
Lotus
1­
2­
3
program
files
as
e­
mail
attachments
to:
bonner.
patricia@
epa.
gov.
If
you
do
so,
Implementation
Staff
will
be
able
to
track
the
use
of
the
questionnaires,
and
learn
from
you
what
is
and
is
not
working
well,
and
gauge
whether
and
how
the
questionnaire
was
useful
to
your
efforts
to
improve
public
involvement
activities.
We
will
not
use
the
information
you
send
to
judge
the
strength
of
individual
public
involvement
efforts.

When
you
conduct
another
public
involvement
event,
repeat
the
same
steps
listed
above.
However,
after
doing
this,
be
sure
to
go
to
the
same
Lotus
1­
2­
3
spreadsheet
and
follow
the
steps
required
to
link
the
data
from
your
first
public
involvement
event
to
your
second
in
order
to
summarize
and
compare
your
data
over
a
period
of
time.
Draft
Draft
Draft
75
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Listening
Session
Participant
Assessment
Background
This
questionnaire
is
designed
to
help
Agency
staff
better
understand
what
worked
well
and
what
improvements
to
consider
implementing
before
holding
future
Listening
Sessions.
Once
the
data
are
summarized,
Agency
staff
can
assess
whether
goals
for
success
were
met,
make
modifications
if
necessary,
and
compare
progress
over
time.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
occasionally
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.

1a.
Pre­
listening
session
publicity
availability
Please
indicate
how
you
heard
about
the
listening
session
(
check
all
that
apply):
radio
announcements
television
announcements
newspapers
ads
flyers
in
store
windows/
bulletin
boards
website
information
Other
(
please
specify)

Was
the
information
about
the
listening
session
made
available
in
locally
used
languages?
Yes
No__

If
you
received
a
pre­
meeting
notice
via
regular
or
electronic
mail,
how
much
in
advance
did
the
notice
arrive?
___
a
few
days
one
week
2
weeks
2­
4
weeks
more
than
1
month___
other
(
please
specify)

Please
suggest
any
additional
ways
you
think
EPA
should
publicize
future
public
listening
sessions:

Please
explain
what
you
learned
about
the
listening
session
or
issue
that
made
you
decide
to
attend:
Draft
Draft
Draft
76
1b.
Pre­
listening
session
background
information
Was
anything
missing
from
the
pre­
session
background
information?
Yes___
No___
If
yes,
please
specify:

Did
you
call
to
ask
for
specific
information?
Yes
No
If
yes,
what
did
you
ask
for?

2.
Session
Logistics
Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
Str
on
gly
agr
ee
Do
n't
kno
w
Pre­
meeting
invitation
arrived
early
enough
for
me
to
plan
attending.
1
2
3
4
5
6
7
Location
was
convenient.
1
2
3
4
5
6
7
Public
transportation
was
close
by.
1
2
3
4
5
6
7
Facility
was
comfortable
for
participants.
1
2
3
4
5
6
7
Time
during
which
session
was
held
was
convenient.
1
2
3
4
5
6
7
The
length
of
the
session
was
about
right.
1
2
3
4
5
6
7
Translators
were
available
(
if
needed).
1
2
3
4
5
6
7
Signers
for
hearing
impaired
were
available
(
if
needed).
1
2
3
4
5
6
7
Provisions
were
made
for
handicap
access.
1
2
3
4
5
6
7
Facility
was
well
equipped
for
all
planned
activities
(
tables,
work
space,
break­
outs,
supplies,
etc).
1
2
3
4
5
6
7
Refreshments
are
important
for
a
successful
session.
1
2
3
4
5
6
7
Draft
Draft
Draft
77
3.
Conduct
of
Listening
Session
Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
Str
on
gly
agr
ee
Do
n't
kno
w
Greeters
made
participants
feel
welcome.
1
2
3
4
5
6
7
The
registration
process
was
efficient.
1
2
3
4
5
6
7
I
understood
the
purpose
of
the
session.
1
2
3
4
5
6
7
I
understood
how
the
session
would
be
conducted.
1
2
3
4
5
6
7
Session
planners
focused
on
the
right
questions.
1
2
3
4
5
6
7
I
was
comfortable
with
the
session
format.
1
2
3
4
5
6
7
The
format
gave
all
ample
opportunity
to
be
heard.
1
2
3
4
5
6
7
My
ideas
were
heard.
1
2
3
4
5
6
7
There
was
good
interaction
among
participants.
1
2
3
4
5
6
7
It
was
easy
to
sign
up
for
follow­
up
information.
1
2
3
4
5
6
7
There
was
a
good
mix
of
viewpoints.
1
2
3
4
5
6
7
I
understand
the
"
next
steps"
in
the
process.
1
2
3
4
5
6
7
Action
items
were
documented
(
flip
charts).
1
2
3
4
5
6
7
Time­
frames
and
accountable
persons
were
listed
(
flip
charts).
1
2
3
4
5
6
7
Participants'
input
will
make
a
difference.
1
2
3
4
5
6
7
Please
suggest
how
EPA
can
improve
the
next
listening
session.
Draft
Draft
Draft
78
Please
indicate
which
group
you
represent:

neighborhood
group
local
non­
profit
organization
(
not
including
local
neighborhood
non­
profit
groups)

national
environmental
organization
business
or
industry
or
trade
association
state
or
tribal
or
local
government
federal
government
other
(
please
specify)

no
organizational
affiliation
(
e.
g.,
interested
local
resident)

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
79
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Listening
Session
EPA/
Contractor
Assessment
Background
This
questionnaire
is
designed
to
obtain
an
EPA/
contractor
perspective
on
the
effectiveness
of
a
Listening
Session.
Agency
staff
will
use
this
information
to
better
understand
what
is
working
well
and
what
improvements
could
make
future
Listening
Sessions
more
successful.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
Str
on
gly
agr
ee
Do
n't
kno
w
The
registration
process
was
efficient.
1
2
3
4
5
6
7
Participants
understood
the
purpose
of
the
session.
1
2
3
4
5
6
7
Participants
understood
how
the
session
would
be
conducted.
1
2
3
4
5
6
7
The
right
topics
were
addressed
during
the
session.
1
2
3
4
5
6
7
The
format
gave
all
ample
opportunity
to
be
1
2
3
4
5
6
7
There
was
good
interaction
among
participants.
1
2
3
4
5
6
7
The
number
of
people
that
participated
was
appropriate.
1
2
3
4
5
6
7
There
was
a
good
mix
of
viewpoints.
1
2
3
4
5
6
7
EPA
gained
new,
or
clarifying,
information,
that
will
enable
it
to
more
effectively
develop,
revise,
or
implement
rules
or
programs.
1
2
3
4
5
6
7
This
listening
session
was
a
success.
1
2
3
4
5
6
7
Please
describe
the
most
important
reason
for
this
listening
session's
success
or
lack
of
success.
Draft
Draft
Draft
80
Please
suggest
how
EPA
can
improve
the
next
listening
session.

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
81
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
Listening
Session
Follow­
Up
Participant
Assessment
Background
This
questionnaire
is
designed
to
help
Agency
staff
better
understand
what
worked
well
and
what
improvements
to
consider
before
implementing
future
Listening
Sessions.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
Str
on
gly
agr
ee
Do
n't
kno
w
I
received
session
notes.
1
2
3
4
5
6
7
Notes
were
clearly
written.
1
2
3
4
5
6
7
Action
items
I
remember
were
documented.
1
2
3
4
5
6
7
Actions,
due
dates
and
accountable
persons
were
noted.
1
2
3
4
5
6
7
Issues
discussed
and
decisions
made
were
accurately
recorded.
1
2
3
4
5
6
7
I
received
the
information
I
requested
in
a
timely
manner
(
in
_______
days).
1
2
3
4
5
6
7
The
packet
I
received
was
complete.
1
2
3
4
5
6
7
Now
that
some
time
has
passed
since
the
listening
session
you
may
have
thought
about
what
could
have
made
it
a
better
experience.
Please
suggest
how
you
think
EPA
can
improve
listening
sessions
and
follow­
up
actions.
Draft
Draft
Draft
82
Please
indicate
which
group
you
represent:

neighborhood
group
local
non­
profit
organization
(
not
including
local
neighborhood
non­
profit
groups)

national
environmental
organization
business
or
industry
or
trade
association
state
or
tribal
or
local
government
federal
government
other
(
please
specify)

no
organizational
affiliation
(
e.
g.,
interested
local
resident)

Please
return
the
completed
questionnaire
in
the
pre­
addressed,
stamped
envelope
provided.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
83
Assessing
EPA's
Small
Discussion
Groups
A
User's
Guide
with
Questionnaires
Enclosed
are
information
and
tools
for
EPA
staff
to
use
in
generating
feedback
on
the
effectiveness
of
small
discussion
groups.
Specifically,
this
user
guide
contains
the
following:


Background
information
for
EPA
Staff

Checklist
for
Administering
Questionnaires
and
Using
Results

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Small
Discussion
Group
(
participant
assessment)


Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Small
Discussion
Group
(
EPA/
contractor
assessment)


Questionnaire
for
Effectiveness
of
Small
Discussion
Group
Follow­
Up
(
participant
assessment)

For
questions
contact:
Patricia
Bonner
(
bonner.
patricia@
epa.
gov)/
202­
566­
2204
DRAFT:
Last
Updated
6/
28/
04
Draft
Draft
Draft
84
Assessing
EPA's
Small
Discussion
Groups
Background
Information
for
EPA
Staff
Introduction
Enclosed
are
three
questionnaires
for
EPA
staff
to
use
in
generating
feedback
on
the
effectiveness
of
public
small
discussion
groups.
The
questionnaires
are
a
component
of
the
evaluation
section
in
the
Framework
for
Implementing
EPA's
Public
Involvement
Policy.
The
questionnaires
are
for
people
who
attend
a
small
discussion
group
to
complete
immediately
after
the
sessions,
for
EPA/
contractor
staff
responsible
for
coordinating
the
work
of
a
small
discussion
group,
and
for
small
discussion
group
participants
who
receive
follow­
up
information
following
the
small
discussion
group.
The
first
questionnaire
focuses
on
participant
perspectives
of
the
effectiveness
of
a
small
discussion
group.
The
second
questionnaire
focuses
on
EPA/
contractor
perspectives
about
the
success
of
the
same
small
discussion
group.
The
third
questionnaire
focuses
on
participant
perspectives
on
the
quality
of
the
follow­
up
information
received
after
the
same
small
discussion
group.

What
is
a
small
discussion
group?
A
small
discussion
group
usually
has
6­
15
participants.
It
is
often,
but
not
always,
composed
of
like­
minded
individuals,
members
of
a
particular
interest
group,
or
a
segment
of
the
population
(
such
as
seniors).
These
participants
are
usually
members
of
the
general
public,
rather
than
representatives
from
stakeholder
organizations
or
non­
governmental
organizations.
EPA
often
conducts
a
series
of
small
discussion
group
sessions
about
a
specific
topic
or
Agency
activity
to
solicit
input
from
a
range
of
different
like­
minded
individuals
or
interest
groups,
and/
or
to
ensure
that
the
input
gathered
represents
viewpoints
from
various
regions
of
the
country.
The
EPA
staff
who
hold
these
sessions
should
use
experienced
facilitators
who
are
familiar
with
participatory
methods
for
generating
lively
discussions
and
soliciting
input
from
all
the
participants.
Small
discussion
groups
are
most
successful
when
the
EPA
staff
who
are
intimately
familiar
with
the
selected
topic
lead
the
discussion,
with
the
assistance
of
a
facilitator
and
an
experienced
recorder.
The
purpose
of
small
discussion
group
sessions
is
to
generate
a
discussion,
and
solicit
face­
to­
face
input
about
a
specific
EPA­
related
issue,
project,
activity,
policy
or
program
from
various
sectors
of
the
general
public.
Session
facilitator(
s)
encourage
all
participants
to
express
their
opinions
about
the
topic
under
discussion.

How
can
the
small
discussion
group
questionnaires
be
useful
to
you?

Use
of
the
questionnaires
should
make
the
process
of
gathering
feedback
for
your
public
involvement
activity
easier
for
you
to
implement
directly,
without
spending
contract
dollars.
The
questionnaires
provide
an
easy
way
to
get
useful
feedback
from
participants
and
EPA
about
the
small
discussion
groups
they
participated
in.
This
feedback
should
help
you
to
better
understand
whether
a
particular
small
discussion
group
worked
well,
whether
there
are
opportunities
for
improvement,
or
whether
certain
problems
need
to
be
addressed.
Once
you
have
summarized
the
data,
Agency
staff
can
assess
whether
they
met
goals,
make
modifications,
and
compare
progress
over
time.

When
should
I
use
the
questionnaires?
Draft
Draft
Draft
85
You
can
use
the
first
questionnaire
(
for
small
discussion
group
participants)
at
the
end
or
towards
the
end
of
a
small
discussion
group.
You
can
use
the
second
questionnaire
(
for
EPA
staff
or
their
contractors
who:
assisted
in
planning
the
session,
lead
the
session,
or
facilitated
the
session)
after
the
same
small
discussion
group.
You
can
use
the
third
questionnaire
(
for
small
discussion
group
participants)
when
sending
out
follow­
up
information
to
all
small
discussion
group
participants.
It
is
ultimately
up
to
you,
however,
to
decide
when
to
use
them.
The
key
is
to
use
the
questionnaires
to
the
extent
that
they
provide
you
with
valuable
information
and
improve
your
ability
to
design
and
implement
more
effective
small
discussion
groups
in
the
future.

Are
the
questionnaires
in
compliance
with
the
Paperwork
Reduction
Act?
Not
yet.
The
questionnaires
will
be
cleared
with
the
Office
of
Management
and
Budget
(
OMB)
under
ICR
#
XXXXX,
OMB
Control
No:
2010­
XXXX.
This
clearance
will
allow
EPA
to
collect
information
(
via
the
questionnaires)
from
more
than
nine
non­
federal
entities.
Currently,
these
questionnaires
are
undergoing
a
pre­
test.

Who
designed
these
questionnaires?

The
Evaluation
Task
Group
of
EPA's
Public
Involvement
Workgroup
designed
these
questionnaires
after
consulting
with
EPA
staff
who
are
regularly
involved
in
the
designing
and
holding
of
small
discussion
groups.

What
should
I
do
when
preparing
to
administer
a
questionnaire?
See
"
Checklist
for
Administering
the
Questionnaire."

What
do
I
do
with
the
questionnaire
data
once
the
forms
are
returned?

You
can
collect,
collate
and
analyze
the
data
from
the
forms
using
a
pre­
formatted
Lotus
1­
2­
3
spreadsheet
program.
In
addition,
the
spreadsheet
program
allows
you
to
summarize
data
over
a
period
of
time,
making
it
easy
to
generate
annual
summary
reports
for
your
management.
More
information
on
how
to
use
the
program
is
available
in
the
"
Checklist
for
Administering
the
Questionnaire
and
Using
Results"
and
within
the
Lotus
1­
2­
3
spreadsheet
program.
Once
completed,
Evaluation
Task
Group
members
encourage
you
to
use
your
findings
and
send
your
completed
data
sheet
files
as
e­
mail
attachments
to:
bonner.
patricia@
epa.
gov.
If
you
do
so,
we
will
be
able
to
track
the
use
of
the
questionnaires,
and
learn
from
you
what
is
and
is
not
working
well,
and
gauge
whether
and
how
the
questionnaires
are
useful
to
your
efforts
to
improve
public
involvement
activities.
We
will
not
use
the
information
you
send
to
judge
the
strength
of
individual
public
involvement
efforts.

How
can
I
use
the
results?
The
completed
questionnaires
should
provide
you
with
information
that
can
help
you
design
improvements
to
activities
and
to
your
overall
involvement
process.
You
can
learn
what
worked
well,
what
participants
feel
is
important
and
what
should
change
to
better
accommodate
their
needs.
It
will
be
up
to
you
to
decide
how
much
data
you
need
to
prompt
changes.
While
it
is
prudent
to
be
cautious
about
making
changes
based
upon
a
small
amount
of
feedback
data,
you
may
find
a
number
of
the
respondents'
ideas
useful.
If
you
can
Draft
Draft
Draft
86
act
on
their
suggestions,
even
in
a
small
way,
participants
will
appreciate
that
you
listened.

How
should
I
communicate
the
results?
Use
the
performance
and
summary
reports
from
the
Lotus
1­
2­
3
spreadsheet
program
to
talk
with
managers
and
staff
about
the
effectiveness
of
the
small
discussion
groups
and
where
changes
may
be
necessary.
Communicate
overall
results
and
what
changes
you
intend
to
make
to
small
discussion
group
participants
in
the
small
discussion
group
summary
report
or
other
follow­
up
materials
distributed
to
session
participants.
A
few
brief
sentences
describing
overall
what
participants
thought
of
the
session,
their
suggestions
for
improvement,
and
what
you
will
do
to
improve
the
next
small
discussion
group
would
let
participants
know
that
you
valued
their
input.

What
is
EPA's
Public
Involvement
Policy
and
where
can
I
get
more
information?

EPA's
2003
Public
Involvement
Policy
underscores
EPA's
commitment
to
improving
its
practices
and
is
based
on
seven
steps
for
effective
public
involvement:
1.
Plan
and
budget
for
public
involvement
activities
2.
Identify
the
interested
and
affected
public
3.
Consider
providing
technical
or
financial
assistance
to
the
public
to
facilitate
involvement
4.
Provide
information
and
outreach
to
the
public
5.
Conduct
public
consultation
and
involvement
activities
6.
Review
and
use
input,
and
provide
feedback
to
the
public
7.
Evaluate
public
involvement
activities
The
Framework
for
Implementing
EPA's
Public
Involvement
Policy
urges
development
of
tools
to
improve
public
involvement
information
sharing,
training
and
evaluation
at
EPA.
This
questionnaire
and
the
database
program
are
examples
of
such
tools.
Cross­
agency
work
groups
developed
the
Policy
and
Framework
using
public
comments
and
ideas
gathered
through
a
two­
week,
Internet­
based
Dialogue
on
Public
Involvement
in
EPA
Decisions.
Copies
of
the
Policy
and
Framework
are
available
@
http://
www.
epa.
gov/
publicinvolvement/
public/
index.
htm.
Draft
Draft
Draft
1Note:
The
instructions
above
are
designed
primarily
for
questionnaires
that
seek
the
perspective
of
participants
about
the
effectiveness
of
a
single
event.
Depending
upon
the
type
of
questionnaire
you
wish
to
distribute
and
the
type
of
public
involvement
activity
you
wish
to
acquire
feedback
for,
you
will
need
to
tailor
the
steps
of
this
checklist
accordingly.

87
Assessing
EPA's
Public
Involvement
Activities
Checklist
for
Administering
the
Questionnaire
and
Using
Results1
Review
questionnaire.

Determine
whether
questionnaire
should
be
administered.

Determine
what
questionnaire(
s)
should
be
administered.

Identify
one
person
to
be
responsible
for
coordinating
the
overall
dissemination
and
collection
of
the
questionnaire
and
to
input
the
data
into
the
Lotus
1­
2­
3
data
input,
analysis,
and
reporting
program
once
the
questionnaires
are
collected.

Make
plans
to
include
the
questionnaire
feedback
process
in
the
agenda
of
your
public
involvement
event.
Specifically,
plan
to
set
aside
a
few
minutes
before
the
conclusion
of
your
public
involvement
event
and
have
participants
fill
out
their
questionnaires.
You
should
also
plan
to
make
an
announcement
at
the
beginning
of
your
public
involvement
event
explaining
the
questionnaire
feedback
process
and
how
feedback
is
needed
from
all
participants
in
order
to
develop
a
more
comprehensive
understanding
of
the
overall
effectiveness
of
the
event
and
how
best
to
make
improvements.
Feedback
is
needed
from
participants
who
leave
early
and
those
who
stay
to
the
end.
If
you
don't
build
the
questionnaire
feedback
process
directly
into
your
event,
and
instead
wait
to
the
very
end,
chances
are
few
participants
will
take
the
time
to
fill
out
their
questionnaires.
Participants
may
suggest
that
they
would
prefer
to
fill
out
their
questionnaires
at
home
or
work.
While
this
option
is
appealing
for
participants,
chances
are
only
a
few
will
take
the
time
to
actually
do
this.
The
best
chance
for
good
feedback
is
when
participants
are
actually
at
the
event.
Take
the
time
to
plan
for
this!

Estimate
the
number
of
participants
who
will
be
attending
the
public
involvement
event
for
which
questionnaires
will
be
distributed.

Make
appropriate
number
of
copies
of
the
questionnaire.

Bring
questionnaire
copies
and
extra
pens
or
pencils
to
the
public
involvement
event.

Identify
official
who
will
be
responsible
for
physically
collecting
the
questionnaires.

Clearly
identify
areas
where
participants
should
return
their
questionnaires.

Hand
out
questionnaires
with
other
public
involvement
activity
materials
either
at
the
beginning
of
the
event,
or
at
a
designated
time
during
the
event.
Draft
Draft
Draft
1Note:
Separate
Lotus
1­
2­
3
programs
have
been
developed
that
match
most
types
of
questionnaires
available
(
e.
g.,
one
for
single
event
effectiveness
for
participants,
one
for
overall
effectiveness
for
participants,
etc.).
Be
sure
to
use
the
appropriate
Lotus
1­
2­
3
program
to
input
your
data.
The
only
type
of
questionnaire
for
which
a
spreadsheet
program
has
not
been
developed
is
for
the
questionnaire
dealing
with
post­
event
follow­
up.
However,
if
you
need
a
spreadsheet
program
for
this
questionnaire,
contact
Pat
Bonner
(
bonner.
patricia@
epa.
gov).

88
Make
it
very
clear
to
participants
to
whom,
or
to
where,
they
should
return
their
questionnaire.
Urge
participants
to
fill
out
their
questionnaire
even
if
they
need
to
leave
early.

Identify
the
total
number
of
participants
attending
(
e.
g.,
the
total
number
of
persons
who
could
possibly
fill
out
questionnaires).

Collect
questionnaires.

Input
data
into
the
Lotus
1­
2­
3
data
input,
analysis,
and
reporting
program
within
one
month
of
session
and
share
results.

Detach
the
Lotus
1­
2­
3
program
onto
your
computer.
1
Open
the
Lotus
1­
2­
3
program.

Click
on
the
"
Getting
Started"
tab
and
follow
the
instructions.

Review
performance
report
and
identify
whether
certain
changes
should
be
considered
in
preparation
for
the
next
public
involvement
event.

Use
the
Lotus
1­
2­
3
performance
reports
to
talk
with
managers
and
staff
about
the
effectiveness
of
your
public
involvement
activity
and
where
changes
may
be
necessary.
Communicate
overall
results
and
what
changes
you
intend
to
make
to
participants
who
attended
your
last
public
involvement
event
in
the
public
involvement
event
summary
report
or
other
follow­
up
materials
distributed
to
participants.
A
few
brief
sentences
describing
overall
what
participants
thought
of
the
event,
their
suggestions
for
improvement,
and
what
you
will
do
to
improve
the
next
event
would
let
participants
know
that
you
valued
their
input.

Sharing
data...
EPA's
Public
Involvement
Policy
Implementation
Staff.
strongly
encourages
you
to
send
your
completed
Lotus
1­
2­
3
program
files
as
e­
mail
attachments
to:
bonner.
patricia@
epa.
gov.
If
you
do
so,
Implementation
Staff
will
be
able
to
track
the
use
of
the
questionnaires,
and
learn
from
you
what
is
and
is
not
working
well,
and
gauge
whether
and
how
the
questionnaire
was
useful
to
your
efforts
to
improve
public
involvement
activities.
We
will
not
use
the
information
you
send
to
judge
the
strength
of
individual
public
involvement
efforts.

When
you
conduct
another
public
involvement
event,
repeat
the
same
steps
listed
above.
However,
after
doing
this,
be
sure
to
go
to
the
same
Lotus
1­
2­
3
spreadsheet
and
follow
the
steps
required
to
link
the
data
from
your
first
public
involvement
event
to
your
second
in
order
to
summarize
and
compare
your
data
over
a
period
of
time.
Draft
Draft
Draft
89
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
occasionally
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Small
Discussion
Group
Participant
Assessment
Background
This
questionnaire
is
designed
to
help
Agency
staff
better
understand
what
worked
well
and
what
improvements
to
consider
implementing
before
holding
future
Small
Discussion
Groups.
Once
the
data
are
summarized,
Agency
staff
can
assess
whether
goals
for
success
were
met,
make
modifications
if
necessary,
and
compare
progress
over
time.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
Str
on
gly
agr
ee
Do
n't
kno
w
Materials
(
i.
e.,
agenda
and
background
materials)
provided
prior
to
the
session
were:
1
2
3
4
5
6
7
a)
clear
1
2
3
4
5
6
7
b)
useful
1
2
3
4
5
6
7
Pre­
session
materials
were
delivered
early
enough
to
review
before
the
session.
1
2
3
4
5
6
7
Session
time
was
convenient.
1
2
3
4
5
6
7
Session
location
was
convenient.
1
2
3
4
5
6
7
Session
location
was
comfortable.
1
2
3
4
5
6
7
Session
objectives
were:

a)
clear
1
2
3
4
5
6
7
b)
achieved
1
2
3
4
5
6
7
Draft
Draft
Draft
90
Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
Don'
t
kno
w
Facilitator
ensured
that:

a)
the
session
ran
smoothly
1
2
3
4
5
6
7
b)
all
participants
had
a
chance
to
fully
1
2
3
4
5
6
7
EPA
staff
(
if
different
from
or
in
addition
to
facilitator)
provided:

a)
knowledgeable
support
1
2
3
4
5
6
7
b)
courteous
support
1
2
3
4
5
6
7
Contractors
(
if
different
from
or
in
addition
to
facilitator)
provided:

a)
knowledgeable
support
1
2
3
4
5
6
7
b)
courteous
support
1
2
3
4
5
6
7
The
session
was
well
organized.
1
2
3
4
5
6
7
Session
length
was
appropriate.
1
2
3
4
5
6
7
Participation
in
session
was
worthwhile.
1
2
3
4
5
6
7
Please
suggest
how
EPA
can
improve
its
next
small
discussion
group
session:

Please
indicate
which
group
you
represent:

neighborhood
group
local
non­
profit
organization
(
not
including
local
neighborhood
non­
profit
groups)

national
environmental
organization
business
or
industry
or
trade
association
Draft
Draft
Draft
91
state
or
tribal
or
local
government
federal
government
other
(
please
specify)

no
organizational
affiliation
(
e.
g.,
interested
local
resident)

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
92
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
occasionally
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
Small
Discussion
Group
EPA/
Contractor
Assessment
Background
This
questionnaire
is
designed
to
obtain
an
EPA/
contractor
perspective
on
the
effectiveness
of
a
Small
Discussion
Group
session.
Agency
staff
will
use
this
information
to
better
understand
what
is
working
well
and
what
improvements
could
make
future
Small
Discussion
Group
sessions
more
successful.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
Str
on
gly
agr
ee
Do
n't
kno
w
Materials
(
i.
e.,
agenda
and
background
materials)
provided
prior
to
the
session
were:
1
2
3
4
5
6
7
a)
clear
1
2
3
4
5
6
7
b)
useful
1
2
3
4
5
6
7
Pre­
session
materials
were
delivered
early
enough
for
participants
to
review
before
the
1
2
3
4
5
6
7
Session
time
was
convenient.
1
2
3
4
5
6
7
Session
location
was
convenient.
1
2
3
4
5
6
7
Session
location
was
comfortable.
1
2
3
4
5
6
7
Participants
represented
a
sufficient
cross­
section
1
2
3
4
5
6
7
Participants
had
the
necessary
expertise
to
fully
participate.
1
2
3
4
5
6
7
Participants
provided
useful
information.
1
2
3
4
5
6
7
Session
objectives
were
clear.
1
2
3
4
5
6
7
Draft
Draft
Draft
93
Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
Don'
t
kno
w
Facilitator
ensured
that:

a)
session
ran
smoothly
1
2
3
4
5
6
7
b)
all
participants
had
a
chance
to
fully
1
2
3
4
5
6
7
Contractors
(
if
different
from
or
in
addition
to
facilitator)
provided:

a)
knowledgeable
support
1
2
3
4
5
6
7
b)
courteous
support
1
2
3
4
5
6
7
The
session
was
well
organized.
1
2
3
4
5
6
7
Session
length
was
appropriate.
1
2
3
4
5
6
7
Session
fulfilled
its
stated
objectives.
1
2
3
4
5
6
7
Please
describe
the
most
important
reason
for
this
small
discussion
group
session's
success
or
lack
of
success:

Please
suggest
how
EPA
can
improve
the
next
small
discussion
group
session:

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
94
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
occasionally
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
Small
Discussion
Group
Session
Follow­
Up
Participant
Assessment
Background
This
questionnaire
is
designed
to
help
Agency
staff
better
understand
what
worked
well
and
what
improvements
to
consider
before
implementing
future
Small
Discussion
Group
sessions.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
Str
on
gly
agr
ee
Do
n't
kno
w
I
received
session
notes.
1
2
3
4
5
6
7
Notes
were
clearly
written.
1
2
3
4
5
6
7
Action
items
I
remember
were
documented.
1
2
3
4
5
6
7
Actions,
due
dates
and
accountable
persons
were
noted.
1
2
3
4
5
6
7
Issues
discussed
and
decisions
made
were
accurately
recorded.
1
2
3
4
5
6
7
I
received
the
information
I
requested
in
a
timely
manner
(
in
_______
days).
1
2
3
4
5
6
7
The
packet
I
received
was
complete.
1
2
3
4
5
6
7
Now
that
some
time
has
passed
since
the
small
discussion
group
session
you
may
have
thought
about
what
could
have
made
it
a
better
experience.
Please
suggest
how
you
think
EPA
can
improve
small
discussion
group
follow­
up
actions.
Draft
Draft
Draft
95
Please
indicate
which
group
you
represent:

neighborhood
group
local
non­
profit
organization
(
not
including
local
neighborhood
non­
profit
groups)

national
environmental
organization
business
or
industry
or
trade
association
state
or
tribal
or
local
government
federal
government
other
(
please
specify)

no
organizational
affiliation
(
e.
g.,
interested
local
resident)

Please
return
the
completed
questionnaire
in
the
pre­
addressed,
stamped
envelope
provided.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
96
Assessing
Public
Involvement
in
EPA's
Stakeholder
Negotiations
A
User's
Guide
with
Questionnaires
Enclosed
are
information
and
tools
for
EPA
staff
to
use
in
generating
feedback
on
the
effectiveness
of
public
involvement
in
EPA's
Stakeholder
Negotiations.
Specifically,
this
user
guide
contains
the
following:


Background
information
for
EPA
Staff

Checklist
for
Administering
Questionnaires
and
Using
Results

Questionnaire
for
Feedback
on
Public
Involvement
in
EPA's
Stakeholder
Negotiations
(
stakeholder
assessment)


Questionnaire
for
Feedback
on
Public
Involvement
in
EPA's
Stakeholder
Negotiations
(
public's
assessment)

For
questions
contact:
Patricia
Bonner
(
bonner.
patricia@
epa.
gov)/
202­
566­
2204
DRAFT:
Last
Updated
6/
28/
04
Draft
Draft
Draft
1
EPA's
Conflict
Prevention
and
Resolution
Center
will
be
performing
program
evaluations
of
stakeholder
negotiations
and
other
alternative
dispute
resolution
processes
using
the
OMB
approved
program
evaluation
system
of
the
US
Institute
for
Environmental
Conflict
Resolution.

97
Assessing
Public
Involvement
in
EPA's
Stakeholder
Negotiations
Background
Information
for
EPA
Staff
Introduction
Enclosed
are
two
questionnaires
for
EPA
staff
to
use
in
generating
feedback
on
stakeholder
negotiations
where
public
involvement
is
either
part
of
a
negotiation
process
or
contributes
to
it.
The
questionnaires
are
a
component
of
the
evaluation
section
in
the
Framework
for
Implementing
EPA's
Public
Involvement
Policy.
The
questionnaires
are
for
people
who
participate
in
stakeholder
negotiations
and
those
involved
in
stakeholder
negotiations
but
who
do
not
participate
in
the
actual
negotiations.
The
first
questionnaire
is
for
identified
stakeholders
participating
in
a
negotiation
process
and
measures
their
satisfaction
with
access
to
information
from
the
public
relevant
to
a
completed
negotiation
process.
The
second
questionnaire
is
for
use
with
members
of
the
public
who
are
involved
in
a
stakeholder
negotiation
process,
but
who
are
not
participating
directly
at
the
table,
and
measures
their
satisfaction
with
their
involvement
in
various
aspects
of
the
negotiation
process.
1
What
is
a
Stakeholder
Negotiation?
Stakeholder
negotiation
addresses
environmental
issues
that
require
resolution.
At
EPA,
stakeholder
negotiations
range
from
settling
enforcement
actions
to
environmental
planning
activities
(
e.
g.,
watershed
planning,
Brownfields/
Superfund
site
reuse).
Stakeholder
negotiations
may
be
subject
to
the
Federal
Advisory
Committee
Act,
the
Administrative
Dispute
Resolution
Act,
and/
or
the
Negotiated
Rulemaking
Act.
If
you're
not
sure
which
statutes
apply
to
your
process,
consult
your
Office
of
Regional
Counsel
or
the
Office
of
General
Counsel.

What
roles
do
stakeholders
play
in
negotiation
processes?

Generally,
stakeholders
are
individuals
and
organizations
that
have
strong
interest
in
the
Agency's
work
and
policies.
In
a
stakeholder
negotiation,
a
convenor
(
e.
g.,
a
neutral
third
party
or
EPA)
invites
a
group
of
stakeholders
who
represent
identified
interests
in
a
particular
environmental
issue
to
participate
directly
in
the
negotiation
to
develop
an
agreement.
The
convenor
may
invite
other
members
of
the
public
to
provide
input
to
the
stakeholder
negotiation
process,
but
not
directly
participate
in
the
negotiation.

What
is
EPA's
role
in
a
stakeholder
negotiation
process?
EPA
may
have
two
different
roles
in
a
stakeholder
negotiation
process:
initiator
and/
or
participant.
When
the
initiators,
EPA
may
or
may
not
participate
directly
in
negotiation.
In
other
instances,
EPA
may
participate
as
a
stakeholder
in
a
negotiation
that
another
party
initiates.
In
either
case,
EPA
should
comply
with
applicable
laws
and
regulations.

How
do
stakeholders
participate
in
a
negotiation
process?
Draft
Draft
Draft
98
Depending
upon
the
issues
in
question,
and
the
interests
or
initial
involvement
of
the
stakeholders
themselves,
stakeholder
participation
spans
the
spectrum
from
observation
and
discussion,
to
substantive
contributions,
to
full
participation
in
a
decision­
making
process.

How
is
the
public
involved
in
stakeholder
a
negotiation
process?
Most
often,
the
public
does
not
have
the
authority
to
make
decisions.
However,
the
public
has
an
important,
and,
at
times,
a
crucial
role
in
a
stakeholder
negotiation.
EPA
is
aware
that
the
best
decisions
are
those
based
on
the
best
information;
often
the
public
has
information
not
available
at
the
state
or
federal
level.
Not
only
do
decision
makers
need
public
support
for
implementing
environmental
policies
and
programs,
but
the
public
often
can
provide
information
that
makes
such
policies
and
programs
more
complete
and
likely
to
succeed.

How
can
the
Stakeholder
Negotiations
questionnaires
be
useful
to
you?

Information
from
these
questionnaires
should
help
Agency
staff
better
understand
what
worked
well
with
respect
to
public
involvement
in
stakeholder
negotiations,
and
what
improvements
they
can
make
for
future
stakeholder
negotiations.
Specifically,
the
questionnaires
can
help
EPA
staff
understand
how
stakeholder
participants
and
other
members
of
the
public
viewed
mechanisms
for
accessing
information,
types
of
information
provided,
and
the
impacts
that
information
had
on
the
negotiation
process
and
outcome.
Once
Agency
staff
have
the
summarized
data
they
can
assess
whether
the
public
involvement
process
has
met
its
goals
for
success,
make
modifications
if
necessary,
and
compare
progress
over
time.

When
should
I
use
the
questionnaires?
Use
both
questionnaires
following
the
conclusion
of
a
stakeholder
negotiation
process
that
included
a
public
involvement
component.
The
key
is
to
use
the
questionnaires
to
the
extent
that
they
can
help
you
design
and
implement
effective
public
involvement
in
a
stakeholder
negotiation.

Are
the
questionnaires
in
compliance
with
the
Paperwork
Reduction
Act?

Not
yet.
The
questionnaires
will
be
cleared
with
the
Office
of
Management
and
Budget
(
OMB)
under
ICR
#
XXXXX,
OMB
Control
No:
2010­
XXXX.
This
clearance
will
allow
EPA
to
collect
information
(
via
the
questionnaires)
from
more
than
nine
non­
federal
entities.

Who
designed
these
questionnaires?
The
Evaluation
Task
Group
of
EPA's
Public
Involvement
Workgroup
designed
these
questionnaires.
Task
Group
members
consulted
EPA
staff
who
are
regularly
involved
in
the
design
and
implementation
of
stakeholder
negotiations
about
this
effort.
The
Task
Group
used
their
suggestions
and
advice
to
help
shape
the
questionnaires.

What
should
I
do
when
preparing
to
administer
a
questionnaire?

See
"
Checklist
for
Administering
the
Questionnaire."
Draft
Draft
Draft
99
What
do
I
do
with
the
questionnaire
data
once
the
forms
are
returned?
You
can
collect,
collate
and
analyze
the
data
from
the
forms
using
a
pre­
formatted
Lotus
Notes
spreadsheet
program.
Here
are
the
simple
steps
you
can
take
to
use
that
program:
(
1)
enter
summary
information
regarding
your
hearing
(
e.
g.,
number
of
participants,
location,
hearing
officer,
etc.),
(
2)
input
questionnaire
data,
and
(
3)
print
an
executive
summary
regarding
questionnaire
results
and
success
in
meeting
overall
performance
goals.
In
addition,
the
spreadsheet
program
allows
you
to
summarize
data
over
a
period
of
time,
making
it
easy
to
generate
annual
summary
reports
for
your
management.
More
information
on
how
to
use
the
program
is
available
in
the
spreadsheet.
Evaluation
Task
Group
members
encourage
you
to
send
your
completed
data
sheet
files
as
e­
mail
attachments
to:
bonner.
patricia@
epa.
gov.
If
you
do
so,
we
will
be
able
to
track
the
use
of
the
templates,
and
learn
from
you
what
is
and
is
not
working
well,
and
gauge
whether
and
how
the
questionnaires
are
useful
to
your
efforts
to
improve
public
involvement
activities.
We
will
not
use
the
information
you
send
to
judge
the
strength
of
individual
public
involvement
efforts.

How
can
I
use
the
results?

The
completed
questionnaires
should
provide
you
with
information
that
can
help
you
design
improvements
to
activities
and
to
your
overall
involvement
process.
You
can
learn
what
worked
well,
what
participants
feel
is
important
and
what
should
change
to
better
accommodate
their
needs.
It
will
be
up
to
you
to
decide
how
much
data
you
need
to
prompt
changes.
While
it
is
prudent
to
be
cautious
about
making
changes
based
upon
a
small
amount
of
feedback
data,
you
may
find
a
number
of
the
respondents'
ideas
useful.
If
you
can
act
on
their
suggestions,
even
in
a
small
way,
participants
will
appreciate
that
you
listened.

How
can
I
communicate
the
results?
Use
the
performance
and
summary
reports
from
the
Lotus
1­
2­
3
spreadsheet
program
to
talk
with
managers,
staff,
and
those
directly
and
indirectly
involved
in
stakeholder
negotations
about
the
effectiveness
of
the
public
involvement
component
of
the
stakeholder
negotiation
and
where
changes
may
be
necessary.
Communicate
overall
results
to
those
involved
in
the
stakeholder
negotiation
in
follow­
up
materials
distributed
to
stakeholder
negotiation
stakeholders.
A
few
brief
sentences
describing
overall
what
participants
thought
of
public
involvement
in
the
stakeholder
negotiation,
their
suggestions
for
improvement,
and
what
might
be
considered
to
improve
the
process
would
let
stakeholders
know
that
you
valued
their
input.

What
is
EPA's
Public
Involvement
Policy
and
where
can
I
get
more
information?

EPA's
2003
Public
Involvement
Policy
underscores
EPA's
commitment
to
improving
its
practices
and
is
based
on
seven
steps
for
effective
public
involvement:
1.
Plan
and
budget
for
public
involvement
activities
2.
Identify
the
interested
and
affected
public
3.
Consider
providing
technical
or
financial
assistance
to
the
public
to
facilitate
involvement
4.
Provide
information
and
outreach
to
the
public
5.
Conduct
public
consultation
and
involvement
activities
6.
Review
and
use
input,
and
provide
feedback
to
the
public
Draft
Draft
Draft
100
7.
Evaluate
public
involvement
activities
The
Framework
for
Implementing
EPA's
Public
Involvement
Policy
urges
development
of
tools
to
improve
public
involvement
information
sharing,
training
and
evaluation
at
EPA.
This
questionnaire
and
the
database
program
are
examples
of
such
tools.
Cross­
agency
work
groups
developed
the
Policy
and
Framework
using
public
comments
and
ideas
gathered
through
a
two­
week,
Internet­
based
Dialogue
on
Public
Involvement
in
EPA
Decisions.
Copies
of
the
Policy
and
Framework
are
available
@
http://
www.
epa.
gov/
publicinvolvement/
public/
index.
htm.
Draft
Draft
Draft
1Note:
The
instructions
above
are
designed
primarily
for
questionnaires
that
seek
the
perspective
of
participants
about
the
effectiveness
of
a
single
event.
Depending
upon
the
type
of
questionnaire
you
wish
to
distribute
and
the
type
of
public
involvement
activity
you
wish
to
acquire
feedback
for,
you
will
need
to
tailor
the
steps
of
this
checklist
accordingly.

101
Assessing
EPA's
Public
Involvement
Activities
Checklist
for
Administering
the
Questionnaire
and
Using
Results1
Review
questionnaires.

Determine
whether
a
questionnaire
should
be
administered.

Determine
what
questionnaire(
s)
should
be
administered.

Identify
one
person
to
be
responsible
for
coordinating
the
overall
dissemination
and
collection
of
the
questionnaire
and
to
input
the
data
into
the
Lotus
1­
2­
3
data
input,
analysis,
and
reporting
program
once
the
questionnaires
are
collected.

Make
plans
to
include
the
questionnaire
feedback
process
in
the
agenda
of
your
public
involvement
event.
Specifically,
plan
to
set
aside
a
few
minutes
before
the
conclusion
of
your
public
involvement
event
and
have
participants
fill
out
their
questionnaires.
You
should
also
plan
to
make
an
announcement
at
the
beginning
of
your
public
involvement
event
explaining
the
questionnaire
feedback
process
and
how
feedback
is
needed
from
all
participants
in
order
to
develop
a
more
comprehensive
understanding
of
the
overall
effectiveness
of
the
event
and
how
best
to
make
improvements.
Feedback
is
needed
from
participants
who
leave
early
and
those
who
stay
to
the
end.
If
you
don't
build
the
questionnaire
feedback
process
directly
into
your
event,
and
instead
wait
to
the
very
end,
chances
are
few
participants
will
take
the
time
to
fill
out
their
questionnaires.
Participants
may
suggest
that
they
would
prefer
to
fill
out
their
questionnaires
at
home
or
work.
While
this
option
is
appealing
for
participants,
chances
are
only
a
few
will
take
the
time
to
actually
do
this.
The
best
chance
for
good
feedback
is
when
participants
are
actually
at
the
event.
Take
the
time
to
plan
for
this!

Estimate
the
number
of
participants
who
will
be
attending
the
public
involvement
event
for
which
questionnaires
will
be
distributed.

Make
appropriate
number
of
copies
of
the
questionnaire.

Bring
questionnaire
copies
and
extra
pens
or
pencils
to
the
public
involvement
event.

Identify
official
who
will
be
responsible
for
physically
collecting
the
questionnaires.

Clearly
identify
areas
where
participants
should
return
their
questionnaires.

Hand
out
questionnaires
with
other
public
involvement
activity
materials
either
at
the
beginning
of
the
event,
or
at
a
designated
time
during
the
event.
Draft
Draft
Draft
1Note:
Separate
Lotus
1­
2­
3
programs
have
been
developed
that
match
most
types
of
questionnaires
available
(
e.
g.,
one
for
single
event
effectiveness
for
participants,
one
for
overall
effectiveness
for
participants,
etc.).
Be
sure
to
use
the
appropriate
Lotus
1­
2­
3
program
to
input
your
data.
The
only
type
of
questionnaire
for
which
a
spreadsheet
program
has
not
been
developed
is
for
the
questionnaire
dealing
with
post­
event
follow­
up.
However,
if
you
need
a
spreadsheet
program
for
this
questionnaire,
contact
Pat
Bonner
(
bonner.
patricia@
epa.
gov).

102
Make
it
very
clear
to
participants
to
whom,
or
to
where,
they
should
return
their
questionnaire.
Urge
participants
to
fill
out
their
questionnaire
even
if
they
need
to
leave
early.

Identify
the
total
number
of
participants
attending
(
e.
g.,
the
total
number
of
persons
who
could
possibly
fill
out
questionnaires).

Collect
questionnaires.

Input
data
into
the
Lotus
1­
2­
3
data
input,
analysis,
and
reporting
program
within
one
month
of
session
and
share
results.

Detach
the
Lotus
1­
2­
3
program
onto
your
computer.
1
Open
the
Lotus
1­
2­
3
program.

Click
on
the
"
Getting
Started"
tab
and
follow
the
instructions.

Review
performance
report
and
identify
whether
certain
changes
should
be
considered
in
preparation
for
the
next
public
involvement
event.

Use
the
Lotus
1­
2­
3
performance
reports
to
talk
with
managers
and
staff
about
the
effectiveness
of
your
public
involvement
activity
and
where
changes
may
be
necessary.
Communicate
overall
results
and
what
changes
you
intend
to
make
to
participants
who
attended
your
last
public
involvement
event
in
the
public
involvement
event
summary
report
or
other
follow­
up
materials
distributed
to
participants.
A
few
brief
sentences
describing
overall
what
participants
thought
of
the
event,
their
suggestions
for
improvement,
and
what
you
will
do
to
improve
the
next
event
would
let
participants
know
that
you
valued
their
input.

Sharing
data...
EPA's
Public
Involvement
Policy
Implementation
Staff
strongly
encourages
you
to
send
your
completed
Lotus
1­
2­
3
program
files
as
e­
mail
attachments
to:
bonner.
patricia@
epa.
gov.
If
you
do
so,
Implementation
Staff
will
be
able
to
track
the
use
of
the
questionnaires,
and
learn
from
you
what
is
and
is
not
working
well,
and
gauge
whether
and
how
the
questionnaire
was
useful
to
your
efforts
to
improve
public
involvement
activities.
We
will
not
use
the
information
you
send
to
judge
the
strength
of
individual
public
involvement
efforts.

When
you
conduct
another
public
involvement
event,
repeat
the
same
steps
listed
above.
However,
after
doing
this,
be
sure
to
go
to
the
same
Lotus
1­
2­
3
spreadsheet
and
follow
the
steps
required
to
link
the
data
from
your
first
public
involvement
event
to
your
second
in
order
to
summarize
and
compare
your
data
over
a
period
of
time.
Draft
Draft
Draft
103
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
Public
Involvement
in
EPA's
Stakeholder
Negotiations
Stakeholder
Assessment
Background
This
questionnaire
is
designed
to
measure
your
satisfaction
with
access
to
information
from
the
public
relevant
to
a
completed
negotiation
process.
This
questionnaire
asks
how
you
received
information
from
the
public
and
the
nature
and
adequacy
of
such
information.
The
questionnaire
also
asks
about
the
extent
to
which
you
believe
that
public
involvement
contributed
value
to
the
outcome
of
the
negotiation.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
occasionally
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
Don'
t
kno
w
To
what
extent
do
you
agree
that
the
following
mechanisms
for
accessing
information
from
the
public
about
the
issues
discussed
during
the
negotiation
process
are
useful?
Please
mark
"
Don't
know"
if
the
mechanism
was
not
used.

a)
Surveys
1
2
3
4
5
6
7
b)
Interviews
1
2
3
4
5
6
7
c)
Mailings
requesting
information
1
2
3
4
5
6
7
d)
Public
meetings
1
2
3
4
5
6
7
e)
Information
repositories
(
e.
g.,
library,
docket)
1
2
3
4
5
6
7
f)
Other,
please
specify___________________________________________________
Draft
Draft
Draft
104
Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
Don'
t
kno
w
To
what
extent
do
you
agree
that
the
following
types
of
information
that
the
public
provided
during
the
stakeholder
negotiation
were
useful?
Please
mark
"
Don't
Know"
if
the
mechanism
was
not
used.

a)
Technical/
scientific
information
1
2
3
4
5
6
7
b)
Preferred
outcomes
1
2
3
4
5
6
7
c)
Past
or
future
impacts
1
2
3
4
5
6
7
d)
New
options
1
2
3
4
5
6
7
e)
Environmental
justice
concerns
1
2
3
4
5
6
7
f)
Legal
information
1
2
3
4
5
6
7
g)
Funding/
resource
issues
1
2
3
4
5
6
7
h)
Other,
please
specify___________________________________________________

To
what
extent
do
you
agree
with
the
following
statements?
Please
mark
"
Don't
know"
if
the
statement
is
not
applicable.

The
information
received
from
the
public
during
the
negotiation
process...

a)
Enhanced
your
satisfaction
with
the
negotiation
process.
1
2
3
4
5
6
7
b)
Narrowed
issues
of
disagreement
with
other
stakeholders.
1
2
3
4
5
6
7
c)
Improved
communication
with
other
stakeholders.
1
2
3
4
5
6
7
d)
Improved
collaboration
with
other
stakeholders.
1
2
3
4
5
6
7
Draft
Draft
Draft
105
Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
Don'
t
kno
w
To
what
extent
do
you
agree
with
the
following
statements?
Please
mark
"
Don't
Know"
if
the
statement
is
not
applicable.

The
information
you
received
from
the
public
during
the
negotiation
process
had
an
impact
on...

a)
Creation
of
options
other
than
the
negotiation
process
for
addressing
issues.
1
2
3
4
5
6
7
b)
Reaching
an
agreement.
1
2
3
4
5
6
7
c)
Your
satisfaction
with
the
agreement.
1
2
3
4
5
6
7
Please
provide
any
additional
comments
you
have
on
how
to
improve
access
to
information
from
the
public
during
stakeholder
negotiations:

Please
indicate
which
group
you
represent:

neighborhood
group
local
non­
profit
organization
(
not
including
local
neighborhood
non­
profit
groups)

national
environmental
organization
business
or
industry
or
trade
association
state
or
tribal
or
local
government
federal
government
other
(
please
specify)

no
organizational
affiliation
(
e.
g.,
interested
local
resident)

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
106
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
Public
Involvement
in
EPA's
Stakeholder
Negotiations
Public
Participant
Assessment
Background
This
questionnaire
is
designed
to
measure
your
satisfaction
with
involvement
in
the
stakeholders'
decision­
making
process
about
an
important
issue
that
affects
your
life.
This
questionnaire
asks
about
the
extent
to
which
you
believe
you
were
involved
in
various
aspects
of
the
stakeholder
negotiation
process.
The
questionnaire
also
asks
about
the
extent
to
which
you
believe
that
your
contributions
had
an
impact
on
the
outcome
of
the
negotiation.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
occasionally
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.

Stro
ngly
disa
gre
e
Dis
agr
ee
So
me
wha
t
disa
gre
e
So
me
wha
t
agr
ee
Agr
ee
Str
on
gly
ag
re
e
Don't
know
Please
rate
the
extent
to
which
you
are
satisfied
with
the
following
aspects
of
your
involvement
with
the
identified
stakeholder
negotiation
process.
Please
mark
"
Don't
know"
if
the
listed
aspect
is
not
applicable.

a)
Your
ability
to
contribute
information
to
the
stakeholder's
decision.
1
2
3
4
5
6
7
b)
The
influence
of
your
contribution
on
the
stakeholders'
decision.
1
2
3
4
5
6
7
c)
Communication
from
the
stakeholders
to
you
about
the
decisionmaking
process.
1
2
3
4
5
6
7
d)
The
effort
made
by
stakeholders
to
solicit
your
input
to
the
decisionmaking
process.
1
2
3
4
5
6
7
e)
Your
ability
to
influence
how
you
provided
input
to
the
decision­
making
process.
1
2
3
4
5
6
7
f)
Access
to
information
that
you
needed
to
effectively
provide
input
to
the
decision­
making
process.
1
2
3
4
5
6
7
g)
The
influence
of
your
input
on
the
outcome
of
the
decision­
making.

process
1
2
3
4
5
6
7
Draft
Draft
Draft
107
Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
Don'
t
kno
w
To
what
extent
do
you
agree
that
your
input
in
the
following
areas
impacted
the
outcome
of
the
decision
making
process?
Please
mark
"
Don't
know"
if
the
mechanism
was
not
used.

a)
Technical/
scientific
information
1
2
3
4
5
6
7
b)
Preferred
outcomes
1
2
3
4
5
6
7
c)
Past
or
future
impacts
1
2
3
4
5
6
7
d)
New
options
1
2
3
4
5
6
7
e)
Environmental
justice
concerns
1
2
3
4
5
6
7
f)
Legal
information
1
2
3
4
5
6
7
g)
Funding/
resource
issues
1
2
3
4
5
6
7
h)
Other,
please
specify___________________________________________________

Please
provide
any
additional
comments
you
have
on
how
to
improve
the
public's
involvement
in
stakeholder
negotiations:

Please
indicate
which
group
you
represent:
neighborhood
group
local
non­
profit
organization
(
not
including
local
neighborhood
non­
profit
groups)
national
environmental
organization
business
or
industry
or
trade
association
state
or
tribal
or
local
government
federal
government
other
(
please
specify)
no
organizational
affiliation
(
e.
g.,
interested
local
resident)

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
108
Assessing
EPA
Federal
Advisory
Committee
Act
Committees
A
User's
Guide
with
Questionnaires
Enclosed
are
information
and
tools
for
EPA
staff
to
use
in
generating
feedback
on
the
effectiveness
of
EPA
Federal
Advisory
Committee
Act
(
FACA)
Committees.
Specifically,
this
user
guide
contains
the
following:


Background
information
for
EPA
Staff

Checklist
for
Administering
Questionnaires
and
Using
Results

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
FACA
Meeting
(
participant
assessment)


Questionnaire
for
Feedback
on
the
Effectiveness
of
a
FACA
Meeting
(
EPA/
contractor
assessment)


Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
FACA
Process
(
participant
assessment)


Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
FACA
Process
(
EPA/
contractor
assessment)


Questionnaire
for
Feedback
on
the
Effectiveness
of
FACA
Committee
Outputs
For
questions
contact:
Patricia
Bonner
(
bonner.
patricia@
epa.
gov)/
202­
566­
2204
DRAFT:
Last
Updated
6/
28/
04
Draft
Draft
Draft
109
Assessing
EPA's
Federal
Advisory
Committee
Act
Committees
Background
Information
for
EPA
Staff
Introduction
Enclosed
are
five
questionnaires
for
EPA
staff
to
use
in
generating
feedback
on
the
effectiveness
of
EPA­
supported
Federal
Advisory
Committee
Act
(
FACA)
Committees,
and
the
effectiveness
of
any
FACA
process.
The
questionnaires
are
a
component
of
the
evaluation
section
in
the
Framework
for
Implementing
EPA's
Public
Involvement
Policy.
The
questionnaires
are
for
people
who
regularly
attend
FACA
meetings
and
for
EPA
staff
and
contractors
who
facilitate
and/
or
participate
in
the
FACA
process
over
a
period
of
time.
The
first
and
second
questionnaires
(
one
for
FACA
participants
and
one
for
EPA
staff/
contractor)
focus
on
the
effectiveness
of
specific
FACA
meetings.
The
third
and
fourth
questionnaires
(
one
for
FACA
participants
and
one
for
EPA
staff/
contractor),
which
should
be
administered
once
or
twice
a
year
as
needed,
focus
on
FACA
participant
and
EPA/
contractor
perspectives
on
the
FACA's
success.
The
fifth
questionnaire,
which
should
be
administered
once
or
twice
a
year
as
needed,
focuses
on
the
perspectives
of
the
FACA
EPA
Designated
Federal
Official
(
DFO)
regarding
the
effectiveness
of
FACA
committee
outputs
for
EPA
purposes.

What
is
a
FACA
Committee?
When
EPA
seeks
advice
or
recommendations
from
a
group
that
includes
one
or
more
individuals
who
are
not
federal
government
employees,
the
Agency
should
determine
whether
the
Federal
Advisory
Committee
Act
(
FACA),
5
U.
S.
C.
App.
2,
applies.
EPA
staff
should
consult
with
the
Office
of
General
Counsel
or
the
Office
of
Regional
Counsel
to
make
that
determination.
FACA
requires,
among
other
things,
that
such
groups
be
chartered,
have
a
balanced
membership,
hold
open
meetings
and
make
written
materials
available
to
the
public.
EPA
staff
should
contact
the
Committee
Management
Officer
in
the
Office
of
Cooperative
Environmental
Management
for
advice
on
complying
with
the
FACA
requirements.
The
primary
function
of
a
FACA
committee
is
to
provide
advice
and
recommendations
to
federal
officials.
FACA
committees
also
can
provide
a
forum
for
addressing
issues,
promoting
constructive
dialogue
among
the
various
interests
represented
on
the
group,
and
enhancing
community
understanding
of
the
Agency's
action.
EPA
FACA
committees
address
a
wide
variety
of
scientific,
technical
and
policy
issues.

How
can
the
FACA
questionnaires
be
useful
to
you?

Use
of
the
questionnaires
should
make
the
process
of
gathering
feedback
for
your
public
involvement
activity
easier
for
you
to
implement
directly,
without
spending
contract
dollars.
These
questionnaires
should
help
Agency
staff
better
understand
what
worked
well
and
what
improvements
they
can
make
prior
to
conducting
future
FACA
meetings
or
staffing
another
FACA
process.
Specifically,
the
questionnaires
can
help
EPA
staff
understand
participant
perspectives
regarding
pre­
session
preparation,
clarity
of
objectives,
facilitator
effectiveness,
EPA
support,
and
value
of
participation.
Once
summarized,
Agency
staff
can
use
the
data
to
assess
if
goals
were
met,
make
modifications
if
necessary,
and
compare
progress
over
time.
This
feedback
should
help
you
to
better
understand
whether
a
particular
FACA
committee
is
working
out
well,
whether
you
are
missing
opportunities
for
improvement,
or
whether
you
need
to
address
certain
problems.
Once
you
have
summarized
the
data,
Agency
staff
can
assess
Draft
Draft
Draft
110
whether
they
met
goals,
make
modifications,
and
compare
progress
over
time.

When
should
I
use
the
questionnaires?
You
can
use
the
first
and
second
questionnaires
(
one
for
participants
and
one
for
EPA
staff/
contactor)
after
every
meeting
or
every
couple
of
meetings.
Use
the
third
and
fourth
questionnaires
(
one
for
participants
and
one
for
the
EPA
FACA
DFO)
once
or
twice
a
year,
depending
on
the
level
of
activity
of
the
FACA.
Use
the
fifth
questionnaire
to
gain
perspective
on
the
effectiveness
of
various
FACA
committee
outputs.
The
questionnaires
may
also
be
helpful
in
situations
where
a
particular
problem
hinders
the
successful
working
of
the
group.
They
may
be
especially
useful
when
EPA
seeks
advice
from
a
FACA
that
needs
particularly
effective
support
to
gel
quickly
and
develop
findings.
It
is
ultimately
up
to
you,
however,
to
decide
when
to
use
them.
The
key
is
to
use
the
questionnaires
to
the
extent
that
they
provide
you
with
valuable
information
and
improve
your
ability
to
design
and
implement
more
effective
FACA
committee
processes
in
the
future.

Are
the
questionnaires
in
compliance
with
the
Paperwork
Reduction
Act?

Not
yet.
The
questionnaires
one
through
four
will
be
cleared
with
the
Office
of
Management
and
Budget
(
OMB)
under
ICR
#
XXXXX,
OMB
Control
No:
2010­
XXXX.
This
clearance
will
allow
EPA
to
collect
information
(
via
the
questionnaires)
from
more
than
nine
non­
federal
entities.
The
fifth
questionnaire
does
not
need
OMB
approval
(
since
it
is
designed
for
federal
entities),
and
is
ready
for
use.

Who
designed
these
questionnaires?
The
Evaluation
Task
Group
of
EPA's
Public
Involvement
Workgroup
designed
these
questionnaires
after
consulting
with
EPA
staff
who
are
regularly
involved
in
working
with
FACA
committees.

What
should
I
when
preparing
to
administer
a
questionnaire?
See
"
Checklist
for
Administering
the
Questionnaire."

What
do
I
do
with
the
questionnaire
data
once
the
forms
are
returned?

You
can
collect,
collate
and
analyze
the
data
from
the
forms
using
a
pre­
formatted
Lotus
1­
2­
3
spreadsheet
program.
In
addition,
the
spreadsheet
program
allows
you
to
summarize
data
over
a
period
of
time,
making
it
easy
to
generate
annual
summary
reports
for
your
management.
More
information
on
how
to
use
the
program
is
available
in
the
"
Checklist
for
Administering
the
Questionnaire
and
Using
Results"
and
within
the
Lotus
1­
2­
3
spreadsheet
program.
Once
completed,
Evaluation
Task
Group
members
encourage
you
to
use
your
findings
and
send
your
completed
data
sheet
files
as
e­
mail
attachments
to:
bonner.
patricia@
epa.
gov.
If
you
do
so,
we
will
be
able
to
track
the
use
of
the
questionnaires,
and
learn
from
you
what
is
and
is
not
working
well,
and
gauge
whether
and
how
the
questionnaires
are
useful
to
your
efforts
to
improve
public
involvement
activities.
We
will
not
use
the
information
you
send
to
judge
the
strength
of
individual
public
involvement
efforts.

How
can
I
use
the
results?
Draft
Draft
Draft
111
The
completed
questionnaires
should
provide
you
with
information
that
can
help
you
design
improvements
to
activities
and
to
your
overall
involvement
process.
Aggregated
data
should
help
you
better
understand
what
worked
well
and
what
improvements
can
be
made
prior
to
conducting
future
FACA
meetings
or
staffing
another
FACA
process.
You
can
learn
what
worked
well,
what
participants
feel
is
important
and
what
should
change
to
better
accommodate
their
needs.
It
will
be
up
to
you
to
decide
how
much
data
you
need
to
prompt
changes.
While
it
is
prudent
to
be
cautious
about
making
changes
based
upon
a
small
amount
of
feedback
data,
you
may
find
a
number
of
the
respondents'
ideas
useful.
If
you
can
act
on
their
suggestions,
even
in
a
small
way,
your
participants
will
appreciate
that
you
listened.

How
should
I
communicate
the
results?
Use
the
performance
and
summary
reports
from
the
Lotus
1­
2­
3
spreadsheet
program
to
talk
with
managers,
staff,
and
FACA
members
about
the
effectiveness
of
FACA
meetings
and
FACAs
overall
and
where
changes
may
be
necessary.
Communicate
overall
results
to
FACA
participants
in
the
FACA
performance
report
or
other
follow­
up
materials
distributed
to
FACA
participants.
A
few
brief
sentences
describing
overall
what
participants
thought
of
the
FACA,
their
suggestions
for
improvement,
and
what
might
be
considered
to
improve
the
FACA
process
would
let
participants
know
that
you
valued
their
input.

What
is
EPA's
Public
Involvement
Policy
and
where
can
I
get
more
information?

EPA's
2003
Public
Involvement
Policy
underscores
EPA's
commitment
to
improving
its
practices
and
is
based
on
seven
steps
for
effective
public
involvement:
1.
Plan
and
budget
for
public
involvement
activities
2.
Identify
the
interested
and
affected
public
3.
Consider
providing
technical
or
financial
assistance
to
the
public
to
facilitate
involvement
4.
Provide
information
and
outreach
to
the
public
5.
Conduct
public
consultation
and
involvement
activities
6.
Review
and
use
input,
and
provide
feedback
to
the
public
7.
Evaluate
public
involvement
activities
The
Framework
for
Implementing
EPA's
Public
Involvement
Policy
urges
development
of
tools
to
improve
public
involvement
information
sharing,
training
and
evaluation
at
EPA.
This
questionnaire
and
the
database
program
are
examples
of
such
tools.
Cross­
agency
work
groups
developed
the
Policy
and
Framework
using
public
comments
and
ideas
gathered
through
a
two­
week,
Internet­
based
Dialogue
on
Public
Involvement
in
EPA
Decisions.
Copies
of
the
Policy
and
Framework
are
available
@
http://
www.
epa.
gov/
publicinvolvement/
public/
index.
htm.
Draft
Draft
Draft
1Note:
The
instructions
above
are
designed
primarily
for
questionnaires
that
seek
the
perspective
of
participants
about
the
effectiveness
of
a
single
event.
Depending
upon
the
type
of
questionnaire
you
wish
to
distribute
and
the
type
of
public
involvement
activity
you
wish
to
acquire
feedback
for,
you
will
need
to
tailor
the
steps
of
this
checklist
accordingly.

112
Assessing
EPA's
Public
Involvement
Activities
Checklist
for
Administering
the
Questionnaire
and
Using
Results1
Review
questionnaires.

Determine
whether
a
questionnaire
should
be
administered.

Determine
what
questionnaire(
s)
should
be
administered.

Identify
one
person
to
be
responsible
for
coordinating
the
overall
dissemination
and
collection
of
the
questionnaire
and
to
input
the
data
into
the
Lotus
1­
2­
3
data
input,
analysis,
and
reporting
program
once
the
questionnaires
are
collected.

Make
plans
to
include
the
questionnaire
feedback
process
in
the
agenda
of
your
public
involvement
event.
Specifically,
plan
to
set
aside
a
few
minutes
before
the
conclusion
of
your
public
involvement
event
and
have
participants
fill
out
their
questionnaires.
You
should
also
plan
to
make
an
announcement
at
the
beginning
of
your
public
involvement
event
explaining
the
questionnaire
feedback
process
and
how
feedback
is
needed
from
all
participants
in
order
to
develop
a
more
comprehensive
understanding
of
the
overall
effectiveness
of
the
event
and
how
best
to
make
improvements.
Feedback
is
needed
from
participants
who
leave
early
and
those
who
stay
to
the
end.
If
you
don't
build
the
questionnaire
feedback
process
directly
into
your
event,
and
instead
wait
to
the
very
end,
chances
are
few
participants
will
take
the
time
to
fill
out
their
questionnaires.
Participants
may
suggest
that
they
would
prefer
to
fill
out
their
questionnaires
at
home
or
work.
While
this
option
is
appealing
for
participants,
chances
are
only
a
few
will
take
the
time
to
actually
do
this.
The
best
chance
for
good
feedback
is
when
participants
are
actually
at
the
event.
Take
the
time
to
plan
for
this!

Estimate
the
number
of
participants
who
will
be
attending
the
public
involvement
event
for
which
questionnaires
will
be
distributed.

Make
appropriate
number
of
copies
of
the
questionnaire.

Bring
questionnaire
copies
and
extra
pens
or
pencils
to
the
public
involvement
event.

Identify
official
who
will
be
responsible
for
physically
collecting
the
questionnaires.

Clearly
identify
areas
where
participants
should
return
their
questionnaires.

Hand
out
questionnaires
with
other
public
involvement
activity
materials
either
at
the
beginning
of
the
event,
or
at
a
designated
time
during
the
event.
Draft
Draft
Draft
1Note:
Separate
Lotus
1­
2­
3
programs
have
been
developed
that
match
most
types
of
questionnaires
available
(
e.
g.,
one
for
single
event
effectiveness
for
participants,
one
for
overall
effectiveness
for
participants,
etc.).
Be
sure
to
use
the
appropriate
Lotus
1­
2­
3
program
to
input
your
data.
The
only
type
of
questionnaire
for
which
a
spreadsheet
program
has
not
been
developed
is
for
the
questionnaire
dealing
with
post­
event
follow­
up.
However,
if
you
need
a
spreadsheet
program
for
this
questionnaire,
contact
Pat
Bonner
(
bonner.
patricia@
epa.
gov).

113
Make
it
very
clear
to
participants
to
whom,
or
to
where,
they
should
return
their
questionnaire.
Urge
participants
to
fill
out
their
questionnaire
even
if
they
need
to
leave
early.

Identify
the
total
number
of
participants
attending
(
e.
g.,
the
total
number
of
persons
who
could
possibly
fill
out
questionnaires).

Collect
questionnaires.

Input
data
into
the
Lotus
1­
2­
3
data
input,
analysis,
and
reporting
program
within
one
month
of
session
and
share
results.

Detach
the
Lotus
1­
2­
3
program
onto
your
computer.
1
Open
the
Lotus
1­
2­
3
program.

Click
on
the
"
Getting
Started"
tab
and
follow
the
instructions.

Review
performance
report
and
identify
whether
certain
changes
should
be
considered
in
preparation
for
the
next
public
involvement
event.

Use
the
Lotus
1­
2­
3
performance
reports
to
talk
with
managers
and
staff
about
the
effectiveness
of
your
public
involvement
activity
and
where
changes
may
be
necessary.
Communicate
overall
results
and
what
changes
you
intend
to
make
to
participants
who
attended
your
last
public
involvement
event
in
the
public
involvement
event
summary
report
or
other
follow­
up
materials
distributed
to
participants.
A
few
brief
sentences
describing
overall
what
participants
thought
of
the
event,
their
suggestions
for
improvement,
and
what
you
will
do
to
improve
the
next
event
would
let
participants
know
that
you
valued
their
input.

Sharing
data...
EPA's
Public
Involvement
Policy
Implementation
Staff
strongly
encourages
you
to
send
your
completed
Lotus
1­
2­
3
program
files
as
e­
mail
attachments
to:
bonner.
patricia@
epa.
gov.
If
you
do
so,
Implementation
Staff
will
be
able
to
track
the
use
of
the
questionnaires,
and
learn
from
you
what
is
and
is
not
working
well,
and
gauge
whether
and
how
the
questionnaire
was
useful
to
your
efforts
to
improve
public
involvement
activities.
We
will
not
use
the
information
you
send
to
judge
the
strength
of
individual
public
involvement
efforts.

When
you
conduct
another
public
involvement
event,
repeat
the
same
steps
listed
above.
However,
after
doing
this,
be
sure
to
go
to
the
same
Lotus
1­
2­
3
spreadsheet
and
follow
the
steps
required
to
link
the
data
from
your
first
public
involvement
event
to
your
second
in
order
to
summarize
and
compare
your
data
over
a
period
of
time.
Draft
Draft
Draft
114
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
occasionally
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
OMB
CONTROL
NO:

EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
FACA
Committee
Meeting
Participant
Assessment
Background
This
questionnaire
is
designed
to
help
Agency
staff
better
understand
what
worked
well
and
what
improvements
to
consider
implementing
before
holding
future
Federal
Advisory
Committee
Act
Committee
(
FACA)
meetings.
Once
the
data
are
summarized,
Agency
staff
can
assess
whether
goals
for
success
were
met,
make
modifications
if
necessary,
and
compare
progress
over
time.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Stro
ngly
disa
gre
e
Dis
agr
ee
So
me
wha
t
disa
gre
e
So
me
wha
t
agr
ee
Agr
ee
Str
on
gly
ag
re
e
Do
n't
kn
ow
Meeting
objectives
were:

a)
clear
1
2
3
4
5
6
7
b)
achieved
1
2
3
4
5
6
7
I
was
given
enough
notice
to
attend
upcoming
meeting.
1
2
3
4
5
6
7
Enough
notice
is
___
days
in
advance
of
meeting.

Agenda
was
clear.
1
2
3
4
5
6
7
Agenda
was
used
to
conduct
meeting.
1
2
3
4
5
6
7
I
was
given
enough
time
to
review
materials
before
attending.
1
2
3
4
5
6
7
Enough
time
is
___
days
in
advance
of
meeting.

Meeting
was
run
effectively.
1
2
3
4
5
6
7
Background
materials
were:

a)
clear
1
2
3
4
5
6
7
b)
helpful
1
2
3
4
5
6
7
c)
written
at
an
appropriate
technical
level
1
2
3
4
5
6
7
I
prefer
more
less
technical
language.
Draft
Draft
Draft
115
Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
D
on
'
t
kn
o
w
Staff
support
was:

a)
helpful
1
2
3
4
5
6
7
b)
courteous
1
2
3
4
5
6
7
c)
accessible
1
2
3
4
5
6
7
d)
knowledgeable
1
2
3
4
5
6
7
Meeting
time
was
convenient.
1
2
3
4
5
6
7
Meeting
location
was
comfortable.
1
2
3
4
5
6
7
Meeting
length
was
appropriate.
1
2
3
4
5
6
7
Participants
could
express
their
points
of
view
and
be
heard.
1
2
3
4
5
6
7
Participation
in
meeting
was
worthwhile.
1
2
3
4
5
6
7
Please
provide
any
comments
or
suggestions
for
improving
the
next
FACA
meeting:

Please
indicate
which
group
you
represent:

neighborhood
group
local
non­
profit
organization
(
not
including
local
neighborhood
non­
profit
groups)

national
environmental
organization
business
or
industry
or
trade
association
state
or
tribal
or
local
government
federal
government
other
(
please
specify)

no
organizational
affiliation
(
e.
g.,
interested
local
resident)

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
116
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.
OMB
CONTROL
NO:
EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Effectiveness
of
a
FACA
Committee
Meeting
EPA/
Contractor
Assessment
Background
This
questionnaire
is
designed
to
obtain
an
EPA/
contractor
perspective
on
the
effectiveness
of
a
FACA
committee
meeting
or
set
of
meetings.
Agency
staff
will
use
this
information
to
better
understand
what
is
working
well
and
what
improvements
could
make
future
FACA
committee
meetings
more
successful.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
D
on
'
t
kn
o
w
Meeting
objectives
were
a)
clear
1
2
3
4
5
6
7
b)
achieved
1
2
3
4
5
6
7
Meeting
was
run
effectively.
1
2
3
4
5
6
7
Sufficient
technical
information
was
available.
1
2
3
4
5
6
7
EPA
participation
was
useful.
1
2
3
4
5
6
7
Participants
could
express
their
points
of
view
and
be
heard.
1
2
3
4
5
6
7
Participants
were
able
to
influence
meeting
outcome.
1
2
3
4
5
6
7
This
FACA
meeting
was
a
success.
1
2
3
4
5
6
7
Please
explain
your
ranking
of
the
previous
statement
about
success:

Please
provide
any
additional
comments
or
suggestions
for
improving
future
FACA
meetings:

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
117
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
occasionally
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.
OMB
CONTROL
NO:
EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
FACA
Committee
Process
Participant
Assessment
Background
This
questionnaire
is
designed
to
help
Agency
staff
better
understand
the
overall
effectiveness
of
a
FACA
committee
process.
Agency
staff
will
use
this
information
to
better
understand
what
is
working
well
and
what
improvements
could
make
this
FACA
committee
process
more
successful.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
D
on
'
t
kn
o
w
The
mission
of
the
committee
was
clearly
defined.
1
2
3
4
5
6
7
Committee
procedures
were
clearly
defined.
1
2
3
4
5
6
7
All
points
of
view
relevant
to
the
subject
and
function
of
the
committee
were
represented.
1
2
3
4
5
6
7
Committee
membership
had
necessary
expertise.
1
2
3
4
5
6
7
I
knew
what
was
expected
of
me
on
the
committee.
1
2
3
4
5
6
7
The
overall
process
was:
1
2
3
4
5
6
7
a)
well
organized
1
2
3
4
5
6
7
b)
efficient
1
2
3
4
5
6
7
c)
productive
1
2
3
4
5
6
7
d)
participatory
1
2
3
4
5
6
7
Members
influenced:
1
2
3
4
5
6
7
a)
the
meeting
logistics/
process
1
2
3
4
5
6
7
b)
the
meeting
process
1
2
3
4
5
6
7
c)
the
group's
report
or
recommendations
1
2
3
4
5
6
7
Members
had
access
to
necessary
information.
1
2
3
4
5
6
7
Draft
Draft
Draft
118
Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
D
on
'
t
kn
o
w
The
resource
needs
(
staffing/
contracting)
of
the
committee
were
met.
1
2
3
4
5
6
7
Committee
fulfilled
its
original
mandate.
1
2
3
4
5
6
7
Committee
produced
carefully
considered
recommendations.
1
2
3
4
5
6
7
Personal
participation
was
worthwhile.
1
2
3
4
5
6
7
Committee
had
sufficient
feedback
from
Agency
as
it
developed
its
recommendations.
1
2
3
4
5
6
7
The
overall
process
resulted
in:
1
2
3
4
5
6
7
a)
more
trust
in
EPA
processes
1
2
3
4
5
6
7
b)
enhanced
understanding
of
different
viewpoints
1
2
3
4
5
6
7
c)
increased
ability
to
work
together
1
2
3
4
5
6
7
EPA
appreciated
the
committee's
work.
1
2
3
4
5
6
7
Please
provide
any
comments
or
suggestions
for
improving
this,
or
future,
FACA
processes:

Please
indicate
which
group
you
represent:

neighborhood
group
local
non­
profit
organization
(
not
including
local
neighborhood
non­
profit
groups)

national
environmental
organization
business
or
industry
or
trade
association
state
or
tribal
or
local
government
federal
government
other
(
please
specify)

no
organizational
affiliation
(
e.
g.,
interested
local
resident)

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
119
Directions
Rate
the
following
statements
on
a
scale
from
1­
7
and
write
your
own
opinion
to
questions
in
the
space
provided.
If
you
have
no
opinion,
circle
the
number
representing
"
Don't
know."
Please
return
your
questionnaire
to
an
EPA
official
before
you
leave.
OMB
CONTROL
NO:
EXPIRATION
DATE:

Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
FACA
Committee
Process
EPA/
Contractor
Assessment
Background
This
questionnaire
is
designed
to
obtain
an
EPA/
contractor
perspective
on
the
overall
effectiveness
of
a
FACA
committee
process.
Agency
staff
will
use
this
information
to
better
understand
what
is
working
well
and
what
improvements
could
make
this
FACA
process
more
successful.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!
Draft
Draft
Draft
120
Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
Don'
t
kno
w
The
mission
of
the
committee
was
clearly
defined.
1
2
3
4
5
6
7
Committee
procedures
were
clearly
defined.
1
2
3
4
5
6
7
All
points
of
view
relevant
to
the
subject
and
function
of
the
committee
were
represented.
1
2
3
4
5
6
7
Committee
membership
had
necessary
expertise.
1
2
3
4
5
6
7
The
overall
process
was:
1
2
3
4
5
6
7
a)
well
organized
1
2
3
4
5
6
7
b)
efficient
1
2
3
4
5
6
7
c)
productive
1
2
3
4
5
6
7
d)
participatory
1
2
3
4
5
6
7
Members
influenced:
1
2
3
4
5
6
7
a)
the
meeting
logistics
and
meeting
process
1
2
3
4
5
6
7
b)
schedule
for
completing
mission/
producing
product
1
2
3
4
5
6
7
c)
the
group's
report
or
recommendations
1
2
3
4
5
6
7
Members
had
access
to
necessary
information.
1
2
3
4
5
6
7
Draft
Draft
Draft
121
Str
on
gly
dis
agr
ee
Dis
agr
ee
So
me
wh
at
dis
agr
ee
So
me
wh
at
agr
ee
Agr
ee
St
ro
ng
ly
ag
re
e
D
on
'
t
kn
o
w
Committee
fulfilled
its
original
mandate.
1
2
3
4
5
6
7
Committee
produced
carefully
considered
recommendations.
1
2
3
4
5
6
7
Committee
had
sufficient
feedback
from
Agency
as
it
developed
its
recommendations.
1
2
3
4
5
6
7
The
overall
process
resulted
in:
1
2
3
4
5
6
7
a)
more
trust
in
EPA
processes
1
2
3
4
5
6
7
b)
enhanced
understanding
of
different
viewpoints
1
2
3
4
5
6
7
c)
increased
ability
to
work
together
1
2
3
4
5
6
7
EPA
appreciated
the
committee's
work.
1
2
3
4
5
6
7
The
FACA
process,
thus
far,
has
been
a
success.
1
2
3
4
5
6
7
Please
explain
your
ranking
of
the
previous
statement
about
success:

Please
provide
any
additional
comments
or
suggestions
for
improving
future
FACA
meetings:

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
Draft
Draft
Draft
122
Questionnaire
for
Feedback
on
the
Overall
Effectiveness
of
a
FACA
Committee
DFO
Assessment
Background
This
questionnaire
is
designed
to
obtain
a
DFO
perspective
on
the
overall
effectiveness
of
a
FACA
committee
by
examining
a
FACA
committee's
outputs.
Agency
staff
will
use
this
information
to
better
understand
what
is
working
well
and
what
improvements
could
be
make
this
FACA
process
more
successful.
Thanks
in
advance
for
taking
time
to
fill
out
the
questionnaire.
We
value
your
input!

1.
Name
of
Advisory
Committee
2.
Goals
of
Advisory
Committee
3.
Over
the
past
6
months,
please
describe
the
key
outputs
you
have
received
from
this
advisory
committee
(
e.
g.,
reports,
recommendations,
minutes,
etc.)
in
the
following
table
and
then
rate
them
on
the
following
measures:
timeliness,
relevance
to
your
agency's
needs,
quality,
and
completeness.

Ver
y
dis
sati
sfie
d
Dis
sati
sfie
d
So
me
wh
at
dis
sati
s.
So
me
wh
at
sati
sfie
d
Sat
isfi
ed
V
er
y
sa
tis
fie
d
D
on
'
t
kn
o
w
Output
1:

a)
Timeliness
1
2
3
4
5
6
7
b)
Relevance
to
your
agency's
needs
1
2
3
4
5
6
7
c)
Quality
1
2
3
4
5
6
7
d)
Completeness
1
2
3
4
5
6
7
Draft
Draft
Draft
123
Ver
y
dis
sati
sfie
d
Dis
sati
sfie
d
So
me
wh
at
dis
sati
s.
So
me
wh
at
sati
sfie
d
Sat
isfi
ed
V
er
y
sa
tis
fie
d
D
on
'
t
kn
o
w
Output
2:

a)
Timeliness
1
2
3
4
5
6
7
b)
Relevance
to
your
agency's
needs
1
2
3
4
5
6
7
c)
Quality
1
2
3
4
5
6
7
d)
Completeness
1
2
3
4
5
6
7
Output
3:
1
2
3
4
5
6
7
a)
Timeliness
1
2
3
4
5
6
7
b)
Relevance
to
your
agency's
needs
1
2
3
4
5
6
7
c)
Quality
1
2
3
4
5
6
7
d)
Completeness
1
2
3
4
5
6
7
Output
4:

a)
Timeliness
1
2
3
4
5
6
7
b)
Relevance
to
your
agency's
needs
1
2
3
4
5
6
7
c)
Quality
1
2
3
4
5
6
7
Output
5:
1
2
3
4
5
6
7
a)
Timeliness
1
2
3
4
5
6
7
b)
Relevance
to
your
agency's
needs
1
2
3
4
5
6
7
c)
Quality
1
2
3
4
5
6
7
d)
Completeness
1
2
3
4
5
6
7
Overall,
how
satisfied
are
you
with
the
outputs
over
the
past
6
months?
1
2
3
4
5
6
7
4.
Do
you
plan
to
implement
any
of
the
findings
of
the
advisory
committee?

___
all
(
skip
to
#
6)
___
most
___
some
___
a
few
___
none
5.
If
you
did
not
answer
"
all"
in
#
6,
please
explain
why
you
do
not
plan
to
implement
all
of
the
Draft
Draft
Draft
124
findings:

6.
Please
provide
any
additional
comments
or
suggestions
for
improving
the
FACA
committee's
usefulness
or
effectiveness:

Date
completed:

Please
return
your
completed
questionnaire
to
an
EPA
official
before
you
leave.
Thank
you
for
helping
EPA
improve
its
public
involvement
practices.
