Evaluation of Effectiveness of Lead Safe Work Practices (LSWP) Training

Executive Summary

Authors:

Alexa Fraser, Ph. D.

Susan Viet, Ph. D., C.I.H.

March 23, 2004

Prepared for:

Department of Housing and Urban Development

Office of Healthy Homes and Lead Hazard Control

451 Seventh Street, SW

Room P3206

Washington, DC 20410	Prepared by:

WESTAT

Rockville, Maryland





EXECUTIVE SUMMARY

As part of its implementation plan for the Lead Safe Housing Rule (24
Part 35) and capacity building efforts, HUD’s Office of Healthy Homes
and Lead Hazard Control (OHHLHC) funded delivery of Lead Safe Work
Practices (LSWP) training nationwide to help develop a workforce trained
to work around lead-based paint safely and efficiently.  OHHLHC's goals
of ensuring accountability, improving work quality and efficiency, and
planning new programs require execution and periodic evaluation of
activities to indicate whether objectives of the initiative have been
met.  As a result, this systematic review and assessment of the
effectiveness of LSWP training was conducted. 

Methodology

The comprehensive design focused on evaluating both process and outcomes
for two curricula - the National Environmental Training Association
(NETA) Lead-based Paint Maintenance Training Program (Lead Maintenance
curriculum) and the joint HUD-EPA recently-developed curriculum Lead
Safety for Remodeling, Repair, and Painting (RRP curriculum) (without
exercises).  The first step was to evaluate the curriculum development
process.  This involved comparing the knowledge, skills, and abilities
(KSAs) and learning objectives (LOs) and mapping the LOs to the course
materials for each curriculum.  The second step was to train workers and
evaluate their performance on written tests and when conducting work on
lead-based paint (or presumed lead-based paint).  A total of 83 workers
were trained in Lead Maintenance, with 37 being observed.  A total of
108 workers were trained in RRP, with 36 being observed.  Various
internal and external factors were recorded in order to assess their
effect on test and job performance.  In addition, past and current
trainers of Lead Maintenance and RRR curricula were interviewed for
their insights about the effectiveness of the training.  The performance
of the trainees was then analyzed in relation to the curricula and
internal and external factors to determine whether training was valuable
and whether there was a transfer of training.

Overall Judgments about Effectiveness of Training

Overall, the two LSWP course curricula, Lead Maintenance and RRP, were
found to be effective in imparting the general knowledge needed as a
basis for performing lead-based paint related work in a safe manner. 
Trainees were satisfied with their new knowledge and performed well on
the posttests, showing a paper-an-pencil knowledge of LSWPs.  The RRP
curriculum is somewhat less effective than the Lead Maintenance
curriculum due to reduced regulation and health effect discussion and
poor audiovisuals.  We consider these courses to provide good and
effective awareness type training. 

However, neither curriculum prepared the trainee to perform actual work
in a lead safe manner.  Numerous lead-safe behaviors were not followed
by trainees following completion of both curricula.  While some external
factors affected their behavior, the LSWP steps are relatively simple
such that had the trainees understood, they could still have easily
incorporated most of the steps despite such external factors.   Where
external factors were not a problem, e.g., among landlords, an equal
number of poor behaviors were observed. 

We could not confidently address one of HUD’s critical questions: Can
the trainees pass clearance?  Most jobs were not cleared (90% of Lead
Maintenance jobs and 63% of RRP jobs) when it was required and even more
often (e.g., small jobs) it was not required.  For those that were
cleared, it was doubtful that the clearance was properly performed. 

Recommendations for Modifying Curriculum

A number of recommendations for modifying the curriculum should be
considered based on the study findings.  Some of the recommendations are
repeated below as they were identified in multiple evaluation
components. 

From Process Evaluation findings: 

Review KSA-LO mapping deficiencies.  Revise course materials to include
LOs that address each KSA and add or revise materials to more adequately
address KSAs where the LO is not addressed by the course materials.

Require a hands-on component – Good course design principles suggest
that “how to” topics are more effectively transferred via
demonstration and/or hands-on format.  

Review and revise LOs and/or performance evaluation criteria – The
evaluation criteria should align with stated LOs and goals for the
training.  If the goal is to be able to perform LSWP and Pass clearance,
then the performance criteria must include performing and passing.  This
suggests a hands-on type performance evaluation.  

From the Value of Training findings

Lead Maintenance – Trainees suggested increasing the font size in the
course materials, and providing more practice and hands-on experience
during class time.

RRP- Trainees suggested more animated visuals and providing more
practice and hands-on experience during class time.  Slow readers and
people with less education require graphics rather than text to learn
effectively.

Reduce the reading level of important appendix materials – Planning a
job, HUD requirements, Pre-renovation rule.  While we can assume that
comprehension is enhanced through the trainer’s explanations and
visuals, all materials provided should be at the same level -  “Plain
English” - particularly in the case of the RRP texts.

From the Internal/External Factors findings:

More than half of the past and current course trainers mentioned the
need for a hands-on training segment or optional workshops.  A few
mentioned that some sort of combination of the best components from the
LM and RRR courses would result in a stronger training.  

In addition, trainers believe that all trainees should receive a copy of
both the planning tool and the field guide, as well as the EPA pamphlet,
regardless of which course they attend. 

Specify conditions for the classroom environment – While we specified
and worked toward very specific conditions for this evaluation, trainees
commented on such things as room lighting and temperature.  We did not
see an affect on learning due to these conditions.  However, trainers
reported that some internal factors, e.g., number of trainees, do affect
training effectiveness. 

From the Transfer of Training findings:

Add new text to clearly explain some of the terms and key concepts that
were identified to be confusing – abatement, LWSP training
requirements, clearance. 

Include an LSWP equipment/supplies kit and HEPA vacuum, and perhaps zip
poles as mandatory demonstration – The field observations showed us
that the trainees did not understand what some of the supplies were
based on a list, line drawings, or quick photos.  

Include a hands-on component for trainees – Field observations showed
that trainees did not follow many of the safe work practices.  

Note:  This recommendation will either require additional time in the
classroom (i.e., another half to full day) or a complete revamping of
course format to focus on hands-on, while learning the knowledge
information as part of a full day hands-on exercise.

The one recommendation that was made during every component in the
evaluation, from process evaluation to trainee and past and current
trainer comments to observations of work following training, was the
need for hands-on training.  HUD should seriously consider this
recommendation.  The exercises presented in the RRP curricula are not
hands-on training. 

Other Recommendations

Additional recommendations that arose during the project, but are not
aimed at modification of existing curriculum, included:

Develop package for putting on course for potential sponsors – There
are still many untrained individuals who require LSWP training.  We
found that sponsors appreciated receiving help with putting on the LSWP
courses.  A package that might get more people trained would useful. 
This could include sample advertising flyers and letters, training room
criteria, clear definition of people who must be trained, registration
forms, sign-up sheets, etc. 

Training Other Populations – It is important to train other
populations who conduct LWSP work.  During this evaluation, we had a
number of Spanish-speaking trainees who, while they had a basic
understanding of spoken English, had a difficult time understanding all
the English text materials.  Training other populations requires more
than simply translating the materials however.  Some recommendations
include: 

Develop Spanish versions of materials, train Spanish-speaking trainers,
and evaluate whether other modifications are needed through pilot
trainings. 

Develop as many self-explanatory visuals as possible – for use with
all courses, but especially for trainees with different languages
(Spanish is not the only other language).

Identify and outreach to other populations people who need the training,
e.g., rural and tribal Populations. 

Require Supervisors be trained in LSWP as well – If supervisors of
workers who work with lead-based paint take these courses as well, there
may be more supervisor support and supply of correct supplies. 

Develop a technical bulletin that clearly describes the rules for who
needs LSWP training and when and how clearance testing must be done. 

Develop separate course for sponsors, lead grantee administrators, etc.
– Such a course would focus not so much on details of work as on what
is required. 

Some sort of LSWP enforcement – While we recognize that this could be
an expensive proposition, perhaps some modest enforcement type activity
could be employed, e.g., requiring a copy of the LWSP training
certificate for workers on jobs that are HUD funded, or a signed
statement by lead grantees that all jobs are conducted by trained
workers and cleared by licensed risk assessors. 

 PAGE   

 PAGE   ii 

 PAGE   

ES- PAGE   5 

