MARCH 2009 DOCUMENT DATA ANALYSIS FOR THE SCIENCE AND TECHNOLOGY FOR
SUSTAINABILITY (STS) RESEARCH PROGRAM

Analysis of Publication Use in Rulemaking, Policy Making, and
Decision-Making

 Summary of Document Data Analysis

Background

The Document Data Analysis (DDA) is a companion to the Bibliometric
Analysis for the Science and Technology for Sustainabiity (STS) Research
Program and it is designed to determine how the Program’s products are
being used by the Office of Research and Development’s (ORD) primary
partners for EPA rulemaking and policy making. The Bibliometric Analysis
focuses on the citation of the Program’s publications in the
peer-reviewed scientific literature; the DDA focuses on the use of the
Program’s publications in rule-, policy-, and decision-making.  The
DDA will serve as an indicator of the relative quality, relevance, and
impact of the science produced by the Program.  

This DDA report is presented in five sections.  The first section
provides a summary of the results of the analysis.  The second section
identifies those STS Program publications that have been cited by EPA
program offices or regions in rulemaking, policy, guidelines, and other
decision-making documents, as well as the risk assessments prepared by
ORD.  The third section presents the results of a Google search of a
relatively small subset of the STS Program’s publications to determine
their use by federal, state, and local governments; the international
community; academia; associations; and others.  The fourth section of
the report identifies the STS Program’s publications that have been
cited in the Toxicological Profiles of the Agency for Toxic Substances
and Disease Registry (ATSDR).

Process

National Program Directors (NPDs) provide a list of all program journal
articles and products to the National Center of Environmental Research
Center (NCER).  NCER utilizes a software (datamining) tool to determine
which research publications are cited by EPA program offices or regions
and, as appropriate, in NCEA risk assessments. To provide additional
context for assessing the validity of the DDA results for demonstrating
program performance, a Google search and analysis of a relatively small
subset of the research products is performed to determine the extent to
which other federal, state, and local agencies, as well as the
international community, academia, associations, and others may be
citing ORD research.

Summary of EPA Program and Regional Office Decision Documents Citing STS
Program Publications (Results of Datamining Search of EPA Web Sites)

Total Number of STS Program Publications Searched Using Tool:  865 

794 journal publications and 71 non-journal publications (books, book
chapters, EPA reports)

Total Number and Percent of STS Program Publications Cited:  19 (2.2%)

Total Number and Percent of STS Program Publications Cited Excluding
Citations by ORD Documents:  3 (0.3%)

Number and Percent of STS Program Publications Cited Two or More Times: 
1 (0.1%)

Total Number of EPA Decision Documents Citing STS Program Publications: 
10 (the 19 publications were cited 20 times in the 10 publications and 1
document cited 10 of the STS Program publications)

Total Number of EPA Decision Documents Citing STS Program Publications
Excluding Citations by ORD Documents:  3

Types of EPA Decision Documents Citing STS Program Publications: 2
Federal Rulemaking/ Regulatory Publications and 8 Technical/Guidance
Documents.

Summary of Federal and Other Governmental/Organizational Resources
Citing STS Program Publications (Google/Google Scholar Search of Subset
of STS Program Publications) 

Total Number of STS Program Publications Searched: 30 (27 journal
publications and 3 non-journal publications selected at random) 

Total Number and Percent of the 30 STS Program Publications Cited:  3
(10%) (the 3 publications were cited 5 times by 5 different decision
resources) 

Number and Percent of the 30 STS Program Publications Cited Two or More
Times:   1 (3.3%)

Total Number of Publications Citing the 30 STS Program Publications:  5
(the 3 STS Program publications were cited by 5 different publications)

Total Number of Publications Citing the 30 STS Program Publications
Excluding ORD Publications:  3 

Types of Publications Citing the 30 STS Program Publications:  4
International documents (including 1 Government Policymaking Document, 2
Government Technical/Guidance documents, and 1 International World
Health Organization Technical/Guidance document) and 1 U.S. Federal
Government Technical/Guidance document.

Summary of ATSDR Toxicological Profile Search

Total Number of STS Program Publications Searched: 865 (794 journal and
71 non-journal publications)

Total Number and Percent of the STS Program Publications Cited in ATSDR
Toxicological Profiles:  1 (0.1%) 

Number and Percent of the STS Program Publications Cited Two or More
Times in ATSDR Toxicological Profiles:  0 (0%)

Total Number of ATSDR Toxicological Profiles Citing the STS Program
Publications:  1 

II. EPA Program and Regional Office Decision Documents Citing STS
Program Publications—Results of the Datamining Search of EPA Web Sites

This section identifies those STS Research Program publications that
have been cited by EPA program offices or regions in rulemaking, policy,
guidelines, and other decision-making documents, as well as the risk
assessments prepared by ORD’s National Center for Environmental
Assessment (NCEA).  Nineteen of the STS Program publications were cited
by 10 different decision documents.  The results are tallied and
summarized in Table 1.  The citing publications are listed in Table 2
and the publication numbers in the first column of Table 2 are
hyperlinked to the citations of the STS Program publications being
cited, which provided in Appendix A.

Table 1.  Tally Summary of STS Program Publications Cited by EPA Program
Offices and Regions in Decision Documents

Data on STS Program Publications

Total Number of STS Program Publications Searched Using Tool	865 (794
journal publications and 71 non-journal publications

Total Number and Percent of STS Program Publications Cited in EPA
Decision Documents	19 (2.2%)

Total Number and Percent of STS Program Publications Cited in EPA
Decision Documents Excluding Citations by ORD Documents	3 (0.3%)

Number and Percent of STS Program Publications Cited Two or More Times
in Decision Documents	1 (0.1%)

Data on EPA Decision Documents Citing STS Program Publications

Total Number of EPA Decision Documents Citing STS Program Publications
10 (the 19 publications were cited 20 times in the 10 publications and 1
document cited 10 of the STS publications)

Total Number of EPA Decision Documents (Excluding the ORD Documents)
Citing STS Program Publications	3

Types of EPA Decision Documents Citing STS Program Publications	2
Federal Rulemaking/Regulatory Publications and 8 Technical and Guidance
Documents



Table 2.  EPA Program Office and Regional Office Decision Documents
Citing 

STS Program Publications

Pub #		Cited in	Type of Citing Pub	Office

  HYPERLINK  \l "A1"  1 ,   HYPERLINK  \l "A2"  2 ,   HYPERLINK  \l "A3"
 3 ,   HYPERLINK  \l "A4"  4 ,   HYPERLINK  \l "A5"  5 ,   HYPERLINK  \l
"A6"  6 ,   HYPERLINK  \l "A7"  7 ,   HYPERLINK  \l "A8"  8 ,  
HYPERLINK  \l "A9"  9 ,   HYPERLINK  \l "A10"  10 

	  HYPERLINK
"http://www.epa.gov/ord/NRMRL/std/cppb/greenchem/greenchemaltsys.htm" 
Alternative Synthesis and Pathways | Green Chemistry & Engineering |
Clean Processes | Sustainable Technology | Risk Management Research |
Research and Development | US EPA   “Alternative Chemical Synthetic
Pathways” Dr. Rajender Varma, US Environmental Protection Agency,
Office of Research and Development, National Risk Management Research
Laboratory,

(Online Publication)   	Technical/ Guidance	NRMRL

  HYPERLINK  \l "A11"  11 ,  HYPERLINK  \l "A12"  12 	  HYPERLINK
"http://www.epa.gov/nrmrl/std/cppb/pervapor/"  Pervaporation-based
Separation Processess | Clean Processes | Sustainable Technology | Risk
Management Research | Research and Development | US EPA  – US EPA/ ORD
– Pervaporation-based Separation Processes for Pollution Prevention
and Waste Treatment (Online Publication)	Technical/ Guidance	NRMRL

  HYPERLINK  \l "A13"  13 	  HYPERLINK
"http://www.epa.gov/nrmrl/std/mtb/p2tools/cape.htm"  CAPE-OPEN |
Pollution Prevention (P2) Research Tools | Industrial Multimedia |
Sustainable Technology | Risk Management Research | Research and
Development | US EPA .   US EPA/ORD. Computer Aided Process Engineering
– Online Pollution Prevention Research Tool	Technical/ Guidance	OIAA

  HYPERLINK  \l "A14"  14 	  HYPERLINK
"http://www.epa.gov/EPA-WATER/2003/August/Day-11/w18295c.htm"  EPA:
Federal Register: National Primary Drinking Water Regulations: Long Term
2 Enhanced Surface Water Treatment Rule  - Federal Register: Volume  68,
Number 154, Page 47739-47788, August 11, 2003. [Proposed Rules]	Federal
Rulemaking/ Regulatory	OIAA

  HYPERLINK  \l "A14"  14 	  HYPERLINK
"http://www.epa.gov/EPA-WATER/2006/January/Day-05/w04c.htm"  EPA:
Federal Register: National Primary Drinking Water Regulations: Long Term
2 Enhanced Surface Water Treatment Rule   - Federal Register: January 5,
2006 (Volume 71, Number 3) – Rules and Regulations	Federal Rulemaking/
Regulatory	OIAA

  HYPERLINK  \l "A15"  15 	  HYPERLINK
"http://www.epa.gov/nrmrl/std/cppb/qsar/"  QSAR | Sustainable Technology
| US EPA  – US EPA/ORD – NRMRL – (online publication) - QSAR
models are useful for estimating toxicities needed for green process
design algorithms such as the   HYPERLINK
"http://www.epa.gov/nrmrl/std/cppb/war/sim_war.htm"  Waste Reduction
Algorithm 	Technical/ Guidance	NRMRL

  HYPERLINK  \l "A16"  16 	  HYPERLINK
"http://cfpub.epa.gov/caddis/info_sources.cfm?Section=31"  Information
Sources | CADDIS | US EPA  – The publication below (#16) is listed as
a reference used in the development of the Stressor Identification
Guidance Document and the CADDIS Web site. CADDIS is an online
application that helps scientists and engineers in the regions, states
and tribes find, access, organize, use, and share information to conduct
causal evaluations in aquatic systems. It is based on the U.S.
Environmental Protection Agency   HYPERLINK
"http://www.epa.gov/waterscience/biocriteria/stressors/stressorid.html" 
Stressor Identification  process. US EPA/NCEA (online Publication)
Technical/ Guidance	NCEA

  HYPERLINK  \l "A17"  17 	  HYPERLINK
"http://www.epa.gov/nrmrl/std/cppb/war/sim_war.htm"  WAR Algorithm |
Sustainable Technology | US EPA   US. EPA/ORD.  Chemical Process
Simulation for Waste Reduction  (online publication)	Technical/ Guidance
NRMRL

  HYPERLINK  \l "A18"  18 	  HYPERLINK
"http://www.epa.gov/nrmrl/std/mtb/metal_finishing.htm"  Metal Finishing
| Industrial Multimedia | Sustainable Technologies | Risk Management
Research | Research and Development | US EPA    US EPA/ORD. Guide to
Pollution Prevention in Metal Finishing  (online publication)	Technical/
Guidance	NRMRL

  HYPERLINK  \l "A19"  19 	  HYPERLINK
"http://www.epa.gov/nrmrl/std/mtb/paris.htm"  Program for Assisting the
Replacement of Industrial Solvents PARIS II | Industrial Multimedia |
Sustainable Technologies | Risk Management Research | Research and
Development | US EPA    US EPA/ORD. PARIS II - Computer Aided Solvent
Design For Pollution Prevention (online publication)	Technical/ Guidance
NRMRL





Federal and Other Governmental/Organizational Resources Citing STS
Program Publications — Results of Google/Google Scholar Searches of 

30 STS Program Publications

This section presents the results of a Google search of 30 of the STS
Program’s publications to determine their use by federal, state, and
local governments; the international community; academia; associations;
and others.  Three of the 30 STS Program publications searched were
cited and the results are tallied and summarized in Table 3 below. The
citing resources are listed in Table 4 and the publication numbers in
the first column of Table 4 are hyperlinked to the citations of the STS
Program publications being cited provided in Appendix B.

Table 3.  Tally Summary of STS Program Publications Cited in Federal and
Other Governmental/Organizational Decision Resources

Data on STS Program Publications Searched

Total Number of STS Program Publications Searched Using Google/Google
Scholar	30 (includes 27 journal publications and 3 non-journal
publications selected at random)  

Total Number and Percent of the 30 STS Program Publications Cited in
Decision Resources	3 (10.0%)—the 3 publications were cited 5 times by
5 different decision resources

Number and Percent of the 30 STS Program Publications Cited Two or More
Times	1 (33.3%)

Data on Federal and Other Governmental/Organizational Decision Resources
Citing STS Program Publications

Total Number of Federal and Other Governmental/Organizational Decision
Resources Citing the 30 STS Program Publications 	5 (3 of the 30 STS
Program publications were cited 5 times by 5 different decision
resources; none of the citing publications were ORD documents)

Types of Publications Citing the 30 STS Publications	1 U.S. Federal
Resource—National Institute for Occupational Safety and Health
Technical/Guidance Document

4 International Resources—1 Government Policymaking Document, 2
Government Technical Guidance Documents, and 1 World Health Organization
Technical/Guidance Document



Table 4.  Federal and Other Governmental/Organizational Decision
Resources Citing 

STS Program Publications

Pub #		Cited in	Type of Citing Pub

  HYPERLINK  \l "B1"  1 	Science and Technology Foresight Center.  The
8th Science and Technology Foresight Survey. Study on Rapidly-developing
Research Area.  National Institute of Science & Technology Policy
(NISTEP), Ministry of Education, Culture, Sports, Science & Technology
(MEXT), June 2005.

(  HYPERLINK
"http://www.nistep.go.jp/achiev/ftx/jpn/rep082j/pdf/rep082j.pdf" 
http://www.nistep.go.jp/achiev/ftx/jpn/rep082j/pdf/rep082j.pdf )
International Policymaking

  HYPERLINK  \l "B2"  2 	Health Canada.  Guidelines for Canadian
Drinking Water Quality:  Supporting Documentation.  Protozoa:  Giardia
and Cryptosporidum.  Federal-Provincial-Territorial Committee on
Drinking Water of the Federal-Provincial-Territorial Committee on Health
and the Environment, Ottawa, Ontario, April 2004.

(  HYPERLINK
"http://www.hc-sc.gc.ca/ewh-semt/alt_formats/hecs-sesc/pdf/pubs/water-ea
u/protozoa/protozoa.pdf" 
http://www.hc-sc.gc.ca/ewh-semt/alt_formats/hecs-sesc/pdf/pubs/water-eau
/protozoa/protozoa.pdf )	International Technical/ Guidance

  HYPERLINK  \l "B2"  2 	New Zealand Ministry of Health.  The Guidelines
for Drinking-water Quality Management for New Zealand, October 2005.
International Technical/ Guidance

  HYPERLINK  \l "B2"  2 	Aertgeerts R, Angelakis A, eds.  State of the
Art Report:  Health Risks in Aquifer Recharge Using Reclaimed Water.
World Health Organization, SDE/WSH/03.08, 2003 (  HYPERLINK
"http://www.who.int/water_sanitation_health/wastewater/wsh0308/en/index.
html" 
http://www.who.int/water_sanitation_health/wastewater/wsh0308/en/index.h
tml )	International Technical/ Guidance

  HYPERLINK  \l "B3"  3 	National Institute for Occupational Safety and
Health.  Emerging Technologies and the Safety and Health of Working
People:  Knowledge Gaps and Research Directions.  Centers for Disease
Control and Prevention, NIOSH Publication No. 2006-136, August 2006.

(  HYPERLINK "http://www.cdc.gov/niosh/docs/2006-136/pdfs/2006-136.pdf" 
http://www.cdc.gov/niosh/docs/2006-136/pdfs/2006-136.pdf )	Federal
Technical/ Guidance



ATSDR Toxicological Profiles Citing STS Program Publications

This section identifies those STS Program publications that have been
cited in Agency for Toxic Substances and Disease Registry (ATSDR)
Toxicological Profiles.

One of the STS Program publications was cited by one ATSDR Toxicological
Profile.  The results are presented in Table 5.  The citing ATSDR
Toxicological Profile is listed in Table 6.  

Table 5.  Tally Summary of STS Program Publications Cited in ATSDR
Toxicological Profiles 

Data on STS Program Publications

Total Number of STS Program Publications Searched Using Tool	865 (794
journal publications and 71 non-journal publications

Total Number and Percent of STS Program Publications Cited in ATSDR
Profiles	1 (0.1%)

Number and Percent of STS Publications Cited Two or More Times in ATSDR
Toxicological Profiles 	0 (0%)

Data on ATSDR Toxicological Profiles Citing STS Program Publications

Total Number of ATSDR Toxicological Profiles Citing STS Program
Publications	1 



Table 6.  ATSDR Toxicological Profiles Citing STS Publications

STS Program Publication Cited 		ATSDR Profile Citing the STS Publication

Kominsky JR, Freyberg RW, Clark PJ, Edwards A, Wilmoth RC, Brackett KA. 
Asbestos exposures during routine floor tile maintenance.  Part I: 
Spray-buffing and wet-stripping.  Applied Occupational and Environmental
Hygiene 1998;13(2):101-106.	Toxicological Profile for Asbestos.  U.S.
Department of Health and Human Services, Public Health Service, Agency
for Toxic Substances and Disease Registry, September 2001.

  HYPERLINK "http://www.atsdr.cdc.gov/toxprofiles/tp61.pdf" 
http://www.atsdr.cdc.gov/toxprofiles/tp61.pdf 



Appendix A:  STS Research Program Publications Cited by 

EPA Decision Documents

 Yang XF, Wang MW, Varma RS, Li CJ. Aldol- and Mannich-type reactions
via in situ olefin migration in ionic liquid. Organic Letters
2003;5(5):657-660.

  HYPERLINK  \l "R1"  [Return to Table] 

Keh CCK, Namboodiri VV, Varma RS, Li CJ. Direct Formation of
tetrahydropyranols via catalysis in ionic liquid. Tetrahedron Letters
2002;43(28):4993-4996.

  HYPERLINK  \l "R2"  [Return to Table] 

Patonay T, Varma RS, Vass A, Levai A, Dudas J. Highly diastereoselective
Michael reaction under solvent-free conditions using microwaves: 
conjugate addition of flavanone to its chalcone precursor. Tetrahedron
Letters 2001;42(8):1403-1406.

  HYPERLINK  \l "R1"  [Return to Table] 

Loupy A, Varma RS. Microwave effects in organic synthesis - Mechanistic
and reaction medium considerations. Chimica Oggi-Chemistry Today
2006;24(3):36+.

  HYPERLINK  \l "R1"  [Return to Table] 

Ju Y, Varma RS. Aqueous N-heterocyclization of primary amines and
hydrazines with dihalides:  microwave-assisted syntheses of
N-azacycloalkanes, isoindole, pyrazole, pyrazolidine, and phthalazine
derivatives. Journal of Organic Chemistry 2006;71(1):135-141.

  HYPERLINK  \l "R1"  [Return to Table] 

Smuleac V, Butterfield DA, Sikdar SK, Varma RS, Bhattacharyya D.
Polythiol-functionalized alumina membranes for mercury capture. Journal
of Membrane Science 2005;251(1-2):169-178.

  HYPERLINK  \l "R1"  [Return to Table] 

Varma RS. Solvent-free accelerated organic syntheses using microwaves.
Pure and Applied Chemistry 2001;73(1):193-198.

  HYPERLINK  \l "R1"  [Return to Table] 

Varma RS. Greener organic syntheses under non-traditional conditions.
Indian Journal of Chemistry Section B-Organic Chemistry Including
Medicinal Chemistry 2006;45(10):2305-2312.

  HYPERLINK  \l "R1"  [Return to Table] 

Varma RS. Clay and clay-supported reagents in organic synthesis.
Tetrahedron 2002;58(7):1235-1255.

  HYPERLINK  \l "R1"  [Return to Table] 

Vass A, Dudas J, Toth J, Varma RS. Solvent-free reduction of aromatic
nitro compounds with alumina-supported hydrazine under microwave
irradiation. Tetrahedron Letters 2001;32(42):5347-5349.

  HYPERLINK  \l "R1"  [Return to Table] 

Abou-Nemeh I, Majumdar S, Saraf A, Sirkar KK, Vane LM, Alvarez FR,
Hitchens L. Demonstration of pilot-scale pervaporation systems for
volatile organic compound removal from a surfactant enhanced aquifer
remediation fluid. II: Hollow fiber membrane modules. Environmental
Progress 2001;20(1):64-73.

  HYPERLINK  \l "R11"  [Return to Table] 

Alvarez FR, Vane LM, Hitchens L. Demonstration of pilot-scale
pervaporation systems for volatile organic compound removal from a
surfactant enhanced aquifer remediation fluid. I: Spiral wound membrane
modules. Environmental Progress 2001;20(1):53-63.

  HYPERLINK  \l "R11"  [Return to Table] 

Barrett WM, Harten P. Development of the Metal Finishing Facility Risk
Screening Tool (MFFRST). Plating and Surface Finishing 2002;89(9):77-84.

  HYPERLINK  \l "R13"  [Return to Table] 

Bukhari Z, Hargy TM, Bolton JR, Dussert B, Clancy JL. Medium-pressure UV
for oocyst inactivation. Journal of the American Water Works Association
1999;91(3):86-94.

  HYPERLINK  \l "R14a"  [Return to Table] 

Martin TM, Harten P, Venkatapathy R, Das S, Young DM. A hierarchical
clustering methodology for the estimation of toxicity. Toxicology
Mechanisms and Methods 2008;18(2-3):251-266.	

  HYPERLINK  \l "R15"  [Return to Table] 

Walsh CJ, Roy AH, Feminella JW, Cottingham PD, Groffman PM, Morgan RP.
The urban stream syndrome: current knowledge and the search for a cure.
Journal of the North American Benthological Society 2005;24(3):706-723.

  HYPERLINK  \l "R16"  [Return to Table] 

Young DM, Cabezas H. Designing sustainable processes with simulation:
the waste reduction (WAR) algorithm. Computers & Chemical Engineering
1999;23(10):1477-1491.

  HYPERLINK  \l "R17"  [Return to Table] 

Chen AS, Stencel N, Ferguson D. Using ceramic membranes to recycle two
nonionic alkaline metal-cleaning solutions. Journal of Membrane Science
1999;162(1-2):219-234.

  HYPERLINK  \l "R18"  [Return to Table] 

Cabezas H, Harten PF, Green MR. Designing greener solvents. Chemical
Engineering 2000;107(3):107-109.

  HYPERLINK  \l "R19"  [Return to Table] 

Appendix B:  STS Program Publications Cited by Federal and Other
Governmental/Organizational Decision Resources

Huddleston JG, Visser AE, Reichert WM, Wilauer HD, Broker GA, Rogers RD.
 Characterization and comparison of hydrophilic and hydrophobic room
temperature ionic liquids incorporating the imidazolium cation.  Green
Chemistry 2001;3(4):156-164.

  HYPERLINK  \l "RB1"  [Return to Table] 

Bukhari Z, Hargy TM, Bolton JR, Dussert B, Clancy JL.  Medium-pressure
UV for oocyst inactivation.  Journal of the American Water Works
Association 1999;91(3):86-94.

  HYPERLINK  \l "RB2"  [Return to Table] 

Anastas PT, Williamson TC, eds.  Green Chemistry:  Frontiers in Benign
Chemical Syntheses and Processes. Oxford:  Oxford University Press,
1998.

  HYPERLINK  \l "RB3"  [Return to Table] 

 

  PAGE   \* MERGEFORMAT  6 

