MEETING MINUTES
WORKPLACE DISPUTES SECTION
JUNE 15, 1999
EVALUATION OF AN ADR PROGRAM: PART TWO
The Workplace Dispute Section session,
AEvaluation of an ADR Program: Part Two,@ was held on Tuesday, 15 June 1999 in the Auditorium at the Federal Deposit Insurance Corporation, 801 17th Street N.W., Washington, D.C. The session was chaired by Martha McClellan, Counsel, FDIC, and attended by representatives of 20 federal agencies. The session was opened at 9:15 a.m. and closed at 11:30 a.m.
INTRODUCTION AND OVERVIEW OF SESSION
Martha McClellan
Counsel
FDIC
Welcomed the group then explained the purpose and process of the meeting.
_
Participants are informed of upcoming Workplace Disputes Section meetings sponsored by FDIC and USPS. They are also reminded that the Department of Justice will soon be sending out a letter to all agencies requesting an end of the year report describing their new or improved ADR program._
The purpose of this session is to provide an opportunity for participants to consult with ADR evaluation experts on specific issues that concern agencies that are either beginning or in the process of evaluating their ADR programs.
GROUP CONSULTATIONS
For the first half of the session, attendees participated in one of five pre-chosen consultation groups, each of which covered a separate evaluation topic. An expert in the field of evaluation lead each group with a brief presentation on their particular topic, after which the discussion was opened up for individual questions. After approximately 50 minutes, participants changed groups for the second round of consultations.
I. Pre-Design Issues
Identifying goals and audience, choosing strategy
Martha McClellan
Counsel
Federal Deposit Insurance Corporation
Round One Participants:
_
Patricia Abdullah_
Dan Callahan_
Karen Kimball_
Cindy Mazur_
Edward Novak
Round Two Participants:
_
Alberto Cornejo_
Joanne Crowder_
Emily Hickey_
Gary Lawson_
Alan Weinstein
Group Consultation Notes:
Pre-design = planning + preparation
Issues:
_
Determining Goals/Objectives_
Identifying Audiences_
Determining How results are to be used_
Considerations of Time/Expense_
Selecting an Evaluator
VI. Data Gathering I:
Surveys, questionnaires, focus groups, interviews
Dolores Crawford
Mediation Program Manager
General Accounting Office
Round One Participants:
_
Carolyn Brooks_
Regina Budd_
Fred Conway_
Alberto Cornejo_
Joanne Crowder_
Alan Weinstein
Round Two Participants:
_
Patricia Abdullah_
Eileen Connolly_
Sharon Hogan_
Karen Kimball_
Sheila Walcott
Group Consultation Notes:
How do you collect data?
_
Individuals-
Interview-
Focus Group-
Survey_
Documents-
Records-
Case files-
Accounting records-
Existing studies_
Observation
XI. Data Gathering II:
Surveys, questionnaires, focus groups, interviews
Kate Dorrell
Training Officer
Department of Labor
Round One Participants:
_
Nicola Goren_
Tanya Luckett_
Paula Small
Round Two Participants
_
Roderick Locklear_
Cindy Mazur_
Edward Novak_
Kathy Recker
Group Consultation Notes:
Data Uses:
_
Marketing_
Manage_
Resources_
Improve
Data Sources
_
Documents-
Case files-
Program plan-
Budget/staffing-
Case tracking data_
People-
Participants-
Mediators-
Program staff-
Other stakeholders-
Experts_
Tools-
Surveys-
Participant interviews-
Focus groups-
Post evaluations
_
What and how to capture to evaluate Y
|
Criterion |
Measure |
Data Point/Source |
|
Efficiency |
_ time_ people_ cost |
|
|
Effectiveness |
Outcomes Durability Effect |
|
|
Satisfaction with Y |
Process Relationship Outcome |
|
|
Administration (Does the design work?) |
-organization responsibilities -access understanding -quality
|
_
If program is facing resistance, use focus groups to find out what the resistance is. If departmental cooperation is a problem, gather data to sell a global approach._
After producing a survey, test it before implementing its use in the program. Hand out copies to 20 friends, then set up a spreadsheet to see if the questions can generate data that is easy to enter. If not, change the questions._
Always use short questionnaires. Using ten of the most important questions is usually best.
IV. Data Analysis:
Performance indicators, quantitative, qualitative
Tom Louthan
Director, ADR Programs
National Office Appeals
Round One Participants:
_
Sharon Hogan_
Gary Lawson_
Beverly Moore_
Dominica Parks
_
Kathy Recker_
Linda Williams
Round Two Participants:
_
Bob Amrhein_
Regina Budd_
Dan Callahan_
Fred Conway
_
Nicola Goren_
Clint Janes_
Teena Johnson_
Tanya Luckett
Group Consultation Notes:
_
It is important to set up a system for data analysis. Use the same people who designed the data gathering tools to do the analysis._
Routine analysis of data allows a head start on complaints. Monthly analysis is best and can be done easily if surveys are automated and put on the Internet. This way data is calculated automatically._
Comments on surveys are important and the best way to analyze them is by grouping them in categories after having read through them carefully.
IV. Reporting Data:
Organization, methods, audience, implications, distribution
Lee Scharf
ADR Program Coordinator
Office of Site Remediation Enforcement
Environmental Protection Agency
Round One Participants:
_
Bob Amrhein_
Eileen Connolly_
Emily Hickey_
Teena Johnson_
Roderick Locklear_
Sheila Walcott
Round Two Participants:
_
Carolyn Brooks_
Beverly Moore_
Dominica Parks_
Paula Small_
Linda Williams
Group Consultation Notes:
_
Writing a report gives the program justification. It is a reflection process because you are forced to ask the question, "What are we really trying to do?"_
There are several issues to consider once you have reached the reporting stage. First, will the report be in the form of a briefing or a written report. If it will be a briefing, it's important to remember to explain both your methods and the data as simply as possible by using bullets or a PowerPoint presentation. Be prepared for questions and invite feedback in your briefing._
Statutes and legislative acts often require written reports, but they are also useful because they provide a single text for reference. First, determine what information needs to be communicated. Not only discuss the strengths, but also admit the weakness and always include recommendations._
Early on in the evaluation process, determine who will disseminate the report. The evaluator or program managers are two choices._
The following are some tips on how to enhance a report:-
- involve stakeholders
-
tailor to audience needs-
be clear and accurate
-
be honest and direct-
note limitations-
remember, "critics are a gift"