Extension Outcomes
Evaluation
Project
Updated Progress Report
March 31, 2001
During the first eight months of the Enhancing and
Supporting Positive Outcomes project, University of Missouri - Kansas City
Institute for Human Development (UMKC-IHD) personnel established a working
relationship with University Outreach and Extension Division (U/OE) regional
directors and staff. Together they have
laid the groundwork for meeting the primary project goal, objectives and
outcomes of this project (see Appendix A for sample letters of appreciation).
Due to differing needs and levels of knowledge, the regions are at different
stages of meeting various objectives and outcomes of this project.
Personnel and Regions
|
UMKC-IHD
|
U/OE Region
|
Regional Director
|
Counties in Region
|
|
Dr. Carl Calkins
|
North East
|
Nancy Flood
|
Adair, Clark, Knox, Lewis, Linn, Macon,
Marion, Monroe, Pike, Ralls, Putnam, Randolph, Schuyler, Scotland, Shelby,
Sullivan (16 Counties)
|
|
Dr. Carl Calkins
|
North West
|
Glen Easter
|
Andrew, Atchison, Buchanan, Caldwell,
Clinton, Daviess, DeKalb, Gentry, Grundy, Harrison, Holt, Livingston, Mercer,
Nodaway, Worth (15 Counties)
|
|
Dr. Chris Rinck
|
Central
|
Mark Stillwell
|
Audrain, Benton, Boone, Callaway, Carroll,
Chariton, Cole, Cooper, Howard, Montieau, Morgan, Osage, Pettis, Saline (14
Counties)
|
|
Dr. Chris Rinck
|
West Central
|
Albert Black
|
Bates, Cass, Clay, Henry, Jackson, Johnson,
Lafayette, Platt, Ray, St. Clair, Vernon, (11 Counties)
|
|
Dr. Carl Calkins
|
East Central
|
Bud Reber
|
Franklin, Jefferson, Lincoln, Montgomery,
St. Charles, St. Francois, St. Louis, St. Genevieve, Warren, Washington (10
Counties)
|
|
Vim Horn
|
South West
|
Beverly Coberly
|
Barry, Barton, Cedar, Christian, Dade,
Dallas, Greene, Hichory, Jasper, Lawrence, McDonald, Newton, Polk, Stone,
Taney, Webster (16 Counties)
|
|
Mike McCarthy
|
South Central
|
Pat Snodgrass
|
Camden, Crawford, Dent, Douglas, Gasconade,
Howell, LaClede, Maries, Miller, Oregon, Ozark, Phelps, Pulaski, Shannon,
Texas, Wright (16 Counties)
|
|
Mike McCarthy
|
South East
|
Janet Kline
|
Bollinger, Butler, Cape Girardeau, Carter,
Dunklin, Iron, Madison, Pemiscot, Mississippi, New Madrid, Perry, Reynolds,
Ripley, Scott, Scoddard, Wayne, (16 Counties)
|
August 8, 2000
Outreach & Extension Video Staff Meeting
Dr. Chris Rinck and Dr. Carl
Calkins from the University of Missouri - Kansas City Institute for Human
Development (UMKC-IHD) held a video conference with all the University Outreach
and Extension Division (U/OE) regional directors, and presented an overview of:
§
Project Goal ,
§
Project Objectives, and
§
Project Outcomes.
|
|
PROJECT GOAL
The goal of Enhancing
and Supporting Positive Outcomes is to provide
evaluation resources and support to the University Outreach and Extension
Division.
|
|
|
|
|
OBJECTIVE ONE
Identify
the evaluation needs of the faculty and staff through an extension survey
process
|
|
Activity 1: Do
a mail survey
§
A
preliminary instrument was developed to survey the evaluation needs of the
staff and faculty of the University Outreach and Extension Division
(U/OE). The survey was designed to
assess the current level of evaluation skill and knowledge and identify content
areas and types of learning styles that will be most conducive to facilitating
acquisition of these skills and knowledge.
§
A
pilot survey was conducted in the Central and South Central Regions.
§
The
results of the pilot test were detailed in a report titled “Piloting of an
Evaluation Needs Assessment Survey” prepared by Christine Rinck, Ph.D. (see
Appendix B).
§
The
survey instrument was revised based on the pilot test and a finalized version
has been developed (see Appendix C).
§
UMKC-IHD
has talked with the U/OE Division Regional Directors to develop a distribution
plan and prepare for distribution of the survey.
§
The
survey has been placed on the UMKC-IHD web page and the U/OE Regional Directors
have been contacted with information on downloading the instrument.
§
UMKC-IHD
has received completed surveys from the West Central, North East, South Central
and South West Regions.
§
Reports
detailing the results from the Evaluation Needs Assessment Surveys have been
prepared for each reporting region and distributed to the regional director
(see Appendix D, E, F, & G).
Activity
2: In-depth phone interview with
regional directors
§
A
telephone interview format has been developed and interviews are being
scheduled and conducted with regional directors (see Appendix H).
Activity
3: Conduct a focus group
Activity 4: Examination
of current reporting requirements, types of information being gathered, and how
data is used
§
A
review of all written materials pertaining to Extension reporting requirements
and practices, has begun and is in its finishing stages.
|
|
OBJECTIVE TWO
Provide training to U/OE faculty and staff on
evaluation issues.
|
|
Activity 1: Conduct basic
evaluation workshops
§
A
Basic evaluation workshop was developed.
§
The
Basic evaluation workshop was conducted with:
1.
Regional
agronomists at their PDE at Tan-Tar-A conference
2.
U/OE
South West Regional Staff
§
A
one-hour presentation on evaluation was conducted for the U/OE Central Region
staff.
§
On January 24, 2001, Vim Horn from UMKC-IHD met with
UO/E’s Statewide Community Development Team. He presented an overview of the
Extension Outcome Evaluation Project goals, objectives, and projected outcomes
to state specialists and faculty.
Activity
2: Provide advanced topical
evaluation workshop.
Activity
3: Evaluation training infusion at
Extension Conference.
|
|
OBJECTIVE
THREE
Assist one major
educational program in each of the eight regions throughout the year period
from conceptualization to report writing.
|
|
Activity 1: Work with Extension Regions to identify a
project which meets the identified program development/evaluation criteria.
§
UMKC-IHD
personnel has been in contact with University Outreach and Extension Division
Regional office in the South East region and has scheduled a meeting for late
April to meet with the regional director to start the process of identifying a
preliminary project.
§
Vim
Horn from UMKC-IHD met with U/OE South West Region staff and initiated the
process to identify an appropriate project.
An “estate planning” education project was selected for evaluation.
§
Dr.
Chris Rinck from UMKC-IHD met with the West Central Region and an “info line”
evaluation project has been selected.
§
Dr.
Carl Calkins and Vim Horn from UMKC-IHD met with U/OE North West Region staff
on August 15, 2000, provided an overview of the evaluation project, facilitated
a discussion of possible regional priorities for evaluation, and an evaluation
project was selected (Farm Families).
§
Dr.
Carl Calkins from UMKC-IHD met with U/OE Central Region staff on September 15,
2000 and an evaluation project has been selected (Farm Taxation Information).
§
Dr.
Chris Rinck from UMKC-IHD has also been working with Chris Thompson from the
U/OE Central Region on a Business and Industry needs survey.
§
UMKC-IHD
and a group including agronomists from various regions have begun the process
of designing a statewide evaluation.
§
Dr.
Carl Calkins from UMKC-IHD and Bud Reber, U/OE East Central Regional Director
met at the annual Outreach & Extension conference in Columbia on October
23, 2000. They discussed levels of technical assistance that could be provided
and intended outcomes of the evaluation program. A strategy was developed to work with regional staff on establishing
prioritized activities.
§
Dr.
Carl Calkins from UMKC-IHD met with U/OE North East Region staff on October 31,
2000 and two evaluation project were discussed. The first project they approved
for evaluation was their “Tilling the Soil” program and curriculum. The second
project they decided to design a comprehensive evaluation program for was their
Estate Planning project, but in a broader context.
§
Mike
McCarthy from UMKC-IHD met with UO/E South Central Regional staff on January
16, 2001 in Rolla, provided an overview of the project and presented ways in
which UMKC-IHD can support UO/E evaluation efforts. An educational program,
which is in early development, was selected to use as a pilot effort.
§
On
January 24, 2001 Dr. Carl Calkins from UMKC-IHD met with Bud Reber, U/OE East
Central Regional Director and further discussed their strategy for working with
regional staff on establishing prioritized activities.
§
On
February 22, 2001, Dr. Calkins met with the East Central Staff for a day long
seminar on program evaluation. He
outlined the project goals, discussed the evaluation logic model, and addressed
questions regarding particular needs assessment/evaluation problems presented
by the staff. The East Central Regional
staff will now work to identify a regional project in which to implement an
evaluation component. Dr. Calkins also
agreed to assist two smaller, more local projects with their evaluation
components.
Activity 2: Work
with the Eight Extension Regional teams to identify program outcomes.
§
In
the U/OE North West Region, a committee was established, a Chair selected
(Kristi Michael), and a series of meetings planned to begin the overall
evaluation process.
§
UMKC-IHD
personnel met with U/OE North West Region staff on September 15, 2000 and
assisted them in identifying program outcomes for their Farm Families project.
§
In
a meeting with between Dr. Carl Calkins from UMKC-IHD and U/OE North East
Region staff on October 31, 2000, the group began identifying those groups with
an interest in the program (stakeholders) for their “Tilling the Soil” program
and curriculum and their Estate Planning project.
§
Vim
Horn from UMKC-IHD met with UO/E South West Region staff and discussed the
process for development of outcomes. It
was decided to shift the focus of the project from “estate planning” to looking
at family financial planning in a broader context with estate planning as one
aspect of the evaluation.
§
Mike
McCarthy from UMKC-IHD met with UO/E South Central Regional staff in Rolla, on
February 22, 2001 and they established desired goals and outcomes for their
educational project.
§
Dr.
Chris Rinck from UMKC-IHD met with UO/E West Central Regional staff and helped
them identify outcomes for their “Info Line” evaluation project. She also consulted with Chris Thompson on
the Business and Industry needs survey he is developing.
Activity 3: Work
with the Eight Extension Regional teams to develop a logic model and outcome
measures
§
UMKC-IHD
personnel met with the U/OE North West Evaluation Task Force on Farm Families
on September 19, 2000 and discussed establishing a logic model for evaluation,
outcome measures, and the necessary tasks that would need to be completed.
§
UMKC-IHD
personnel met with U/OE North West Region staff on October 17, 2000 and
assisted them in developing a needs assessment. A preliminary needs assessment
survey was designed for their 21st Century Farm Families project
that could be used in both focus groups and for individual responses (see
Appendix I).
§
In
a meeting between Dr. Carl Calkins from UMKC-IHD and U/OE North East Region
staff on October 31, 2000, the group decided to use a survey tool to identify
needs for their “Tilling the Soil” program and curriculum. The group discussed
using the same survey topics after each of the ten training sessions and again
after completion of the entire program.
The use and development of focus groups was also discussed. Strategies for defining needs for the target
population for the Estate Planning project were also discussed, including the
use of focus groups. Dr. Calkins
outlined the “Person Centered Planning Approach to Futures Planning.”
§
On
November 9, 2000 Dr. Calkins from UMKC-IHD met with Parmen Green of the UO/E
Central Regional and developed a preliminary needs assessment survey for
stakeholders in their Farm Taxation Program (see Appendix J).
§
Vim
Horn from UMKC-IHD met with UO/E South West Region staff and discussed the
process for developing a logic model, program design, and outcome measures
based on client need.
§
On
February 22, 2001 Mike McCarthy from UMKC-IHD met with UO/E South Central
Regional staff in Rolla and they established desired goals and outcomes for
their educational project.
§
Dr.
Chris Rinck from UMKC-IHD met with UO/E West Central Regional staff and helped
them develop a tool to evaluate info line messages and a process for using the
evaluation tool. Four Health
Ambassadors were designated to call 40 info messages rating them to see how
they like the messages, how useful the information is, and whether the
information and messages are culturally appropriate. A plan to have residents
in area housing projects rate the info line messages is also being
developed.
§
Dr.
Chris Rinck from UMKC-IHD has provided suggestions and feedback concerning
methods for developing outcome measures with Chris Thompson on the Business and
Industry needs survey.
§
On
March 22, 2001, Dr. Calkins met with the East Central Staff for a half day
seminar on program evaluation. He
outlined the project goals, discussed the evaluation logic model, and addressed
questions regarding particular needs assessment/evaluation problems presented
by the staff. The staff were given
copies of the needs assessment report prepared by IHD staff and it was
discussed. The Central Regional staff
will now work to identify a regional project in which to implement an
evaluation component.
Activity 4: Support
the Eight Extension Regional teams with implementation and follow-a-long of
program and evaluation.
§
In
a meeting with U/OE North West Region staff on October 17, 2000, UMKC-IHD
personnel started assisting North West Region staff in implementing the
evaluation process for two separate ongoing projects. Assignments were made to
begin the needs assessment process.
§
On
November 16, 2000 Dr. Carl Calkins from UMKC-IHD met with U/OE North East
Region staff and discussed the functional evaluation of their Tilling the Soil
program. The discussion looked beyond
the pre-designed evaluation materials and developed a questionnaire focusing on
each of the ten sessions of the curriculum.
Further work was continued after the questionnaires were mailed to Dr.
Calkins.
§
On
November 21, 2000 Dr. Carl Calkins from UMKC-IHD met with the Farm Families
2000 Evaluation Committee from the U/OE North West Region. Responses from focus
groups were discussed and after integrating feedback from the focus group
participants, the needs assessment survey for the 21st Century Farm
Families Evaluation program was finalized.
§
UMKC-IHD
personnel held a telephone conference with Darla Campbell from the U/OE North
East Region to discuss their two projects.
Her approach to Tilling the Soil was to team teach the program and use
the evaluation questions to assist in determining whether there was an actual
match of the program to the needs of the participants. Bob Wells was intending to teach two courses
and do the same with the evaluation questions.
A preliminary review of information from the Estate Planning focus
groups was also discussed and organized.
Several outcomes were generated from this discussion. One of which was a differential level of
technical assistance and training, depending on the level of need. An assessment device is being developed by
UMKC to determine level of need and programming. Additional needs assessment is also being gathered and organized
(see Appendix K).
§
On
January 17, 2001, Dr. Carl Calkins met with Parmen Green, of the UO/E Central
Region. He collected the completed needs survey for data compilation, and they
discussed curriculum development for the Farm Tax program.
§
On
January 22, 2001, UMKC-IHD personnel presented the first quarter Evaluation
Project report.
§
Mike
McCarthy from UMKC-IHD met with UO/E South Central Regional staff and discussed
issues and approaches to ways to gather more information about the needs of the
target audience. They concluded with a plan of action, part of which was to
review and summarize past evaluations and to develop a simple survey
instrument.
§
Mike
McCarthy and Kellie Ellerbusch from UMKC-IHD met with UO/E staff at the North
Central Regional office in Macon on February 1, 2001 where they focused on
identifying the target audience and clarifying characteristics of the target
group for their training program on long range financial planning. They also
discussed the development of a survey tool in which participants would indicate
priorities with reference to the proposed content and approach. A follow-up
meeting has been scheduled for next quarter.
§
On
February 9, 2001, Dr. Calkins met with Parmen Green, of the UO/E Central
Region, and the region’s project group to discuss preliminary results from the
needs survey and begin developing the project curriculum. They also discussed
development of a consumer evaluation form for technical assistance provided by
UO/E staff (See Appendix Q).
§
UMKC-IHD
is analyzing the data from the UO/E West Central Region’s evaluation of “info
line” messages and will provide the regional director a report on the findings.
§
Dr.
Chris Rinck from UMKC-IHD has also consulted with Jean VanBooven from the UO/E
West Central Region in development of a multilevel impact evaluation of youth
involved in community gardening. The
evaluation will include a participant pre and post assessment, a pre and post
assessment completed by the young person’s teacher, and a post assessment
completed by the parents. UMKC-IHD is analyzing the data and will provide the
regional director a report on the findings.
§
On
March 22, 2001, Dr. Calkins met with the East Central Region’s Farm Tax project
group to discuss the survey report and continue planning curriculum
development.
OBJECTIVE
FOUR
Provide resources and
technical assistance to faculty and staff through Internet tutorials and
other resources related to evaluation.
|
|
Activity 1: Develop
basic written and interactive evaluation training resources.
Activity 2: Develop resource materials addressing
specific regional evaluation needs that can be used by other regions.
Activity 3: Create
a website to contain resource materials.
§
A
UO/E Outcome Evaluation Supports website, has been created to enable program
leaders to ask questions and request information from UMKC-IHD personnel.
Reports and resources have also been posted on the new site (see Appendix L).
§
The
website has been updated to include links to each of the UO/E Region’s website.
§
On
November 9, 2000 Kellie Ellerbusch from UMKC-IHD met with Parmen Green and
discussed the development of web pages to highlight the activities of the Central
Region’s Farm Taxation Program.
§
Christina
Karakoussis from UMKC-IHD met with U/OE North West Regional staff and developed
a web page for their Farm Families 2001 Project (see Appendix M).
Activity 4: Provide
Region based topical reports and literature reviews.
§
UMKC-IHD
personnel provided the U/OE Central Office with assistance in coding data,
providing analysis and tables, and Dr. Walt Boulden prepared an evaluation
report for a completed 4-H Training project (see Appendix N).
§
Dr.
Walt Boulden from UMKC-IHD prepared an evaluation report of a “Water Quality
Education Project” for the Buchanan County U/OE office in the North West Region
(see Appendix O).
Activity 5: Establish
an Extension evaluation message board.
§
An
internet message board have been created to enable program leaders to ask
questions and request information from UMKC-IHD personnel (see Appendix L).
§
The
internet message board is being Hosted/Staffed by Dr. Walt Boulden.
§
UMKC-IHD
personnel have started a discussion on how to best develop procedures to
promote awareness of the internet message board with U/OE regional staff.
Activity 6: Consult
with program leaders on evaluation questions.
§
Individual
technical assistance strategies were developed with U/OE North West Region
Staff regarding using the “Tilling the Soil” curriculum.
Activity 7: Produce
a quarterly Extension evaluation newsletter.
