REU Evaluation: Determining what works!

The National Science Foundation (NSF) has a wonderful program which provides funding to research institutions so that undergraduate students can receive educational opportunities directly conducting research. The funding program is titled Research Experiences for Undergraduates (REUs). The REUs provide valuable hands-on learning opportunities for future scientists. 

As scientists, faculty involved in REUs want to know which components of the experience function at capacity, as well as where improvement efforts should be focused. MEREA Consulting has developed an effective evaluation process for REU awards. This process provides formative and summative results intended to improve the research experiences. We believe front-loading an effective evaluation system provides timely program monitoring. Timely program monitoring yields appropriate adjustments that can maximize the REU experience for the participants, leading to achievement of the NSF goal – more participation in science!

Below is an outline of the evaluation report format we use:

Program Background — Title of REU program, number of participants, dates of program, where participants lodged, where participants worked, overview of research projects, and overview of project faculty.

Evaluation Components — Formative (obtain feedback regarding quality of current program), summative (inform next program), pre- and post-program self-evaluative process measuring changes in skill and knowledge as a result of program experience, evaluation questions for participant (to include, but not limited to, extent of increase in skills, extent of increase in knowledge, program experience satisfaction, research experience satisfaction, faculty knowledge satisfaction, faculty pedagogy satisfaction, lodging satisfaction, meals satisfaction, transportation satisfaction, and suggested program changes)

Evaluation Methods — Electronic survey emailed to participants prior to arrival at program facility and after program conclusion, survey hosted on SALG (Student Assessment of their Learning Goals), pre-program analysis will be provided electronically to program coordinator within 3 weeks of the beginning of program, post-program analysis will be provided within 3 weeks of the conclusion of program.

Data Analysis — Graphs of all questions for participants by percentage using Likert Scales, along with anonymous comment highlights of all questions for participants.

Ancillary Information — Executive summary, list of participants with contact information, list of faculty, copies of the pre- and post-surveys, overview of participant projects (including titles, authors, mentors, and brief descriptions).

Cost — The full evaluation services are provided at 4.5% of annual award.

Additional Services — Additional customized evaluation methods are available. Please contact us with details of your request for a quote.

Evaluation Works!

As the director of licensure for the Public Education Department of the State of New Mexico, I knew I had accepted a huge responsibility. I was responsible for serving every public school district and every public school in the state. Superintendents and principals depend on having appropriately licensed teachers in every classroom to ensure that learning at the appropriate pace occurs. For superintendents and principals, having a licensed teacher is a must! My responsibility was to support these superintendents and principals with their challenging, yet rewarding job.

When I accepted this responsibility, it took the Professional Licensure Bureau 12 weeks to issue a license – from the time the application was received until the license was viewable on the state website. As a life-long educator, I knew this was not acceptable. The students, teachers, and administrators in New Mexico schools deserved better. As an experienced teacher, coach, and administrator, I knew that the problem was a systems issue. A systems issue that had to be fixed. New Mexico students, teachers, and administrators were depending on it. I knew that we needed to evaluate our internal systems at the Professional Licensure Bureau. I knew that we needed to evaluate what was working and what was not.

Fortunately, I had a dedicated staff of 10 individuals who were willing to embark on this evaluation journey. I said at the beginning of our efforts that the evaluation of our daily work would be hard but it would be rewarding. I said that, if we stayed the course, we would like the results and New Mexico schools would benefit. Through our evaluation process and constant monitoring and adjustment, that dedicated staff cut the time in issuing licenses from 12 weeks to one week! 

Who was the winner for implementing this evaluation effort? New Mexico students, teachers, and administrators. Evaluation works!

Phil Baca