by Libby Whittington-West and Ainsley Goldman
the challenges: Like many folks working in student life, we have a lot of data about student learning that takes place during our programs but are unsure of the long-term impact of our programming . What learning stays with them?
the solution: Realizing this was a great learning experience for a work study project, we hired work study students to conduct phone interviews with students who participated in experiential learning job shadowing programs one year ago. We chose to keep questions open-ended to gather qualitative information about what their experience in the program meant to them. We were in the process of re-imagining our program goals so we wanted these conversations to be more exploratory in nature rather than measure-y ™ Libby and Ainsley
working with student colleagues: the summer work study period is very short! We wanted our work study students to be collaborators from start to finish so we did not have much time to test out our questions before interviews began, meaning that we made adjustments to the questions along the way. Thankfully, both work study students had skills and experience interviewing, but the tight timelines meant that we had to rely on the less than ideal data collection method of interviewer note-taking.
analysis: Though the open-ended nature of the questions meant themes naturally emerged, the qualitative data made it hard to analyze afterwards. It would have been more effective to start open-ended and then drill down to some very specific questions tied to our program goals. However, it was only by using these open-ended questions that we noticed something new: students didn’t really like using the word ‘networking’ – they preferred to say things like ‘keeping in touch’ or ‘developing relationships’. If we had only asked a likert-scale question ‘how comfortable are you networking?’ we may not have captured this learning.
We also struggled with a low response rate, likely some response bias and not much time to follow up to increase the number of participants, and not enough time to build in a control group.
the results: at first glance, the results are kind of boring as they echo the results of students immediately after finishing the program. Common themes include: the program prompted some sort of realization; the program reinforced prior interests; the experiential learning method introduced them to work culture; co-curricular activities and developing a network are important. But hearing the same themes emerge for students a year after they participated in the program is also quite rewarding – phew, they still feel the same way.
an added bonus: after the interviews were finished, we paired our interviewers up with our work study social media guru to make a short promo video for our program, inviting in the students who had the most rave reviews about the program to include their testimonials in the video.
tl;dr: doing a very informal temperature check of student learning over time is worthwhile, but there will be sacrifices in data integrity. However, this insight helped shape goal-setting and program planning for next year.
Libby Whittington-West and Ainsley Goldman work in the University of Toronto Career Centre as Coordinators of Career Exploration Programs.