by Jeff Burrow
Finally! Your Assessment Project is almost done. You had a goal, you designed a project, you collected and analyzed your data from the (let’s say) five focus student groups. And now you have written up a report including some implications and suggestions for change, and are ready to share with your team and director to start talking about next steps.
But wait! The analyzing and writing process can take several weeks or months, meaning that a large gap between when you heard the student voices in the focus groups and when you finish a summary and interpretation of what they shared. And in this process of preparing a draft report, what might have seemed very important initially can become a bit less clear and evident as you are immersed in all the data. Our goal is to develop a report that is based upon an accurate representation of what the students shared with us. There are lots of ways to help ensure a fair representation of the data from the students like having audio transcripts and taking personal notes, but there are two other steps (that don’t take very long) that can add to the trustworthiness and credibility of your final reports.
One approach is to share a summary of the key findings and representative quotes with your participants. This is often called a member check. In this case we are asking the members (students in the focus groups) if they ‘see’ themselves in the findings and summaries you have created. You want them to tell you ‘What we got right and what we might have (unintentionally) under or overstated.” This process also gives the students one more chance to share with you. Think about how many times you have been in an interview and had that feeling afterwards of “Why didn’t I mention that?” Using this process, you get the benefit of their feedback for accuracy and the time they have had to reflect on the focus group discussion.
A second approach is to share your transcripts with a colleague who is acting as a ‘peer debriefer’. Ask them to read the transcripts, or whatever summaries of the focus groups you have. Have your colleague identify what they think the major findings and learning’s from these same conversations. This is called a peer member check or debrief and can be helpful to check against some of our own bias and potential blind spots. Who should you choose to help out? A good person to ask if someone who knows of, but isn’t totally involved in your program. When they have shared back with you, you can then discuss where your interpretations are aligned and where they might diverge. This process will not likely lead to total agreement, and it may lead to some re-writing on your part. However, it is a very useful step that should make you even more confident that what you share with your team, the data you use to inform decisions about what to do and not do are based on the best analysis possible.
These are two ways we can help to ensure that the process we use to learn more about our programs and services results in the fairest representation of the stories the students share before we begin to make changes.
Jeff Burrow is Manager, Analysis and Assessment and Co-chair of the Learning Outcomes and Assessment Committee