Darby

In student affairs, we don’t often talk about benchmarking and how it can be used for improvement. I know I spend more time thinking about student learning or satisfaction as my focus, but I wanted to plant the seed that benchmarking should be in the mix of options, depending on what you need to know. (Assessment projects should always start with what you need to know.

Benchmarking can be used in a couple of ways. One, you can benchmark against a standard or baseline that you want to meet. For example, the Council for the Advancement of Standards in Higher Education has created a lot of functional-specific self-assessment guides that provide guidance on how well a functional area performs based on a set of criteria that a unit should meet. This evaluation tells you where your strengths and areas of improvement are in relation to an external standard or requirement. It can provide a comprehensive view of unit performance, leading to decisions about where to take action. This can be done fairly informally as a staff discussion or part of a more formal review process, such as accreditation application.

Benchmarking can also be used to compare one functional unit to other, similar functional units. The comparison groups could be similar units at similar institution types, similar functions outside of higher education, or units that might be considered to have best practices. For example, a health center at a large, public institution might compare themselves to other large, public institutions in the country or region. But, they also might look at similar institutions who have the same accreditation they do. Even further, they could look at local health care facilities, especially if they are looking at a specific part of their operation. Maybe the health center sees a need to improve their scheduling process to be sure they can see as many student patients as possible, as quickly as possible. They could ask local facilities about the scheduling software they use, how they schedule their providers, and what the average/acceptable wait time is between making an appointment and seeing a health care professional. Based on the information collected, the health center could implement changes that they think would work in their environment.

The process of benchmarking does not have to be overly complex. Sometimes, it’s as easy as looking at other units’ websites to find the information you need. Alternatively, it might be reaching out through your networks (listservs, professional associations, informal peer groups, etc.) with several focused questions. Obviously, it could me more complex and time-intensive if you are looking at a comprehensive program review or accreditation process.

I hope that gives you a glimpse of benchmarking as an assessment tool. If you need assistance, Student Life Studies is always here to help.

The National Institute for Learning Outcomes Assessment (https://www.learningoutcomesassessment.org/) has an amazing wealth of resources for campuses interested in all aspects of assessment of student learning in higher education. There are articles, case studies, webinars, frameworks, and more. You really should check it out!

Earlier in 2020, NILOA sponsored a webinar on evidence-based storytelling (https://youtu.be/TutW9VCam8A). While I cannot do it justice (you really should check it out!), the premise is that evidence-based storytelling is supporting claims of student learning using evidence through stories to persuade a particular audience (time 6:07-7:35). It promotes improvement and accountability. Stories are different than just sharing assessment results. They are about sharing the decisions we make and who is impacted by what we do. The webinar provides a number of tips and ideas in the webinar that can help your guide the narrative you are developing.

NILOA created an evidence based storytelling toolkit, which can be accessed at https://www.learningoutcomesassessment.org/wp-content/uploads/2019/10/EBST-Toolkit.pdf. You really should check it out! The toolkit gives steps to make the process fairly easy: defining your audience(s), determining the goals and arguments for your story, choosing compelling evidence, figuring out the type of story (i.e., compliance, improvement, etc.), creating the characters, writing the plot, selecting the visuals that will best tell your story, and determining the medium to get your story out to your audience(s).

If you are an assessment person, reach out to your marketing/communications people. If you are a marketing/communications person, reach out to your assessment people. If you are a program coordinator, bring your assessment and marketing/communications people together. If you need a place to start, contact Student Life Studies, so we can help you. You really should check us out!

Many student affairs/co-curricular programs have moved to a virtual environment, commonly on Zoom. This has changed the way we can do data collection with a “captive audience.” Before the pandemic, programs could have students complete an assessment form in person, which usually yielded a high response rate and immediate feedback. Because we are not all physically in the same room, collecting data in the moment can be a challenge.

If you have not checked out the Zoom polling function, you might want to do that. They have a step-by-step tutorial at https://support.zoom.us/hc/en-us/articles/213756303-Polling-for-meetings. Like any good assessment, you will want to plan ahead. It’s much easier to create your questions ahead of time, rather than while you are in your meeting. Remember, if you are asking bad questions, it doesn’t matter how fancy your platform is—you will still get bad answers that won’t help you.

You can ask both choose one and choose all of the above types of questions. Unfortunately, you cannot use a text response type of question. You can specify if you want the responses to be anonymous or connected to your participants’ information. You can also have up to 10 response options in each question. Plus, you can have multiple questions inside each poll.

When you are in the meeting, you can launch the poll by clicking on the poll icon at the bottom of the screen. Your participants can then answer the question. When you end the poll question, you can also share it with your participants if you would like to spark discussion or clarify information. You might want to have several poll questions in each program, so you keep participants engaged.

After the meeting, you can sign into the Zoom web portal to access your results, which will be generated in an Excel spreadsheet. If you chose for the responses to be anonymous, you will see that indicated in the spreadsheet. If you did not choose that option, you will see the user name and user email. This allows you to follow up if you need to contact participants. You can then analyze the results for any information that will help you improve what you are doing.

While no one asked to shift all of our program delivery to an online format, it does give us an opportunity to learn and use new technology in data collection. It’s okay to not be perfect in it every time, but lots of people can empathize and help. We can stop assessment, or we can embrace the opportunity. I hope that you embrace the opportunity!

This academic year will be like no other that we have experienced in our lifetimes. So much change, so much still unknown. While it can cause us stress and anxiety, it can also be an opportunity to look at assessment differently. It’s easy to fall into a pattern of program—assessment—decisions—planning—program—assessment….That pattern has been disrupted because we can’t provide our programs and services in the same way, which necessitates us changing how we assess.

As always, it is important to understand who our students are, what they need, what they are learning, and how satisfied they are (to name a few). Right now, it might be even more important to understand our students’ sense of belonging, when many of our programs and services are being offered remotely and the quality of those virtual engagements. How are students connecting with their peers, which is especially important for new students? How are students’ connecting with organization advisors? How do you know students are learning something when you cannot necessarily observe them as frequently?

It’s time to be a little creative in your big questions, your data collection, and your analysis. Take a breath, grab your favorite beverage, and reflect on the following question: What is the most important thing we need to know from the students we serve (and those we don’t serve) about their experience in our program or service during a global pandemic? Don’t rush this process. Jot down some words, draw a picture, think of a theme song….

When you have that question answered, think about how you are going to gather information about what you want to know. In the past, you might have had an in person audience who could fill out a survey or provide verbal feedback on the spot. Without that, we have to figure out different ways to collect information. Maybe it’s using Zoom polls or chats, doing something in the learning management system, having students tweet using a hashtag, create a video. I’m not that creative, so I know people can come up with even more interesting ways to collect data.

When you have the data collected, think about the best way to analyze it. You might have done a poll, so you have quantitative data, but you could also have saved the chats which give you qualitative data. A great way to get a fresh perspective is to ask students to help you analyze the data to make meaning out of it. When you have the data analyzed think of creative ways to share it. Can you do something graphically with charts or word clouds?

Your changes may not be earth shattering, but I hope they are meaningful for you and the students you work with. It’s going to take all of us to get through a pandemic, and I hope we take this opportunity to think about what is really important and how we can be of service to the students we serve on a daily basis.

Last month I was asked to present at a virtual conference related to COVID-19 with a colleague from NYU. If you know me, you might be saying, “Darby, you are a doctor, but you are not that kind of doctor.” So, true. The point of the presentation, though, was to talk about the value of rapid data collection in times of crisis or the unknown when information, decisions, actions can and do change very quickly.

I thought I would share some of the take-aways from that session, in case you find yourself in a global pandemic again or some other major campus crisis.

Good is better than perfect. (Okay, there is no perfect assessment, as much as we try!) In times of rapid change, we don’t always have time to plan the perfect data collection tool, and we can’t wait until we have all the information that we need to plan. While that doesn’t give you an excuse to launch a really bad survey, it does give you a little latitude to do the best you can with the information you have.

Collaborate. Collaborate. Collaborate. Other people may already have information that you need, so it would be detrimental to re-collect data. It is a waste of your time and annoying to your audience. While everyone may feel pressure to immediately collect data and make decisions in their own silo, there should be some coordinated effort that makes sense and streamlines the assessment and communication process.

Small samples and few questions are okay. Sometimes in assessment and research, we get caught up in having a large, representable sample and a high response rate. Those are great things, but in times of crisis, you may not have that luxury. If you can collect reasonably accurate and representative responses from a small group in a short period of time, that gives you the ability to make a decision, test an action, and determine whether that new action is working. When you re-assess using another small sample, you’ll generally know whether that new intervention worked. If it doesn’t, then you can try something else fairly quickly. Think about the minimum data points you need to maximize the information you can use.

Assessment can be low-tech. It could be as easy as tracking the topic of phone calls/emails that you get from students. Within a day, you probably could make a check sheet of common topics. Fairly quickly (within days or a week), you probably could summarize the main concerns and communicate them to the appropriate person/office. When some action is taken (e.g., putting more information on your website), you can determine if the number of questions about that topic have decreased. If you saw students in person, you could ask them to write main concerns on an index card. If you are calling students, you can also track their concerns/questions.

Equity is still a key component in assessment. In times of crisis, we need to recognize that some groups are impacted more than others. We need to collect data from a variety of populations, while not adding a burden to marginalized populations who are already experiencing greater trauma. In addition, when we are making decisions, we need to include multiple voices at the table. Policy decisions, new procedures, and new requirements can have an inequitable impact, even when we think they might be fair to all (e.g., wearing a face covering, requiring a laptop, etc.).

I hope that gives you some insight about collecting data in these times of rapid change. Student Life Studies is always here to help you in that process. Please let us know how we can help.

 

As we are gearing up for students to return to campus (in person or virtually), it’s time to start thinking about our program and service delivery. Obviously, I think that assessment should be a part of the planning and decision making process. But, I also strongly advocate for including students in the assessment process, especially if the assessment is about learning and student engagement.

A simple assessment cycle would go something like this: Determine your mission, goals, and outcomes. Plan how to deliver and assess your program/service/experience. Deliver that program. Collect data. Analyze and interpret data. Make a decision about improvements to your program. Tell people about your assessment. Implement those changes. Start the cycle again, being sure to reassess to know whether your changes made a positive difference with your participants. Let’s look at those steps individually to see how you could include student voices in each step.

Determine your mission, goals, and outcomes. Do you ask students about what they want to learn and be able to do following some experience? Do you engage them in updating the mission statement? Do you ask student leaders what they want their organization to accomplish? Do you all agree on what success looks like? How can you include multiple voices that you may not normally hear from?

Plan how to deliver and assess the experience. This gets into the logistics of the process. How would students like to deliver their experience (in person, virtually, hybrid)? How many people will be there? What is the goal of the program? What assessment methods will be used to determine success? Have you asked students how they want to be assessed? If you are planning on teaching effective verbal communication skills to 300 people in a one-hour workshop, you may not have time for each participant to practice a skill and get feedback on their performance. That’s why planning and assessment should be happening together. Have you gotten feedback from your participants, employees, or student leaders about how they could best represent their learning and development? A survey may be easy, but it may not be the best way to measure knowledge and learning.

Deliver the program. Is everyone clear about what the outcome and content are? How are you including students in program delivery? Are you able to use students to engage with other students? (Doing that will also increase the skills of the students who are presenting material.) Have you talked ahead of time about where assessment happens in the program delivery?

Collect data. Hopefully, you asked students ahead of time about how best they can demonstrate their learning. At this point in the cycle, can you engage students in the data collection? Maybe you are using a rubric to measure a leadership skill (verbal communication, event planning, meeting facilitation, etc.). Can the executive student leadership be trained to use the rubric to provide feedback to the next level of leaders? Can your students be assessment cheerleaders in talking to their peers about the importance of participating in the assessment?

Analyze and interpret data. Do you include students in analyzing the data and making meaning of it? Have you scheduled a meeting with students to do that? Do you bring multiple voices to the table to give you a variety of perspectives? This can be particularly useful and insightful if you have qualitative data. Students are good about providing the feedback about the student experience because they are living it.

Make a decision about improvements to your program. Based what you are hearing from students, what do you need to do to improve? Have you asked students what specific changes they would make based on the peer feedback? What are their change priorities? What options might they think of that have not crossed your mind?

Tell people about your assessment. When you have assessment results and plans for change, do you tell students about it? Do you present to student leaders, event participants, student employees, student advisory groups? Do you post assessment information on your website to be transparent with students? Have you asked student leaders to present information to stakeholders? All of this helps participants know that you listened to their feedback, which may encourage them to participate in future assessment.

Implement those changes. How have you included students in implementing changes? It could be something that changes in the student organization or with a service you provide. Is it something that they can help implement, so they have ownership in the changes? Students can also be cheerleaders here by telling their peers about the positive changes that have been made.

Start the cycle again and reassess. Have you included students in how you will reassess to be sure the changes are working? Have expressed a desire to have them involved in future changes and assessment?

If you are not incorporating students into your assessment process, you are losing important student voices. Assessment becomes something you do to students, rather than something you do with and for students. People buy into what they help create. Do not underestimate students’ ability to engage in every step of the assessment process. They can be really insightful and develop skills they can use after graduation.  Student Life Studies is always here to help you in that process. Please let us know how we can help.

 

My last blog was about identifying your audience(s) to help prepare you to share assessment results. This time, I would like to address what you want them to know or do. Obviously, that is very dependent on the particular audience you are addressing. Tailoring your message is incredibly important.

Knowledge

Based on your assessment results, you might identify people that have an interest in what you are doing and assessing, but you don’t necessarily need for them to do something (at this point). At the very least, you can probably identify several people you think should know something about your program right now.

You want your audiences to speak knowledgeably, accurately, and positively about what you do. Again, they may want different pieces of information. You may want the department director to know that you came in under budget and that participants/clients/users were satisfied with their experience. You may want to Vice President to know that students learned something specific in your program and positively contributed to student success. The Vice President, in turn, may share that with a new faculty member who wants to know about programs in student affairs. You may want users (and non-users) to know that you listened to their suggestions and changed something about what you do.

Action

On the other hand, you may want some people to take specific action based on your communication of assessment results. Without knowledge of your program, assessment data, and plans to move forward, it may be difficult for people to act.

An obvious action would be to get more financial support for what you want to do. You may be requesting money from donors; they need to know the positive impact you have had on students. Your director may be requesting funding from the Student Affairs Fee Advisory Board (SAFAB). The board needs to know the value and effect on the student body. Because of SAFAB process, you may want students to make positive comments on the SAFAB website when it is open for feedback on requests. You may need to increase participation fees; you need to let people know what their money will go toward to make it easier to support the change. The more information people have about what you do will encourage them to care about your program.

You may also be interested in having more people participate in your programs and services.  How will you get them to take that action? One way is to tell them how attentive you have been to assessment and what changes you have made based on their feedback. This information could be on your website or in other recruitment material that you send out. Another way is to work through past users/participants to spread the word. You could communicate with them through email or social media, encouraging them to tell their friends.

Maybe you want to forge new partnerships to expand or enhance your current program or service. If someone asked you to do more/different work and use already scare resources, wouldn’t you want to know as much as you could about the program? Being able to approach people with accurate, relevant, usable data will build other people’s confidence in your program.

Those are only a few reasons to share assessment results. Be sure to keep those reasons, and others, in mind as you undertake an assessment project. Student Life Studies is always here to help you in that process. Please let us know how we can help.

As the academic year winds down, it’s a good time to reflect on what programs have accomplished, what students have learned, and what changes you want to make moving forward. Wrapped up in all of that is how and what you want to share with others about the great things you have accomplished and why they should be invested in what you do.

According to dictionary.com, a stakeholder is “a person or group that has an investment, share, or interest in something, as a business or industry.” In other words, who cares (or should care) about what you do? Take a moment and jot down (even in your head) who has a stake in what you do. I’ll wait….

How many entities did you come up with? One, two, five, ten? You may have listed a combination of these stakeholders: students, potential participants, student leaders, your supervisor, your department head, other staff, parents, donors/sponsors, faculty, administrators, the Vice President’s Office, state legislature, Board of Regents, funding agencies, local business owners, the alumni association…the list goes on. You may have even thought of one or more not on that list.

Because your stakeholders can vary drastically in their need for information, your communication to them about results should also vary. Not everyone is interested in a five page report with lots of tables and charts and individual quotes. You have lots of options: power point presentation, infographic, one page executive summary, social media post, website, full report, newsletter, word cloud, poster, etc. The Vice President may not have time to read an extensive report, but she may be really interested in the impact on student learning and success and/or the cost per student in a one-page executive summary. New student leaders planning this year’s program may be interested in what past participants thought went well and what did not, so a word cloud may be applicable. Future participants may look at the program’s website to see what the learning outcomes are and how well they have been accomplished in the past. Donors want to see that their money is spent wisely and educating students—they may want to hear individual student voices/quotes talk about the impact of their experiences.

Here are a few pointers to help you decide how to share your news:

1. Identify and prioritize your stakeholders.
2. Determine what you want each stakeholder to know, do, or feel.
3. Decide what information is most pertinent to each stakeholder based on what you want them to know, do, or feel.
4. Understand the level of complexity and detail that you need to get the information communicated accurately. This will help you choose the best method.
5. When you have drafted your message, have a trusted colleague give you feedback on your draft.
6. Determine the best time to communicate results (immediately following a program, before budget decisions are made, leadership transition time).
7. Reach out to your stakeholders with the information. Offer to meet with them in person, if applicable.
8. Build this step into your annual planning process.

If you are fortunate enough to have marketing and communications staff available to you, consult with them for additional resources. As always, Student Life Studies is here to help you maximize your assessment. Let us know what you need, so we can better serve you.