April Office Hours: Storytelling with Anecdotal Evidence and Data

Apurva: Hello everyone, welcome to another Office Hours session. I’m Apurva Ashok, I’m the project lead at the Rebus Community, and I’m joined today by my wonderful colleague, Monica Brown, who is Rebus’ new assistant program manager as well as one of our favorite collaborators, the Open Education Network. We have Karen Lauritsen from the Open Education Network and Barb Thees as well. 

Thank you, Barb, so for anyone who doesn’t know about the Rebus Community, we are a charity based in Canada and offer programs and resources to support all kinds of open publishing efforts. Today, we’re going to be talking about storytelling with anecdotal evidence and data, which is quite an exciting topic. And also, if this is your first time attending Office Hours, these are pretty casual conversations. 

We typically give our guests five minutes each to share their experiences about this topic, and any advice they might have for you. And then, we turn things over to all of you, for your questions, for your thoughts, for your experiences. We are always open to your suggestions for future topics or for future guests. So, if you have someone in mind that you’d like to be featured on Office Hours or something that you’d like us to dig into as a community, please do drop in your suggestions in the form that we’ll link to in the chat. 

As I said, we have a great line up of guests today, but I want to hand it off to Karen to tell you a little more about OEN and then introduce our guests and topic. 

Karen: Thank you, Apurva. It’s always so nice to co-host Office Hours with you and the Rebus Community and we’re so excited that that now includes Monica. So, welcome everybody. We are happy to be here with all of you as well. We have a great line up with four guests. And as usual, I’m sure that there’s also a lot of expertise in the community and in the group that has gathered here today. 

So, the Open Education Network is a community of professionals who are working to make higher education more open and they share strategies and support for doing so. Today, as Apurva mentioned, we are talking about storytelling with anecdotal evidence and data. It is kind of a part two from last month’s session when we talked about social media storytelling. So, hopefully those two will complement each other. 

And I just want to highlight again the form that Barb dropped into the chat, we are always thinking and talking about how we can include more voices, plan conversations around the topics that matter to you. And we really need your help to do that the best we can, so any suggestions you can make for topics or colleagues who you think would be great to talk with at Office Hours are deeply appreciated. 

So, without further ado, I will go ahead and introduce our guests today. We are joined by Urooj Nizami, who is open education strategist at Kwantlen Polytechnic University. Erin Milanese, who is affordable learning project coordinator at PALNI, which is Private Academic Library Network of Indiana. Jeff Gallant who is program director with Affordable Learning Georgia. And Genya O’Gara, who is virtual library of Virginia, otherwise known as VIVA deputy director. So, to kick us off, I will turn things over to Urooj. 

Urooj: Thank you so much. I am also going to share my screen and I hope to stick to five minutes, and if I don’t, I’m just going to wrap up really quickly. Okay. So, hi all, in the next five minutes I’m going to try and share four ways that KPU demonstrates impact of our various open ed initiatives using data. But first, I want to start with a land acknowledgment. 

At KPU, we work, study and live in the region south of the Frazier River, which overlaps with the unseated traditional and ancestral lands of the Kwantlen, Musqueam, Katzie, Semiahmoo, Tsawwassen, Qayqayt and Kwikwetlem peoples. I also want to say that the foundation for a lot of what I’m going to share today was laid by an amazing open ed working group at KPU, led by Rajiv Jhangiani.

I’m just lucky to be able to share some of what folks have been able to come up with. So, first it’s really tough to talk about just one of our open ed initiatives at KPU without talking about another, because we’ve tried to create an ecosystem where our initiatives really rely on one another. So, please bear with me as I sometimes speak around our OER initiative in particular. 

And this first example is a good case of that. So, our zero textbook cost program, our courses, and even soon to be eight full programs that do not require a textbook to earn a credit or a credential. So in working with the office of the registrar, along with each faculty in our campuses, we have been able to add a filter on our scheduling data that allows students to select courses based on this attribute in the course selection timetable. 

So, of course, a large part of our ZTC courses are made up of those that make use of OER. This is a very short video, and in the spirit of today’s theme data and storytelling, I wanted to share this micro campaign we launched with the goal of informing students that they can search ZTC courses in the course timetable. And we used an all too familiar scenario to demonstrate this. 

And I want to touch on the fact that I’m talking about an attribute and it’s not always seen as data, but we’re applying it to data on the backend. This doesn’t necessarily have volume, it’s a texting video, so I hope you’re able to see it. 

[VIDEO PLAYS]

Okay, so the ZTC attribute, as I hope we’ve just demonstrated, is worth it just because it increases transparency by allowing students to be informed about their choices when it comes to course selection. But it really has an added benefit, and that is that we’re able to use the data collected by the office of the registrar to compare and contrast student enrolment persistence and performance across ZTC and non-ZTC courses. 

And because of this, we’re able to make claims around student enrolment persistence and performance at KPU. And we’re able to show that OER are better for students. This is recent data that I analyzed for the school of business, using this data dashboard that the office of the registrar makes available. And it clearly shows that the withdrawal rate for ZTC courses is a full percentage point lower than that of non-ZTC courses. 

And for the school of business this means bringing down the withdrawal rate could mean 400 students staying enrolled in a course, which is quite significant. Here again, we see that in terms of student outcomes and performance, those in ZTC courses, so those that perhaps use OER instead of commercial textbooks perform better 0.18 GPA points better. Next, we’re able to demonstrate the popularity of KPU published OER. 

So, we’ve configured Google Analytics for Pressbooks and just one of our KPU published OER, learning to learn online received close to 70,000 page views during the pandemic. This tells a pretty compelling story for why supporting OER authors is so vital, both for our administration to understand that these programs are not all for naught. They really do make an impact, people are really seeing them but also, for those considering creating or adapting OER. 

And finally, like so many of our colleagues across North America and the world, we make an effort to showcase faculty and student champions by gathering and sharing testimonials. And I think it’s really important to think of data not just in numbers, but also that rich, thick data that faculty provide perhaps in the form of testimonials. It’s been key for us to continue our open ed initiatives at KPU. I’m really excited to hear what my colleagues have to say and learn from them. So, better stop sharing. 

Karen: Thanks very much, Urooj. The video was a hit! Okay, now turning things to Erin. 

Erin: Yeah, that video was awesome. So, as Karen mentioned, my title is affordable learning project coordinator for PALNI. And PALNI is 26 private universities and seminaries in Indiana. And a lot of what I do is data wrangling, so I help develop the tools to collect data, analyze it, and do a lot of reporting on it. And it was really interesting, in preparing for this session today, to really think about how our data and narratives have changed over the last few years. 

I’ll also share my screen here in a minute to show some examples. But at the beginning of our program, when we didn’t really have a lot of what you’d call hard data, quantitative data about our institutions, we relied a lot on anecdotal stories we were hearing, both from students and from faculty. And we basically made a big Google Doc and we shared it with our brilliant marketing coordinator at PALNI, Megan. 

And she created these really great infographics for us, one is a faculty journey and one is a student journey. So, I’ll share my screen and just show you those quickly. And then, I’ll pop them into chat. And let me know if the share doesn’t work, because there’s a weird glitch with Macs right now. Are you seeing that? 

Apurva: We see it. 

Erin: Okay, thank you. So, this is the faculty journey, and again, this is an amalgamation of seven or eight stories that Megan took and created this very great infographic with an example of a professor who adopted a resource and how her perceptions changed. And then, it ends with what can PALSave do for you? And here’s the student journey. So, similarly this is an amalgamation, not any particular student. 

But we talk about a student in a health sciences major, at a PALNI school, the impact of textbook prices. But then, oh, there’s a class with an open textbook and it’s great. That’s this experience, and again, ends with this is how we can help you. So, early on, this is how we promoted the program, just to gain traction at our various institutions, and it worked. We had great success with being brought to various campuses to give presentations on open educational resources. 

We also used a variation of these stories in the front part of our grant proposal to the Lilly Endowment Grant. And I don’t know if the stories helped us get the grant, but we did get the grant, so we can at least say it didn’t hurt us. But yeah, I think that was one way we used storytelling and specifically with anecdotal data early on. So, since then, now we have a lot of data about our institutions. 

And so, our reporting feels a little less story and more just reporting. But there are always those qualitative bits in our survey data that I like to highlight. So, one example that jumps out, up until two months ago I worked at a PALNI institution, Goshen College. And as I was compiling data for the administrators that I knew, I was trying to highlight some of the student voices that came through. 

And one in particular, it was actually really similar to the example in the texting video you just shared. But it was a student who said, “I’m a student athlete, and I still have to work three part-time jobs to afford textbooks and pay the rest of my tuition.” And I think having those examples, even in the midst of a giant report that’s really quantitatively focused really brings that voice across. 

The last thing I wanted to mention is that along with Amanda Hurford our scholarly communications director at PALNI, we’ve put together a really thorough data plan. And I can put a link to that in the chat. But that’s just really helped us hone in on what our research questions are and make sure we’re collecting the right kinds of data and how that might be shared. So, I’ll stick that link in chat here in a second. 

Actually, do it right now as just an example. And this is a Google Doc that you can copy, if you are interested and I’m happy to answer any questions or talk more about that. But I’ll stop there. 

Karen: Thanks so much, Erin, and congratulations on the grant. Okay, over to you, Genya. 

Genya: Hello, I’m muted here and started. Thanks for having me, this is such a great group, I’m already inspired by the things that have been shared. I’m not going to be able to address everything in five minutes about our program from the prompt I was given. But I’m going to try you all a general flavor of our approach at VIVA. I don’t have any slides to share. For those of you that don’t know, we are the academic library consortium here in Virginia. 

And there is a quote by Tutaleni Asino that I’ve been using a lot in presentations about our program that is, “There are many gates through which people enter the open house and it’s okay for those many different gates to exist.” And when I’m thinking about how to effectively tell the story of our program and our efforts to different stakeholders and communities, I think about this a lot because it’s really true. 

We have 71 different member institutions in VIVA and their reasons for engaging in the open and affordable world are as diverse as they are. This means that we really have to have a multi-pronged approach to both assessment and to storytelling with the data that we do collect. So, our VIVA open education network program was really our first consortially supported OER program. 

I think it was back in 2016 that we started it. And the initial success and take up of the pilot program gave us the data and the stories to take to the general assembly to ask for support for broadening our open and affordable program across the state. And so, for this, we really used a combination of student cost avoidance, stories of the type of support faculty truly needed to actually be able to redesign their courses with open materials. 

Anecdotal data about the transformation in classrooms and student success when they had access on day one, to really tease out the story of what this program might be, beyond just savings. And we were able to do that effectively, we were funded, we received $600,000 to our base budget, which is earmarked specifically for open and affordable programs. This really allowed us to expand the program and hire dedicated staffing for it. 

Which I’m the biggest proponent of dedicated staffing, dedicated staffing. I’m sure some of you have worked with our amazing colleagues, we’ve got Dr Stephanie Westcott and Sophie Rondeau, who truly shepherd this program in the ins and outs of every day. And briefly, our open and affordable initiative now includes VIVA Open, which is a searchable OER repository aligned with Virginia higher ed curricula. 

We’ve got a VIVA open grants program, where we try to empower Virginia faculty in adopting, adapting and creating open content. We still have the VIA OEN program, which trains leaders at our member institutions to support their teaching faculty. And we provide all of the stipends for faculty for this work. 

And then, we have what we think of as the VIVA affordable component of our program, which provides a way to search and match ebooks in a faculty portal that are free to students and it includes open content alongside library owned content and for purchase content. So, the way we track our work, it’s pretty straightforward. From each of these programs, we track the classes where OER is being adopted, what texts they replace, how many students are impacted. 

These big numbers are really great for administrators and for the legislature, so we can say things like as of December 2020, the OEN component of our program alone has resulted in 150 known adoptions of OER. It’s impacted over 18,000 students. It’s resulted in over $2 million in savings. And that’s incredibly satisfying to rattle off, but it isn’t really why or how the majority of our community engages in open or how it is really embedded on our campuses. 

So, we always try to go beyond savings. We collect a lot of quotes and stories about how the resources are being used, most often in the final reports from our grant recipients and from our library leaders. This type of information is really how we humanize those huge numbers. We have faculty stories that range from talking about how critical redesigning with open resources has been to aligning assessment and instruction. 

To rethinking pedagogical practices and more, including data about student performance improvement. And the information we get from students is often about ease of use. More than even about whether or not the resource was free, which I think is a really important thing to note. So, I think of savings numbers as a gateway conversation to a much deeper engagement about how open impacts our community, if that makes any sense. 

We’re also doing a lot to ask questions to get the data, it sounds like PALNI is, to help us ensure that this program is actually aligned with our consortial goals. Particularly in our anti-racism and equity diversity and inclusion efforts. So, to that end, we’ve done things like in the RFPs for all of our open grants, they must include how they demonstrate very clearly how they are going to impact educational equity and student outcomes in traditionally underserved communities. 

We’re also embarking on new kinds of assessment, we’re in the beginning stages of launching a statewide student survey about textbook costs, which we know Florida and others have done for years. But these surveys have really focused predominantly on the impact of textbook costs on students without really connecting how the costs impact the broader questions of educational equity, which is what we’re going to try to do. 

And we’re hoping by addressing this more specifically as we collect our data, we’re going to be able to tailor our program and advocate for the resources to support the specific needs we find. And then, I know my five minutes, I was like ah! Five minutes. Briefly, we’re also building our own database to track program information across the different components. So, we can better review our program as a whole, and across institutions and subject areas. 

We think it’s going to help us better identify gaps and overlaps in both disciplines, subjects and really the communities that we’re not reaching right now, because there are communities that we’re not reaching. So, these are all examples of the ways we try to use both data and community stories to both help us build and advocate for OER in ways that are most meaningful for our community. And that’s all I have. 

Karen: Thank you. And I appreciate what you’re saying about humanizing numbers. There’s a need and a place for both the data side and the story side. Okay, Jeff, over to you. 

Jeff: Thanks. I am presenting from Lawrenceville Georgia, originally home to the Muscogee and Timucua first nations. I could describe Affordable Learning Georgia for a very long time. But I’ve got five minutes, so I am going to share a link for the About Us page. It’s got our current strategic direction. So, from there I’m going to try to share some stuff on my screen. Here we go. All right. 

So, there is a lot of ways that we’ve used data since 2014. One of the first ones is one that you’d be very familiar with. How many students have been affected by a grant program over the entire time that we’ve been in existence? And how much estimated costs that has saved students during that time combines with our textbook transformation grants program and our eCore and eMajor partnerships, that’s now over $100 million. 

That sounds really cool, but as each presenter has said, that’s only one part of the puzzle. There are a lot of different ways that you can use your data, not just to tell the story externally, but to tell it internally and drive your next strategic goals. One of the big things that we did in 2016, when we knew that we were going to be a long-term program was create a new strategic plan. 

It was going to be three years, one of the first things that we said was well, we don’t know what we need until we go out there and find it. We’re going to do a needs assessment survey. That survey had a lot of quantitative questions, but it also had a lot of open ended response questions. I was partnered up with Marie Lassiter, one of the pioneers in open educational resources here in the USG from eCore way, way back. 

Wrote a white paper on it in 2011. So, what she did was she brought her experience in the psychology field over and did qualitative coding and this is inductive coding. So, we did not have a code book before. We took the quotes from every faculty member and professional staff member through the survey and we added codes to them. We started to find different trends among them. 

This one, for example, is the one that says, “OER are poor quality”. So, we’re looking at not just a quantitative number of we think that this is poor quality, but we’re looking at the different perspectives in here from everyone. Not only that, but we’ve identified the number of codes for each quote. So, some of them go into other fields, for example, accessibility, lack of ancillary materials, student efficacy perceptions. 

Another thing that we found here in the survey was one that we didn’t plan for, we didn’t have any questions about it, but it was a print versus digital trend. A lot of people were very passionate in their responses that students learn better with prints. They always need the option, the research says that it is that way. The research is varied and more needs to be done, but that’s neither here nor there, we’re looking at the perceptions. 

So, we knew that print versus digital was now a big thing, because we were taking this anecdotal advice, these open ended responses and bringing them together into categories, and then looking at the why towards these things. So, anyway, all of this results in a USG OER survey that includes the Babson survey research thing from before that said what are the three most significant deterrents to the use of open educational resources?

Our first one was not high quality, it was even the biggest one among people who use OER and our grantees. So, we wanted to go into what does that mean? So, we already had all of these OER poor quality quotes. So, we were like okay, what are we trying to sum here? So, we found that determinants of quality included a whole bunch of different things. So, we’re trying to then address all of them in the next strategic plan, especially prioritizing the most important ones. 

So, that’s one way of conveying to the public what’s going on in the USG and conveying to us what we need to do next. Also, let me just go into our tracking document. So, we’ve been tracking all of our grants for a very long time. One of the big questions is how do you know what happens four years down the line? Well, we survey everyone, we have our champions reach out, we really try to get them to respond. 

If they don’t respond, we zero it out, if they’ve discontinued, we zero it out for the year. That’s all done through some if then statements. You either put in the students because the project discontinued or you don’t and then, you don’t multiply them. So, that winds up leading over to our data center. I’m going to show you that very quickly. In here, we use PowerBI. I’ve got a data dashboard over here. 

Just uses that same Excel sheet, and the institution reports pull right from PowerBI. So, it could tell you what’s going on at Georgia Gwinett College. Total savings, where are the subjects it’s affecting, how many low cost and no cost materials are implemented, regardless of our grants. And where are they happening. That’s for our administrators. And then, if you just want a big, overall look at it, we’ve embedded the PowerBI data into our dashboard. 

And you can select your institution, and it will show exactly that share of the data within that institution. So, we’ve done a lot of things, and I’m more here to answer questions than to explain our entire data strategy today. So, thank you very much, and I’ll move right on. 

Karen: Thanks Jeff. And thanks for sharing the behind the scenes spreadsheets with us. Thank you to all four of our guests. There’s been a great chat already happening. And so, as you think of your questions or follow up for our guests or for one another, I would like to turn things over to Barb Thees, the OEN community manager, she’s been working closely with Dave Ernst, our executive director and members on developing a data dashboard. 

And when we started talking about today’s topic, we thought hey, wait a minute, maybe this is an opportunity to sneak preview some of the dashboard. So, over to you, Barb. 

Barb: Thanks, Karen and thank you everybody for sharing so far, this is really inspiring. And I might follow up with some of you about our data dashboard process, which I already have talked to some of you about it. But as it currently stands, we do have a data dashboard that we offer to our members, that tracks faculty workshops, like the ones Genya mentioned. 

And follows up with faculty after their workshops, asking them to review a textbook in the Open Textbook Library and then tracking any reviews that they ended up doing for open textbooks as well as sending out a survey asking them about their plans to engage with or to adapt an open textbook as a result of engaging with these workshops. That’s where we’re at right now. 

It’s a great system, it does give some of these snapshots that are helpful in terms of advocating and storytelling around OER programming. But that being said, we’ve heard from our members that as we’ve seen here, there’s a lot more work in OER initiatives than just these faculty workshops. So, as Karen mentioned, that’s what we’ve currently been working on, is adding to the functionality of our data dashboard so that it will track grant programming. 

It will track other kinds of events and cohorts that our members will be engaging faculty campus partners with. And then, allowing a more robust method of tracking adaptions as well as updating enrolments. And not only for the administrators to do that themselves, but also to allow and invite faculty to be updating that information regularly. Another feature that I’m really excited about that, Jeff, I’m going to follow up with you on is that we want to expand our reporting capabilities as well. 

And allow the system to create some of these visuals, that as you could see in Jeff’s example are really impactful in illustrating the information. So, we’re working on that now, Dave Ernst and I. And that’s something that we hope to rollout this summer in conjunction with our OEN summit. So, stay tuned for that, our members will get communications about that via the Google group. 

Those who aren’t OEN members, if you want to talk about that offline or chat at all, I’ll drop my email in the chat here. And I’m happy to either tell you more about what we currently offer for the data dashboard or where we’re going with it. 

Apurva: Thanks so much, Barb. I hope a lot of people follow up and we’re excited to see all of the work that comes out of you folks at OEN. I know that you’ve been working long and hard on that dashboard. So, everyone, this is really the time when we turn it over to you for questions and comments and thoughts and scenarios. I know that there’s already been a few in the chat that might have been answered by our guests. 

I’m going to give you all a few minutes, you’re welcome to unmute or drop your question in the chat. And I might just kick us off with the first one, while you do some thinking. Erin, I think you mentioned a data plan when you were sharing your pieces about the topic earlier today. And I know Urooj, you and Genya both talked about teams and just various considerations that you take going into collecting the data. 

And I’m just curious about the types of conversations that might be happening within the team, as you’re putting together that data plan. What sort of questions are you asking one another? And I know Jeff, you too, you have quite the robust set of spreadsheets and processes and workflows. What are you asking before you dive into all of that content? How do you begin putting together that plan? What leads you to it? 

Erin: I guess I can start. We started our data plan in large part because we did get a grant from the Lilly Endowment Foundation. And we knew we had to report various things back to them, and we wanted to make sure we had the tools in place to do so. So, that was really where we started. But even without the grant, I think we would have needed to have those sorts of conversations. 

Like what do our institutions need to hear from us to be interested? What’s most valuable to them? What do professors want to know when they’re adopting these in their classes? What do they want to hear from students? But for us it really was that Lilly grant. And then, the other question we asked is what are other consortia doing? What are other institutions doing? 

So, we spent a lot of time looking around and talking to people who had had programs longer than us. And as we put the plan together, we looked at various tools. We looked a lot at that Florida textbook survey that’s been published and other student perception surveys that are out there. Because we didn’t want to start from scratch when there’s so many good surveys already out there, plus is allows us to compare our data really closely when the questions are the same. 

And then, we were really conscious of what data we maybe didn’t need. There’s a lot of survey creep that can happen. And making sure that we’re keeping our survey short so that people will fill them out and not overwhelming people by asking too many questions. So, that was where we started, at least. 

Apurva: I think that’s helpful to hear about the guiding questions behind making the plan itself. Urooj, what about you? I know you’ve talked about weaving in a strategy across all of the open initiatives. How have you made sure that you’re not collecting the same kind of data from different surveys or duplicating information?

Urooj: I think to start answering that question, I got so excited when I saw what Jeff was doing, because it’s something that we’ve been doing, but want to also integrate PowerBI. But at the same time, I have all of these questions about that hard numerical data. Oftentimes, we need to do that to prove our programs to our administrators or governmental bodies as some have mentioned. But that thick qualitative data is not valued in the same way. 

But to me, it seems so impactful so, I guess the way we think about it at KPU on our working group is we’re speaking to different audiences, and I know this is a perspective that a lot of us take. Students may not care about how many adoptions there have been, they may have different concerns. So, speaking to them has to be different than our administration or faculty. 

So, I think those are our guiding questions where we know we’re speaking to different audiences. So, what sort of data do we have to collect and present and how do we present it to be appealing to various stakeholders on our campus? I hope that began to answer your question. 

Apurva: It did. Genya, what about you? I know you talked about dedicated staffing and the team. So, what are the three of you, and more, I’m guessing, the VIVA team is much larger, talking about as you’re making your data plans and setting some of those goals across the organization?

Genya: It’s a really good question. I like very much the way you all have framed it. I think Jeff said something about internal versus external audiences. We’ve been talking a lot about there are those big savings numbers that everybody likes to get on. And we do collect that data, I don’t mean to imply that that isn’t important data to us, because it is. 

But those stories and understanding where are the intersections in the things that our communities need internally, that’s really the data that changes the way our program is run and how we support different groups. So, what we’ve been looking a lot about how do we talk about sustainability at a much higher level beyond savings, really? Because there are so many different kinds of sustainability. 

And for us, the most important part of sustainability of this program because it really started as a grassroots program in Virginia. The things that are successful in VIVA are grassroots programs that we see our community needs, and then we support deeply and spread the infrastructure. So, when we’re talking about sustainability and what to collect right now, it’s about how do we better figure out what our community needs and how we need to support them in getting there.

So, we’re looking at a lot of things like program overlap, so we have this grant program, and we have an open and affordable program. We have a grant program and then we have the affordable component. Where are the places where we’re not seeing a lot of applications for grants and people aren’t finding a lot of open educational resources, where we’re not seeing a lot that are aligned to different curriculum?

Is that a place for dedicated requests for proposals for grants, to help create more? What is the reason behind that piece of data? So, that’s the sort of stuff that we’re trying to look at. Because if our community is not deeply engaged in this effort, then it almost doesn’t matter how much the savings are, because they’re our advocates. This program is for them. And by community be mean our librarians, our faculty and our students. 

And those are very different kinds of data, that we have to somehow weave them all together. So, I don’t know if that was too convoluted. We actually have a planning meeting coming up with our open and affordable program in a couple of weeks, where this is all we’ll be doing is taking the different data that we have and seeing if it really is answering the questions that we have. Are we actually meeting our goals in terms of sustainability and community engagement in this program? 

Urooj: I think what Genya just mentioned, I’m sorry for interrupting perhaps, is so important that we can use data to identify the gaps that we have, as well as demonstrate the successes. So, just noting at our institutions which OER are getting picked up, but in which courses they’re not. I mean, that’s not something that you necessarily are going to write in a report to demonstrate how great your program is, but it’s stuff you can use to innovate your program, and I think that angle is so important. 

Genya: I’m sorry, I’m interrupting again. Also, to build on the OEN program of faculty reviews, we’ve taken that and started reaching out to faculty that have been involved in various components of our program to do reviews of what we’re calling our course alignments, where we’re aligning courses with either existing OER, OER created in Virginia that works for all, for example for our VCCS, which is our community college systems to say here are potential open educational resources in this arena. 

And so, taking a way it works in one program and using it to support faculty and really interrogating this stuff in another area has also been really helpful in figuring out where we could use one approach with another group. 

Karen: I know it’s always a little awkward in a Zoom call, but as much as possible, feel free to jump in whenever someone says something that inspires a comment. Thank you all for that discussion. One of the things that jumped out at me when listening to the four of you was the commonalities across wanting to show return on investment, student savings, how much the resources are used, for example. 

And then, Genya, you actually mentioned one that struck me as oh, I hadn’t thought about that as a data point, which maybe isn’t quite right. But you mentioned wanting to show the general assembly the support that’s truly needed for faculty to convert their courses. Can you say a little bit more about that? 

Genya: Sure. I know that these other programs do that as well in different ways, so I’m sure everyone has stuff to add here. But it is about the stories of it’s one thing to say you should adopt this open educational resource, and it’s another thing to talk through what the process looks like for a faculty member who has to completely redesign an entire course or courses in order to incorporate any open educational resources or any no cost resources into it. 

This is a huge and heavy lift we ask, even when it is just the adoption of one book, even if the benefits can be long term. And so, talking through the kind of support that they needed for that in addition to the savings numbers were really how we got our grant program funded, because it’s not that faculty are uninspired or uninterested in doing this work. 

It’s that there is not a support system in place to give them the semester off to pay for their time and their expertise in doing this redesign that is often not supported by a homework platform or the other kinds of things that they already have in place within the department. And so, telling that story has been critical into how we get support for our grant program where it’s never enough. 

But at least are allowed to do some financial support and get a dedicated letter of support from the department of the faculty member applying for the grant saying that they support this work. And that they’re funded at least in some part to do the heavy lift of a redesign of a course. 

Karen: Anyone else care to comment on the faculty support piece?

Apurva: I also picked up on a few measures when Urooj was talking about not just enrolment and completion but also persistence and performance. Some of these non-academic measures of OER impact, those are pretty cool. And Genya, I’m really looking forward to that state survey that you’ll be conducting and the types of measures you can bring in and maybe any big gaps that you’re able to identify. 

Jeff, I was wondering when you were sharing your various spreadsheets and reports and workflows it looks like you have an extremely established system at Affordable Learning Georgia. I’m wondering what did you start with, and how did you end up to where you are right now? I imagine that there are a few people on the call who might not have the support capacity resources to be using all of the different tools that you have. What would you suggest they start with?

Jeff: Yeah, I was almost going to chime in on the first question, because often when people are like, “Oh, that’s a cool data strategy, it’s great you got that started from the beginning.” It’s like, “No, that’s not where we started.” When we first got money to run this program, it was program funding, they gave us one year. We didn’t know if we were going to last another year. 

They gave it to us and said, “Go.” So, we had to demonstrate the potential impact of the grant projects that were going to happen, in a very compressed period of time. One semester to get OER in the classroom, one semester to implement and measure it and report it. And then, we had those results immediately. Yeah. State government. So, the impact was communicated only through potential annual students affected and annual per student savings. 

And that’s what we had at first, we had our grants, here’s the number of the grant in one column. We had the annual savings in another column. We had the annual students in another column. There wasn’t much else, that was about it. And we got caught behind the ball. 2015 comes around we got another round of funding. Once again, one year, we have no idea if we’re going to last. 

And in the middle of I believe it was Richmond Virginia, the open ed conference there, right when there was a reception with free food and all this stuff, I got the phone call from one of our high up vice chancellors, asking for some data that we did not plan for. Because suddenly, we had these annual impact numbers, and they wanted to know, well, okay it’s been maybe three semesters of implementation. 

So, how much have you really saved students up until this point? And we’re like, “Well, we don’t exactly have what semester they started and what semester they ended.” We can start guessing, but we had only gathered annual. And we had only gathered the savings and students affected. We were going to have some real trouble. So, our reports came with a lot of caveats after we normed a lot of data from various sources. 

eCore had like five different reports and one of them said this, one of them said that. And they’re like, “Well but the numbers are different here.” And I’m like, “Yeah, that’s because eCore sent this one and this one.” Now, we finally have a good data strategy, I just want to show you something real quick here on screen. Don’t worry about this PowerBI thing, I might show this later. 

So, goal five of our first ever strategic plan, which was developed in 2016, when we finally got funding that said, “You’re going to be an ongoing program.” It was all about expanding and improving data collection, because we were way off from where we wanted to be. The first one was expand the scope and improve the accuracy and granularity of the measurement of the quantitative impact of ALG programs. 

That meant being able to answer the question what semester did savings start? If it got discontinued, where did it get discontinued? And you are no longer tracking that, right? And so, that part of it was what you’re starting to see in that ALG tracking sheet, the if then formulas, the sustainability checks. That all goes into it. And then, tracking all the usage of open, no cost and library materials across the USG, that was a different thing. 

We needed course designators for that. I’m sure many of you are very familiar with that, there is an open textbook on course markings at this point. I believe it is a Rebus text. One of the things that we did with it, even though the numbers were kind of inconsistent, some places were not recording it correctly. We were asked to identify some of the gaps between courses that were high enrolment but didn’t have any no cost usage. 

So, what we did was we did these bubble charts here for each institution and this is just through an Excel spreadsheet feeding into PowerBI. And I can click on Dalton State here, and I know what’s going on. So, college algebra, zero no cost materials at Dalton. Pretty highly enrolled, 640, Dalton’s a small place. But if you look at this one, fundamentals of speech, the enrolment number which is that green circle and the no cost number which is that gray circle are overlapped. 

That’s because Dalton State had a team that authored the exploring public speaking text, which is now in its fourth edition and is the highest downloaded open textbook created by the USG. So, obviously, at Dalton State that’s going to be a big one. But what we did find were that the top four courses across the system that had some gaps that we could possibly look at were psych 1101, American government, composition one and composition two. 

What we found when we asked professors about these was that American government, composition one and composition two had some real issues to making big departmental wide scale ups. Partially due to American government just having very subjective and opinionated materials that everybody latched onto. And composition one and two had varied materials across the board, that weren’t necessarily high cost materials. 

So, we dove into psych instead, and we have a psychology adaptive learning pilot that came from interviews and all kinds of anecdotal stuff that happened. So, this stuff doesn’t come out of a vacuum, or it doesn’t come out of me being a huge experienced data analyst. I was a musician, I became a librarian, and now I do this data stuff because we ran into various fires and had to put them out. 

And then, put up fireproof things all over the place, as a response. So, that’s why. We really started with nothing, we started with very small Excel sheets. The Excel sheets are big now, I’m glad Excel can handle that amount of data and that my computer doesn’t explode. And that just feeds into PowerBI at that point. It takes time, just like any OER project. It takes a lot of time. 

It could take any staff member, they can get to know Excel and they can get to know how if then works. And they can get to know how to put stuff in PowerBI. You don’t have to have a degree in data visualization to do it. But you do need a lot of time. And conveying that if you do not have any funding for that or any full time people is a tough one. But that’s the stuff they’ll ask for afterwards, so you will need somebody to put in that time. 

Karen: Thanks so much, Jeff, for demystifying how you came to be at this particular point and echoing some of the other things that our guests have talked about today in terms of the value of staffing and showing the impact of your program. We are nearing the end of our hour together. So, last call for any questions, either from our guests for one another, or from our attendees for our guests. 

And another request to please fill out our very short form on ideas for future Office Hours sessions. We greatly appreciate that. And then, finally, please join us in thanking our guests, Jeff, Urooj, Genya, and Erin for joining us today and sharing their strategies and techniques for storytelling with anecdotal evidence and data. Apurva?

Apurva: I might actually turn it over to our guests to see if they have any final thoughts, and feel free to jump in, any of you, if there was something you wanted to share, but didn’t get a chance to. All right, Jeff thinks he’s good, Urooj, Genya, Erin. 

Genya: I did want to say one thing that we’re trying to do right now, we do have dedicated staff which is wonderful. But also, there is so much work that can be done in the open field. Part of what we want to use data for is to also understand what we should stop doing, what’s not working for our community. Because we also have limited time, we can’t do all the things, so we’re trying to do a couple of the things that we can do well enough. 

So, I think that’s another important way to use the data, for your own internal. Thank you all for the opportunity, I learned so much from everyone today. 

Apurva: Really useful and likewise. And I’m hoping that going forward we can continue to share our reports and share our strategies and what’s working and not working. And what we’ve decided to start doing and stop doing. I will echo what Karen said, thank you all four of you for sharing your expertise today. And to everyone else for attending and joining, looking forward to seeing you at our next Office Hours in May. Take care everybody, bye bye. 

00:11:49 Barb Thees, she/her: https://www.rebus.community
00:12:00 Barb Thees, she/her: https://open.umn.edu/oen/
00:12:33 Barb Thees, she/her: Recordings of past Office Hours events: https://www.youtube.com/watch?v=hfclOrdD6-8&list=PLWRE6ioG4vdahUrKaiHvsCA2J6NQLysU6
00:12:51 Barb Thees, she/her: Here’s where you can request future topics or even nominate someone to be a speaker: https://docs.google.com/forms/d/e/1FAIpQLScaGr1NCvVnk1C6uKiwkfYWvJcK0QDfwJIZJJV-ckmGK19Wpg/viewform
00:13:25 Apurva Ashok: Welcome everyone!
00:14:34 Barb Thees, she/her: Social Media Storytelling Office Hours session recording: https://www.youtube.com/watch?v=0xkQaJ8hUn0&list=PLWRE6ioG4vdahUrKaiHvsCA2J6NQLysU6&index=41
00:14:59 Apurva Ashok: Lots of acronyms today ?
00:18:32 April Akins (She/Her): Love that!
00:18:41 Monica Brown (she/her/hers): Great video!
00:18:58 Judith Sebesta: Simple, elegant, profound!
00:19:02 James Glapa-Grossklag: fantastic video! URL please, so that I can ask my college’s folks to do one!
00:20:06 Apurva Ashok: +1! We’ll share all the resources in the chat, and will also link to them from the recording.
00:22:42 Jeff Gallant: Journey maps! Love it!
00:23:07 Urooj Nizami: Finding OER— Texting Video: https://media.kpu.ca/media/Finding+Zero+Textbook+Cost+%28ZTC%29+Courses/0_oxam8ndb
00:23:42 Judith Sebesta: Wow! Already thinking how we can adapt that for our own use (assuming an open license). Love infographics.
00:25:22 Erin Milanese: https://docs.google.com/document/d/1P6lkqCjcRnTJftyTDHWMk_Xg5iVRU_pByA1hFJyfXUM/edit
00:26:28 Erin Milanese: faculty journey info graphic link https://drive.google.com/file/d/1SsU3J7S7BRWDTQSmSiVsMjnZrgIuUAnF/view?usp=sharing
00:26:49 Erin Milanese: student journey https://drive.google.com/file/d/1MRsvXln14kmk71RR_l4UnqPr79zIjgih/view?usp=sharing
00:29:22 Judith Sebesta: Erin: Are the infographics you shared openly licensed? I didn’t see on them… TIA
00:31:08 Erin Milanese: Judith – great question. I think they *should* be but don’t have the cc icon on them right now. I’ll be sure to follow up with a cc licensed version
00:31:41 Judith Sebesta: Thanks, Erin! That would be great if possible.
00:32:37 Jeff Gallant: https://www.affordablelearninggeorgia.org/about/about_us
00:34:14 Genya (she/hers) VIVA: Yes, internal is as critical as external in our experience.
00:37:05 Urooj Nizami: This is SUCH good local information to have.
00:37:50 Judith Sebesta: Most impressive, Jeff.
00:38:39 Urooj Nizami: Jeff, you’ll be hearing from me! ?
00:38:49 Jeff Gallant: Woohoo! Feel free – jeff.gallant@usg.edu.
00:41:51 Barb Thees, she/her: thee0017@umn.edu
00:42:31 Karen Lauritsen: Guests, please also feel free to ask one another questions, or share trade secrets ?
00:45:39 Jeff Gallant: When we were first getting started with data visualization, I was talking with Rajiv about his ideas on Tableau, so that’s interesting!
00:46:58 Urooj Nizami: Yup! We’re using Tableau now.
00:58:38 Apurva Ashok: https://uta.pressbooks.pub/markingopenandaffordablecourses/
01:02:10 Deidre Tyler: Thanks
01:02:11 Karen Lauritsen: Let us know: https://docs.google.com/forms/d/e/1FAIpQLScaGr1NCvVnk1C6uKiwkfYWvJcK0QDfwJIZJJV-ckmGK19Wpg/viewform
01:02:20 Apurva Ashok: Topics and Guests for Future Office Hours: https://docs.google.com/forms/d/e/1FAIpQLScaGr1NCvVnk1C6uKiwkfYWvJcK0QDfwJIZJJV-ckmGK19Wpg/viewform
01:02:40 Erin Owens (she/her): Thank you, this was great!
01:02:54 Jeff Gallant: I think I already shared it!
01:02:58 Urooj Nizami: Thank you for organizing!
01:02:58 Monica Brown (she/her/hers): Thanks for sharing your experience and expertise today, everyone!
01:03:03 Rumyana: Thank you, very useful information!