Member Spotlight: How PALSave’s Data Plan Demonstrates Open Ed Program Success

One of the challenges we often hear from our members is how to demonstrate the success of open education initiatives. In this member spotlight, we examine how one member uses an ambitious data plan to meet this need.

PALNI, or the Private Academic Library Network of Indiana, serves 24 private academic institutions across Indiana, and is a member of the Open Education Network (OEN). PALNI’s affordable education initiative, PALSave, works to “improve student success and retention by supporting the use of course materials that are free to students.” To make its case for the positive impact of zero cost textbook adoption, PALSave sought to create an ambitious data plan.

PALSave launched in 2018 after PALNI’s Affordable Education Initiatives Task Force’s scan of peer programs demonstrated the need for such a program. The PALSave team kicked off by doing workshops and offering faculty the opportunity to write reviews for the Open Textbook Library; hosting two OEN Train-the-Trainer trainings for their campuses; and sending their PALSave leaders to the OEN’s Summit Institute and Summit. In 2019, PALSave was awarded a $520,000 grant by Lilly Endowment, Inc. to fund the program for 5 years. 

Understanding the need to gather and evaluate massive amounts of vital program data for the grant, Amanda Hurford and Erin Milanese, PALSave Leads, set out to create a data plan to measure the impact of textbook prices, zero cost textbook adoption, and perceptions of open textbook quality.

PALSave’s metrics for success and impact were many: impact on student success and retention, cost savings, open educational resource (OER) awareness, and more, across its diffuse network of campuses. To properly assess these metrics they needed a data-driven approach. These data would also support performance indicators in PALSave’s grant reporting and stakeholder communication. 

Amanda Hurford, PALNI’s Scholarly Communications Director, says there’s a misconception that private schools don’t care about the cost of educational resources. In reality, she explains, private institutions have a particular need to be cost-sensitive in order to recruit and retain students, making impact data valuable in securing administrative buy-in. With this in mind, her team leveraged OEN data tools to create a data plan.

The team began by analyzing their OEN workshop data, including the number of faculty open textbook reviews and adoptions they had tracked using the OEN’s Data Dashboard. The Data Dashboard is a tool for OEN members to examine and analyze statistics about faculty open textbooks reviews, OER adoption, student cost savings and more. The team credits the data available to them as OEN members for laying the foundation of their plan, and helping them frame the research questions that would guide the plan’s development. 

Using a combination of PALSave’s course tracker, student success & perception surveys, OEN data, and an IRB-approved plan to survey students in comparison with the widely-used 2018 Florida Virtual Campus Student Textbook and Course Materials Study, the team completed their data plan.

PALSave’s data plan was created in February 2020, and its public rollout is ongoing. It seeks to collect data demonstrating the need for the program, assess its impact, and gather further information about the use of zero cost resources at private colleges in Indiana. The plan consists of eight research questions and identifies tools to answer those questions. The plan will guide PALSave’s future work in workshops and redesign grants, and the increased faculty support it has generated will enable the team to reach even more ambitious goals.

To view and/or adapt the PALSave Data Plan, visit https://tinyurl.com/palsave-data-plan.

Erin Milanese, PALNI’s Affordable Learning Project Coordinator and Head of Learning Technologies at Goshen College, reflects that the biggest lesson she learned through her part in creating the data plan, is that there is no one silver bullet approach to collecting these various forms of data. Instead, the best course of action is to use a variety of research tools.

Erin and Amanda describe having to work within the prescribed confines of grant goals as being a kind of blessing in disguise. They point to the unexpected value of constraints necessitating creativity. Rather than succumbing to “survey creep”—dumping an endless stream of questions into a single survey to get at every conceivable research question they were interested in—the team had to focus on targeted methods of gathering data using a number of instruments.

The data plan allows PALNI supported institutions to demonstrate the need for expanded OER programs. Some of their key findings include that most (66%) students felt that open/free textbooks helped their studies, over 3,700 students have been impacted, and the program is projected to save $1.25 million over 5 years.

Learn more about the PALSave Data Plan at the Open Education Conference 2020 Lightning Talk: Studying an Affordable Learning Program with Data: PALNI’s Plan, Tools, and Results.