As most public schools struggle to recover from the profound learning loss tied to the COVID-19 pandemic, students in Colorado’s Boulder Valley School District are racing ahead, leaving the dark days of COVID-19 behind.
Boulder Valley’s notable 2024 test scores, which easily surpassed its math and reading marks in 2019, are not the result of a newfangled education theory but instead involve a bedrock principle of American business: return on investment.
Academic ROI (AROI) is a data-driven approach that analyzes the costs and outcomes of programs to identify those that deliver the most educational bang for the buck. Hidebound educators might bristle at the use of AROI, conjuring alarming images of assembly-line learning. But Boulder Valley is one of a small but growing number of districts that are adopting the business practice to target funding for the biggest effect on student performance.
What’s a more cost-effective way to help elementary students who are behind in reading: one-on-one tutoring or four-on-one? AROI helped Boulder Valley figure that out.
It was a no-brainer that tutoring students together rather than individually was less costly. When the district’s research found that small groups were also more effective in helping students learn to read, it shifted its tutoring resources to the four-on-one program as the most cost-effective option.
“The results of four-on-one were a little bit better at a quarter of the cost, so we can now do four times as much literacy intervention,” Bill Sutter, chief financial officer at Boulder Valley, said.
Smaller Classes Don’t Always Pay Off
District spending is often driven by what’s trendy in education or in the interest of staff, without examining whether it’s also the best use of resources for student achievement. New York state, for example, recently mandated that New York City Public Schools shrink class sizes in all grades by 2028 even though there’s plenty of evidence that the multibillion-dollar program will not deliver significant academic gains across K–12, according to researchers who have studied similar efforts elsewhere.“Our IT folks would go to conferences and were told by salespeople that some software was the best thing on the planet,” Carol MacLeod, then the chief financial officer of the Lake County School District in Florida and now an AROI advocate, said. “But we found that some of these software programs were never used. They were probably sitting in a closet.”
AROI aims to ensure that each education dollar is more wisely spent. While the business practice has spread to other fields, including law enforcement, social services, and nonprofit groups, it can face resistance among teachers who are wed to their programs.
The practice emphasizes data-driven accountability, a touchy subject among educators since the controversial federal No Child Left Behind Act of 2002 imposed performance reporting on districts. Even though AROI is controlled by districts, escaping the heavy hand of federal and state overseers, some educators still criticize data-based evaluations as not holistic enough to capture the full benefits of programs.
“The challenge with the AROI process is the pushback that comes with publicly saying that a particular instructional program is not as cost effective as another,” Sutter said. “Educators and parents can be passionate about specific educational programs and the AROI process can get sidetracked quickly if you don’t bring them along carefully.”
Boulder Valley Rolls Out New Approach
At the relatively affluent Boulder Valley district, which enrolls about 30,000 students, the AROI process began about six years ago with a much-needed inventory of its programs. Over many years, the district’s semi-autonomous schools and leaders had frequently added new programs—without ending the older ones—in a process called layering, Sutter said.As a result, no one in the central office, including the superintendent, knew of all of the ongoing programs and whether they were effective. In this way, Boulder Valley was typical of decentralized districts everywhere.
The central office expected that principals and teachers would question its requests to identify their programs and provide performance information, if it existed. Administrators are often seen as bumbling bureaucrats, Sutter said, so Boulder Valley hired an education consulting firm to play the role of the “bad guy.” The consultants collected information on programs and helped develop methods to estimate their costs, performance based on surveys of principals, and alignment with the district’s broad goals.
The inventory uncovered 251 mostly academic and extracurricular programs, such as elementary Honors Choir and Democracy Day, at its 56 schools. They varied widely in effectiveness. The ongoing cost: $46 million.
The revelation spurred the district to change the way it operated. Just before the COVID-19 pandemic, it installed AROI at the center of its decision-making on big and important programs.
When the COVID-19 pandemic hit, the practice helped Boulder Valley strategize about how to maximize the effectiveness of its share of the massive federal relief program. The district decided to build a program that could live on after the aid was gone.
Federal dollars were narrowly focused on hiring teachers, tutors, and coaches, who provided training in data-driven evaluation, in six low-performing schools. Math and English proficiency test scores rose in about half of the schools between 2022 and 2024, and the district is now tweaking the program elements, based on what has worked best, to further improve results over the next three years.
A Culture of Accountability
At the large Mansfield Independent School District in Texas, AROI has created a culture of accountability based on regular data-driven evaluations of programs—a rarity in public education.Every quarter, principals from Mansfield’s 49 schools go before the leadership cabinet in what’s called a “cadence of accountability,” according to Michele Trongaard, an associate superintendent of business and finance who is part of the group. They present their progress in reaching a goal as part of the district’s objectives, such as reading proficiency by the beginning of third grade or algebra II mastery by the end of 11th grade. Almost half of the district’s 35,000 students come from low-income families.
The principals must reveal why they did or didn’t make progress and what they might need to improve. This is where AROI comes into play. The cabinet might not approve a request for additional staffing based on whether it’s cost effective.
The funding decisions are not always black and white because some program effects are hard to measure with data, but the process is crucial to improving the academic performance of students, Trongaard said. The district has made steady gains in math and reading test scores since the COVID-19 pandemic but has yet to fully recover the lost learning.
“We hold everybody accountable,” she said. “We can show the community where the money is going. Absolutely, we are getting more bang for the buck.”
Funding up, Performance Down
Nationwide, the AROI doesn’t look so strong. The influential National Education Association, a union representing millions of teachers, has long been focused on increasing school funding and, for the most part, has succeeded.The academic benefit to students has been mixed. In the first decade of the No Child Left Behind era of accountability, test scores rose in tandem with more funding, producing “steady” AROI, according to the lab. In the past decade, as test-based accountability was rolled back, student performance in math and reading fell, indicating “lousy” AROI.
Mississippi has been a surprising AROI success story, bucking the downward trend. The state was ranked the worst academic performer in the nation a decade ago when Carey Wright took over as state superintendent. She promptly lifted academic standards and then leaned on data to identify the worst-performing districts, providing them instructional reading coaches and measuring the effects on student outcomes. The tight focus on data created accountability and turned Mississippi’s rapid reading improvement into the envy of the nation.
When Wright presented the data to the Board of Education, she spoke in business terms.
An AROI approach to class size reduction might have started with test programs in a few elementary schools to measure the costs against student outcomes. Those findings would then be compared with other approaches, such as high-dosage tutoring, to accelerate learning.
Chingos said the biggest flaw in the state mandate is that it sidelined school leaders who know better than lawmakers about the needs of their students.
Steep Learning Curve
AROI comes with a learning curve for school leaders, which helps explain the slow rollout. Districts don’t typically have the in-house expertise to run formal research studies, but they can tap financial officers such as Trongaard at Mansfield to help provide a general estimate of the cost effectiveness of academic programs, researchers say.Districts also have plenty of data—federal and state test scores; student grading, behavior, and attendance records; and more recently, school expenditure reports and digital educational platforms that track academic growth—to help make plausible AROI estimates.
Roza is bullish on AROI based on her experience in training hundreds of school leaders. Her lab provides them with free software tools to analyze AROI data. When they realize that their spending often isn’t correlated with academic achievement—some low-performing schools get a disproportionate share of money while some higher performers get much less—they tend to say, “‘Wow, we need to do something about this now,’” she said.
Now at the alliance, which supports more than 100 districts implementing AROI, MacLeod sees growing interest in the practice.
“I am receiving more requests to speak at conferences about AROI, and more and more people come to listen and avail themselves of the information we make available,” she said.