Articles, conferences and books have been devoted to the concept and strategies behind evaluating the success of programs, but there’s surprisingly little information available about how to fit data tracking and analysis software into such efforts. Until now...
Because there’s no single software specific to evaluating programs, different types of systems can help with the different parts of the evaluation process.But what parts? And what software? To find out, we talked to a number of nonprofit program evaluation experts to explore the types of systems that can aid with program evaluation, and how they might fit in to the overall process. What we learned is that it requires more than just a software program to evaluate programs effectively, but the right software can help.
Defining Your Evaluation Process
Program evaluation is not primarily a technology issue. Software can help you track and analyze the data you’ll need, but in order to make any meaningful conclusions, you need to first carefully identify certain things. What are you trying to accomplish? How can it be effectively measured? Which data points will you actually use, and what will be most helpful in truly understanding what’s working? No matter how powerful it is, software can’t help with evaluation until you’ve defined the process that will be most effective for your organization.
This article focuses on how software can help you track, interrelate and make sense of the data you’ve collected once you’re ready to work with it, and assumes you’ve already defined what you want to measure, why it’s important, and how you’ll collect it.
If you need help conceptualizing evaluation, relating it to your overall mission and day-to-day work, and defining the most useful indicators, there’s a tremendous amount of information available elsewhere. As a starting point, consider Theoryofchange.org or the Innovation Network. The Foundation Center also has a very helpful library of possible conceptual tools and indicators in their TRASI database.
Tracking Constituent Data and Outcomes
Many nonprofits—especially those focusing on human services—track much of their data at a client level and then roll it up across multiple clients to help them better understand the broader impacts of their programs.
For instance, a nonprofit that seeks to obtain permanent housing for homeless clients is likely to track the individual demographics and outcomes for each client—like whether they’re still in permanent housing a year after they left the program. In order to track overall effectiveness, this nonprofit might then look at the average number of clients still in permanent housing after a year and seek to boost that average as a method of program improvement.
If the data you want to track is mostly at the client level, a client- or case-management system is one of the best ways to manage it. Such systems provide an interface to allow case workers or field staff to enter information about their interactions with each client, and methods to summarize that individual information to provide some organizational performance metrics. A single system that allows your front-line staff to enter and see day-to-day data, and that also lets you report on the metrics you’d like to use for program evaluation, makes tracking data for evaluation a core part of your daily work as opposed to an extra task to be done in addition to other routines.
Of course, before the system can help, you must first define which metrics are useful to track and identify the data that must be collected to calculate them. Nearly every client management system has some ability to summarize data into performance metrics, but their ability to do exactly what you want will depend on how specialized your needs are, and how flexible the system is.
There are a number of case management systems specific to particular subsets of human services—for instance, homelessness work, healthcare or educational programs. Such packages can provide more tailored processes right out of the box, but may be difficult to further customize to match more specific needs. (For more information on case management systems, read our article, “A Few Good Case Management Tools.”)
On the other hand, you could look to a more flexible Constituent Relationship Management platform like Salesforce or Microsoft Dynamics CRM. Because these packages aren’t specifically built to support human service organizations, they’ll likely provide less support for your processes out of the box. But if you’re knowledgeable about the systems, they’re flexible enough to allow you to build screens, workflows and customized reports to match the way you work, and how you’d like to measure your programs. They also tend to allow you to easily get data in and out of the system, an important consideration for those working with multiple databases. (For more information on CRMs, see our article “In Search of CRM Part 2: Searching for Software.”)
Tracking Non-Constituent Data
But what if you’d like to track data that isn’t specifically related to a particular constituent? Perhaps you’d like to track water quality over time, the results of a public perception survey, or the outcomes of 30 different programs conducted by 30 different organizations, for example.
As a first step, check to see if you’re using any other systems that track information at the same level. Say you’re tracking air quality and pollution level over time—it would be logical to store your water quality data in the same way. If you’re storing information about the 30 different programs—maybe as grantees of your foundation in a grants management system, for example—you can likely also store and report on the outcomes of those programs in that same system.
If you’re not tracking anything similar, decide how complex your data needs are. Can you easily track everything you need about each point of data in a spreadsheet row? If so, don’t shy away from a simple Microsoft Excel or online Google Docs spreadsheet. There’s no need to overcomplicate data tracking if a spreadsheet will work.
If you’re tracking a number of interrelated things, you may need to use a spreadsheet or database platform to build your own system. Consider Microsoft Access or FileMaker Pro, which give you a base set of tools on which to build systems to enter and report on data. For more sophisticated systems, consider building from scratch on a platform like SQL Server or Oracle. It’s possible to build quite sophisticated systems with any of these tools, but before you do, consider it carefully. While it can be useful to build your own internal capacity to understand and manage data, you’ll also need to train your staff on how to use it and fix any issues and make any needed updates yourself. By building a custom system, you’re committing yourself to software development work for the long haul.
What if you need to track and interrelate both constituent and non-constituent metrics? CRMs might work well in this circumstance, as both allow you to set up records for anything you like and then relate them to people or other records in the database. This approach is likely to be less complex than building a similar system out of Access or FileMaker Pro. You’re also more likely to easily find qualified people to configure and support it.
Analyzing and Displaying Your Measures
It’s one thing to track all this data, but another to be able to make sense of it or display it in meaningful ways to your staff and funders. The easier you can make it for staff to see the metrics for which you’re tracking, the more likely they are to get on board with data tracking to improve the services you provide.
The best scenario would be to create easy-to-read reports with the same system you’re using to track data—this keeps the process of entering data directly connected (both physically and mentally) to the resulting metrics. It’s important, though, to make sure the reports are not just easy to find, but easy to understand. Providing a “dashboard” of numbers that are straightforward to read can be a good way to help all your staff focus only on the most important figures.
Unfortunately, it’s not always possible to produce the reports you want out of the system you’re using. If that’s the case, consider connecting your data to an external report builder such as Crystal Reports or Jasper Reports, or even a powerful business intelligence tool like IBM’s Cognos or Business Objects. You may need someone with substantial database expertise to set up and maintain the reports built with any of these systems, but once they’re set up, you can create reports that are relatively easy to read.
There are also systems geared toward helping you make and visualize meaning in your data. Tools like ManyEyes or Tableau can help you create interactive charts or visualizations, and dashboard tools like ClickView and GoodData allow you to import data from your database and create visual summaries of multiple metrics with relative ease.
If you’re looking to provide statistical evidence of efficacy, then your data is just a starting point—you’ll also need statistical analysis tools like SPSS, R or SAS for quantitative data, or NVivo, STATA or Cognitive Edge SenseMaker for such qualitative data as interviews or free text.
Larger organizations that want to compile metrics from various sources might also consider building a data “warehouse.” Typically just a large database, data warehouses pull information from multiple sources to a central place for reporting or analysis. For more on data warehouses, see our article “Data Warehousing for Nonprofits.”
There’s no substitute for a careful evaluation strategy to help you define what you should track, and why—and there’s no software silver bullet that will take care of all your program evaluation needs.
Choosing the right software for your situation can help you collect data, whether it’s a case management system, a Constituent Relationship Management system or something custom-built to your organization. But your work doesn’t end there. You also need to give some thought to how you’ll analyze and report on that data to provide all your stakeholders the information they need to keep improving your services. Software can help with this stage of the process, too, but it still won’t do the work for you.
Software is just part of the equation when it comes to successful program evaluation. When integrated thoughtfully into a careful strategy, the right tools can help you learn more about the efficacy of your programs and help you act on what you learn.
This article was funded by the Annie E. Casey Foundation. We thank them for their support, but acknowledge that the findings and conclusions are Idealware’s and do not necessarily reflect the opinions of the foundation. Thanks as well as to the following nonprofit technology professionals who provided recommendations, advice and other help: