Program Evaluation, Social Impact

Impact = factual – counterfactual

August 17, 2020

Is our program REALLY making a difference?

This question is deceptively simple. “Yes, of course…” and we share stories or output measurements like, “200 people attended the course,” or “92% completed the program.”

Unfortunately, the simplicity of the concept belies its complexity in practice. Estimating the real effects of interventions is challenging work. The good news is there are accessible methods and software that can greatly improve how we evaluate our programs.

In this series of posts, we will introduce the concept of counterfactuals and how our organizations can take steps toward improving our measurement of outcomes and impact. Before we get into counterfactuals and the potential outcome framework, let’s use a logic model to clarify terms.

Logic models help define the relationship between our planned work and intended results.

Inputs: Resources needed for program activities.

Activities: Services or components of a program.

Outputs: Direct result of activities.

Outcomes: Intended benefits of activities.

It’s important to distinguish outputs and outcomes because outputs can easily inflate our perceived impact.

During my freshman year of college, I started a weekly meeting in a local prison with men approaching reentry. It was a form of kind-hearted social malpractice that I thought was making a real impact. Why? The room was packed every week! And the feedback of inmates was always positive. The difference between outputs (attendance) and outcomes (benefits) could be revealed with the follow-up question to the outputs, “so that…” “Well, they come to the meetings, so that they can develop a reentry plan, identify housing and job prospects, and strengthen positive relationships.” These were some of the (unmeasured) program outcomes. How many inmates obtain stable housing and employment? Such an outcome measurement could be distinguished by short and long-term estimates.

But do outcomes equate to impact? 


No, outcomes alone cannot fully answer the question, “Is your program making a difference?” We could show some convincing pre and post-test data that is statistically significant. However, what outcomes alone cannot tell us is what would have happened if the person did not participate in the program. This is known as the counterfactual.

The impact of a program is simply “what happened” minus “what would’ve happened without it.”

Impact = factual — counterfactual

Since counterfactuals aren’t directly observable, causal inference methods are necessary to try to create the context where we can come as close as possible to observe the unobservable.

Think for a minute about a pre-post test where we calculate a simple difference. We’re running a rapid rehousing program for homeless Veterans and one outcome metric is income. Say Bill comes into our program with monthly income of $800 and when he finishes the program his income is $1000. Can the simple difference of $200 be attributed to the program? In other words, can we equate the simple difference with a treatment effect (impact) of the program?




Treatment effects (impact) are changes in outcomes due to changes in treatment (activities) holding all other variables constant.

This last phrase is really important. In order to answer the question, “Is the program REALLY making a difference?” we need to isolate the effect of the program, which means that a simple difference between pre and post-test scores cannot be the treatment effect. Post-test data could show significant positive change during the time of the program, but how do we know the program is responsible for the change? We need a way to isolate the effects of the program to obtain the real treatment effects.



The gold standard for such an evaluation is a randomized controlled trial (RCT); however, our focus in this series will be social programs where resources are limited and RCT’s aren’t feasible. How can we identify the effects of our programs with limited resources?

In the next few posts, we’ll introduce and illustrate the primary concepts, terms, and methods for measuring social impact.


Don't miss an article! Get them sent right to your email.

Blog, Featured Posts, Healthy Relationships, Miscellaneous

When to Say ‘Yes’ and When to Say ‘No’

February 17, 2017

A Look at the Tension Between Selfless Love and Healthy Boundaries.

What would you say to the young couple, recently married, with a 6-month old baby who wants to foster a sibling group of five children, ranging in ages between 8 and 17? How do you hold well the tension between selfless love and healthy boundaries?

Jesus says, ‘Don’t resist the one who is evil. If he takes your tunic, give him your cloak. If someone forces you to go one mile, go two.’ Continue Reading…

Don't miss an article! Get them sent right to your email.

Blog, Featured Posts, Transition Well

One Step from Stupid

January 22, 2016

graphic copy

The goal in transitions is the right place at the right time for the right reason and in the right way. It’s a high bar. There are many unknowns and much could go wrong. As I thought about the many variables, I started to question my own perspective.

Continue Reading…

Don't miss an article! Get them sent right to your email.

Blog, Transition Well

Helping Your Family Navigate A Transition

January 16, 2016

How do you best navigate major life transitions as a family? Our oldest daughter is not a huge fan of change and we were about to uproot her from everyone and everything she knew. Before we broke the news of our upcoming move, I had a terrible image in my mind: My daughter in a counseling office many years from now saying, “Well…it all started when we moved to Pennsylvania. Everything was good before that move.” Fortunately, my worst fears aren’t playing out. Our kids seem resilient and responding well to the move so far. Continue Reading…

Don't miss an article! Get them sent right to your email.

Blog, Featured Posts, Transition Well

A Dream Worth Pursuing (or Before you Kill the Vision)

January 7, 2016

Do you have a dream on the bubble? Do you see the potential but not sure it’s worth the risk? One thing is certain: if your dream has a chance, you’ll need to eradicate disordered fear from your life. Most of our emotional, spiritual, and mental struggles are rooted in disordered fear.

There’s a children’s book called, “What Do You Do With An Idea?” that deals with the delicate nature of beautiful visions. A young boy is confronted by “an idea”… Continue Reading…

Don't miss an article! Get them sent right to your email.