Walking the talk: Are evaluators learning from evaluation successes and failures?

8 December, 2021
Evaluation Matters Magazine:
Learning from Successes and Failures in Evaluation

Nadini Persaud, The University of the West Indies, Cave Hill Campus, Barbados
Ruby Dagher, The University of the West Indies, Cave Hill Campus, Barbados

Abstract

Contemporary society cherishes success and stigmatizes failure. But success in evaluation is a relative concept–a grey area subject to interpretation and depends on the question, “success for whom?” This article stresses the importance of evaluators practising what they preach. The literature suggests that most evaluators learn from their mistakes and mishaps and aempt to avoid repeating them. But for most evaluators, talking about their failures is not easy. It may take some time for the embarrassment to fade, even when an evaluation’s mitigated success–or complete failure–was beyond the evaluator’s control. Nonetheless, openness about failure is important to the discipline and practice of evaluation.

Key Messages 

  • Learning from successes and failures is an integral part of the evaluation discipline, for clients and evaluators alike. Yet evaluators have lile willingness to report a failure or a failed approach publicly.
  • Success and failure can be defined in many ways. The definition depends on the context in which the evaluator operates as well as on other factors, such as the relationship between the evaluator and the client. 
  • The many definitions of success and failure are vital for collating an array of knowledge about evaluations that can be adapted to diverse situations and help build a range of tools on which evaluators can draw. 

Learning: A foundational aspect of evaluation 

Evaluators help clients become more accountable and improve their interventions, policies, and programs. With every evaluation, the evaluator provides their client with an assessment of the client’s interventions, policies, and programs. The evaluation also recommends ways to improve the next set of interventions, policies, and programs. As such, identifying lessons learned is integral to evaluators’ work.

Since learning is such an important pillar of evaluations, it is vital to determine whether the learning process is part of evaluators’ self-assessment as well. Do evaluators use lessons learned from their evaluations to build their knowledge and improve their performance? Are evaluators as open to learning from failure as they are from success? Do evaluators report their failures as well as their successes, or are they biased only to disclose what works? 

The contemporary societal lens: Success is celebrated, failure is stigmatized 

The contemporary societal lens: Success is celebrated, failure is stigmatized The term “failure” has evolved. Formerly used to designate inadequacy in financial business, it has come to be associated with personal shortfalls. Today, society cherishes success and generally perceives failure as something to be avoided (Loscalzo, 2014). Since success and failure cannot coexist, success is celebrated and failure is kept low-key. Specifically, many people experience failure as ego-threatening (Eskreis-Winkler and Fishbach, 2019), even as a personal shortcoming. As such, most people who feel that they have “goofed up”–whether through carelessness, a lack of effort, or a lack of foresight–try to avoid making the same mistake in the future. That said, they are usually too humiliated to talk about their mishap until their pain, disappointment, and embarrassment about it have faded. This is the case whether the incident concerns the evaluator’s perception of an unacceptable personal failure or a client’s perception of an unacceptable professional failure on the part of the evaluator. In either case, the evaluator may fear that the incident will cause them to be stigmatized and stripped of future work possibilities. For this reason, evaluators find it easier and more beneficial to share their successes. For evidence, consider the Fail Forward initiative, which was established in 2010 to help organizations learn from their failures. To date, only 28 failure reports and failure sharing events have been posted on the Fail Forward website, and only 16 reports are accessible on the Admiing Failure website.

success and failure

Of course, the predicament is that success and failure can be interpreted differently, depending on the profession, organization, task, or activity at hand. Although failure in some situations may be devastating for example, if an engineering mistake causes a space shuttle to explode in mid-air–in other situations, a certain amount of failure may be tolerated, even accepted. For example, the harsh critique that greets many scholarly research papers can be discouraging. Even though many academics may want to give up,  they recognize that publications are the means to achieving tenure, so they persevere, mute their egos, transform their discouragement into creative synergy, and do what is necessary to be published again. In general, success is often gauged by some type of measurement. That said, contextualization may be appropriate, 

In general, success is often gauged by some type of measurement. That said, contextualization may be appropriate, even with measurement. For instance, should programs that did not achieve most of their aims in 2020 be considered as failures, even though COVID‑19 restrictions affected their operations? 

Are evaluators learning from successes and failures in their evaluations? 

The literature on whether evaluators learn from successes and failures in their evaluations is limited. Nonetheless, evaluators would be hypocritical if they preached one thing to their clients–i.e., the importance of learning from one’s mistakes–but ignored their own gospel. Moreover, in a profession that is still not well established or even accepted in several parts of the world, evaluators cannot afford to repeat the same mistakes, or not learn from their failures, since this will compromise the growth and integrity of the nascent profession. And since evaluation is still an emerging discipline, we must constantly strive to promote evaluation as something good and vital to decision-making. This means that we have little choice but to build on our successes and learn from our mistakes so that we are taken seriously.  

 

[[up till here]]

A review of the literature reveals two noteworthy compilations on evaluation successes and failures. The first is a special edition compiled for the 25th anniversary of the Canadian Journal of Program Evaluation in 2012. Edited by Michael Quinn Patton, the edition published stories of incomplete successes. The second is a more recent compilation edited by Kylie Hutchinson and titled Evaluation Failures: 22 Tales of Mistakes Made and Lessons Learned, which shared lessons on evaluation pitfalls and ways to avoid them. The main takeaway was that even the most experienced evaluators can make mistakes and errors of judgment in the course of their work. In contrast to the two publications just cited, the theme of the 2015 annual conference of the American

Evaluation Association was “Exemplary Evaluations in a Multicultural World: Learning from Evaluation’s Successes.” The omission of the word “failures” from the theme is unsurprising, given the negative connotation generally associated with failures. Interestingly, the conference held one session on failure (it was titled “Learning from Failures”). According to Hutchinson (2018), the session sought to counterbalance the conference theme by having evaluators share stories of failures. The session was well attended.

In any given evaluation, many things can go wrong. Even the most meticulous planning does not exclude the possibility that problems occur. The evaluator’s experience often determines how problems are managed and resolved. Some problems are easily fixed, while others require negotiating with the client to reach a compromise. And other problems are simply unfixable and quickly spiral out of control. Whether caused by inexperience, a lack of foresight, or events outside of an evaluator’s control, problems during an evaluation can be costly and can carry profound consequences and repercussions.

Still, incomplete success or outright failure can present opportunities for learning, yield good insight (Dahlin, Chuang, and Roulet, 2017), be a great teacher (Loscalzo, 2014), or become a prerequisite for future success (Noonan, n.d.). In fact, most if not all research evolves from considerable trial and error, with scores of failures along the way. Talking about problems, admitting to mistakes, sharing in a public forum, and reflecting carefully on what happened, can help us understand why failure took place and how to avoid errors in the future. Reflecting on failure also helps us build resilience and stamina, and become wiser and more adaptable. Yet creating space for reflection can be a momentous task (Hutchinson, 2018)–this, even though the Canadian Evaluation Society’s Competencies for Canadian Evaluation

Practice considers reflective practice to be a competency domain. And even if not all voluntary organizations for professional evaluations agree with that position, we believe that most evaluators engage in self-reflection to avoid falling into the same pitfalls as they did before. Ultimately, having an evaluation contract terminated, losing money on a job, or sustaining damage to one’s professional reputation affect an evaluator’s livelihood and are not experiences that the evaluator wishes to repeat. Errors of judgment are a fact. As evaluators, we must accept that we are human and become comfortable with the possibility that we make errors of judgment. We must also acknowledge that in our profession, the politics surrounding evaluations are ever-pervasive and we have little choice than to learn to navigate this obstacle.

Defining success and failure in evaluations: A grey area

Concluding that an evaluation mission has succeeded or succeeded in part (has failed to different degrees) is nebulous–a grey area (see Figure 1). Success and failure are relative concepts that are riddled with methodological nuances (e.g., benchmarks, perceptions). Specifically, success for whom and success as fact or interpretation, are questions that require context to answer (McConnell, 2011). Different stakeholders, including evaluators, view these concepts differently. According to Patton (2012), success is determined by the criteria (or the lack thereof) that is used to define it. Evaluators adhere to various evaluation and research standards and follow different guidelines to execute their work. For many, success therefore corresponds to the extent to which those guidelines and standards were used to protect and preserve the integrity and credibility of their evaluation. For others, success stands for the level of methodological rigor used in the evaluation and the credibility of the evaluation’s findings (Owen, 2012). For still others, success is equated with effective communication (Perrin, 2012) or with how findings are used (Lee, 2012).

The literature documents a host of issues that can compromise an evaluation’s success: contextual challenges, inadequacies on the part of the client, people problems, scope creep, inadequate buffers for the evaluation’s timeline and budget, a lack of control by the evaluator as a result of heavy reliance on the client’s approval for most aspects of the evaluation, lack of an evaluation culture, unrealistic expectations on the part of the client, the evaluator’s failure to set clear priorities with the client, an unclear purpose for the evaluation, misaligned expectations, an evaluator working alone, an inadequate paper trail, turbulent organizational environments with constant staff turnover, insufficient stakeholder meetings to garner buy-in, the involvement of diverse stakeholders, inappropriate assumptions, an insufficient understanding of the local context, a methodology and approach that are not sufficiently adapted to the local context, insensitive reporting language, reluctance by the evaluator to communicate concerns early, and contracts that are not detailed meticulously. These issues, combined with power dynamics and the ever-present politics of evaluations, can be a recipe for incomplete success or outright failure.

Notwithstanding the difficulty of defining success and failure in evaluation, evaluators’ willingness to share all of their experience–positive and negative–in an ethical manner is key to learning within the discipline of evaluation. When evaluators share their rawness and full experience, other evaluators can develop lists, benchmarks, and tools and adapt them to different situations. And let us not hide by evoking grey zones. Grey zones exist because every situation is unique. It is evaluators’ responsibility to learn from each other’s experience and apply the lessons from all situations, unique otherwise.

Figure 1 a grey area

Conclusion 

Most evaluators recommend that their clients use the lessons learned from the intervention that has been evaluated to improve interventions in the future. As evaluators, we need to “walk the talk” ourselves. This is important for our own professional growth and development as well as for the growth and development of our profession. Still, when analyzed critically, success, incomplete success, or failure are grey areas that are prone to interpretation. This interpretation depends on the program, on one’s perspective, on criteria, and on other elements. It is therefore important that evaluators be as forthcoming as possible about all aspects of their experiences. Only by doing so will they give other evaluators the context they need to assess the usability and the applicability of the lessons they share.

Author(s)


Nadini

Nadini Persaudis a lecturer in evaluation at the University of the West Indies, Cave Hill Campus, Barbados. She is a Fulbright scholar and holds a bachelor of science in accounting (First Class Honors), a master’s of science in project management and evaluation (with distinction), and a PhD in evaluation from Western Michigan University. Nadini also holds professional designations from Canada, namely, Certified General Accountant (CGA) (Gold Award) and a Chartered Professional Accountant (CPA). She serves as a board member on several regional and international organizations. Nadini has published several papers in reputable journals and is the co-author of Research Methodology Basics: From Conceptualization to Write-Up, the author of Strengthening Program Evaluation Culture in the English Speaking Caribbean: A Guide for Evaluation Practitioners and Decision‑Makers in the Public, Private, and NGO Sectors (in press), and the co-author of The Role of Monitoring and Evaluation in the UN 2030 SDGs Agenda. 

Ruby

Ruby Dagher is an international development professional, a researcher, a consultant, and a replacement assistant professor. Her professional experience in international development and her research over the last decade have given her significant expertise in public policy, state capacity, evaluations, donor-recipient relationships, power dynamics, and economic development. She holds a PhD in public policy and administration from Carleton University, a master’s degree in public policy and administration from Carleton University, and a bachelor’s degree in political science and economics from McGill University. Ruby's research interests lie in foreign aid policy and donor programming, evaluation, post‑conflict development, fragile states, post-conflict legitimacy, decentralization and state legitimacy, and the role of civil society in non-Western cultures, the Middle East, and the Caribbean. She has published several papers in reputable journals. She is the author of Building Locally Recognized State Legitimacy in Post-Conflict Countries and the co-author of The Role of Monitoring and Evaluation in the UN 2030 SDGs Agenda.