EBM: What It Is and Why Leaders Struggle Using It

By Brad Winn

“Executives sometimes make decisions based on a disciplined review of the available evidence. Unfortunately often they do not—too often opinions, knee-jerk-reactions, and ‘doing what everyone else is doing’ drive decision making.”ⁱ Evidence-based management (EBM) has been discussed for decades as a superior way for managers and leaders to make decisions. Yet, after all these years, executives still struggle to implement EBM and utilize the best available evidence in their roles as decision-makers-in-chief. In this installment of “Linking Theory + Practice” we’ll look at the evidence for evidence-based management. In doing so, we’ll review the what, why, and how of EBM. First, we’ll discuss what EBM is and what it is not. Next, we’ll discuss why leaders struggle to implement EBM in the real world of practice. Finally, we’ll propose some practical steps for closing the divide between EBM theory and practice.

This is a synthesis review of several key publications on evidence-based management. Special appreciation is given to Dr. Denise Rousseau of Carnegie Mellon University for providing insights during a personal interview and to the following researchers whose published articles contributed to this review and the broader discussion on evidence-based management: Eric Barends (Center for Evidence-Based Management), Denise Rousseau (Carnegie Mellon University), Rob Briner (University of London), David Denyer (Cranfield University), Sara Rynes and Amy Colbert (University of Iowa), Ernest O’Boyle (Indiana University), Trish Reay (Alberta School of Business), Whitney Berta and Melanie Kazman Kohn (University of Toronto), and Jeffrey Pfeffer and Robert Sutton (Stanford University).

EBM: What It Is (and Is Not)

As stewards of the strategic direction of the firm, leaders are responsible for making key decisions for long-term success, sustainability, and growth of their companies. But what do leaders rely upon to help them make these key decisions? According to Barends, Rousseau, and Brinerⁱⁱ, leaders should base their decisions on sound evidence. To support their claim, they offer the following:

“Consider this hypothetical situation. You pay a visit to a dietitian after gaining a bit of weight over the holiday season. The dietitian advises you to try diet X. It’s very expensive and demands a radical change in lifestyle, but the prospect of having a slim and healthy body motivates you to stick to the diet. After a few weeks, however, you have gained five pounds and suffer serious side effects that require medical treatment. After searching the Internet, you learn that most scientific studies find diet X to be ineffective and fraught with such side effects. When you confront the diet consultant with these findings, he replies, ‘Why should I pay attention to scientific studies? I have 20 years of experience. Besides, the diet was developed by a famous American nutritionist, whose book sold more than a million copies.ⁱⁱⁱ

“Does that sound like malpractice? It probably does. Unfortunately, in management, disregarding sound evidence and relying on personal experience or the popular ideas of management gurus is daily practice. Yet managerial decisions affect the working lives and well-being of people around the world. As Henry Mintzberg^iv said, “No job is more vital to our society than that of a manager. It is the manager who determines whether they serve us well or whether they squander our talents and resources.”

So what exactly is evidence-based management? A helpful place to begin is with the notion of evidence itself. The Oxford Dictionary defines evidence as the available body of facts or information indicating whether a belief or proposition is true or valid. This assumes there is an available body of knowledge that can inform practice, such as medical practice, educational practice and, of course, management practice.

Since evidence-based practice began in medicine and then education, let’s begin by looking at how these ideas are described there. In the medical field, evidence-based medicine includes the “conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients. It integrates clinical experience with patient values with the best available research information. It is a movement which aims to increase the use of high-quality clinical research in clinical decision-making.”^v In the educational arena, evidence-based education is known as the “what works” agenda. This agenda is meant to “narrow the gap between research and practice and be an area in which research can make itself directly useful to practice.”^vi In evidence-based education, relevant research is utilized by educators and policy-makers with the purpose to improve teaching and learning practices. In the practice of management, EBM involves (1) a mindset, (2) multi-source mining, and (3) methods for decision-making that values taking time to consult the best available information before coming to a conclusion.

1. EBM is a Mindset. EBM is a mindset based on a set of attitudes that balance the need for deciding quickly (an important priority for any manager who works in the increasingly fast-paced real world) with the need for deciding carefully (an important priority for any manager who has ever wasted time and resources trying to implement a bad decision). EBM is a mindset that tends to question ideas that are assumed to be “best practice.” It’s a way of thinking about how to make decisions by asking the right questions before jumping to conclusions based on past assumptions.

In a time of greatly expanded dissemination of information, wise decision-makers value a “mindful mindset” that is conscientious and discerning of the myriad sources of data that compete for their attention. It’s what judicious leaders, managers and practitioners do in their daily decision-making.

Briner, Denyer and Rousseau^vii clarified the concept of evidence-based management and also discussed some common misconceptions. Their explanations are summarized in Figure 1:

Figure 1: The EBM Mindset

Evidence-Based Management Is...	Evidence-Based Management Is Not...
Something managers and practitioners do	Something management scholars do
Something practitioners already do to some extent	A brand-new way of making decisions
About the practice of management	About conducting particular types of academic research
A family of related approaches to decision making	A single decision-making method
A way of thinking about how to make decisions	A rigid, one-size-fits-all decision-making formula
About using different types of information	About privileging evidence from academic research
About using a wide range of different kinds of research evidence depending on the problem	About using only certain types of research evidence irrespective of the problem
Practitioners using research evidence as just one of several sources of information	Scholars or research evidence telling practitioners what they should do
A means of getting existing management research out to practitioners	About conducting research only about management practices
Likely to help both the process and outcome of practitioner decision making	The solution to all management problems
About questioning ideas such as “best practice”	About identifying and promoting “best practice”

2. EBM involves Multi-source Mining. One of the common myths of EBM is that it relies solely on mining published scientific research articles to uncover valid evidence. Nothing could be further from the truth. Research evidence is just one source of information among many sources. In management settings, evidence typically comes primarily from four sources: practitioner expertise, local experimentation, research evidence, and stakeholder information. In summary, Briner, Denyer and Rousseau^vii offer the following:

“Evidence-based management is about making decisions through the conscientious, explicit, and judicious use of four sources of information: practitioner expertise and judgement, evidence from the local context, a critical evaluation of the best available research evidence, and the perspectives of those people who might be affected by the decision.”

Decision-makers utilize different types of information and evidence depending on the problem as shown in Figure 2.

Figure 2 - EBM: Multiple-Source Mining^vii

3. EBM involves a Method. Evidence-based management can be enacted through a methodology or process. The method involves a set of steps that help to ensure the best available information is consulted before a decision is made. According to Briner, Denyer, and Rousseau^vii “Although no single or agreed-upon process exists, we suggest that the following steps might constitute one approach...

1. The start of the process is the practitioner’s or manager’s problem, question, or issue. The accuracy of the problem statement would be discussed and challenged so that it could be articulated as clearly and explicitly as possible.
2. Next, internal organizational evidence or data about the problem or issue would be gathered and examined to check its relevance and validity. At this stage the problem may be restated or reformulated or made more specific.
3. External evidence from published research about the problem would also be identified and critically appraised in the form of something like a systematic review or a rapid evidence assessment (a quicker and more tightly constrained version of a systematic review, which similarly adopts an explicit and systematic method).
4. The views of stakeholders and those likely to be affected by the decision would be considered, along with ethical implications of the decision.
5. When all these sources of information had been gathered and critically appraised, a decision would be made that attempts to take account of and integrate these four sources of information.”

EBM: Why Leaders Struggle Using It

In 1948 when the world was rebuilding after the ravages of WWII, President James Conant of Harvard University suggested that industry leaders were not adequately using the best available evidence in making strategic decisions toward economic recovery. In his HBR article entitled, ‘The Place of Research in Our National Life,’ he pointed out the gap between research evidence and practitioner decision-making. Raey, Berta, and Kohn chronicle the next 60 years’ worth of calls for closing the research-practice divide through EBM. They noted another prominent voice, Peter Drucker, who in 1955 advocated a more scientific approach to management decision-making and stated that management science should, “arm the manager’s imagination and supply him with the vision needed to make rational decisions.”

Yet throughout their review of another 160 plus articles written over the next half century, Rey et al. did not find evidence for general utilization of EBM among executives and middle managers who were the primary target audiences of these calls to utilize evidence. They stated, “…the existence of evidence is critical, but not sufficient to inspire and guide action…. As a result, innovative and creative methods must be developed to bring evidence to practicing clinicians, managers, or other decision makers in order to overcome barriers to change.”

Yet, this doesn’t tell the whole story as to why leaders struggle to utilize EBM. The lead article in the latest issue of the Journal of Management is entitled, “When the Best Available Evidence Doesn’t Win: How Doubts about Science and Scientists Threaten the Future of Evidence-Based Management.” The authors, Rynes et. al, state that, “the assumption has been that if researchers produce stronger, more replicable research findings and disseminate them in practitioner-accessible venues, the results will be believed and implanted if appropriate to the context. However, an abundance of prior research indicates that some of the strongest management research findings have not been widely adopted by mangers…because (1) people are unaware of research findings, (2) even if they are aware of findings, they don’t believe them, or (3) even if they are aware of the findings, and believe them, they don’t (implement them because) they may threaten people’s cherished beliefs, self-image, self-interest, or social identity.”^ix

In an era of exponentially growing fake news and hyper-information availability, where do leaders turn for trusted evidence? Apparently, they are no longer turning to experts (doctors, lawyers, scientists, etc.) as readily as they once did. In The Death of Expertise, Nichols writes about the notion people have that “my ignorance is just as good as your knowledge.” This general rejection of evidence or facts is, “a Google-fueled, Wikipedia-based, blog-sodden collapse of any division between professionals and laypeople, students and teachers, knowers and wonderers…not just a rejection of existing knowledge, [but rather] fundamentally a rejection of science and dispassionate rationality which are the foundations of modern civilization.”^x

Rynes et al. cited the following examples of evidence that is often rejected even when supported by a vast majority of research studies:

1. Despite strong scientific consensus that human activity is a driver of rising global temperatures, 39% of Americans do not believe.
2. 98% of scientists believe that natural selection plays a role in evolution, 34% of Americans entirely reject this.
3. HR professionals generally do not believe the research findings that:
   • Goal setting is more effective than employee participation for improving organizational performance
   • Most errors in performance appraisal cannot be eliminated by error-reduction training
   • Intelligence is a better predictor of performance than either values or conscientiousness
   • Intelligence improves performance even on low-skilled jobs
   • Tests and actuarial selection models are superior to unstructured interviews
4. Despite evidence to the contrary, investors believe that:
   • There is a negative relationship between corporate social responsibility and corporate financial performance
   • Firms undergoing IPOs so not have a significantly larger chance of 5-year survival if they place a higher value on human resources

In general, the theory-practice divide is still gaping with regard to practitioners utilizing formal research evidence to guide decision-making. Although decades of management science suggest an EBM approach to decision-making, leaders continue to struggle with accessing and implementing external research evidence. That said, external published research is only one source of evidence. Practitioners are most likely doing a better job at utilizing the other types of evidence to guide decision-making, including practitioner expertise, evidence from the local context, and the perspectives of stakeholders.

EBM: How Leaders Move Forward

Evidence-based management is an indispensable tool for the long-term success of leaders and organizations. It is a relentless focus on what works and why. It requires managers to take the time to test their gut instincts by accessing and assessing the best available evidence, whether that evidence comes from external research studies, internal investigation and experimentation, stakeholder interests, or professional judgement.

The best leaders are good at trusting their intuition, but only after they’ve done due diligence. It’s one thing to be an activist for a new direction and quite another to be a credible activist.^xi Executives set the example by simply and continuously asking the questions, “Why & What?” That is, “Why do we believe our strategic decisions will work and what is the evidence for that belief?” Continually asking these questions throughout the company will embed credibility into the culture of “how we make decisions around here.” To this end, we propose a few practical tips that will help leaders move forward and close the divide between EBM theory and practice.

• Begin with the mindset that EBM is the “way we do business” and is a core competency embedding credibility into strategic decision-making. Exemplify using evidence to confirm your gut instincts.
• Don’t short change the way you define your strategic questions. Back up, reframe, and ask questions about the nature of your problem. Look at your problems and opportunities from different perspectives and engage divergent points of view.
• Dispel the myth that sound evidence is solely created externally to your organization in published scientific research. That said, you might consider working with local university faculty members. They are likely looking for opportunities to partner with real-world organizations to conduct relevant research.
• Implement internal inquiry. Emphasize that the most relevant data may be developed fairly quickly within HR through short surveys or focus groups (employees, customers, stakeholders, benchmarking, etc.) Remember, EBM is a multi-source mining practice.
• Challenge the status quo. Foster an atmosphere for candid discussions and allow for failure and risk-taking.
• As a matter of course, conduct after-action reviews of your own and your organizational decisions. Make this standard procedure to review what worked and what didn’t. Then ask why. Keep a list of decisions gone well and decisions gone bad.

Insights for Executives and HR Leaders

Executives are discovering for themselves (sometimes the hard way) that it pays to check their instincts. We are leading in an era of unprecedented easy access to information. That’s the good news and the bad news. It’s good news because it doesn’t take long to access evidence on almost any topic. It’s bad news because we live in a time of questionable sources of information, over-reliance upon opinion, and blatantly fake news. In this fast-paced era of instant access we are also expected to engage in instant wise decision-making. This may be an unrealistic expectation.

Employees throughout our organizations and customers throughout the world are hungry for leaders who make brilliant decisions that solve real problems and create new opportunities. We need to exemplify the discipline of being credible activists through evidence-based management. We need to embed this in our organizational culture and teach it to our managers by continually asking the simple question, “What evidence do you have that this works?” Executives can better utilize evidence-based management as a vital tool to become even more effective in their roles as able decision-makers-in-chief.

ⁱ Sokol, M. (2018). Call for Papers. People + Strategy 42.1.
ⁱⁱ BRB: Barends, E., Rousseau, D.M., & Briner, R.B. (2014). Evidence-Based Management: The Basic Principles. Amsterdam: Center for Evidence-Based Management.
ⁱⁱⁱ Adapted by BRB from Pfeffer, J. & Sutton, R., Trust the Evidence, Not Your Instincts: New York Times, Sep 3, 2011
^iv Mintzberg, H. The manager’s job: folklore and fact. Harvard Business Review, 1990, Vol 53 (4).
^v Sacket, et al. (2000). Evidence-Based Medicine: How to Practice and Teach EBM. London: BMJ Publishing
^vi Kvernbekk, T. (2017). Evidence-Based Educational Practice. Oxford Research Encyclopedias.
^vii Briner, R., Denyer, D. & Rousseau, D. (2009). Evidence-based management: Concept cleanup time? Academy of Management Perspectives, 23 (4).
^viii Reay, T., Berta, W. & Kohn, M (2009). What’s the evidence on evidence-based management. Academy of Management Perspectives, 23 (4).
^ix Rynes, S., Colbert, A. & O’Boyle, E. (2018). When the ‘best available evidence’ doesn’t win: How doubts about science and scientists threaten the future of evidence-based management. The Journal of Management 44 (8).
^x Nichols T. (2017). The death of expertise: The campaign against established knowledge and why it matters. New York: Oxford University Press.
^xi Winn, B. and Lake, D. (2013). Common & Customized Competencies. In Ulrich D. and Brockbank W., Global HR Competencies. New York, NY: McGraw Hill.