I'm Sorry I broke Your Company

Scientific method

Usually the scientific method is written as four steps, but sometimes people break the four steps down into finer detail with six or seven resulting steps. I'm going with the four-step approach I was taught in grade school.

1. Research, analyze, investigate, and define the problem

I don't know if this is symptomatic of the American culture, but we are so action oriented and impatient to get to the solutions that we spend very little time understanding the problem. The first step of the scientific method is to define the problem, and to define the problem, you need to understand it. How you define the problem sets the course of all the following actions. Finding the root causes of problems and fixing those can save countless hours and money compared to constantly addressing the symptoms of problems. This habit of addressing symptoms always results in new symptoms that need fixing. Spending a little time finding out the «why» before moving on to the «how» can save time in the long run.

Several years ago, I was asked to participate in a cross-functional, problem-solving session to brainstorm ideas on how to get more employees involved in community volunteer activities. The session came up with some very creative public relations ideas and actions to take. However, no one spent any time researching why employees weren't getting involved. Instead, according to the consultants advice to phrase problems in a «how to» format to solicit actions, we worked on the wrong problem. The real issue was that employees equated involvement in these activities with being a low-level or unimportant employee. In addition, many employees thought that these activities were meant to provide leadership opportunities for the administrative staff. Our list of brainstorming ideas and actions didn't try to change that perception. All we had to do to solve the problem was to get some senior management involved in these activities.

2. Develop hypotheses based on the research

My experience with human-caused problems is that often you’ll find the cause of the problem in the first step, so this one isn’t often needed. However, when you can’t easily find the solution, developing hypotheses helps expand your thinking. Notice that this step involves multiple hypotheses based on your research, not just one. The problem with our thinking is that we like to narrow down our options too early and adopt an initial solution just because it is the first one that might work. Then, once we’ve adopted it as the solution, we are unlikely to change our minds even with new information. If you formulate multiple hypotheses, you expand your search and have a higher probability of finding the correct option than if you narrow down your solutions early on.

3. Experiment and test the hypotheses

In new product development, the key to success is fast failure, not early perfection. The point is to figure out as quickly as possible all the product features that don’t work and eliminate those from further inquiry. Management theorists need to adopt the habit of eliminating unsound theories. Instead, each model and method adds to the body of knowledge, and businesses try to implement everything. Worse, one clever consultant finds a new way of combining unrelated ideas and then other consultants build on it until it becomes a mishmash of conceptual frameworks. No manner of adjustment is ever going to get a faulty assumption to work right. Scientists know that when a hypothesis doesn’t work, they need to move on to the next one.

4. Discern findings, formulate conclusions, and repeat step 3

Most often, this step is written as just «formulate conclusions.» I have separated out findings from conclusions because I have found that many businesspeople don't understand the difference. The scientific method is an iterative process; it is unlikely that one experiment will provide enough evidence to allow you to draw conclusions. Rather, it should provide information, aka findings, on where to investigate further. This point is becoming more important because the ability to survey large groups of people and perform a variety of data analyses on the information gathered has become infinitely easier in the last decade. However, people frequently misinterpret the data and see conclusions where none exist. The term «highly correlated» is bandied about as if it means two data points have an irrefutable cause-and-effect relationship, but two data points could be highly correlated due to a number of inconclusive reasons.

For example, one stock market predictor is the Super Bowl Index (SBI). The SBI states that the stock market will rise in years in which the NFC (National Football Conference) team wins the Super Bowl, and it has an 80 percent correlation. Statistically, that's a very high correlation, so make sure you buy stocks whenever the NFC wins! However, when you examine the SBI more closely, you realize that there is no relationship between the variables at all. The NFC teams win the Super Bowl more often than the AFC (American Football Conference), and the stock market has more bull years (it goes up over time) than bear years. Any randomly chosen year should see a bull market and an NFC win. The two highly correlated variables have no actual relationship to each other.

Not too long ago, I had a phone conversation with a representative from a large talent management consultancy on It's menu of leadership assessments. The woman assured me that using her firms leadership assessments was highly correlated with business success. When I asked her to explain this, she said that statistical analyses showed that clients who used these assessments performed above the industry average in revenue growth.

Therefore, using these assessments would make a company successful, at least in terms of revenue growth. However, a much more plausible explanation is that successful companies, those with money, are more likely to buy leadership assessments than those without money. Another more plausible hypothesis is that companies that invest in employee development are likely to outperform those that don’t. Making the conclusion that company success was due to the use of leadership assessments is more a leap of faith than an evidence-based result. The only conclusion you can reach from two highly correlated variables is that you need to do more research.