The Skeptical Business Analyst – 9 Tools to Build a Baloney Detection Kit

Skeptical is a highly-charged word. Are you skeptical? In business being skeptical is akin to being a nay-sayer, difficult to work with, not being a team player or a trouble maker. Skeptical does not need to be destructive or negative. You do not have to be a jerk to be skeptical. Skeptical can mean looking at things with a critical eye respectfully and politely to create thoughtful and meaningful discussions to elicit business requirements and build business solutions.

Carl Sagan (November 9, 1934–December 20, 1996) was an American astronomer, cosmologist, and astrophysicist who was known as a master of the balance between skepticism and openness.   In Carl’s last book, he wrote a chapter titled “The Fine Art of Baloney Detection” where he outlines the “Baloney Detection Kit.”  With the introduction of new ideas, Carl Sagan used the kit to evaluate them. If the new idea survives examination by the tools in the kit, the idea was given a warm reception and sometimes a provisional acceptance of the idea. If the new idea was challenged by the kit and didn’t fare well, the idea was refined and modified.

The Baloney Detection Kit is a collection of tools that can apply elegantly to business solutions and requirements elicitation in the Business Analyst profession. Although these tools were designed for the larger realm of cosmic exploration, physics, and science, they are equally at home with a Business Analysts thinking. Let’s focus on how these tools are used for elicitation and design as a part of the Business Analyst role. 

1.  Wherever Possible There Must Be Independent Confirmation of The Facts

Several hard lessons were learned in dealing with Commercial Off-the-Shelf (COTS) vendors and when dealing with reporting dashboards for business intelligence projects. The mistake was accepting the facts at face value and not verifying those facts.

For the COTS project, it meant accepting the vendor's statement that the solution would perform the desired function without verifying the desired function worked. Building a sandbox to use as a playground to verify the desired function worked in the COTS application is very beneficial. 

For the Business Intelligence project, it was confirmation of the meaning of the data, and it’s overall quality and accuracy. A simple data discussion has turned into a spirited debate about what a data field means and the data it contains. Learning to avoid useless reports by carefully evaluating the quality and accuracy of the data has been my greatest lesson. Just because data is captured doesn’t mean it is captured accurately or that it has quality. I have had many cases where all stakeholders trusted the quality of the data without question, but after thoughtful and careful review it was determined the data was of low quality. 

2. Encourage Substantive Debate by Knowledgeable Proponents of All Points of View

A forest is seen in many ways. From above you see the canopy of green leaves.  From the ground, you see the tree trunks and branches.  Looking from below ground, you see the roots of the trees.  Different perspectives give you a deeper understanding of the overall situation that leads to more efficient and meaningful solutions.

When eliciting the requirements or validation of the business solution, a Business Analyst needs to see the whole picture. By looking more globally, you can determine how the solution will fit into the organization.  An example of this would be only looking at the needs for a Customer Relationship Management tool from the perspective of customer service and ignoring marketing and sales teams in eliciting requirements.  Marketing and Sales have specific requirements when interacting with customers just as the Customer Service Team.  Even though the scope of the project is entirely within Customer Service, care should be taken to understanding Marketing and Sales teams that could in the future utilize or connect with the system.

Systems created in a silo was a pitfall for many of my clients by not looking at the larger organization and how it would use a CRM solution.  So, each department that interacted with a customer – 9 different teams in all – created their separate systems which created a costly solution to maintain 9 CRM solutions and 60+ interfaces to pass data between them.  The solution was to combine all 9 CRM solutions into a single CRM solution to save costs.  It took two years to complete the consolidation. It was very costly for the organization not to look at a more global approach to creating a CRM application.

Another example of this would be from another client that had three different locations for product testing laboratories. The labs mostly did not talk with each other because they were in different parts of the world. The project was to upgrade a Laboratory Information Management System (LIMS) in one of the locations because the vendor no longer supported the version the lab was working on and it did not connect to all their new lab equipment. The lab manager assured me that talking with the other labs would be a waste of time. The team saw an article on the web about one of the other labs upgrading to a new system. We reached out to them to get a better understanding of the cost and project effort that would be required. Surprisingly we discovered they had a tool that would meet all our needs, an enterprise license for the software that would save our project money and desired a better connection to our lab. The overall result was the reduction of the project budget and project duration cut in half.

3. Arguments from Experts Carry Little Weight

This tool is the most charged of the tools in the kit. The statement here is that no one is all knowing or God-like in his or her knowledge of a system or process. Experts have made assumptions based on their experience, but handle assumptions of all kinds with great care. Systems and processes are ever evolving and changing. This situation seems to happen on projects where a business team member is considered the all-knowing expert for a process or system that doesn’t like to be challenged. These individuals begin to “spin” the facts to avoid looking like they made a mistake and their desire to maintain the status of all-knowing.  They do not want to lose face.  It becomes tough to understand the overall current state but probing gently and thoughtfully in a non-threatening way can lead to the discovery of the current state issues and root causes.  Care if taken to avoid the expert feeling that you do not trust them.  Seek to understand first and gain the agreement of the current or future states.

No one is infallible or all-knowing. Care should be taken to verify what an expert gives you facts or requirements. It is not disrespectful to challenge an expert. If done thoughtfully and gently it can build a strong business relationship.

4. Spin More Than One Hypothesis

In Business Analysis, we create solutions to problems, and we help build designs to solve those business problems. In a way, a business design or a business solution is a hypothesis.  It is a potential way to solve a business problem. In Business Analysis, we typically don’t create multiple business solutions and then compare them against each other, but one of the strengths of a COTS project is that you are comparing multiple vendor solutions against each other to ensure you get the best possible. In several recent projects, I have taken to building multiple potential internal solutions and then comparing them against each other in a similar fashion to comparing COTS solutions to each other.

The outcome of this was that I could see a broader range of solutions.  Additionally, my stakeholders and sponsors felt they had an opportunity to pick a solution, feel invested in that solution and not feel Technology was forcing them to accept only one solution.  The other interesting outcome of this approach is stakeholders and sponsors pulling ideas from one solution into another solution to create a whole new approach.

If there’s something to be explained, think of all the different explanations. Then think of tests by which you might systematically disprove each of the alternatives. What survives has a much better chance of being an effective solution than if you had simply run with the first idea.

5. Try Not to Get Attached to a Hypothesis Just Because It is Yours

A better way to state this would don't get hooked on a solution you came up with and ignore other potential solutions.  Be open to other potential business solutions and be open to modifications or changes to the business solution you have put forward.  Building the best business solution needs flexibility and collaboration.

Ask yourself why you like the idea. Compare it fairly with the alternatives. See if you can find reasons for rejecting it and modify the idea to strengthen it.

6. Quantify

If whatever it is you are explaining has some measure, some numerical quantity attached to it, you will be much better able to discriminate among competing hypotheses or solutions. Quantification of requirements and business solutions are important. Ensure the requirement or solution can be quantified by analysis. Every solution should have an objective measurement that can be used to determine its value when compared to other solutions.

Anything that is vague and open to many different explanations will result in requirements and business solutions being confusing, frustrating, and easily misinterpretation.  At the beginning of a project either when we are creating the business case or just been assigned to a project there is a lot of uncertainty and assumptions.  As you move through the project, it is important to track all the assumptions made and verify them as you move forward. I have had some assumptions that were never resolved until the project was almost completed. Challenge assumptions made and resolve them continuously throughout. A bad assumption can lead the project down a path to a bad or ineffective business solution.

7. Every Chain Link Must Work

If there’s a chain of argument or solution, every link in the chain must work (including the premise) — not just most of them. Business solutions are systems, hardware, software, processes, data, interfaces, integrations and business rules that are chained together. This is where traceability shines brightly. Ensuring that scope aligns with requirements and requirements to test cases is the basic traceability.  Advanced Traceability is making sure every part of the business solution is in alignment and supporting each other and that nothing was missed from the overall solution design.

Business process neatly falls into this discussion. Each task in a business process is connected by the task ahead of it or after it. To verify a business process each task's input, processing, and output are verified. The SIPOC can be a good technique for verification.

8. Occam’s Razor

When faced with two hypotheses or solutions that solve the business problem equally well, it is wiser to choose the simpler or less complicated solution.  Simplicity has great power in that it can be understood more easily.  The greatest challenge of any business analyst is taking complex business problems and solving them with simple business solutions.

9. Always Ask Whether the Hypothesis Can Be, At Least In Principle, Falsified.

This tool is the second most charged tool in the kit because of the word falsified which has a strong negative reaction.  The hypothesis in this context can be related to a business solution. In the Business Analysis context, this is the verification or validation of the business solution.

Let's take the word falsified in a new direction and away from a negative space and any ethical discussions.  To apply this tool to Business Analysis the proposed business solution should be testable and verified. Building a sandbox or prototype helps test a business solution without having to completely build it. Conference room pilots and simulations are other ways of testing to ensure the business solution will solve the business problem. In Business Analysis terms this is the verification of the proposed business solution. Testing is one part of the equation. Verification and validation which can be managed in requirements traceability are equally important. Resolving assumptions and verification of those assumption used in the business solution should be addressed.

Review the solution and open it up to critical feedback from stakeholders. This opens up collaboration with stakeholders and sponsors on the business solution. Allow your assumptions, experiments, prototypes to be interacted with by stakeholders and sponsors to see if they get the same results you did when interacting with the business solution.

Summary

There are a lot of good things that can come from being a positive skeptical Business Analyst.  Are you a skeptical Business Analyst?  Do you think you can be skeptical without being seen as mean spirited?  Let me know your thoughts in the comments below.

References

In The Demon-Haunted World: Science as a Candle in the Dark by Carl Sagan published February 27, 1997.  Available on Amazon.com.  ISBN-10: 0345409469

Carl Sagan Autobiography - https://en.wikipedia.org/wiki/Carl_Sagan

The Baloney Detection Kit: Carl Sagan’s Rules for BS-Busting and Critical Thinking - https://www.brainpickings.org/2014/01/03/baloney-detection-kit-carl-sagan/