Best time to improve your ad hoc analysis workflow.

Anyone who has worked on a data team has been there – that frantic moment when an unexpected question about your data pops up, and you're left scrambling to provide an answer. If you're a data professional, handling ad hoc requests might be a daily struggle for you. These seemingly innocent questions can often take up a massive chunk of your time, drawing your focus away from more strategic projects.

Let's dive deep into when organizations should ideally address these unplanned queries and when they should prioritize other tasks.

The Ad Hoc Query Cycle

At first glance, ad hoc queries appear as simple questions. But they soon turn into deep dives where you're required to understand the underlying purpose (the “why”) of that query, test various hypotheses in SQL, and iterate the process repeatedly until the answer fits the requester's needs perfectly.

Two Types of Organizations

From our experience, companies can be broadly placed into two baskets:

1. Young, Growing Organizations: Here, the setup is still fresh. The company is likely in its early stages of creating an internal data warehouse for analytics. You'll probably find a data engineer or two who are multitasking – building data pipelines while simultaneously catering to reporting needs. The real challenge surfaces when the initial data set is ready. Business teams start shooting questions even before comprehensive reports are developed. This is the golden moment when a solution for ad hoc queries can be a game-changer. It offers the business team the autonomy to query the data independently, ensuring the data team isn't unnecessarily disrupted. As a matter of fact, embracing these ad hoc questions can be a huge advantage, helping you clarify business requirements. Think of this as a way to build MVP reporting before you pour in hours of work into a “golden” BI dashboard that misses the mark.
2. The Seasoned Players: These organizations are well into their data journey. They have multiple data sources, a comprehensive BI solution, and probably a myriad of dashboards and reports. The problem? Finding the right dashboard becomes akin to finding a needle in a haystack. Also, a stream of ad hoc queries for exploratory data analysis is a daily occurrence. These organizations employ a sizable force of data analysts and scientists. They're often assigned to specific teams and find it hard to maintain a holistic view of the data.

When Should You Hold Off On Ad Hoc Queries?

There are moments when addressing ad hoc queries shouldn't be at the forefront:

• If your internal data warehouse is still in its infancy. At this stage, trying to answer questions on poor or non-existent data may simply be a fools errand. Getting the fundamentals right should be your top priority.
• If you're in the middle of revamping your data set. Often, a surge in ad hoc queries during this period is a mere reflection of changes in your BI dashboard. Generally these types of projects are managed with carefully planned migration plans. Addressing questions on a moving target can be extremely challenging and slow down the entire project.

The Magic of Modern Technology

Here’s the good news: building a solution and system to address ad hoc requests no longer requires hiring an army of data analysts as a pre-requisite. Data teams will continue to play a crucial role and be worth their weight in gold, but thanks to AI and tools like Fabi.ai, exploratory data analysis (EDA) has become accessible to anyone so long as the underlying data is there. Even if you're only armed with the skill of posing a question in simple English or making minor SQL edits, you're good to go.

‍