How to get the most out of AI analytics by asking the right questions
AI is getting better and better at answering data questions, but it's not magic (at least not yet). Knowing what questions to ask to tackle your most important business issues can mean the difference between actionable insights and errors / hallucinations. Let's look at a few examples of common questions and how we can reword them to get better answers:
What’s my user retention?
Problem: This question isn’t specific enough. Different companies calculate retention differently, and there may be multiple metrics by which retention can be measured. How long do you wait before considering a user churned?
Solution: Be specific - provide column names where possible, parameters for the calculations, and the timeframe for your question.
Why aren’t customers buying my product?
Problem: This isn’t really a data question, but rather a question that requires strategic thinking, informed by data. This is your job, not the AI’s.
Solution: Ask questions that have answers in the data. There may be a line of questioning that reveals the answer - e.g. sales over time, sales by product category, marketing budget over time, etc.
How can I increase monetization?
Problem: Again, the answer to this isn’t in the data. The AI doesn’t know why your customers buy your product, but you do.
Solution: The AI can easily highlight areas where reality differs from your expectations. How many of your users are on the free tier? How many convert to paid? Are there any metrics like engagement or usage of a certain feature that seem to correlate well with monetization?
Which features are the most underutilized?
Problem: Who’s to say how much a feature should be used? How is the AI supposed to know that?
Solution: Ask for a graph showing how often each feature is used, and then compare that to your own expectations of product usage. Do the numbers match what you had in mind when you created the feature?
How does the distribution of spending differ between new and existing customers in the US over the last 6 months?
Problem: The question has a lot of parts - just like a human, the AI does its best work when you ask concise questions with a clear answer.
Solution: Break the question down into a series of questions - first ask the AI to split your customers into two groups, then ask for a histogram showing spending by each group.
What balance changes should I make to my game?
Problem: The AI isn’t a game designer and doesn’t know what kind of experience you want to deliver to your users.
Solution: Choose some things that you’d like to balance, such as character or weapon win rates, match duration, etc, and ask the AI to graph those so you can find areas of imbalance where the players’ experience doesn’t match expectations.
In summary, AI data analysis tools should be treated like a junior data analyst - a resource for quickly answering data questions, but not one for strategic thinking. We don’t know how this will change in the future, but as of right now, the best way to get value out of such a tool is to ask questions that allow you test your expectations vs. the reality in the data. You’ll be much better served asking relatively simple questions that the AI can consistently answer correctly rather than complex questions that even humans can get wrong.
AI won’t do your job for you, but it can make it easier.