(Sonu Goswami) I Let ChatGPT Do My Market Research. It Cost Me Three Months.
Last quarter, I did something stupid. I fed ChatGPT 50 customer interviews, asked it to “find the patterns,” and built features based on what it told me.
The AI was confident. The summaries were clean. The recommendations seemed logical.
None of it worked.
Adoption rate? 14%. Customer feedback? “This doesn’t solve my actual problem.” Engineering team morale? Let’s not talk about it.
Here’s what I learned the expensive way: AI is really good at telling you what most people say. It’s terrible at finding what matters.
The Day I Realized AI Was Making Me Dumber
Three months ago, I was reviewing user research for a new workflow feature. I had:
- 50 interview transcripts
- 200+ support tickets
- Community forum discussions
- Usage analytics from 5,000 users
I threw it all into GPT-4, asked for insights, and got back a beautiful summary. It told me users wanted “better collaboration features” and “improved notifications.”
So we built better collaboration features and improved notifications.
Launch day: crickets.
Then I did something I should’ve done first. I actually read the transcripts myself. All of them. Took me two full days.
Buried in interview #37, a user mentioned something casually: “I usually just screenshot the data and text it to my team because your export feature makes the file too big for email.”
That throwaway comment appeared once. In one interview. Out of fifty.
ChatGPT never mentioned it. Why would it? It’s a single data point. Statistically irrelevant. Not a pattern.
But when I went back and checked our analytics, 23% of our users were screenshotting instead of using our export feature. We’d built this whole “export to CSV” thing that nobody used because the files were 15MB and Gmail bounces anything over 25MB.
That one “irrelevant” comment? It explained a massive adoption problem. AI missed it because it wasn’t popular enough to be a pattern.
Why AI Keeps Missing the Good Stuff
LLMs work by pattern matching. They’re basically really sophisticated “what do most people say” machines.
Here’s what that means in practice:
If 40 people say “I want dark mode” and 2 people say “your export feature breaks my workflow,” the AI will tell you everyone wants dark mode. Those 2 people? Noise in the data.
Except sometimes those 2 people are telling you about a real problem that affects 20% of your users. They just don’t phrase it the same way, so it doesn’t show up as a “pattern.”
I’ve seen this play out repeatedly:
The Reddit comment problem: Some of our best product insights came from a thread with 6 upvotes in r/ProductManagement. GPT-4 had never seen it. Too obscure. Not indexed well enough. When I asked it about that specific pain point, it gave me generic advice that sounded right but was completely useless.
The “everyone knows this” trap: We had users complaining about onboarding in support tickets. Not many. Maybe 8 out of 200 tickets. Claude told me “most users are satisfied with onboarding.” Technically true. But those 8 users? They all churned within 30 days. The AI saw frequency. I needed to see consequence.
The specialist blog blindness: There’s this tiny blog run by a guy who works at a Fortune 500 company. Maybe 200 readers. He wrote one post about a workflow problem that’s specific to enterprise teams. It’s exactly the problem we’re trying to solve. I found it through manual research. Asked ChatGPT about it specifically. Response: “I don’t have information about that.” Of course you don’t. It’s not popular enough to matter in your training data.
This is the core problem: AI assumes popular = important. In SaaS, the best insights are usually unpopular.
What Humans Still Do Better (And Probably Always Will)
I’m not anti-AI. I use it every day. But there are things my brain does that AI genuinely can’t.
Connecting weird dots: Last month, I noticed our power users were exporting data, editing it in Excel, then re-uploading it. Separately, I saw a support ticket asking if we supported bulk editing. Separately, our analytics showed people spending 6+ minutes on the export page before leaving.
Three unrelated observations. AI would report all three separately. My brain went: “Oh shit, people are using Excel as their bulk editor because ours is too slow.”
We built bulk editing. Adoption: 54% in week one.
Knowing who to trust: I read a blog post from someone with 50 Twitter followers claiming a specific onboarding pattern reduces churn by 30%. Then I saw a viral thread from a “growth expert” claiming the opposite.
AI would probably weight the viral thread more. I checked the small blog author’s background. He’s head of product at a company that grew from 0 to 10M ARR in 18 months. The viral thread guy? He sells courses.
I trusted the small blog. We implemented his approach. Churn dropped 22%.
Reading between the lines: User interview, someone says: “Yeah, the feature works fine, I guess.”
AI categorizes this as positive feedback.
I heard the pause. The “I guess.” The tone. That’s not satisfaction. That’s resignation. That’s someone who’s already mentally checked out but is too polite to say the feature sucks.
We dug deeper. Turns out the feature technically worked but was so slow people stopped using it. They just didn’t want to hurt our feelings.
How I Actually Use AI Now (Without Letting It Make Me Stupid)
I still use AI. But I changed how.
AI’s job: First pass, not final answer
I feed it all the data and ask: “What are the common themes?”
Then I read everything it missed. The outliers. The single mentions. The weird edge cases.
Last sprint, GPT gave me 5 themes from customer interviews. All valid. Then I manually read the transcripts and found 3 more insights it completely missed. Those 3 insights became our highest-performing features.
AI’s job: Find needles, I decide which ones matter
I use AI to search through huge datasets for specific problems I’m investigating.
Example: I suspected users were struggling with a specific workflow. I asked Claude to find every mention of that workflow across 100 interviews. It found 12 mentions.
But here’s the thing: AI just listed them. It didn’t tell me that 8 of those 12 users churned within 60 days. I had to connect that dot myself. Once I did, that workflow became our top priority.
The hybrid approach that actually works
Here’s my current process:
- AI summarization: Feed everything to GPT/Claude, get the obvious patterns (saves me 6 hours)
- Manual deep dive: Spend 2-3 hours reading the raw data myself, looking for what AI missed
- AI-assisted search: Use AI to find specific things I’m investigating (”find all mentions of export problems”)
- Human synthesis: I connect the dots, prioritize based on impact, decide what we build
This combo gives me speed without stupidity.
The Tools I Actually Use (And Why I Picked Them)
I’m not going to pretend there’s one perfect tool. It depends on what you’re doing.
For broad research: I use Claude or GPT-4. They’re good at summarizing large amounts of qualitative data quickly. But I always manually verify the insights they surface.
For accuracy: When I need to analyze specific terminology or domain-specific content, I use RAG setups (Retrieval-Augmented Generation). Basically, I give AI access to only my data, not the whole internet. Way fewer hallucinations. Way more relevant answers.
For finding rare insights: Honestly? Google, manual searching, and reading things myself. I’ve found more breakthrough insights on page 3 of Google than any AI has given me.
The setup that works:
- Claude for summarizing interview transcripts (fast, good at nuance)
- Custom RAG system for analyzing our product docs and support tickets (accurate, doesn’t make stuff up)
- My brain for reading between the lines and connecting non-obvious dots
- Spreadsheets for tracking which insights actually led to successful features (so I know what to trust)
What This Actually Means for You
If you’re a SaaS founder using AI for research, here’s what I wish someone had told me:
AI is your research assistant, not your researcher. It can read faster than you. It can organize better than you. It cannot think better than you.
The best insights are almost always unpopular. If everyone’s talking about it, it’s probably not your competitive advantage. The gold is in the stuff that 3% of users mention once.
Hybrid is the only approach that works. AI for speed and scale. Human brain for judgment and connection. Anyone telling you to go all-in on either is selling something.
Trust your gut when AI’s answer feels wrong. I’ve had AI tell me something was a “low priority” issue that my gut said was actually critical. Every time I trusted my gut over the AI, I was right. Every time I trusted the AI over my gut, I wasted engineering time.
The Real Lesson
Three months ago, I let AI do my thinking. It was faster. It was easier. It was wrong.
Now I use AI as a tool, not a brain replacement. It handles the grunt work. I handle the insight work.
Our feature adoption rate went from 14% to 61%. Our development cycle got shorter because we stopped building wrong things. Our team is happier because they’re building stuff that actually ships and gets used.
AI didn’t do that. AI + my brain did that.
Don’t let efficiency make you stupid. Use AI to go faster. Use your brain to go in the right direction.
The data’s easy to find. The insight is hard to see. That’s why we still get paid.