Gathering requirements: the curious case of the echo chamber

In this article, I want to share a personal experience from years ago when I uncovered that the entire business completely misunderstood the customer request (and nearly made it a strategic blunder!) We all know and keep hearing how business analysts and product managers need to be able to capture feedback from stakeholders. And for the most part, that seems easy. But let me demonstrate how it can go awry.😱

The customer call

Years ago, I was on a customer call with one of the executives. We were a B2B SaaS (software provider for businesses), and the customer was the team lead at a company using our software. Because that customer was one of the early adopters and a target persona for us, the call was deemed a significant learning opportunity.

The customer praised many things about the software but mentioned one problem — one of our collaboration features duplicated what his team had already built themselves in Salesforce. He asked if it would be possible to integrate our feature with theirs to continue using their solution but keep feeding data into our system.

Therefore, we documented that request in our system and other feedback and decided to revisit it all at a later date.

The sales-product call 

A couple of days later, I attended the call with our sales team. It was a regular meeting scheduled between the members of the sales team and myself, as a product manager. We used it to discuss any recurring or recent customer requests and feedback.

One salesperson mentioned that they have heard from customers that they don’t want to use our collaboration features because they already had a working solution. He said this created a problem as this data/interaction was crucial to our system’s workflow. However, if we could allow our customers to integrate with their existing solutions, it would address the problem.

This immediately piqued my interest. “That was exactly what I heard from a customer just last week!” I said excitedly. To my delight, every other team member on the call concurred:” I’ve heard the same”, "Me too", and “I came across the same feedback”.

It genuinely felt like we had a mounting amount of evidence that there was an urgent problem worth solving and an obvious solution to address it.

The executive call 

Another week or so went by when we had the product strategy call. Again, all executives were present, and we were discussing and debating what our priorities should be for the coming quarter.

One of the founders made the point that of all the opportunities, one that seems to be the most well documented and consistently recurring is the need for an integration between our collaboration feature and other solutions. The head of sales vigorously agreed with that assessment.

Long story short, addressing that particular problem soon became one of my top strategic priorities.

The missing leads

At that moment, I decided to do what product managers should do: make sure I correctly understand the problem, the job to be done, and what other solutions our customers use for collaboration. It felt like with so many recurring requests, and it wouldn’t take too long.

To my surprise, I was right, but not for the reasons I suspected. 

I reached out to one of the executives who spoke passionately about this problem to gather which customers have raised this issue. However, he only said that he heard this from the sales team, and the only case where he heard this directly was on the call we both attended weeks ago (the customer call).

So I went to speak with the head of sales. I got the same answer — he has not heard this feedback directly from any customer but the sales team have raised this. And hence I went on to speak with the sales team. And to my surprise, nobody could recollect ANY customer who has mentioned this need! How was this possible?! Merely 2 weeks ago on the product-sales call everyone said they have heard the same. 

But that was just it — they literally did hear the same feedback. 

The echo chamber

After a few calls and a few exchanged emails, I pieced together what transpired. It turns out, shortly after the original customer call, the executive shared the customer feedback with one member of the sales team. That member mentioned it to other salespeople individually during different calls.

This is how the echo chamber was created. Since everybody only mentioned the feedback casually to another team member and never mentioned the customer by name, nobody knew exactly who raised the request. So when the salespeople started discussing customer requests among themselves, they were unaware that they were essentially referring to the same request. 

When the executive and I heard back the sales team’s feedback, we were blissfully unaware that we were hearing back exactly what we shared ourselves in the first place. But the confirmation bias has already been fed and sent us down a perilous path.

The 2nd customer call

Even though the perception of mounting evidence has been disproven, I still decided to follow up with the customer who raised the original request to dig deeper and understand the need. After all, we still could have decided to pursue the integration.

But that’s when the echo chamber experience finished on a hilarious twist — the customer said to me, ‘Forget about it, we decided to use your feature instead.’

That’s when I learned — always check your sources and follow-up on leads!

How to avoid the echo chamber?

There are many ways to ensure the 'echo' chamber does not happen to you and your organization:

• Standardize on feedback mechanism: no 'I've heard', no slack messages and emails. You should have a centralized library of feedback records, ideally in your CRM system. 
• Ensure feedback data quality: Specify what information needs to be captured (who, which account, when, what was requested, what is the problem etc.) for the record to be accepted for consideration.
• Document evidence: Salespeople and Customer success people, who are often the first point of contact with prospects and customers, are very busy. They have a lot of calls in a day. And so their memory of what was said exactly by a customer will naturally fade. So investing in call recordings (Meet/Zoom), with transcribing capabilities (we use Gong) is the single best way of ensuring you collect actual market intelligence, not Chinese whispers. 

While my personal experience is one of a product manager in a commercial software company, the lesson of this story is just as applicable to business analysts working with internal technology stack. It is all too easy to rely on 'I've heard...', 'They said...' type of communication. You need to build a reliable mechanism for capturing well-evidenced stakeholder needs. Otherwise you'll end up chasing false leads more often than you realize.

What’s next?

Having a standardized feedback capture mechanism and capability to capture transcripts/recordings of user feedback is just the first step. All of that market intelligence needs to be properly analyzed by a skilled business analyst or product manager.

If you would like to master ability to break customer feedback down to the underlying needs and drivers, and design truly useful, innovative solutions, then I would like to recommend my masterclass in Total Story Visualization: