Product Development Uncategorized

Making an ass of your assumptions

When faced with a problem it’s very easy to go with the obvious answer. If we expected X to happen but instead we got Y, then we can only assume Z must be the reason.

If presented with a hard fact, we tend to fill in the blanks to make sense of why something didn’t happen the way we expected. Sometime we’re right. There is a reason why it’s the obvious answer. But like stereotypes, which can be based on repeated traits, assumptions can also be wide of the mark.

We need to think differently. We need to dig beneath the surface.

To do so, it’s important to understand the data, but we also need to put ourselves in our customers shoes and look at it from their perspective. Data can give you the what, but not the why.

Scrap it, start again

Let’s look at an example. My previous employer,, launched an iPhone job search app.

It was built in response to the considerable mobile traffic growth they were experiencing. It was a basic app, nothing fancy, but focused on the core need of the jobseeker – to search and apply for jobs.

It had several hundred thousand downloads, drove a lot of activity, but it only averaged 1.5 stars out of 5. This was pretty much in line with the other job hunting apps in the App Store.

The initial reaction to the score?

The app is not good – we need to either upgrade it or build a new one.

But that didn’t sit well with me. I didn’t think the score was a fair reflection of the app. Yes, I knew there were things that needed to be improved, and we were working on them in the background, but I felt the app was better than its rating.

Why was it so low?

Digging for treasure

It didn’t take too long to find the answer. Aside from the star rating, Apple allow customers to write a review of their app experience. When your app is getting positive reviews – 4 and 5 stars – then this is a great piece of social proof that is likely to encourage other shoppers to download the app. However, negative social proof can have the complete opposite effect and put prospective customers off.

Once you get over the initial sting of reading the reviews you realise how much of a gold mine of information it can be if you’re interested in improving the customer experience.

With a little reading it became apparent that most of the reviews were from people frustrated that they couldn’t find or get a job, and were projecting that frustration in their review of the app.

It’s an unfortunate fact within recruitment that most people are unsuccessful with a job application. At best, one person is going to be really happy (if they got the job), some might be disappointed but had a positive experience during the process (e.g. good recruiter service or getting short listed), but most are frustrated due to not even getting any feedback to their application let alone invited to interview.

The initial thought at this point is “well, we can’t do anything about that, that bit is out of our control”. Which is true, a job board plays little to no role in that part of the recruitment process.

So I guess we can’t affect the scores then? We’re stuck at 1.5 out of 5. Let’s move on.

Not so fast. Let’s look at this differently.

Finding out the why

We’ll start with user behaviour. When do users leave feedback on apps?

It’s actually not that easy to do. If a user is being proactive they could visit the App Store, find the app and leave a review. This does requires a bit of effort on behalf of the user, and I’d hazard a guess most people don’t have the time or inclination to do that.

So when else?

When you delete the app from your device.

And when are you likely to do that? Perhaps when you’re frustrated because you can’t find a job?

So its possible that businesses, regardless of offering, are getting poor ratings because customers are being asked to rate the app at a time when they’re unhappy.

You can test this hypothesis quite simply – flip the scenario and ask your customers at a time when they are happy and see what happens.

With job hunting the most positive times would either be when the jobseeker gets the job or when they’ve just applied for a job. Unfortunately neither were options for Jobsite due to process and technical limitations, so we theorised that the next best time would be after using the app on three occasions. With the high abandonment rate of apps, we reckoned if you’re using it for the third time, you must be finding it useful.

So we introduced a timed pop up – see example image – inviting them to feedback on the app.


The big reveal

The results?

The app rating jumped from 1.5 to 4 out of 5 immediately!

We hadn’t improved the app in any way, we’d just captured the views of the silent – and happy – segment, who until that point had had an obstacle strewn path to review the app.

One of the great things about having to resubmit the app to the App Store following a change is that the ratings displayed are for the new version. So any positive impact is felt immediately, as you’re not having your average score dragged down by all the previous low scores.

It’s worth noting though, you’ll still get the bad reviews – the product still needs work – but now they’re just balanced out with the good ones. In the meantime you’ve built strong social proof in the App Store and a better star rating equals more downloads.

So what are the lessons learned?

For a start, assumptions can be dangerous. When we’re pushed for time, it’s easy to look at ‘facts’ and take them as gospel. The star ratings said the app was no good, but your future business planning cannot be determined by hundreds or thousands of bits of customer feedback distilled down into a simple 5 star graphic. You need to go a step further and unearth richer data to make informed decisions.

You’ll come across roadblocks – such as an industry working a particularly unhelpful way – so you’ll need the resolution to not accept things as set in stone. You can either work out how to disrupt that industry (tip: focus on what would really make the customer happy) or you can perservere and find a way to work the system to your advantage.

This all takes time, time you perhaps don’t have. Consider this though, do you want to get lots of things done and maintain the status quo or do you want to get less things done but do them really well and shift the needle?

Leave a Reply

Your email address will not be published. Required fields are marked *