With a current project the client agreed that getting reaction to the wholesale changes we were doing on the website was a good idea. He had a new logo done by another designer that he was happy with and I told him I would work with that in my new designs.  One thing that came out of the design exploration was that the colors the other designer had used were a bit garish.  I showed him some alternatives that I felt fit in with his audience more.

We worked out the survey questions and he sent it out to some of his customers to get their reaction.  To our surprise they seemed to choose the bright colored original that we were both starting to dislike. In most cases it would have stopped there and that would have been it. Luckily, this client was open to keep prodding the database to see why we were getting responses that we just didn’t expect.

We changed the questions, tweaked the new colors a bit and sent out the new survey. Sure enough it came back with the same verdict – the original bright colors were the choice. But… as we looked at the open ended feedback we found something interesting: they didn’t like the design of the logo and the color of it was of no consequence.

We hadn’t even asked whether they liked it – they just offered up that opinion. This intrigued him (and me) enough that he wanted me to design some alternatives we could show and I did. There were two that he loved, but we wanted to query the customers again with these new logos. Frankly, inside I felt they would probably choose the one they had been choosing (although each time we queried new people.)

I was wrong.  They loved the one both he and I liked best. And the logo they had been choosing as the winner was dead last!

This points out that just asking questions is almost as dangerous as not doing any surveying. Had we stopped with one round we would have been confident in the design choices based on feedback that was tainted.  And simply because we asked the wrong question (or didn’t include the right ones) we would have ended up with a design that was less than ideal.

So remember:

1. Listen to your gut when all else fails. If you are getting results back that just don’t seem to make sense you could be damaging the results with poorly written questions, confusing choices or plainly not asking them what needs to be asked.
2. Always provide some catch-all place to allow participants to tell you something other than what you are asking.
3. Always do a few rounds of questioning to make sure you aren’t leading the participants to certain conclusions.
4. Don’t just look at the fancy pie charts and tables. Reading the free-form data is a pain and it’s hard to quantify, but there is likely some good stuff in there.

And potentially stuff that could change the entire site for the better.