Beware the Query Effect

Whenever you do ask users for their opinions, watch out for the query effect: People can make up an opinion about anything, and they'll do so if asked. You can thus get users to comment at great length about something that doesn't matter, and which they wouldn't have given a second thought to if left to their own devices.

It's dangerous to make big design changes because "users didn't like this" or "users asked for that." If you ask leading questions or press respondents for answers, they might make up opinions that don't reflect their real preferences in the slightest.

For example, if you quiz people about your visual design, you'll inevitably get comments about the colors, even if they're not particularly important to the users. On the other hand, if you hear people mention the colors (unprompted) while they're using the site, then there's probably something to consider. (Say, a comment like "Wow, this neon blue really hurts the eyes," or a more positive statement like "The ultramarine is nice and calming.")