As qualitative researchers we sometimes “know” we are getting the answers but we don’t have the statistical precision of a quant methodologist to prove our point. Subjective arguments just don’t carry the same gravitas as objective, quantitative ones. With that frustration in mind, I have lifted a portion of a post by Victoria Else from her blog, “Behind the Two Way Mirror.” The entire post can be found at http://themarketartist.blogspot.com/2009/04/illusion-of-quantitative-result.html
So here’s my plea. First, if you haven’t already, read The Black Swan; it’s both necessary and delightful. Second, ask yourself some serious questions about quantitative research. It may be–heresy though this is–that qualitative is nearly always a much better basis for the development of marketing messages.
I am not completely anti-quant. Segmentation and behavioral models can lift your results if they are narrative-free (i.e., reflect no assumptions), easy to validate in real-time, and frequently refreshed. However, quantitative messaging studies over-complicate and even distort our understanding of human attitudes and behaviors.
There, I said it.
Why? Qualitative research, done properly (which means more interviews and fewer groups), forces you to deal with the complexity of human reactions in a way that humans are reasonably good at–face to face. In the qualitative setting:
1) You seek the simplest important conclusions. You look for a simple preponderance of evidence that seems consistent or reliable, not “data” connections between human attitudes that are either statistical phantoms or too complex to be replicated in the actual marketplace.
2) You treat the result as temporary. In interviewing actual people, you are confronted with the fact that they are responding to specific stimulus at a specific point in time, and that as the marketplace or external factors change, their responses would probably change.
3) You are somewhat less likely to end up focused on the wrong data or ignoring surprising data. When people are able to speak at length, relevant facts emerge that you would never have considered incorporating in a quantitative study. (That’s also why interviews are better than groups.)
To corroborate this, by the way, I have been told that in the case of branding research, decisions made based on twelve in-depth interviews can produce better in-market results than a quantitative study. (I would infinitely prefer to write a creative brief based on twelve in-depth interviews than on a quantitative study, that’s for sure.)
Allow me to repeat that segmentation and modeling can definitely lift your results. I have seen some excellent models lift results for my clients. However, the excellent models were refreshed frequently, sometimes based on real-time behavioral data, using actual data from actual marketing activities. They avoided making long-term predictions based on data collected at a single point in time. Also, being purely statistical creatures, they contained no narratives; no assumptions of cause and effect, just correlations. That approach, I think, minimizes both the confirmation and narrative errors Taleb refers to.