In this article I’ll try to guide you through how Choice Modelling works and why we believe it is a revolution in Market Research. No prior knowledge is required and of course if you still have questions at the end, you can always drop me a line.

Ben White
Ben WhiteFounder

But first a personal disclaimer.  Having run hundreds of choice experiments at SurveyEngine over the past 15 years, we are convinced Choice Modelling can replace nearly all current popular methods of Market Research. The results of a well defined choice experiment make most market research look like the reading of tea-leaves.

But it wasn’t always so. I was first introduced to the concept by Hikaru Phillips of Virtual Vegas fame who clumsily explained that he’d met an American professor who could predict my behaviour. I was naturally affronted. Me? I’m an individual. You can’t predict what I’ll do. The following week I met Professor Jordan Louviere, who I would work alongside for many years, and after grappling with the mathematics, I realised it was so. People may not be rational in their decisions, but they are (mostly) systematic. And if there is a system it can be discovered through a systematic experiment.

So what is Choice Modelling?

Choice Modelling is simply running experiments on people to discover their decision process.

If you remember the game ‘mastermind’ from the 70’s it may give you the flavour of what choice modelling is trying to do.

MastermindIn the game of mastermind, the first player, lets call her ‘the subject’, chooses 4 coloured pegs hidden from view. The second player, the ‘experimenter’, has to work out what the hidden pegs are by trying different combinations. Each turn the ‘subject’ gives clues to how close the ‘experimenter’ is by awarding a white marker for a correct colour and a red marker if both the colour and position are correct. The experimenter has only 12 turns to work out what the hidden pegs are.

To play to win, just guessing each time won’t work. There are far too many combinations, but by systematically testing patterns, most experimenters can win in less than 12 turns.

Choice Modelling is similar in many ways. ‘Subjects’ have hidden preferences, so hidden in-fact that they are often unaware of them or cannot express them. A good experimenter, through careful but limited testing, can.

A more concrete example

Imagine the last time you made a non trivial purchase, such as buying a car. There was some explicit set of rules you came up with to base your decision on or at least to justify it. You could explain these, though probably not quantify them. Something like – not too expensive, low mileage, 2 years free service and not brown or pink.

There were also most likely a set of subconscious rules that had an influence, such as a likeable sales-guy or a pleasing thud as the door closed. But ultimately you made a decision based on all of these factors and that was the best you could do. You successfully ‘maximised your utility’ as an economist would say.

But what if I owned a car dealership and wanted to know, accurately and even better than even you knew yourself, how your decision was made? And why would I want to know?

Knowing your decision process exactly,  I would also know how much to reduce the sales price of those pink station wagons. Or I would know how much more I could increase the sales price if I added an additional 2 years ‘free’ service. I would have a huge advantage of being able to see every possible sales outcome in advance. I could then find the one that was the most profitable. Or I might just want to make the most number of sales.

Having a model, a model of consumer choice, puts that power of foresight close to hand. Choice Modelling is the process to create that model.

What does a model look like?

models_small

Above is a detail from a recent choice model on a financial product. Such a choice model shows how each part of the product offer affects whether once choice will be made over another. The ‘parts’ of the offer are called factors and can be anything conceivable; like prices, mileages, a colour range or even the aftershave brand of the sales-guy.

Now we could almost do the same thing with a questionnaire by ranking or rating car attributes. We may find that pink is less popular than silver, or that 3 years service is preferred to 2 years. But we wouldn’t be able to combine this information in a rigourous or useable way.

The case for a questionnaire becomes more and more shaky as we ramp up the experiment to a complete and realistic description of car buying choices.

A Combinatorial Explosion

We are about to enter territory that is both complex and astounding, so stay with me.

A realistic description of a car for sale, as in a classified ad, would have the following:

  • Make, Model and Year
  • Price
  • Mileage
  • Colour
  • Condition
  • Used or New
  • Warranty
  • Accessories

It may also have a number of ‘soft’ factors as well

  • descriptions – such as ‘pre-loved’
  • keywords – like ‘classic’ or ‘muscle-car’
  • inspection times – like weekends only
  • contact methods – such as email or phone
  • locations – close or near to the city

If each of these factors has a modest 4 possible levels, e.g  colour could be either red, green, white or pink, then a quick calculation says there are 4x4x4x4x4x4x4x4x4x4x4x4x4x4x4 = 1,073,741,824 possible combinations of cars. Over a trillion possible ways a car ‘could be described.

A questionnaire attempting to differentiate between any of these combinations would either fail outright or worse still give spurious results to base a strategy on. A choice model however would achieve this easily. A choice model would show how large the effect of each factor was compared to every other factor, which factors dominated and which were irrelevant.

Is it worth the added complexity?

Without doubt. Unlike 15 years ago, technology has advanced so that all that is required is to clearly describe the question you want answered – the rest is taken care of with technology. Questions like:

  • What is the optimal pricing and packing for my product to maximise profit?
  • How much more can I charge for my pizzas if I offer free delivery?
  • Which features of my product do customers not care about at all? (and can therefore drop them)

However, often the answers are much richer than expected so I’ll finish with a true anecdote.

Fighter pilots and Choice Modelling

A number of years ago we were contracted by a ‘leading military power’ who had a problem. Many of their highly trained pilots who they had spent millions on training, were leaving to join to commercial airlines.

A knee-jerk analysis was that ‘clearly the pay was better at Qantas, so we should pay our pilots more’.

Fortunately Brigadier-General Nichols Jans was on the remuneration commission. Nick Jans was also an econometrician in civilian life and advocated a choice modelling approach. After all there was no harm, if it really was just about remuneration, a model of choice would support that hunch.

SurveyEngine ran a large scale choice experiment on active Air Force pilots. As with the car example above, we modelled realistic choices between Military and Civilian service. Current and hypothetical factors were included alike: location, airline, superannuation, medical plan, pay scales, flying hours, domestic, training, spousal support and many others.

When the experiment was complete and the models generated, we had an realistic description of how Air Force pilots made career decisions. The usual suspects were there of course, paying them more would partially halt the loss of pilots.

But one factor stood out and looked wrong. Flying hours. We had all expected increased flying hours to have a negative effect, but the model had it round the other way. The model predicted that more flying hours meant more pilots would stay in service. A check with some pilots confirmed that the model  was indeed correct.

We all felt a little stupid. Of course, these young guys joined to fly fighter jets at supersonic speeds, yet the military bureaucracy had gradually been increasing desk-bound office work. Of course they were leaving.

The policy was quickly reversed and without having to increase pay, the attrition of pilots was halted.

This result is characteristic of the power of Choice Modelling, it is agnostic about pre-existing ideas, is evidence based Science and like all Science it uncovers unexpected, but in hindsight obvious, truths.

And the obvious was, as Val Kilmer said, ‘I feel the need … the need for speed’.

Top_Gun_large_verge_medium_landscape

 {:}