Parametric studies
The problem with parametric studies
15/10/09 07:13 Filed in: Analysis
A few days ago I was in a meeting reviewing another
teams fairly complex piece of analysis. It was a
fairly obvious question, but someone asked ‘how do
you know if your result is correct’. It’s probably
the most fundamental question you can ever ask when
analysing a problem, and in this case the answer
was a common one ‘we undertook a lot of parametric
studies’.
There is a really major issue with this answer. Just because you have done parametric studies with your model, it still doesn’t actually mean that any of the answers the model is giving are correct. Assuming that parametric studies covers the variability and inaccuracy in your model is not enough.
Lets consider a some working on marketing Cuke Cola trying to work out how much they should spend on advertising. To do this they build themselves a nice little spreadsheet model. The basic assumption in the model is that for every dollar spent on advertising, each person who sees the advert spends an extra 0.01 cents on Cuke Cola on average. So in this case provided 10,000 people see the advert, we will get all the money spent on advertising back. That all makes sense so far, so basically we make a return on the money spent on advertising provided we get at least 10,000 people to see the advert.
We can even do parametric studies on this model, looking at how the break even point varies as the income from each person who sees the advert varies. The problem is that the model is massively flawed. If we assume that 20,000 people will see the advert and we spend 100 dollars on advertising we will get 100 x 0.01 cents x 20,000 = $200. The net profit will therefore be $100. The problem is if we double the amount we spend on advertising, the net profit will also double, and so on. So this model predicts that we should spend every penny we have on advertising because we will double our money. This is clearly flawed for many reasons, but fundamentally, it doesn’t matter how good our advertising is, there is only so much Cuke Cola that a person can drink in a day. The model needs to have some sort of law of diminishing returns built in, and it doesn’t matter how many parametric studies we do, this would not be reflected in the results.
It is not just the inputs to our models, but the models themselves that can be flawed, no matter how much time we spend creating them and how sophisticated they are. Simple use of parametric studies never guarantees that you are getting good results from your model.
There is a really major issue with this answer. Just because you have done parametric studies with your model, it still doesn’t actually mean that any of the answers the model is giving are correct. Assuming that parametric studies covers the variability and inaccuracy in your model is not enough.
Lets consider a some working on marketing Cuke Cola trying to work out how much they should spend on advertising. To do this they build themselves a nice little spreadsheet model. The basic assumption in the model is that for every dollar spent on advertising, each person who sees the advert spends an extra 0.01 cents on Cuke Cola on average. So in this case provided 10,000 people see the advert, we will get all the money spent on advertising back. That all makes sense so far, so basically we make a return on the money spent on advertising provided we get at least 10,000 people to see the advert.
We can even do parametric studies on this model, looking at how the break even point varies as the income from each person who sees the advert varies. The problem is that the model is massively flawed. If we assume that 20,000 people will see the advert and we spend 100 dollars on advertising we will get 100 x 0.01 cents x 20,000 = $200. The net profit will therefore be $100. The problem is if we double the amount we spend on advertising, the net profit will also double, and so on. So this model predicts that we should spend every penny we have on advertising because we will double our money. This is clearly flawed for many reasons, but fundamentally, it doesn’t matter how good our advertising is, there is only so much Cuke Cola that a person can drink in a day. The model needs to have some sort of law of diminishing returns built in, and it doesn’t matter how many parametric studies we do, this would not be reflected in the results.
It is not just the inputs to our models, but the models themselves that can be flawed, no matter how much time we spend creating them and how sophisticated they are. Simple use of parametric studies never guarantees that you are getting good results from your model.