A few days ago I was in a meeting reviewing another teams fairly complex piece of analysis. It was a fairly obvious question, but someone asked ‘how do you know if your result is correct’. It’s probably the most fundamental question you can ever ask when analysing a problem, and in this case the answer was a common one ‘we undertook a lot of parametric studies’.

There is a really major issue with this answer. Just because you have done parametric studies with your model, it still doesn’t actually mean that any of the answers the model is giving are correct. Assuming that parametric studies covers the variability and inaccuracy in your model is not enough.

Lets consider a some working on marketing Cuke Cola trying to work out how much they should spend on advertising. To do this they build themselves a nice little spreadsheet model. The basic assumption in the model is that for every dollar spent on advertising, each person who sees the advert spends an extra 0.01 cents on Cuke Cola on average. So in this case provided 10,000 people see the advert, we will get all the money spent on advertising back. That all makes sense so far, so basically we make a return on the money spent on advertising provided we get at least 10,000 people to see the advert.

We can even do parametric studies on this model, looking at how the break even point varies as the income from each person who sees the advert varies. The problem is that the model is massively flawed. If we assume that 20,000 people will see the advert and we spend 100 dollars on advertising we will get 100 x 0.01 cents x 20,000 = $200. The net profit will therefore be $100. The problem is if we double the amount we spend on advertising, the net profit will also double, and so on. So this model predicts that we should spend every penny we have on advertising because we will double our money. This is clearly flawed for many reasons, but fundamentally, it doesn’t matter how good our advertising is, there is only so much Cuke Cola that a person can drink in a day. The model needs to have some sort of law of diminishing returns built in, and it doesn’t matter how many parametric studies we do, this would not be reflected in the results.

It is not just the inputs to our models, but the models themselves that can be flawed, no matter how much time we spend creating them and how sophisticated they are. Simple use of parametric studies never guarantees that you are getting good results from your model.

Tags: Analysis, Parametric studies, Sensitivity analysis

You can’t beat actually seeing something being done to help you understand how to do something so here’s my first Video tutorial on the use of Risk Engine. It based on a simple example that can be found in the book How to measure anything by Douglas Hubbard, but you won’t need the book to follow along. Let me know what you think.

Risk Engine video tutorial from Ian Turner on Vimeo.

Tags: Risk Engine, Tutorial, Video, Douglas Hubbard

Probably the feature that is requested for Risk Engine with the most energy is the ability to shuffle ranges of data for regression analysis in biological sciences. I've been working on this feature for the last couple of weeks and today I'm releasing a beta version. This is partly to allow users to test the software if they wish, and partly to give users access to the functionality as soon as possible. This version currently only works with iWork Numbers and I have some development work on validation, documentation and adding the functionality for Microsoft Excel, but I'm hoping to address all of this in the near future.

If you would like to use the latest version you can download it here: Risk Engine Beta

Instructions for how to use analyses using ranges are available on the Risk Engine Wiki: Analysis using ranges of data

Tags: Risk Engine, Beta

A few weeks ago I saw Malcolm Gladwell speak in Brighton on expert failure and the credit crunch. It was a stunning and mesmerising talk and as ever really thought provoking. It's not the full talk he gave in Brighton, but here is part of the same talk given at the New Yorker Summit on 5th May.

Tags: Malcom Gladwell, Expert failure

I had a sudden realisation today about the poor environmental credentials of this website. I've seen a few reports recently that have honestly shocked me about the carbon emissions from websites and emails. I guess because it is so cheap, I don't realise the cost on the environment. Today I read an article saying that each item of junk email produces an average of 0.3g of carbon. I don't know how acurate that number is, but I find it truly amazing that something with such a small piece of information can produce enough carbon to actually see.

I've decided to try and make this website and all of it's surrounding activities carbon neutral. I suspect this is not going to be an easy thing to achieve, because the information and tools just are not available to make it a simple fill in a form and make a few payments exercise. Somehow I need to work out, for each activity, be it website views, emails, application releases or just support activities, how much it will cost to do the release. My other problem is that I want to be carbon neutral at the source rather than by mitigating carbon emissions. I feel that this is a little bit more holisitic than simply working out emmisions and then paying to have trees planted that offset the emmisions.

This is not going to be an easy problem to deal with and it certainly is not going to be something that I can complete overnight. I do however want it to be as open as possible so I will openly publish the work I do on this. In the mean time, if anyone has any ideas or useful tools for how to assess carbon emissions from websites or emails please drop me a line and let me know.

Tags: Risk Engine

In the last few weeks I've heard a few discussions about 'solution spaces'. I thought it might be time to update a post I wrote a few years ago on my interpretation of the solution space and the importance of boundaries in creativity.

There is a fairly commonly held belief that boundaries are bad for creativity; the more boundaries you have the less creative your outcome. Creative endeavours are often described in terms of 'pushing the boundaries', opening up the space in which to create something and thus being creative simply by exploring something that has never been explored before.

So what does this mean for engineers and other real world problem solvers. If we push the boundaries of what has been achieved before, does this mean we are being creative? And what does it actually mean to push the boundaries in terms of creativity. It can not be the limits of technology, otherwise the only source of creativity in real world problem solving would be through scientists rather than engineers and designers.

For me, the solution space is a really important concept in helping to understand these issues. I tend to think of the solution space as a multi-dimensional space where each variable of the clients requirements, such as cost, time, accessibility or reliability, is situated in one of the dimensions.

Consider a client who requests the services of an engineer to design a bridge from an island to the mainland. He gives the engineer nothing more in the brief than the request for a bridge. What sort of bridge should the engineer design? In this situation, she must do her best to satisfy the client, but with such a limited brief she shouldn't do anything particularly creative. The design will be fairly standard, maybe a simple concrete two lane highway bridge. Nothing particularly special, but at least an attempt at giving the client what he wants.

In this situation, the clients brief was clearly a problem. The lack of definition meant that there was little chance that the bridge would actually give the client what he actually wanted. It also means there is little chance for the engineer to be truly creative, trying to satisfy the clients requirements.

So let us say the client is a little more definitive about the brief, and define the cost. The client knows his budget for the bridge and says the bridge should cost between £3 million and £4 million. Now things are a little more defined and the engineer has some sort of idea how best to satisfy the clients requirements. In this instance we can also start to represent the clients requirements in terms of a solution space:



There is still not much room for creativity, and the chance of satisfying the clients requirements is still fairly low. Does he want a footbridge, a two line highway bridge or even a twelve lane highway bridge?

For a better design, that satisfies his desires better, the client adds a little more detail to the request. The brief is now that the costs should be around £3.5 million and there should be four carriageways. With two dimensions to the clients request, the engineer now has a much more interesting solution space to work with. The graphical representation goes from one dimension to two dimensions. With such a defined brief from the client there is also a single point on the graph that represents exactly what the client requested:



With this brief the engineer can now work on a solution which we can also represent graphically in the solution space:



Off all the possible solutions, there might only be a handful of solutions that actually satisfies the clients requirements. We are now in a situation where the engineer is producing designs that may genuinely have never been created before. This allows the engineer to express some of their creativity in developing a solution and yet in this instance, the boundaries have not been pushed, the boundaries are actually allowing the engineer to express some of her creativity.

For me, this is the essence of creativity as an engineer. By defining your boundaries better, it actually forces the engineer to come up with a more creative solution to satisfy the clients requirements.

The real world is of course much more complicated than this. What if the client requested a bridge costing around £2.5 million? Suddenly the engineers solution space is very different:



Now it seems like we can not deliver the clients desires. This is the sort of situation where true creativity is required, the engineer has a few options for dealing with this situation. One option is to push the boundaries of the possible solutions, expanding the possible range until it encompasses the point which satisfies the client. Another option might be to consider the clients original request for the cost of the bridge. He requested a bridge of around £2.5 million, so there is obviously some scope to vary the cost, although the further the final cost is from the original, the less likely that the clients requirements are satisfied. It is therefore very important to not only understand each of the variables, or dimensions, in the clients brief, but also how client satisfaction varies within that dimension. Does the client want it maximised, minimise, is there an absolute floor or ceiling to the request, and if there is some variation allowed, how much?

Ideally we would want to represent all of these dimensions graphically as it is much easier to understand the problem. Most clients desires are however heavily multi-dimensional. As such they are very difficult to plot graphically. One document should however be used by every design team to lay out the solution space. In this instance I'm going to call that document the design specification, although it may be know by many other titles. The design specification lays out in detail how the clients desires will be satisfied. It gives both the dimensions which need to be considered to provide a design that satisfies the client, as well as the range within each dimension that can be considered. Given the real world nature of the problems that the specification defines, it is important that it gives the designer some wriggle room when dealing with a problem so some of the dimensions in the specification must be flexible, and yet some will be fixed with no room for changes; few clients let designers compromise on health and safety in delivering a solution. The design specification therefore defines the solution space that the designer must search in order to provide a solution to the client, and by laying out the boundaries clearly, this document allows the engineer to be truly creative in the search for a solution.

By way of closing, I'd like to go back to the client who wanted a bridge between the mainland and an island. The client now comes with a modified brief that they want a minimum cost bridge, with a very low capacity and which is not classified as a critical link so it does not necessarily need to be open every day. In this instance, a cheap, single lane, bridge design might somewhat satisfy the client. But would a simple vehicle ferry give more satisfaction being much cheaper and yet still satisfying the other requirements? A ferry would be a truly creative solution to a request for a bridge from a client and yet this creativity is only possible by truly understanding the clients requirements and having a good design specification at the outset.

Tags: Creativity, Solution space, Design