Following Public Health England’s inability to understand Excel file sizes and thereby under report Covid cases by 16,000 we need to consider, again, whether Microsoft’s spreadsheet is in fact the most dangerous software on the planet.
A million-row limit on Microsoft’s Excel spreadsheet software may have led to Public Health England misplacing nearly 16,000 Covid test results, it is understood.
The data error, which led to 15,841 positive tests being left off the official daily figures, means than 50,000 potentially infectious people may have been missed by contact tracers and not told to self-isolate.
The danger isn’t that the software itself is awful. It is, rather, that it is so widely used and so widely used badly. On which point here is something I did earlier as with the sticky back plastic things of our childhood:
Microsoft’s Excel Might Be The Most Dangerous Software On The Planet
Feb 13, 2013,09:37am EST
No, really, it’s possible that Microsoft‘s Excel is the most dangerous software on the planet. Yes, more dangerous than rogue code running a nuclear power plant, than the Stuxnet that was deliberately sent off to sabotage Iran‘s nuclear program, worse, even, than whatever rent in the fabric of space time led to the invention of Lolcats. Really, that serious.
There’s a danger at one level: it’s all become so complex and it’s handled in such a slapdash manner that no one is really on top of it anymore. And don’t just take it from me as an assertion, there are very serious people indeed warning about this:
Both the Switzerland-based Basel Committee on Banking Supervision1 (BCBS) and the Financial Services Authority2 (FSA) in the UK have recently made it clear that when relying on manual processes, desktop applications or key internal data flow systems such as spreadsheets, banks and insurers should have effective controls in place that are consistently applied to manage risks around incorrect, false or even fraudulent data. The citation by the BCBS is the first time that spreadsheet management has ever been specifically referenced at such a high level, a watermark in the approach to spreadsheet risk.
To give you an idea of how important this is here’s a great tale from James Kwak:
The issue is described in the appendix to JPMorgan’s internal investigative task force’s report. To summarize: JPMorgan’s Chief Investment Office needed a new value-at-risk (VaR) model for the synthetic credit portfolio (the one that blew up) and assigned a quantitative whiz (“a London-based quantitative expert, mathematician and model developer” who previously worked at a company that built analytical models) to create it. The new model “operated through a series of Excel spreadsheets, which had to be completed manually, by a process of copying and pasting data from one spreadsheet to another.” The internal Model Review Group identified this problem as well as a few others, but approved the model, while saying that it should be automated and another significant flaw should be fixed.** After the London Whale trade blew up, the Model Review Group discovered that the model had not been automated and found several other errors. Most spectacularly,
“After subtracting the old rate from the new rate, the spreadsheet divided by their sum instead of their average, as the modeler had intended. This error likely had the effect of muting volatility by a factor of two and of lowering the VaR . . .”
To translate that into the vernacular, the bank, JP Morgan, was running huge bets (tens of billions of dollars, what we might think of a golly gee gosh that’s a lot of money) in London. The way they were checking what they were doing was playing around in Excel. And not even in the Masters of the Universe style that we might hope, all integrated, automated and self-checking, but by cutting and pasting from one spreadsheet to another. And yes, they got one of the equations wrong as a result of which the bank lost several billion dollars (perhaps we might drop the gee here but it’s still golly gosh that’s a lot of money).
And the various financial market regulators are rather waking up to how these decisions are being made in the markets. And thus the warning at the top: guys, do you think you could pay a little more attention to the tools you are using to move these billions and tens of billions around? For as we can see getting it wrong can be painfully expensive.
So that’s one sense in which Excel could be dangerous: that we’ve tens of thousands, hundreds of thousands, of financiers and bankers throwing trillions of dollars around the markets on the basis of their incomplete spreadsheets and their ignorance of how they’re doing it wrong. Pretty scary really.
But there’s another deeper level of risk here. That very throwing of trillions a day around the markets (and it really is trillions a day: the foreign exchange market in London alone is over $2 trillion a day) is dependent upon the existence of Excel itself.
Well, OK, on the existence of spreadsheets perhaps, so we’d need to include VisiCalc, Lotus 1 2 3, Open Office and all in there. But the only spreadsheet that anyone uses in any quantity in business or finance is indeed Excel. And the thing is, if the spreadsheet, or Excel, didn’t exist, then a lot of what the financial markets do couldn’t be done. There would be no collaterialised debt obligations, (CDOs), no credit default swaps (CDS), indeed much of the complexity of the financial markets would simply disappear in a puff of smoke. For if you cannot model these things (however badly they are modelling them) then you simply could not be trading them as they are.
Quite simply, without Excel we’d not have had the incredible financialisation of the economy over the past 30 odd years. And if we hadn’t had that then we also wouldn’t have had the financial crash of 2007. So there’s a dangerous piece of software for you.
It’s possible to take this too far of course: but it is still true that without spreadsheets then the financial markets just would not look as they do and much of the history of the past 30 years would be rather different. The interesting argument is whether it would all look better without all that finance: I tend to think not but you’re entirely free to disagree with me.