Home Business Microsoft Excel Truly Is The Most Dangerous Software On The Planet

Microsoft Excel Truly Is The Most Dangerous Software On The Planet

Author

Comments

Following Public Health England’s inability to understand Excel file sizes and thereby under report Covid cases by 16,000 we need to consider, again, whether Microsoft’s spreadsheet is in fact the most dangerous software on the planet.

Yes.

A million-row limit on Microsoft’s Excel spreadsheet software may have led to Public Health England misplacing nearly 16,000 Covid test results, it is understood.

The data error, which led to 15,841 positive tests being left off the official daily figures, means than 50,000 potentially infectious people may have been missed by contact tracers and not told to self-isolate.

The danger isn’t that the software itself is awful. It is, rather, that it is so widely used and so widely used badly. On which point here is something I did earlier as with the sticky back plastic things of our childhood:

Microsoft’s Excel Might Be The Most Dangerous Software On The Planet

Feb 13, 2013,09:37am EST

No, really, it’s possible that Microsoft‘s Excel is the most dangerous software on the planet. Yes, more dangerous than rogue code running a nuclear power plant, than the Stuxnet that was deliberately sent off to sabotage Iran‘s nuclear program, worse, even, than whatever rent in the fabric of space time led to the invention of Lolcats. Really, that serious.

There’s a danger at one level: it’s all become so complex and it’s handled in such a slapdash manner that no one is really on top of it anymore. And don’t just take it from me as an assertion, there are very serious people indeed warning about this:

Both the Switzerland-based Basel Committee on Banking Supervision1 (BCBS) and the Financial Services Authority2 (FSA) in the UK have recently made it clear that when relying on manual processes, desktop applications or key internal data flow systems such as spreadsheets, banks and insurers should have effective controls in place that are consistently applied to manage risks around incorrect, false or even fraudulent data. The citation by the BCBS is the first time that spreadsheet management has ever been specifically referenced at such a high level, a watermark in the approach to spreadsheet risk.

To give you an idea of how important this is here’s a great tale from James Kwak:

The issue is described in the appendix to JPMorgan’s internal investigative task force’s report. To summarize: JPMorgan’s Chief Investment Office needed a new value-at-risk (VaR) model for the synthetic credit portfolio (the one that blew up) and assigned a quantitative whiz (“a London-based quantitative expert, mathematician and model developer” who previously worked at a company that built analytical models) to create it. The new model “operated through a series of Excel spreadsheets, which had to be completed manually, by a process of copying and pasting data from one spreadsheet to another.” The internal Model Review Group identified this problem as well as a few others, but approved the model, while saying that it should be automated and another significant flaw should be fixed.** After the London Whale trade blew up, the Model Review Group discovered that the model had not been automated and found several other errors. Most spectacularly,

“After subtracting the old rate from the new rate, the spreadsheet divided by their sum instead of their average, as the modeler had intended. This error likely had the effect of muting volatility by a factor of two and of lowering the VaR . . .”

To translate that into the vernacular, the bank, JP Morgan, was running huge bets (tens of billions of dollars, what we might think of a golly gee gosh that’s a lot of money) in London. The way they were checking what they were doing was playing around in Excel. And not even in the Masters of the Universe style that we might hope, all integrated, automated and self-checking, but by cutting and pasting from one spreadsheet to another. And yes, they got one of the equations wrong as a result of which the bank lost several billion dollars (perhaps we might drop the gee here but it’s still golly gosh that’s a lot of money).

And the various financial market regulators are rather waking up to how these decisions are being made in the markets. And thus the warning at the top: guys, do you think you could pay a little more attention to the tools you are using to move these billions and tens of billions around? For as we can see getting it wrong can be painfully expensive.

So that’s one sense in which Excel could be dangerous: that we’ve tens of thousands, hundreds of thousands, of financiers and bankers throwing trillions of dollars around the markets on the basis of their incomplete spreadsheets and their ignorance of how they’re doing it wrong. Pretty scary really.

But there’s another deeper level of risk here. That very throwing of trillions a day around the markets (and it really is trillions a day: the foreign exchange market in London alone is over $2 trillion a day) is dependent upon the existence of Excel itself.

Well, OK, on the existence of spreadsheets perhaps, so we’d need to include VisiCalc, Lotus 1 2 3, Open Office and all in there. But the only spreadsheet that anyone uses in any quantity in business or finance is indeed Excel. And the thing is, if the spreadsheet, or Excel, didn’t exist, then a lot of what the financial markets do couldn’t be done. There would be no collaterialised debt obligations, (CDOs), no credit default swaps (CDS), indeed much of the complexity of the financial markets would simply disappear in a puff of smoke. For if you cannot model these things (however badly they are modelling them) then you simply could not be trading them as they are.

Quite simply, without Excel we’d not have had the incredible financialisation of the economy over the past 30 odd years. And if we hadn’t had that then we also wouldn’t have had the financial crash of 2007. So there’s a dangerous piece of software for you.

It’s possible to take this too far of course: but it is still true that without spreadsheets then the financial markets just would not look as they do and much of the history of the past 30 years would be rather different. The interesting argument is whether it would all look better without all that finance: I tend to think not but you’re entirely free to disagree with me.

SUPPORT US WITH A SUBSCRIPTION?

8 COMMENTS

  1. Completely correct. Excel’s low barrier to entry for non-programmers is both its great power and its Achilles heel. Excel is just as capable as any other development environment when used correctly (e.g. without formulae, with sophisticated add-ins and extensions, and with a proper development cycle including extensive automated testing). What is characteristic of all these “Excel is bad” headlines is the extension of a quick-and-dirty manual process into an automated enterprise scale – something that is much harder to do in other languages simply because a typical user would have no idea where to even start with them, so professional development (and all its higher costs and delays) becomes necessary.

    In this case, Excel does seem the wrong solution – whilst it would have made perfect sense for the manual data analysis and process design, something like Python would have been much safer and quicker for the automation and productionisation of the process once fixed, with Excel then used for any additional manual analysis of the resultant outputs for non-programmers. Particularly, falling for the very well known .xls 65535 limitation is embarrassing, and splitting into batches rather than simply moving to the new file formats seems similarly ill advised.

    • The right solution for the NHS would have been SQL Server and using SSIS packages for importing. SSIS is a tool built for this job. You can define CSV as a source, map fields to a table, have it run for multiple files in a folder and set it up to trigger automatically. You can define fail actions and all sorts of things. I’ve built this sort of thing in a couple of days.

      This ultimately comes down to Matt Hancock, and his inexperience in large scale management. Most people running multibillion pound enterprises have decades of experience. They maybe started as trolley boys or management trainees in the organisation and worked their way up. Probably cocked up and learned some things. They’re not against a bit of Excel, but not for critical systems. They had months and plenty of budget to set this all up. Hancock hasn’t managed shit. I’m not even saying he’s stupid, but experience really counts.

  2. Is it wise to run mission-critical potentially business-killing apps on a piece of software that costs thirty quid? Written by self-taught traders with no idea of validation and verification or any other software development practices? Probably not. Best leave it to IT professionals…..

    • In the case of traders, the value of Excel is how quickly you can build and change the models. And they often use IT professionals. I’ve seen Excel VBA jobs for “front office”. They hire developers to do things like write functions that their traders then use.

      Ultimately, they use it because it works. If something else was better, they’d use that. How many months would it take to build something professional in Java or C# for the model, and does that mean you lose the trade?

      • As a junior level C# developer, it probably took me 4 weeks in total to learn everything required to do that you require. Programming wise anyway.
        I work for a small banking consultancy, the actual building of solutions takes no time at all – if you have a proper data model to begin with. A lot of the work is basically just replacing tactical spreadsheet solutions with VBA scripts, because they no longer employ the person that wrote it and have no idea what the hell they are looking at.
        I’ve met Excel VBA from HMRC – it seems like they are all being forced to learn C# and SQL at the moment.

    • Is it good practice? No.
      Is it common practice? Unfortunately, yes.

      One way to reduce the risk is to offer training to staff who need to use Excel. But it’s a tough sell to spend that money, because the assumption is that everyone knows how to use Office.

  3. For a small model of a couple of hundred rows, Excel works just fine. For real data you need a real database, which will always be a flavour of SQL. Excel can be used as the front end to that database, though VBA is better, and any number of modern tools like Python better still. Long, long ago I used MS FoxPro to build a database of the six and a half billion possible bridge hands and it coped quite nicely. SQL queries by default use the latest version of the data and a query once written can be parameterised as needed. Click click and bob’s your auntie.

    Anyone who can do Excel can learn MS Access in a day.

  4. Some years ago I did some consultancy for PHE. They supplied the necessary data by way of an Excel spreadsheet of manay interlinked tables. Someone had obviously devoted many hours (weeks, months?) to to its development; and I spent many frustrating hours unpicking it to extract anything usable.
    For the past half year I have been looking at the Covid stats from ONS and PHE/NHS. The lack of competence and common sense in the use of Excel to bee seen in the PHE output is obvious: tables with half a dozen rows but a separate column for every day of the year!

LEAVE A REPLY

Please enter your comment!
Please enter your name here

expunct

in British English
expunct (ɪkˈspʌŋkt)
VERB (transitive)
1. to delete or erase; blot out; obliterate
2. to wipe out or destroy

Support Us

Recent posts

Aren’t We Glad We Didn’t Join The Euro?

It's possible to argue that this following is projecting a little too much onto just the monetary system of the remnant European Union. It's...

If Only The Media Understood The Economics Of The Media

Of course a newspaper will be all in favour of the antitrust dunning of Google. They see the search company as a competitor and...

The BBC Isn’t Grasping This Economics Stuff

True, the World Economic Forum isn't grasping this economics stuff either but that's no excuse. The BBC's remit is to explain to us proles...

This Just In From An Economics Professor

It's all terrible that we measure the economy by what is actually produced, consumed, in the economy. We must, of course, start to measure...

What’s Wrong With Modern Monetary Theory?

Richard Murphy, he of the three professorial positions, asks us what is wrong with his exposition of Modern Monetary Theory. First, in a country...

Recent comments