Introduction

In our experience, each client is different, context is always king. However, because many financial institutions start off with end-user programmers, people who program for themselves, usually in Excel with VBA, the organisations tend to follow similar growth paths. In the end, financial institutions end up with a consolidated engineering team, what is often called the ‘Engineering Hypermarket’. The hypermarket is a place where the financial institution can ‘shop’ for:

• A bug fix.
• A new pricer.
• A desk quant who can sit in the trading room assisting with features, analysis or coding.

The hypermarket is characterised by high levels of automation, discipline and rapidity; they do things quickly. Building a hypermarket is a bootstrapping process that starts with the smoke test.

Smoke Test

In electronic engineering, the first test for a new piece of equipment, for example a television, was to turn it on. If smoke started coming out the back, then the test failed! This endearing name made its way over to financial engineering sometime in the 1990s. A product, for example a spreadsheet, is said to be smoking if:

• It doesn’t compile or link to its libraries.
• The sheet does not reproduce the values of an older version.
• The sheet crashes at run time.

The smoke test becomes part of the rhythm of an engineering team. It can be done once a month, once a week or once a day. In mature teams, the tests are fully automated and executed every time a sheet or library changes.

Prerequisites for Smoke Test

A smoke test is a collection of tests that are executed together. For it to make any sense, a number of tests should be available. Therefore, if a financial engineer is using VBA, then he should provide a unit and functional test in either Excel or a tool like VBAUnit. If the engineering is using Excel, they should provide a test in Excel. And, if the engineer is using C#, then they should provide a test written in C#. For these reasons, when a team decides to introduce smoke testing, they usually have to spend some time learning about unit testing. This can be done with lunch time sessions, once a week is enough, or with workshops. The creation of a smoke test is incremental, it also follows a bootstrapping process; you start out with a simple test, then you learn better testing techniques, and then you write better tests. As time goes on, the tests get better and better, the operational risk associated with sheets gets less and less.

Once there are some tests available, they need to be executed. A test runner is what does the execution. In the first instance, the test runner can be a person. This is because the process of building, testing and reporting results is not easily automated. Once a week, for example, the ‘smoke tester’ takes the sheets whose content has changed and runs all the tests against them. If they are correct, he reports his results to interested engineers and customers and the sheet can be safely released.

Figure 1 - a smoke test is the combination of tests and a test runner. In the above example, three tests are used - FitNesse, NUnit and a tailor made test written in VBA. Fred interprets the results and reports any issues to the other engineers. If Fred is satisfied, the smoke test is passed.

Benefits of the Smoke Test

The smoke test has a number of key benefits:

• Increased quality.
• Defect diagnosis.
• Realising risks early.
• Regulatory.

In addition, the smoke test is fantastic for morale. It forms the ‘heart beat’ of a project, helps management track progress, and via the tests the engineers write, becomes a source of pride.

The Square Law of Spreadsheets & Next Steps

One thing financial companies often end up asking themselves is, ‘how did things get like this’. There is a misunderstood dynamic in spreadsheet and tool development. Consider the following diagram:

Figure 2 - The Square Law of Spreadsheets Effects Diagram.

There is, roughly, a set number of defects per spreadsheet. Therefore, for each new sheet, a number of faults are introduced. This creates labour to locate (and remove) faults. However, for each new sheet, there are extra places to look. In this dynamic, labour and risk are considered the same thing. This is known as the ‘square law of spreadsheets’ and it leads to this:

The table shows that 10 sheets or tools are manageable for a financial engineering team. However, as the number of sheets or tools tends towards 100, the potential amount of work needed to maintain them is an order of magnitude higher. This is how some financial engineering teams ‘grind to a halt’; at one point, the maintenance work simply explodes. Often, a ‘capacity problem’ is blamed. However, it’s often a case of the engineering team relying on manpower instead of machine power to do a lot of the work. The smoke test is the beginning of moving over to machine power, we call this the Matrix Principle - never send a human to do a machine’s job. It’s also the beginning of dealing with the square law of spreadsheets, therefore, this technical practice has a direct influence on operational risk.

Other Factors: Requirements and Equation Reuse

The smoke test helps reduce the manpower to locate faults by being partially automated and by focussing the team on writing basic tests. However, there are two more things that can reduce the operational risk/labour:

1. Reduce the number of sheets.
2. Reduce the amount of code in each sheet.

By strictly managing the creation of new sheets, remember for 10 new sheets there is 100 work units of maintenance - the total amount of work can be limited. With the same logic, retiring older sheets is also a way to reduce the number of sheets. By removing duplicated equations, for example by using VBA instead of copying and posting equations, the amount of labour can also be reduced.

Conclusion

Financial institutions often start off from similar beginnings, with spreadsheets and the odd bit of Visual Basic. Before long, these institutions fall prey to the ‘square law of spreadsheets’ and find that their operations are seriously at risk. In order to get the process managed, to create an ‘engineering hypermarket’, financial companies start to adopt practices from more traditional engineering. In our experiences, a regular build and smoke test is the most cost-effective and effective way to start to fight operational risks that the square law of spreadsheets creates. When coupled with lightweight requirements management, a little bit of code reuse and a little bit of education, in the form of workshops and/or coaching, great gains can be made. The smoke test is a tangible first step towards the engineering hypermarket, towards dealing with operational risks.

Further Reading

Crosby, P. B. (1980) Quality is Free, Signet.Humble J., Farley, D. (2010) Continuous Delivery, Pearson Education.

Feathers, M. (2004) Working Effectively with Legacy Code, Prentice-Hall.

Matyas, S., Voit, M., Schneider N. (2007) Continuous Integration, Addison-Wesley ProfessionalMcConnel, S. (1996) Rapid Development: Taming Wild Software Schedules. Microsoft.

Weinberg, G. M. (1992) Quality Software Management, Volume 1: Systems Thinking, Dorset House.