Over the last few years I have had a few sales pitches from vendors in the automated FIX testing market place.
So – what would make an ideal FIX based testing system?
I have never been able to construct a viable business case to take up to senior management to make an investment in one of these products worthwhile. A few times I have been asked to explain this to vendors and I have taken the time to explain why no deal was made. After the passage of time it's worth explaining a few problems I have seen. Please note that all of the below comments are in no particular order and are sufficiently anonymised that no vendor should be able to spot their own product.
(1) More than one FIX testing package has been inconsistent in look and feel between different modules. Multiple executables that look like a mix of technologies and development stacks. Not a joined up approach.
(2) No true DSL for testing. FIX testing applications have tended to have a workflow model along the lines of: do your testing manually initially. At a later date the application then replays log files and has some fancy implementation of grep/awk/sed to update some tag=value pairs such as 42, 52, 122 and so on. Replay then requires essentially macro style code to be executed to handle anything. Classic example – roll a future test case – see how much nasty code you have to write to get from a March futures contract in 2012 to a June contract in 2013.
(3) Focus on a buy-side testing an internal workflow, using the application to act as a simulator of a sell-side. And/or the reverse of this...
(4) Focus on a sell-side testing an internal workflow, using the application to act as a simulator of a buy-side.
(5) Focus on a sell-side testing an internal workflow, using the application to act as a simulator of an exchange.
(6) Focus on allowing a sell-side to offer a service to buy-side clients to connect for initial testing without a human in the loop.
(7) No coverage for FIX ATDL
(8) No coverage for Foreign Exchange or Fixed Income workflow such as RFQ or RFS
(9) No coverage of Equity Program Trading workflow
(10)Buy the software, then spend a fortune implementing it and then upgrade it manually and painfully every time.
In the FX and Fixed Income world we have a different set of problems versus Equity business.
(1) FX and FI Execution venues are not (usually) exchanges. This is a massive change and the impact of this fact is one that many testing vendors don’t understand. An exchange will prepare a great big specification. For example, this link http://www.londonstockexchange.com/products-and-services/millennium-exchange/millennium-exchange-migration/mit202issuev11-1new.pdf gets you the London Stock Exchange FIX 5.0 specification. Eighty three pages. Certain FX venues produce specifications that are closer to eight pages than eighty. So there is not the level of clarity required to actually analyse the document and then, in advance of testing, prepare a complete set of test cases that can be automated. Testing in these cases requires an iterative approach with a human-in-the-loop to pick up the defects in the documentation. Which means that you needed a skilled manual tester or test team to start. So where's the value added by the testing software?
(2) FX and FI Execution venues will change their FIX interface with little or no notice. This is not some malicious action, it’s typically that they are not always aware of the workflows that their clients use. If you consider the case of a Fixed Income execution venue that is linked to say, a popular market data provider, the testing performed in conjunction with the venue is very basic. Any advanced testing is performed in isolation, buy-side or sell-side running two instances of the market data provider desktop application, one to act as a buy-side and one as a sell-side. So there is little empathy for valid use-cases by clients.
(3) FX and FI Execution venues that are moving to actually become “exchange-like” (SEF, OTF and similar) are often using equity trading platforms as the basis of an enhanced system. These systems are being modified and tweaked to fit a workflow. But in many cases I have heard of the documentation is not keeping up with the business or the code changes.
(a) Server based software – runs a web browser front-end. No client to deploy. Can run on Linux, Solaris, Windows.
(b) Run a web-based front end with an internet connection to a SaaS back-end system at a vendor
(c) Domain Specific language rather than something clunky and macro based.
(d) Full audit trail of who tested what, when and how from start to finish. So not just a yes/no but a rich source of data to see how long testing took, how many attempts were made and the elapsed time.
(e) Database backend to persist testing results.
(f) Integration with document management software to store Rules of Engagement documents with full versioning.
(g) Ideally integration with email so that related email traffic can be captured
(h) Integrated analysis of independent runs of tests. So we can see that Test Case 27 executed six months ago gets different results today. So we can spot added or removed fields and/or repeating groups.
(i) Intelligent replay engine – allow for futures to roll without writing a mountain of code, allow a set of cases to be replayed and a regression analysis performed.
(j) Load based testing capability. How many executions can a buy-side receive and process before it crashes. Allow distribution of execution dispatch to fit to a curve and/or set a mean and standard deviation. Or what happens if a large amount of executions are DK’d by a buy-side – what happens at the sell-side? Or if a sell-side amends a large number of executions?
(k) Load based testing meta-data. Analysis of bandwidth in use at message level and at TCP/IP level. So see impact when hitting an application with a microburst amount of data in excess of the bandwidth procured.
(l) Standard workflows for different asset classes shipped as part of the distribution
(m) Integration with packet capture technology and or tools such as wireshark to see what is going on in the layers below the FIX session.
This is just a sub-set of my observations – I could write War and Peace on this topic…
Comments
Post a Comment