Sell-side Fixed Income connectivity reference implementation

Within the world of Fixed Income we are seeing a rapid proliferation of new venues that are attempting to bring about structural change in the Fixed Income market place.  How does a sell-side operate in this new paradigm of Fixed Income trading venue proliferation?

One impact is that the typical sell-side connectivity implementation faces structural difficulties.  In the past a sell-side would typically use a well known vendor for all of their connectivity or build in-house.  In either case the implementation would be expensive to run and not very flexible.
The days of a sell-side having a handful of connections are dying and time-to-market, ease of support, minimised capex and opex and high availability are driving change. On this blog we have covered a lot of ground in this field, for a refresher, have a look at the list of links at the end of this post.
We now see that Fixed Income shops are looking at their connectivity implementation and realising that while it worked when the number of connections was stable it is creaking under load and unable to adapt to the pace of change in this market. The answer is to build the adaptable FIX implementation.  The key characteristics of the adaptable FIX implementation are 
Designed to remove sources of procedural latency
Procedural latency is the often observed phenomenon whereby a system operates in such as way that there is a requirement to follow a particular procedure to effect change.  an easy way to describe this is with some examples of procedural latency:  System requires "stop and restart of *nix service/daemon" to effect a change.  This will require a *nix system administrator and require a change request/testing/release weekend scheduling and so on. Data has to be updated in several places and requires scripts to be executed against a database, then services stopped and restarted to pick up the new data.  Again - change request and further procedural latency.   
Designed to offer high availability
This is essential to allow for continual market presence - few things look worse that a firm offering quotes which disappear during service interruptions.  Buy-side firms will spot this and think it's either an intrusive risk management strategy or a technology failure.  Either way, not a good way to face the market 
Designed to allow data enrichment and enhancement
Platform A uses CUSIPs, platform B uses Sedols, platform C uses ISIN and so on.  All need to be handled and mapped to a common identifier to allow for correct pricing and accurate risk management.  This needs to be done in a manner that ensures that the bank has a single view of the risk book and the client flow book. 
Designed with diverse physical connectivity
This blog does not name and shame, so no names mentioned here.  But if a sell-side uses a single vendor of physical connectivity and receives sub-optimal service, that's to be expected.  Go buy a bunch of connectivity from a bunch of vendors and put them into competition for every new connection.  
Partnership with the FIX engine vendor
Don't just buy a FIX engine and then install it.  Partner up with the FIX engine vendor, meet the team from bottom to top of the organisation.  Negotiate a contract that is tough but fair. 
Implement high quality middleware
The FIX engine has to be connected to the bank systems, typically using middleware.  This has to be done in a manner that does not add in excessive round-trip latency, procedural latency or brittleness in implementation. 
Designed for an abstraction layer between FIX engine and core systems
Allow for message modification, enrichment, transformation from an abstraction layer.  Allow the abstraction layer to stream out data in real time on system performance and any issues, such that the abstraction layer can then be used as the point of interface to pricing engines, risk management, tick database and so on.  
Designed for automated regression testing
The build/test/implement cycle must be minimised in order to allow for rapid deployment of changes to the system.  The best way to do this is to allow the system to be tested in a manner that minimises human-in-the-loop testing and automates as much as possible.  Think of this as a further step in the DevOps world alongside source code control, continuous integration and other techniques. 
Designed for location independence
This is a more subtle point but any implementation should allow for the hardware and software to be relocated physically to and from a specific datacentre (internal or third party).  The implementation should also allow for diverse location - running in multiple locations around the world to cater for the globally distributed client base and sources of liquidity. 
Designed for manageability and supportability
Allow the system to either integrate with DevOps software or bake it into the solution.  Allow for system support staff to see what is going on from anywhere in the world using a mobile device. 
Designed to take advantages of hardware based processing
Per FIX sell-side reference implementation (not low latency) use technology such as full-fidelity packet capture rather than application logging, kernel bypass network interface cards, minimal implementations of OS rather than the full stack, 10/40/100GigE networking.
Design for modular replacement of components
Allow for every vendor supplied or open source component of the system to be replaced through the correct use of interfaces and abstraction design patterns.  In an ideal scenario every component should be a rip-and-replace job with automated regression testing to check it functions as required. There is of course a trade-off between using portable implementations and taking full benefit of the specific implementation in one tool. 
Designed for graceful degradation of service rather than failure
Again, a subtle point, but allow the system to fail with grace.  If a connection to a pricing source is lost - send alerts, widen published spreads and send all RFQ/RFS to a human for intervention.  It may be possible to trade with a human-in-the-loop rather than simply have the system auto-reject everything.   
Design for multiple protocols
Each Fixed Income venue has their own flavour of protocol - whether it is FIX 4.2 with custom field, FIX 4.4 with some 5.0 retrofit fields, a hybrid mish-mash of FIX 4.3, ITCH/OUCH or some unpleasant proprietary protocol.  All must be handled and managed and that's the job of the abstraction layer as mentioned above.  
Any sell-side looking at their Fixed Income connectivity would do well to look at the adaptable FIX implementation design pattern as a starting point for their systems renewal and enhancement. 

List of references