Big data methodology for the buy-side

Within an institutional asset management context there is a huge amount of data available to organisations.  We see repeated instances where firms do not have the capability to actually use that data in a meaningful way.

We therefore propose an easy to remember catchphrase to cover big data within Asset Management:
Ingest, Digest, Suggest

What does Ingest, Digest, Suggest mean?

There are three stages to big data usage within Asset Management:

1. Ingest.  Get the data from the relevant sources.  This may be direct exchange connections, Factset, ThomsonReuters, Bloomberg, InteractiveData, FactEntry, FIX connections, in-house platforms and third party platforms. This is the "plumbing" side of big-data and typically is not the glamorous part - most data scientists view this as being non-core.
2. Digest.  Once the data has been received from whatever source in whatever format we then generally need to ensure that the data can be persisted - this may involve cleaning, normalisation, de-duplication, conflation and persistence.  In simple terms, clean it up and store it somewhere.
3. Suggest.  When the data is available in usable format, now make use of it to suggest what to do and what not to do.

Observation: Hiring a Phd to assist with this process is generally a good idea for "Suggest". But stages "Ingest" and "Digest" require a different skillset. Furthermore, without domain knowledge the learning curve can be steep. 

Example 1: explanation of why persisting all quotes received on a Fixed Income RFQ is a good idea. Logical to anyone with a trading background, but considered noise in the signal/noise paradigm by some.


Example 2: explanation of why trying to capture market data into a standard implementation of a relational database is not such a good idea.  The "but it works in a test system" answer can be painful to hear.

Ingest can be hard, but we suggest digestion may be tricky in this case...