We therefore propose an easy to remember catchphrase to cover big data within Asset Management:
Ingest, Digest, Suggest
What does Ingest, Digest, Suggest mean?
There are three stages to big data usage within Asset Management:
- Ingest. Get the data from the relevant sources. This may be direct exchange connections, Factset, ThomsonReuters, Bloomberg, InteractiveData, FactEntry, FIX connections, in-house platforms and third party platforms. This is the "plumbing" side of big-data and typically is not the glamorous part - most data scientists view this as being non-core.
- Digest. Once the data has been received from whatever source in whatever format we then generally need to ensure that the data can be persisted - this may involve cleaning, normalisation, de-duplication, conflation and persistence. In simple terms, clean it up and store it somewhere.
- Suggest. When the data is available in usable format, now make use of it to suggest what to do and what not to do.
Example 1: explanation of why persisting all quotes received on a Fixed Income RFQ is a good idea. Logical to anyone with a trading background, but considered noise in the signal/noise paradigm by some.
Example 2: explanation of why trying to capture market data into a standard implementation of a relational database is not such a good idea. The "but it works in a test system" answer can be painful to hear.
|Ingest can be hard, but we suggest digestion may be tricky in this case...|