Big Data and the Rise of Machines in Financial Markets
— June 20, 2014
By Vasant Dhar
I exploited the slow stock market for several years through clever algorithms that used high-frequency data to identify short-term patterns in prices. The program had on average one losing day a month. Ironically, Reg NMS unceremoniously eliminated my advantage and conferred it on others who studied the regulation carefully, particularly the incentives it created for the different market participants and their behaviors manifested in the data.
The key to exploiting big data is to ask the right questions. In my recent article on data science and prediction,2 I illustrated this with a healthcare example, but finance is no different. The torrent of data emanating from financial markets—such as quotes, orders, trades, cancellations, prices, and messages—is enormous. Adding to this complexity is the fact that securities trade on multiple exchanges simultaneously, with each marketplace offering different incentives for orders being routed to them and regulation such as Reg NMS mandating that an order must be routed to the marketplace where it will receive the best prevailing price* (lowest for buys and highest for sells). When combined with news releases and economic data, the fire hose of information is overwhelming to all but the most analytically astute. A player with the ability to discover buried, but exploitable, patterns is at a huge advantage over the significant majority who are not well positioned to find them.
Read the full article as published in Big Data.
Vasant Dhar is Professor and Head of the Information Systems Group and Co-Director of Stern's Center for Business Analytics.
Follow @VasantDhar on Twitter