LOBSTER generates a 'message' and an 'orderbook' file for each active trading day of a selected ticker. The 'orderbook' file contains the evolution of the limit order book up to the requested number of levels. The ’message’ file contains indicators for the type of event causing an update of the limit order book in the requested price range. All events are timestamped to seconds after midnight, with decimal precision of at least milliseconds and up to nanoseconds depending on the requested period.
Both the 'message' and 'orderbook' files are provided in the .CSV format and can easily be read with any statistical software package.
Below the structure of the message and orderbook file are described in detail.
order book file.
The term 'level' refers to occupied price levels. The difference between two levels in the LOBSTER output is not necessarily the minimum tick size.
message and orderbook file.
The 'message' and 'orderbook' files can be viewed as matrices of size (Nx6) and (Nx(4xNumLevel)), respectively, where N is the number of events in the requested price range and NumLevel is the number of levels requested.
The k-th row in the 'message' file describes the limit order event causing the change in the limit order book from line k-1 to line k in the 'orderbook' file.
Consider the 'message' and 'orderbook' figures above. The limit order deletion (event type 3) in the second line of the 'message' file removes 100 shares from the ask side at price 118600. The change in the 'orderbook' file from line one to two corresponds to this removal of liquidity. The volume available at the best ask price of 118600 drops from 9484 to 9384 shares.
The initial period for which limit order books are available covers all trading days on NASDAQ from January 6th 2009 to the day before yesterday. In the future, we aim to increase the available period further into the past.
unoccupied price levels.
When the selected number of levels exceeds the number of levels available, the empty order book positions are filled with dummy information to guarantee a symmetric output. The extra bid and/or ask prices are set to -9999999999 and 9999999999, respectively. The corresponding volumes are set to 0.
When trading halts, a message of type '7' is written into the 'message' file. The corresponding price and trade direction are set to '-1' and all other properties are set to '0'. Should the resume of quoting be indicated by an additional message in NASDAQ's Historical TotalView-ITCH files, another message of type '7' with price '0' is added to the 'message' file. Again, the trade direction is set to '-1' and all other fields are set to '0'. When trading resumes a message of type '7' and price '1' (Trade direction '-1' and all other entries '0') is written to the 'message' file. For messages of type '7', the corresponding order book rows contain a duplication of the preceding order book state.
The figure below illustrates the trading halt messages as they appear in the message file.
To keep the output structure as clean as possible, the reason for the trading halt is not included in the output.
Beyond the usual limit order activities - submissions, cancellations, deletions and executions - NASDAQ's Historical TotalView-ITCH files contain other messages such as net order imbalances indicators. These messages are ignored and skipped by the limit order book reconstruction algorithm, since they do not affect the evolution of the limit order book.