NSE TBT Market Data and High-Performance Software Stack: A Practical Insight

Author: Naveen Kumar

In the world of High-Frequency Trading (HFT), microseconds are the difference between a profitable trade and a missed opportunity. At the heart of India's trading ecosystem lies the NSE Tick-by-Tick (TBT) data feed a high-velocity stream of market information that demands a specialized software and hardware stack to harness effectively

To use TBT feed effectively, firms must build specialized low-latency software and hardware stacks capable of processing massive data volumes with deterministic speed.

1 Understanding NSE Tick-by-Tick (TBT) Data

Traditional market data (Level 1 or Level 2) provides snapshots or the "Top 5" best bids and offers. In contrast, NSE TBT is a low-level, sequence-perfect stream disseminated via 10G connectivity within the Exchange’s colocation (colo) facility. The TBT Engine also includes Order Flow Insights, a tool that helps traders see where their limit orders stand in the exchange’s order book. You can view the number of orders ahead of yours, along with the total quantity of shares waiting to be traded. This insight helps traders modify their orders in real-time, ensuring better execution, especially in fast-moving markets

NSE TBT provides a sequence-perfect event stream delivered through high-speed connectivity, commonly via exchange colocation infrastructure.

Rather than snapshots, TBT build every order book event in real time

TBT provides a granular view of the Limit Order Book (LOB). It doesn't just show the price; it broadcasts every single event:

• NewOrder: A participant places a new limit order.
• ModifyOrder: An existing order's price or quantity is changed.
• CancelOrder: An order is withdrawn from the book.
• Trade: A match occurs between a buyer and a seller.

This gives traders a full view of the Limit Order Book (LOB) and market microstructure.

2 Why TBT Matters for HFT

For HFT systems, granular market data creates opportunities such as:

• Order flow analysis
• Queue position modeling
• Short-term price prediction
• Spread capture strategies
• Arbitrage detection
• Liquidity sensing

But these advantages only exist if the data is processed faster than competitors.

3 Software Foundations of Low Latency

Processing millions of TBT messages per second requires moving away from "standard" programming paradigms toward ultra-low-latency engineering.

Traditional designs often fail because they prioritize convenience over latency consistency.

Key principles include:

• Minimal system calls
• Low memory allocation overhead
• CPU cache efficiency
• Predictable thread scheduling
• Lock-free concurrency
• Batch packet processing

4. Avoiding the Thread-Per-Channel Mistake

The NSE distributes sequenced TBT data across a multitude of multicast channels. A common architectural pitfall is spawning a dedicated receiving thread for every single channel. This approach leads to significant thread-scheduling overhead, increased context switching, and cache contention, all of which degrade performance during high-traffic bursts.

A more robust and scalable model utilizes non blocking I/O Multiplexing via epoll. By centralizing the monitoring of multiple channels into a single, high-performance network thread, you can manage the entire feed with deterministic timing.

5. The Architecture: epoll + Batch Processing

To achieve maximum throughput, the receiver should implement a pipeline that minimizes the interaction between the application and the Linux kernel. The optimal workflow involves:

Epoll Monitoring: Epoll allows a single thread to monitor multiple multicast sockets simultaneously. When any channel has data, epoll_wait() returns. Instead of just reading one packet, the thread enters a draining loop, pulling as much data as possible from that specific socket buffer before moving to the next.

Batch Retrieval via recvmmsg(): Instead of fetching one packet at a time, use recvmmsg() to drain a large batch of datagrams from the socket buffer in a single system call. This drastically reduces the overhead of crossing the user-kernel boundary.

Standard recv() or recvfrom() involves a context switch between User Space and Kernel Space for every single packet.

• The Problem: If you receive 1,000 packets, you perform 1,000 system calls.
• The Solution: recvmmsg() (receive multiple messages). This system call allows you to transfer a large batch of datagrams from the kernel to your application in a single system call.

6. Lock-Free Data Structures

Standard applications use mutexes for shared access.

In HFT, mutexes are dangerous because blocked threads sleep and wake unpredictably.

Better Option

Use lock-free queues based on atomic operations.

Examples:

• Single Producer Single Consumer (SPSC) queue
• Multi Producer Single Consumer queue

Benefits

• No blocking
• No scheduler waits
• Deterministic latency
• Better CPU efficiency

7. Optimizing the TCP Stack for Order Routing

Receiving market data is only half the problem. Orders must also be sent quickly back to the exchange.

Disable Nagle’s Algorithm

Nagle batches small TCP packets to save bandwidth.

For HFT, this can add hidden delays.

Use:

int one = 1; 
setsockopt(fd, IPPROTO_TCP, TCP_NODELAY, &one, sizeof(one)); 

This forces immediate packet transmission.

8. Kernel Bypass and Zero-Copy Networking

Elite trading systems often bypass the normal Linux kernel network stack.

Examples include:

• Solarflare Onload
• NVIDIA Mellanox VMA / accelerated networking
• FPGA-based NICs

Benefits

• Lower latency
• Lower jitter
• Reduced kernel overhead
• Direct NIC-to-user-space delivery

9. Eliminating Jitter

Latency matters, but variance in latency matters even more.

Jitter occurs when the OS interrupts your process.

Common Solutions

CPU Isolation

Reserve CPU cores using Linux boot parameters such as:

isolcpus=

CPU Affinity

Pin critical threads to dedicated cores.

NUMA Awareness
Keep memory and CPU locality aligned.

10. Throughput vs Latency

A serious HFT platform must excel in both.

Latency
Time to process one event.

Measured in:

• Nanoseconds
• Microseconds

Throughput
Ability to survive heavy bursts at market open or during volatile events.

Measured in:

• Messages per second
• Packets per second

11 Standard Trading vs HFT Stack

Conclusion

Building a production-grade system to process NSE TBT data is an exercise in relentless optimization. By combining kernel bypass hardware, lock-free software architectures, and efficient I/O multiplexing via epoll, developers can transform a chaotic stream of millions of orders into a clean, actionable competitive advantage.

In this domain, success is defined by deterministic performance, not just the ability to be fast, but the ability to be consistently fast under all market conditions.

For institutions with colocation presence, our TBT solutions and HFT consulting services integrate seamlessly into your existing trading infrastructure. Additionally, source code options are available for institutions seeking complete transparency into the software stack.