After a recent backend revamp, we delivered 1.5+ million PDF e-mail contract notes in less than 25 minutes at the end of the trading day. We have been incrementally improving the performance of customer-facing features on Kite and Console as well, and we have more behind-the-scenes enhancements in the coming months.
You can follow all the product updates here: Updates – Z-Connect by Zerodha
What was the time taken to process these many notes before the revamp? Just curious on the tech improvements
Trading day ends at 5:00 PM right?
I received my contract note at 03:24 AM today
Certainly has gotten better than previous years.
I parse CN everyday before market open, earlier many times it wont be there. But for many months its always there.
While the currency market closes at 5 PM, brokers need the exchange files like trade, positions, STT & stamp duty, etc, to figure out the obligation and charges to carry out the trade process and generate contract notes & margin statements. We’ve been working on several optimisations and ways to reduce external dependency to carry out all the EOD processes, provide the required reports and analytics as quickly as possible to our users at our scale.
So this is still in implementation ?
This is live. Its one of the many optimisations being done in the back end.
then why did i receive my contract note early this morning?.
I should get it yesterday right?
As I said earlier, there could be a delay in receiving exchange files, without which ou our trade process. We’re working on reducing such external dependency.
Why does the broker need to exchange files like trades, positions, STT, stamp duty, and other charges every day? The exchange has already published all charges. As per Zerodha’s list of all charges: List of all fees, charges, and taxes on trading and investing – Zerodha
What is the average size of a contract note? ie. the pdf in KB/MB
What would be the largest size top 1% avg?
Does 25min include computation and creation of files(contract note) or is it just email delivery?
technical blog for the curious
@nithin im not sure who’s in dev team but I’d like to mention few points regards to further optimizations that can be achieved.
Write an asynchronous server based out of OpenSSL library that can take in as many signing requests and output back in async.
You can accelerate the entire signing process if you are using an HSM or a hardware acceleration. RSA 2048 takes 800ms on a typical i5, while RSA 4096 takes 5 seconds to sign. With hardware acceleration (possibly check for cloudHSM) you could improve throughput many folds. With EC keys there is much higher latency.
If no hardware acceleration , you could still use a thread pool and scale threads dynamically based on the incoming load. You could have dedicated queue for each thread. And balance incoming work on the queues may be in round robin or the size of your HTML package.
I hope you already doing CPU pinning of the tasks. If not, this helps you a little bit more with cache.
Delay your work of writing the pdfs in a drive. You’d keep them in RAM so long the emailer can pick them up directly from the RAM. You’d avoid disk writes and reads. If RAM is your limitation, try writing them to a fast drive (SSD) but in a dedicated thread or a task or an instance.
Try to code them above in either C++ or Rust. C++ is very much possible.
thanks. this blog is the way to explain, and it is quite detailed.
Satisfied to study it.
Good job Z!