Zerodha's sensex expiry issue in June and July 2023

[YS_Bhargav]

Saw this tweet regarding the issue that Zerodha and its clients faced in BSE options for an expiry in June and July.

https://x.com/BandiShreyas/status/1712370213696782756?s=20

I have some questions to @nithin and @Zerodha-Staff

• Is the tweet and the order entirely true or are we missing something?

• It says the BSE informed Zerodha that the Bandwidth had crossed 70% on June 30th and inspite of being aware of the bandwidth issue, Zerodha failed to upgrade its bandwidth causing much severe issue on 7th July.

• Bandwidth of 4 Mbps is very slow considering the volumes and speed that Zerodha would require for orders to go through - I remember reading that it takes much lesser than mili seconds for orders to go through and trades to execute. What is the general industry standard regarding this and can’t you and other brokers upgrade this to the maximum possible speed?

[VenuMadhav]

Hey Bhargav,

It’s hard to deduce & easier to make assumptions without having complete knowledge of the incident that has occurred and the subsequent follow ups made. The order that has been shared is a GRC order for an issue that occurred on 07th July 2023. Let me attempt to explain so there’s clarity for everyone to understand the situation correctly.

It’s true that we did have an issue on the 30th of June for BFO orders. The issue lasted for 14 minutes between 3:08 and 3:22 pm affecting a smaller set of users. Users were unable to place, modify or cancel their orders in this 14 minute period after which things resumed to normal.

On the same day evening, when we reached out to our technology vendor, Refinitiv, we were informed that this disconnection happened due to a network connectivity issue in the line connected between the Stock Exchange and our OMS/RMS systems. There are P2P leased lines connected between the Stock Exchange & the broker OMS systems and these lines were flapping on account of which the issue had occurred. Despite this, on the same day, we reached out to the Stock exchange and asked them to increase the bandwidth capacity on all our lines to eliminate any possibility of bandwidth-related disruptions leading to the primary disconnection, which eventually happened only on July 12, 2023.

I do not want to sound confrontational or indulge in mud slinging, but were we informed by the MII that there was a bandwidth issue on 30th June? The simple answer is ‘no’. Matter of fact, till date, there isn’t a system for members to proactively track bandwidth utilization during the day. It was only on July 21, 2023, 14 days after the bigger issue of 07th July, that the stock exchange issued a circular that they would be sending reports via email to trading members notifying them of the bandwidth utilization.

My colleague, who is also the compliance officer at Zerodha, was a party to the meeting representing Zerodha in the said matter. In the presence of the panel member, no such claim of having informed Zerodha of the bandwidth utilization was made by any officials of the Stock Exchange. Adding the line “As per exchange officials’ alerts were sent to them on 70% utilization, still no corrective actions were taken by the Trading Member” in the GRC order was a blatant mistake which we’d missed to spot, which we’ve now brought to the Stock Exchange’s notice for them to act upon. If we’re able to obtain a copy of the corrected order, we’ll make it a point to share it here concealing all other client details, if any.

Trading on the BFO segment has picked up in the recent past. While on the face of it, 4MBPS seems less in today’s world, we’ll need to understand the nuances between a P2P line and a broadband line. A private leased line to an exchange (of which several are pulled by brokers to the same exchange) cannot be compared with a home broadband line in terms of “Mbps”. These leased lines are very contextual and only transmit compressed exchange protocol messages, which typically only come up to a couple of hundred BYTES. Typically at any given moment, only a small percentage of such a dedicated leased line is utilized.

To substantiate, here’s a link to a NSE circular: https://nsearchives.nseindia.com/content/circulars/MSD55333.pdf on market data. Although this only talks about broadcast and not interactive (the one used to send orders and receive confirmation), everything that is broadcast by the Stock Exchange (feeds, 5 depths, market open/close status) only consumes 600 Kbps in the Cash market segment.

Further, on 26th July, 2023, when we finally received our Bandwidth utilization report for 30th June, none of the reports indicated any excess utilization. I’ve attached the Bandwidth graphs from one of the silos here indicating the utilization:

As is seen in the above charts, the bandwidth utilization was lower than 2MBPS. This means that the issue of 30th June, cannot be attributed to bandwidth utilization as has wrongly been claimed by the Stock exchange. The fact that we were running a 4MBPS leased line is not a “discovery” or a “revelation” through the GRC order. We’re among the only brokers today that has a dedicated section on its website where a root cause summary of all technical glitches are shared with the public for transparency. Even for the issue of 30th June, the RCA that was put up on the website, the report did indicate that we were running a 4MBPS Leased line and had requested for an upgrade. Also, it’s not true that Zerodha hasn’t refunded losses, wherever the claims are legitimate, the team has been given a mandate to assess and refund losses suitably.

[Saurabh_Agrawal]

Technically I dont see any issue with 4mbps lease line. I have worked in this industry. The outage issue they are highlighting must be mostly due to server issue at either Zerodha or Exchange side. Or if they are using any intermediary then over there.

[Randhir]

Dear @VenuMadhav ,
Appreciate you taking time out to clarify things. On the outset I would say, honestly admitting to mistakes goes a long way in building trust. So thanks for publishing those RCAs that you do.

I am in Software Engineering profession and it’s a standard procedure in production systems to alert on >70% capacity utilization - not just for network, but CPU, Memory, storage etc. Even if BSE didn’t alert, shouldn’t your own system monitoring alert? You are saying you never went above 2Mbps utilization; but what’s in the charts above that indicates these charts are of the private leased line to BSE? Could be of any other line. I appreciate all you folks @zerodha for what and how you have built. And I am sorry for sounding suspicious in this instance. I have some reasons to do so:

1. It’s hard to believe that BSE inserted that observation regarding 70% capacity utilization alert, out of thin air. We will believe it when we see it, please do share once you get the updated judgement. If you don’t, we would have to assume that there was a reason for this claim.

2. Secondly, you say these leased lines between OMS and BSE were flapping, which caused this issue. Agreed, I have personally faced this in production, and it may happen sometimes. But then, why on the very first instance you asked for bandwidth increase on the 30th June, if you were running much below 50% capacity utilization? And on both days, it was network disconnections, that too during peak load times??

3. Fact that you agreed to refund 100% of the loss amount in this case, also indicates some culpability on your end. It’s true that you refund losses at times, but never more than 50%, unless forced by an institution.

Will be happy (and frankly, relieved) to see a point by point rebuttal from your end.

Regards,
Randhir

[VenuMadhav]

@Randhir

Even if BSE didn’t alert, shouldn’t your own system monitoring alert?

The leased line I’m talking about is the P2P line between the stock exchange and our infra setup commissioned by the stock exchange. We had 0 visibility on bw utilisation of this P2P until recently that trading members started receiving alerts. Otherwise, all internal systems are adequately provisioned for with suitable alert generating systems when thresholds are breached.

@Randhir

You are saying you never went above 2Mbps utilization; but what’s in the charts above that indicates these charts are of the private leased line to BSE?

This was supposed to be a private communication, I didn’t want to share it but since it got to this, I had to. Here’s a SS of the email received, is that proof enough ?

@Randhir

please do share once you get the updated judgement

We have asked for it, should we received an amended copy, we will share.

@Randhir

why on the very first instance you asked for bandwidth increase on the 30th June, if you were running much below 50% capacity utilization?

Quoting from my above response. Since traction was picking up on derivative trading on the exchange, this was done proactively

@VenuMadhav

to eliminate any possibility of bandwidth-related disruptions leading to the primary disconnection, which eventually happened only on July 12, 2023.

@Randhir

Fact that you agreed to refund 100% of the loss amount in this case, also indicates some culpability on your end. It’s true that you refund losses at times, but never more than 50%, unless forced by an institution.

IMO, this specific complaint shouldn’t have been referred to the GRC, the team should have refunded the user. Each case is different, we’ll need to assess and figure if there’s merit in the claim and then take suitable action. A number of users got affected on 07th July, only a small number were referred to GRC, that’s because most were resolved amicably.

[Krishnakumar]

@VenuMadhav

specific complaint shouldn’t have been referred to the GRC, the team should have refunded the user.

Those who didn’t pursue with exchange didn’t get full refund either. Myself is one among them. My case was no different to this
Does this imply that in future (i wish it doesn’t happen) if it happens, people should never accept what zerodha’s team tries to negotiate? because you have now set an example which shows those who escalate will give full refunds and those who don’t, will have to settle for lesser even though their claims are genuine

You refunding one person fully and not refunding the other even after you know that a case deserves 100% refund is unethical business practice. You knew that some of your clients are entitled for a full refund and yet you forced them to settle for lesser. This is really bad

Note: I’m not trying to re open my ticket using this new update in any way, I’m here for constructive discussion

[VenuMadhav]

Unless a client approaches, there’s no way for a broker to know that a loss is made. Assume a situation where a client has bought a contract at 100, attempted to sell it at 145 (price was available), but couldn’t due to tech issues and had to subsequently close position at 104.

What is the loss of the investor here? The investor thinks it’s 41, the broker could argue there’s no loss and the client has made a profit of 4. This is why, it’s always important to reach out to your broker, be it anyone, in case you’re aggrieved. The broker refunding or not refunding is secondary, but as an investor you ought to take that route.

About settling, all brokers (some may not say publicly), will attempt to negotiate with the client regardless. Remember, enabling trading involves a complex system dependent on multiple players - exchanges, clearing corps, depositories, leased line providers, data centers etc and some of the issues are completely beyond the control of the broker. I don’t want to delve further, but the discretion to settle or not settle is left to the investor. No broker can “force” a settlement on the client. I think the market regulator has played a vital role in shaping a robust system that ensures fairness and transparency in the Indian financial markets, so investors should make use in case they feel their rights are violated.

[Krishnakumar]

Please read the reply assuming a calm tone. My language used may accidentally sound harsh (unintentionally), but I’m a happy customer of zerodha and a wellwisher

The main point as per the GRC was that if price was available favourable to the user to exit for sufficient time, the loss calculations are taken as per what the client gave.

Ideally I agree with what the GRC body says. It makes complete sense in that matter. Can’t say because you made profit in the strike at the close, you didn’t lose money. Because that leg might be part of other strategies put together (those other legs might be winning or losing is another part of the story, we will not get into that).

@VenuMadhav

there’s no way for a broker to know that a loss is made.

When you have the server/ terminal’s logs from your end and when the client is giving you all the calculations and you feel it is legitimate, ideally you should have refunded without much hassle ?
Sorry to say this didn’t happen. The team called 10s of times to the point that I got frustrated and accepted their negotiation because I could take it no longer.
( If it was done on purpose, I don’t have any suggestions. Otherwise, please take it as a feedback on your client relations )
This is what I meant by “Forcing”

@VenuMadhav

I think the market regulator has played a vital role in shaping a robust system that ensures fairness and transparency in the Indian financial markets, so investors should make use in case they feel their rights are violated.

I think this is the major takeaway. Don’t accept anything other than 100% refund from the broker if you feel your case is genuine. Keep escalating to the highest level

Personally, I feel that the next disaster if any might hit hard on zerodha (be it any broker) because now people are never going to trust your first level of contact (customer care) and every one would want to escalate to the end, which will ultimately cost you resources

The only reason why I’m spending this much time writing is because you people directly from the board level are interacting with the customers which is not the usual case with some other broker, and I feel its important for you to know how the end user felt after being affected from the glitch

[Randhir]

Hello @VenuMadhav ,

Appreciate your response. And being transparent enough to share an internal email screenshot. Means a thing.

I trust you - it was not a bandwidth exhaustion case.

@VenuMadhav

Quoting from my above response. Since traction was picking up on derivative trading on the exchange, this was done proactively

Appreciate that. That’s a sign of a mature tech team.

Now the question is - what are the odds of 2 occurrences of connections flaking for an extended period - on an expiry day itself, that too during peak load times? Connection problems can occur, but it would be on random days, which is not the case here.

Connection failures during peak load times indicate some sort of resource exhaustion on the server. This could manifest as an internal server error e.g HTTP 503, in case of Web. Since it was not bandwidth, it could be any other server resource - viz CPU, I/O ports/threads, RAM, storage etc. If this server setup is not in your control - I would suggest you ask your tech team to work with the server infra team to dig in and RCA that. If you have not done so already, that is.

Because if you don’t - merely upgrading the leased line bandwidth may not help and you may end up getting into this situation again.

@VenuMadhav

Otherwise, all internal systems are adequately provisioned for with suitable alert generating systems when thresholds are breached.

I am sure they are, given how stable your system is, almost all the time. That was the reason I was wondering, as to why you couldn’t detect bandwidth exhaustion, even if BSE didn’t alert.

Regards!

[whity]

When it comes to load testing, how do exchanges or brokers make sure, they are capable to handle the customer trading order load, especially in expiry days? Plus, given we have more daily expiries, what are the performance testing and monitoring that are followed in Zerodha and how can we as customers be stress free, that this will not happen with NSE also in the future?

BSE saying 4Mbps line was 70% overloaded and here you are saying it was not the case. It was the case of flapping issue. You have not mentioned what was the reason for flapping here or any fix applied for the same to avoid this issue in the future. Does increasing the bandwidth solve the flapping issue? From your statement, I get the feeling it will not.

Waiting for your kind response.

[VenuMadhav]

@whity

how do exchanges or brokers make sure, they are capable to handle the customer trading order load,

You can read point 4.2 with point 6.1 of this SEBI circular that requires brokers to be adequately provisioned, with stock exchanges tracking the load through the LAMA system.

@whity

Does increasing the bandwidth solve the flapping issue?

No, flapping as I understand is intermittent disconnection in the line for any reason. Can’t be solved with increasing bandwidth.

[sureshmjss]

Hi, on 07/07/2023 zerodha had a technical glitch and you had 14.5k affected customers. You have received 3411 complaints(2672 calls and 739 tickets across all segments) There are many articles online available stating you have refunded couple of customers. May I know out of 3411 complaints for how many customers you have refunded. I raised a ticket in zerodha but they did not answer and said it is internal information. I believe as a trader I have a right to know how many are refunded.