The limit is applicable on overall OI of all derivative contracts of that underlying, this will include Calls, Puts as well as Futures.
For example: if Call option XYZ has an OI of 100,000, the Put option has OI of 50,000, and the Futures contract has OI of 200,000. The the total OI is 350,000, the 5% client-wise OI limit is applicable on this.
The client-level OI limit is calculated at the PAN level by Clearing Corporation. If the 5% OI limit is breached, the Clearing Corporation will notify the broker, and after that broker will notify the client asking to reduce the position. If the position isn’t reduced by the client then it is reduced by the RMS.