We have been facing a weird issue lately for our GRPC service. During some time period, we're seeing significants numbers of timeouts on our HAProxy dashboard, however when i checked the api response times from our application dashboard, there are no threshold breaches. Our clients are experiencing the timeouts and threshold breaches. All the timeouts are distributed on all our servers, so it means it's not some server specific issue. It also gets fixed after some time, and then we start seeing these during the same time period again. During that period, there is no increasing in load, in-fact the load is way too less on our service compared to other times.

There is nothing in error-logs or degradation of any other metric on application side as GC pauses, thread allocation, java heap etc... Not sure how to approach this issue now.

As from the hap side, the load is perfectly distributed on all the servers. For the take time for backend in hap there are spikes very less in number though present, but most of the time for all hap servers it's not exceeding 30 ms.

Would appreciate if someone can share any fresh idea on how to debug this issue further.