Polycom Real Presence Group series call drop issues.
In this article, I would like to share my recent video conferencing troubleshooting experience.
Last week, we spent some time in troubleshooting some strange Call drops issue.
Issue: Polycom Real Presence Group series call drops exactly at 45th minute.
Firmware version: 6.1.1
Reason: As per the Polycom device logs, TIMER Expired and call dropped.
Setup: In a Conference room, a Polycom Real
presence Group series device is used for Video conferencing. Last week we received
complaints from few users that while the users had call for around two hours
the call were dropped
twice. One more user complained that the call was around one hour and
then call dropped and had to connect again. However, other users who had meeting for did not complain about any issue. Later when we checked with the users, we found that the users who did not complained about the issue had a meeting
less than 40 minutes.
Model: Polycom Real presence Group 500 Series
Firmware version: 6.1.1
The call flow is as follows:
PSTN ßà AudioCodes SBC ßà SfB Mediation server ßà Polycom Group Series Device.
If you are in a hurry to fix the problem, please go the Solution section at the bottom of the Page :-)
In case, if you have more time then before we go to the solution, let us discuss few basics like the
- What is SIP Session refresh (using either RE-INVITE or UPDATE ) ?
- What are the well known reasons for call drops ?
- What is dead call and how we can detect it ?
- What is SIP Session refresh interval and how it is
useful in a SIP Stateful proxy ?
A SIP session when started would have a time frame (you might have seen the header
Session Expires:1800 and it can get refreshed by using
UPDATE or RE-INVITE for a session
(
Min-SE: 90). Note that, the Polycom device supports the
UPDATE method for a SIP session refresh and that is the reason the Call works for the first 44 minutes.
The Call drops can happen due to multiple reasons like
- Compatibility
with SIP Proxy.
- Remote
SIP endpoint.
- Network Firewall that has SIP
inspection enabled.
- Issue with the end point (could be a defective device)
- Issue with the Software running on that device.
So in this case, the call drop was
NOT due to a SIP Proxy, or a Firewall. Because it is not impacting all the users and other end points. Instead, the call drop issues are noticed only on one type of device (Polycom Real Presence Group Series 500). In addition to that, it was interesting to note that we were able to find the issue on two different Polycom devices (with same model and firmware version). So we can rule out the other options like SIP Proxy, Firewall SIP Inspection here or it is not due to a defective device because it is happening on multiple devices .
When a SIP Client fails to send a BYE message at the end of a session, or when the BYE message gets lost due to network problems, a Stateful SIP proxy server will not know when the session has ended. This is called as Dead call/session. And worse, it will maintain such dead sessions and if there are several such Dead sessions/calls, the Stateful proxy may not be even able to process the legitimate new requests. Because it still considers that the Dead Session are active and does not end the session unless it gets a BYE message. Also, from Security point of view, if we have don’t have a session limit then it would be used for Denial of Service (DOS) Attack. So, having the UPDATE method for a SIP session is very helpful.
Dead Call
detection:
There are two ways to detect the Dead call session.
- RTCP based detection: With this option, by listening and gathering the RTCP packet details for a session, we can detect if the call
was dead or not.
- Session Timers based detection: The Session
Timer based detection provide an alternate methodology for the detection of
dead calls using SIP Signaling. If the SIP peer fails to respond to an UPDATE
or INVITE message, this enhancement cleans up the call by sending a BYE which
will enable the call Stateful SIP proxies (Lync/SfB in this example) to clean
up the resources associated with the call.
Let us see what is a Session Interval ? Session Interval is the maximum amount of time that can occur between the session refresh requests
in a SIP dialog, before the SIP session will be considered timed out. This session interval is conveyed in the Session-Expires header field. So lets
us see about this Session interval in detail and why do we need this ?
So, how does Min-SE: and Session-Expires: headers help a STATEFUL SIP Proxy ?
A Client Initiates a SIP session by sending an INVITE. This INVITE includes a
Supported: header field with the option tag
'timer', indicating support for this extension. When this Client
INVITE
request passes through SIP proxies, any one of the SIP Proxies may
have an interest in establishing a session timer. A SIP Proxy in that path can insert a
Session-Expires header field and a
Min-SE header field into the request (
if none is already there) or
alter the value of existing
Session-Expires and
Min-SE header fields as described
below.
As per the RFC 4028,
The Min-SE: header
field establishes the lower bound
for the session refresh interval. The purpose of this header field is to
prevent hostile SIP Proxies from setting short refresh intervals so that their neighbours
(SIP Proxies or Clients or other SIP Servers) are overloaded.
Each SIP Proxy processing the request can raise this lower bound (increase the
period between refreshes) but is not allowed to lower it.
The Session-Expires:
header field establishes the upper bound for the session refresh interval;
i.e., the time period after processing a request for which any session-stateful
SIP proxy must retain its state for this session. Any SIP proxy servicing this request can
lower this value, but it is not allowed to decrease it below the value
specified in the Min-SE header field.
Now, you can understand how the session timer comes handy in maintaining only the active session.
By knowing these basics, now lets get back to the Polycom Group Series Call drop issue
So collected the Polycom Group Series 500 device log and went through it once again. Since we know that UPDATE SIP method is used to refresh the session. In the logs we see several UPDATE and a corresponding 200 OK for this call. However during the 44th minute a call refresh state is also success (that is, the UPDATE SIP Method and after that we see
a corresponding 200 OK for that UPDATE method). But the call fails with the message Timer Expired at the 45 th minute immediately after the 200 OK (for the UPDATE). So where is the problem ?
Looks like the Polycom Real Presence Group series does not
honor this 44th minute UPDATE, so it is not refreshing the session further (after the 45th minute). Instead, the Polycom device has its own
timer and hence it disconnects the call. I say this because in the Polycom Real Presence Group 500 series device logs we saw Timer expired -- Disconnecting call. So in this case, the root
cause of the problem is the Polycom Group Series accepts Update method and the Session refresh
request from its peer (the Mediation server component) till 44th minute. This happens for all the calls. So for all the calls that exceeds 45th minute, the call drops at the 45th
minute (and works fine for all calls within the 45 minute).
Solution:
Resolution for
Call drop exactly at the 45 th minute:
Raised a Support Case with the Polycom Support - Polycom
Support asked to update the firmware to version 6.1.4. It
should fix the call drop issue that occurs exactly at the 45th minute of each call.
After upgrading to the latest
firmware 6.1.4, we found that few
calls were lasting for couple of hours (one call for 2 hours, another for over
4 hours) without any problem. However, soon we ran into another issue that is random call drops even within 45 minutes (like within 35 minutes, the test call gets dropped).
Resolution for
random call drop (after upgrading to version 6.1.4):
Now after updating to the firmware 6.1.4 randomly some calls were disconnected even
within the
45 th minute (this time we had a call drop at 35 th minute itself). So, followed up with the Polycom Support again. The Polycom support escalated the case and confirmed that they are able to duplicate
the random call drop issues at their testing lab in spite of the Session refresh
interval using the UPDATE SIP method. So, Polycom Support acknowledged that it is
a bug and they are working on getting this issue fixed in their upcoming firmware update (6.2.0).
So, just in case if anyone experiences such call drop issue
exactly at the 45th minute (then check if you have firmware version 6.1.1 and try to update the Polycom firmware to 6.1.4) and it should reduce the call drops. However, if you will still have random call drops.
Check the Polycom Group series logs and if it is due to the Timer Expired issue then we need wait for any firmware
version updates (which will be released by end of February 2018).
Hope this post was informative. Please feel free to share
your suggestions or comments.