Calls made after long idle hours are failing 

Recently, we had customer reporting about a call failure issues. Since the scenario and the root cause of the issue was different I would like to share that experience here.

Issue Description: 

Every morning, whenever users  try to place calls from their desk phones to the PSTN,  the call rings. However, the call gets disconnected as soon as  the call gets accepted/answered.

Technical description: In this scenario, for  a SIP based VoIP Call flow, the SIP Signaling works and the phone rings fine. However, when the callee/called person answers the call, it gets disconnected.
 Hence in this issue, the SIP signaling works fine but  the Media path always fails.

Lync Platform: Lync 2013 MT platform.

Desk Phone - Yealink

Recent change: Firmware update. At the local site, the firmware was upgraded on the Yealink phones. But after the firmware update all the test cases were  successful.

Phone Firmware versions: 

Firmware version without the issue:   66.9.0.25

New firmware version (that caused the issue) :   66.9.0.42

Workaround (when you are using the firmware 66.9.0.42)

• Reboot the phone after several hours of idle time. Then the issues does not occur. (OR)
• Downgrade to another version in our case it was 66.9.0.25.

Recent Changes: 

Scenario: 

The customer was migrated from existing PBX to Lync 2013 MT several months ago. The users are using Yealinks Desk Phone to make calls. The phones were working fine for the last few weeks. However, users reported that they are always not able to make calls in the morning.  Once they reboot the phone then the calls are working fine throughout the day. However, the next morning again we have the same issue and it gets resolved once the phones are rebooted.


Troubleshooting:

• Confirmed that the port 3478 for the STUN (UDP) was allowed in the Firewall.
• Confirmed that Lync/SfB server was listening on the port 3478 for new sessions and there were no server related issues.
• There were no connectivity issues between the phones and the Lync/ SfB (Skype for Business) servers.

Issue: After troubleshooting the issue with the customer, it was obvious that the issue occurred only on the phones which had the latest firmware 66.9.0.42.


Network trace collection and Analysis:

In order to collect the Wireshark trace,  I connected to the Yealink phone using its IP Address and collected the Wireshark trace for the not-working and the working scenario.

The following are the snapshots of the network trace collected while the calls were failing (after several hours of the idle time). First let us see network trace from the phone on a morning when the calls are failing. From the  snapshots (of the Wireshark trace of the failure scenario) - we could see that there were several STUN binding requests but no successful response. Moreover there were  several strange errors for STUN binding requests like.


1. Allocate Error Response error-code: 401 (unauthorized) the request did not contain a Message-Integrity attribute”.   

 And sometimes the STUN binding requests failed with the other STUN binding errors like,

2. “Allocate Error Response Code : 436” – The username supplied in the request is not known.“

So, it is evident that, the issue was due to some STUN Binding requests and the lack of successful STUN responses. 

While searching in the internet based on STUN error message ( that we got from the Wireshark trace),  understood that the issue was due to the STUN response or the ICE keep alive related issues.

So, tried to collected the Wire Shark trace for a working scenario. Hence, rebooted the phone and then collected the Wire Shark trace (when the calls were working fine after the phone reboot).

When the phones were rebooted, found that the phones were connected to the same Lync server but the phones started working (after the reboot). Hence, this implies that there were no problems with the Phones and the Lync server. This is because after the rebooting the phone, it was sending out a new STUN biding requests and receiving a  STUN Allocate Success  response immediately, for the Media flow.

 Hence, contacted the Yealink support and provided the Wireshark trace for the working and not-working scenarios.

The Yealink support checked wire shark traces I provided. They also confirmed the issue after performing the tests at their end. So, their Yealink Product development team worked on a hot-fix and   provided us a hot-fix in couple of days and it fixed our problem.

Root cause of the issue:  As per the Yealink support, the root cause of the issue "the phones don’t update the STUN user information in time, new firmware hot-fix would let the phones update the STUN information every 10 minutes."

A quick word about Yealink support in this case:

I must admit that since I have worked with several other UC Phone vendors, I can tell you that Yealink support was great in this case. Because, earlier when I faced similar firmware issues with other Microsoft UC vendors, my experience with their support was really time consuming and bad.

After all, the other premier UC vendor for Microsoft was in total denial mode for months rather than accepting about the issues with their firmware. During those instances, not only I need to wait for several months for them to fix the issue. But also, the vendor would take several months to even acknowledge the issue on their product. On the other hand, the Yealink support was very quick on confirming the issue and providing us the hot-fix immediately in order to fix this issue. So, a big thank  you to the Yealink Support  :-)

Lessons learnt:

From this issue we understood that we have one more scenario that needs to be tested after a firmware update. The take away from this experience is, you need to test the call flow after long idle hours as well (at least after a time frame of 12 hours since the desk phones was rebooted).

                          SIP Back-to- Back User Agent Role  [ Signaling B2BUA Role ]

Recently, while I was working on SIP Call flow issue we had to work a custom built application. Unfortunately, there were no documentation  or diagrams available for us in order to understand that application. Later while discussing about the application, we got to know that it was a custom application designed for a specific purpose like masking the Caller's Identity (before leaving the network), identifying and tearing down idle sessions etc for security reasons.


From the SIP logs that we collected from that application, we realized that the SIP Call flow looked different than the predominant SIP server roles (like SIP Registrar, SIP Proxy or SIP Redirect server  Roles) which we were well aware.  Moreover, it was not an SBC connecting to the PSTN network either.Instead it was a SIP B2BUA. Since I have worked mostly on more on Microsoft UC products the only B2BUA that I was aware was Mediation server Role in Microsoft Lync or SfB infrastructure.
What is a Mediation server ? For readers who are new to Microsoft UC platform, the Mediation Server is considered the last point of contact for the Lync/SfB environment before communicating to the telephony world for audio communication, whether its is a inbound  or outbound VoIP calls from/to the Public Switched Telephone Network (PSTN) network.  But later while learning about the B2BUA understood that there are several categories with in B2BUA.

So, started searching and reading about B2BUA in the Internet and i would like to share some of the information which i gathered while trying to understand the functionalities of a SIP B2BUA.


Back to Back User Agent:

What is a SIP B2BUA Role ?

Back to Back User Agent (B2BUA) is the logical combination of a UAS and UAC.

UAS :    User  Agent Server.

UAC :    User Agent Client.

In SIP deployments, there are several Back to Back User Agents (B2BUA). So, it is very important to understand the different types and what a B2BUA Role can do or Cannot do in a SIP infrastructure.  

Note, the Back to Back User Agents are further classified in several types. Again, it is a SIP server Role not a single system. That is, a system or a server can perform all of the B2BUA roles mentioned below in one server and not necessarily each Back to Back UA Role should run on a separate server.

 The B2BUA is broadly classified into two categories:

• Signaling Plane B2BUA
• Signaling + Media Plane B2BUA

The SIP B2BUA Role is in itself a vast topic. So in this post, we will discuss about the Signaling Plane B2BUA and discuss about the Signaling + Media Plane B2BUA  in a different post.

what is a Signaling Plane B2BUA ?

A  Signaling Plane B2BUA as it name implies it ONLY operates on the SIP Messages and SIP Headers.  and  NOT on the Media.

Again there are several classification within the Signaling plane B2BUA like

    1)  Proxy-B2BUA 

    2)  Signaling Only B2BUA 

3  3)  SDP Modifying Only Signaling 

     1)  Proxy B2BUA: (REPLACES only the VIA: and Record-Route: SIP Headers)

The Proxy B2BUA maintains the Sufficient SIP Dialog state in order to (or if required to) generate the In-Dialog SIP messages on its own. If the Proxy B2BUA can generate In-Dialog SIP messages then it can also MODIFY the CSEQ: header after it has generated its own.

Example of this B2BUA is, sending the BYE requests in order to tear down a dead SIP session.

so what are all the SIP headers that a SIP Proxy B2BUA can modify ?

The Proxy B2BUA role can only modify the Via: and Record-Route: SIP header fields.

What SIP headers a SIP Proxy B2BUA cannot modify ?

The Proxy B2BUA role does not modify the TO: ,  FROM:  , Contact:  SIP headers. 

2)  Signaling – only B2BUA: (Replaces all the SIP Headers)

A Signaling Only B2BUA is the one, that operates at the SIP layer but in ways beyond those of the SIP Proxies.

That is, the Signaling -Only B2BUA can  replace the Contact URI  along with modifying or removing the Via and Record-Route headers.

Also, in this Signaling Only B2BUA role - No SIP headers are guaranteed to be copied from the Received SIP request messages from the UAS and generated on the UAC side.

So if you want to completely create a new call leg between two different System or Networks, then you need to have Signaling-only B2BUA. (The Mediation servers in the Lync/SfB infrastrucure)

Example:

Like a Application Server or a PBX which actually Processes the REFER methods locally and then generates a new INVITE on Behalf of the REFER’s target.

Another example is a  Privacy Service Proxy performing the ‘Header’ Privacy function.

This kind of  B2BUA,  a Singaling only B2BUA is useful if you want to hide the caller's identity before it leaves your SIP infrastructure. Or may be for billing purposes if you want to convert all the Call transfer (like REFER) to a new INVITE session. Then, you can have a Signaling only B2BUA Server sit in the Perimeter of a network and makes sure that any call transfers that made within the network to any outside network should be treated as a new Call. Thus, the Billing server would only consider the INVITE sessions generated with unique Call-ID and charge the calls transferred outside of a system as a new Call. So, may be then you might need this kind of feature.

3) SDP modifying Signaling-Only (can modify the SDP in SIP message).

An SDP –Modifying Signaling Only B2BUA is one that operates in the Signaling Plan only AND NOT in the Media Path. However, it can MODIFY the SDP. Thus, this type of B2BUA is aware of the SDP semantics.

Purpose:

This SDP modifying B2BUA does NOT  make changes to the Media Path.  That is, it does not stay or INSERT themselves in the PATH of the Media (like a Third Party Call control servers).

However, it will make SDP changes that affects 

• what is sent on the Media Plane ?  (like the SDP offer changes like removing the unsupported Codecs ) OR 
• It can MERGE two separate Media end points into one SDP offer etc.  

Certain Application servers or SIP PBX or SIP PSTN Gateways act in this role (SDP modifying B2BUA). So that they can remove the unsupported Codecs from the SDP.

            

        In SIP, what is GRUU and how it helps in Call Transfer             

                   

Most of us would have worked on various SIP Call transfer related issues. And we already know that the SIP Method REFER is used for the call transfer. So, we are not going to discuss about REFER method here. Instead, today we are going to discuss about how a SIP server would identify the exact SIP client correctly and then routes the call to the target location (when users are signed to multiple devices). In order to route the call to the end point, the information in the Contact header is used. So, Let us see the contents of a  Contact: header in detail.

The Contact: header

In a SIP  Contact: header you will find the SIP Client IP address, port number, the protocol used, expires  value, URN and GRUU.

For example, When a client sends a SIP REGISTER request, would be similar to the one shown here

In the SIP Client Register request:

Contact: <sip:10.2.2.210:49872;transport=tls;ms-opaque=cc851bcfca>;methods="INVITE, MESSAGE, INFO, OPTIONS, BYE, CANCEL, NOTIFY, ACK, REFER, BENOTIFY";proxy=replace;+sip.instance="<urn:uuid:DBCB7786-ACCF-5829-8AF1-6928B6C14315>"

Then, the response from SIP Registrar server would be like

Contact: <sip:10.2.2.210:49872;transport=tls;ms-opaque=cc851bcfca;ms-received-cid=C020900>;expires=7200;+sip.instance="<urn:uuid:dbcb7786-accf-5829-8af1-6928b6c14315>";gruu="sip:yogesh.s@sipdomain.com;opaque=user:epid:hnfL28-sKViK8WkotsFDFQAA;gruu"

Did you notice that the Client sent a URN value and the Server had returned some GRUU value ?

Purpose of GRUU:

Normally, an administrator new to VoIP systems will have this question, why we need GRUU ?  is having a SIP Address (which is unique) coupled with IP address (which is also unique) is not sufficient to route to the correct SIP end point ? - No. Let us see the reasons.

Reason #1: No, this is because, just with the IP address we cannot route to the correct SIP client if a client is behind a NAT.  Not only the limitation in case of NAT, but also, it should remain unique even when the SIP client is connected to different IP network.

Reason #2:  What if the IP address of the SIP client (which is signed in with a SIP account) changes later after a client restart. In this case the same user with the same SIP client may get a different IP Address (if the IP address lease has expired).

So, we need a different mechanism in order to uniquely identify a SIP client (other than the IP address).  Such that, the SIP client (User Agent) instance remains globally unique within the SIP Infrastructure.

Hence, we depend on other mechanisms like URN and GRUU in the Contact: header.

Before discussing about URN and GRUU let us try to understand their purpose. 

Alright ! so, what is the purpose of URN and GRUU ? and how do they help in a SIP Call flow ? In order to understand this, we need to first understand the challenges in a blind call transfer scenario. So, let us consider a scenario and try to understand the challenges in it.

SIP URI:

We know that the advantage of SIP system is that, it provides a SIP URI which is unique within a SIP domain. Moreover, it allows a user to login to multiple devices with the same SIP URI. So, consider that I am logged to a desktop computer, a laptop computer and two mobile devices (like a tablet and one mobile phone device) with my SIP account (which is unique but same SIP URI is on all the four devices). Then what if,  I am already on a call with  my colleague using a (SIP client running on my laptop) and waiting for a call to be transferred (from one of my colleague) ? In this case, the call should be transferred to the end point where I am already active isn't ? Only then, I can attend the call immediately.So,  is it simple for a SIP server to find exact end point where I am already on a call or is there any challenge in it ?  Yes, the challenge here is, how a SIP server can identify exactly my laptop client and route to it. ? So, how can we solve this problem ?

SIP URI limitations:

Can a SIP URI can solve this problem, because it is unique  isn't ? No ! because, though the SIP URI is unique within the SIP domain, it only helps in  identifying the unique SIP user account. Whereas, in this scenario I am using the same SIP URI on multiple devices. Hence, we have two challenges here.

Firstly, we need a mechanism to uniquely identify each SIP client instance though I am using the same SIP URI on multiple devices. That is, the SIP client instance running on the Desktop, Laptop and mobile devices has to differentiated (though I am using the same SIP URI).

Secondly, we also need a way to identify the exact SIP client instance along with the SIP URI that is used on the device. (why ? because the same SIP client can be used by different users with different SIP URI as well). In order to address these challenges we use URN and GRUU.

URN: 

 As per RFC 5031, Each SIP Client MUST have an Instance Identifier Uniform Resource Name (URN) that uniquely identifies the device. Furthermore, a URN has the following characteristics.

• Usage of a URN provides a persistent and unique name for the User Agent  instance. 
• It also provides an easy way to guarantee uniqueness within the AOR when signed to multiple devices. 
• This URN MUST be persistent across power cycles of the device. 
• The instance ID MUST NOT change as the device moves from one network to another.
• The SIP client is responsible to create a instance ID.
• In Soft client, during the SIP client installation the instance id is created.

Alright ! So does having a  persistent URN and with all the above characteristics, would solve our problem? Not really ! Why ? Notably it leaves us with another challenge here, what  if  the same SIP client is used by a different user with different SIP URI  ? So, the SIP server actually needs a mechanism to route to the Unique SIP Client Instance along with the SIP URI successfully.
Hence, we use GRUU – Globally Routable User agent URI.

Next, let us see this GRUU  in detail.  What is GRUU ?

GRUU:

A SIP URI that routes to a specific UA instance is called a Globally Routable User Agent URI (GRUU). That is, we need a globally routable mechanism in order to reach each SIP client Instance.

Now, lets see how the GRUU is generated ?

During the SIP client Registration phase,  the SIP client would send a Register request to the SIP REGISTRAR server with the +sip.instance and its URN value. Also the SIP client would send with the Supported: header with the value gruu, thus  indicating that it can support GRUU. Thus, a SIP Registrar will understand that the SIP client can support GRUU and it had to create one for the SIP client with the specific client instance.

As per RFC 5627, "The basic unit of reference is the Address of Record (AOR).  However, in SIP systems a single user can have a number of user agents (handsets,soft phones, voicemail accounts, etc.) that are all referenced by the same AOR. There are a number of contexts in which it is desirable to  have an identifier that addresses a single user agent rather than the group of user agents indicated by an AOR."  And it also says that "Every GRUU is associated with a single AOR and a single instance ID.  A SIP registrar MUST be able to determine the instance ID and AOR when presented with a GRUU.  In addition, the GRUU, like an AOR, resolves to zero or more contacts.  While the AOR resolves to all registered contacts for an AOR, a GRUU resolves only to those contacts whose instance ID matches the one associated with the GRUU. "

GRUU properties:

• It routes to a specific UA instance.
• It can be successfully dereferenced by any user agent on the Internet, not just ones in the same domain or IP network as the UA instance to which the GRUU points.

Once the SIP client received the GRUU from the SIP Registrar server,it uses them as the contents of the Contact: header field in non-REGISTER requests and responses that it emits (for example, an INVITE request and 200 OK response).

Contact: <sip:yogesh.s@sipdomain.com;opaque=user:epid:hnfL28-sKViK8WkotsFDFQAA;gruu>

Also, we have two types of GRUUs a) Public GRUU and b) Temporary GRUU based on the requirements and the purpose. When a SIP client refreshes this registration prior to its expiration, the SIP Registrar will return back the same public GRUU. However, it will create a new temporary GRUU only when the contact for the instance expires, either through  explicit de-registration or timeout, all of the temporary GRUUs become invalidated.

NOTE: The SIP client would use one of its temporary GRUUs for anonymous calls (because it does not have the user's SIP Address), and use its public GRUU for all the other calls. 

How a SIP Proxy would treat a GRUU ?

From RFC 5627 we see how a SIP Proxy would de-reference the GRUU. Since it is self-explanatory I will like to just mention it here. It says that "A GRUU is simply a URI, a UA  dereferences it in exactly the same way as it would any other URI.  However, once the request has been routed to the appropriate proxy, the behavior is slightly different.  The proxy will map the GRUU to the AOR and determine the set of contacts that the particular UA instance has registered.  The GRUU is then mapped to those contacts, and the request is routed towards the UA".

Analogy:

If you feel that it is difficult to follow, I could help you with an analogy that we are well aware  in the TCP/IP suite. In IP network, though the MAC address is unique and the IP address is also unique within the network, you need both MAC and IP address to reach the correct host - isn't ?

Similarly, in SIP 

• URN is like the MAC address  -  because URN does not change over reboot or while changed to another network.
• GRUU is like the IP address (in this case, I mean a dynamic IP with the DHCP leasing time).Since this is provided by the SIP servers (at the time of Registration phase). Hence, this may gets changed for some session, but remains unique within the SIP infrastructure. 

The SIP proxy, is just like the Routers and Switches in the IP network (which is used for routing the Packets and Frames to the respective hosts correctly). Similarly, the SIP proxy has its own  mechanism to determine the exact SIP client instance using the GRUU. As a result, it could then routes the calls to all the contacts that are registered with same SIP URI. However, once a call transfer request is made from a particular sip client instance, the SIP proxy can uniquely identify the SIP client instance and route the call accordingly.

Therefore, by using GRUU (which in turn requires URN) in the Contact: header a SIP server can uniquely identify and route to exact SIP client instance. Thus, when a blind call transfer is initiated, the calls get routed to the exact end point (from where the request was made, even though when a user is signed to multiple devices with the same SIP URI).

In summary,

• The SIP client creates a globally unique Instance ID at the time of installation (in Soft clients).
• During the SIP client registration phase, the SIP client contacts the SIP REGISTRAR server with Contact: header which contains details like IP address, port number, SIP methods it can support and protocol  used by the SIP client. Besides that it also provides its URN value in +sip.instance and with Supported: header with the value gruu. 
• The SIP Registrar replies with the GRUU value in the Contact: header. Such that, later a SIP server  by checking the GRUU value, it would be able to uniquely identify the SIP URI and its exact sip client instance.
• Then, the SIP client later uses this GRUU value (which it received from SIP Registrar's response) in its Contact: header for any Non-Register communication (Invite, or 200 OK). 
• Thus, while routing a call, a SIP Proxy server or (a SIP client) identifies the exact SIP client instance using GRUU and directs the call exactly to the requested end point.

Thank you for reading !

Reference:    RFC 5031, 5627.

                                   Identifying Active Speakers in a conference

Hello Readers,

Have you ever had this question in your mind -  how a Conference server would detect the active speakers in a conference and then displays their name or photo or video while they are speaking in a conference call ? Today we are going to discuss about the technical details behind highlighting the active speakers in a Conference.

I had this question  in mind for quite some time and was trying to find an answer to it. So, today let us discuss about it. I am sure that you might know that in a real time communication we use RTP along with the transport protocol UDP (User Datagram Protocol)  in order to carry the media from one end point to the other.  RTP helps in several ways than merely carrying the traffic with the help of UDP.  

RTP (Real Time Protocol):

Apart from carrying the media from one end point to the other, RTP also helps in identifying the active speakers in a conference calls. It does that by using Synchronization Source (SSRC) and Contribution Source (CSRC) identifier. Let us see these in detail by looking at the RTP header.

In the RTP header (snapshot shown below) we have several fields like sequence number, timestamp, Marker bit, and the Synchronization source (SSRC) and contributing source (CSRC) identifiers. For today's topic let us discuss about the Synchronization source and contributing source identifiers here.

Let us see what actually happens in a conference call. A user will use a SIP Address to join the Conference call. However, the SIP is a application layer protocol. So, it cannot help in detecting the media or in identifying the end points that is used to send the media traffic. Moreover, what if a user uses two video cameras for a session. In that case, you need a mechanism to differentiate the signals from the two different devices. So that, after Sampling the analog signal and converting them to digital it can be placed them in a RTP packet with the captured device details. In the RTP packet is there a way to notify which device was used?

Yes, the Synchronization Source (SSRC) identifier in the RTP header, helps in the identifying the actual device that was used to send the media in a RTP session. Also this Synchronization Source identifier is globally unique within a RTP session. This is true even  if you have multiple Audio devices – a headset or a laptop microphone and speaker. This does not mean that the synchronization source identifier would remain same for all the RTP sessions, it may change for the next RTP session.

So, having a synchronization source (SSRC) identifier for each device would help in identifying the exact device and its input from the other device. For example, if a user uses the headset then the Synchronization source  identifier for the RTP session value would be the headset. So, with the help of SSRC identifier the Conference server would identify the active speaker and then shows the active speaker accordingly. This look simple isn't? Alright, now let us see  a real world scenario. 

Example:

Let us consider that, in a conference we have 5 participants. And chances are that, all the participants may join from different networks, countries and would have different bandwidth limits. Let us say 2 participants have excellent bandwidth and 1 has average bandwidth and 2 members are connected from a network which has low bandwidth.

In this case, if you want to choose a common codec then obviously it  would be one  which is used in the low frequency network can support. But by doing so we don’t want the users who have the excellent bandwidth to have poor video quality. So how to overcome this situation ? Here comes the role of a Mixer (Conference Server does that) 

RTP Mixer.

 A RTP mixer (in the conference sever) would be actually collecting all the inputs from all the participants. Then it would convert them to a new RTP packet and send it to all the endpoints. Thus,  the users in the poor network location would receive the quality which their network can support. Likewise, the other participants who are having excellent bandwidth can choose the one which has the best quality.

Here comes the tricky part. If you need to just differentiate the RTP stream using the source of the device using the Synchronization Source, then in this case the Synchronization source would be Mixer (conference server). So having only a Synchronization Source value in the RTP header is not an optimal solution to find the active speaker in a conference scenario. Hence we have another identifier called the Contributing Source (CSRC) identifier which helps in this situation.

The Contribution source identifier (CSRC) plays a very significant role while collecting the RTP streams from multiple users RTP stream and converting to a new RTP packet. While RTP Mixer (the conference server) creating the new RTP packet, it would also include the list of the active speakers in that instance, like participant 1 - who was talking AND  at the same time participant # 5 was trying to ask a question, while others were silent. So in this Packet the SSRC will have the mixer/conference server value and the CSRC will have the value of the Participant #1 and Participant #5. Thus, we get to see the active speaker in the conference even if hear sounds or noise from multiple users. That is great, but does the RTP Mixer work if a user is behind a NAT or a Firewall ?  No! So, here comes another important component called  - RTP Translator. Let us check that scenario now. As usual let us check why we need it and how it help us ?

RTP Translator:

The RTP Mixer can help only if the participants are directly reachable. However, if they are behind a NAT/ firewall then obviously a participant cannot reach the Mixer (Conference server). Hence, we have another component called RTP Translator. Consider this translator is like a server who sits in the DMZ and with a funnel. Then, it funnels the RTP traffic from all the participants to the Mixer and gets the new RTP stream from the Conference. Also it funnels  out to the other participants who are in the internet.

A Participant Leaving or Exiting Scenario:

Alright this sounds like a good option, but what happens if a person is leaving the conf. session  ? Well, in order to address this scenario, we have RTCP BYE packet. An RTCP sends a RTCP - BYE message when a person leaves a conference. Hence, others get notified that a user is leaving the conference.

NOTE:  We have not discussed about RTCP here yet. Let us discuss about it on some other day :-)

Reference: RFC 3550

           

To summarize, using the Synchronization source and Contribution Source identifiers in the RTP header we get to know the active speaker details in a conference call. I hope that you liked this topic and the discussion.  Thank you for reading !

Polycom Group Series Remote and Real Presence Touch Screen are not working.


We have couple of Polycom Real Presence Group series 500 devices in our conference rooms. Those devices are really good and working fine for several months without any problem. Recently, we received a complaint from user saying the Polycom Remote is not working.

Issue: Polycom Group Screen Remote and Real Presence Touch Screen are not working.
I can guess what you are thinking - was it a battery issue ? it is strange, because the Power On and Power button was working but the other options like  Volume functions were not working on the remote. Moreover, if it was a battery issue even the Power On and Power Off buttons would not have worked. So, no, it was definitely not a battery issue :-)

While working with Polycom Support regarding this issue, they asked us to check if the Polycom Touch screen device was working. We found that both the Touch screen device and then Polycom Remote was not working. As per the Polycom Support, if you have Skype Mode option enabled the Polycom Remote functionality would not work. However, in our case both the Real Presence Touch screen and the Polycom Remote  was not working.

Since, the Polycom Support provided this hint (Skype Mode option) my colleague (Kit) and I decided to check the device configuration once again. We noticed that the option  Enable Skype Mode was selected and hence felt like disabling this option and test it.  So, we Unchecked that option  (shown in the Snapshot), click Save and  rebooted the device, the issue was resolved.

 Though the issue got resolved by not selecting the Enable Skype and Polycom Support also confirmed that "When Group series is in Skype mode you can only use Real Presence Touch". 
Polycom Support later replied saying Skype Mode works differently for SfB account (for Online and on-Prem accounts).

If the SfB account (which is used in the Polycom Real Presence device) is in SfB Online and if you have the option Skype Mode enabled then the Real Presence touch will work.

If the SfB account (that is used in the Polycom Real Presence device) on a On-Prem server then if you have the option Skype Mode Enabled then both the Polycom Remote and the Polycom Real Presence Touch would NOT work.

 Polycom Real Presence Group series call drop issues.

In this article, I would like to share my recent video conferencing troubleshooting experience.

Last week, we spent some time in troubleshooting some strange Call drops issue.

Issue: Polycom Real Presence Group series call drops exactly at 45th minute.

Firmware version:    6.1.1

Reason: As per the Polycom device logs, TIMER Expired and call dropped.

Setup: In a Conference room,  a Polycom Real presence Group series device is used for Video conferencing. Last week we received complaints from few users that while the users had call for around two hours the call were dropped twice. One more user complained that the call was around one hour and then call dropped and had to connect again. However, other users who had meeting for  did not complain about any issue.  Later when we checked with the users, we found that the users who did not complained about the issue had a meeting less than 40 minutes.

Model: Polycom Real presence Group 500 Series

Firmware version: 6.1.1

The call flow is as follows:

PSTN ßà AudioCodes SBC  ßà SfB Mediation server ßà Polycom Group Series Device.

If you are in a hurry to fix the problem, please  go the Solution section at the bottom of the Page :-)

In case, if you have more time then before we go to the solution,  let us  discuss few basics like the

1. What is SIP  Session refresh (using either RE-INVITE or UPDATE ) ?
2. What are the well known reasons for call drops ?
3. What is  dead call  and how we can  detect it ?
4. What is SIP  Session refresh interval and how it is useful in a SIP Stateful proxy ?

A SIP session when started would have a time frame (you might have seen the header
Session Expires:1800 and it can get refreshed by using UPDATE or RE-INVITE for a session
(Min-SE: 90). Note that, the Polycom device supports the UPDATE method for a SIP session refresh and that is the reason the Call works for the first 44 minutes.

The Call drops can happen due to multiple reasons like

1.  Compatibility with SIP Proxy.
2.  Remote SIP endpoint.
3.  Network Firewall that has SIP inspection enabled.
4.  Issue with the end point (could be a defective device) 
5.  Issue with the  Software running on that device.

So in this case, the call drop was NOT due to a SIP Proxy, or a Firewall. Because  it is not impacting all the users and other end points. Instead, the call drop issues are noticed only on one type of device (Polycom Real Presence Group Series 500). In addition to that, it was interesting to note that we were able to find the issue on two different Polycom devices (with same model and firmware version). So we can rule out the other options like SIP Proxy, Firewall SIP Inspection here or it is not due to a defective device because it is happening on multiple devices .

When a SIP Client fails to send a BYE message at the end of a session, or when the BYE message gets lost due to network problems, a Stateful SIP proxy server will not know when the session has ended.  This is called as Dead call/session.  And worse,  it will maintain such dead sessions and if there are several such Dead sessions/calls, the Stateful proxy may not be even able to process the legitimate new requests. Because it still considers that the Dead Session  are active and does not end the session unless it gets a BYE message. Also, from Security point of view, if we have don’t have a session limit then it would be used for Denial of Service (DOS) Attack. So, having the UPDATE method for a SIP session is very helpful.

Dead Call detection:

There are two ways to detect the Dead call session.

•    RTCP based detection: With this option, by listening and gathering the RTCP  packet details for a session, we can detect if the call was dead or not.

•    Session Timers based detection: The Session Timer based detection provide an alternate methodology for the detection of dead calls using SIP Signaling. If the SIP peer fails to respond to an UPDATE or INVITE message, this enhancement cleans up the call by sending a BYE which will enable the call Stateful SIP proxies (Lync/SfB  in this example) to clean up the resources associated with the call.

Let us see what is a Session Interval ? Session Interval is the maximum amount of time that can occur between the session refresh requests in a SIP dialog, before the SIP session will be considered timed out.   This session interval is conveyed in the Session-Expires header field. So lets us see about this Session interval in detail and why do we need this ?

So, how does Min-SE: and Session-Expires: headers help a STATEFUL SIP Proxy ?

A Client Initiates a SIP session by sending an INVITE.  This INVITE includes a Supported: header field with the option tag 'timer', indicating support for this extension. When this Client INVITE request passes through SIP proxies,  any one of the SIP Proxies may have an interest in establishing a session timer. A SIP Proxy in that path can insert a Session-Expires header field and a Min-SE header field into the request (if none is already there) or alter the value of existing Session-Expires and Min-SE header fields as described below.

As per the RFC 4028,

The Min-SE: header field establishes the lower bound for the session refresh interval. The purpose of this header field is to prevent hostile SIP Proxies from setting short refresh intervals so that their neighbours (SIP Proxies or Clients or other SIP Servers) are overloaded. 

Each SIP Proxy processing the request can raise this lower bound (increase the period between refreshes) but is not allowed to lower it.

The Session-Expires: header field establishes the upper bound for the session refresh interval; i.e., the time period after processing a request for which any session-stateful SIP proxy must retain its state for this session.  Any SIP proxy servicing this request can lower this value, but it is not allowed to decrease it below the value specified in the Min-SE header field.

Now, you can understand how the session timer comes handy in maintaining only the active session.

By knowing these basics, now lets get back to the Polycom Group Series Call drop issue 

So  collected the Polycom Group Series 500 device log and went through it once again. Since we know that UPDATE SIP method is used to refresh the session. In the logs we see several  UPDATE and a corresponding 200 OK  for this call. However during the 44^th minute a call refresh state  is also success (that is, the UPDATE SIP Method and after that we see a corresponding 200 OK for that UPDATE method). But the call fails with the message Timer Expired at the 45 th minute immediately after the 200 OK (for the UPDATE). So where is the problem ?

Looks like the Polycom Real Presence Group series does not honor this 44th minute UPDATE, so it is not refreshing the session further (after the 45th minute). Instead, the Polycom device has its  own timer and hence it disconnects the call. I say this because in the Polycom Real Presence Group 500 series device logs we saw Timer expired  -- Disconnecting call.  So in this case, the root cause of the problem is the Polycom Group Series accepts Update method and the Session refresh request from its peer (the Mediation server component) till 44th minute. This happens for all the calls. So for all the calls that exceeds 45th minute, the call drops at the 45^th minute (and works fine for all calls within the 45 minute).

Solution:

Resolution for Call drop exactly at the 45 th minute:

Raised a Support Case with the Polycom Support - Polycom Support asked to update the firmware to version 6.1.4. It should fix the call drop issue that occurs exactly at the 45th minute of each call.

After upgrading to the latest firmware 6.1.4, we found that few calls were lasting for couple of hours (one call for 2 hours, another for over 4 hours) without any problem.  However, soon we ran into another issue that is random call drops even within 45 minutes (like within 35 minutes, the test call gets dropped).

Resolution for random call drop (after upgrading to version 6.1.4):

Now after updating to the firmware 6.1.4  randomly some calls were disconnected even within the
45 th minute (this time we had a call drop at 35 th minute itself). So, followed up with the Polycom Support again. The Polycom support escalated the case and  confirmed that they are able to duplicate the random call drop issues at their testing lab in spite of the Session refresh interval using the UPDATE SIP method. So, Polycom Support acknowledged that it is a bug and they are working on getting this issue fixed in their upcoming firmware update (6.2.0).

So, just in case if anyone experiences such call drop issue exactly at the 45th minute (then check if you have firmware version 6.1.1 and try to update the Polycom firmware to 6.1.4) and it should reduce the call drops. However,  if you will still have random call drops. Check the Polycom Group series logs and if it is due to the Timer Expired  issue then we need wait for any firmware version updates (which will be released by end of February 2018).

Hope this post was informative. Please feel free to share your suggestions or comments.

Call not releasing / disconnecting properly with analog devices in a SfB Enterprise Voice deployment !

Recently we deployed a SfB Enterprise in a Production site (We used AudioCodes Mediant 1000 PSTN Gateway + SfB SBA + AudioCodes Analog Gateway) in this site. The Analog gateway was to integrate the Analog devices with the SfB clients and Polycom VVX Phones.

Requirement: In a SfB Enterprise Voice deployment scenario, users from their SfB client, Polycom VVX phones should be able to dial Announcement system (by dialing a phone number) and make announcements successfully.

We deployed AudioCodes SBA for a Branch site. Since it is a manufacturing site it had a requirement to integrate the announcement systems. These Analog phones are required to meet the regulatory requirements of any Manufacturing sites).


So, we also installed an Analog Gateway (AudioCodes Media Pack MP124) and connected the analog devices like (analog phones and the announcement systems) to it.

In the SfB Environment, We had the SfB Dial plan created with all the normalized rules.

Like users dial only the last 4 digits and it gets converted to the + < area code> Once the Number gets normalized  we have a PSTN Usage and matching Route Pattern and add the Analog Gateway as the Trunk.

Issue: A call made from a SfB client or VVX Polycom Phone to the announcement works but when the Call ends (either from SfB client you disconnect or when you disconnect the call from the Phone VVX phone), the phone line does not get disconnected or released.  Instead, we get the long busy tone.

Troubleshooting:

We connected an Analog Phone to the Media pack port and then test by dialing the number for example 8xxx.

The analog phone rang and when we hung up the phone it also disconnected we did not get the busy or fast busy tone this time. However, only when we connect the actual Announcement device (Which is an analog device) and when we call the announcement device it accepts the call and we can make the announcement through it. However, When we disconnect the call from our end the call does not get released instead we get a fast busy.

To fix the issue:


1) Created a new Coder (Gateway --> Configuration --> Coders and Profiles --> Coders created a new Proifle ID 1.

2) And then in the Profile ID 1, enabled two properties in the Analog MediaPack Gateway.

a)  Enabled Polarity Reversal to Enable

When this feature is enabled, the analog port (FXS) interface polarity is reversed to indicate the start of a VoIP session and it is reversed back when the VoIP session ends.

b) Enable Current Disconnect to Enable

By enabling these two parameters the call is immediately disconnected after polarity reversal or current disconnect is detected on the Tel side (assuming the PBX / CO produces this signal). 


For the snapshots for creating new Coders and Tel Profile

Creating a new Tel Profile:

Once you create the new Tel Profile 1 then you need to assign it to the required phone number in my case it is Analog Announcement system like shown below.

Planning for Mediation server Role

Mediation server Role plays a crucial role in Lync/SfB system for establishing a call between a Lync/SfB and PSTN or any VoIP system.  There is a Microsoft TechNet article which discusses about the Capacity planning for Mediation server Role.
 [ MS articles

https://technet.microsoft.com/en-in/library/gg398399.aspx  

https://technet.microsoft.com/en-us/library/gg615015.aspx ]. 

However, after going through it i felt that the information provided in the TechNet article is not conclusive. So before we have the Design work shop for the SIP Trunk I went through some of Channel 9 videos regarding the Enterprise Voice features found that there are several important points to consider while planning for Mediation server Role.  So, we will discuss about  some scenarios where Media Bypass can be used and CANNOT be used.

Media Bypass:

One of the main question people have is, can we reduce the number of servers required for the Mediation server role if we have Media By pass enabled for a network.

What is Media ByPass?

Source: https://technet.microsoft.com/en-us/library/gg412740.aspx

Media bypass refers to removing the Mediation Server from the media path whenever possible for calls whose signaling traverses the Mediation Server.  Media bypass can improve voice quality by reducing latency, needless translation, possibility of packet loss, and the number of points of potential failure. Scalability can be improved, because elimination of media processing for bypassed calls reduces the load on the Mediation Server. This reduction in load complements the ability of the Mediation Server to control multiple gateways. Picture from the TechNet article about the Media bypass.

Media By pass is a feature which when enabled will allow the Agents to talk directly to the certified Gateways (bypassing the Mediation server). However, if Media By pass feature is not enabled or not configured for the network, then all the media traffic has to traverse Mediation server.

Media By pass is a great feature and it significantly reduces the load on the Mediation servers and improves the call quality. However, it is good to understand the certain scenarios where the Media Bypass is irrelevant. It is also important to consider the following scenarios (where though the Media By pass feature is enabled) but it will not work even if a Lync/SfB call is made from a Network where Media Bypass is enabled.

Features  / applications / devices  which does not work with Media Bypass:

•  Mobile devices does not use Media By pass. So if you have several thousands of Mobile devices in your infrastructure, then the all the Media traffic has to go through the Mediation servers.  Hence planning for Mediation servers needs to consider this situation too.

•  Response Group does not use Media By pass. So if you have a user calling a Response Group number then though the Calls are originated from a network segment where Media By pass is enabled, the Media Bypass feature will not be used and the media traffic will be going to and from the Mediation server.

•  If you have any trusted third party application like for Contact center solutions      then again Media bypass will not work.

•  CAC issues:As per TechNet artcile [source: https://technet.microsoft.com/en-us/library/gg413028(v  =ocs.15).aspx ]  If you have already optionally configured call admission control (CAC) note that the bandwidth reservation performed by call admission control does not apply to any calls for which media bypass is employed. 

• The Media Bypass feature does not work for the  Dial in Conferencing scenarios.

• The UCWA Applications and AV MCU does not support Media Bypass.

I hope  this post was intersting and please share your views. In case, if you feel that i have missed any other points to be considered while planning for Mediation server Role please share.

How to find the number of devices a user is Logged into using Lync/SfB Client logs ?

Recently, we had issue reported regarding the trusted application. While troubleshooting the issue, we had a requirement to understand the number of devices on which a user is logged in.

In our client environment we are using a third party Contact center solution as a trusted application. It was a kind of weird one, where the callers of a Contact center number where complaining they are not able to reach any agents. Whereas, the agents complained us that they are NOT receiving any calls.  

While we were discussing this issue with my colleagues we had a question, can we find the total number of registered end points for a sip account using the Lync/SfB client logs. That was a very good question. Because we have seen some third party applications actually providing the details like a user with a sip address say UserA@uctest.com has signed to 3 devices. So,  it was interesting to find out how a third party reporting applications can get the list of  devices to which a users had signed in to ? So, I tried to understand if there are anyways to get this information from the client/server  logs and understand the number of End points on which a user is logged into. 

NOTE:

I am writing this article in order to help those administrators who don't have Monitoring/Reporting solution available at their environment. Since, some Lync/SfB environment do not have the Monitoring server role or any third party Monitoring/Reporting solutions. 

There are two ways  to find out the list of get this information one using the Server logs and the other using the Client logs.

Solution #1: From the Server  logs

My first suggestion was to enable logging on all the Lync/SfB FE server and then make a test call to that the end point. Since Front End server will fork the incoming calls to all the registered end points we will get multiple 180 Ringing SIP Response if a user has signed on to multiple devices. 

So, if a user is signed on two devices like computer and a mobile phone, then I will get two 180 Ringing in the FE server logs.  Though this is helpful, this requires little bit of time and effort (to collect the logs on all the Lync/SfB servers during business hours will end up generating more logs). Moreover, if you have 10 servers in a pool then the amount of logs generated and analyzed from all the servers in the Lync/SfB FE pool will require more effort.

Solution #2: Using the Lync/SfB Client logs

But later, I realized that there is much better way to get this information and with less effort. All you need to do is, ask the user is to Sign in to another computer once again and provide the Lync/SfB client logs.  So, is it possible to get the find the number of devices on which a user is logged into using the recently gathered Lync/SfB client logs ? The answer is Yes J

I will explain this with an example. I singed into one computer, say Computer #1, then a mobile device. Later,  I singed to another computer,  Computer #2. So, now I am signed into my Lync accounts from 2 computers and one mobile client, so I have multiple registered end points for my sip account.

I collected the Lync client logs from the second computer Computer #2 (this is the latest computer on which I signed in). Then, I looked for the response 200 OK for the SIP REGISTER packet (In this 200 OK response there are three Contact: headers in it.

NOTE: You have to ask the user to Sign-in to Computer once again and then collect the client logs.This is a very important step, if you skip this step then you might not get the correct details.

07/04/2016|21:46:11.303 1834:1838 INFO  :: Data Received -10.1.92.181:5061 (To Local Address: 172.24.41.159:59266) 1640 bytes:

07/04/2016|21:46:11.303 1834:1838 INFO  :: SIP/2.0 200 OK

ms-keep-alive: UAS; tcp=no; hop-hop=yes; end-end=no; timeout=300

Authentication-Info: TLS-DSK qop="auth", opaque="F8084D77", srand="468DD1A4", snum="1", rspauth="2891b96442e3e7c7af0ef004a7babb9063fd48ba", targetname="FE01.uctest.com", realm="SIP Communications Service", version=4

From: "SOMASUNDARAM Yogesh"<sip:yogesh.somasundaram@uctest.com>;tag=cc0f68ce68;epid=6c1409c121

To: <sip:yogesh.somasundaram@uctest.com>;tag=99C17913B0B71378DCADCD792D587E29

Call-ID: d8060b5307124a1787e243e9ff587edc

CSeq: 12 REGISTER

Via: SIP/2.0/TLS 172.24.41.159:59266;ms-received-port=59266;ms-received-cid=39F2FF00

Contact: <sip:172.24.41.159:59266;transport=tls;ms-opaque=61fe165797;ms-received-cid=39F2FF00>;expires=7200;+sip.instance="<urn:uuid:aa39a11c-7a67-5d34-a985-7418ff06799c>";gruu="sip:yogesh.somasundaram@uctest.com;opaque=user:epid:HKE5qmd6NF2phXQY_wZ5nAAA;gruu"

Contact: <sip:FEPOOL01.uctest.com:5088;ms-fe=FE02.uctest.com;transport=Tls;ms-opaque=43b8303fcf7aa2b4>;expires=1295940;+sip.instance="<urn:uuid:aad6c08c-7dd8-5733-8b9e-cdcb8b0a1827>";gruu="sip:yogesh.somasundaram@uctest.com;opaque=user:epid:jMDWqth9M1eLns3LiwoYJwAA;gruu"

Contact: <sip:172.24.41.172:50900;transport=tls;ms-opaque=353dd698df;ms-received-cid=3A42B400>;expires=6311;+sip.instance="<urn:uuid:b9cc1792-d0cf-51ae-8c7e-6a0504a8b81c>";gruu=sip:yogesh.somasundaram@uctest.com;opaque=user:epid:khfMuc_QrlGMfmoFBKi4HAAA;gruu

If you have a mobile client then you will find that the Contact: header has the  FE pool name with the port number 5088 (unlike the IP address of computer).  This is because, the UCWA sip Listening port is 5088. In order to find this port number, on a FE server if you run the Get-csService | Fl *UC*  this will let you know that the UcwaSIP PrimaryListeningPort  is 5088.

So with the Contact header field present in the 200 OK, we can confirm that a user is logged into two different computer and one mobile client. And there is no need to collect the logs from the FE server’s logs in order to find this out J

Hope you find this information helpful.

NOTE:  I am sure, there are many ways to get his information and it is possible from SQL database too. However, if you are not having access to write a SQL query (like me) then it is easy to check the Client SIP logs rather than writing a SQL query.