VoIP Mailing List Archives

[cary at ytel.com]

 We recently started having a problem with our FreeSWITCH servers where calls are working well, with about 200 calls operating sometimes for a couple of days, and then something happens and all call progress stops. Then any new inbound calls that come into the server get stuck in initial status, new inbound get stuck at RINGING. This happens until the max sessions are hit and of course nothing else works from there. Our thought is something is creating a lock and stopping anything else from progressing.
 We are seeing the stuck sessions using "show channels". Doing a uuid_kill on the calls seems to make no difference to the output of "show channels", but we have noticed in the logs that some events do happen to some of the calls.
 We had ran into a similar issue when writing a custom C module and made some mistakes with session_rwlock etc. But in this case, all of this is using LUA.  
Some general information:
FreeSWITCH Versions 1.10.6 and 1.10.1
Modules used:
LUA, xml_curl(dialplan), event_socket,

 Most of this is just application playing audio files and or creating a new call to bridge to.

 Really looking for some help on what debugging we can enable to help identify what is getting stuck to identify where a bug might be on our usage or possible issue with FreeSWITCH.

 We have captured a core dump of the running process when it got into the bad state as well, if that is helpful, we can send it in.

Thank you for your help.

[brian at freeswitch.com]

It's an issue we've been searching for, if you uuid_kill it, and it says no session, it only means the record was NOT removed from the sqlite database.

Nobody can seem to replicate the steps required to get into that condition, maybe you could look thru the logs and see if you can replicate what ever happened on that lingering UUID


/b




On Fri, Aug 27, 2021 at 5:40 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

[cary at ytel.com]

 Yes, that is what we are seeing when running uuid_kill.  We cannot "replicate" the issue, but it happens multiple times a day for us across our servers.   We have looked through logs across multiple systems, taken SIP traces to see if that showed anything.  Last we got a Core dump of the running process after the failure. Nothing seems to stand out as the cause.

 
 Are there any other debug steps you can think of or what other information to look at?  We could capture all traffic on the server, or recompile with special debug flags whatever it is that might help.
 
Steps we have taken:
sofia loglevel all 9
sofia global siptrace on
fsctl loglevel 7
# Set environment variable TPORT_LOG=1




On Sat, Aug 28, 2021 at 12:54 AM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

[brian at freeswitch.com]

Check the UUID of the session in question, and extract its logs, and let's diagnose it

On Mon, Aug 30, 2021 at 12:39 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

[brian at freeswitch.com]

and to make sure when you uuid kill one it says it didn't exist but show channels had it still listed correct?

On Tue, Aug 31, 2021 at 2:58 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

[brian at freeswitch.com]

Guessing you create sessions in lua? ProxyOutbound.luac?

On Tue, Aug 31, 2021 at 3:33 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

[brian at freeswitch.com]

I you do sessions from your lua, I need to see how you do that.

On Tue, Aug 31, 2021 at 4:20 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

[brian at freeswitch.com]

Can you try to create one where this happens without all this mess?

On Wed, Sep 1, 2021 at 2:41 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

[cary at ytel.com]

[cary at ytel.com]

Hello,


We have got the logs for a few calls that were stuck.
(Show channels fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/NTnte7J7


(Extracted from logs fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/sAhFiGjY

(Show channels 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)
https://pastebin.com/bmcdjzAr



(Extracted from logs 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)

https://pastebin.com/aGyY3m9F



(Show channels 624cac90-0a7b-11ec-b7d0-6f543a588268)

https://pastebin.com/0cnQaRhC



(Extracted from logs 624cac90-0a7b-11ec-b7d0-6f543a588268)
https://pastebin.com/atX0VXdZ



Please let us know if there is anything else we can provide that would be useful. Thank you.




On Mon, Aug 30, 2021 at 8:20 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

[cary at ytel.com]

Here is the LUA code regarding creating sessions and bridging. It is broken up into 5 sections.
The top part is the common functions, and the 4 sections below are different call scenarios.
https://pastebin.com/qqGw3tms









On Wed, Sep 1, 2021 at 3:53 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

[cary at ytel.com]

The first time you run fs_cli uuid_kill <UUID here> freeswitch returns +OK
The second time you run it -ERR No such channel!
If you check show channels after that the call still show up the same as before running uuid_kill.


On Tue, Aug 31, 2021 at 8:33 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

[cary at ytel.com]

Here are two originate samples.


https://pastebin.com/YKRKrJLu


https://pastebin.com/TEX3yfX6





On Wed, Sep 1, 2021 at 8:00 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

[brian at freeswitch.com]

And these exhibit the issue on command?

On Wed, Sep 1, 2021 at 5:06 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

[brian at freeswitch.com]

Gotta get it replicated somehow.

On Thu, Sep 2, 2021 at 3:40 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote: