Sponsor: VoiceMeUp - Corporate & Wholesale VoIP Services

VoIP Mailing List Archives
Mailing list archives for the VoIP community
 SearchSearch 

[Freeswitch-users] FreeSWITCH Stops Responding

Goto page Previous  1, 2
 
Post new topic   Reply to topic    VoIP Mailing List Archives Forum Index -> freeSWITCH Users
View previous topic :: View next topic  
Author Message
cary at ytel.com
Guest





PostPosted: Fri Sep 03, 2021 3:07 pm    Post subject: [Freeswitch-users] FreeSWITCH Stops Responding Reply with quote

Quote:
We have not been able to reproduce this in testing. It only happens in our production environment.
Are there any other debug options that could be enabled, or is there any code in FreeSWITCH that you think may be causing this that we could look into. We could get you access to a server if that would help.
Server Details:
Gentoo-5.10.27
RAM: 15GB
CPU: 4
FreeSWITCH-1-.10.6
Sqlite-3.34.1



On Thu, Sep 2, 2021 at 7:24 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
And these exhibit the issue on command?

On Wed, Sep 1, 2021 at 5:06 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here are two originate samples.


https://pastebin.com/YKRKrJLu


https://pastebin.com/TEX3yfX6





On Wed, Sep 1, 2021 at 8:00 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Can you try to create one where this happens without all this mess?

On Wed, Sep 1, 2021 at 2:41 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here is the LUA code regarding creating sessions and bridging. It is broken up into 5 sections.
The top part is the common functions, and the 4 sections below are different call scenarios.
https://pastebin.com/qqGw3tms









On Wed, Sep 1, 2021 at 3:53 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
I you do sessions from your lua, I need to see how you do that.

On Tue, Aug 31, 2021 at 4:20 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Quote:
Yes, it bridges the inbound call to an outbound one.(ProxyOutbound.luac)
https://pastebin.com/RDUMy473



On Tue, Aug 31, 2021 at 8:39 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Guessing you create sessions in lua? ProxyOutbound.luac?

On Tue, Aug 31, 2021 at 3:33 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
and to make sure when you uuid kill one it says it didn't exist but show channels had it still listed correct?

On Tue, Aug 31, 2021 at 2:58 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Hello,


We have got the logs for a few calls that were stuck.
(Show channels fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/NTnte7J7


(Extracted from logs fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/sAhFiGjY

(Show channels 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)
https://pastebin.com/bmcdjzAr



(Extracted from logs 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)

https://pastebin.com/aGyY3m9F



(Show channels 624cac90-0a7b-11ec-b7d0-6f543a588268)

https://pastebin.com/0cnQaRhC



(Extracted from logs 624cac90-0a7b-11ec-b7d0-6f543a588268)
https://pastebin.com/atX0VXdZ



Please let us know if there is anything else we can provide that would be useful. Thank you.




On Mon, Aug 30, 2021 at 8:20 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Check the UUID of the session in question, and extract its logs, and let's diagnose it

On Mon, Aug 30, 2021 at 12:39 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
 Yes, that is what we are seeing when running uuid_kill.  We cannot "replicate" the issue, but it happens multiple times a day for us across our servers.   We have looked through logs across multiple systems, taken SIP traces to see if that showed anything.  Last we got a Core dump of the running process after the failure. Nothing seems to stand out as the cause.

 
 Are there any other debug steps you can think of or what other information to look at?  We could capture all traffic on the server, or recompile with special debug flags whatever it is that might help.
 
Steps we have taken:
sofia loglevel all 9
sofia global siptrace on
fsctl loglevel 7
# Set environment variable TPORT_LOG=1




On Sat, Aug 28, 2021 at 12:54 AM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
It's an issue we've been searching for, if you uuid_kill it, and it says no session, it only means the record was NOT removed from the sqlite database.

Nobody can seem to replicate the steps required to get into that condition, maybe you could look thru the logs and see if you can replicate what ever happened on that lingering UUID


/b




On Fri, Aug 27, 2021 at 5:40 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
 We recently started having a problem with our FreeSWITCH servers where calls are working well, with about 200 calls operating sometimes for a couple of days, and then something happens and all call progress stops. Then any new inbound calls that come into the server get stuck in initial status, new inbound get stuck at RINGING. This happens until the max sessions are hit and of course nothing else works from there. Our thought is something is creating a lock and stopping anything else from progressing.
 We are seeing the stuck sessions using "show channels". Doing a uuid_kill on the calls seems to make no difference to the output of "show channels", but we have noticed in the logs that some events do happen to some of the calls.
 We had ran into a similar issue when writing a custom C module and made some mistakes with session_rwlock etc. But in this case, all of this is using LUA.  
Some general information:
FreeSWITCH Versions 1.10.6 and 1.10.1
Modules used:
LUA, xml_curl(dialplan), event_socket,

 Most of this is just application playing audio files and or creating a new call to bridge to.

 Really looking for some help on what debugging we can enable to help identify what is getting stuck to identify where a bug might be on our usage or possible issue with FreeSWITCH.

 We have captured a core dump of the running process when it got into the bad state as well, if that is helpful, we can send it in.

Thank you for your help.


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]


















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]
















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]













Back to top
krice at freeswitch.org
Guest





PostPosted: Fri Sep 03, 2021 6:13 pm    Post subject: [Freeswitch-users] FreeSWITCH Stops Responding Reply with quote

Having massive experience here, you might try replacing your linux distro.

(Yes i know some people are going to take this as brushing off the problem or trying to start a distro war. If this starts a distro war I will moderate you and only allow replies are not headed to a flame war.)

However, as someone who has ran an extremely large Gentoo based cluster, we had to abandon gentoo due to random crashes, unexplainable bugs that could not be reproduced or explained.


If you can not replace the distro I VERY HiGHLY recommend running vanilla libc and other libs without the gentoo portage included patches. (hint the use flag: vanilla)

Ken
Sent from my iPhone

Quote:
On Sep 3, 2021, at 14:49, Cary Shouse <cary@ytel.com> wrote:


Quote:
We have not been able to reproduce this in testing. It only happens in our production environment.
Are there any other debug options that could be enabled, or is there any code in FreeSWITCH that you think may be causing this that we could look into. We could get you access to a server if that would help.
Server Details:
Gentoo-5.10.27
RAM: 15GB
CPU: 4
FreeSWITCH-1-.10.6
Sqlite-3.34.1



On Thu, Sep 2, 2021 at 7:24 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
And these exhibit the issue on command?

On Wed, Sep 1, 2021 at 5:06 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here are two originate samples.


https://pastebin.com/YKRKrJLu


https://pastebin.com/TEX3yfX6





On Wed, Sep 1, 2021 at 8:00 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Can you try to create one where this happens without all this mess?

On Wed, Sep 1, 2021 at 2:41 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here is the LUA code regarding creating sessions and bridging. It is broken up into 5 sections.
The top part is the common functions, and the 4 sections below are different call scenarios.
https://pastebin.com/qqGw3tms









On Wed, Sep 1, 2021 at 3:53 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
I you do sessions from your lua, I need to see how you do that.

On Tue, Aug 31, 2021 at 4:20 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Quote:
Yes, it bridges the inbound call to an outbound one.(ProxyOutbound.luac)
https://pastebin.com/RDUMy473



On Tue, Aug 31, 2021 at 8:39 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Guessing you create sessions in lua? ProxyOutbound.luac?

On Tue, Aug 31, 2021 at 3:33 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
and to make sure when you uuid kill one it says it didn't exist but show channels had it still listed correct?

On Tue, Aug 31, 2021 at 2:58 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Hello,


We have got the logs for a few calls that were stuck.
(Show channels fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/NTnte7J7


(Extracted from logs fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/sAhFiGjY

(Show channels 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)
https://pastebin.com/bmcdjzAr



(Extracted from logs 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)

https://pastebin.com/aGyY3m9F



(Show channels 624cac90-0a7b-11ec-b7d0-6f543a588268)

https://pastebin.com/0cnQaRhC



(Extracted from logs 624cac90-0a7b-11ec-b7d0-6f543a588268)
https://pastebin.com/atX0VXdZ



Please let us know if there is anything else we can provide that would be useful. Thank you.




On Mon, Aug 30, 2021 at 8:20 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Check the UUID of the session in question, and extract its logs, and let's diagnose it

On Mon, Aug 30, 2021 at 12:39 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Yes, that is what we are seeing when running uuid_kill. We cannot "replicate" the issue, but it happens multiple times a day for us across our servers. We have looked through logs across multiple systems, taken SIP traces to see if that showed anything. Last we got a Core dump of the running process after the failure. Nothing seems to stand out as the cause.


Are there any other debug steps you can think of or what other information to look at? We could capture all traffic on the server, or recompile with special debug flags whatever it is that might help.

Steps we have taken:
sofia loglevel all 9
sofia global siptrace on
fsctl loglevel 7
# Set environment variable TPORT_LOG=1




On Sat, Aug 28, 2021 at 12:54 AM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
It's an issue we've been searching for, if you uuid_kill it, and it says no session, it only means the record was NOT removed from the sqlite database.

Nobody can seem to replicate the steps required to get into that condition, maybe you could look thru the logs and see if you can replicate what ever happened on that lingering UUID


/b




On Fri, Aug 27, 2021 at 5:40 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
We recently started having a problem with our FreeSWITCH servers where calls are working well, with about 200 calls operating sometimes for a couple of days, and then something happens and all call progress stops. Then any new inbound calls that come into the server get stuck in initial status, new inbound get stuck at RINGING. This happens until the max sessions are hit and of course nothing else works from there. Our thought is something is creating a lock and stopping anything else from progressing.
We are seeing the stuck sessions using "show channels". Doing a uuid_kill on the calls seems to make no difference to the output of "show channels", but we have noticed in the logs that some events do happen to some of the calls.
We had ran into a similar issue when writing a custom C module and made some mistakes with session_rwlock etc. But in this case, all of this is using LUA.
Some general information:
FreeSWITCH Versions 1.10.6 and 1.10.1
Modules used:
LUA, xml_curl(dialplan), event_socket,

Most of this is just application playing audio files and or creating a new call to bridge to.

Really looking for some help on what debugging we can enable to help identify what is getting stuck to identify where a bug might be on our usage or possible issue with FreeSWITCH.

We have captured a core dump of the running process when it got into the bad state as well, if that is helpful, we can send it in.

Thank you for your help.


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]


















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]
















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]














_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com
Back to top
david.villasmil.work a...
Guest





PostPosted: Sat Sep 04, 2021 5:50 am    Post subject: [Freeswitch-users] FreeSWITCH Stops Responding Reply with quote

Hello all,


Interesting point about Gentoo, but I highly doubt it’s OS related. I’ve been living this issue (about those “stuck channels”, not a frozen FS, that’s indeed weird) for year and years. I’ve seen in on CentOS, Debian, Ubuntu, etc. 


In my experience this became an issue with the rise of VM. Back in the day you would see this every now and then, like months apart (this, or a crashed MySQL database), but wasn’t big enough to be a critical problem. Today day it’s basically an everyday thing if you have enough nodes.



On Sat, 4 Sep 2021 at 00:15, Ken Rice <krice@freeswitch.org (krice@freeswitch.org)> wrote:

Quote:
Having massive experience here, you might try replacing your linux distro.

 (Yes i know some people are going to take this as brushing off the problem or trying to start a distro war. If this starts a distro war I will moderate you and only allow replies are not headed to a flame war.)

However, as someone who has ran an extremely large Gentoo based cluster, we had to abandon gentoo due to random crashes, unexplainable bugs that could not be reproduced or explained. 


If you can not replace the distro I VERY HiGHLY recommend running vanilla libc and other libs without the gentoo portage included patches. (hint the use flag: vanilla)

Ken
Sent from my iPhone

Quote:
On Sep 3, 2021, at 14:49, Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:





Quote:
Quote:
We have not been able to reproduce this in testing. It only happens in our production environment.
Are there any other debug options that could be enabled, or is there any code in FreeSWITCH that you think may be causing this that we could look into. We could get you access to a server if that would help.
Server Details:
Gentoo-5.10.27
RAM: 15GB
CPU: 4
FreeSWITCH-1-.10.6
Sqlite-3.34.1



On Thu, Sep 2, 2021 at 7:24 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
And these exhibit the issue on command?

On Wed, Sep 1, 2021 at 5:06 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here are two originate samples.


https://pastebin.com/YKRKrJLu


https://pastebin.com/TEX3yfX6





On Wed, Sep 1, 2021 at 8:00 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Can you try to create one where this happens without all this mess?

On Wed, Sep 1, 2021 at 2:41 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here is the LUA code regarding creating sessions and bridging. It is broken up into 5 sections.
The top part is the common functions, and the 4 sections below are different call scenarios.
https://pastebin.com/qqGw3tms









On Wed, Sep 1, 2021 at 3:53 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
I you do sessions from your lua, I need to see how you do that.

On Tue, Aug 31, 2021 at 4:20 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Quote:
Yes, it bridges the inbound call to an outbound one.(ProxyOutbound.luac)
https://pastebin.com/RDUMy473



On Tue, Aug 31, 2021 at 8:39 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Guessing you create sessions in lua? ProxyOutbound.luac?

On Tue, Aug 31, 2021 at 3:33 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
and to make sure when you uuid kill one it says it didn't exist but show channels had it still listed correct?

On Tue, Aug 31, 2021 at 2:58 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Hello,


We have got the logs for a few calls that were stuck.
(Show channels fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/NTnte7J7


(Extracted from logs fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/sAhFiGjY

(Show channels 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)
https://pastebin.com/bmcdjzAr



(Extracted from logs 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)

https://pastebin.com/aGyY3m9F



(Show channels 624cac90-0a7b-11ec-b7d0-6f543a588268)

https://pastebin.com/0cnQaRhC



(Extracted from logs 624cac90-0a7b-11ec-b7d0-6f543a588268)
https://pastebin.com/atX0VXdZ



Please let us know if there is anything else we can provide that would be useful. Thank you.




On Mon, Aug 30, 2021 at 8:20 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Check the UUID of the session in question, and extract its logs, and let's diagnose it

On Mon, Aug 30, 2021 at 12:39 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
 Yes, that is what we are seeing when running uuid_kill.  We cannot "replicate" the issue, but it happens multiple times a day for us across our servers.   We have looked through logs across multiple systems, taken SIP traces to see if that showed anything.  Last we got a Core dump of the running process after the failure. Nothing seems to stand out as the cause.

 
 Are there any other debug steps you can think of or what other information to look at?  We could capture all traffic on the server, or recompile with special debug flags whatever it is that might help.
 
Steps we have taken:
sofia loglevel all 9
sofia global siptrace on
fsctl loglevel 7
# Set environment variable TPORT_LOG=1




On Sat, Aug 28, 2021 at 12:54 AM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
It's an issue we've been searching for, if you uuid_kill it, and it says no session, it only means the record was NOT removed from the sqlite database.

Nobody can seem to replicate the steps required to get into that condition, maybe you could look thru the logs and see if you can replicate what ever happened on that lingering UUID


/b




On Fri, Aug 27, 2021 at 5:40 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
 We recently started having a problem with our FreeSWITCH servers where calls are working well, with about 200 calls operating sometimes for a couple of days, and then something happens and all call progress stops. Then any new inbound calls that come into the server get stuck in initial status, new inbound get stuck at RINGING. This happens until the max sessions are hit and of course nothing else works from there. Our thought is something is creating a lock and stopping anything else from progressing.
 We are seeing the stuck sessions using "show channels". Doing a uuid_kill on the calls seems to make no difference to the output of "show channels", but we have noticed in the logs that some events do happen to some of the calls.
 We had ran into a similar issue when writing a custom C module and made some mistakes with session_rwlock etc. But in this case, all of this is using LUA.  
Some general information:
FreeSWITCH Versions 1.10.6 and 1.10.1
Modules used:
LUA, xml_curl(dialplan), event_socket,

 Most of this is just application playing audio files and or creating a new call to bridge to.

 Really looking for some help on what debugging we can enable to help identify what is getting stuck to identify where a bug might be on our usage or possible issue with FreeSWITCH.

 We have captured a core dump of the running process when it got into the bad state as well, if that is helpful, we can send it in.

Thank you for your help.


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]


















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]
















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]














_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com


--
Regards,


David Villasmilemail: david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)
phone: +34669448337
Back to top
krice at freeswitch.org
Guest





PostPosted: Sat Sep 04, 2021 11:32 am    Post subject: [Freeswitch-users] FreeSWITCH Stops Responding Reply with quote

“stuck channels” usually arent stuck once you realize things like “show calls” and “show channels” are selects against the coredb. this usually happens on busy systems timing put waiting for db delete’s to run that fail or get skipped.

Sent from my iPhone

Quote:
On Sep 4, 2021, at 05:22, David Villasmil <david.villasmil.work@gmail.com> wrote:

Hello all,


Interesting point about Gentoo, but I highly doubt it’s OS related. I’ve been living this issue (about those “stuck channels”, not a frozen FS, that’s indeed weird) for year and years. I’ve seen in on CentOS, Debian, Ubuntu, etc.


In my experience this became an issue with the rise of VM. Back in the day you would see this every now and then, like months apart (this, or a crashed MySQL database), but wasn’t big enough to be a critical problem. Today day it’s basically an everyday thing if you have enough nodes.



On Sat, 4 Sep 2021 at 00:15, Ken Rice <krice@freeswitch.org (krice@freeswitch.org)> wrote:

Quote:
Having massive experience here, you might try replacing your linux distro.

(Yes i know some people are going to take this as brushing off the problem or trying to start a distro war. If this starts a distro war I will moderate you and only allow replies are not headed to a flame war.)

However, as someone who has ran an extremely large Gentoo based cluster, we had to abandon gentoo due to random crashes, unexplainable bugs that could not be reproduced or explained.


If you can not replace the distro I VERY HiGHLY recommend running vanilla libc and other libs without the gentoo portage included patches. (hint the use flag: vanilla)

Ken
Sent from my iPhone

Quote:
On Sep 3, 2021, at 14:49, Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:





Quote:
Quote:
We have not been able to reproduce this in testing. It only happens in our production environment.
Are there any other debug options that could be enabled, or is there any code in FreeSWITCH that you think may be causing this that we could look into. We could get you access to a server if that would help.
Server Details:
Gentoo-5.10.27
RAM: 15GB
CPU: 4
FreeSWITCH-1-.10.6
Sqlite-3.34.1



On Thu, Sep 2, 2021 at 7:24 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
And these exhibit the issue on command?

On Wed, Sep 1, 2021 at 5:06 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here are two originate samples.


https://pastebin.com/YKRKrJLu


https://pastebin.com/TEX3yfX6





On Wed, Sep 1, 2021 at 8:00 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Can you try to create one where this happens without all this mess?

On Wed, Sep 1, 2021 at 2:41 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here is the LUA code regarding creating sessions and bridging. It is broken up into 5 sections.
The top part is the common functions, and the 4 sections below are different call scenarios.
https://pastebin.com/qqGw3tms









On Wed, Sep 1, 2021 at 3:53 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
I you do sessions from your lua, I need to see how you do that.

On Tue, Aug 31, 2021 at 4:20 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Quote:
Yes, it bridges the inbound call to an outbound one.(ProxyOutbound.luac)
https://pastebin.com/RDUMy473



On Tue, Aug 31, 2021 at 8:39 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Guessing you create sessions in lua? ProxyOutbound.luac?

On Tue, Aug 31, 2021 at 3:33 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
and to make sure when you uuid kill one it says it didn't exist but show channels had it still listed correct?

On Tue, Aug 31, 2021 at 2:58 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Hello,


We have got the logs for a few calls that were stuck.
(Show channels fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/NTnte7J7


(Extracted from logs fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/sAhFiGjY

(Show channels 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)
https://pastebin.com/bmcdjzAr



(Extracted from logs 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)

https://pastebin.com/aGyY3m9F



(Show channels 624cac90-0a7b-11ec-b7d0-6f543a588268)

https://pastebin.com/0cnQaRhC



(Extracted from logs 624cac90-0a7b-11ec-b7d0-6f543a588268)
https://pastebin.com/atX0VXdZ



Please let us know if there is anything else we can provide that would be useful. Thank you.




On Mon, Aug 30, 2021 at 8:20 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Check the UUID of the session in question, and extract its logs, and let's diagnose it

On Mon, Aug 30, 2021 at 12:39 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Yes, that is what we are seeing when running uuid_kill. We cannot "replicate" the issue, but it happens multiple times a day for us across our servers. We have looked through logs across multiple systems, taken SIP traces to see if that showed anything. Last we got a Core dump of the running process after the failure. Nothing seems to stand out as the cause.


Are there any other debug steps you can think of or what other information to look at? We could capture all traffic on the server, or recompile with special debug flags whatever it is that might help.

Steps we have taken:
sofia loglevel all 9
sofia global siptrace on
fsctl loglevel 7
# Set environment variable TPORT_LOG=1




On Sat, Aug 28, 2021 at 12:54 AM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
It's an issue we've been searching for, if you uuid_kill it, and it says no session, it only means the record was NOT removed from the sqlite database.

Nobody can seem to replicate the steps required to get into that condition, maybe you could look thru the logs and see if you can replicate what ever happened on that lingering UUID


/b




On Fri, Aug 27, 2021 at 5:40 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
We recently started having a problem with our FreeSWITCH servers where calls are working well, with about 200 calls operating sometimes for a couple of days, and then something happens and all call progress stops. Then any new inbound calls that come into the server get stuck in initial status, new inbound get stuck at RINGING. This happens until the max sessions are hit and of course nothing else works from there. Our thought is something is creating a lock and stopping anything else from progressing.
We are seeing the stuck sessions using "show channels". Doing a uuid_kill on the calls seems to make no difference to the output of "show channels", but we have noticed in the logs that some events do happen to some of the calls.
We had ran into a similar issue when writing a custom C module and made some mistakes with session_rwlock etc. But in this case, all of this is using LUA.
Some general information:
FreeSWITCH Versions 1.10.6 and 1.10.1
Modules used:
LUA, xml_curl(dialplan), event_socket,

Most of this is just application playing audio files and or creating a new call to bridge to.

Really looking for some help on what debugging we can enable to help identify what is getting stuck to identify where a bug might be on our usage or possible issue with FreeSWITCH.

We have captured a core dump of the running process when it got into the bad state as well, if that is helpful, we can send it in.

Thank you for your help.


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]


















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]
















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]














_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com


--
Regards,


David Villasmilemail: david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)
phone: +34669448337


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com
Back to top
david.villasmil.work a...
Guest





PostPosted: Sat Sep 04, 2021 12:39 pm    Post subject: [Freeswitch-users] FreeSWITCH Stops Responding Reply with quote

That is correct. But we who have worked with FS since around 2002 know this, newer people get all worked up until they’re told this Smile


It’s a learning thing..

On Sat, 4 Sep 2021 at 17:31, Ken Rice <krice@freeswitch.org (krice@freeswitch.org)> wrote:

Quote:
“stuck channels” usually arent stuck once you realize things like “show calls” and “show channels” are selects against the coredb. this usually happens on busy systems timing put waiting for db delete’s to run that fail or get skipped.

Sent from my iPhone

Quote:
On Sep 4, 2021, at 05:22, David Villasmil <david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)> wrote:



Quote:
Hello all,


Interesting point about Gentoo, but I highly doubt it’s OS related. I’ve been living this issue (about those “stuck channels”, not a frozen FS, that’s indeed weird) for year and years. I’ve seen in on CentOS, Debian, Ubuntu, etc. 


In my experience this became an issue with the rise of VM. Back in the day you would see this every now and then, like months apart (this, or a crashed MySQL database), but wasn’t big enough to be a critical problem. Today day it’s basically an everyday thing if you have enough nodes.



On Sat, 4 Sep 2021 at 00:15, Ken Rice <krice@freeswitch.org (krice@freeswitch.org)> wrote:

Quote:
Having massive experience here, you might try replacing your linux distro.

 (Yes i know some people are going to take this as brushing off the problem or trying to start a distro war. If this starts a distro war I will moderate you and only allow replies are not headed to a flame war.)

However, as someone who has ran an extremely large Gentoo based cluster, we had to abandon gentoo due to random crashes, unexplainable bugs that could not be reproduced or explained. 


If you can not replace the distro I VERY HiGHLY recommend running vanilla libc and other libs without the gentoo portage included patches. (hint the use flag: vanilla)

Ken
Sent from my iPhone

Quote:
On Sep 3, 2021, at 14:49, Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:





Quote:
Quote:
We have not been able to reproduce this in testing. It only happens in our production environment.
Are there any other debug options that could be enabled, or is there any code in FreeSWITCH that you think may be causing this that we could look into. We could get you access to a server if that would help.
Server Details:
Gentoo-5.10.27
RAM: 15GB
CPU: 4
FreeSWITCH-1-.10.6
Sqlite-3.34.1



On Thu, Sep 2, 2021 at 7:24 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
And these exhibit the issue on command?

On Wed, Sep 1, 2021 at 5:06 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here are two originate samples.


https://pastebin.com/YKRKrJLu


https://pastebin.com/TEX3yfX6





On Wed, Sep 1, 2021 at 8:00 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Can you try to create one where this happens without all this mess?

On Wed, Sep 1, 2021 at 2:41 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here is the LUA code regarding creating sessions and bridging. It is broken up into 5 sections.
The top part is the common functions, and the 4 sections below are different call scenarios.
https://pastebin.com/qqGw3tms









On Wed, Sep 1, 2021 at 3:53 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
I you do sessions from your lua, I need to see how you do that.

On Tue, Aug 31, 2021 at 4:20 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Quote:
Yes, it bridges the inbound call to an outbound one.(ProxyOutbound.luac)
https://pastebin.com/RDUMy473



On Tue, Aug 31, 2021 at 8:39 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Guessing you create sessions in lua? ProxyOutbound.luac?

On Tue, Aug 31, 2021 at 3:33 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
and to make sure when you uuid kill one it says it didn't exist but show channels had it still listed correct?

On Tue, Aug 31, 2021 at 2:58 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Hello,


We have got the logs for a few calls that were stuck.
(Show channels fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/NTnte7J7


(Extracted from logs fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/sAhFiGjY

(Show channels 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)
https://pastebin.com/bmcdjzAr



(Extracted from logs 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)

https://pastebin.com/aGyY3m9F



(Show channels 624cac90-0a7b-11ec-b7d0-6f543a588268)

https://pastebin.com/0cnQaRhC



(Extracted from logs 624cac90-0a7b-11ec-b7d0-6f543a588268)
https://pastebin.com/atX0VXdZ



Please let us know if there is anything else we can provide that would be useful. Thank you.




On Mon, Aug 30, 2021 at 8:20 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Check the UUID of the session in question, and extract its logs, and let's diagnose it

On Mon, Aug 30, 2021 at 12:39 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
 Yes, that is what we are seeing when running uuid_kill.  We cannot "replicate" the issue, but it happens multiple times a day for us across our servers.   We have looked through logs across multiple systems, taken SIP traces to see if that showed anything.  Last we got a Core dump of the running process after the failure. Nothing seems to stand out as the cause.

 
 Are there any other debug steps you can think of or what other information to look at?  We could capture all traffic on the server, or recompile with special debug flags whatever it is that might help.
 
Steps we have taken:
sofia loglevel all 9
sofia global siptrace on
fsctl loglevel 7
# Set environment variable TPORT_LOG=1




On Sat, Aug 28, 2021 at 12:54 AM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
It's an issue we've been searching for, if you uuid_kill it, and it says no session, it only means the record was NOT removed from the sqlite database.

Nobody can seem to replicate the steps required to get into that condition, maybe you could look thru the logs and see if you can replicate what ever happened on that lingering UUID


/b




On Fri, Aug 27, 2021 at 5:40 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
 We recently started having a problem with our FreeSWITCH servers where calls are working well, with about 200 calls operating sometimes for a couple of days, and then something happens and all call progress stops. Then any new inbound calls that come into the server get stuck in initial status, new inbound get stuck at RINGING. This happens until the max sessions are hit and of course nothing else works from there. Our thought is something is creating a lock and stopping anything else from progressing.
 We are seeing the stuck sessions using "show channels". Doing a uuid_kill on the calls seems to make no difference to the output of "show channels", but we have noticed in the logs that some events do happen to some of the calls.
 We had ran into a similar issue when writing a custom C module and made some mistakes with session_rwlock etc. But in this case, all of this is using LUA.  
Some general information:
FreeSWITCH Versions 1.10.6 and 1.10.1
Modules used:
LUA, xml_curl(dialplan), event_socket,

 Most of this is just application playing audio files and or creating a new call to bridge to.

 Really looking for some help on what debugging we can enable to help identify what is getting stuck to identify where a bug might be on our usage or possible issue with FreeSWITCH.

 We have captured a core dump of the running process when it got into the bad state as well, if that is helpful, we can send it in.

Thank you for your help.


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]


















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]
















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]














_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com


--
Regards,


David Villasmilemail: david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)
phone: +34669448337


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com

_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com


--
Regards,


David Villasmilemail: david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)
phone: +34669448337
Back to top
gmaruzz at gmail.com
Guest





PostPosted: Sat Sep 04, 2021 4:07 pm    Post subject: [Freeswitch-users] FreeSWITCH Stops Responding Reply with quote

"son, there was a time when we had to deal with Asterisk's DeadLocks..."




On Sat, Sep 4, 2021 at 7:53 PM David Villasmil <david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)> wrote:

Quote:
That is correct. But we who have worked with FS since around 2002 know this, newer people get all worked up until they’re told this Smile


It’s a learning thing..

On Sat, 4 Sep 2021 at 17:31, Ken Rice <krice@freeswitch.org (krice@freeswitch.org)> wrote:

Quote:
“stuck channels” usually arent stuck once you realize things like “show calls” and “show channels” are selects against the coredb. this usually happens on busy systems timing put waiting for db delete’s to run that fail or get skipped.

Sent from my iPhone

Quote:
On Sep 4, 2021, at 05:22, David Villasmil <david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)> wrote:



Quote:
Hello all,


Interesting point about Gentoo, but I highly doubt it’s OS related. I’ve been living this issue (about those “stuck channels”, not a frozen FS, that’s indeed weird) for year and years. I’ve seen in on CentOS, Debian, Ubuntu, etc. 


In my experience this became an issue with the rise of VM. Back in the day you would see this every now and then, like months apart (this, or a crashed MySQL database), but wasn’t big enough to be a critical problem. Today day it’s basically an everyday thing if you have enough nodes.



On Sat, 4 Sep 2021 at 00:15, Ken Rice <krice@freeswitch.org (krice@freeswitch.org)> wrote:

Quote:
Having massive experience here, you might try replacing your linux distro.

 (Yes i know some people are going to take this as brushing off the problem or trying to start a distro war. If this starts a distro war I will moderate you and only allow replies are not headed to a flame war.)

However, as someone who has ran an extremely large Gentoo based cluster, we had to abandon gentoo due to random crashes, unexplainable bugs that could not be reproduced or explained. 


If you can not replace the distro I VERY HiGHLY recommend running vanilla libc and other libs without the gentoo portage included patches. (hint the use flag: vanilla)

Ken
Sent from my iPhone

Quote:
On Sep 3, 2021, at 14:49, Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:





Quote:
Quote:
We have not been able to reproduce this in testing. It only happens in our production environment.
Are there any other debug options that could be enabled, or is there any code in FreeSWITCH that you think may be causing this that we could look into. We could get you access to a server if that would help.
Server Details:
Gentoo-5.10.27
RAM: 15GB
CPU: 4
FreeSWITCH-1-.10.6
Sqlite-3.34.1



On Thu, Sep 2, 2021 at 7:24 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
And these exhibit the issue on command?

On Wed, Sep 1, 2021 at 5:06 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here are two originate samples.


https://pastebin.com/YKRKrJLu


https://pastebin.com/TEX3yfX6





On Wed, Sep 1, 2021 at 8:00 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Can you try to create one where this happens without all this mess?

On Wed, Sep 1, 2021 at 2:41 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here is the LUA code regarding creating sessions and bridging. It is broken up into 5 sections.
The top part is the common functions, and the 4 sections below are different call scenarios.
https://pastebin.com/qqGw3tms









On Wed, Sep 1, 2021 at 3:53 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
I you do sessions from your lua, I need to see how you do that.

On Tue, Aug 31, 2021 at 4:20 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Quote:
Yes, it bridges the inbound call to an outbound one.(ProxyOutbound.luac)
https://pastebin.com/RDUMy473



On Tue, Aug 31, 2021 at 8:39 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Guessing you create sessions in lua? ProxyOutbound.luac?

On Tue, Aug 31, 2021 at 3:33 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
and to make sure when you uuid kill one it says it didn't exist but show channels had it still listed correct?

On Tue, Aug 31, 2021 at 2:58 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Hello,


We have got the logs for a few calls that were stuck.
(Show channels fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/NTnte7J7


(Extracted from logs fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/sAhFiGjY

(Show channels 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)
https://pastebin.com/bmcdjzAr



(Extracted from logs 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)

https://pastebin.com/aGyY3m9F



(Show channels 624cac90-0a7b-11ec-b7d0-6f543a588268)

https://pastebin.com/0cnQaRhC



(Extracted from logs 624cac90-0a7b-11ec-b7d0-6f543a588268)
https://pastebin.com/atX0VXdZ



Please let us know if there is anything else we can provide that would be useful. Thank you.




On Mon, Aug 30, 2021 at 8:20 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Check the UUID of the session in question, and extract its logs, and let's diagnose it

On Mon, Aug 30, 2021 at 12:39 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
 Yes, that is what we are seeing when running uuid_kill.  We cannot "replicate" the issue, but it happens multiple times a day for us across our servers.   We have looked through logs across multiple systems, taken SIP traces to see if that showed anything.  Last we got a Core dump of the running process after the failure. Nothing seems to stand out as the cause.

 
 Are there any other debug steps you can think of or what other information to look at?  We could capture all traffic on the server, or recompile with special debug flags whatever it is that might help.
 
Steps we have taken:
sofia loglevel all 9
sofia global siptrace on
fsctl loglevel 7
# Set environment variable TPORT_LOG=1




On Sat, Aug 28, 2021 at 12:54 AM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
It's an issue we've been searching for, if you uuid_kill it, and it says no session, it only means the record was NOT removed from the sqlite database.

Nobody can seem to replicate the steps required to get into that condition, maybe you could look thru the logs and see if you can replicate what ever happened on that lingering UUID


/b




On Fri, Aug 27, 2021 at 5:40 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
 We recently started having a problem with our FreeSWITCH servers where calls are working well, with about 200 calls operating sometimes for a couple of days, and then something happens and all call progress stops. Then any new inbound calls that come into the server get stuck in initial status, new inbound get stuck at RINGING. This happens until the max sessions are hit and of course nothing else works from there. Our thought is something is creating a lock and stopping anything else from progressing.
 We are seeing the stuck sessions using "show channels". Doing a uuid_kill on the calls seems to make no difference to the output of "show channels", but we have noticed in the logs that some events do happen to some of the calls.
 We had ran into a similar issue when writing a custom C module and made some mistakes with session_rwlock etc. But in this case, all of this is using LUA.  
Some general information:
FreeSWITCH Versions 1.10.6 and 1.10.1
Modules used:
LUA, xml_curl(dialplan), event_socket,

 Most of this is just application playing audio files and or creating a new call to bridge to.

 Really looking for some help on what debugging we can enable to help identify what is getting stuck to identify where a bug might be on our usage or possible issue with FreeSWITCH.

 We have captured a core dump of the running process when it got into the bad state as well, if that is helpful, we can send it in.

Thank you for your help.


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]


















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]
















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]














_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com


--
Regards,


David Villasmilemail: david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)
phone: +34669448337


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com

_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com


--
Regards,


David Villasmilemail: david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)
phone: +34669448337


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



--
Sincerely,

Giovanni Maruzzelli
OpenTelecom.IT
cell: +39 347 266 56 18
Back to top
shaun at sysconfig.cloud
Guest





PostPosted: Tue Sep 07, 2021 12:13 pm    Post subject: [Freeswitch-users] FreeSWITCH Stops Responding Reply with quote

Back to top
brian at freeswitch.com
Guest





PostPosted: Tue Sep 07, 2021 7:45 pm    Post subject: [Freeswitch-users] FreeSWITCH Stops Responding Reply with quote

Ya'll are too funny...  but I'd love to replicate and fix this issue.

On Sat, Sep 4, 2021 at 4:12 PM Giovanni Maruzzelli <gmaruzz@gmail.com (gmaruzz@gmail.com)> wrote:

Quote:
"son, there was a time when we had to deal with Asterisk's DeadLocks..."




On Sat, Sep 4, 2021 at 7:53 PM David Villasmil <david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)> wrote:

Quote:
That is correct. But we who have worked with FS since around 2002 know this, newer people get all worked up until they’re told this Smile


It’s a learning thing..

On Sat, 4 Sep 2021 at 17:31, Ken Rice <krice@freeswitch.org (krice@freeswitch.org)> wrote:

Quote:
“stuck channels” usually arent stuck once you realize things like “show calls” and “show channels” are selects against the coredb. this usually happens on busy systems timing put waiting for db delete’s to run that fail or get skipped.

Sent from my iPhone

Quote:
On Sep 4, 2021, at 05:22, David Villasmil <david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)> wrote:



Quote:
Hello all,


Interesting point about Gentoo, but I highly doubt it’s OS related. I’ve been living this issue (about those “stuck channels”, not a frozen FS, that’s indeed weird) for year and years. I’ve seen in on CentOS, Debian, Ubuntu, etc. 


In my experience this became an issue with the rise of VM. Back in the day you would see this every now and then, like months apart (this, or a crashed MySQL database), but wasn’t big enough to be a critical problem. Today day it’s basically an everyday thing if you have enough nodes.



On Sat, 4 Sep 2021 at 00:15, Ken Rice <krice@freeswitch.org (krice@freeswitch.org)> wrote:

Quote:
Having massive experience here, you might try replacing your linux distro.

 (Yes i know some people are going to take this as brushing off the problem or trying to start a distro war. If this starts a distro war I will moderate you and only allow replies are not headed to a flame war.)

However, as someone who has ran an extremely large Gentoo based cluster, we had to abandon gentoo due to random crashes, unexplainable bugs that could not be reproduced or explained. 


If you can not replace the distro I VERY HiGHLY recommend running vanilla libc and other libs without the gentoo portage included patches. (hint the use flag: vanilla)

Ken
Sent from my iPhone

Quote:
On Sep 3, 2021, at 14:49, Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:





Quote:
Quote:
We have not been able to reproduce this in testing. It only happens in our production environment.
Are there any other debug options that could be enabled, or is there any code in FreeSWITCH that you think may be causing this that we could look into. We could get you access to a server if that would help.
Server Details:
Gentoo-5.10.27
RAM: 15GB
CPU: 4
FreeSWITCH-1-.10.6
Sqlite-3.34.1



On Thu, Sep 2, 2021 at 7:24 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
And these exhibit the issue on command?

On Wed, Sep 1, 2021 at 5:06 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here are two originate samples.


https://pastebin.com/YKRKrJLu


https://pastebin.com/TEX3yfX6





On Wed, Sep 1, 2021 at 8:00 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Can you try to create one where this happens without all this mess?

On Wed, Sep 1, 2021 at 2:41 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here is the LUA code regarding creating sessions and bridging. It is broken up into 5 sections.
The top part is the common functions, and the 4 sections below are different call scenarios.
https://pastebin.com/qqGw3tms









On Wed, Sep 1, 2021 at 3:53 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
I you do sessions from your lua, I need to see how you do that.

On Tue, Aug 31, 2021 at 4:20 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Quote:
Yes, it bridges the inbound call to an outbound one.(ProxyOutbound.luac)
https://pastebin.com/RDUMy473



On Tue, Aug 31, 2021 at 8:39 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Guessing you create sessions in lua? ProxyOutbound.luac?

On Tue, Aug 31, 2021 at 3:33 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
and to make sure when you uuid kill one it says it didn't exist but show channels had it still listed correct?

On Tue, Aug 31, 2021 at 2:58 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Hello,


We have got the logs for a few calls that were stuck.
(Show channels fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/NTnte7J7


(Extracted from logs fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/sAhFiGjY

(Show channels 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)
https://pastebin.com/bmcdjzAr



(Extracted from logs 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)

https://pastebin.com/aGyY3m9F



(Show channels 624cac90-0a7b-11ec-b7d0-6f543a588268)

https://pastebin.com/0cnQaRhC



(Extracted from logs 624cac90-0a7b-11ec-b7d0-6f543a588268)
https://pastebin.com/atX0VXdZ



Please let us know if there is anything else we can provide that would be useful. Thank you.




On Mon, Aug 30, 2021 at 8:20 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Check the UUID of the session in question, and extract its logs, and let's diagnose it

On Mon, Aug 30, 2021 at 12:39 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
 Yes, that is what we are seeing when running uuid_kill.  We cannot "replicate" the issue, but it happens multiple times a day for us across our servers.   We have looked through logs across multiple systems, taken SIP traces to see if that showed anything.  Last we got a Core dump of the running process after the failure. Nothing seems to stand out as the cause.

 
 Are there any other debug steps you can think of or what other information to look at?  We could capture all traffic on the server, or recompile with special debug flags whatever it is that might help.
 
Steps we have taken:
sofia loglevel all 9
sofia global siptrace on
fsctl loglevel 7
# Set environment variable TPORT_LOG=1




On Sat, Aug 28, 2021 at 12:54 AM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
It's an issue we've been searching for, if you uuid_kill it, and it says no session, it only means the record was NOT removed from the sqlite database.

Nobody can seem to replicate the steps required to get into that condition, maybe you could look thru the logs and see if you can replicate what ever happened on that lingering UUID


/b




On Fri, Aug 27, 2021 at 5:40 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
 We recently started having a problem with our FreeSWITCH servers where calls are working well, with about 200 calls operating sometimes for a couple of days, and then something happens and all call progress stops. Then any new inbound calls that come into the server get stuck in initial status, new inbound get stuck at RINGING. This happens until the max sessions are hit and of course nothing else works from there. Our thought is something is creating a lock and stopping anything else from progressing.
 We are seeing the stuck sessions using "show channels". Doing a uuid_kill on the calls seems to make no difference to the output of "show channels", but we have noticed in the logs that some events do happen to some of the calls.
 We had ran into a similar issue when writing a custom C module and made some mistakes with session_rwlock etc. But in this case, all of this is using LUA.  
Some general information:
FreeSWITCH Versions 1.10.6 and 1.10.1
Modules used:
LUA, xml_curl(dialplan), event_socket,

 Most of this is just application playing audio files and or creating a new call to bridge to.

 Really looking for some help on what debugging we can enable to help identify what is getting stuck to identify where a bug might be on our usage or possible issue with FreeSWITCH.

 We have captured a core dump of the running process when it got into the bad state as well, if that is helpful, we can send it in.

Thank you for your help.


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]


















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]
















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]














_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com


--
Regards,


David Villasmilemail: david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)
phone: +34669448337


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com

_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com


--
Regards,


David Villasmilemail: david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)
phone: +34669448337


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



--
Sincerely,

Giovanni Maruzzelli
OpenTelecom.IT
cell: +39 347 266 56 18


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]
Back to top
cary at ytel.com
Guest





PostPosted: Thu Sep 09, 2021 11:56 am    Post subject: [Freeswitch-users] FreeSWITCH Stops Responding Reply with quote

We have set up a Debian 10 FreeSWITCH-1.10.6 server for testing/debugging. 


On Sat, Sep 4, 2021 at 5:51 PM David Villasmil <david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)> wrote:

Quote:
That is correct. But we who have worked with FS since around 2002 know this, newer people get all worked up until they’re told this Smile


It’s a learning thing..

On Sat, 4 Sep 2021 at 17:31, Ken Rice <krice@freeswitch.org (krice@freeswitch.org)> wrote:

Quote:
“stuck channels” usually arent stuck once you realize things like “show calls” and “show channels” are selects against the coredb. this usually happens on busy systems timing put waiting for db delete’s to run that fail or get skipped.

Sent from my iPhone

Quote:
On Sep 4, 2021, at 05:22, David Villasmil <david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)> wrote:



Quote:
Hello all,


Interesting point about Gentoo, but I highly doubt it’s OS related. I’ve been living this issue (about those “stuck channels”, not a frozen FS, that’s indeed weird) for year and years. I’ve seen in on CentOS, Debian, Ubuntu, etc. 


In my experience this became an issue with the rise of VM. Back in the day you would see this every now and then, like months apart (this, or a crashed MySQL database), but wasn’t big enough to be a critical problem. Today day it’s basically an everyday thing if you have enough nodes.



On Sat, 4 Sep 2021 at 00:15, Ken Rice <krice@freeswitch.org (krice@freeswitch.org)> wrote:

Quote:
Having massive experience here, you might try replacing your linux distro.

 (Yes i know some people are going to take this as brushing off the problem or trying to start a distro war. If this starts a distro war I will moderate you and only allow replies are not headed to a flame war.)

However, as someone who has ran an extremely large Gentoo based cluster, we had to abandon gentoo due to random crashes, unexplainable bugs that could not be reproduced or explained. 


If you can not replace the distro I VERY HiGHLY recommend running vanilla libc and other libs without the gentoo portage included patches. (hint the use flag: vanilla)

Ken
Sent from my iPhone

Quote:
On Sep 3, 2021, at 14:49, Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:





Quote:
Quote:
We have not been able to reproduce this in testing. It only happens in our production environment.
Are there any other debug options that could be enabled, or is there any code in FreeSWITCH that you think may be causing this that we could look into. We could get you access to a server if that would help.
Server Details:
Gentoo-5.10.27
RAM: 15GB
CPU: 4
FreeSWITCH-1-.10.6
Sqlite-3.34.1



On Thu, Sep 2, 2021 at 7:24 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
And these exhibit the issue on command?

On Wed, Sep 1, 2021 at 5:06 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here are two originate samples.


https://pastebin.com/YKRKrJLu


https://pastebin.com/TEX3yfX6





On Wed, Sep 1, 2021 at 8:00 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Can you try to create one where this happens without all this mess?

On Wed, Sep 1, 2021 at 2:41 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Here is the LUA code regarding creating sessions and bridging. It is broken up into 5 sections.
The top part is the common functions, and the 4 sections below are different call scenarios.
https://pastebin.com/qqGw3tms









On Wed, Sep 1, 2021 at 3:53 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
I you do sessions from your lua, I need to see how you do that.

On Tue, Aug 31, 2021 at 4:20 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Quote:
Yes, it bridges the inbound call to an outbound one.(ProxyOutbound.luac)
https://pastebin.com/RDUMy473



On Tue, Aug 31, 2021 at 8:39 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Guessing you create sessions in lua? ProxyOutbound.luac?

On Tue, Aug 31, 2021 at 3:33 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
and to make sure when you uuid kill one it says it didn't exist but show channels had it still listed correct?

On Tue, Aug 31, 2021 at 2:58 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
Hello,


We have got the logs for a few calls that were stuck.
(Show channels fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/NTnte7J7


(Extracted from logs fe8f206c-2bd9-4af2-b329-35f53228a38a)
https://pastebin.com/sAhFiGjY

(Show channels 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)
https://pastebin.com/bmcdjzAr



(Extracted from logs 46c6e04b-a58e-4b82-9968-fb0e9c2c0fb3)

https://pastebin.com/aGyY3m9F



(Show channels 624cac90-0a7b-11ec-b7d0-6f543a588268)

https://pastebin.com/0cnQaRhC



(Extracted from logs 624cac90-0a7b-11ec-b7d0-6f543a588268)
https://pastebin.com/atX0VXdZ



Please let us know if there is anything else we can provide that would be useful. Thank you.




On Mon, Aug 30, 2021 at 8:20 PM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
Check the UUID of the session in question, and extract its logs, and let's diagnose it

On Mon, Aug 30, 2021 at 12:39 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
 Yes, that is what we are seeing when running uuid_kill.  We cannot "replicate" the issue, but it happens multiple times a day for us across our servers.   We have looked through logs across multiple systems, taken SIP traces to see if that showed anything.  Last we got a Core dump of the running process after the failure. Nothing seems to stand out as the cause.

 
 Are there any other debug steps you can think of or what other information to look at?  We could capture all traffic on the server, or recompile with special debug flags whatever it is that might help.
 
Steps we have taken:
sofia loglevel all 9
sofia global siptrace on
fsctl loglevel 7
# Set environment variable TPORT_LOG=1




On Sat, Aug 28, 2021 at 12:54 AM Brian West <brian@freeswitch.com (brian@freeswitch.com)> wrote:

Quote:
It's an issue we've been searching for, if you uuid_kill it, and it says no session, it only means the record was NOT removed from the sqlite database.

Nobody can seem to replicate the steps required to get into that condition, maybe you could look thru the logs and see if you can replicate what ever happened on that lingering UUID


/b




On Fri, Aug 27, 2021 at 5:40 PM Cary Shouse <cary@ytel.com (cary@ytel.com)> wrote:

Quote:
 We recently started having a problem with our FreeSWITCH servers where calls are working well, with about 200 calls operating sometimes for a couple of days, and then something happens and all call progress stops. Then any new inbound calls that come into the server get stuck in initial status, new inbound get stuck at RINGING. This happens until the max sessions are hit and of course nothing else works from there. Our thought is something is creating a lock and stopping anything else from progressing.
 We are seeing the stuck sessions using "show channels". Doing a uuid_kill on the calls seems to make no difference to the output of "show channels", but we have noticed in the logs that some events do happen to some of the calls.
 We had ran into a similar issue when writing a custom C module and made some mistakes with session_rwlock etc. But in this case, all of this is using LUA.  
Some general information:
FreeSWITCH Versions 1.10.6 and 1.10.1
Modules used:
LUA, xml_curl(dialplan), event_socket,

 Most of this is just application playing audio files and or creating a new call to bridge to.

 Really looking for some help on what debugging we can enable to help identify what is getting stuck to identify where a bug might be on our usage or possible issue with FreeSWITCH.

 We have captured a core dump of the running process when it got into the bad state as well, if that is helpful, we can send it in.

Thank you for your help.


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]


















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]
















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]

















--



Brian West | Co-founder and Developer
Need Commercial support? email sales@freeswitch.com (sales@freeswitch.com)
FreeSWITCH Solutions | 17345 Civic Drive #2531 Brookfield, WI 53045
Email: brian@freeswitch.com (brian@freeswitch.com)
Mobile: 918-424-9378
Website: https://www.FreeSWITCH.com
[/url] [url=https://twitter.com/freeswitch]














_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com


--
Regards,


David Villasmilemail: david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)
phone: +34669448337


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com

_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com


--
Regards,


David Villasmilemail: david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)
phone: +34669448337


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com
Back to top
Display posts from previous:   
Post new topic   Reply to topic    VoIP Mailing List Archives Forum Index -> freeSWITCH Users All times are GMT - 5 Hours
Goto page Previous  1, 2
Page 2 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group

VoiceMeUp - Corporate & Wholesale VoIP Services