Sponsor: VoiceMeUp - Corporate & Wholesale VoIP Services

VoIP Mailing List Archives
Mailing list archives for the VoIP community
 SearchSearch 

[Freeswitch-users] BLF under high load


 
Post new topic   Reply to topic    VoIP Mailing List Archives Forum Index -> freeSWITCH Users
View previous topic :: View next topic  
Author Message
lists at telefaks.de
Guest





PostPosted: Wed Mar 10, 2021 11:16 am    Post subject: [Freeswitch-users] BLF under high load Reply with quote

Hello


we have a Freeswitch installation for a multi tenant environment. We have extensively tested our Freeswitch for all call scenarios and all worked fine. This is the good news.

However, under higher load (>250 Channels and  >6000 Subscriptions), presence does not seem to work reliably. I've found some ealier posts on this issue, which seem to lead to the following 2 solutions
  • use PostgreSQL (we use Mariadb via ODBC, also tested mod_mariadb, which made things worse)
  • Move presence handling to Kamailio

As your environment is based on MySQL, we do not want to move to PostgreSQL without knowing, whether this will improve the situation.
So far we are catching events via ESL or even grepping the network and correct the status of presence informatons sent to the phone. However we discover, that even if we send corrected PRESENCE_IN via event socket, our Freeswitch keeps on sending the wrong BLF information afterwards to the phone. Means BLFs are blinking or in state red event if the watched phone is not in a call. So at present we are grepping all presence informations on the network, crosscheck this with the channels table and resend the corrected BLF status via event_socket if needed. But still this is not sufficiant.

So I am now asking the community: What is your approach to overcome this?
  • Do newer Freeswitch versions perform better for this case?(Ours is "FreeSWITCH Version 1.10.5-release+git~20200818T185121Z")
  • which work have you done to overcome this?
  • does anybody have an idea how to query all internal presence status from Freewitch or maybe how to correct it?

Thank you in advance. Any help is appreciated.

Peter
Back to top
grcamauer at gmail.com
Guest





PostPosted: Wed Mar 10, 2021 12:41 pm    Post subject: [Freeswitch-users] BLF under high load Reply with quote

From your description, it sounds like the DB is not keeping up with the updates FreeSwitch is making.  Have you tried tuning the db?  More memory, faster/more disks?

Guillermo


On Wed, Mar 10, 2021 at 1:16 PM Peter Steinbach <lists@telefaks.de (lists@telefaks.de)> wrote:

Quote:

Hello


we have a Freeswitch installation for a multi tenant environment. We have extensively tested our Freeswitch for all call scenarios and all worked fine. This is the good news.

However, under higher load (>250 Channels and  >6000 Subscriptions), presence does not seem to work reliably. I've found some ealier posts on this issue, which seem to lead to the following 2 solutions
  • use PostgreSQL (we use Mariadb via ODBC, also tested mod_mariadb, which made things worse)
  • Move presence handling to Kamailio

As your environment is based on MySQL, we do not want to move to PostgreSQL without knowing, whether this will improve the situation.
So far we are catching events via ESL or even grepping the network and correct the status of presence informatons sent to the phone. However we discover, that even if we send corrected PRESENCE_IN via event socket, our Freeswitch keeps on sending the wrong BLF information afterwards to the phone. Means BLFs are blinking or in state red event if the watched phone is not in a call. So at present we are grepping all presence informations on the network, crosscheck this with the channels table and resend the corrected BLF status via event_socket if needed. But still this is not sufficiant.

So I am now asking the community: What is your approach to overcome this?
  • Do newer Freeswitch versions perform better for this case?(Ours is "FreeSWITCH Version 1.10.5-release+git~20200818T185121Z")
  • which work have you done to overcome this?
  • does anybody have an idea how to query all internal presence status from Freewitch or maybe how to correct it?

Thank you in advance. Any help is appreciated.

Peter
_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



--
Guillermo Ruiz Camauer
Back to top
lists at telefaks.de
Guest





PostPosted: Thu Mar 11, 2021 7:43 am    Post subject: [Freeswitch-users] BLF under high load Reply with quote

We made some investigations on our Production database. We set mysql slow log to 0.02 sec and we did not discover any long running qeuries concerning presence.
The command

    select * from INFORMATION_SCHEMA.PROCESSLIST  where command <> 'Sleep';

shows, that presence queries run between 0.1 and 1 milliseconds. So I do not really feel, that the Mysql database is the bottleneck here. MySQL show about 10-15%CPU on an 8 core machine. Freeswitch runs at about 50% CPU on a different 8 core machine.
Nevertheless we have severe problems with presence. I have e.g. a Snom phone and a Yealink phone on my desk with a BLF for a 3rd phone, which is not in a call at present. BLF on Snom is "on" and BLF on Yealink is "blinking". Both should be "off". This is very strange to me. How can this happen? BTW when this system was under no load, before we migrated our customers, everything worked fine.

So any help is very much appreciated here.

Another question is still:
  • does anybody have an idea how to query all internal presence status from Freewitch or maybe how to correct it?
    • e.g. "sofia_presence_data list number@domain" returns e.g.

status|rpid|user_agent|network_ip|network_port
unknown|unknown|N510 IP PRO/42.250.00.000.000|xx.xxx.xxx.xx|21549
unknown|unknown|Yealink SIP-T41S 66.83.0.35 805EC025FDA9|xx.xx.xxx.xxx|20901


Quote:
  • and "sofia_presence_data status number@domain" always returns

unknownunknown

Quote:
  • Database table "sip_subscriptions" however is fine



Best regards
Peter


Am 10.03.21 um 18:07 schrieb Guillermo Ruiz Camauer:

Quote:
From your description, it sounds like the DB is not keeping up with the updates FreeSwitch is making.  Have you tried tuning the db?  More memory, faster/more disks?

Guillermo


On Wed, Mar 10, 2021 at 1:16 PM Peter Steinbach <lists@telefaks.de (lists@telefaks.de)> wrote:

Quote:

Hello


we have a Freeswitch installation for a multi tenant environment. We have extensively tested our Freeswitch for all call scenarios and all worked fine. This is the good news.

However, under higher load (>250 Channels and  >6000 Subscriptions), presence does not seem to work reliably. I've found some ealier posts on this issue, which seem to lead to the following 2 solutions
  • use PostgreSQL (we use Mariadb via ODBC, also tested mod_mariadb, which made things worse)
  • Move presence handling to Kamailio

As your environment is based on MySQL, we do not want to move to PostgreSQL without knowing, whether this will improve the situation.
So far we are catching events via ESL or even grepping the network and correct the status of presence informatons sent to the phone. However we discover, that even if we send corrected PRESENCE_IN via event socket, our Freeswitch keeps on sending the wrong BLF information afterwards to the phone. Means BLFs are blinking or in state red event if the watched phone is not in a call. So at present we are grepping all presence informations on the network, crosscheck this with the channels table and resend the corrected BLF status via event_socket if needed. But still this is not sufficiant.

So I am now asking the community: What is your approach to overcome this?
  • Do newer Freeswitch versions perform better for this case?(Ours is "FreeSWITCH Version 1.10.5-release+git~20200818T185121Z")
  • which work have you done to overcome this?
  • does anybody have an idea how to query all internal presence status from Freewitch or maybe how to correct it?

Thank you in advance. Any help is appreciated.

Peter
_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



--
Guillermo Ruiz Camauer


Quote:
_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com

Back to top
lists at telefaks.de
Guest





PostPosted: Thu Mar 11, 2021 8:47 am    Post subject: [Freeswitch-users] BLF under high load Reply with quote

While debugging presence, I found a lot of the following lines on the freeswitch console


2021-03-11 13:26:34.496493 [CRIT] sofia_presence.c:1377 CHECK SQL: 9886@customer.mydomain.net (9886@customer.mydomain.net) [select state,status,rpid,presence_id,uuid from sip_dialogs where call_info_state != 'seized' and hostname='fs02.mydomain.net' and profile_name='internal' and ((sip_from_user='9886' and sip_from_host='customer.mydomain.net') or presence_id='9886@customer.mydomain.net (9886@customer.mydomain.net)') order by rcd desc]
2021-03-11 13:26:34.496493 [ERR] sofia_presence.c:1435 PRES SQL update sip_subscriptions set version=version+1 where hostname='fs02.mydomain.net' and profile_name='internal' and sip_subscriptions.event != 'line-seize' and sip_subscriptions.proto='sip' and (event='presence' or event='dialog') and sub_to_user='9886' and (sub_to_host='customer.mydomain.net' or sub_to_host='xxx.xx.x.xxx' or sub_to_host='N/A' or presence_hosts like '%customer.mydomain.net%') and (sip_subscriptions.profile_name = 'internal' or presence_hosts like '%customer.mydomain.net%')



  • The first query returns 0 entries, when I run it manually, line is marked as [CRIT]
    • my impression is that this query is always empty because _all_ values of sip_dialogs.call_info_state are empty, allthough we have about 100 entries in sip_dialogs table. Running 100k continuous queries against this table, I could not find any other value than '' for sip_dialogs.call_info_state
  • The second query updates 7 entries, when I run it manually, but line is marked as [ERR]


Maybe this info creates some new ideas.


Best regards
Peter





Am 11.03.21 um 13:08 schrieb Peter Steinbach:

Quote:

We made some investigations on our Production database. We set mysql slow log to 0.02 sec and we did not discover any long running qeuries concerning presence.
The command

    select * from INFORMATION_SCHEMA.PROCESSLIST  where command <> 'Sleep';

shows, that presence queries run between 0.1 and 1 milliseconds. So I do not really feel, that the Mysql database is the bottleneck here. MySQL show about 10-15%CPU on an 8 core machine. Freeswitch runs at about 50% CPU on a different 8 core machine.
Nevertheless we have severe problems with presence. I have e.g. a Snom phone and a Yealink phone on my desk with a BLF for a 3rd phone, which is not in a call at present. BLF on Snom is "on" and BLF on Yealink is "blinking". Both should be "off". This is very strange to me. How can this happen? BTW when this system was under no load, before we migrated our customers, everything worked fine.

So any help is very much appreciated here.

Another question is still:
  • does anybody have an idea how to query all internal presence status from Freewitch or maybe how to correct it?
    • e.g. "sofia_presence_data list number@domain" returns e.g.

status|rpid|user_agent|network_ip|network_port
unknown|unknown|N510 IP PRO/42.250.00.000.000|xx.xxx.xxx.xx|21549
unknown|unknown|Yealink SIP-T41S 66.83.0.35 805EC025FDA9|xx.xx.xxx.xxx|20901


Quote:
  • and "sofia_presence_data status number@domain" always returns

unknownunknown

Quote:
  • Database table "sip_subscriptions" however is fine



Best regards
Peter


Am 10.03.21 um 18:07 schrieb Guillermo Ruiz Camauer:

Quote:
From your description, it sounds like the DB is not keeping up with the updates FreeSwitch is making.  Have you tried tuning the db?  More memory, faster/more disks?

Guillermo


On Wed, Mar 10, 2021 at 1:16 PM Peter Steinbach <lists@telefaks.de (lists@telefaks.de)> wrote:

Quote:

Hello


we have a Freeswitch installation for a multi tenant environment. We have extensively tested our Freeswitch for all call scenarios and all worked fine. This is the good news.

However, under higher load (>250 Channels and  >6000 Subscriptions), presence does not seem to work reliably. I've found some ealier posts on this issue, which seem to lead to the following 2 solutions
  • use PostgreSQL (we use Mariadb via ODBC, also tested mod_mariadb, which made things worse)
  • Move presence handling to Kamailio

As your environment is based on MySQL, we do not want to move to PostgreSQL without knowing, whether this will improve the situation.
So far we are catching events via ESL or even grepping the network and correct the status of presence informatons sent to the phone. However we discover, that even if we send corrected PRESENCE_IN via event socket, our Freeswitch keeps on sending the wrong BLF information afterwards to the phone. Means BLFs are blinking or in state red event if the watched phone is not in a call. So at present we are grepping all presence informations on the network, crosscheck this with the channels table and resend the corrected BLF status via event_socket if needed. But still this is not sufficiant.

So I am now asking the community: What is your approach to overcome this?
  • Do newer Freeswitch versions perform better for this case?(Ours is "FreeSWITCH Version 1.10.5-release+git~20200818T185121Z")
  • which work have you done to overcome this?
  • does anybody have an idea how to query all internal presence status from Freewitch or maybe how to correct it?

Thank you in advance. Any help is appreciated.

Peter
_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



--
Guillermo Ruiz Camauer


Quote:
_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com
Back to top
Display posts from previous:   
Post new topic   Reply to topic    VoIP Mailing List Archives Forum Index -> freeSWITCH Users All times are GMT - 5 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group

VoiceMeUp - Corporate & Wholesale VoIP Services