Sponsor: VoiceMeUp - Corporate & Wholesale VoIP Services

VoIP Mailing List Archives
Mailing list archives for the VoIP community
 SearchSearch 

[Freeswitch-users] Jitter and dropping MOS score


 
Post new topic   Reply to topic    VoIP Mailing List Archives Forum Index -> freeSWITCH Users
View previous topic :: View next topic  
Author Message
dragos at freeswitch.org
Guest





PostPosted: Wed Oct 06, 2021 6:07 am    Post subject: [Freeswitch-users] Jitter and dropping MOS score Reply with quote

if it really consumes 100% CPU with few calls then it's abnormal. kill it with SIGABRT (make sure it will dump core before that), get a backtrace with gdb and open a github issue. 
before doing this you need to make sure it's really eating 100 % CPU, this screenshot is not very relevant. 
Also, you need to make sure you don't have real packet loss. Perhaps it's a matter of dimensioning, eg: how many simultaneous calls do you expect to have on that hardware, what codec, etc. 
 


On Wed, Oct 6, 2021 at 2:26 AM Christoph Russow <russow@emtex.de (russow@emtex.de)> wrote:

Quote:
Hi all,

we are currently experiencing jitter and dropping MOS score on an
increasing amount of calls on our system.

i can't see any increased load on our system (neither CPU nor I/O wise).

Current stats:

~ 15Mbit/s network traffic (on a 1Gbit/s network)
load average: 1,10, 0,82, 0,91 (16 core machine)
memory used: 2,26GB/16GB

freeswitch@lc1> status
UP 0 years, 12 days, 13 hours, 30 minutes, 24 seconds, 649 milliseconds,
831 microseconds
FreeSWITCH (Version 1.10.1 -release-12-f9990221e6 64bit) is ready
137634 session(s) since startup
218 session(s) - peak 245, last 5min 222
1 session(s) per Sec out of max 30, peak 8, last 5min 2
1000 session(s) max
min idle cpu 0.00/95.37
Current Stack Size/Max 244K/244K

System ist not virtualized.

the only "strange" thing i see is the freeswitch "main"process that
consumes (according to htop) >=100% cpu (see attached screenshot).
Can this be the problem? if yes what can i do against it.

if not what are possible bottlenecks i should weed out?

if you need any additional informations to help me sort out this problem
please don't hesitate to ask.

Thanks in advance and best regards
Christoph



--
________________________________________________________________

EMTEX GmbH
Christoph Russow
Software Engineer

Bischof-Otto-Weg 9
D-91086 Aurachtal
                           Geschäftsführer:       Markus Enzinger
Tel. +49 9132 7490 0      Sitz der Gesellschaft: 91086 Aurachtal
Fax. +49 9132 7490 900    Amtsgericht Fürth:     HRB6804
_________________________________________________________________________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com
Back to top
david.villasmil.work a...
Guest





PostPosted: Wed Oct 06, 2021 6:14 am    Post subject: [Freeswitch-users] Jitter and dropping MOS score Reply with quote

First thing to look at is check your lua scripts if you have any. It is very strange for fs to take 100% of cpu on such low loads. So check your call flows.

On Wed, 6 Oct 2021 at 00:58, Christoph Russow <russow@emtex.de (russow@emtex.de)> wrote:

Quote:
Hi all,

we are currently experiencing jitter and dropping MOS score on an
increasing amount of calls on our system.

i can't see any increased load on our system (neither CPU nor I/O wise).

Current stats:

~ 15Mbit/s network traffic (on a 1Gbit/s network)
load average: 1,10, 0,82, 0,91 (16 core machine)
memory used: 2,26GB/16GB

freeswitch@lc1> status
UP 0 years, 12 days, 13 hours, 30 minutes, 24 seconds, 649 milliseconds,
831 microseconds
FreeSWITCH (Version 1.10.1 -release-12-f9990221e6 64bit) is ready
137634 session(s) since startup
218 session(s) - peak 245, last 5min 222
1 session(s) per Sec out of max 30, peak 8, last 5min 2
1000 session(s) max
min idle cpu 0.00/95.37
Current Stack Size/Max 244K/244K

System ist not virtualized.

the only "strange" thing i see is the freeswitch "main"process that
consumes (according to htop) >=100% cpu (see attached screenshot).
Can this be the problem? if yes what can i do against it.

if not what are possible bottlenecks i should weed out?

if you need any additional informations to help me sort out this problem
please don't hesitate to ask.

Thanks in advance and best regards
Christoph



--
________________________________________________________________

EMTEX GmbH
Christoph Russow
Software Engineer

Bischof-Otto-Weg 9
D-91086 Aurachtal
                           Geschäftsführer:       Markus Enzinger
Tel. +49 9132 7490 0      Sitz der Gesellschaft: 91086 Aurachtal
Fax. +49 9132 7490 900    Amtsgericht Fürth:     HRB6804
_________________________________________________________________________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com


--
Regards,


David Villasmilemail: david.villasmil.work@gmail.com (david.villasmil.work@gmail.com)
phone: +34669448337
Back to top
john.hardiman at missi...
Guest





PostPosted: Wed Oct 06, 2021 12:08 pm    Post subject: [Freeswitch-users] Jitter and dropping MOS score Reply with quote

Hi Christoph,

For the CPU part: On linux boxes if a process in top goes over 100% or top shows >1.0 in load, this means that more than one core is occupied. Since you have 16 cores this is fine as you are using 1/16 of your CPU for the FreeSWITCH process. So 200% would be 2 cores fully used etc, etc.

For the Jitter part/dropping MoS, we might need a bit more info, like times or patterns that is happens or even what the MoS drops by and how much jitter is seen. I personally use SNGrep and a combination of Homer/VoIPMonitor to get all of the stuff I need.

Accurate monitoring will point you in the right direction if there is an issue (Homer 7 is free and really good as a side note)

Thanks,

John.

-----Original Message-----
From: Christoph Russow <russow@emtex.de>
Sent: 05 October 2021 10:34
To: freeswitch-users@lists.freeswitch.org
Subject: [Freeswitch-users] Jitter and dropping MOS score

Hi all,

we are currently experiencing jitter and dropping MOS score on an increasing amount of calls on our system.

i can't see any increased load on our system (neither CPU nor I/O wise).

Current stats:

~ 15Mbit/s network traffic (on a 1Gbit/s network) load average: 1,10, 0,82, 0,91 (16 core machine) memory used: 2,26GB/16GB

freeswitch@lc1> status
UP 0 years, 12 days, 13 hours, 30 minutes, 24 seconds, 649 milliseconds,
831 microseconds
FreeSWITCH (Version 1.10.1 -release-12-f9990221e6 64bit) is ready
137634 session(s) since startup
218 session(s) - peak 245, last 5min 222
1 session(s) per Sec out of max 30, peak 8, last 5min 2
1000 session(s) max
min idle cpu 0.00/95.37
Current Stack Size/Max 244K/244K

System ist not virtualized.

the only "strange" thing i see is the freeswitch "main"process that consumes (according to htop) >=100% cpu (see attached screenshot).
Can this be the problem? if yes what can i do against it.

if not what are possible bottlenecks i should weed out?

if you need any additional informations to help me sort out this problem please don't hesitate to ask.

Thanks in advance and best regards
Christoph



--
________________________________________________________________

EMTEX GmbH
Christoph Russow
Software Engineer

Bischof-Otto-Weg 9
D-91086 Aurachtal
Geschäftsführer: Markus Enzinger
Tel. +49 9132 7490 0 Sitz der Gesellschaft: 91086 Aurachtal
Fax. +49 9132 7490 900 Amtsgericht Fürth: HRB6804
________________________________________________________________
_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com
Back to top
Jerry.Kendall at Bisho...
Guest





PostPosted: Wed Oct 06, 2021 12:09 pm    Post subject: [Freeswitch-users] Jitter and dropping MOS score Reply with quote

Christoph,

If you are running any lua scripts, you might want to look at those...

I had a huge load issue I went through in testing and found out my lua scripts were killing the CPU during "high" usage of 3-5 calls...

Search for my postings on "High CPU usage and Slow BLFs with only 8 channels" or... https://lists.freeswitch.org/pipermail/freeswitch-users/2021-September/135066.html


The issue I had was looking at the EVENTS in the lua script.... Every channel created 20-30 events
I changed the strategy for catching the EVENTS and the load has never been over 0.2. The MOS is now steady at 4+ and ZERO jitter....

While I never commented on the jitter in my postings, there were issues and I knew it was due to the high CPU usage. My issue, why the high load with only a few calls.

Hope it helps,
Jerry

On 10/5/2021 5:33 AM, Christoph Russow wrote:
Quote:
Hi all,

we are currently experiencing jitter and dropping MOS score on an increasing amount of calls on our system.

i can't see any increased load on our system (neither CPU nor I/O wise).

Current stats:

~ 15Mbit/s network traffic (on a 1Gbit/s network)
load average: 1,10, 0,82, 0,91 (16 core machine)
memory used: 2,26GB/16GB

freeswitch@lc1> status
UP 0 years, 12 days, 13 hours, 30 minutes, 24 seconds, 649 milliseconds, 831 microseconds
FreeSWITCH (Version 1.10.1 -release-12-f9990221e6 64bit) is ready
137634 session(s) since startup
218 session(s) - peak 245, last 5min 222
1 session(s) per Sec out of max 30, peak 8, last 5min 2
1000 session(s) max
min idle cpu 0.00/95.37
Current Stack Size/Max 244K/244K

System ist not virtualized.

the only "strange" thing i see is the freeswitch "main"process that consumes (according to htop) >=100% cpu (see attached screenshot).
Can this be the problem? if yes what can i do against it.

if not what are possible bottlenecks i should weed out?

if you need any additional informations to help me sort out this problem please don't hesitate to ask.

Thanks in advance and best regards
Christoph




_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com



_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com
Back to top
jurijs.ivolga at gmail...
Guest





PostPosted: Wed Oct 06, 2021 12:18 pm    Post subject: [Freeswitch-users] Jitter and dropping MOS score Reply with quote

Hi,


I can not tell you about everybody, but usually when I get this problem 99% of the time the issue is on the provider side. I would try to reproduce this issue by myself and capture pcap files and analyze those.



With kind regards,



Jurijs







On Wed, Oct 6, 2021 at 7:43 PM John Hardiman <john.hardiman@missionlabs.co.uk (john.hardiman@missionlabs.co.uk)> wrote:

Quote:
Hi Christoph,

For the CPU part: On linux boxes if a process in top goes over 100% or top shows >1.0 in load, this means that more than one core is occupied. Since you have 16 cores this is fine as you are using 1/16 of your CPU for the FreeSWITCH process. So 200% would be 2 cores fully used etc, etc.

For the Jitter part/dropping MoS, we might need a bit more info, like times or patterns that is happens or even what the MoS drops by and how much jitter is seen. I personally use SNGrep and a combination of Homer/VoIPMonitor to get all of the stuff I need.

Accurate monitoring will point you in the right direction if there is an issue (Homer 7 is free and really good as a side note)

Thanks,

John.

-----Original Message-----
From: Christoph Russow <russow@emtex.de (russow@emtex.de)>
Sent: 05 October 2021 10:34
To: freeswitch-users@lists.freeswitch.org (freeswitch-users@lists.freeswitch.org)
Subject: [Freeswitch-users] Jitter and dropping MOS score

Hi all,

we are currently experiencing jitter and dropping MOS score on an increasing amount of calls on our system.

i can't see any increased load on our system (neither CPU nor I/O wise).

Current stats:

~ 15Mbit/s network traffic (on a 1Gbit/s network) load average: 1,10, 0,82, 0,91 (16 core machine) memory used: 2,26GB/16GB

freeswitch@lc1> status
UP 0 years, 12 days, 13 hours, 30 minutes, 24 seconds, 649 milliseconds,
831 microseconds
FreeSWITCH (Version 1.10.1 -release-12-f9990221e6 64bit) is ready
137634 session(s) since startup
218 session(s) - peak 245, last 5min 222
1 session(s) per Sec out of max 30, peak 8, last 5min 2
1000 session(s) max
min idle cpu 0.00/95.37
Current Stack Size/Max 244K/244K

System ist not virtualized.

the only "strange" thing i see is the freeswitch "main"process that consumes (according to htop) >=100% cpu (see attached screenshot).
Can this be the problem? if yes what can i do against it.

if not what are possible bottlenecks i should weed out?

if you need any additional informations to help me sort out this problem please don't hesitate to ask.

Thanks in advance and best regards
Christoph



--
________________________________________________________________

EMTEX GmbH
Christoph Russow
Software Engineer

Bischof-Otto-Weg 9
D-91086 Aurachtal
                           Geschäftsführer:       Markus Enzinger
Tel. +49 9132 7490 0      Sitz der Gesellschaft: 91086 Aurachtal
Fax. +49 9132 7490 900    Amtsgericht Fürth:     HRB6804
________________________________________________________________
_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com (sales@freeswitch.com)
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org (FreeSWITCH-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com
Back to top
russow at emtex.de
Guest





PostPosted: Thu Oct 07, 2021 4:45 am    Post subject: [Freeswitch-users] Jitter and dropping MOS score Reply with quote

Hi all,

thanks for all the input.

We could sort out the Jitter/MOS problem as a temporary Network problem
with one of our carriers.

about the CPU:
how should i make sure that it is really eating 100% CPU?

"ps aux" shows the freeswitch process at arround 30% so it seems to not
really consume 100%cpu as htop still tells me.

USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND
freeswi+ 30074 30.6 3.9 8582604 636456 ? SNsl Sep22 6431:58
/usr/bin/freeswitch -ncwait -u freeswitch -g emtex-files -nonat

best regards
Christoph

Am 06.10.21 um 12:33 schrieb Dragos Oancea:
Quote:
if it really consumes 100% CPU with few calls then it's abnormal.
kill it with SIGABRT (make sure it will dump core before that), get a
backtrace with gdb and open a github issue.
before doing this you need to make sure it's really eating 100 % CPU,
this screenshot is not very relevant.
Also, you need to make sure you don't have real packet loss. Perhaps
it's a matter of dimensioning, eg: how many simultaneous calls do you
expect to have on that hardware, what codec, etc.

On Wed, Oct 6, 2021 at 2:26 AM Christoph Russow <russow@emtex.de
<mailto:russow@emtex.de>> wrote:

Hi all,

we are currently experiencing jitter and dropping MOS score on an
increasing amount of calls on our system.

i can't see any increased load on our system (neither CPU nor I/O wise).

Current stats:

~ 15Mbit/s network traffic (on a 1Gbit/s network)
load average: 1,10, 0,82, 0,91 (16 core machine)
memory used: 2,26GB/16GB

freeswitch@lc1> status
UP 0 years, 12 days, 13 hours, 30 minutes, 24 seconds, 649
milliseconds,
831 microseconds
FreeSWITCH (Version 1.10.1 -release-12-f9990221e6 64bit) is ready
137634 session(s) since startup
218 session(s) - peak 245, last 5min 222
1 session(s) per Sec out of max 30, peak 8, last 5min 2
1000 session(s) max
min idle cpu 0.00/95.37
Current Stack Size/Max 244K/244K

System ist not virtualized.

the only "strange" thing i see is the freeswitch "main"process that
consumes (according to htop) >=100% cpu (see attached screenshot).
Can this be the problem? if yes what can i do against it.

if not what are possible bottlenecks i should weed out?

if you need any additional informations to help me sort out this
problem
please don't hesitate to ask.

Thanks in advance and best regards
Christoph



--
________________________________________________________________

EMTEX GmbH
Christoph Russow
Software Engineer

Bischof-Otto-Weg 9
D-91086 Aurachtal
                           Geschäftsführer:       Markus Enzinger
Tel. +49 9132 7490 0      Sitz der Gesellschaft: 91086 Aurachtal
Fax. +49 9132 7490 900    Amtsgericht Fürth:     HRB6804
_________________________________________________________________________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire
https://signalwire.com <https://signalwire.com>
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN
services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time
https://signalwire.community <https://signalwire.community>

Professional FreeSWITCH Services
sales@freeswitch.com <mailto:sales@freeswitch.com>
https://freeswitch.com <https://freeswitch.com>

Official FreeSWITCH Sites
https://freeswitch.com/oss <https://freeswitch.com/oss>
https://freeswitch.org/confluence <https://freeswitch.org/confluence>
https://cluecon.com <https://cluecon.com>

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
<mailto:FreeSWITCH-users@lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
<http://lists.freeswitch.org/mailman/listinfo/freeswitch-users>
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
<http://lists.freeswitch.org/mailman/options/freeswitch-users>
https://freeswitch.com <https://freeswitch.com>


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com


_________________________________________________________________________

The FreeSWITCH project is sponsored by SignalWire https://signalwire.com
Enhance your FreeSWITCH install with disruptive priced SMS and PSTN services.
Build your next product on our scalable cloud platform.

Join our online community to chat in real time https://signalwire.community

Professional FreeSWITCH Services
sales@freeswitch.com
https://freeswitch.com

Official FreeSWITCH Sites
https://freeswitch.com/oss
https://freeswitch.org/confluence
https://cluecon.com

FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
https://freeswitch.com
Back to top
Display posts from previous:   
Post new topic   Reply to topic    VoIP Mailing List Archives Forum Index -> freeSWITCH Users All times are GMT - 5 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group

VoiceMeUp - Corporate & Wholesale VoIP Services