VoIP Mailing List Archives :: View topic - [asterisk-users] Problems with the voice quality under load

Sponsor: VoiceMeUp - Corporate & Wholesale VoIP Services

VoIP Mailing List Archives
Mailing list archives for the VoIP community

[asterisk-users] Problems with the voice quality under load

VoIP Mailing List Archives Forum Index -> Asterisk Users

View previous topic :: View next topic

Author

Message

mkaganer at gmail.com
Guest

Posted: Mon Mar 02, 2015 10:27 am Post subject: [asterisk-users] Problems with the voice quality under load

B.H. Hello, all We have a cluster of Asterisk (v. 11.9) servers that host IVR applications. The servers work behind SIP proxy (kamailio) for load balancing. All servers are in 2 processor configuration, 8-10 cores per CPU. When a particular server gets about 500 concurrent calls, the sound quality begins to degrade, the sound plays slowly and with clicks. As far as i understand, it's because asterisk is unable to send the voice stream in time i.e. the server is overloaded. What i don't understand is, at the time that the server appears to be overloaded and the audio quality is bad, actual server's load is no more than 30-40% (60-70% idle CPU on average). IMHO, this indicates that for some reason the server is unable to use it's CPU capacity efficiently. May be because of some kind of thread contention inside asterisk? I have read blogs that advice to divide physical server into several VMs and they claim that this will improve the total capacity. In my own experience, this did not work very well and seems like the visualization actually made the quality worse. Do you have any advice for me (other than purchasing more servers ? Thanks! -- משיח NOW! Moshiach is coming very soon, prepare yourself!יחי אדוננו מורינו ורבינו מלך המשיח לעולם ועד!

rwheeler at artifact-s...
Guest

Posted: Mon Mar 02, 2015 10:57 am Post subject: [asterisk-users] Problems with the voice quality under load

Have you done the math for the network connections? BTF and external
What bit rates for the sound?
What codecs?
How are calls coming in - SIP - analogue

Disks OK(low IO per second)? Caching working OK?

CPU may not be the problem if your CPU utilization is really that low.

Ron

On 02/03/2015 10:26 AM, Mordechay Kaganer wrote:

Quote:

B.H.

Hello, all Smile

We have a cluster of Asterisk (v. 11.9) servers that host IVR applications. The servers work behind SIP proxy (kamailio) for load balancing.

All servers are in 2 processor configuration, 8-10 cores per CPU.

When a particular server gets about 500 concurrent calls, the sound quality begins to degrade, the sound plays slowly and with clicks. As far as i understand, it's because asterisk is unable to send the voice stream in time i.e. the server is overloaded.

What i don't understand is, at the time that the server appears to be overloaded and the audio quality is bad, actual server's load is no more than 30-40% (60-70% idle CPU on average). IMHO, this indicates that for some reason the server is unable to use it's CPU capacity efficiently. May be because of some kind of thread contention inside asterisk?

I have read blogs that advice to divide physical server into several VMs and they claim that this will improve the total capacity. In my own experience, this did not work very well and seems like the visualization actually made the quality worse.

Do you have any advice for me (other than purchasing more servers Wink

?

Thanks!

--
משיח NOW!

Moshiach is coming very soon, prepare yourself! יחי אדוננו מורינו ורבינו מלך המשיח לעולם ועד!

--
Ron Wheeler
President
Artifact Software Inc
email: rwheeler@artifact-software.com (rwheeler@artifact-software.com)
skype: ronaldmwheeler
phone: 866-970-2435, ext 102

asterisk_list at earth...
Guest

Posted: Mon Mar 02, 2015 11:05 am Post subject: [asterisk-users] Problems with the voice quality under load

On Monday 02 Mar 2015, Mordechay Kaganer wrote:

Quote:

When a particular server gets about 500 concurrent calls, the sound quality
begins to degrade, the sound plays slowly and with clicks. As far as i
understand, it's because asterisk is unable to send the voice stream in
time i.e. the server is overloaded.

What i don't understand is, at the time that the server appears to be
overloaded and the audio quality is bad, actual server's load is no more
than 30-40% (60-70% idle CPU on average). IMHO, this indicates that for
some reason the server is unable to use it's CPU capacity efficiently. May
be because of some kind of thread contention inside asterisk?

Do you have any advice for me (other than purchasing more servers Wink

Perhaps it's not CPU that is blocking things, but I/O? It should be visually
and audibly obvious if this is the case -- the disk activity lights will be
illuminated, and the HDDs will be making noises.

If you can spare the RAM, consider setting up a tmpfs to hold your IVR prompts
in memory.

--
AJS

Note: Originating address only accepts e-mail from list! If replying off-
list, change address to asterisk1list at earthshod dot co dot uk .

--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
New to Asterisk? Join us for a live introductory webinar every Thurs:
http://www.asterisk.org/hello

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users

asterisk.org at sedwar...
Guest

Posted: Mon Mar 02, 2015 11:15 am Post subject: [asterisk-users] Problems with the voice quality under load

On Mon, 2 Mar 2015, Mordechay Kaganer wrote:

Quote:

When a particular server gets about 500 concurrent calls, the sound
quality begins to degrade, the sound plays slowly and with clicks. As
far as i understand, it's because asterisk is unable to send the voice
stream in time i.e. the server is overloaded.

What i don't understand is, at the time that the server appears to be
overloaded and the audio quality is bad, actual server's load is no more
than 30-40% (60-70% idle CPU on average). IMHO, this indicates that for
some reason the server is unable to use it's CPU capacity efficiently.
May be because of some kind of thread contention inside asterisk?

You may be using a feature (i.e., meetme) that is single threaded. If you
view the system using 'htop' instead of 'top' (or, press '1' while running
top) you may see that a single CPU is maxed out while the others are
relatively idle.

--
Thanks in advance,
-------------------------------------------------------------------------
Steve Edwards sedwards@sedwards.com Voice: +1-760-468-3867 PST
Newline Fax: +1-760-731-3000

--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
New to Asterisk? Join us for a live introductory webinar every Thurs:
http://www.asterisk.org/hello

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users

mkaganer at gmail.com
Guest

Posted: Mon Mar 02, 2015 12:32 pm Post subject: [asterisk-users] Problems with the voice quality under load

B.H.

Hi, really thanks for all the relies Smile

Here's my answers:

On Mon, Mar 2, 2015 at 5:56 PM, Ron Wheeler <rwheeler@artifact-software.com (rwheeler@artifact-software.com)> wrote:

Quote:

There's no problem with network speed - we have 3 port bonding of 1G connections on each server and 1G external cross connect to the SIP providers (which are located in the same server farm). Even single 1G link is sufficient for more than 5000 calls with G711.

All disk IO is off-loaded to the central server with NFS and as far as i can see, the load is low. In most cases, the people listen to the same files and caching is OK - each server has 32GB of RAM most of it is used for caching.

We are connected to several SIP trunk providers. All the setup is SIP only. The codec is G711.

On Mon, Mar 2, 2015 at 6:02 PM, A J Stiles <asterisk_list@earthshod.co.uk (asterisk_list@earthshod.co.uk)> wrote:

Quote:

The prompts are anyway cached in in memory by the disk cache so i don't think it's the issue in this case. As stated above, i don't see much of disk load on the NFS server as well.

On Mon, Mar 2, 2015 at 6:14 PM, Steve Edwards <asterisk.org@sedwards.com (asterisk.org@sedwards.com)> wrote:

Quote:

I did check the CPU load and it is distributed evenly between all the cores. Most of the load is in "system", probably network handling in the kernel.

We do use ConfBridge a lot, but all conferences are on a single server and the issue is on all the servers.

What other features are single-threaded?

Asterisk is almost the only thing that runs on the PBX servers. There are also some PHP AGI scripts (5-10 per minute per server on average) and most of the AGI functions are off-loaded to a java application on the central server with fast AGI. But there is still a lot of dialplan logic that runs on the PBX's. Maybe, huge dialplan is the key for the issue?

--

משיח NOW!

Moshiach is coming very soon, prepare yourself!יחי אדוננו מורינו ורבינו מלך המשיח לעולם ועד!

Display posts from previous:

	VoIP Mailing List Archives Forum Index -> Asterisk Users	All times are GMT - 5 Hours
Page 1 of 1

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum