Sponsor: VoiceMeUp - Corporate & Wholesale VoIP Services

VoIP Mailing List Archives
Mailing list archives for the VoIP community
 SearchSearch 

[asterisk-users] Exceptionally long queue length queuing


 
Post new topic   Reply to topic    VoIP Mailing List Archives Forum Index -> Asterisk Users
View previous topic :: View next topic  
Author Message
pwakano at gmail.com
Guest





PostPosted: Wed Jul 22, 2020 7:06 pm    Post subject: [asterisk-users] Exceptionally long queue length queuing Reply with quote

Hello list,
Hope you are all doing well!


Sorry for the long email but I tried to explain all I've seen regarding this issue.... I am going to open a ticket for this issue but I found it useful to already explain it here.


I've also recently faced this "Exceptionally long queue length queuing" error in some servers running 16.8.0 and after A LOT of investigation, I've discovered what is causing it in my case (not sure if is the same case as initially reported by Dovid)
So, what I discovered is that in case the diaplan has a Wait() and during this wait period many "deferrable frames" are received, the final piece of the Wait function (ast_safe_sleep_conditional) can throw the "Exceptionally long queue length queuing" error message.

It is not really common to get a lot of "deferrable frames" (see ast_is_deferrable_frame) but we can have that by simply putting on hold and off hold while the channel is on "wait".
My production case is a little more complex and happens while on Wait and with the AST_CONTROL_SRCCHANGE event. This happens when an old Asterisk 13.13.0 receives a hold REINVITE and starts a new RTP stream (with different SSRC) towards some other victim Asterisk with the Music on Hold while still sending the original caller RTP (so at this point 2 RTP streams with different SSRCs are sent to the destination that happens to be a channel currently on Wait). I could not find any ticket for such bug, but version 16.8.0 does not have this problem and instead seems to inject the Music on Hold in the existing RTP stream so there aren't 2 streams at the same time to the same destination. The problem with 2 streams is that each RTP package of the new stream generates a source change frame (AST_CONTROL_SRCCHANGE), and at 50pps this builds up a lot of events fast and so if the Wait is high the issue pops up (I agree having a channel on Wait for too long is not a good practice but noone is perfect...). Also if multiple channels are facing this situation then the problem just escalates very bad.
When the "Exceptionally long queue length queuing" is happening all AMI commands seem to fail, BYE messages seem to be missed/delayed, internal timeouts expire and all sort of weird things happen to all channels in the server, apparently the whole Asterisk process gets locked in this loop affecting everything else... and this is what makes this issue really bad because Asterisk can become completely unresponsive while the error message is happening... if it only could throw the error and keep working fine I guess it would be reasonable leaving it up to the dialplan logic to avoid a long Wait()....


Anyway, to replicate the issue is enough a simple dialplan like this:
exten => 8888,1,Answer()
same => n,Wait(30)
same => n,NoOp(After wait)
same => n,Playback(goodbye)
same => n,Hangup()

After calling in and while in the Wait(30), repeatedly press Hold/Unhold in the telephone (with Linphone just press the hold once and then press and hold the spacebar which will repeatedly do the hold and unhold several times). After Wait is finished the "Exceptionally long queue length queuing" will show up but gets resolved very fast just because I think there weren't enough frames queued to really freeze Asterisk for too long.


Thank you,

Kind regards,
Patrick Wakano



On Fri, 3 Jul 2020 at 19:29, Joshua C. Colp <jcolp@sangoma.com (jcolp@sangoma.com)> wrote:

Quote:
On Fri, Jul 3, 2020 at 3:32 AM Dovid Bender <dovid@telecurve.com (dovid@telecurve.com)> wrote:

Quote:



On Mon, Jun 29, 2020 at 6:46 AM Joshua C. Colp <jcolp@sangoma.com (jcolp@sangoma.com)> wrote:

Quote:
On Sun, Jun 28, 2020 at 2:26 PM Dovid Bender <dovid@telecurve.com (dovid@telecurve.com)> wrote:

Quote:
Hi,


We have a box up and we are starting to see a lot of "Exceptionally long queue length queuing" in the logs. From all the research so far it seems like this leads to their systems crashing and being unreachable. In our case the box remains up and takes calls. We are running Asterisk 16.6.1. We are using MusicOnHold to play online music streams via ffmpeg. Any idea on how to troubleshoot this further to see why this is happening?




The message occurs when either a channel is deadlocked or the processing time to handle what a channel is doing is substantial, and things build up. Getting a backtrace[1] at the time it occurs will show what the system is actually doing and can provide insight into what is causing it. 



[1] https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace#GettingaBacktrace-Runningast_coredumperfordeadlocks,taskprocessorbackups,etc.

--

I ended up re-writing my code to use less Local channels and most of the errors went away. I also noticed that my load average and CPU usage is way down. I will open a ticket since it seems  that it may be a bug that others are experiencing.





A backtrace would still be needed for the issue, otherwise there is no way to know what exactly is going on our where things are getting held up. 



--
Joshua C. Colp
Asterisk Technical Lead
Sangoma Technologies
Check us out at www.sangoma.com and www.asterisk.org











--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --

Check out the new Asterisk community forum at: https://community.asterisk.org/

New to Asterisk? Start here:
      https://wiki.asterisk.org/wiki/display/AST/Getting+Started

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
   http://lists.digium.com/mailman/listinfo/asterisk-users
Back to top
Display posts from previous:   
Post new topic   Reply to topic    VoIP Mailing List Archives Forum Index -> Asterisk Users All times are GMT - 5 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group

VoiceMeUp - Corporate & Wholesale VoIP Services