VoIP Mailing List Archives :: View topic - [Freeswitch-users] Freeswitch wedges in voicemail?

Sponsor: VoiceMeUp - Corporate & Wholesale VoIP Services

VoIP Mailing List Archives
Mailing list archives for the VoIP community

[Freeswitch-users] Freeswitch wedges in voicemail?

Goto page 1, 2 Next

VoIP Mailing List Archives Forum Index -> freeSWITCH Users

View previous topic :: View next topic

Author

Message

marc at avvatel.com
Guest

Posted: Mon Oct 27, 2008 12:43 pm Post subject: [Freeswitch-users] Freeswitch wedges in voicemail?

I have a strange problem that I've been dealing with for the last week or two. Once or twice a day, Freeswitch will "wedge", in what appears to be voicemail. The switch is sort of still responding, meaning I can issue various commands at the console, but no new sofia channels will be handled, registrations start failing on the internal profile and no new calls can come in from the external profile. When it happens, existing channels stay up, but once they are completed, no new channels can be created. There is always one channel that remains that looks something like (DID obfuscated): api show channels Content-Type: api/response Content-Length: 394 uuid,created,created_epoch,name,state,cid_name,cid_num,ip_addr,dest,application,application_data,dialplan,context,read_codec,read_rate,write_codec,write_rate ea758bcf-25f5-4cde-9f6c-5e3f1226130f,2008-10-27 10:02:27,1225126947,sofia/external/253941xxxx@204.11.200.165,CS_EXECUTE,253941xxxx,253941xxxx,199.245.214.150,328,voicemail,default pgm.cisvp.com 246,XML,pgm,PCMU,8000,PCMU,8000 1 total. I've been updating every weekend with the latest code, this is currently trunk rev 10146. Anyone have any thoughts on this? Its driving me crazy. - Marc -- Marc Lewis Avvatel Corporation _______________________________________________ Freeswitch-users mailing list Freeswitch-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

anthony.minessale at g...
Guest

Posted: Mon Oct 27, 2008 1:13 pm Post subject: [Freeswitch-users] Freeswitch wedges in voicemail?

next time it happens
go into another shell

and enter

gcore -o dump `cat /usr/local/freeswitch/log/freeswitch.pid`

then

gdb /usr/local/bin/freeswitch/bin/freeswitch dump.*

then enter these 3 commands:

set pagination off
thread apply all bt
thread apply all pt full

and send me the entire output

also are you updating with "make current" to avoid build skew?

On Mon, Oct 27, 2008 at 12:32 PM, Marc Lewis <marc@avvatel.com (marc@avvatel.com)> wrote:

Quote:

I have a strange problem that I've been dealing with for the last week
or two. Once or twice a day, Freeswitch will "wedge", in what appears
to be voicemail. The switch is sort of still responding, meaning I can
issue various commands at the console, but no new sofia channels will be
handled, registrations start failing on the internal profile and no new
calls can come in from the external profile.

When it happens, existing channels stay up, but once they are completed,
no new channels can be created. There is always one channel that
remains that looks something like (DID obfuscated):

api show channels

Content-Type: api/response
Content-Length: 394

uuid,created,created_epoch,name,state,cid_name,cid_num,ip_addr,dest,application,application_data,dialplan,context,read_codec,read_rate,write_codec,write_rate
ea758bcf-25f5-4cde-9f6c-5e3f1226130f,2008-10-27
10:02:27,1225126947,sofia/external/253941xxxx@204.11.200.165 (253941xxxx@204.11.200.165),CS_EXECUTE,253941xxxx,253941xxxx,199.245.214.150,328,voicemail,default
pgm.cisvp.com 246,XML,pgm,PCMU,8000,PCMU,8000

1 total.

I've been updating every weekend with the latest code, this is currently
trunk rev 10146.

Anyone have any thoughts on this? Its driving me crazy.

- Marc

--
Marc Lewis
Avvatel Corporation

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org (Freeswitch-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

--
Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/

AIM: anthm
MSN:anthony_minessale@hotmail.com ([email]MSN%3Aanthony_minessale@hotmail.com[/email])
GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com ([email]PAYPAL%3Aanthony.minessale@gmail.com[/email])
IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference
sip:888@conference.freeswitch.org ([email]sip%3A888@conference.freeswitch.org[/email])
iax:guest@conference.freeswitch.org/888
googletalk:conf+888@conference.freeswitch.org ([email]googletalk%3Aconf%2B888@conference.freeswitch.org[/email])
pstn:213-799-1400

marc at avvatel.com
Guest

Posted: Mon Oct 27, 2008 1:29 pm Post subject: [Freeswitch-users] Freeswitch wedges in voicemail?

I'll send the files the next time it happens.

As for my builds, I'm building fresh each time using a script I wrote to create RPM files. I do an 'svn update' then a './bootstrap.sh' and then run my script. My script creates a tarball from the checked out files and creates RPM files from that tarball.

The only other thing I can think that might be causing this since I haven't read about anyone else having this problem is a lua script that I wrote. The script is for creating auto attendants dynamically out of the XML dialplan pulled from curl. The channels don't show that the script itself is running, but it is transferred to voicemail from the script.

The lua code that transfers to voicemail looks like this:

elseif (action == "voicemail") then
session:execute("voicemail", "default ".. aaDomain .." ".. target)
session:hangup()

Where the aaDomain and target are set from the XML dialplan.

I'm happy to share the auto attendant lua script with anyone that thinks it might be useful to them. Its not well documented yet, though.

- Marc

Anthony Minessale wrote:

Quote:

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org (Freeswitch-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

--
Marc Lewis
Avvatel Corporation

anthony.minessale at g...
Guest

Posted: Mon Oct 27, 2008 1:40 pm Post subject: [Freeswitch-users] Freeswitch wedges in voicemail?

we can give you a folder in contrib or check it in for you if you wish
we may also be able to find some issue in the script if we see it.

are you using session:ready in all your loops to avoid a death spiral?

On Mon, Oct 27, 2008 at 1:22 PM, Marc Lewis <marc@avvatel.com (marc@avvatel.com)> wrote:

Quote:

--
Marc Lewis
Avvatel Corporation

marc at avvatel.com
Guest

Posted: Mon Oct 27, 2008 1:50 pm Post subject: [Freeswitch-users] Freeswitch wedges in voicemail?

I'll write up some quick and dirty documentation for it and send it to you (I'm gottafixthat on irc). Its quite useful when combined with curl.

There are a total of 8 session:ready() checks in the script, at least one in each loop, often two because I'm paranoid. It is my first time programming in lua, so its entirely possible I botched it.

Thanks, I'll send you a couple files later today.

- Marc

Anthony Minessale wrote:

Quote:

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

--
Marc Lewis
Avvatel Corporation

Quote:

--
Marc Lewis
Avvatel Corporation

marc at avvatel.com
Guest

Posted: Mon Oct 27, 2008 2:53 pm Post subject: [Freeswitch-users] Freeswitch wedges in voicemail?

Here is the autoattendant.lua script and a brief README file that outlines its basic operation.

As I mentioned, my irc nick is 'gottafixthat' should you decide to put this into contrib.

Thanks.

- Marc

Anthony Minessale wrote:

Quote:

--
Marc Lewis
Avvatel Corporation

Quote:

--
Marc Lewis
Avvatel Corporation

marc at avvatel.com
Guest

Posted: Thu Oct 30, 2008 9:42 am Post subject: [Freeswitch-users] Freeswitch wedges in voicemail?

Here is the gdb output from the commands. Its actually wedged up twice since I first sent my email.

Anthony Minessale wrote:

Quote:

--
Marc Lewis
Avvatel Corporation

marc at avvatel.com
Guest

Posted: Thu Oct 30, 2008 1:15 pm Post subject: [Freeswitch-users] Freeswitch wedges in voicemail?

Got my second wedge of the day. Last time the channel state showed in my lua autoattendant script. This time the channel state showed in voicemail -- which is where it usually shows when its wedged.

Here is the gdb output from this one. Hopefully this will help track this down. I'm anxious to get this particular problem behind me.

Thanks.

- Marc

Anthony Minessale wrote:

Quote:

--
Marc Lewis
Avvatel Corporation

mike at jerris.com
Guest

Posted: Thu Oct 30, 2008 1:28 pm Post subject: [Freeswitch-users] Freeswitch wedges in voicemail?

What svn revision was this?

Mike

On Oct 30, 2008, at 2:00 PM, Marc Lewis wrote:

Quote:

Got my second wedge of the day. Last time the channel state showed
in my lua autoattendant script. This time the channel state showed
in voicemail -- which is where it usually shows when its wedged.

Here is the gdb output from this one. Hopefully this will help
track this down. I'm anxious to get this particular problem behind
me.

Thanks.

- Marc

marc at avvatel.com
Guest

Posted: Thu Oct 30, 2008 1:37 pm Post subject: [Freeswitch-users] Freeswitch wedges in voicemail?

Its rev 10146, but it also happened under 10067.

Its now happened to me 4 times today. Twice in the last half hour. I have dumps from those, too.

- Marc

Michael Jerris wrote:

Quote:

What svn revision was this?

Mike

On Oct 30, 2008, at 2:00 PM, Marc Lewis wrote:

Quote:

Got my second wedge of the day. Last time the channel state showed
in my lua autoattendant script. This time the channel state showed
in voicemail -- which is where it usually shows when its wedged.

Here is the gdb output from this one. Hopefully this will help
track this down. I'm anxious to get this particular problem behind
me.

Thanks.

- Marc

--
Marc Lewis
Avvatel Corporation

anthony.minessale at g...
Guest

Posted: Fri Oct 31, 2008 8:18 am Post subject: [Freeswitch-users] Freeswitch wedges in voicemail?

can you update to latest trunk please and test again?

On Thu, Oct 30, 2008 at 1:23 PM, Marc Lewis <marc@avvatel.com (marc@avvatel.com)> wrote:

Quote:

Its rev 10146, but it also happened under 10067.

Its now happened to me 4 times today. Twice in the last half hour. I have dumps from those, too.

- Marc

Michael Jerris wrote:

Quote:

What svn revision was this?

Mike

On Oct 30, 2008, at 2:00 PM, Marc Lewis wrote:

Quote:

Got my second wedge of the day. Last time the channel state showed
in my lua autoattendant script. This time the channel state showed
in voicemail -- which is where it usually shows when its wedged.

Here is the gdb output from this one. Hopefully this will help
track this down. I'm anxious to get this particular problem behind
me.

Thanks.

- Marc

--
Marc Lewis
Avvatel Corporation

marc at avvatel.com
Guest

Posted: Fri Oct 31, 2008 9:56 am Post subject: [Freeswitch-users] Freeswitch wedges in voicemail?

Compiling now. I'll update my production system as soon as it is possible.

Thanks.

- Marc

Anthony Minessale wrote:

Quote:

can you update to latest trunk please and test again?

On Thu, Oct 30, 2008 at 1:23 PM, Marc Lewis <marc@avvatel.com (marc@avvatel.com)> wrote:

Quote:

Its rev 10146, but it also happened under 10067.

Its now happened to me 4 times today. Twice in the last half hour. I have dumps from those, too.

- Marc

Michael Jerris wrote:

Quote:

What svn revision was this?

Mike

On Oct 30, 2008, at 2:00 PM, Marc Lewis wrote:

Quote:

Got my second wedge of the day. Last time the channel state showed
in my lua autoattendant script. This time the channel state showed
in voicemail -- which is where it usually shows when its wedged.

Here is the gdb output from this one. Hopefully this will help
track this down. I'm anxious to get this particular problem behind
me.

Thanks.

- Marc

--
Marc Lewis
Avvatel Corporation

Quote:

--
Marc Lewis
Avvatel Corporation

marc at avvatel.com
Guest

Posted: Fri Oct 31, 2008 12:51 pm Post subject: [Freeswitch-users] Freeswitch wedges in voicemail?

My systems are all running svn rev 10207 now. I'll let you know if things hang again.

Thanks again.

- Marc

Anthony Minessale wrote:

Quote:

can you update to latest trunk please and test again?

On Thu, Oct 30, 2008 at 1:23 PM, Marc Lewis <marc@avvatel.com (marc@avvatel.com)> wrote:

Quote:

Its rev 10146, but it also happened under 10067.

Its now happened to me 4 times today. Twice in the last half hour. I have dumps from those, too.

- Marc

Michael Jerris wrote:

Quote:

What svn revision was this?

Mike

On Oct 30, 2008, at 2:00 PM, Marc Lewis wrote:

Quote:

Got my second wedge of the day. Last time the channel state showed
in my lua autoattendant script. This time the channel state showed
in voicemail -- which is where it usually shows when its wedged.

Here is the gdb output from this one. Hopefully this will help
track this down. I'm anxious to get this particular problem behind
me.

Thanks.

- Marc

--
Marc Lewis
Avvatel Corporation

Quote:

--
Marc Lewis
Avvatel Corporation

marc at avvatel.com
Guest

Posted: Tue Nov 04, 2008 7:06 pm Post subject: [Freeswitch-users] Freeswitch wedges in voicemail?

Vast improvements on svn rev 10207. I haven't had a single wedge like the ones I experienced before since then.

I did however, just a few minutes ago have it completely freeze up on me. The console became unresponsive and registrations were no longer being accepted. This is different than the times before, so I don't think it is the same problem. I have only had freeswitch become totally unresponsive maybe twice before in the last 2 months that I've been working with it in a production environment.

I did manage to get a backtrace of it in the hopes that it will help.

Thanks.

- Marc

Anthony Minessale wrote:

Quote:

can you update to latest trunk please and test again?

On Thu, Oct 30, 2008 at 1:23 PM, Marc Lewis <marc@avvatel.com (marc@avvatel.com)> wrote:

Quote:

Its rev 10146, but it also happened under 10067.

Its now happened to me 4 times today. Twice in the last half hour. I have dumps from those, too.

- Marc

Michael Jerris wrote:

Quote:

What svn revision was this?

Mike

On Oct 30, 2008, at 2:00 PM, Marc Lewis wrote:

Quote:

Got my second wedge of the day. Last time the channel state showed
in my lua autoattendant script. This time the channel state showed
in voicemail -- which is where it usually shows when its wedged.

Here is the gdb output from this one. Hopefully this will help
track this down. I'm anxious to get this particular problem behind
me.

Thanks.

- Marc

--
Marc Lewis
Avvatel Corporation

Quote:

--
Marc Lewis
Avvatel Corporation

anthony.minessale at g...
Guest

Posted: Tue Nov 04, 2008 7:23 pm Post subject: [Freeswitch-users] Freeswitch wedges in voicemail?

can you update again to make sure its still ok on even more latest.
also can you turn off crash protection? that looks like it hit crash protection and it did not end well.
If you can please disable it just for a bit so we can see if you get a real core.

On Tue, Nov 4, 2008 at 6:05 PM, Marc Lewis <marc@avvatel.com (marc@avvatel.com)> wrote:

Quote:

can you update to latest trunk please and test again?

On Thu, Oct 30, 2008 at 1:23 PM, Marc Lewis <marc@avvatel.com (marc@avvatel.com)> wrote:

Quote:

Its rev 10146, but it also happened under 10067.

Its now happened to me 4 times today. Twice in the last half hour. I have dumps from those, too.

- Marc

Michael Jerris wrote:

Quote:

What svn revision was this?

Mike

On Oct 30, 2008, at 2:00 PM, Marc Lewis wrote:

Quote:

Got my second wedge of the day. Last time the channel state showed
in my lua autoattendant script. This time the channel state showed
in voicemail -- which is where it usually shows when its wedged.

Here is the gdb output from this one. Hopefully this will help
track this down. I'm anxious to get this particular problem behind
me.

Thanks.

- Marc

--
Marc Lewis
Avvatel Corporation

Quote:

--
Marc Lewis
Avvatel Corporation

[root@trelane ~]# gdb /opt/freeswitch/bin/freeswitch dump.14291
GNU gdb Red Hat Linux (6.5-37.el5_2.2rh)
Copyright (C) 2006 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type "show copying" to see the conditions.
There is absolutely no warranty for GDB. Type "show warranty" for details.
This GDB was configured as "i386-redhat-linux-gnu"...Using host libthread_db library "/lib/i686/nosegneg/libthread_db.so.1".

warning: Can't read pathname for load map: Input/output error.
Reading symbols from /lib/i686/nosegneg/libm.so.6...done.
Loaded symbols for /lib/i686/nosegneg/libm.so.6
Reading symbols from /opt/freeswitch/lib/libfreeswitch.so.1...Reading symbols from /usr/lib/debug/opt/freeswitch/lib/libfreeswitch.so.1.0.0.debug...done.
done.
Loaded symbols for /opt/freeswitch/lib/libfreeswitch.so.1
Reading symbols from /lib/libuuid.so.1...done.
Loaded symbols for /lib/libuuid.so.1
Reading symbols from /lib/i686/nosegneg/librt.so.1...done.
Loaded symbols for /lib/i686/nosegneg/librt.so.1
Reading symbols from /lib/libdl.so.2...done.
Loaded symbols for /lib/libdl.so.2
Reading symbols from /lib/libcrypt.so.1...done.
Loaded symbols for /lib/libcrypt.so.1
Reading symbols from /lib/i686/nosegneg/libpthread.so.0...done.
Loaded symbols for /lib/i686/nosegneg/libpthread.so.0
Reading symbols from /usr/lib/libncurses.so.5...done.
Loaded symbols for /usr/lib/libncurses.so.5
Reading symbols from /lib/i686/nosegneg/libc.so.6...done.
Loaded symbols for /lib/i686/nosegneg/libc.so.6
Reading symbols from /lib/ld-linux.so.2...done.
Loaded symbols for /lib/ld-linux.so.2
Reading symbols from /usr/lib/libstdc++.so.6...done.
Loaded symbols for /usr/lib/libstdc++.so.6
Reading symbols from /lib/libgcc_s.so.1...done.
Loaded symbols for /lib/libgcc_s.so.1
Reading symbols from /usr/lib/libodbc.so.1...done.
Loaded symbols for /usr/lib/libodbc.so.1
Reading symbols from /opt/freeswitch/mod/mod_console.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_console.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_console.so
Reading symbols from /opt/freeswitch/mod/mod_logfile.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_logfile.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_logfile.so
Reading symbols from /opt/freeswitch/mod/mod_syslog.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_syslog.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_syslog.so
Reading symbols from /opt/freeswitch/mod/mod_enum.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_enum.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_enum.so
Reading symbols from /opt/freeswitch/mod/mod_xml_rpc.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_xml_rpc.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_xml_rpc.so
Reading symbols from /opt/freeswitch/mod/mod_xml_curl.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_xml_curl.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_xml_curl.so
Reading symbols from /usr/lib/libcurl.so.3...done.
Loaded symbols for /usr/lib/libcurl.so.3
Reading symbols from /usr/lib/libgssapi_krb5.so.2...done.
Loaded symbols for /usr/lib/libgssapi_krb5.so.2
Reading symbols from /usr/lib/libkrb5.so.3...done.
Loaded symbols for /usr/lib/libkrb5.so.3
Reading symbols from /usr/lib/libk5crypto.so.3...done.
Loaded symbols for /usr/lib/libk5crypto.so.3
Reading symbols from /lib/libcom_err.so.2...done.
Loaded symbols for /lib/libcom_err.so.2
Reading symbols from /lib/libresolv.so.2...done.
Loaded symbols for /lib/libresolv.so.2
Reading symbols from /usr/lib/libidn.so.11...done.
Loaded symbols for /usr/lib/libidn.so.11
Reading symbols from /lib/libssl.so.6...done.
Loaded symbols for /lib/libssl.so.6
Reading symbols from /lib/libcrypto.so.6...done.
Loaded symbols for /lib/libcrypto.so.6
Reading symbols from /usr/lib/libz.so.1...done.
Loaded symbols for /usr/lib/libz.so.1
Reading symbols from /usr/lib/libkrb5support.so.0...done.
Loaded symbols for /usr/lib/libkrb5support.so.0
Reading symbols from /lib/libkeyutils.so.1...done.
Loaded symbols for /lib/libkeyutils.so.1
Reading symbols from /lib/libselinux.so.1...done.
Loaded symbols for /lib/libselinux.so.1
Reading symbols from /lib/libsepol.so.1...done.
Loaded symbols for /lib/libsepol.so.1
Reading symbols from /opt/freeswitch/mod/mod_xml_cdr.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_xml_cdr.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_xml_cdr.so
Reading symbols from /lib/libnss_files.so.2...done.
Loaded symbols for /lib/libnss_files.so.2
Reading symbols from /lib/libnss_dns.so.2...done.
Loaded symbols for /lib/libnss_dns.so.2
Reading symbols from /opt/freeswitch/mod/mod_cdr_csv.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_cdr_csv.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_cdr_csv.so
Reading symbols from /opt/freeswitch/mod/mod_event_socket.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_event_socket.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_event_socket.so
Reading symbols from /opt/freeswitch/mod/mod_sofia.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_sofia.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_sofia.so
Reading symbols from /usr/lib/gconv/ISO8859-1.so...done.
Loaded symbols for /usr/lib/gconv/ISO8859-1.so
Reading symbols from /usr/lib/libmyodbc3.so...done.
Loaded symbols for /usr/lib/libmyodbc3.so
Reading symbols from /usr/lib/mysql/libmysqlclient.so.15...done.
Loaded symbols for /usr/lib/mysql/libmysqlclient.so.15
Reading symbols from /lib/libnsl.so.1...done.
Loaded symbols for /lib/libnsl.so.1
Reading symbols from /usr/lib/libltdl.so.3...done.
Loaded symbols for /usr/lib/libltdl.so.3
Reading symbols from /usr/lib/libodbcinst.so.1...done.
Loaded symbols for /usr/lib/libodbcinst.so.1
Reading symbols from /opt/freeswitch/mod/mod_commands.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_commands.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_commands.so
Reading symbols from /opt/freeswitch/mod/mod_conference.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_conference.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_conference.so
Reading symbols from /opt/freeswitch/mod/mod_dptools.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_dptools.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_dptools.so
Reading symbols from /opt/freeswitch/mod/mod_expr.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_expr.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_expr.so
Reading symbols from /opt/freeswitch/mod/mod_fifo.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_fifo.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_fifo.so
Reading symbols from /opt/freeswitch/mod/mod_voicemail.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_voicemail.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_voicemail.so
Reading symbols from /opt/freeswitch/mod/mod_limit.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_limit.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_limit.so
Reading symbols from /opt/freeswitch/mod/mod_esf.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_esf.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_esf.so
Reading symbols from /opt/freeswitch/mod/mod_fsv.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_fsv.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_fsv.so
Reading symbols from /opt/freeswitch/mod/mod_dialplan_xml.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_dialplan_xml.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_dialplan_xml.so
Reading symbols from /opt/freeswitch/mod/mod_dialplan_asterisk.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_dialplan_asterisk.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_dialplan_asterisk.so
Reading symbols from /opt/freeswitch/mod/mod_voipcodecs.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_voipcodecs.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_voipcodecs.so
Reading symbols from /opt/freeswitch/mod/mod_ilbc.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_ilbc.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_ilbc.so
Reading symbols from /opt/freeswitch/mod/mod_speex.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_speex.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_speex.so
Reading symbols from /opt/freeswitch/mod/mod_h26x.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_h26x.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_h26x.so
Reading symbols from /opt/freeswitch/mod/mod_sndfile.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_sndfile.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_sndfile.so
Reading symbols from /opt/freeswitch/mod/mod_native_file.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_native_file.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_native_file.so
Reading symbols from /opt/freeswitch/mod/mod_local_stream.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_local_stream.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_local_stream.so
Reading symbols from /opt/freeswitch/mod/mod_tone_stream.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_tone_stream.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_tone_stream.so
Reading symbols from /opt/freeswitch/mod/mod_spidermonkey.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_spidermonkey.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_spidermonkey.so
Reading symbols from /opt/freeswitch/lib/libnspr4.so...Reading symbols from /usr/lib/debug/opt/freeswitch/lib/libnspr4.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/lib/libnspr4.so
Reading symbols from /opt/freeswitch/lib/libjs.so.1...done.
Loaded symbols for /opt/freeswitch/lib/libjs.so.1
Reading symbols from /opt/freeswitch/mod/mod_spidermonkey_teletone.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_spidermonkey_teletone.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_spidermonkey_teletone.so
Reading symbols from /opt/freeswitch/mod/mod_spidermonkey_core_db.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_spidermonkey_core_db.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_spidermonkey_core_db.so
Reading symbols from /opt/freeswitch/mod/mod_spidermonkey_socket.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_spidermonkey_socket.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_spidermonkey_socket.so
Reading symbols from /opt/freeswitch/mod/mod_lua.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_lua.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_lua.so
Reading symbols from /opt/freeswitch/mod/mod_say_en.so...Reading symbols from /usr/lib/debug/opt/freeswitch/mod/mod_say_en.so.debug...done.
done.
Loaded symbols for /opt/freeswitch/mod/mod_say_en.so
Core was generated by `/opt/freeswitch/bin/freeswitch'.
#0 0x004ab402 in __kernel_vsyscall ()
(gdb) set pagination off
(gdb) thread apply all bt

Thread 30 (process 14291):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
#3 0x0022c80d in switch_sleep (t=1000000) at src/switch_time.c:143
#4 0x001ee91b in switch_core_runtime_loop (bg=1) at src/switch_core.c:643
#5 0x08049ee0 in main (argc=2, argv=0xbfd03a64) at src/switch.c:626
#6 0x004c1dec in __libc_start_main () from /lib/i686/nosegneg/libc.so.6
#7 0x08048d91 in _start ()

Thread 29 (process 26501):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
#3 0x0022c80d in switch_sleep (t=30000) at src/switch_time.c:143
#4 0x00201e51 in rtp_common_read (rtp_session=0x8df502a8, payload_type=0x82c1d38 "", flags=0x82c1d4c, io_flags=0) at src/switch_rtp.c:1559
#5 0x00202d63 in switch_rtp_zerocopy_read_frame (rtp_session=0x8df502a8, frame=0x82c1d14, io_flags=0) at src/switch_rtp.c:1704
#6 0x00cf21e8 in sofia_read_frame (session=0xeb49808, frame=0x7a5fe064, flags=0, stream_id=0) at mod_sofia.c:648
#7 0x001ebbaf in switch_core_session_read_frame (session=0xeb49808, frame=0x7a5fe064, flags=0, stream_id=0) at src/switch_core_io.c:146
#8 0x002061bb in audio_bridge_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_ivr_bridge.c:332
#9 0x002076a3 in switch_ivr_multi_threaded_bridge (session=0xeb49808, peer_session=0x857d8938, input_callback=0, session_data=0x0, peer_session_data=0x0) at src/switch_ivr_bridge.c:876
#10 0x00207a22 in uuid_bridge_on_soft_execute (session=0xeb49808) at src/switch_ivr_bridge.c:604
#11 0x001e9641 in switch_core_session_run (session=0xeb49808) at src/switch_core_state_machine.c:436
#12 0x001e6ef5 in switch_core_session_thread (thread=0x82c26c8, obj=0xeb49808) at src/switch_core_session.c:854
#13 0x0024a3a6 in dummy_worker (opaque=0x82c26c8) at threadproc/unix/thread.c:138
#14 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#15 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 28 (process 26487):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0058bbfe in __lll_mutex_lock_wait () from /lib/i686/nosegneg/libc.so.6
#2 0x00519a28 in _L_lock_14107 () from /lib/i686/nosegneg/libc.so.6
#3 0x00518cb4 in free () from /lib/i686/nosegneg/libc.so.6
#4 0x0052f59f in tzset_internal () from /lib/i686/nosegneg/libc.so.6
#5 0x0052febd in tzset () from /lib/i686/nosegneg/libc.so.6
#6 0x00534646 in strftime_l () from /lib/i686/nosegneg/libc.so.6
#7 0x005345a6 in strftime () from /lib/i686/nosegneg/libc.so.6
#8 0x00000050 in ?? ()
#9 0x002c405a in __func__.11670 () from /opt/freeswitch/lib/libfreeswitch.so.1
#10 0x7cdfcd20 in ?? ()
#11 0x005ee380 in __libc_utmp_file_name () from /lib/i686/nosegneg/libc.so.6
#12 0x003013e8 in ?? () from /opt/freeswitch/lib/libfreeswitch.so.1
#13 0x7cdfcd58 in ?? ()
#14 0x0024c1a5 in apr_strftime (s=0x0, retsize=0x0, max=0, format=0x0, xt=0x0) at time/unix/timestr.c:151
#15 0x00000000 in ?? ()

Thread 27 (process 26486):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0058bbfe in __lll_mutex_lock_wait () from /lib/i686/nosegneg/libc.so.6
#2 0x0052ff5b in _L_lock_1783 () from /lib/i686/nosegneg/libc.so.6
#3 0x0052fce4 in __tz_convert () from /lib/i686/nosegneg/libc.so.6
#4 0x0052e30c in localtime_r () from /lib/i686/nosegneg/libc.so.6
#5 0x0024bfb7 in explode_time (xt=0x787fa1cc, t=1225842636452329, offset=0, use_localtime=1) at time/unix/time.c:90
#6 0x0024c043 in apr_time_exp_lt (result=0x787fa1cc, input=1225839500853248) at time/unix/time.c:133
#7 0x001d13cb in switch_time_exp_lt (result=0x787fa1cc, input=1225843795820540) at src/switch_apr.c:298
#8 0x00222b20 in switch_log_printf (channel=SWITCH_CHANNEL_ID_LOG, file=0x2c971c "src/switch_core_state_machine.c", func=0x2c9bf2 "print_trace", line=225, userdata=0x0, level=SWITCH_LOG_CRIT, fmt=0x2c99b0 "Obtained %zd stack frames.\n") at src/switch_log.c:267
#9 0x001e9e65 in handle_fatality (sig=11) at src/switch_core_state_machine.c:225
#10 <signal handler called>
#11 0x00515e30 in _int_malloc () from /lib/i686/nosegneg/libc.so.6
#12 0x00516fe5 in malloc () from /lib/i686/nosegneg/libc.so.6
#13 0x001fb77f in my_dup (s=0xb29aad70 "2062803316") at src/switch_event.c:91
#14 0x001fcde5 in switch_event_add_header_string (event=0xa6f145a8, stack=SWITCH_STACK_BOTTOM, header_name=0x787fa738 "Caller-Caller-ID-Name", data=0x18 <Address 0x18 out of bounds>) at src/switch_event.c:698
#15 0x001d3363 in switch_caller_profile_event_set_data (caller_profile=0xb29aacf0, prefix=0x2c4010 "Caller", event=0xa6f145a8) at src/switch_caller.c:250
#16 0x001d57bf in switch_channel_event_set_data (channel=0xece0dc0, event=0xa6f145a8) at src/switch_channel.c:1132
#17 0x002071bd in switch_ivr_multi_threaded_bridge (session=0xecd8660, peer_session=0x8c474488, input_callback=0, session_data=0x0, peer_session_data=0x0) at src/switch_ivr_bridge.c:907
#18 0x00aac549 in audio_bridge_function (session=0xecd8660, data=0xe0094b8 "sofia/internal/101@206.124.151.179:5101;fs_nat=yes,sofia/internal/103@206.124.151.179:5103;fs_nat=yes,sofia/internal/105@206.124.151.179:5105;fs_nat=yes,sofia/internal/106@206.124.151.179:5200;fs_nat="...) at mod_dptools.c:1924
#19 0x001e472b in switch_core_session_exec (session=0xecd8660, application_interface=0xb27b1f38, arg=0xe0094b8 "sofia/internal/101@206.124.151.179:5101;fs_nat=yes,sofia/internal/103@206.124.151.179:5103;fs_nat=yes,sofia/internal/105@206.124.151.179:5105;fs_nat=yes,sofia/internal/106@206.124.151.179:5200;fs_nat="...) at src/switch_core_session.c:1192
#20 0x001e7348 in switch_core_session_execute_application (session=0xecd8660, app=0xb29ab058 "bridge", arg=0xb29ab060 "${sofia_contact(101@sikeslaw.cisvp.com (101@sikeslaw.cisvp.com))},${sofia_contact(103@sikeslaw.cisvp.com (103@sikeslaw.cisvp.com))},${sofia_contact(105@sikeslaw.cisvp.com (105@sikeslaw.cisvp.com))},${sofia_contact(106@sikeslaw.cisvp.com (106@sikeslaw.cisvp.com))},${sofia_contact(107@sikeslaw.cisvp.c"...) at src/switch_core_session.c:1143
#21 0x001e9592 in switch_core_session_run (session=0xecd8660) at src/switch_core_state_machine.c:152
#22 0x001e6ef5 in switch_core_session_thread (thread=0xb29aaa68, obj=0xecd8660) at src/switch_core_session.c:854
#23 0x0024a3a6 in dummy_worker (opaque=0xb29aaa68) at threadproc/unix/thread.c:138
#24 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#25 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 26 (process 26420):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
#3 0x0022c80d in switch_sleep (t=30000) at src/switch_time.c:143
#4 0x00201e51 in rtp_common_read (rtp_session=0xa7b53ca0, payload_type=0xb29d9d70 "\r", flags=0xb29d9d84, io_flags=0) at src/switch_rtp.c:1559
#5 0x00202d63 in switch_rtp_zerocopy_read_frame (rtp_session=0xa7b53ca0, frame=0xb29d9d4c, io_flags=0) at src/switch_rtp.c:1704
#6 0x00cf21e8 in sofia_read_frame (session=0x8612ad78, frame=0x7d9fe134, flags=0, stream_id=0) at mod_sofia.c:648
#7 0x001ebbaf in switch_core_session_read_frame (session=0x8612ad78, frame=0x7d9fe134, flags=0, stream_id=0) at src/switch_core_io.c:146
#8 0x002061bb in audio_bridge_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_ivr_bridge.c:332
#9 0x00208173 in audio_bridge_on_exchange_media (session=0x8612ad78) at src/switch_ivr_bridge.c:447
#10 0x001e92e0 in switch_core_session_run (session=0x8612ad78) at src/switch_core_state_machine.c:433
#11 0x001e6ef5 in switch_core_session_thread (thread=0xb29da6d8, obj=0x8612ad78) at src/switch_core_session.c:854
#12 0x0024a3a6 in dummy_worker (opaque=0xb29da6d8) at threadproc/unix/thread.c:138
#13 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#14 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 25 (process 26419):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
#3 0x0022c80d in switch_sleep (t=30000) at src/switch_time.c:143
#4 0x00201e51 in rtp_common_read (rtp_session=0x8d75f860, payload_type=0xb29cdd40 "\r", flags=0xb29cdd54, io_flags=0) at src/switch_rtp.c:1559
#5 0x00202d63 in switch_rtp_zerocopy_read_frame (rtp_session=0x8d75f860, frame=0xb29cdd1c, io_flags=0) at src/switch_rtp.c:1704
#6 0x00cf21e8 in sofia_read_frame (session=0x85796190, frame=0x7e3fefa4, flags=0, stream_id=0) at mod_sofia.c:648
#7 0x001ebbaf in switch_core_session_read_frame (session=0x85796190, frame=0x7e3fefa4, flags=0, stream_id=0) at src/switch_core_io.c:146
#8 0x002061bb in audio_bridge_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_ivr_bridge.c:332
#9 0x002076a3 in switch_ivr_multi_threaded_bridge (session=0x85796190, peer_session=0x8612ad78, input_callback=0, session_data=0x0, peer_session_data=0x0) at src/switch_ivr_bridge.c:876
#10 0x00aac549 in audio_bridge_function (session=0x85796190, data=0xb29cea08 "sofia/gateway/osercr/9076442402") at mod_dptools.c:1924
#11 0x001e472b in switch_core_session_exec (session=0x85796190, application_interface=0xb27b1f38, arg=0xb29cea08 "sofia/gateway/osercr/9076442402") at src/switch_core_session.c:1192
#12 0x001e7348 in switch_core_session_execute_application (session=0x85796190, app=0xb29cea00 "bridge", arg=0xb29cea08 "sofia/gateway/osercr/9076442402") at src/switch_core_session.c:1143
#13 0x001e9592 in switch_core_session_run (session=0x85796190) at src/switch_core_state_machine.c:152
#14 0x001e6ef5 in switch_core_session_thread (thread=0xb29ce6d8, obj=0x85796190) at src/switch_core_session.c:854
#15 0x0024a3a6 in dummy_worker (opaque=0xb29ce6d8) at threadproc/unix/thread.c:138
#16 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#17 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 24 (process 26240):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
#3 0x0022c80d in switch_sleep (t=20000) at src/switch_time.c:143
#4 0x00201e51 in rtp_common_read (rtp_session=0x863e1fa8, payload_type=0x824aec8 "", flags=0x824aedc, io_flags=0) at src/switch_rtp.c:1559
#5 0x00202d63 in switch_rtp_zerocopy_read_frame (rtp_session=0x863e1fa8, frame=0x824aea4, io_flags=0) at src/switch_rtp.c:1704
#6 0x00cf21e8 in sofia_read_frame (session=0x857d8938, frame=0x791fc134, flags=0, stream_id=0) at mod_sofia.c:648
#7 0x001ebbaf in switch_core_session_read_frame (session=0x857d8938, frame=0x791fc134, flags=0, stream_id=0) at src/switch_core_io.c:146
#8 0x002061bb in audio_bridge_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_ivr_bridge.c:332
#9 0x00208173 in audio_bridge_on_exchange_media (session=0x857d8938) at src/switch_ivr_bridge.c:447
#10 0x001e92e0 in switch_core_session_run (session=0x857d8938) at src/switch_core_state_machine.c:433
#11 0x001e6ef5 in switch_core_session_thread (thread=0x824b7d8, obj=0x857d8938) at src/switch_core_session.c:854
#12 0x0024a3a6 in dummy_worker (opaque=0x824b7d8) at threadproc/unix/thread.c:138
#13 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#14 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 23 (process 24933):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
#3 0x0022c80d in switch_sleep (t=30000) at src/switch_time.c:143
#4 0x00201e51 in rtp_common_read (rtp_session=0x88b89968, payload_type=0xb29c7d28 "", flags=0xb29c7d3c, io_flags=0) at src/switch_rtp.c:1559
#5 0x00202d63 in switch_rtp_zerocopy_read_frame (rtp_session=0x88b89968, frame=0xb29c7d04, io_flags=0) at src/switch_rtp.c:1704
#6 0x00cf21e8 in sofia_read_frame (session=0x857e3948, frame=0x7f1ff134, flags=0, stream_id=0) at mod_sofia.c:648
#7 0x001ebbaf in switch_core_session_read_frame (session=0x857e3948, frame=0x7f1ff134, flags=0, stream_id=0) at src/switch_core_io.c:146
#8 0x002061bb in audio_bridge_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_ivr_bridge.c:332
#9 0x00208173 in audio_bridge_on_exchange_media (session=0x857e3948) at src/switch_ivr_bridge.c:447
#10 0x001e92e0 in switch_core_session_run (session=0x857e3948) at src/switch_core_state_machine.c:433
#11 0x001e6ef5 in switch_core_session_thread (thread=0xb29c8688, obj=0x857e3948) at src/switch_core_session.c:854
#12 0x0024a3a6 in dummy_worker (opaque=0xb29c8688) at threadproc/unix/thread.c:138
#13 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#14 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 22 (process 24932):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
#3 0x0022c80d in switch_sleep (t=30000) at src/switch_time.c:143
#4 0x00201e51 in rtp_common_read (rtp_session=0x8ddf1390, payload_type=0x8328548 "", flags=0x832855c, io_flags=0) at src/switch_rtp.c:1559
#5 0x00202d63 in switch_rtp_zerocopy_read_frame (rtp_session=0x8ddf1390, frame=0x8328524, io_flags=0) at src/switch_rtp.c:1704
#6 0x00cf21e8 in sofia_read_frame (session=0x8d713350, frame=0x77df9fa4, flags=0, stream_id=0) at mod_sofia.c:648
#7 0x001ebbaf in switch_core_session_read_frame (session=0x8d713350, frame=0x77df9fa4, flags=0, stream_id=0) at src/switch_core_io.c:146
#8 0x002061bb in audio_bridge_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_ivr_bridge.c:332
#9 0x002076a3 in switch_ivr_multi_threaded_bridge (session=0x8d713350, peer_session=0x857e3948, input_callback=0, session_data=0x0, peer_session_data=0x0) at src/switch_ivr_bridge.c:876
#10 0x00aac549 in audio_bridge_function (session=0x8d713350, data=0x8329138 "sofia/gateway/osercr/8005628192") at mod_dptools.c:1924
#11 0x001e472b in switch_core_session_exec (session=0x8d713350, application_interface=0xb27b1f38, arg=0x8329138 "sofia/gateway/osercr/8005628192") at src/switch_core_session.c:1192
#12 0x001e7348 in switch_core_session_execute_application (session=0x8d713350, app=0x8329130 "bridge", arg=0x8329138 "sofia/gateway/osercr/8005628192") at src/switch_core_session.c:1143
#13 0x001e9592 in switch_core_session_run (session=0x8d713350) at src/switch_core_state_machine.c:152
#14 0x001e6ef5 in switch_core_session_thread (thread=0x8328ec8, obj=0x8d713350) at src/switch_core_session.c:854
#15 0x0024a3a6 in dummy_worker (opaque=0x8328ec8) at threadproc/unix/thread.c:138
#16 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#17 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 21 (process 21291):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0058bbfe in __lll_mutex_lock_wait () from /lib/i686/nosegneg/libc.so.6
#2 0x0052ff5b in _L_lock_1783 () from /lib/i686/nosegneg/libc.so.6
#3 0x0052fce4 in __tz_convert () from /lib/i686/nosegneg/libc.so.6
#4 0x0052e30c in localtime_r () from /lib/i686/nosegneg/libc.so.6
#5 0x0024bfb7 in explode_time (xt=0x8c2fe474, t=1225842688776584, offset=0, use_localtime=1) at time/unix/time.c:90
#6 0x0024c043 in apr_time_exp_lt (result=0x8c2fe474, input=1225839500853248) at time/unix/time.c:133
#7 0x001d13cb in switch_time_exp_lt (result=0x8c2fe474, input=1225843795820540) at src/switch_apr.c:298
#8 0x001fd294 in switch_event_fire_detailed (file=0x2cb878 "src/switch_loadable_module.c", func=0x2cbfef "switch_api_execute", line=1439, event=0x8c2fe594, user_data=0x0) at src/switch_event.c:1016
#9 0x001f3c57 in switch_api_execute (cmd=0x8e7859f4 "show", arg=0x8e7859f9 "channels", session=0x0, stream=0x8c2fe574) at src/switch_loadable_module.c:1439
#10 0x00ca9108 in api_exec (thread=0x0, obj=0x8c2fea30) at mod_event_socket.c:1026
#11 0x00ca9f48 in parse_command (listener=0xb29c2ad8, event=0x8c2feb00, reply=0x8c2ff10c "", reply_len=512) at mod_event_socket.c:1350
#12 0x00cabb26 in listener_run (thread=0xaaccfb68, obj=0xb29c2ad8) at mod_event_socket.c:1722
#13 0x0024a3a6 in dummy_worker (opaque=0xaaccfb68) at threadproc/unix/thread.c:138
#14 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#15 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 20 (process 4862):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
#3 0x0022c80d in switch_sleep (t=1000) at src/switch_time.c:143
#4 0x00ca87e6 in read_packet (listener=0xb29b0338, event=0xa3cf9b00, timeout=0) at mod_event_socket.c:988
#5 0x00cabaf6 in listener_run (thread=0xb2869b68, obj=0xb29b0338) at mod_event_socket.c:1712
#6 0x0024a3a6 in dummy_worker (opaque=0xb2869b68) at threadproc/unix/thread.c:138
#7 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#8 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 19 (process 14490):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x00574dc3 in poll () from /lib/i686/nosegneg/libc.so.6
#2 0x00448a4e in chanSwitchAccept (chanSwitchP=0x8277f10, channelPP=0xac0f50a0, channelInfoPP=0xac0f509c, errorP=0xac0f50a4) at libs/xmlrpc-c/lib/abyss/src/socket_unix.c:630
#3 0x0043b384 in ChanSwitchAccept (chanSwitchP=0x8277f10, channelPP=0xac0f50a0, channelInfoPP=0xac0f509c, errorP=0xac0f50a4) at libs/xmlrpc-c/lib/abyss/src/chanswitch.c:149
#4 0x0044744e in ServerRun (serverP=0x481e74) at libs/xmlrpc-c/lib/abyss/src/server.c:908
#5 0x00438702 in mod_xml_rpc_runtime () at mod_xml_rpc.c:822
#6 0x001f40b3 in switch_loadable_module_exec (thread=0x8247ae0, obj=0x82478d0) at src/switch_loadable_module.c:92
#7 0x0024a3a6 in dummy_worker (opaque=0x8247ae0) at threadproc/unix/thread.c:138
#8 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#9 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 18 (process 14489):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x00156148 in accept () from /lib/i686/nosegneg/libpthread.so.0
#2 0x00248f8d in apr_socket_accept (new=0xacaf630c, sock=0x82bcff8, connection_context=0xb29be180) at network_io/unix/sockets.c:187
#3 0x001d09db in switch_socket_accept (new_sock=0xacaf630c, sock=0x82bcff8, pool=0xb29be180) at src/switch_apr.c:635
#4 0x00cab060 in mod_event_socket_runtime () at mod_event_socket.c:1916
#5 0x001f40b3 in switch_loadable_module_exec (thread=0x8247878, obj=0x8247668) at src/switch_loadable_module.c:92
#6 0x0024a3a6 in dummy_worker (opaque=0x8247878) at threadproc/unix/thread.c:138
#7 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#8 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 17 (process 14488):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
#3 0x0022c80d in switch_sleep (t=1000) at src/switch_time.c:143
#4 0x0022cbc0 in softtimer_runtime () at src/switch_time.c:371
#5 0x001f40b3 in switch_loadable_module_exec (thread=0x8247610, obj=0x8247400) at src/switch_loadable_module.c:92
#6 0x0024a3a6 in dummy_worker (opaque=0x8247610) at threadproc/unix/thread.c:138
#7 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#8 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 16 (process 14485):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
#3 0x0022c80d in switch_sleep (t=1000) at src/switch_time.c:143
#4 0x0022cf8e in timer_next (timer=0xadef6038) at src/switch_time.c:260
#5 0x001dd81c in switch_core_timer_next (timer=0x3013e8) at src/switch_core_timer.c:75
#6 0x006c561c in read_stream_thread (thread=0xb296fd10, obj=0xb296fc38) at mod_local_stream.c:206
#7 0x0024a3a6 in dummy_worker (opaque=0xb296fd10) at threadproc/unix/thread.c:138
#8 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#9 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 15 (process 14484):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
#3 0x0022c80d in switch_sleep (t=100000) at src/switch_time.c:143
#4 0x00d1c2de in sofia_presence_event_thread_run (thread=0xb2a68ee0, obj=0x0) at sofia_presence.c:584
#5 0x0024a3a6 in dummy_worker (opaque=0xb2a68ee0) at threadproc/unix/thread.c:138
#6 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#7 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 14 (process 14373):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
#3 0x0022c80d in switch_sleep (t=10000) at src/switch_time.c:143
#4 0x00cfc1a1 in sofia_profile_worker_thread_run (thread=0xb2910ac8, obj=0xb290fe98) at sofia.c:464
#5 0x0024a3a6 in dummy_worker (opaque=0xb2910ac8) at threadproc/unix/thread.c:138
#6 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#7 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 13 (process 14372):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
#3 0x0022c80d in switch_sleep (t=10000) at src/switch_time.c:143
#4 0x00cfc1a1 in sofia_profile_worker_thread_run (thread=0xb29171b0, obj=0xb2916460) at sofia.c:464
#5 0x0024a3a6 in dummy_worker (opaque=0xb29171b0) at threadproc/unix/thread.c:138
#6 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#7 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 12 (process 14371):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0058bbfe in __lll_mutex_lock_wait () from /lib/i686/nosegneg/libc.so.6
#2 0x00519a28 in _L_lock_14107 () from /lib/i686/nosegneg/libc.so.6
#3 0x00518cb4 in free () from /lib/i686/nosegneg/libc.so.6
#4 0x00dccd35 in _su_home_deinit (home=0x84bca9a0) at su_alloc.c:1009
#5 0x00dce85c in su_home_unref (home=0x84bca9a0) at su_alloc.c:679
#6 0x00da97f6 in soa_destroy (ss=0x84bca9a0) at soa.c:360
#7 0x00d70c68 in nua_session_usage_remove (nh=0x82f300e0, ds=0x82f30118, du=0x7fce0ad0, cr0=0x0, sr0=0xb06fbe44) at nua_session.c:312
#8 0x00d7f044 in nua_dialog_usage_remove_at (own=0x82f300e0, ds=0x82f30118, at=<value optimized out>, cr0=0x0, sr0=0xb06fbe44) at nua_dialog.c:377
#9 0x00d7809b in nua_base_server_report (sr=0xb06fbe44, tags=0x0) at nua_stack.c:1891
#10 0x00d7494d in nua_bye_server_report (sr=0xb06fbe44, tags=0x0) at nua_session.c:3860
#11 0x00d76643 in nua_server_report (sr=0x0) at nua_stack.c:1827
#12 0x00d7d066 in nua_stack_process_request (nh=0x82f300e0, leg=0x7fce0a00, irq=0x8289a608, sip=0x9e0138fc) at nua_stack.c:1464
#13 0x00d66970 in leg_recv (leg=0x7fce0a00, msg=0x9e013860, sip=0x9e0138fc, tport=0x82932a0) at nta.c:4884
#14 0x00d67c47 in agent_recv_request (agent=0x8294d98, msg=0x9e013860, sip=0x9e0138fc, tport=0x82932a0) at nta.c:2472
#15 0x00d68c04 in agent_recv_message (agent=0x8294d98, tport=0x82932a0, msg=0x9e013860, tport_via=0x8293858, now={tv_sec = 3434831441, tv_usec = 372449}) at nta.c:2254
#16 0x00dd2d04 in tport_base_deliver (self=0x82932a0, msg=0x9e013860, now={tv_sec = 4294967292, tv_usec = 372449}) at tport.c:3013
#17 0x00ddedde in tport_deliver (self=0x82932a0, msg=0x9e013860, next=0x0, sc=0x0, now={tv_sec = 3434831441, tv_usec = 372449}) at tport.c:3002
#18 0x00ddf0d2 in tport_parse (self=0x82932a0, complete=1, now={tv_sec = 3434831441, tv_usec = 372449}) at tport.c:2919
#19 0x00ddf24d in tport_recv_event (self=0x82932a0) at tport.c:2861
#20 0x00ddf548 in tport_base_wakeup (self=0x82932a0, events=1) at tport.c:2763
#21 0x00dc641c in su_epoll_port_wait_events (self=0x82941e8, tout=1000) at su_epoll_port.c:506
#22 0x00dbc082 in su_base_port_run (self=0x82941e8) at su_base_port.c:342
#23 0x00dc4899 in su_root_run (self=0x8294118) at su_port.h:310
#24 0x00dc7dea in su_pthread_port_clone_main (varg=0xb1afe084) at su_pthread_port.c:321
#25 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#26 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 11 (process 14370):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0058bbfe in __lll_mutex_lock_wait () from /lib/i686/nosegneg/libc.so.6
#2 0x00519a28 in _L_lock_14107 () from /lib/i686/nosegneg/libc.so.6
#3 0x00518cb4 in free () from /lib/i686/nosegneg/libc.so.6
#4 0x00dccd35 in _su_home_deinit (home=0x11529d40) at su_alloc.c:1009
#5 0x00dce85c in su_home_unref (home=0x11529d40) at su_alloc.c:679
#6 0x00d31e0a in msg_destroy (msg=0x11529d40) at msg.c:168
#7 0x00d576b0 in incoming_reclaim_queued (rm=0x0, msg=0x0, u=0xb10fd128) at nta.c:5111
#8 0x00d64d98 in incoming_timer (sa=0xb27118c8) at nta.c:6493
#9 0x00d65bff in agent_timer (rm=0x828f258, timer=0xb2711100, agent=0xb27118c8) at nta.c:762
#10 0x00dc003f in su_timer_expire (timers=<value optimized out>, timeout=0xb10fd31c, now={tv_sec = 3434831437, tv_usec = 188331}) at su_timer.c:533
#11 0x00dbc062 in su_base_port_run (self=0x8292b70) at su_base_port.c:334
#12 0x00dc4899 in su_root_run (self=0xb2711e28) at su_port.h:310
#13 0x00dc7dea in su_pthread_port_clone_main (varg=0xb26ff084) at su_pthread_port.c:321
#14 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#15 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 10 (process 14369):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0058bbfe in __lll_mutex_lock_wait () from /lib/i686/nosegneg/libc.so.6
#2 0x0052ff5b in _L_lock_1783 () from /lib/i686/nosegneg/libc.so.6
#3 0x0052fce4 in __tz_convert () from /lib/i686/nosegneg/libc.so.6
#4 0x0052e30c in localtime_r () from /lib/i686/nosegneg/libc.so.6
#5 0x0024bfb7 in explode_time (xt=0xb1afd85c, t=1225842639412327, offset=0, use_localtime=1) at time/unix/time.c:90
#6 0x0024c043 in apr_time_exp_lt (result=0xb1afd85c, input=1225839500853248) at time/unix/time.c:133
#7 0x001d13cb in switch_time_exp_lt (result=0xb1afd85c, input=1225843795820540) at src/switch_apr.c:298
#8 0x00222b20 in switch_log_printf (channel=SWITCH_CHANNEL_ID_LOG, file=0x2c3eb2 "src/switch_channel.c", func=0x2c53b7 "switch_channel_set_name", line=551, userdata=0x0, level=SWITCH_LOG_NOTICE, fmt=0x2c41fe "New Channel %s [%s]\n") at src/switch_log.c:267
#9 0x001d7601 in switch_channel_set_name (channel=0xeada210, name=0xb1afd988 "sofia/external/2537977557@204.11.200.165 (2537977557@204.11.200.165)") at src/switch_channel.c:551
#10 0x00d11304 in sofia_glue_attach_private (session=0xead1ab0, profile=0xb2916460, tech_pvt=0xb29c0ff0, channame=0xb29c1628 "2537977557@204.11.200.165 (2537977557@204.11.200.165)") at sofia_glue.c:429
#11 0x00d05828 in sofia_handle_sip_i_invite (nua=0x8292858, profile=0xb2916460, nh=0x83803fc0, sofia_private=0x0, sip=0x828e5da4, tags=0x85fdcc84) at sofia.c:3732
#12 0x00d09af8 in sofia_event_callback (event=nua_i_invite, status=100, phrase=0x85fdcc8c "Trying", nua=0x8292858, profile=0xb2916460, nh=0x83803fc0, sofia_private=0x0, sip=0x828e5da4, tags=0x85fdcc84) at sofia.c:290
#13 0x00d797bd in nua_application_event (dummy=0x0, sumsg=0xb1afe158, ee=0x85fdcc6c) at nua_stack.c:405
#14 0x00dbb8fc in su_base_port_execute_msgs (queue=<value optimized out>) at su_base_port.c:276
#15 0x00dbbf28 in su_base_port_step (self=0x828f440, tout=0) at su_base_port.c:448
#16 0x00dc4b70 in su_root_step (self=0x828f8c8, tout=1000) at su_port.h:324
#17 0x00d0da9d in sofia_profile_thread_run (thread=0xb29170c8, obj=0xb2916460) at sofia.c:636
#18 0x0024a3a6 in dummy_worker (opaque=0xb29170c8) at threadproc/unix/thread.c:138
#19 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#20 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 9 (process 14366):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0058bbfe in __lll_mutex_lock_wait () from /lib/i686/nosegneg/libc.so.6
#2 0x0052ff5b in _L_lock_1783 () from /lib/i686/nosegneg/libc.so.6
#3 0x0052fce4 in __tz_convert () from /lib/i686/nosegneg/libc.so.6
#4 0x0052e30c in localtime_r () from /lib/i686/nosegneg/libc.so.6
#5 0x0024bfb7 in explode_time (xt=0xb26fe544, t=1225842636572334, offset=0, use_localtime=1) at time/unix/time.c:90
#6 0x0024c043 in apr_time_exp_lt (result=0xb26fe544, input=1225839500853248) at time/unix/time.c:133
#7 0x001d13cb in switch_time_exp_lt (result=0xb26fe544, input=1225843795820540) at src/switch_apr.c:298
#8 0x001fd294 in switch_event_fire_detailed (file=0xe0b460 "sofia_reg.c", func=0xe0bc80 "sofia_reg_handle_register", line=833, event=0xb26fe6b4, user_data=0x0) at src/switch_event.c:1016
#9 0x00d24f9a in sofia_reg_handle_register (nua=0x828f258, profile=0xb290fe98, nh=0x9bd90da0, sip=0xa492433c, regtype=REG_REGISTER, key=0xb26fede8 "29a4ac13-6574-46c7-9647-35b2912a33e2", keylen=128, v_event=0xb26fede4, is_nat=0xe055b8 "via received") at sofia_reg.c:833
#10 0x00d255c6 in sofia_reg_handle_sip_i_register (nua=0x828f258, profile=0xb290fe98, nh=0x9bd90da0, sofia_private=0x0, sip=0xa492433c, tags=0x7f275a0c) at sofia_reg.c:1037
#11 0x00d09ab1 in sofia_event_callback (event=nua_i_register, status=100, phrase=0x7f275a14 "Trying", nua=0x828f258, profile=0xb290fe98, nh=0x9bd90da0, sofia_private=0x0, sip=0xa492433c, tags=0x7f275a0c) at sofia.c:296
#12 0x00d797bd in nua_application_event (dummy=0x0, sumsg=0xb26ff158, ee=0x7f2759f4) at nua_stack.c:405
#13 0x00dbb8fc in su_base_port_execute_msgs (queue=<value optimized out>) at su_base_port.c:276
#14 0x00dbbf28 in su_base_port_step (self=0x8257758, tout=0) at su_base_port.c:448
#15 0x00dc4b70 in su_root_step (self=0x824e688, tout=1000) at su_port.h:324
#16 0x00d0da9d in sofia_profile_thread_run (thread=0xb29109e0, obj=0xb290fe98) at sofia.c:636
#17 0x0024a3a6 in dummy_worker (opaque=0xb29109e0) at threadproc/unix/thread.c:138
#18 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#19 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 8 (process 14352):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0058bbfe in __lll_mutex_lock_wait () from /lib/i686/nosegneg/libc.so.6
#2 0x0052ff5b in _L_lock_1783 () from /lib/i686/nosegneg/libc.so.6
#3 0x0052fce4 in __tz_convert () from /lib/i686/nosegneg/libc.so.6
#4 0x0052e30c in localtime_r () from /lib/i686/nosegneg/libc.so.6
#5 0x0024bfb7 in explode_time (xt=0xb3efff34, t=1225842638036314, offset=0, use_localtime=1) at time/unix/time.c:90
#6 0x0024c043 in apr_time_exp_lt (result=0xb3efff34, input=1225839500853248) at time/unix/time.c:133
#7 0x001d13cb in switch_time_exp_lt (result=0xb3efff34, input=1225843795820540) at src/switch_apr.c:298
#8 0x001fd294 in switch_event_fire_detailed (file=0x2ca1ed "src/switch_core.c", func=0x2cb409 "send_heartbeat", line=78, event=0xb3f00084, user_data=0x0) at src/switch_event.c:1016
#9 0x001f0884 in heartbeat_callback (task=0x822c608) at src/switch_core.c:78
#10 0x001f1b42 in switch_scheduler_execute (tp=0xfffffffc) at src/switch_scheduler.c:61
#11 0x001f1ebc in task_thread_loop (done=0) at src/switch_scheduler.c:127
#12 0x001f2134 in switch_scheduler_task_thread (thread=0x82420d0, obj=0x0) at src/switch_scheduler.c:168
#13 0x0024a3a6 in dummy_worker (opaque=0x82420d0) at threadproc/unix/thread.c:138
#14 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#15 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 7 (process 14351):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
#3 0x0022c80d in switch_sleep (t=1000) at src/switch_time.c:143
#4 0x001e3363 in switch_core_sql_thread (thread=0xb2afdae8, obj=0x0) at src/switch_core_sqldb.c:223
#5 0x0024a3a6 in dummy_worker (opaque=0xb2afdae8) at threadproc/unix/thread.c:138
#6 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#7 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 6 (process 14346):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x001532d6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i686/nosegneg/libpthread.so.0
#2 0x002443aa in apr_thread_cond_wait (cond=0xb72e34b8, mutex=0xb72e3488) at locks/unix/thread_cond.c:68
#3 0x0023b50c in apr_queue_pop (queue=0xb72e3458, data=0xb4901368) at misc/apr_queue.c:276
#4 0x001d0354 in switch_queue_pop (queue=0xb72e3458, data=0xb4901368) at src/switch_apr.c:845
#5 0x0022327d in log_thread (thread=0xb4963ae0, obj=0x0) at src/switch_log.c:208
#6 0x0024a3a6 in dummy_worker (opaque=0xb4963ae0) at threadproc/unix/thread.c:138
#7 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#8 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 5 (process 14343):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x001532d6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i686/nosegneg/libpthread.so.0
#2 0x002443aa in apr_thread_cond_wait (cond=0xb749fb38, mutex=0xb749fb08) at locks/unix/thread_cond.c:68
#3 0x0023b50c in apr_queue_pop (queue=0xb749fad8, data=0xb53eb368) at misc/apr_queue.c:276
#4 0x001d0354 in switch_queue_pop (queue=0xb749fad8, data=0xb53eb368) at src/switch_apr.c:845
#5 0x001fc44d in switch_event_thread (thread=0x81f2288, obj=0xb749fad8) at src/switch_event.c:271
#6 0x0024a3a6 in dummy_worker (opaque=0x81f2288) at threadproc/unix/thread.c:138
#7 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#8 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 4 (process 14342):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x001532d6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i686/nosegneg/libpthread.so.0
#2 0x002443aa in apr_thread_cond_wait (cond=0xb7502b38, mutex=0xb7502b08) at locks/unix/thread_cond.c:68
#3 0x0023b50c in apr_queue_pop (queue=0xb7502ad8, data=0xb5dec368) at misc/apr_queue.c:276
#4 0x001d0354 in switch_queue_pop (queue=0xb7502ad8, data=0xb5dec368) at src/switch_apr.c:845
#5 0x001fc44d in switch_event_thread (thread=0x81f2268, obj=0xb7502ad8) at src/switch_event.c:271
#6 0x0024a3a6 in dummy_worker (opaque=0x81f2268) at threadproc/unix/thread.c:138
#7 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#8 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 3 (process 14341):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x001532d6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i686/nosegneg/libpthread.so.0
#2 0x002443aa in apr_thread_cond_wait (cond=0x81e8388, mutex=0x81e8358) at locks/unix/thread_cond.c:68
#3 0x0023b50c in apr_queue_pop (queue=0x81e8328, data=0xb67ed368) at misc/apr_queue.c:276
#4 0x001d0354 in switch_queue_pop (queue=0x81e8328, data=0xb67ed368) at src/switch_apr.c:845
#5 0x001fc44d in switch_event_thread (thread=0x81f2248, obj=0x81e8328) at src/switch_event.c:271
#6 0x0024a3a6 in dummy_worker (opaque=0x81f2248) at threadproc/unix/thread.c:138
#7 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#8 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 2 (process 14340):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x001532d6 in pthread_cond_wait@@GLIBC_2.3.2 () from /lib/i686/nosegneg/libpthread.so.0
#2 0x002443aa in apr_thread_cond_wait (cond=0xb72e33b8, mutex=0xb72e3388) at locks/unix/thread_cond.c:68
#3 0x0023b50c in apr_queue_pop (queue=0xb72e3358, data=0xb71ee368) at misc/apr_queue.c:276
#4 0x001d0354 in switch_queue_pop (queue=0xb72e3358, data=0xb71ee368) at src/switch_apr.c:845
#5 0x001fd979 in switch_event_dispatch_thread (thread=0x81f2228, obj=0xb72e3358) at src/switch_event.c:228
#6 0x0024a3a6 in dummy_worker (opaque=0x81f2228) at threadproc/unix/thread.c:138
#7 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#8 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6

Thread 1 (process 14292):
#0 0x004ab402 in __kernel_vsyscall ()
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
#3 0x0022c80d in switch_sleep (t=100000) at src/switch_time.c:143
#4 0x001dfa84 in pool_thread (thread=0xb7f35da8, obj=0x0) at src/switch_core_memory.c:421
#5 0x0024a3a6 in dummy_worker (opaque=0xb7f35da8) at threadproc/unix/thread.c:138
#6 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
#7 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6
(gdb) thread apply all bt full

Thread 30 (process 14291):
#0 0x004ab402 in __kernel_vsyscall ()
No symbol table info available.
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#3 0x0022c80d in switch_sleep (t=1000000) at src/switch_time.c:143
No locals.
#4 0x001ee91b in switch_core_runtime_loop (bg=1) at src/switch_core.c:643
No locals.
#5 0x08049ee0 in main (argc=2, argv=0xbfd03a64) at src/switch.c:626
pid_path = "/opt/freeswitch/log/freeswitch.pid", '\0' <repeats 221 times>
pid_buffer = "14291", '\0' <repeats 26 times>
pid_len = 5
err = 0x0
nf = 0
runas_user = 0x0
runas_group = 0x0
nc = 1
pid = 14291
x = 2
die = 0
alt_dirs = 0
known_opt = 1
high_prio = 0
flags = 1
ret = <value optimized out>
destroy_status = <value optimized out>
fd = (switch_file_t *) 0x81dc928
pool = (switch_memory_pool_t *) 0x81dc8f0
#6 0x004c1dec in __libc_start_main () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#7 0x08048d91 in _start ()
No symbol table info available.

Thread 29 (process 26501):
#0 0x004ab402 in __kernel_vsyscall ()
No symbol table info available.
#1 0x0053dad6 in nanosleep () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#2 0x0057827c in usleep () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#3 0x0022c80d in switch_sleep (t=30000) at src/switch_time.c:143
No locals.
#4 0x00201e51 in rtp_common_read (rtp_session=0x8df502a8, payload_type=0x82c1d38 "", flags=0x82c1d4c, io_flags=0) at src/switch_rtp.c:1559
sbytes = 11895231
stat = <value optimized out>
bytes = 0
status = SWITCH_STATUS_BREAK
check = 0 '\0'
jb_frame = <value optimized out>
ret = <value optimized out>
sleep_mss = 30000
__func__ = "rtp_common_read"
__PRETTY_FUNCTION__ = "rtp_common_read"
#5 0x00202d63 in switch_rtp_zerocopy_read_frame (rtp_session=0x8df502a8, frame=0x82c1d14, io_flags=0) at src/switch_rtp.c:1704
bytes = <value optimized out>
#6 0x00cf21e8 in sofia_read_frame (session=0xeb49808, frame=0x7a5fe064, flags=0, stream_id=0) at mod_sofia.c:648
status = SWITCH_STATUS_FALSE
channel = (switch_channel_t *) 0xeb51f68
__PRETTY_FUNCTION__ = "sofia_read_frame"
__func__ = "sofia_read_frame"
#7 0x001ebbaf in switch_core_session_read_frame (session=0xeb49808, frame=0x7a5fe064, flags=0, stream_id=0) at src/switch_core_io.c:146
ptr = <value optimized out>
status = 2339599
need_codec = -2055334320
perfect = <value optimized out>
do_bugs = 3150824
do_resample = 0
is_cng = 0
flag = 0
__PRETTY_FUNCTION__ = "switch_core_session_read_frame"
__func__ = "switch_core_session_read_frame"
#8 0x002061bb in audio_bridge_thread (thread=<value optimized out>, obj=<value optimized out>) at src/switch_ivr_bridge.c:332
cb_status = <value optimized out>
dtmf = {digit = 0 '\0', duration = 7}
data = (switch_ivr_bridge_data_t *) 0x82c2ce8
stream_id = 0
pre_b = 0
ans_a = 1
ans_b = 1
input_callback = (switch_input_callback_function_t) 0
message = (switch_core_session_message_t *) 0x7
msg = {from = 0x0, message_id = SWITCH_MESSAGE_REDIRECT_AUDIO, numeric_arg = 0, string_arg = 0x0, string_arg_size = 0, pointer_arg = 0x0, pointer_arg_size = 0, numeric_reply = 0, string_reply = 0x0, string_reply_size = 0, pointer_reply = 0x0, pointer_reply_size = 0, flags = 0}
user_data = (void *) 0x0
chan_a = (switch_channel_t *) 0xeb51f68
chan_b = (switch_channel_t *) 0x857e1098
read_frame = (switch_frame_t *) 0x0
session_a = (switch_core_session_t *) 0xeb49808
session_b = (switch_core_session_t *) 0x857d8938
loop_count = 2782
app_name = <value optimized out>
app_arg = <value optimized out>
hook_var = <value optimized out>
silence_codec = {codec_interface = 0x0, implementation = 0x0, fmtp_in = 0x0, fmtp_out = 0x0, codec_settings = {quality = 0, complexity = 0, enhancement = 0, vad = 0, vbr = 0, vbr_quality = 0, abr = 0, dtx = 0, preproc = 0, pp_vad = 0, pp_agc = 0, pp_agc_level = 0, pp_denoise = 0, pp_dereverb = 0, pp_dereverb_decay = 0, pp_dereverb_level = 0}, flags = 0, memory_pool = 0x0, private_info = 0x0, agreed_pt = 0 '\0'}
silence_frame = {codec = 0x0, source = 0x0, packet = 0x0, packetlen = 0, data = 0x0, datalen = 0, buflen = 0, samples = 0, rate = 0, payload = 0 '\0', timestamp = 0, seq = 0, ssrc = 0, m = SWITCH_FALSE, flags = 0}
silence_data = {0 <repeats 4160 times>}
silence_var = <value optimized out>
var = <value optimized out>
silence_val = 0
bypass_media_after_bridge = 0
vh = {session_a = 0x0, session_b = 0x0, up = 0}
vid_launch = 0
__func__ = "audio_bridge_thread"
__PRETTY_FUNCTION__ = "audio_bridge_thread"
#9 0x002076a3 in switch_ivr_multi_threaded_bridge (session=0xeb49808, peer_session=0x857d8938, input_callback=0, session_data=0x0, peer_session_data=0x0) at src/switch_ivr_bridge.c:876
ext = <value optimized out>
w_state = <value optimized out>
a_leg = (switch_ivr_bridge_data_t *) 0x82c2ce8
b_leg = (switch_ivr_bridge_data_t *) 0x864cf8c0
caller_channel = (switch_channel_t *) 0xeb51f68
peer_channel = (switch_channel_t *) 0x857e1098
status = 976433696
state = <value optimized out>
event = (switch_event_t *) 0x0
br = 1
__func__ = "switch_ivr_multi_threaded_bridge"
#10 0x00207a22 in uuid_bridge_on_soft_execute (session=0xeb49808) at src/switch_ivr_bridge.c:604
ready_a = 1 '\001'
state = <value optimized out>
other_channel = (switch_channel_t *) 0x857e1098
event = (switch_event_t *) 0x0
ready_b = <value optimized out>
channel = (switch_channel_t *) 0xeb51f68
other_session = (switch_core_session_t *) 0x857d8938
other_uuid = <value optimized out>
__func__ = "uuid_bridge_on_soft_execute"
#11 0x001e9641 in switch_core_session_run (session=0xeb49808) at src/switch_core_state_machine.c:436
event = (switch_event_t *) 0xa7700010
state = <value optimized out>
endstate = <value optimized out>
endpoint_interface = <value optimized out>
driver_state_handler = (const switch_state_handler_table_t *) 0xe498e0
application_state_handler = <value optimized out>
thread_id = 2053106576
env = {{__jmpbuf = {3150824, 2006352, 2920701, 2053104424, 644400294, 1546235356}, __mask_was_saved = 0, __saved_mask = {__val = {6117240, 1385740, 3150824, 11, 1, 2053104168, 1381393, 250000, 3074673368, 1385740, 1386752, 2053104200, 1, 3074673420, 3150824, 0, 136178016, 2053104216, 1386752, 2053104232, 2375712, 136178068, 3150824, 2053104264, 2340287, 136178064, 3150824, 246297200, 3150824, 246262456, 136178016, 2053104296}}}}
sig = <value optimized out>
__func__ = "switch_core_session_run"
__PRETTY_FUNCTION__ = "switch_core_session_run"
#12 0x001e6ef5 in switch_core_session_thread (thread=0x82c26c8, obj=0xeb49808) at src/switch_core_session.c:854
session = (switch_core_session_t *) 0xeb49808
event = <value optimized out>
event_str = 0x0
val = <value optimized out>
__func__ = "switch_core_session_thread"
__PRETTY_FUNCTION__ = "switch_core_session_thread"
#13 0x0024a3a6 in dummy_worker (opaque=0x82c26c8) at threadproc/unix/thread.c:138
No locals.
#14 0x0014f492 in start_thread () from /lib/i686/nosegneg/libpthread.so.0
No symbol table info available.
#15 0x0057ee1e in clone () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.

Thread 28 (process 26487):
#0 0x004ab402 in __kernel_vsyscall ()
No symbol table info available.
#1 0x0058bbfe in __lll_mutex_lock_wait () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#2 0x00519a28 in _L_lock_14107 () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#3 0x00518cb4 in free () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#4 0x0052f59f in tzset_internal () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#5 0x0052febd in tzset () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#6 0x00534646 in strftime_l () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#7 0x005345a6 in strftime () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#8 0x00000050 in ?? ()
No symbol table info available.
#9 0x002c405a in __func__.11670 () from /opt/freeswitch/lib/libfreeswitch.so.1
No symbol table info available.
#10 0x7cdfcd20 in ?? ()
No symbol table info available.
#11 0x005ee380 in __libc_utmp_file_name () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#12 0x003013e8 in ?? () from /opt/freeswitch/lib/libfreeswitch.so.1
No symbol table info available.
#13 0x7cdfcd58 in ?? ()
No symbol table info available.
#14 0x0024c1a5 in apr_strftime (s=0x0, retsize=0x0, max=0, format=0x0, xt=0x0) at time/unix/timestr.c:151
tm = {tm_sec = 4, tm_min = 10, tm_hour = 108, tm_mday = 2, tm_mon = 308, tm_year = 0, tm_wday = -28800, tm_yday = 2095042184, tm_isdst = 2375712, tm_gmtoff = 248386428, tm_zone = 0x3013e8 "��\025"}
#15 0x00000000 in ?? ()
No symbol table info available.

Thread 27 (process 26486):
#0 0x004ab402 in __kernel_vsyscall ()
No symbol table info available.
#1 0x0058bbfe in __lll_mutex_lock_wait () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#2 0x0052ff5b in _L_lock_1783 () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#3 0x0052fce4 in __tz_convert () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#4 0x0052e30c in localtime_r () from /lib/i686/nosegneg/libc.so.6
No symbol table info available.
#5 0x0024bfb7 in explode_time (xt=0x787fa1cc, t=1225842636452329, offset=0, use_localtime=1) at time/unix/time.c:90
tm = {tm_sec = 0, tm_min = 3997699, tm_hour = 1385740, tm_mday = 1841239, tm_mon = -1208562432, tm_year = 0, tm_wday = -1208567352, tm_yday = 6217716, tm_isdst = -1, tm_gmtoff = -1300573296, tm_zone = 0x787fa0bc "�\0230"}
tt = 1225842636
#6 0x0024c043 in apr_time_exp_lt (result=0x787fa1cc, input=1225839500853248) at time/unix/time.c:133
No locals.
#7 0x001d13cb in switch_time_exp_lt (result=0x787fa1cc, input=1225843795820540) at src/switch_apr.c:298
No locals.
#8 0x00222b20 in switch_log_printf (channel=SWITCH_CHANNEL_ID_LOG, file=0x2c971c "src/switch_core_state_machine.c", func=0x2c9bf2 "print_trace", line=225, userdata=0x0, level=SWITCH_LOG_CRIT, fmt=0x2c99b0 "Obtained %zd stack frames.\n") at src/switch_log.c:267
date = '\0' <repeats 79 times>
retsize = 0
tm = {tm_usec = 452329, tm_sec = -1208522368, tm_min = 1724416, tm_hour = 1791896, tm_mday = 1911472, tm_mon = -1208522368, tm_year = 1724416, tm_wday = 1793260, tm_yday = 1922208, tm_isdst = -1208522368, tm_gmtoff = 1724416}
data = 0x0
new_fmt = 0x0
ret = <value optimized out>
ap = 0x787fa29c "\n"
handle = (FILE *) 0x0
filep = 0x2c9720 "switch_core_state_machine.c"
funcp = 0x2c9bf2 "print_trace"
content = <value optimized out>
now = 1225842636452329
len = <value optimized out>
__PRETTY_FUNCTION__ = "switch_log_printf"
__func

Display posts from previous:

	VoIP Mailing List Archives Forum Index -> freeSWITCH Users	All times are GMT - 5 Hours Goto page 1, 2 Next
Page 1 of 2

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum