VoIP Mailing List Archives

[helmut.kuper at ewetel.de]

Hello,

since a few days I observe a high CPU load of my FS server, but I have
no idea what it could be. There are only a few sessions running and
there is only a few log activity. 2 days ago I restarted FS, but no
change. The top command shows this:

top - 15:02:33 up 106 days, 30 min, 4 users, load average: 0.24, 0.35,
0.42
Tasks: 190 total, 1 running, 189 sleeping, 0 stopped, 0 zombie
Cpu(s): 7.2%us, 12.2%sy, 0.0%ni, 80.0%id, 0.2%wa, 0.2%hi, 0.2%si,
0.0%st
Mem: 4151776k total, 4003664k used, 148112k free, 414708k buffers
Swap: 15623204k total, 88k used, 15623116k free, 2021412k cached

PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
14048 ippbx 20 0 1555m 1.2g 10m S 43 30.5 333:11.03 freeswitch
14049 ippbx 20 0 1555m 1.2g 10m S 0 30.5 5:06.88 freeswitch
14054 ippbx 20 0 1555m 1.2g 10m S 0 30.5 4:14.38 freeswitch
14055 ippbx 20 0 1555m 1.2g 10m S 0 30.5 4:58.50 freeswitch
14057 ippbx 20 0 1555m 1.2g 10m S 0 30.5 13:05.20 freeswitch
20511 ippbx 20 0 1555m 1.2g 10m S 0 30.5 0:00.14 freeswitch


so only one process (PID: 14048) is causing that load. It's not the
parent process (the initial FS startup process) as ps -elf shows:

ippbx@ippbx-prod-node0:~/ippbx.prod$ ps -eLf | grep frees
ippbx 14033 1 14033 0 28 Mar23 ? 00:00:01
bin/freeswitch -nc
ippbx 14033 1 14034 0 28 Mar23 ? 00:00:08
bin/freeswitch -nc
ippbx 14033 1 14035 0 28 Mar23 ? 00:03:39
bin/freeswitch -nc
ippbx 14033 1 14036 0 28 Mar23 ? 00:00:07
bin/freeswitch -nc
ippbx 14033 1 14037 0 28 Mar23 ? 00:00:00
bin/freeswitch -nc
ippbx 14033 1 14038 0 28 Mar23 ? 00:00:00
bin/freeswitch -nc
ippbx 14033 1 14039 0 28 Mar23 ? 00:00:02
bin/freeswitch -nc
ippbx 14033 1 14042 0 28 Mar23 ? 00:03:41
bin/freeswitch -nc
ippbx 14033 1 14043 0 28 Mar23 ? 00:00:03
bin/freeswitch -nc
ippbx 14033 1 14044 0 28 Mar23 ? 00:00:01
bin/freeswitch -nc
ippbx 14033 1 14045 0 28 Mar23 ? 00:00:25
bin/freeswitch -nc
ippbx 14033 1 14046 0 28 Mar23 ? 00:01:20
bin/freeswitch -nc
ippbx 14033 1 14047 0 28 Mar23 ? 00:05:32
bin/freeswitch -nc
ippbx 14033 1 14048 7 28 Mar23 ? 05:33:35
bin/freeswitch -nc
ippbx 14033 1 14049 0 28 Mar23 ? 00:05:07
bin/freeswitch -nc
ippbx 14033 1 14050 0 28 Mar23 ? 00:01:01
bin/freeswitch -nc
ippbx 14033 1 14051 0 28 Mar23 ? 00:25:43
bin/freeswitch -nc
ippbx 14033 1 14052 0 28 Mar23 ? 00:00:01
bin/freeswitch -nc
ippbx 14033 1 14054 0 28 Mar23 ? 00:04:14
bin/freeswitch -nc
ippbx 14033 1 14055 0 28 Mar23 ? 00:04:58
bin/freeswitch -nc
ippbx 14033 1 14056 0 28 Mar23 ? 00:06:23
bin/freeswitch -nc
ippbx 14033 1 14057 0 28 Mar23 ? 00:13:05
bin/freeswitch -nc
ippbx 14033 1 14058 0 28 Mar23 ? 00:00:00
bin/freeswitch -nc
ippbx 14033 1 14059 0 28 Mar23 ? 00:00:00
bin/freeswitch -nc
ippbx 14033 1 20518 0 28 15:02 ? 00:00:00
bin/freeswitch -nc
ippbx 14033 1 20519 0 28 15:02 ? 00:00:00
bin/freeswitch -nc
ippbx 14033 1 20521 0 28 15:02 ? 00:00:00
bin/freeswitch -nc
ippbx 14033 1 20522 0 28 15:02 ? 00:00:00
bin/freeswitch -nc
ippbx 20526 19854 20526 0 1 15:03 pts/0 00:00:00 grep frees

Doing a strace on PID 14048 prints tons of "epoll_wait(21, {}, 4,
0) = 0" lines on the screen, which eats all of my desktop
pc's cpu power :/

So can a developer say what this is, or what and how should I debug to
find out the cause of this?
Can I shot it down via kill or "kill -9" without crashing FS totally?

regards
helmut




_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

[mrene_lists at avgs.ca]

gcore -o fs [pid here]
gdb /path/to/fs core.file
thread apply all bt

then look for the thread and show me the backtrace.

Math

On 26-Mar-09, at 10:12 AM, Helmut Kuper wrote:

[helmut.kuper at ewetel.de]

Hi Mathieu,

thx for the help

The epoll_wait function on PID 14048 listens on fd 21 which points to
"/anon_inode:[eventpoll]"



On 26.03.2009 15:26, Mathieu Rene wrote:

[helmut.kuper at ewetel.de]

Hi Mathieu

I straced process 14047 as well and I found that it also epolls a
anon_inode. But the process' epoll call lokk slightly different that
14048's:

epoll_wait(13, {}, 4, 1000) = 0

Here a timeout is given. 14048 is not using a timeout (0). Maybe this
helps you ...

regards
Helmut


On 26.03.2009 16:01, Helmut Kuper wrote:

[brian at freeswitch.org]

Before we go any further... what SVN rev are you on? And by heavy load what does your load average say?

/b

On Mar 26, 2009, at 10:01 AM, Helmut Kuper wrote:

[helmut.kuper at ewetel.de]

Hello Brian,

On 26.03.2009 16:07, Brian West wrote:

[helmut.kuper at ewetel.de]

Hello,

today I killed that special thread via "kill -9" a simple kill didn't
helped. Unfortunately this led to a normal shutdown of FS although I
killed not the parent process.

After restart of FS the server has a normal load again.

regards and a nice weekend
Helmut


_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

[rupa at rupa.com]

kill -9 on a thread will kill the process which kills freeswitch.

On Fri, Mar 27, 2009 at 10:09 AM, Helmut Kuper <helmut.kuper@ewetel.de (helmut.kuper@ewetel.de)> wrote:

[helmut.kuper at ewetel.de]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

I still have this problem. From the day of starting up freeswitch two
threads are consuming slowly more and more CPU power. In parallel FS
virtual and physical memory usage grows slowly as well. FS is up for 6
days now and served 3160 sessions. Virtual memory usage has grown from
200MB to 1.1GB (18,1%) and is still growing. CPU is mostly around 25%
with lowest of 17% and a maximum of 50% (all on a 32 bit 4 CPU core
system) and is still growing.

There are two FS-Threads with nearly same CPU usage of currently
around 20% each (I used htop for this):

strace Thread 1 (I guess this is the sofia/sip thread):
epoll_wait(21, {}, 4, 0) = 0
epoll_wait(21, {}, 4, 0) = 0
epoll_wait(21, {{EPOLLIN, {u32=2, u64=2}}}, 4, 0) = 1
ioctl(24, FIONREAD, [268]) = 0
recvmsg(24, {msg_name(16)={sa_family=AF_INET, sin_port=htons(1068),
sin_addr=inet_addr("85.16.245.249")}, msg_iov(1)=[{"SIP/2.0 200
Ok\r\nVia: SIP/2.0/UDP"..., 268}], msg_controllen=0, msg_flags=0}, 0)
= 268
gettimeofday({1239012809, 343580}, NULL) = 0
gettimeofday({1239012809, 343645}, NULL) = 0
epoll_wait(21, {}, 4, 0) = 0
epoll_wait(21, {}, 4, 0) = 0
...


strace Thread 2:
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
select(0, NULL, NULL, NULL, {1, 0}) = 0 (Timeout)
...

I use FreeSWITCH Version 1.0.trunk (12347M)



regards
helmut
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.7 (MingW32)

iD8DBQFJ2ddp4tZeNddg3dwRAlkXAJ9fIwpJw6u18JPhFC4hzB+0Z1iAbgCfW7AE
dnrmpXDLVOnWtjwFKMoVw48=
=zzZ9
-----END PGP SIGNATURE-----


_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

[carthick84 at gmail.com]

It could be due to registrations. I am currently trying to troubleshoot this problem. I used  a sipp scenario to authenticate with fs and register about 2000 different accounts (absolutely no calls made on the test setup). Memory usage increases continuously and does not decrease at all and crosses more than 1 GB in a few hours. On the other hand, there is another fs setup with bypass media turned on and no registrations and is up for almost 45 days without restart and has consumed only about 95 MB of memory and twice as much virtual memory.
B Karthik

[brian at freeswitch.org]

If you guys are not on rev 12914 then you'll need to update.

/b

On Apr 6, 2009, at 9:04 AM, B Karthik wrote:

[helmut.kuper at ewetel.de]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

in my scenario I have a reregistration interval of 60 seconds and 32 sip
phones connected. So I have a good amount of registrations. Additionally
each phone subscribes to itself for MWI and some phone subscribes to
others for BLF.

Registrar database looks fine. No unused entries there.

First I will upgrade to recent svn trunk. If that doesn't help, I will
run valgrind on my production system and hope that my machine is strong
enough to deliver its service even with valrgind.

regards
Helmut


On 06.04.2009 16:04, B Karthik wrote:

[carthick84 at gmail.com]

I updated to the latest revision. No Luck

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hello,

in my scenario I have a reregistration interval of 60 seconds and 32 sip
phones connected. So I have a good amount of registrations. Additionally
each phone subscribes to itself for MWI and some phone subscribes to
others for BLF.

Registrar database looks fine. No unused entries there.

First I will upgrade to recent svn trunk. If that doesn't help, I will
run valgrind on my production system and hope that my machine is strong
enough to deliver its service even with valrgind.

regards
Helmut


On 06.04.2009 16:04, B Karthik wrote:

[anthony.minessale at g...]

did you both follow the policy to upgrade?
stop fs
type make current
restart fs

if you do not rebuild sofia too (only happens in make current)
I just fixed all the problems with these symptoms, 38 million registrations in a 2 day span using 62mb

btw,
did we not make the policy clear enough about not reporting bugs on the mailing list?



On Mon, Apr 6, 2009 at 10:02 AM, B Karthik <carthick84@gmail.com (carthick84@gmail.com)> wrote:

[carthick84 at gmail.com]

yes, i did exactly as you mentioned. I will try building again from a fresh checkout. I am sorry about not following the policy, I didn't intend to report it as a bug since i was still unsure that it could be a problem in Freeswitch.


did you both follow the policy to upgrade?
stop fs
type make current
restart fs

if you do not rebuild sofia too (only happens in make current)
I just fixed all the problems with these symptoms, 38 million registrations
in a 2 day span using 62mb

btw,
did we not make the policy clear enough about not reporting bugs on the
mailing list?



On Mon, Apr 6, 2009 at 10:02 AM, B Karthik <carthick84@gmail.com> wrote: