VoIP Mailing List Archives

[kawarod at laposte.net]

Dear list,

I've been playing with freeswitch for some time (2 months) and the fact
is that I'm very pleased with the functionnalities of this software.

I'd like to use FS as a SBC handling media and I'm doing some tests with
sipp to load the machine but I'm unable to bridge more than 60 calls
without seeing the CPU being loaded at 100%. I'm sure something is going
wrong with my setup but I'm unable to see what.

The test machine has the following specs:
Athlon XP 3500+ with 2GB of memory (I know this is not a high end
machine :p)

Freeswitch:/opt/freeswitch/log# cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 95
model name : AMD Athlon(tm) 64 Processor 3500+
stepping : 2
cpu MHz : 2199.973
cache size : 512 KB
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt
rdtscp lm 3dnowext 3dnow up rep_good pni cx16 lahf_lm svm extapic cr8_legacy
bogomips : 4402.97
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

I installed FS on a fresh debian 64:
Linux Freeswitch 2.6.26-1-amd64 #1 SMP Sat Jan 10 17:57:00 UTC 2009
x86_64 GNU/Linux

I set the ulimit parameters like those on the website:
freeswitch@internal> ...
Freeswitch:/opt/free-svn/bin# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 999999
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
real-time priority (-r) 0
stack size (kbytes, -s) 244
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


My network setup is the following:

SIPP machine (10.10.10.1/24)----------------vlan 55
----------(10.10.10.254/24) FS (10.10.20.254/24)-------------- vlan56
-------------------(10.10.20.100/24) OTHER STOCK FS


I launched sipp with:
sipp -sn uac_pcap -s 9999 -r 10 -l 80 -d 60000 -mi 10.10.10.1 -i
10.10.10.1 -mp 25000 10.10.10.254:5060

The dialplan on FS is very simple:
<?xml version="1.0" encoding="utf-8"?>
<!-- http://wiki.freeswitch.org/wiki/Dialplan_XML -->
<include>
<context name="default">

<extension name="hold_music_bridge">
<condition field="destination_number" expression="^9999$">
<action application="answer"/>
<action application="bridge"
data="sofia/external/9999@10.10.20.100"/>
</condition>
</extension>

</context>

</include>

FreeSWITCH Version 1.0.trunk (11560M) Started.
Crash Protection [Disabled]
Max Sessions[1000]
Session Rate[100]
SQL [Enabled]


The test is very simple: sipp dial 9999 that matches in my FS dialplan
and this is bridged to an other FS machine playing music on hold.
When I launch "top" I see after 30 to 40 s that FS consumes all the CPU
ressources (with a mean of 50-60 % before), with 80 calls.
When I set 70 calls, I have to wait 70-80 s before seeing the same issue.

Presence is set to false on the 2 profile.

I have the same issue with FS 1.0.2 that' s why I tried FS 11560.

When I use the FS machine as a router to test the packet per second
performance, I'm reaching 100Mbps with 8000pps in each direction (from
vlan 55 to vlan56) with less than 12% CPU. So that I don't think there's
an issue with the network.

Here is an "mpstat -P ALL 1" to show you what's happening suddenly with
70 bridge calls:
12:31:26 CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
12:31:27 all 3,00 0,00 3,00 0,00 1,00 4,00
0,00 89,00 6241,00
12:31:27 0 3,00 0,00 3,00 0,00 1,00 4,00
0,00 89,00 6241,00

12:31:27 CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
12:31:28 all 14,14 0,00 56,57 0,00 2,02 5,05
0,00 22,22 6035,35
12:31:28 0 14,14 0,00 56,57 0,00 2,02 5,05
0,00 22,22 6035,35

12:31:28 CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
12:31:29 all 24,75 0,00 67,33 0,00 0,99 6,93
0,00 0,00 5483,17
12:31:29 0 24,75 0,00 67,33 0,00 0,99 6,93
0,00 0,00 5483,17


The CPU is going from 89% idle to 0% in less than 2 seconds.

I know that I don't have to expect too much from this kind of hardware,
but it seems strange that the CPU power vanished so suddenly.

Thanks a lot for the guys that have read this long mail :p

kind regards,
rod





_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

[kawarod at laposte.net]

Dear list,

I've been playing with freeswitch for some time (2 months) and the fact
is that I'm very pleased with the functionnalities of this software.

I'd like to use FS as a SBC handling media and I'm doing some tests with
sipp to load the machine but I'm unable to bridge more than 60 calls
without seeing the CPU being loaded at 100%. I'm sure something is going
wrong with my setup but I'm unable to see what.

The test machine has the following specs:
Athlon XP 3500+ with 2GB of memory (I know this is not a high end
machine :p)

Freeswitch:/opt/freeswitch/log# cat /proc/cpuinfo
processor : 0
vendor_id : AuthenticAMD
cpu family : 15
model : 95
model name : AMD Athlon(tm) 64 Processor 3500+
stepping : 2
cpu MHz : 2199.973
cache size : 512 KB
fpu : yes
fpu_exception : yes
cpuid level : 1
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge
mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt
rdtscp lm 3dnowext 3dnow up rep_good pni cx16 lahf_lm svm extapic
cr8_legacy
bogomips : 4402.97
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management: ts fid vid ttp tm stc

I installed FS on a fresh debian 64:
Linux Freeswitch 2.6.26-1-amd64 #1 SMP Sat Jan 10 17:57:00 UTC 2009
x86_64 GNU/Linux

I set the ulimit parameters like those on the website:
freeswitch@internal> ...
Freeswitch:/opt/free-svn/bin# ulimit -a
core file size (blocks, -c) unlimited
data seg size (kbytes, -d) unlimited
scheduling priority (-e) 0
file size (blocks, -f) unlimited
pending signals (-i) unlimited
max locked memory (kbytes, -l) unlimited
max memory size (kbytes, -m) unlimited
open files (-n) 999999
pipe size (512 bytes, -p) 8
POSIX message queues (bytes, -q) unlimited
real-time priority (-r) 0
stack size (kbytes, -s) 244
cpu time (seconds, -t) unlimited
max user processes (-u) unlimited
virtual memory (kbytes, -v) unlimited
file locks (-x) unlimited


My network setup is the following:

SIPP machine (10.10.10.1/24)----------------vlan 55
----------(10.10.10.254/24) FS (10.10.20.254/24)-------------- vlan56
-------------------(10.10.20.100/24) OTHER STOCK FS


I launched sipp with:
sipp -sn uac_pcap -s 9999 -r 10 -l 80 -d 60000 -mi 10.10.10.1 -i
10.10.10.1 -mp 25000 10.10.10.254:5060

The dialplan on FS is very simple:
<?xml version="1.0" encoding="utf-8"?>
<!-- http://wiki.freeswitch.org/wiki/Dialplan_XML -->
<include>
<context name="default">

<extension name="hold_music_bridge">
<condition field="destination_number" expression="^9999$">
<action application="answer"/>
<action application="bridge"
data="sofia/external/9999@10.10.20.100"/>
</condition>
</extension>
</context>

</include>

FreeSWITCH Version 1.0.trunk (11560M) Started.
Crash Protection [Disabled]
Max Sessions[1000]
Session Rate[100]
SQL [Enabled]


The test is very simple: sipp dial 9999 that matches in my FS dialplan
and this is bridged to an other FS machine playing music on hold.
When I launch "top" I see after 30 to 40 s that FS consumes all the CPU
ressources (with a mean of 50-60 % before), with 80 calls.
When I set 70 calls, I have to wait 70-80 s before seeing the same issue.

Presence is set to false on the 2 profile.

I have the same issue with FS 1.0.2 that' s why I tried FS 11560.

When I use the FS machine as a router to test the packet per second
performance, I'm reaching 100Mbps with 8000pps in each direction (from
vlan 55 to vlan56) with less than 12% CPU. So that I don't think there's
an issue with the network.

Here is an "mpstat -P ALL 1" to show you what's happening suddenly with
70 bridge calls:
12:31:26 CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
12:31:27 all 3,00 0,00 3,00 0,00 1,00 4,00
0,00 89,00 6241,00
12:31:27 0 3,00 0,00 3,00 0,00 1,00 4,00
0,00 89,00 6241,00

12:31:27 CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
12:31:28 all 14,14 0,00 56,57 0,00 2,02 5,05
0,00 22,22 6035,35
12:31:28 0 14,14 0,00 56,57 0,00 2,02 5,05
0,00 22,22 6035,35

12:31:28 CPU %user %nice %sys %iowait %irq %soft
%steal %idle intr/s
12:31:29 all 24,75 0,00 67,33 0,00 0,99 6,93
0,00 0,00 5483,17
12:31:29 0 24,75 0,00 67,33 0,00 0,99 6,93
0,00 0,00 5483,17


The CPU is going from 89% idle to 0% in less than 2 seconds.

I know that I don't have to expect too much from this kind of hardware,
but it seems strange that the CPU power vanished so suddenly.

Thanks a lot for the guys that have read this long mail :p

kind regards,
rod


_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

[anthony.minessale at g...]

Which of the 2 machines has the load issue? You said it was one box calling the other.

You have 2 major things against you, single CPU and AMD, but you should at least be able to get in the vicinity of 800-1000 calls on a box like that.

Are you calling the default 9999? It's not really an appropriate extension for load testing.
On the terminating box you should set up a manual extension that is the first one in the dial plan
to play a wav file from preferably a ram disk or /tmp

If you do plan on using this in production accept nothing less than a multi-core intel machine with at least 4 cores, the more cores the better because that parallel processing is where FS gets it's atvantage.



On Fri, Jan 30, 2009 at 5:56 AM, rod <kawarod@laposte.net (kawarod@laposte.net)> wrote:

[peder at networkoblivi...]

Given the choice between dual core and dual processor, would one work
better than the other for FS? Assuming roughly the same speed. Or
would it not really have a measurable difference assuming same speed and
both Intel?

Anthony Minessale wrote:

[anthony.minessale at g...]

I have not compared them directly to be able to answer. I do know that the more cores the better (even slower ones) because each core divides the entire workload of the scheduler and increases the total threads that can work in parallel.

also you can start FS with -hp (high performance) which activates the round-robin scheduler, increases some ulimits and forces all consumed memory to be permanently resident with mlockall (no swapping)


On Fri, Jan 30, 2009 at 11:05 AM, peder@networkoblivion.com (peder@networkoblivion.com) <peder@networkoblivion.com (peder@networkoblivion.com)> wrote:

[kawarod at laposte.net]

Thanks Anthony,

the setup is like this:

sipp server ---- FS 1 ---- FS2

FS1 is the AMD CPU that has only one extension in dialplan that bridges
9999 to FS2. 9999 is the first extension in FS2 dialplan that plays moh,
FS2 has no CPU pbm.

FS1 is maxing out at 60 bridged calls without your option -hp.

Using -hp, I'm now able to bridge 200 concurrent calls (a great
improvement) and the system is still reactive. CPU load is high but not
100% and as the system responds well, I think that doesn't matter. The
2GB of memory are completely consumed (top command shows 700MB for FS
process).

I understand that FS1 server is not the best hardware platform, and I'm
waiting for new 4 cores server for testing.
I will update those numbers when testing with the new hardware.

regards,
rod.

Anthony Minessale wrote:

[krice at freeswitch.org]

Dont forget there are several things you can do to increase performance...

1) where possible use bypass media or media proxy modes
2) mount freeswitch/db as a ram drive (if you are using voicemail with the internal FS DBs you'll need a way to make this persistant across reboots)
3) see the wiki for setting reasonable ulimits
4) (this is my oppinion others may vary) dont use mod_cdr_csv
5) turn off (or reduce logging) in switch.conf.xml

all of these thing can greatly improve performance.

On Mon, Feb 2, 2009 at 1:04 AM, rod <kawarod@laposte.net (kawarod@laposte.net)> wrote:

[kawarod at laposte.net]

Hi Ken,

1) I'd like to use FS to hide topology, so bypass media is not possible
2) done
3) done
4) not used
5) i'm using this ins switch.xml -> <param name="loglevel"
value="info"/>, if you think an other log level is more suitable.

Regarding logging, I can see in console and in the freeswitch.log that
there is still a lot of NOTICE logging, see below:
2009-02-02 08:33:56 [NOTICE] switch_core_session.c:960
switch_core_session_thread() Session 8721
(sofia/internal/sipp@10.10.10.1:5060) Ended
2009-02-02 08:33:56 [NOTICE] switch_core_session.c:962
switch_core_session_thread() Close Channel
sofia/internal/sipp@10.10.10.1:5060 [CS_HANGUP]
2009-02-02 08:33:56 [NOTICE] switch_core_session.c:960
switch_core_session_thread() Session 8722
(sofia/external/9998@10.10.20.100) Ended
2009-02-02 08:33:56 [NOTICE] switch_core_session.c:962
switch_core_session_thread() Close Channel
sofia/external/9998@10.10.20.100 [CS_HANGUP]
2009-02-02 08:33:56 [NOTICE] sofia.c:3164 sofia_handle_sip_i_state()
Channel [sofia/external/9998@10.10.20.100] has been answered
2009-02-02 08:33:56 [WARNING] mod_sofia.c:740 sofia_read_frame()
Changing codec ptime to 30. I bet you have a linksys/sipura =D

Do you have any idea where I can switch off this kind of logging. I
thought it should be in /dialplan/internal.xml, but I see that in
internal.xml -> <param name="debug" value="0"/>

thanks a lot for your suggestion.

regards,
rod

Ken Rice wrote:

[jaybinks at gmail.com]

for topology hiding, use proxy media.
it means FS ignores the RTP stream totally, and just passes it through.

On Mon, Feb 2, 2009 at 5:36 PM, rod <kawarod@laposte.net (kawarod@laposte.net)> wrote:

[krice at freeswitch.org]

If you don't have to transcode, using proxy media mode will still save you
some CPU time. This is 1/2 way between bypass media and the default media
interactive mode. The other draw back to this mode is if you are using FS to
clean up RTP and DTMF you loose those functions but they are not needed in
most use cases.

As far as the log level goes, I found that once I had things stable setting
the loglevel to helped a good deal... Info is probably a bit too high of a
loglevel I would probably go for CRIT or ERR (2 or 1 respectively) if you
insist on leaving logging turned on... On a busy system these can and will
generate a good deal of activity (and disk IO if using mod_logfile)

Ken

[kawarod at laposte.net]

Hi Ken, Jay,

thanks for pointing to proxy media, I will test.

Ken, you are right, I was brain damaged (a stupid mistake) when setting
INFO cause this kind of level could be very verbose. I'm switching to
CRIT or ERR.

Thanks guys,
rod.

thanks for

Ken Rice wrote:

[saeedahmad1981 at gmai...]

Hi Rod,

Could you please share how you configured Sipp & FS to create a test
environment? Especially the dial plan, sofia settings etc..., actually I am
a newbie. I want to test it on a single FS machine.

Kind Regards
Saeed
-----Original Message-----
From: freeswitch-users-bounces@lists.freeswitch.org
[mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of rod
Sent: Monday, February 02, 2009 11:00 AM
To: freeswitch-users@lists.freeswitch.org
Subject: Re: [Freeswitch-users] Strange Performance when using as SBC

Hi Ken, Jay,

thanks for pointing to proxy media, I will test.

Ken, you are right, I was brain damaged (a stupid mistake) when setting
INFO cause this kind of level could be very verbose. I'm switching to
CRIT or ERR.

Thanks guys,
rod.

thanks for

Ken Rice wrote:

[kawarod at laposte.net]

Some update:
- I switched log level to "err"
- I'm now using proxy-media
- and I erased the directive answer in the dialplan (useless and
seems that it consumes lots of CPU, don't know why)

the dialplan now looks like this:
<extension name="hold_music_bridge">
<condition field="destination_number" expression="^9999$">
<action application="set" data="proxy_media=true"/>
<action application="bridge"
data="sofia/external/9999@10.10.20.100"/>
</condition>
</extension>

instead of
<extension name="hold_music_bridge">
<condition field="destination_number" expression="^9999$">
<action application="answer"/>
<action application="bridge"
data="sofia/external/9999@10.10.20.100"/>
</condition>
</extension>

The box is now able to bridge 300 calls with 20-30% of free CPU. I will
run a long term test.

I see this error in the log and don't understand exactly if somebody
could help (I'm running latest trunk 11592M):
2009-02-02 13:29:54 [ERR] switch_core_io.c:117
switch_core_session_read_frame() sofia/external/9998@10.10.20.100 has no
read codec.

regards,
rodrigue

rod wrote:

[kawarod at laposte.net]

Hi Saeed,

I just created an account to share my setup on the wiki. I will detail
all the steps for a clean install of a debian64 lenny with FS used as a
SBC (next step is to try the new LCR module )and what I'm doing do
stress the server.

I wrote nothing at this time so please be patient, I'm waiting for my
new hardware so that I will detail as much as possible what I'll do.

For beginning I suggest you reading the start page on the wiki,
especially these pages:
-http://wiki.freeswitch.org/wiki/Getting_Started_Guide
-http://wiki.freeswitch.org/wiki/Dialplan_XML

maybe you could tell more about the linux distribution you're using so
that I can give you some pointers for sipp...

regards.
rod.


Saeed Ahmed wrote:

[saeedahmad1981 at gmai...]

Thanks rod for a quick answer,

FS is installed on Ubuntu Server.

I am planning to replace Nextone SBC with FS, Later I'll also use openZAP to
communicate with TDM but this all depends how much calls it can take, or
maybe we can also do something in clustering environment ( I am not sure
about it). But thanks again and any further help will be highly appreciated.


Kind Regards
Saeed Ahmed Tariq



-----Original Message-----
From: freeswitch-users-bounces@lists.freeswitch.org
[mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of rod
Sent: Monday, February 02, 2009 1:53 PM
To: freeswitch-users@lists.freeswitch.org
Subject: Re: [Freeswitch-users] Strange Performance when using as SBC

Hi Saeed,

I just created an account to share my setup on the wiki. I will detail
all the steps for a clean install of a debian64 lenny with FS used as a
SBC (next step is to try the new LCR module )and what I'm doing do
stress the server.

I wrote nothing at this time so please be patient, I'm waiting for my
new hardware so that I will detail as much as possible what I'll do.

For beginning I suggest you reading the start page on the wiki,
especially these pages:
-http://wiki.freeswitch.org/wiki/Getting_Started_Guide
-http://wiki.freeswitch.org/wiki/Dialplan_XML

maybe you could tell more about the linux distribution you're using so
that I can give you some pointers for sipp...

regards.
rod.


Saeed Ahmed wrote: