VoIP Mailing List Archives

[rs at runsolutions.com]

Hello List,

I have read the current thread about scalability and I would need some
advice about a callcenter setup:

First where I come from:
I have lot's of problems with an asterisk solution. I have regular
crash's and lock-ups, with downgrading and other stuff i got it
somewhat stable, but have nevertheless regular hickups. I am desperate
and want to get rid of asterisk and I hope that freeSwitch will
provide me with a more stable solution.


Our Setup (really nothing special):
* 1 Asterisk box, New IBM Hardware (3 month old), 2 HE rack server, 3
GIG of RAM, Xircom analog switch connected to mobile stations for 4
different providers, Digium 4port cards TP400<something>
* 8 queues
* ~60 agents (which logon, logoff, pause, unpause), not more than 40
concurrently online
* ~ 7K - 9K calls (well, CDR entries) a day (not that much for a bpx)
* Music on Hold in the call-queues
* No special announcement
* Transfers between calls in queues and different agents as well as
non agents (i mention this because we have transfer related chrashes
in asterisk)

The current Problems:
* Lockups with different causes (ranging from calls not terminated to
heavy thread locking through the AMI interface)
* Crashes and library aborts (pthread, libc, crashes related to music
on hold, app_queue, transfers)

We used Asterisk: 1.4.23, 1.4.24, 1.4.26rc3, 1.4.26rc5, 1.4.26 and are
now back to 1.4.21.2 (stock debian) as anything beyond that is for
whatever reason highly unstable for our szenario. Maybe we should have
been segmenting the box into one asterisk dedicted to talking to the
hardware, one especial for queue/sip handling, i do not know. (all
issues are well documented in issues.asterisk.org, but it seems to be
very, very difficult to get to the bottom of them as they exist since
1.4.23 as it seems and are open until know and not fixable since month.)


Now, I really would appreciate some success-stories on how you guys
managed to get a stable pbx system with freeSWITCH in regard of HA and
scalability:

* How to segment freeSWITCH? Or is it stable enough to handle all in
one for such a szenario as outlined above?
* What would be the best strategie for High Availability / Failover?
-> I read in the WIKI (featurelist) that Livemigration of calls from
one box to another should be possible?
-> I was thinking about using memcached for storing all state
information so another freeswitch box can takeover calls from the
first box if it dies, is this possible? If so, how?
-> Is there anotherway to somehow configure freeSWITCH that in the
event of a crash i do not loose the current established calls?

Basically I just want a stable PBX where I do not have to fear every
day it will core-dump or abort or Lock up. Is freeSWITCH mature enough
so i can sleep at night for at least 3 month without a crash?



Thank you for your Time and help in advance, and I am more than
willing to take all the information gathered here and create a wiki
page to help other people with the same questions/problems.

best
Ray


--
Raimund Sacherer
-
RunSolutions
Open Source It Consulting
-

Parc Bit - Centro Empresarial Son Espanyol
Edificio Estel - Local 3D
07121 - Palma de Mallorca
Baleares


_______________________________________________
FreeSWITCH-users mailing list
FreeSWITCH-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

[gmaruzz at celliax.org]

Usually you don't need to worry about stability issues with FS.

For scalability, peoples tend to use openser or some other sip
loadbalancer in fron of fs, but you probably would not need that.

Live migration of calls is not yet possible, tough.

-giovanni



On Fri, Aug 28, 2009 at 8:52 PM, Raimund Sacherer<rs@runsolutions.com> wrote:

[msc at freeswitch.org]

On Fri, Aug 28, 2009 at 12:11 PM, Giovanni Maruzzelli <gmaruzz@celliax.org (gmaruzz@celliax.org)> wrote:

[dave at 3c.co.uk]

Hi Raimund,

One FreeSWITCH box will be quite enough to handle the call volumes that
you're talking about, and it ought to be much more stable than the
Asterisk solution which you've outlined below.

It's probably best to forget about live failover without calls dropping
- this isn't something that's supported, and there'd be a lot of work to
do to develop code to keep two boxed in sync. Once you get used to a
stable solution - i.e. something which doesn't crash - then live
failover, HA, etc., will seem somewhat irrelevant.

I recently did some work on an FS-based call center solution - drop me a
note if I could be of any help with yours.

Cheers --

Dave

[diego.viola at gmail.com]

Hi David,

What have you used on FS for call center, mod_fifo?

Can you describe your experience with that, I'm currently interested in call center + FS scenario.

Diego

On Fri, Aug 28, 2009 at 9:01 PM, David Knell <dave@3c.co.uk (dave@3c.co.uk)> wrote:

[diego.viola at gmail.com]

Yes, FreeSWITCH is a system that you can trust 100%. I have switched my Asterisk servers to FreeSWITCH and have peace now.

If I were you I would get rid of Asterisk and use FreeSWITCH, FS will handle all what you want very well.

And I agree with David, fail-over is kinda irrelevant since the FS doesn't crash like Asterisk does.

Diego


On Fri, Aug 28, 2009 at 6:52 PM, Raimund Sacherer <rs@runsolutions.com (rs@runsolutions.com)> wrote:

[dave at 3c.co.uk]

Hi Diego,

We didn't use mod_fifo; we built our own queues using an application
hanging off the event socket. This was partly down to typical
programmer hubris, and partly to allow us to do things that mod_fifo
might not without either (a) trawling through mod_fifo, or (b) pestering
Anthony.

As a brief outline:
- operators each live in their own conference
- supervisors can add themselves to these conferences for monitoring or
coaching
- callers sit in queues for the various skills that the operators have.

When an operator is free, look at each queue for the skills that the
operator has, and choose the caller who's been waiting longest.

I used a one-thread-per-call model, so there's a little bit of thought
needed to get the access to shared structures right but, essentially,
given the low frequency of access to them and the fact that nothing was
time-critical at the sub-second level, I just wrapped all of the
accesses in a single global mutex, which was easy and is pretty
foolproof. The latter is important, given that it was me writing the
stuff

Cheers --

Dave

[steve.kurzeja at gmail...]

On Sat, Aug 29, 2009 at 2:34 PM, Diego Viola <diego.viola@gmail.com (diego.viola@gmail.com)> wrote:

[rs at runsolutions.com]

Oh yeah, that would be so helpfull for my situation, as my client *demands* now a solution where he can press a big red button and all fails over to another box. Hi es totally scared because of the Lockups in Asterisk which under specific situations including AMI, Automated Call Setup, and murphy led to a lockup of the entire machine, no console was working anymore, only cold-reset could do it.

So, IF there is the possibility for life-takeover, / failover etc. I would love to here how has been done.


I am very experienced with openvz and use for about two years now only openvz virtualization servers for anything because of live-migration etc. But as I am new in this company we could not adopt this until now.


So Please Ken, if you can, describe what need's to be done to get a failover / takeover working (an outline would be enough)


Thanks in Advance

--
Raimund Sacherer
-
RunSolutions
Open Source It Consulting
-

Parc Bit - Centro Empresarial Son Espanyol
Edificio Estel - Local 3D
07121 - Palma de Mallorca
Baleares






On Aug 29, 2009, at 11:58 AM, Steve Kurzeja wrote:

[rs at runsolutions.com]

Thinking about it, maybe we can create a solution, if some of us work together:

My strength are in virtualization, linux, development, databases, integration, etc.
What I do not now much about is how SIP (and everything else for that matter in the Voice world) works under the hood, and how it's implemented in FS.


I know that the state information for a call has to be stored and retrieved somewhere and somehow, only I do not know that part. What I know is that it hast to be do-able to store all the stream information (ip's, port's, current state's, etc.) in a very fast database (e.g. my idea would be memcached) so another FS could just take this information and take over the call, maybe you loose a second of voice, maybe you loose the recorded call file or a part of it, but that should be it. (SipFoundry has a boxed opensource PBX, which, of course is not flexible like FreeSWITCH or Asterisk, but has Call Live Migration and Call Live Failover integrated!).


What I want is for my company to be able to sell a 99.99 uptime PBX (we do mostly call-center related stuff), which can scale well, and can grow with the company without lot's of hassles, my Dream would be:


To begin with:


One Hardware Node with the essential hardware (digium cards for example).
On this node are OpenVZ virtualized containers:
[VirtCnt1: FS which only talks to the Hardware and forwards everything] = Could be replaced with hardware media gateway, etc.
[VirtCnt2: FS which handles the PBX] \___ Loadbalanced, with odbc or xml, Failover, Livetakeover
[VirtCnt3: FS which handles the PBX] /
[VirtCnt4: Database for state information] (maybe something as resource-friendly as memcached? ressource heavvy database?)


With this we can achieve all this:


Problem with VirtCnt2 (e.g. crash, lock, ...)
* VirtCnt3 can take over.
-> You are free without stress to investigate the problem, you can debug and analyze whyle the machine is still running
-> you can also create a machine-state-dump of the virtual container, dump the container as well, copy the data to your lab and restore the machine up the state which it was running with the problem, so you can liveinvestigate it in the lab (some prerequirements given, but easy doable)
-> just think about the possibility of better bugreports because someone can take the time to read out all the data with GDB to investigate the proper cause of a machine Lock!


You want to upgrade to a new FreeSWITCH version?
* Take VirtCnt2 out of the LoadBalancing Scheme,
* Stop it, Clone it,
* Upgrade FreeSWITCH in the cloned Container
* Start the cloned container
* if there's something wrong, stop it and restart the original VirtCnt2
-> No problem at all, you can Test on the Live Hardware, with part of the Live users (maybe a low-volume queue) to be sure everything works out fine before you activate the full loadbalance


Server on it's own can't handle the load
* Buy new machine
* Setup Hardware Node
* Livemigrate VirtCnt3 (no downtime)


Now the first Server with the VrtCnt1 and VirtCnt2 as well has to much load
* Buy new machine
* Setup Hardware Node
* Livemigrate VirtCnt2 (no downtime)
-> Now you have a 3 server solution (1 mediaprox, 2 loadbalanced / failover PBXes) out of the first box you bought, without headaches, because the system was built for it from the beginning!


The Database drains to much?
* Buy new machine
* Setup Hardware Node
* Livemigrate database VirtCnt4 (no downtime)


You want to upgrade Hardware/Kernel in Hardware node 1?
* Livemigrate VirtCnt2 to a hotstandby machine, or to the other PBX machine, upgrade the hardware, Re-Livemigrate the containers. (no downtime)
* OR just break the loadbalancing, wait until all current calls are teared down correctly, upgrade machine, reenable the loadbalancer


You want an exact copy of the first server for Hardware HA?
* Buy new machine
* Setup Hardware node
* Buy hardware PRI switchover box
* Clone VirtCnt1 - VirtCnt4 to the new machine
* Make basic failover configuration




-> the sky's the limit, as the saying goes ...




So, I can do all the openvz stuff and the integration with database / memcached / heartbeat / whatever is needed here, someone there to be willing to work with me on this on the FreeSWITCH side? or at least provide me with the necessary information about what's needed / how to talk / what states from FreeSWITCH?


I know this seems very ambitious but if this could be made in a rather relativly easy to setup package, with good documentation, it would be a boost for FreeSWITCH, i am sure, because after all this is what everyone is grown accustomed to from good old phone companys and the good old pbx's: carrier grade uptimes ...


Thanks for everyone reading up until here,
all the best,


Ray






--
Raimund Sacherer
-
RunSolutions
Open Source It Consulting
-

Parc Bit - Centro Empresarial Son Espanyol
Edificio Estel - Local 3D
07121 - Palma de Mallorca
Baleares






On Aug 29, 2009, at 3:17 PM, Raimund Sacherer wrote:

[brian at freeswitch.org]

I was able to do this using OpenVZ, You can get away with it on
smaller instances... like if you're doing one instance per company but
don't expect live migration to work as well on large instances with
thousands of calls up at once. You need a fast network, fast disks and
to follow the howto on the wiki.

/b

On Aug 29, 2009, at 4:58 AM, Steve Kurzeja wrote:

[anthony.minessale at g...]

We have previously estimated the development of live fail over (after a box dies where live migration is no longer possible) to exceed 100k in development costs.
It requires several additions to the sofia sip library, freeswitch and a dependancy on some other code we would have to implement to manage it.
It may or may not be worth it to raise that kind of funding just to avoid an occasional disaster.
Then there is a matter of securing the time of the developers necessary to carry out the implementation.

[dave at 3c.co.uk]

Hi Raimund,

The difficult bit in all of this is having calls seamlessly transferred
from one box to another when the first box dies. There's a lot of state
associated with a call, not all of which is easy to replicate across.

99.99% uptime implies an average of no more than 15 seconds downtime
during a 40 hour business week (or about one unscheduled reboot every
couple of months), which is easily achievable using FreeSWITCH (and, in
my experience, Asterisk) on standard hardware.

People agonize about their four or five nines too much, in my opinion.
Folk are used to their phones crashing, needing rebooting and dropping
calls - we've cellphones to thank for that - and a half-decent VoIP
solution will knock the spots off your favourite mobile carrier for
reliability. Plus there's all the external factors - 99.999% uptime on
your PRI means that someone can only drive a digger through the cable
once every 273 years, assuming that it takes a day to fix, and no
telco's going to give you that in an SLA. And your power can't go off.
And so on.

Lastly, I'm afraid that virtualization and VoIP don't play well
together, at least not if you want to achieve a sensible density. The
large number of small packets being moved around the network interfaces
- both physical and virtual - will quickly chew up your CPU.

Cheers --

Dave

[mitul at enterux.com]

On PRI circuits failover with ha is possible using a red tone
appliance, but I m sure it's going to be almost impossible to transfer
the running calls during the event of fs box going down.

Thanks & Regards,
Mitul Limbani,
Founder & CEO,
Enterux Solutions Pvt. Ltd.,
The Enterprise Linux Company (r),
http://www.enterux.com
http://www.entVoice.com

On 29-Aug-2009, at 11:35 PM, David Knell <dave@3c.co.uk> wrote:

[rs at runsolutions.com]

Hmm, so basically 100 interested companys which each chip in 1000 bucks

sounds like lots of manpower, but on the other hand, I do not know the issues regarding the SIP protocoll, but, basically, isn't it *just* to tell another FS box to listen on port x for voicetraffic, forward it to ip on port y?


ok, i understand there's a lot going on under the hood, i guess it would mean to setup a call, but take care to not really set up the call, just the internal state ...


hmm, could it theoretically be done with the event system? ok, i guess I have to dive further into the internals to fully understand the scope.




But a live migration, where the box is available, be possible right now? Would be a step i would like to implement just to be able to do work on a hardware node if necesary without interrupting the service ...



--
Raimund Sacherer
-
RunSolutions
Open Source It Consulting
-

Parc Bit - Centro Empresarial Son Espanyol
Edificio Estel - Local 3D
07121 - Palma de Mallorca
Baleares






On Aug 29, 2009, at 6:01 PM, Anthony Minessale wrote: