VoIP Mailing List Archives

[krice at freeswitch.org]

It is not possible to do a live migration at all right now... There is no way to move a call and have all the states required for that call to magically re-appear on a different instance. This will require a fair bit of work to get there. It is possible to configure fs via some back end DB magic to share configurations ie: n+1 or “warm standby” style fault tolerance but you are going to loose the calls that are up when the failure occurs. From: Raimund Sacherer <rs@runsolutions.com> Reply-To: <freeswitch-users@lists.freeswitch.org> Date: Sat, 29 Aug 2009 20:33:21 +0200 To: <freeswitch-users@lists.freeswitch.org> Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing Hmm, so basically 100 interested companys which each chip in 1000 bucks sounds like lots of manpower, but on the other hand, I do not know the issues regarding the SIP protocoll, but, basically, isn't it *just* to tell another FS box to listen on port x for voicetraffic, forward it to ip on port y? ok, i understand there's a lot going on under the hood, i guess it would mean to setup a call, but take care to not really set up the call, just the internal state ... hmm, could it theoretically be done with the event system? ok, i guess I have to dive further into the internals to fully understand the scope. But a live migration, where the box is available, be possible right now? Would be a step i would like to implement just to be able to do work on a hardware node if necesary without interrupting the service ... -- Raimund Sacherer - RunSolutions Open Source It Consulting - Parc Bit - Centro Empresarial Son Espanyol Edificio Estel - Local 3D 07121 - Palma de Mallorca Baleares On Aug 29, 2009, at 6:01 PM, Anthony Minessale wrote: We have previously estimated the development of live fail over (after a box dies where live migration is no longer possible) to exceed 100k in development costs. It requires several additions to the sofia sip library, freeswitch and a dependancy on some other code we would have to implement to manage it. It may or may not be worth it to raise that kind of funding just to avoid an occasional disaster. Then there is a matter of securing the time of the developers necessary to carry out the implementation. On Aug 29, 2009 10:19 AM, "Brian West" <brian@freeswitch.org> wrote: I was able to do this using OpenVZ, You can get away with it on smaller instances... like if you're doing one instance per company but don't expect live migration to work as well on large instances with thousands of calls up at once. You need a fast network, fast disks and to follow the howto on the wiki. /b On Aug 29, 2009, at 4:58 AM, Steve Kurzeja wrote: > You still have hardware failures and fail-over... _______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists... _______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org _______________________________________________ FreeSWITCH-users mailing list FreeSWITCH-users@lists.freeswitch.org http://lists.freeswitch.org/mailman/listinfo/freeswitch-users UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users http://www.freeswitch.org

[mgg at giagnocavo.net]

Also please remember this means every _application_ also must be “failover” aware. So, if you’re using a little calling card or billing app to control your routing, that app will have to explicitly enroll its state as well as restore itself.

Live migration is possible via OpenVZ. OVZ does not have the same overhead of other virtualization techniques such as Hyper-V, ESX, Xen, etc. So you can do media without much worry. OVZ can work with other virtualization, too; you can spin it up inside of a VMware machine.

But this is *live* migration of the container. It’s “just” pausing your container, copying the whole memory, and resuming it on another node. This is not “dead” migration or failover after calls break. However, you could try running a very frequent sync in OVZ, so that if there was a node failure, you could resume from a previously saved state. But, I highly doubt performance would be good enough for this to work successfully on any serious deployment.

-Michael

From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Raimund Sacherer
Sent: Saturday, August 29, 2009 12:33 PM
To: freeswitch-users@lists.freeswitch.org
Subject: Re: [Freeswitch-users] FreeSWITCH HA + Loadbalancing



Hmm, so basically 100 interested companys which each chip in 1000 bucks


sounds like lots of manpower, but on the other hand, I do not know the issues regarding the SIP protocoll, but, basically, isn't it *just* to tell another FS box to listen on port x for voicetraffic, forward it to ip on port y?



ok, i understand there's a lot going on under the hood, i guess it would mean to setup a call, but take care to not really set up the call, just the internal state ...



hmm, could it theoretically be done with the event system? ok, i guess I have to dive further into the internals to fully understand the scope.





But a live migration, where the box is available, be possible right now? Would be a step i would like to implement just to be able to do work on a hardware node if necesary without interrupting the service ...




--
Raimund Sacherer

-
RunSolutions
Open Source It Consulting
-

Parc Bit - Centro Empresarial Son Espanyol
Edificio Estel - Local 3D
07121 - Palma de Mallorca
Baleares







On Aug 29, 2009, at 6:01 PM, Anthony Minessale wrote:




We have previously estimated the development of live fail over (after a box dies where live migration is no longer possible) to exceed 100k in development costs.
It requires several additions to the sofia sip library, freeswitch and a dependancy on some other code we would have to implement to manage it.
It may or may not be worth it to raise that kind of funding just to avoid an occasional disaster.
Then there is a matter of securing the time of the developers necessary to carry out the implementation.

[mitul at enterux.com]

Is itnpossible to have a db cluster know the state of each and every call and then use Heartbeat on this db +
fs cluster so that clients see only one ip where as internally all fs boxes refer db for call states, db again is under replication.


This in the thioery can be written, but I am sure if we think bit more on this direction the problem seem to be getting addressed.


Other guys also chip in their 2 cents, we just need 50 of em to make a full dollar.

Thanks & Regards,Mitul Limbani,
Founder & CEO,
Enterux Solutions Pvt. Ltd.,
The Enterprise Linux Company (r),
http://www.enterux.com
http://www.entVoice.com


On 30-Aug-2009, at 12:14 AM, Ken Rice <krice@freeswitch.org (krice@freeswitch.org)> wrote:

[brian at freeswitch.org]

The sip stack needs to be modified to spin that data up into the state machine so that it can take over calls once the fail over takes place... its not an easy task.

/b

On Aug 29, 2009, at 2:56 PM, Mitul Limbani wrote:

[pete at privateconnect...]

I am also in the process of playing around with FS running inside Xen Virtual Machines with "mirrored" VMs on a second failover system. So far, the initial tests are promising. I can see a 2-3 sec "hiccup" in the net traffic during the live migration of the Xen VM. My calls do not drop, but I will stress that I'm only running test cases at this time, I am not using real world traffic.

Once I figure it all out, I'll report it here.

-pete

[pete at privateconnect...]

I guess I should also mention that Xen is a side-project.

When considering this issue for an existing production systems, we chose to put as much HA into hardware as we can. We are not concerned with FS crashing, as so far we've never seen that happen (except when our module caused it So for each of our systems:
- We have dual NIC cards (onboad NIC + PCI card) both bridged together in case one fails
- We have redundant power supplies.
- We use Mirrored Solid State Disks for local storage (far better MTBF than HDD, a lot faster too)
- All but OS and speed-critical data is stored on a NAS device
- We have redundant DBs with Memcache infront for speed

At the same time we chose to use COTS hardware (SuperMicro chassis/MoBo) rather than the big-boys like IBM or Dell. This kept the overall cost per machine low. Initially some were concerned that not having a name like IBM on our servers would be concerning to some potential clients. The solution was to pay a company to deisgn and build a custom face plate for the SuperMicro boxes. Which oddly looks more impressive to clients that a rack full of IBM faceplates. It was suprisingly low cost for the faceplates too.

For scalability, OpenSIPS was our choice. There's a very nice tutorial on their website on how to configure Load Balancing.

-pete

[steveu at coppice.org]

This sounds like so many "redundancy" projects that will probably offer
nothing in the real world.

On 08/30/2009 05:52 AM, Pete Mueller wrote:

[dave at 3c.co.uk]

On Sun, 2009-08-30 at 11:17 +0800, Steve Underwood wrote:

[pete at privateconnect...]

There is a need for ensuring that calls do not drop, but we must balance that with the cost of making the system redundant. We took some small, inexpensive measures, to improve our odds, but we could spend a lot more, for basically nothing more than giving some client a warm fuzzy.

To expand on what's mentioned below, The biggest cause for downtime that we've experienced is human accidents. We only have our solution in Tier 1 co-location facilities, so power/net dying isn't really an issue. (If the power does go out, 1000s of systems are down, and everyone notices). What we do end up with is IT admins tripping over power cords, pulling the wrong Ethernet cable, blowing a fuse on one side of the rack, etc. So we've doubled-up on all our cables.

After that, the next biggest cause has been MoBo/CPU failure due to fan failure. This issue doesn't really have a good solution, and is why we began looking at Xen. This is where SUN systems look attractive, as systems like the E10000 can shut down one CPU or board and keep the rest of the system running. But the cost for that solution is high, and I think that's SPARC-only. I'd love to head others take on a solution for this, as Xen is really a lot of overhead for a rare problem. Though it is at least technically interesting for me

As for storage, this was completely personal experience. Our SSD have had no issues, while SATA drives seem to fail about 1 every month. The NAS storage is connected via dual NICs as well (again for the cabling) and is completely separated from the network, very close to DAS, just using GigE as the "cable". We are always looking for ways to improve, but the newest and greatest from EMC and others just doesn't seem to offer anything significant and cost a LOT more.

I like the idea about the complete custom chassis. I hadn't considered that due to my thinking it would be expensive. Sounds like it's worth a look. As we consider creating an appliance offering, this may become more important.

-pete

[jason at jasonjgw.net]

Pete Mueller <pete@privateconnect.com> wrote:

[rs at runsolutions.com]

So, the first way would be to have a look on the sip stack, which is, in fact, sofia ..., well, sound's like a nice, fun, long-going hobby project to me

I will definitly at least look at the sip code to check if it is something i would myself willingly give over to ...


But I *really* do want a setup where it does not matter if one specific FS instance is UP or NOT ...



--
Raimund Sacherer
-
RunSolutions
Open Source It Consulting
-

Parc Bit - Centro Empresarial Son Espanyol
Edificio Estel - Local 3D
07121 - Palma de Mallorca
Baleares






On Aug 29, 2009, at 10:04 PM, Brian West wrote:

[jim at evolutiontel.net]

High Availability is always an interesting problem. PSTN equipment
generally overcomes it through duplication of messaging and hardware.
Obviously at an E1 (T1 for the US folks) or Subscribers Line Level
this is not possible. This duplication comes at a cost and lowers the
calls per minute a softswitch can handle for the amount of hardware
purchased. Like everything it is a trade-off between cost and
reliability.

Per Dave's comment, duplication on a network level can be achieved
easily through duplicating network switches and using OSPF, VRRP or
network bonding to ensure continuity if a path is compromised. From
experience this is exactly what the vendors do and it does work.
During failover the calls experience a minut breakup of voice quality
but they still hold up ok.

Conceptually in a pure SIP installation of Freeswitch, hot standby or
active standby could be achieved through duplication of messages
(INVITE, 200OK and BYE for calls) between 2 boxes and then using VRRP
to change the IP address when the box falls over. Going this way
allows call state to be as up to date as possible. Obviously this
would require logic added to sofia to transfer the RTP port
information and UUID information between the FS instances (Easily
achieved using additional SIP headers). It would also require that
sofia does not forward duplicated messages to gateways or user agents.
CDR information would also need to be generated correctly (My
experience of Radius with Opensips is that it generates start and stop
messages on INVITE/200OK then BYE respectively so this could be an
easy solution to the CDR issue).

Regards,



On Sun, Aug 30, 2009 at 11:19 PM, Raimund Sacherer<rs@runsolutions.com> wrote:

[krice at freeswitch.org]

Your forgetting that with RTP theres the duplication of RTP that may be
required along with a mechanism that keeps media timers working properly...

Where many things don't support RTP timers they sure come in handy on a
regular basis especially dealing with several of the other software based
platforms out there that just love to have calls end but never want to send
a BYE

[jim at evolutiontel.net]

Hi Ken,

I don't disagree, the scenario you mentioned is just the tip of the
iceberg when it comes to getting something like this working smoothly.
However to try and cover them all in a single post (assuming that I
could) would have resulted in something akin to War and Peace and
probably bored the pants off of most readers For any HA solution
to move from a concept into reality, it would require some serious
brainstorming to identify all scenarios and how to handle them.

Given that many posts in this thread mentioned transfer of call state
at time of failure or some form of virtualisation, I wanted to provide
another solution that would provide real time call state updates
between FS boxes utilising mechanisms that are largely already
in-place and thus (hopefully) reducing software development time if
someone was willing to cough up the $$$$$.

Regards,

On Mon, Aug 31, 2009 at 1:23 PM, Ken Rice<krice@freeswitch.org> wrote:

[krice at freeswitch.org]

I was thinking more like something the size of war and peace but written
with the style of an RFC (and about as interesting and an RFC)

I know there has already been some discussion on several fronts of atleast
getting the core and several other pieces to where they need to be for
stateful failover and I'm not sure if its been mentioned here, but sofia is
going to be a bit of work and estimates run in the 100K USD range. Now if we
could get a get a couple of corporate sponsors to help here it would be
great.

K