VoIP Mailing List Archives

[johnd at defyne.org]

Hi,

I am slowly gaining confidence using FreeSWITCH in production, but there
is one issue that I'm still wondering about: how are people upgrading
their FreeSWITCH installation binaries without dropping all current calls?

So far I have been upgrading in the dead of night, after pausing for 5
minutes then dropping the stragglers, but this is hardly ideal.

What I would like to do is to run an upgraded instance of FreeSWITCH on
the same machine, and have it handle all new call packets, whereas the old
instance continues to handle the existing call packets, until there are no
more old calls left.

I can think of about seven ways to accomplish this, but before I dive into
the code I thought I'd better ask what everyone else has been doing

(The only standard way I can think of doing this is to have a SIP proxy
sitting in front of FS the whole time, just to handle these upgrade
windows. It seems like a bit of a waste.)

So how are you handling your FS software upgrades?

{P^/
John

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

[mgg at giagnocavo.net]

How are you handling your FS box crashing?

-----Original Message-----
From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of John Dalgliesh
Sent: Wednesday, June 10, 2009 9:04 PM
To: freeswitch-users@lists.freeswitch.org
Subject: [Freeswitch-users] Live Upgrade Techniques


Hi,

I am slowly gaining confidence using FreeSWITCH in production, but there
is one issue that I'm still wondering about: how are people upgrading
their FreeSWITCH installation binaries without dropping all current calls?

So far I have been upgrading in the dead of night, after pausing for 5
minutes then dropping the stragglers, but this is hardly ideal.

What I would like to do is to run an upgraded instance of FreeSWITCH on
the same machine, and have it handle all new call packets, whereas the old
instance continues to handle the existing call packets, until there are no
more old calls left.

I can think of about seven ways to accomplish this, but before I dive into
the code I thought I'd better ask what everyone else has been doing

(The only standard way I can think of doing this is to have a SIP proxy
sitting in front of FS the whole time, just to handle these upgrade
windows. It seems like a bit of a waste.)

So how are you handling your FS software upgrades?

{P^/
John

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

[mrene_lists at avgs.ca]

By reporting it on Jira so it doesn't crash anymore


On 11-Jun-09, at 12:04 AM, Michael Giagnocavo wrote:

[michal.bielicki at hal...]

Am 11.06.2009 um 05:04 schrieb John Dalgliesh:

[anthony.minessale at g...]

or you can put a sip proxy in front of 2 boxes where you can control the flow of traffic.
when you want to upgrade one, take all the traffic off of it by forcing all calls to the other box, upgrade it then shift the traffic to the new one.
if that goes well, upgrade the other one too.



On Thu, Jun 11, 2009 at 5:47 AM, Michal Bielicki <michal.bielicki@halo2.pl> wrote:

[mgg at giagnocavo.net]

Exactly. You probably want to have something like this anyways, so that when someone accidentally unplugs the system, or the disks/CPU/RAM crash, you’re not stuck.

That is, until FreeSWITCH can record its internal state to some inter-machine memory so we can have hot failover.

-Michael

From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale
Sent: Thursday, June 11, 2009 10:55 AM
To: freeswitch-users@lists.freeswitch.org
Subject: Re: [Freeswitch-users] Live Upgrade Techniques


or you can put a sip proxy in front of 2 boxes where you can control the flow of traffic.
when you want to upgrade one, take all the traffic off of it by forcing all calls to the other box, upgrade it then shift the traffic to the new one.
if that goes well, upgrade the other one too.



On Thu, Jun 11, 2009 at 5:47 AM, Michal Bielicki <michal.bielicki@halo2.pl> wrote:

Am 11.06.2009 um 05:04 schrieb John Dalgliesh:

[msc at freeswitch.org]

On Thu, Jun 11, 2009 at 10:33 AM, Michael Giagnocavo <mgg@giagnocavo.net (mgg@giagnocavo.net)> wrote:

[johnd at defyne.org]

OK thanks that is what I thought the general way of doing it would be. But
it seems a bit wasteful to have that SIP proxy there the whole time
especially when I am using FS in the role of an SBC.

The problem with the graceful restart of course is that you have to wait
for the calls count to get to zero, which may never happen. It's 3:30am
here in Sydney now and I just checked FS: 20 calls in progress still!

So what I plan to do is add a '--upgrade' cmd line arg to FS. This will
make the new instance contact the old one on a unix socket and receive a
dup of its SIP socket fd(s) via a SCM_RIGHTS sendmsg. It will use those
for sending and the unix socket for receiving. Meanwhile the old instance
will pass any packets with unknown Call-Ids over the unix socket to the
new instance, instead of handling them itself. When the old instance has
no calls left, it shuts down. The new instance detects the unix socket is
closed and switches to reading from the SIP socket (which would have
buffered any unread packets - so nothing is lost).

Sound good? I realise this will be 90% in libsofia but I've read teh code
and it seems very do-able. Anyone interested in my changes will of course
be most welcome to them.

The runner-up approach I considered was to make a kernel module that
extends iptables with a filter that can extract the Call-Id and look it up
in a table that is somehow populated from FS. Maybe this exists already?
Kind of a SIP proxy lite that can be enabled on the server machine when
needed. Anyway that lost out as it's more work and even less portable.

{P^/
John

On Thu, 11 Jun 2009 at 11:54 -0500, Anthony Minessale wrote:

[johnd at defyne.org]

I assume he's talking about hardware failures here

But to answer the question: crashes are easy to deal with. With a crash
you have lost the calls that are in progress anyway; you don't have to
manage a gradual transition.

Currently, since FS is quite quick to start up, I am just relaunching it
immediately.

But when I have a second box up and running what I'll do is just add the
IP of the dead machine as another IP of the second box, and then it will
take all the old machine's traffic. That is the plan anyway. I've seen
some commercial boxes that use a similar trick.

(I've only seen one crash that wasn't my fault. Something to do with
terminating a bridge: when the first leg gets a hangup it hangs up the
other leg on its own thread... which can cause problems if the other leg
was doing something funky at the time. Leads to a heap corruption. Doesn't
happen with MALLOC_CHECK_ set so I'm just leaving it set for now

{P^/

On Thu, 11 Jun 2009 at 00:41 -0400, Mathieu Rene wrote:

[johnd at defyne.org]

On Thu, 11 Jun 2009 at 10:42 -0700, Michael Collins wrote:

[msc at freeswitch.org]

On Thu, Jun 11, 2009 at 11:24 AM, John Dalgliesh <johnd@defyne.org (johnd@defyne.org)> wrote:

[mike at jerris.com]

On Jun 11, 2009, at 2:24 PM, John Dalgliesh wrote:

[kristian.kielhofner at...]

That's exactly what I do.

Between dispatcher and FLAGS/GFLAGS this is easy to do in OpenSIPS/SER.

On Thu, Jun 11, 2009 at 12:54 PM, Anthony
Minessale<anthony.minessale@gmail.com> wrote:

[mgg at giagnocavo.net]

Well, if you're running multiple machines, waiting for it to drainstop isn't that big of a deal unless you're in some sort of hurry, right? Give it an hour or so to drainstop, then kill 'em.

Would it not be simpler to try to do something with re-invites or REFER, assuming your endpoints support it?

-Michael

-----Original Message-----
From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of John Dalgliesh
Sent: Thursday, June 11, 2009 12:14 PM
To: freeswitch-users@lists.freeswitch.org
Subject: Re: [Freeswitch-users] Live Upgrade Techniques


I assume he's talking about hardware failures here

But to answer the question: crashes are easy to deal with. With a crash
you have lost the calls that are in progress anyway; you don't have to
manage a gradual transition.

Currently, since FS is quite quick to start up, I am just relaunching it
immediately.

But when I have a second box up and running what I'll do is just add the
IP of the dead machine as another IP of the second box, and then it will
take all the old machine's traffic. That is the plan anyway. I've seen
some commercial boxes that use a similar trick.

(I've only seen one crash that wasn't my fault. Something to do with
terminating a bridge: when the first leg gets a hangup it hangs up the
other leg on its own thread... which can cause problems if the other leg
was doing something funky at the time. Leads to a heap corruption. Doesn't
happen with MALLOC_CHECK_ set so I'm just leaving it set for now

{P^/

On Thu, 11 Jun 2009 at 00:41 -0400, Mathieu Rene wrote:

[johnd at defyne.org]

Hi,

On Thu, 11 Jun 2009 at 16:33 -0400, Michael Giagnocavo wrote: