VoIP Mailing List Archives

[mcollins at fcnetwork.com]

/me sends Anthony’s post to the printer to be laminated and framed… J


From: freeswitch-users-bounces@lists.freeswitch.org [mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony Minessale
Sent: Thursday, October 30, 2008 6:10 AM
To: freeswitch-users@lists.freeswitch.org
Subject: Re: [Freeswitch-users] mod_cdr revival (or new module maybe)


Actually,

The goal is to not limit the functionality by over thinking how things will be used but to provide building blocks to make much
more possible. Our analogy for this is that if you take the common lego reference, If you have something cool built out of legos but they are super glued together it limits what you *could* have done had then not been.

The system that David described is indeed ideal. I have mentioned it more than once and its no coincidence that FreeSWITCH plays into that model to a tee. I do not mention it very often because I think understanding that concept alone is valuable advice that I'd prefer not to waste on deaf ears so I recommend you heed his suggestion.

Logging direct to DB is not a sin or anything but it's just not very reliable. We have so many ways to process call records as it is.
you can use the cdr_csv if you want legacy support from asterisk that you can eventually tweak with templatable output, you can even make a template that generates the log as ready to roll sql statements you can just cat into mysql.

We have event_socket + channel_hangup event which has all the same info that the cdr_csv uses to generate the file.
We have the XML cdr that is a 3 dimensional view of the call with an account of nearly everything that happend including
each application executed, branching records when the call is transferred complete with a timestamp for each occurance.

--"However it would be great if freeswitch had the options for extra functionality (auto log rotation, db cdrs etc)
so that it meets the peculiarities of every different project."

A single cron entry that does kill -HUP `cat freeswitch.pid` and you have automatic rotation in unix.
In windows there are several cron equivs and instead of kill you can use an xml-rpc script to call the FSAPI call "fsctl send_sighup"
also you can schedule it to happen automaticly with the FSAPI

sched_api @300 mygroup fsctl send_sighup

A script connected to event socket subscribed to channel_hangup can act as your DB cdr engine and has the bonus that it could
connect to N boxes at once as well as be able to tell if the system was in trouble by listening for system heartbeat events and or socket disconnect.

Moral of the story, there is more already there than first appears. I've always hated the M$ paper clip so I strive not to bring him back.
"It looks like you are trying to run a VoIP termination company! Shall I assume how you want your call details?"



On Thu, Oct 30, 2008 at 7:43 AM, regs@kinetix.gr (regs@kinetix.gr) <regs@kinetix.gr (regs@kinetix.gr)> wrote:
I'll try some tests with various combinations first and then decide what's best.

I have to say that I was astonished to find out that freeswitch had so many event handlers from the very beginning.
However it would be great if freeswitch had the options for extra functionality (auto log rotation, db cdrs etc)
so that it meets the peculiarities of every different project. That would make a big difference compared to
other softswitch solutions where the lack of such features prohibits people from using them (especially in
the carrier grade level). User feedback (and wish lists) is the key for the success of an open source (or not)
project...

Thank you all for your replies. You' ve been very helpful.


David Knell wrote:
The sleep's done each time the directory's empty, not each time a file's written. File open and close
are trivial (it's probably still cached), and the contents are going to have to be parsed wherever you
process it.

We've used exactly this to process deliver CDRs on boxes handling in excess of 500K mins/day
without issue. And, looking at one now, the CDR processor's used about 4% of the CPU time
of FreeSWITCH, and about half that of the MySQL database which it writes the records to, also
on the local machine, from which they're simply copied to the main CDR processor.

It performance simply isn't worth worrying about.

--Dave

[regs at kinetix.gr]

Don't get me wrong. I am not trying to undermine the excellent work that
has already been done on the event handling
modules of FS.

I myself am a big fun of modular constructs in order to achieve
something complicated (e.g. the UNIX way).

But there are some situations that we will always need some extra
features in order to accomplish our goal.

As I wrote before, my current system is using the cdr-push (from the
gateway side) method to gather my cdrs.
Many billing systems and many terminating providers are satisfied with
that model alone. I am not. In the end of each hour
I am gathering all the cdrs and checking them one by one against my
database in order to verify that ALL of my cdrs were
handled by my radius servers.

In my future system (where a more batch-like mode is preferable) I am
"forced" to use a cdr-pull method. My billing system will
be responisble for "pulling" the cdrs from the gateways and then process
them. So if we have a look at the event handling weaponry of FS the
following modules cannot meet me needs :

mod_event_multicast
mod_event socket
mod_radius_cdr
mod_xmpp_event

because they all rely on a mechanism where the gateway is pushing my
cdrs to my billing system (and you need a checking mechanism to verify
that all the cdrs were handled).

So now, lets have a look at the alternatives :

mod_xml_cdr : this was my first choice as it had all the info needed
(even more). But the cdr per file approach proved out to be
inadequate in terms of performance. I wrote a few lines of code (in
perl) that did a listing of the directory, parsed the cdrs using
XML::Simple (without even doing anything meaningful, e.g. checks,
inserts to a database etc.) and I was not able to make my system
parse more than a 30 cdrs/sec (reference 1000 USD average system, not
drawing any conclusions, don't get me wrong).
Also while the parser was running I had a high cpu utilization (as
expected) that was competing with my FS service. Wouldn't that make
a bad impact of my freeswitch performance?

so I went to a different module

mod_cdr_csv : the text .csv format with the plethora of attributes was
perfect for my application but it would be a daunting task
for me to make a script tha purges the Master csv while it was being
used by FS. I googled the issue and found about the HUP method
and the rotate directive in the config. So far so good. But it lacked
some functionality.

All I was trying to state in my previous messages was that it would be
nice if the rotation was being initiated by FS, and maybe have different
behavior depending on another directive or something. Let me explain
what I mean.

There are some people who don't care about what cdrs exist in a given
rotated log file, as long as a) no cdrs get lost in the process of rotation
, b) no duplicates exist. I can see that the HUP method satisfied this need.

There is another group of people that DO care about what cdrs are in a
rotated log file. E.g. : The file with the name
Master.csv.2008-01-01-09-00-00
would only contain cdrs that were terminated from "2008-01-01 09:00:00"
to "2008-01-01 09:59:59" if an hourly setup was desirable or from
"2008-01-01 09:00:00" to "2008-01-01 09:04:59" if a five-minutes setup
was desirable. That need is not covered by the current HUP method
because some cdrs might "escape" from the next to the previous file due
to the fact that there is a delay between cron executing the HUP and FS
doing
the file log rotation.

In simple words : I am getting my job done with FS they way it already
is (no question about that). BUT, I (or someone with different needs)
could do
his/her job better if some minor features were present.

I am not saying that you should should embrace modules that meet the
average needs yet sacrifice flexibility. I am merely suggesting
extending the flexibility
of the already existing ones. Put some more lego tiles in your box set


Michael Collins wrote:

[anthony.minessale at g...]

Don't worry, I don't think you are undermining anything. It's just a discussion thread.
If Yossi gets mod_cdr back up to speed we will happily accept it.

BTW, *hint*
You use the perl script to turn the current batch of xml files into a single file or array of sql stmts.
then you pass that on to a single insert / update transaction.

Your realtime insert into the db idea will suffer the same slow effect of
single transaction per insert which is your bottleneck. But it will scale
rather far before it becomes a problem. The bigger issue is the chicken or the
egg on what to do when the db is down. File system is the safest place to put the data
cos if it goes, so does you box and there is little chance of dispare apart from filling it up and
you have the ability to reprocess the files again if the db dies. Single file per record solves
file locking problems and the xml data is so big anyway you would want to seggregate it.

Like I said I don't go on giving out hints all the time so that's the last from me for now.



On Thu, Oct 30, 2008 at 3:27 PM, regs@kinetix.gr (regs@kinetix.gr) <regs@kinetix.gr (regs@kinetix.gr)> wrote: