Sponsor: VoiceMeUp - Corporate & Wholesale VoIP Services

VoIP Mailing List Archives
Mailing list archives for the VoIP community
 SearchSearch 

[Freeswitch-users] mod_cdr revival (or new module maybe)

Goto page 1, 2  Next
 
Post new topic   Reply to topic    VoIP Mailing List Archives Forum Index -> freeSWITCH Users
View previous topic :: View next topic  
Author Message
regs at kinetix.gr
Guest





PostPosted: Wed Oct 29, 2008 10:39 am    Post subject: [Freeswitch-users] mod_cdr revival (or new module maybe) Reply with quote

Hi,

I saw in the wiki that the mod_cdr module is now unsupported. There
is also a note
about a revival of the module. I would like to ask the following :

What is the current state of the revival process? (should we expect
something in the near future?)

Will it have the same functionality as before (DB support for instance)?

Are there any plans for a brand new database specific event handler module?

It would be great if there was one so that developers (especially those
who develop
billing applications) would not have to create their own hacks (cron
scripts etc.)

Thank you for your time,

--
-------------------------------------------
Apostolos Pantsiopoulos
Kinetix Tele.com Support Center

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
Back to top
mike at jerris.com
Guest





PostPosted: Wed Oct 29, 2008 10:48 am    Post subject: [Freeswitch-users] mod_cdr revival (or new module maybe) Reply with quote

Unsure at this time. There has been some work on mod_cdr_odbc. We
generally advise against direct to db cdr methods without a very
robust backup method for when the db is down.

On Oct 29, 2008, at 9:57 AM, "regs@kinetix.gr" <regs@kinetix.gr> wrote:

Quote:
Hi,

I saw in the wiki that the mod_cdr module is now unsupported. There
is also a note
about a revival of the module. I would like to ask the following :

What is the current state of the revival process? (should we expect
something in the near future?)

Will it have the same functionality as before (DB support for
instance)?

Are there any plans for a brand new database specific event handler
module?

It would be great if there was one so that developers (especially
those
who develop
billing applications) would not have to create their own hacks (cron
scripts etc.)

Thank you for your time,

--
-------------------------------------------
Apostolos Pantsiopoulos
Kinetix Tele.com Support Center

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
Back to top
regs at kinetix.gr
Guest





PostPosted: Wed Oct 29, 2008 4:02 pm    Post subject: [Freeswitch-users] mod_cdr revival (or new module maybe) Reply with quote

Yes, I agree. But one could use the two methods combined (csv or xml +
db) for redundancy.

Is there any consideration regarding automatic log rotation (e.g.
hourly, or user specified)
without the need of a HUP? Now, that could make things a lot easier for
the development of
an external csv to db aggregation script because the script would read
from a closed (not used by freeswitch
at the time) CDRs file. And the developer could be sure that the cdrs
contained in that file would
have a hangup timestamp that could be described by the filename (e.g.
20080101_010000.csv).

Michael Jerris wrote:
Quote:
Unsure at this time. There has been some work on mod_cdr_odbc. We
generally advise against direct to db cdr methods without a very
robust backup method for when the db is down.

On Oct 29, 2008, at 9:57 AM, "regs@kinetix.gr" <regs@kinetix.gr> wrote:


Quote:
Hi,

I saw in the wiki that the mod_cdr module is now unsupported. There
is also a note
about a revival of the module. I would like to ask the following :

What is the current state of the revival process? (should we expect
something in the near future?)

Will it have the same functionality as before (DB support for
instance)?

Are there any plans for a brand new database specific event handler
module?

It would be great if there was one so that developers (especially
those
who develop
billing applications) would not have to create their own hacks (cron
scripts etc.)

Thank you for your time,

--
-------------------------------------------
Apostolos Pantsiopoulos
Kinetix Tele.com Support Center

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org


_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org



_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
Back to top
regs at kinetix.gr
Guest





PostPosted: Wed Oct 29, 2008 4:02 pm    Post subject: [Freeswitch-users] mod_cdr revival (or new module maybe) Reply with quote

Yes, I agree. But one could use the two methods combined (csv or xml +
db) for redundancy.

Is there any consideration regarding automatic log rotation (e.g.
hourly, or user specified)
without the need of a HUP? Now, that could make things a lot easier for
the development of
an external csv to db aggregation script because the script would read
from a closed (not used by freeswitch
at the time) CDRs file. And the developer could be sure that the cdrs
contained in that file would
have a hangup timestamp that could be described by the filename (e.g.
20080101_010000.csv).

Michael Jerris wrote:
Quote:
Unsure at this time. There has been some work on mod_cdr_odbc. We
generally advise against direct to db cdr methods without a very
robust backup method for when the db is down.

On Oct 29, 2008, at 9:57 AM, "regs@kinetix.gr" <regs@kinetix.gr> wrote:


Quote:
Hi,

I saw in the wiki that the mod_cdr module is now unsupported. There
is also a note
about a revival of the module. I would like to ask the following :

What is the current state of the revival process? (should we expect
something in the near future?)

Will it have the same functionality as before (DB support for
instance)?

Are there any plans for a brand new database specific event handler
module?

It would be great if there was one so that developers (especially
those
who develop
billing applications) would not have to create their own hacks (cron
scripts etc.)

Thank you for your time,

--
-------------------------------------------
Apostolos Pantsiopoulos
Kinetix Tele.com Support Center

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org


_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org



_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
Back to top
mcollins at fcnetwork.com
Guest





PostPosted: Wed Oct 29, 2008 4:08 pm    Post subject: [Freeswitch-users] mod_cdr revival (or new module maybe) Reply with quote

Quote:
Yes, I agree. But one could use the two methods combined (csv or xml +
db) for redundancy.

Is there any consideration regarding automatic log rotation (e.g.
hourly, or user specified)
without the need of a HUP? Now, that could make things a lot easier
for
Quote:
the development of
an external csv to db aggregation script because the script would read
from a closed (not used by freeswitch
at the time) CDRs file. And the developer could be sure that the cdrs
contained in that file would
have a hangup timestamp that could be described by the filename (e.g.
20080101_010000.csv).

For the record, I've been dumping all my XML CDRs into a particular
directory and letting a script pick them up and process them. I think
this is the best of both worlds: you get individual files with tons of
info on each call and you can have a process that picks up those files
and inserts them into the db. If the db is down then the CDRs aren't
lost - they just accumulate in the directory until you get the db/script
thing working again.

Just my $.02

-MC

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
Back to top
regs at kinetix.gr
Guest





PostPosted: Wed Oct 29, 2008 4:25 pm    Post subject: [Freeswitch-users] mod_cdr revival (or new module maybe) Reply with quote

Yes, the xml files give you tons of info... but isn't it a little
insufficient - performance wise -
to open and close so many files in such a little time. In a PBX
environment that wouldn't be an
issue but if we get to the small-voip-carrier level (some thousand cdrs
per hour)
that could slow things down considerably, wouldn't it?

Thanks again for your prompt replies,

Michael Collins wrote:
Quote:
Quote:
Yes, I agree. But one could use the two methods combined (csv or xml +
db) for redundancy.

Is there any consideration regarding automatic log rotation (e.g.
hourly, or user specified)
without the need of a HUP? Now, that could make things a lot easier

for

Quote:
the development of
an external csv to db aggregation script because the script would read
from a closed (not used by freeswitch
at the time) CDRs file. And the developer could be sure that the cdrs
contained in that file would
have a hangup timestamp that could be described by the filename (e.g.
20080101_010000.csv).


For the record, I've been dumping all my XML CDRs into a particular
directory and letting a script pick them up and process them. I think
this is the best of both worlds: you get individual files with tons of
info on each call and you can have a process that picks up those files
and inserts them into the db. If the db is down then the CDRs aren't
lost - they just accumulate in the directory until you get the db/script
thing working again.

Just my $.02

-MC

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org



_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
Back to top
shawnl at waterwheelne...
Guest





PostPosted: Wed Oct 29, 2008 4:26 pm    Post subject: [Freeswitch-users] mod_cdr revival (or new module maybe) Reply with quote

In regards to auto log rotation - YES YES

ANTHM just completed that item for me, where by you can set the time in
minutes i believe it was.

I have not tested it yet, hope to this week.

Shawn


Michael Collins wrote:
Quote:
Quote:
Yes, I agree. But one could use the two methods combined (csv or xml +
db) for redundancy.

Is there any consideration regarding automatic log rotation (e.g.
hourly, or user specified)
without the need of a HUP? Now, that could make things a lot easier

for

Quote:
the development of
an external csv to db aggregation script because the script would read
from a closed (not used by freeswitch
at the time) CDRs file. And the developer could be sure that the cdrs
contained in that file would
have a hangup timestamp that could be described by the filename (e.g.
20080101_010000.csv).


For the record, I've been dumping all my XML CDRs into a particular
directory and letting a script pick them up and process them. I think
this is the best of both worlds: you get individual files with tons of
info on each call and you can have a process that picks up those files
and inserts them into the db. If the db is down then the CDRs aren't
lost - they just accumulate in the directory until you get the db/script
thing working again.

Just my $.02

-MC

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org



_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
Back to top
regs at kinetix.gr
Guest





PostPosted: Wed Oct 29, 2008 4:41 pm    Post subject: [Freeswitch-users] mod_cdr revival (or new module maybe) Reply with quote

That's very good news. Smile

Shawn Lewis wrote:
Quote:
Quote:
In regards to auto log rotation - YES YES

ANTHM just completed that item for me, where by you can set the time in
minutes i believe it was.

I have not tested it yet, hope to this week.

Shawn


Michael Collins wrote:
Quote:
Quote:
Yes, I agree. But one could use the two methods combined (csv or xml +
db) for redundancy.

Is there any consideration regarding automatic log rotation (e.g.
hourly, or user specified)
without the need of a HUP? Now, that could make things a lot easier

for

Quote:
the development of
an external csv to db aggregation script because the script would read
from a closed (not used by freeswitch
at the time) CDRs file. And the developer could be sure that the cdrs
contained in that file would
have a hangup timestamp that could be described by the filename (e.g.
20080101_010000.csv).

For the record, I've been dumping all my XML CDRs into a particular
directory and letting a script pick them up and process them. I think
this is the best of both worlds: you get individual files with tons of
info on each call and you can have a process that picks up those files
and inserts them into the db. If the db is down then the CDRs aren't
lost - they just accumulate in the directory until you get the db/script
thing working again.

Just my $.02

-MC

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org (Freeswitch-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org



_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org (Freeswitch-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
Back to top
mcollins at fcnetwork.com
Guest





PostPosted: Wed Oct 29, 2008 5:30 pm    Post subject: [Freeswitch-users] mod_cdr revival (or new module maybe) Reply with quote

Quote:
Yes, the xml files give you tons of info... but isn't it a little
insufficient - performance wise -
to open and close so many files in such a little time. In a PBX
environment that wouldn't be an
issue but if we get to the small-voip-carrier level (some thousand
cdrs
Quote:
per hour)
that could slow things down considerably, wouldn't it?

Thanks again for your prompt replies,


At that level of activity then I would assume you'd want a more robust
solution which obviously would involve a server handling the CDRs
separately. That's where XML is a real winner: it can POST CDRs to a web
server and the webserver can handle all the pre-processing and db fun
stuff. And if the connection to the webserver failed, the CDRs would be
put on disk so that they aren't lost forever. Also, the webserver could
cache the CDRs to its disk (or whatever storage) if the db itself went
down but the webserver stayed up.

Just a thought, anyway. It may be extra layers but it's also extra
control.

-MC


Quote:
Michael Collins wrote:
Quote:
Quote:
Yes, I agree. But one could use the two methods combined (csv or
xml +
Quote:
Quote:
Quote:
db) for redundancy.

Is there any consideration regarding automatic log rotation (e.g.
hourly, or user specified)
without the need of a HUP? Now, that could make things a lot easier

for

Quote:
the development of
an external csv to db aggregation script because the script would
read
Quote:
Quote:
Quote:
from a closed (not used by freeswitch
at the time) CDRs file. And the developer could be sure that the
cdrs
Quote:
Quote:
Quote:
contained in that file would
have a hangup timestamp that could be described by the filename
(e.g.
Quote:
Quote:
Quote:
20080101_010000.csv).


For the record, I've been dumping all my XML CDRs into a particular
directory and letting a script pick them up and process them. I
think
Quote:
Quote:
this is the best of both worlds: you get individual files with tons
of
Quote:
Quote:
info on each call and you can have a process that picks up those
files
Quote:
Quote:
and inserts them into the db. If the db is down then the CDRs aren't
lost - they just accumulate in the directory until you get the
db/script
Quote:
Quote:
thing working again.

Just my $.02

-MC

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users

UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
Quote:
Quote:
http://www.freeswitch.org



_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users

UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
Quote:
http://www.freeswitch.org

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
Back to top
regs at kinetix.gr
Guest





PostPosted: Wed Oct 29, 2008 5:56 pm    Post subject: [Freeswitch-users] mod_cdr revival (or new module maybe) Reply with quote

Good point. I have got this kind of behavior (cdrs push model) in my current system (using radius servers).
The only drawback of this method is that if you want to be absolutely sure that all the cdrs were handled by
the web server (or radius server) you have to check at certain intervals every cdr one by one (and handle those left
unhandled for various reasons (network, excessive web server load etc.))

But for my next project I am somewhat forced to use a "cdrs-pull" method where a process will "pull" cdrs
from the server at its own pace making this extra check unnecessary.

I will wait for an automatic log rotation as Shawn Lewis wrote. I think that will do the job.

Michael Collins wrote:
Quote:
Quote:
Quote:
Yes, the xml files give you tons of info... but isn't it a little
insufficient - performance wise -
to open and close so many files in such a little time. In a PBX
environment that wouldn't be an
issue but if we get to the small-voip-carrier level (some thousand
cdrs
Quote:
per hour)
that could slow things down considerably, wouldn't it?

Thanks again for your prompt replies,


At that level of activity then I would assume you'd want a more robust
solution which obviously would involve a server handling the CDRs
separately. That's where XML is a real winner: it can POST CDRs to a web
server and the webserver can handle all the pre-processing and db fun
stuff. And if the connection to the webserver failed, the CDRs would be
put on disk so that they aren't lost forever. Also, the webserver could
cache the CDRs to its disk (or whatever storage) if the db itself went
down but the webserver stayed up.

Just a thought, anyway. It may be extra layers but it's also extra
control.

-MC


Quote:
Michael Collins wrote:
Quote:
Quote:
Yes, I agree. But one could use the two methods combined (csv or
xml +
Quote:
Quote:
Quote:
db) for redundancy.

Is there any consideration regarding automatic log rotation (e.g.
hourly, or user specified)
without the need of a HUP? Now, that could make things a lot easier

for

Quote:
the development of
an external csv to db aggregation script because the script would
cdrs
0
Quote:
Quote:
Quote:
Quote:
cdrs
1
cdrs
2
Quote:
Quote:
Quote:
Quote:
cdrs
3
cdrs
4
Quote:
Quote:
Quote:
Quote:
cdrs
5 cdrs
6
cdrs
7
Quote:
Quote:
Quote:
cdrs
8
cdrs
9
Quote:
Quote:
Quote:
per hour)
that could slow things down considerably, wouldn't it?

Thanks again for your prompt replies,

0
per hour)
that could slow things down considerably, wouldn't it?

Thanks again for your prompt replies,

1
Quote:
Quote:
Quote:
per hour)
that could slow things down considerably, wouldn't it?

Thanks again for your prompt replies,

2
per hour)
that could slow things down considerably, wouldn't it?

Thanks again for your prompt replies,

3
Quote:
Quote:
Quote:
per hour)
that could slow things down considerably, wouldn't it?

Thanks again for your prompt replies,

4
per hour)
that could slow things down considerably, wouldn't it?

Thanks again for your prompt replies,

5
Quote:
Quote:
Quote:
per hour)
that could slow things down considerably, wouldn't it?

Thanks again for your prompt replies,

6 per hour)
that could slow things down considerably, wouldn't it?

Thanks again for your prompt replies,

7 per hour)
that could slow things down considerably, wouldn't it?

Thanks again for your prompt replies,

8
Quote:
Quote:
per hour)
that could slow things down considerably, wouldn't it?

Thanks again for your prompt replies,

9
At that level of activity then I would assume you'd want a more robust
solution which obviously would involve a server handling the CDRs
separately. That's where XML is a real winner: it can POST CDRs to a web
server and the webserver can handle all the pre-processing and db fun
stuff. And if the connection to the webserver failed, the CDRs would be
put on disk so that they aren't lost forever. Also, the webserver could
cache the CDRs to its disk (or whatever storage) if the db itself went
down but the webserver stayed up.

Just a thought, anyway. It may be extra layers but it's also extra
control.

-MC


0
Back to top
dave at 3c.co.uk
Guest





PostPosted: Thu Oct 30, 2008 1:07 am    Post subject: [Freeswitch-users] mod_cdr revival (or new module maybe) Reply with quote

regs@kinetix.gr wrote:
Quote:
Yes, the xml files give you tons of info... but isn't it a little
insufficient - performance wise -
to open and close so many files in such a little time. In a PBX
environment that wouldn't be an
issue but if we get to the small-voip-carrier level (some thousand cdrs
per hour)
that could slow things down considerably, wouldn't it?

Not that you'd notice. We run XML CDR to database scripting on each box
that we use
for switching, and it's a pretty trivial task compared with switching
all that media. Doing it
this way is:-
(a) distributed - one process per box scales nicely;
(b) robust - script down, DB down, no problem: files just queue up;
(c) simple - the script logic is trivial:
- while 1
- for each file in the XML CDR directory
- open it
- parse it (XML::Simple for us)
- insert it in to the DB
- delete it
- sleep for a couple of seconds
Two error cases: can't parse or can't find data which should be there:
move the file in to
another directory to be examined by real eyes; DB insert fails: break
out of inner loop and
it'll be retried after a short pause.

--Dave

--
David Knell, Director, 3C Limited
T: 020 8114 8901 F: 020 3002 7257 M: 001 415 630 3031
http://www.3c.co.uk


_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
Back to top
regs at kinetix.gr
Guest





PostPosted: Thu Oct 30, 2008 5:40 am    Post subject: [Freeswitch-users] mod_cdr revival (or new module maybe) Reply with quote

Quote:
"sleep for a couple of seconds"

But then you could only insert 1800 cdrs per hour...
If I was to insert 36000 cdrs per hour this means that I have to
open
parse
close
10 files per second. Imagine the I/O penalty just for opening - closing the file.
(the persing is the same for both situations)



David Knell wrote:
Quote:
Quote:
regs@kinetix.gr (regs@kinetix.gr) wrote:
Quote:
Yes, the xml files give you tons of info... but isn't it a little
insufficient - performance wise -
to open and close so many files in such a little time. In a PBX
environment that wouldn't be an
issue but if we get to the small-voip-carrier level (some thousand cdrs
per hour)
that could slow things down considerably, wouldn't it?

Not that you'd notice. We run XML CDR to database scripting on each box
that we use
for switching, and it's a pretty trivial task compared with switching
all that media. Doing it
this way is:-
(a) distributed - one process per box scales nicely;
(b) robust - script down, DB down, no problem: files just queue up;
(c) simple - the script logic is trivial:
- while 1
- for each file in the XML CDR directory
- open it
- parse it (XML::Simple for us)
- insert it in to the DB
- delete it
- sleep for a couple of seconds
Two error cases: can't parse or can't find data which should be there:
move the file in to
another directory to be examined by real eyes; DB insert fails: break
out of inner loop and
it'll be retried after a short pause.

--Dave

Back to top
dave at 3c.co.uk
Guest





PostPosted: Thu Oct 30, 2008 7:03 am    Post subject: [Freeswitch-users] mod_cdr revival (or new module maybe) Reply with quote

The sleep's done each time the directory's empty, not each time a file's written. File open and close
are trivial (it's probably still cached), and the contents are going to have to be parsed wherever you
process it.

We've used exactly this to process deliver CDRs on boxes handling in excess of 500K mins/day
without issue. And, looking at one now, the CDR processor's used about 4% of the CPU time
of FreeSWITCH, and about half that of the MySQL database which it writes the records to, also
on the local machine, from which they're simply copied to the main CDR processor.

It performance simply isn't worth worrying about.

--Dave
Quote:
Quote:
"sleep for a couple of seconds"

But then you could only insert 1800 cdrs per hour...
If I was to insert 36000 cdrs per hour this means that I have to
open
parse
close
10 files per second. Imagine the I/O penalty just for opening - closing the file.
(the persing is the same for both situations)



David Knell wrote:
Quote:
Quote:
regs@kinetix.gr wrote:
Quote:
Yes, the xml files give you tons of info... but isn't it a little
insufficient - performance wise -
to open and close so many files in such a little time. In a PBX
environment that wouldn't be an
issue but if we get to the small-voip-carrier level (some thousand cdrs
per hour)
that could slow things down considerably, wouldn't it?

Not that you'd notice. We run XML CDR to database scripting on each box
that we use
for switching, and it's a pretty trivial task compared with switching
all that media. Doing it
this way is:-
(a) distributed - one process per box scales nicely;
(b) robust - script down, DB down, no problem: files just queue up;
(c) simple - the script logic is trivial:
- while 1
- for each file in the XML CDR directory
- open it
- parse it (XML::Simple for us)
- insert it in to the DB
- delete it
- sleep for a couple of seconds
Two error cases: can't parse or can't find data which should be there:
move the file in to
another directory to be examined by real eyes; DB insert fails: break
out of inner loop and
it'll be retried after a short pause.

--Dave



_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org (Freeswitch-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
--
David Knell, Director, 3C Limited
T: 020 8114 8901 F: 020 3002 7257 M: 001 415 630 3031
http://www.3c.co.uk
Back to top
regs at kinetix.gr
Guest





PostPosted: Thu Oct 30, 2008 7:57 am    Post subject: [Freeswitch-users] mod_cdr revival (or new module maybe) Reply with quote

I'll try some tests with various combinations first and then decide what's best.

I have to say that I was astonished to find out that freeswitch had so many event handlers from the very beginning.
However it would be great if freeswitch had the options for extra functionality (auto log rotation, db cdrs etc)
so that it meets the peculiarities of every different project. That would make a big difference compared to
other softswitch solutions where the lack of such features prohibits people from using them (especially in
the carrier grade level). User feedback (and wish lists) is the key for the success of an open source (or not)
project...

Thank you all for your replies. You' ve been very helpful.

David Knell wrote:
Quote:
The sleep's done each time the directory's empty, not each time a file's written. File open and close
are trivial (it's probably still cached), and the contents are going to have to be parsed wherever you
process it.

We've used exactly this to process deliver CDRs on boxes handling in excess of 500K mins/day
without issue. And, looking at one now, the CDR processor's used about 4% of the CPU time
of FreeSWITCH, and about half that of the MySQL database which it writes the records to, also
on the local machine, from which they're simply copied to the main CDR processor.

It performance simply isn't worth worrying about.

--Dave
Quote:
Quote:
"sleep for a couple of seconds"

But then you could only insert 1800 cdrs per hour...
If I was to insert 36000 cdrs per hour this means that I have to
open
parse
close
10 files per second. Imagine the I/O penalty just for opening - closing the file.
(the persing is the same for both situations)



David Knell wrote:
Quote:
Quote:
regs@kinetix.gr wrote:
Quote:
Yes, the xml files give you tons of info... but isn't it a little
insufficient - performance wise -
to open and close so many files in such a little time. In a PBX
environment that wouldn't be an
issue but if we get to the small-voip-carrier level (some thousand cdrs
per hour)
that could slow things down considerably, wouldn't it?

Not that you'd notice. We run XML CDR to database scripting on each box
that we use
for switching, and it's a pretty trivial task compared with switching
all that media. Doing it
this way is:-
(a) distributed - one process per box scales nicely;
(b) robust - script down, DB down, no problem: files just queue up;
(c) simple - the script logic is trivial:
- while 1
- for each file in the XML CDR directory
- open it
- parse it (XML::Simple for us)
- insert it in to the DB
- delete it
- sleep for a couple of seconds
Two error cases: can't parse or can't find data which should be there:
move the file in to
another directory to be examined by real eyes; DB insert fails: break
out of inner loop and
it'll be retried after a short pause.

--Dave


_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
--
David Knell, Director, 3C Limited
T: 020 8114 8901 F: 020 3002 7257 M: 001 415 630 3031
http://www.3c.co.uk

_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org (Freeswitch-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
Back to top
anthony.minessale at g...
Guest





PostPosted: Thu Oct 30, 2008 8:27 am    Post subject: [Freeswitch-users] mod_cdr revival (or new module maybe) Reply with quote

Actually,

The goal is to not limit the functionality by over thinking how things will be used but to provide building blocks to make much
more possible. Our analogy for this is that if you take the common lego reference, If you have something cool built out of legos but they are super glued together it limits what you *could* have done had then not been.

The system that David described is indeed ideal. I have mentioned it more than once and its no coincidence that FreeSWITCH plays into that model to a tee. I do not mention it very often because I think understanding that concept alone is valuable advice that I'd prefer not to waste on deaf ears so I recommend you heed his suggestion.

Logging direct to DB is not a sin or anything but it's just not very reliable. We have so many ways to process call records as it is.
you can use the cdr_csv if you want legacy support from asterisk that you can eventually tweak with templatable output, you can even make a template that generates the log as ready to roll sql statements you can just cat into mysql.

We have event_socket + channel_hangup event which has all the same info that the cdr_csv uses to generate the file.
We have the XML cdr that is a 3 dimensional view of the call with an account of nearly everything that happend including
each application executed, branching records when the call is transferred complete with a timestamp for each occurance.

--"However it would be great if freeswitch had the options for extra functionality (auto log rotation, db cdrs etc)
so that it meets the peculiarities of every different project."

A single cron entry that does kill -HUP `cat freeswitch.pid` and you have automatic rotation in unix.
In windows there are several cron equivs and instead of kill you can use an xml-rpc script to call the FSAPI call "fsctl send_sighup"
also you can schedule it to happen automaticly with the FSAPI

sched_api @300 mygroup fsctl send_sighup

A script connected to event socket subscribed to channel_hangup can act as your DB cdr engine and has the bonus that it could
connect to N boxes at once as well as be able to tell if the system was in trouble by listening for system heartbeat events and or socket disconnect.

Moral of the story, there is more already there than first appears. I've always hated the M$ paper clip so I strive not to bring him back.
"It looks like you are trying to run a VoIP termination company! Shall I assume how you want your call details?"



On Thu, Oct 30, 2008 at 7:43 AM, regs@kinetix.gr (regs@kinetix.gr) <regs@kinetix.gr (regs@kinetix.gr)> wrote:
Quote:
I'll try some tests with various combinations first and then decide what's best.

I have to say that I was astonished to find out that freeswitch had so many event handlers from the very beginning.
However it would be great if freeswitch had the options for extra functionality (auto log rotation, db cdrs etc)
so that it meets the peculiarities of every different project. That would make a big difference compared to
other softswitch solutions where the lack of such features prohibits people from using them (especially in
the carrier grade level). User feedback (and wish lists) is the key for the success of an open source (or not)
project...

Thank you all for your replies. You' ve been very helpful.


David Knell wrote:
Quote:
The sleep's done each time the directory's empty, not each time a file's written. File open and close
are trivial (it's probably still cached), and the contents are going to have to be parsed wherever you
process it.

We've used exactly this to process deliver CDRs on boxes handling in excess of 500K mins/day
without issue. And, looking at one now, the CDR processor's used about 4% of the CPU time
of FreeSWITCH, and about half that of the MySQL database which it writes the records to, also
on the local machine, from which they're simply copied to the main CDR processor.

It performance simply isn't worth worrying about.

--Dave
Quote:
Quote:
"sleep for a couple of seconds"

But then you could only insert 1800 cdrs per hour...
If I was to insert 36000 cdrs per hour this means that I have to
open
parse
close
10 files per second. Imagine the I/O penalty just for opening - closing the file.
(the persing is the same for both situations)



David Knell wrote:
Quote:
Quote:
regs@kinetix.gr (regs@kinetix.gr) wrote:
Quote:
Yes, the xml files give you tons of info... but isn't it a little
insufficient - performance wise -
to open and close so many files in such a little time. In a PBX
environment that wouldn't be an
issue but if we get to the small-voip-carrier level (some thousand cdrs
per hour)
that could slow things down considerably, wouldn't it?

Not that you'd notice. We run XML CDR to database scripting on each box
that we use
for switching, and it's a pretty trivial task compared with switching
all that media. Doing it
this way is:-
(a) distributed - one process per box scales nicely;
(b) robust - script down, DB down, no problem: files just queue up;
(c) simple - the script logic is trivial:
- while 1
- for each file in the XML CDR directory
- open it
- parse it (XML::Simple for us)
- insert it in to the DB
- delete it
- sleep for a couple of seconds
Two error cases: can't parse or can't find data which should be there:
move the file in to
another directory to be examined by real eyes; DB insert fails: break
out of inner loop and
it'll be retried after a short pause.

--Dave


_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org (Freeswitch-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
--
David Knell, Director, 3C Limited
T: 020 8114 8901 F: 020 3002 7257 M: 001 415 630 3031
http://www.3c.co.uk
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org (Freeswitch-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org





_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org (Freeswitch-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org




--
Anthony Minessale II

FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/

AIM: anthm
MSN:anthony_minessale@hotmail.com ([email]MSN%3Aanthony_minessale@hotmail.com[/email])
GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com ([email]PAYPAL%3Aanthony.minessale@gmail.com[/email])
IRC: irc.freenode.net #freeswitch

FreeSWITCH Developer Conference
sip:888@conference.freeswitch.org ([email]sip%3A888@conference.freeswitch.org[/email])
iax:guest@conference.freeswitch.org/888
googletalk:conf+888@conference.freeswitch.org ([email]googletalk%3Aconf%2B888@conference.freeswitch.org[/email])
pstn:213-799-1400
Back to top
Display posts from previous:   
Post new topic   Reply to topic    VoIP Mailing List Archives Forum Index -> freeSWITCH Users All times are GMT - 5 Hours
Goto page 1, 2  Next
Page 1 of 2

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group

VoiceMeUp - Corporate & Wholesale VoIP Services