VoIP Mailing List Archives
Mailing list archives for the VoIP community |
|
View previous topic :: View next topic |
Author |
Message |
kokoska.rokoska at pos... Guest
|
Posted: Fri Feb 20, 2009 6:35 am Post subject: [Freeswitch-users] SIP dump to DB |
|
|
Hi all,
I'm facing the problem I need all SIP messages "going thru" FreeSWITCH
(I know FS i B2BUA - so, better to say just "all SIP messages") logged
somewhere and this log have to be "searchable" (by call-id etc) and I
should be able to simply delete "old" messages...
And more over - it should be done on not trivial SIP messages amount -
say hundreds messages per second.
My questions are:
1. Do you have any suggestion how to do it with FreeSWITCH?
2. Or it is not possible now, and "bounty" is necessary?
3. How hard it will be to implement?
----------
FYI:
I think something like SER/Kamailio/OpenSIPS siptrace is what I'am
(probably) looking for:
http://www.kamailio.org/docs/modules/1.4.x/siptrace.html
It could go (based on my tests) up to 7-10.000 messages per second till
MySQL "dies"...
----------
Thanks for your time
Best regards,
kokoska.rokoska
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org |
|
Back to top |
|
|
anthony.minessale at g... Guest
|
Posted: Fri Feb 20, 2009 9:15 am Post subject: [Freeswitch-users] SIP dump to DB |
|
|
what exact info do you need? That's likely to be a challenge with any database to store at that speed.
On Fri, Feb 20, 2009 at 5:24 AM, kokoska rokoska <kokoska.rokoska@post.cz (kokoska.rokoska@post.cz)> wrote:
--
Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
AIM: anthm
MSN:anthony_minessale@hotmail.com ([email]MSN%3Aanthony_minessale@hotmail.com[/email])
GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com ([email]PAYPAL%3Aanthony.minessale@gmail.com[/email])
IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference
sip:888@conference.freeswitch.org ([email]sip%3A888@conference.freeswitch.org[/email])
iax:guest@conference.freeswitch.org/888
googletalk:conf+888@conference.freeswitch.org ([email]googletalk%3Aconf%2B888@conference.freeswitch.org[/email])
pstn:213-799-1400 |
|
Back to top |
|
|
ajlong at worldlink.net Guest
|
Posted: Fri Feb 20, 2009 9:40 am Post subject: [Freeswitch-users] SIP dump to DB |
|
|
MySQL MEMORY/HEAP table might be ideal for this. This data is prob not
critical and is probably
being used for diagnosing peer connectivity issues anyway.
If it is critical well then... there are always trade offs right
I think in general what he is speaking of is just some sort of temporary SIP
trace setup that can log
that can be controlled or filtered. So its not just all or nothing.
I wonder is it possible to enable the current sip trace functionality via a
variable.
For example something like this.
<action application="bridge"
data="{sip_trace=on}sofia/public/XXXXXXXXXX@10.10.10.1" />
This of course only helps for bridged B2BUA calls.
But if inbound sip tracing is required a user param like sip_trace could
address that, yes?
That could be a good starting point, then perhaps I could help roll a
mod_xml_siptrace module
based on the mod_xml_cdr design/concept that could somehow link into the
existing sip trace logging.
Just some thoughts, no idea how much of this exists today.
Regards,
-Adam
From: freeswitch-users-bounces@lists.freeswitch.org
[mailto:freeswitch-users-bounces@lists.freeswitch.org] On Behalf Of Anthony
Minessale
Sent: Friday, February 20, 2009 8:55 AM
To: freeswitch-users@lists.freeswitch.org
Subject: Re: [Freeswitch-users] SIP dump to DB
what exact info do you need? That's likely to be a challenge with any
database to store at that speed.
On Fri, Feb 20, 2009 at 5:24 AM, kokoska rokoska <kokoska.rokoska@post.cz>
wrote:
Hi all,
I'm facing the problem I need all SIP messages "going thru" FreeSWITCH
(I know FS i B2BUA - so, better to say just "all SIP messages") logged
somewhere and this log have to be "searchable" (by call-id etc) and I
should be able to simply delete "old" messages...
And more over - it should be done on not trivial SIP messages amount -
say hundreds messages per second.
My questions are:
1. Do you have any suggestion how to do it with FreeSWITCH?
2. Or it is not possible now, and "bounty" is necessary?
3. How hard it will be to implement?
----------
FYI:
I think something like SER/Kamailio/OpenSIPS siptrace is what I'am
(probably) looking for:
http://www.kamailio.org/docs/modules/1.4.x/siptrace.html
It could go (based on my tests) up to 7-10.000 messages per second till
MySQL "dies"...
----------
Thanks for your time
Best regards,
kokoska.rokoska
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
--
Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
AIM: anthm
MSN:anthony_minessale@hotmail.com
GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com
IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference
sip:888@conference.freeswitch.org
iax:guest@conference.freeswitch.org/888
googletalk:conf+888@conference.freeswitch.org
pstn:213-799-1400
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org |
|
Back to top |
|
|
kokoska.rokoska at pos... Guest
|
Posted: Fri Feb 20, 2009 9:49 am Post subject: [Freeswitch-users] SIP dump to DB |
|
|
Anthony Minessale napsal(a):
Quote: | what exact info do you need? That's likely to be a challenge with any
database to store at that speed.
|
Thank you very much, Anthony, for your reply!
I should say: Personally I don't need it (I see preformance penalty),
but few people around me need to store somewhere ALL sip messages going
through the server. And ALL means "really all" (well, it will be very
helpful if I can skip OPTIONS and other nat-keep-alive related messages).
So I need something like "sipgrep dump" but I should be able to simply
corelate messages to user (in case of MESSAGE, REGISTER etc.) and to
user+call (in case of INVITE, BYE, CANCEL etc.).
-------------
BTW:
I'm sure it will be challenge for DB - and thus I made some tests with
mentioned siptrace.
One SIP call is about 18-21 SIP messages, so for 100 cps I should make a
little bit over 2.000 INSERTs per second. 900 REGISTERs per second will
generate about 5.500 INSERTs per second => I need to fire cca 8.000
INSERTs per second...
-------------
But, may be, better solution exists - without DB.
Any hint is very appreciated
Best regards,
kokoska.rokoska
Quote: | On Fri, Feb 20, 2009 at 5:24 AM, kokoska rokoska
<kokoska.rokoska@post.cz <mailto:kokoska.rokoska@post.cz>> wrote:
Hi all,
I'm facing the problem I need all SIP messages "going thru" FreeSWITCH
(I know FS i B2BUA - so, better to say just "all SIP messages") logged
somewhere and this log have to be "searchable" (by call-id etc) and I
should be able to simply delete "old" messages...
And more over - it should be done on not trivial SIP messages amount -
say hundreds messages per second.
My questions are:
1. Do you have any suggestion how to do it with FreeSWITCH?
2. Or it is not possible now, and "bounty" is necessary?
3. How hard it will be to implement?
----------
FYI:
I think something like SER/Kamailio/OpenSIPS siptrace is what I'am
(probably) looking for:
http://www.kamailio.org/docs/modules/1.4.x/siptrace.html
It could go (based on my tests) up to 7-10.000 messages per second till
MySQL "dies"...
----------
Thanks for your time
Best regards,
kokoska.rokoska
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
<mailto:Freeswitch-users@lists.freeswitch.org>
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
--
Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
AIM: anthm
MSN:anthony_minessale@hotmail.com
<mailto:MSN%3Aanthony_minessale@hotmail.com>
GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com
<mailto:PAYPAL%3Aanthony.minessale@gmail.com>
IRC: irc.freenode.net <http://irc.freenode.net> #freeswitch
FreeSWITCH Developer Conference
sip:888@conference.freeswitch.org
<mailto:sip%3A888@conference.freeswitch.org>
iax:guest@conference.freeswitch.org/888
<http://iax:guest@conference.freeswitch.org/888>
googletalk:conf+888@conference.freeswitch.org
<mailto:googletalk%3Aconf%2B888@conference.freeswitch.org>
pstn:213-799-1400
------------------------------------------------------------------------
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
|
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org |
|
Back to top |
|
|
kokoska.rokoska at pos... Guest
|
Posted: Fri Feb 20, 2009 9:59 am Post subject: [Freeswitch-users] SIP dump to DB |
|
|
Adam Long napsal(a):
Quote: | MySQL MEMORY/HEAP table might be ideal for this. This data is prob not
critical and is probably
being used for diagnosing peer connectivity issues anyway.
If it is critical well then... there are always trade offs right
|
Thank you very much, Adam, for interest!
You are right - it is for diagnosis. And while the data is not critical,
I can't leave them in MySQL MEMORY table. Based on my counts I have to
store and maintain about 70-90 GiB of SIP messages and I simply don't
have enough RAM
Quote: | I think in general what he is speaking of is just some sort of temporary SIP
trace setup that can log
that can be controlled or filtered. So its not just all or nothing.
|
Exactly!
Quote: | I wonder is it possible to enable the current sip trace functionality via a
variable.
For example something like this.
<action application="bridge"
data="{sip_trace=on}sofia/public/XXXXXXXXXX@10.10.10.1" />
This of course only helps for bridged B2BUA calls.
|
It will be very helpful...
Quote: | But if inbound sip tracing is required a user param like sip_trace could
address that, yes?
|
Yes, I need all SIP packets except nat-keep-alives...
BTW: How to recognize them? I know how could I do it on proxy (SER
like), but what about on FreeSWITCH?
Anyway - for me it shouldn't be an issue, because I filter then on
loadbalancer.
Quote: | That could be a good starting point, then perhaps I could help roll a
mod_xml_siptrace module
based on the mod_xml_cdr design/concept that could somehow link into the
existing sip trace logging.
|
It will be very powerfull, but I'm affraid it can't scale to thousands
request per second.
Quote: | Just some thoughts, no idea how much of this exists today.
|
Thanks once more, Adam!
Best regards,
kokoska.rokoska
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org |
|
Back to top |
|
|
sicfslist at gmail.com Guest
|
Posted: Fri Feb 20, 2009 10:12 am Post subject: [Freeswitch-users] SIP dump to DB |
|
|
Why not just use NGREP and then dump the packets at a more reasonable pace? You aren't going to be able to analysis in real time anyway. |
|
Back to top |
|
|
kokoska.rokoska at pos... Guest
|
Posted: Fri Feb 20, 2009 10:37 am Post subject: [Freeswitch-users] SIP dump to DB |
|
|
Shelby Ramsey napsal(a):
Quote: | Why not just use NGREP and then dump the packets at a more reasonable
pace?
|
Thank you very much, Shelby, for your interest!
I can't use ngrep because the dump is not searchable IMO
See below, please.
You aren't going to be able to analysis in real time anyway.
Some basic diagnosis should by doable realtime. For example:
1. find whole SIP trace for selected call (both a-leg, b-leg)
2. find all succeful user REGISTERs for given period
...
And I have no idea how to do it on 50-100 GiB plain text file.
Best regards,
kokoska.rokoska
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org |
|
Back to top |
|
|
anthony.minessale at g... Guest
|
Posted: Fri Feb 20, 2009 10:47 am Post subject: [Freeswitch-users] SIP dump to DB |
|
|
you could try sippcapdump but i hear it needs work but it snoops the wire and tries to make individual files out of each call.
On Fri, Feb 20, 2009 at 9:27 AM, kokoska rokoska <kokoska.rokoska@post.cz (kokoska.rokoska@post.cz)> wrote:
--
Anthony Minessale II
FreeSWITCH http://www.freeswitch.org/
ClueCon http://www.cluecon.com/
AIM: anthm
MSN:anthony_minessale@hotmail.com ([email]MSN%3Aanthony_minessale@hotmail.com[/email])
GTALK/JABBER/PAYPAL:anthony.minessale@gmail.com ([email]PAYPAL%3Aanthony.minessale@gmail.com[/email])
IRC: irc.freenode.net #freeswitch
FreeSWITCH Developer Conference
sip:888@conference.freeswitch.org ([email]sip%3A888@conference.freeswitch.org[/email])
iax:guest@conference.freeswitch.org/888
googletalk:conf+888@conference.freeswitch.org ([email]googletalk%3Aconf%2B888@conference.freeswitch.org[/email])
pstn:213-799-1400 |
|
Back to top |
|
|
sicfslist at gmail.com Guest
|
Posted: Fri Feb 20, 2009 10:47 am Post subject: [Freeswitch-users] SIP dump to DB |
|
|
Sorry .. I didn't give enough detail. My point was to dump it via NGREP ... parse it using something else to get it into a database where it would be usable. Then you can match calls from the CDR (using the UUID) to the database. The benefit is that you don't have to put the burden on your FS boxes to do it ... Just monitor from another device and then dump it into the database. Of course you better have a beast of database if you want to do 10,000 writes per second or be running something like NDB that scales well.
There are other tools like scapy as well that can be quite useful in fact.
Shelby |
|
Back to top |
|
|
kokoska.rokoska at pos... Guest
|
Posted: Fri Feb 20, 2009 11:07 am Post subject: [Freeswitch-users] SIP dump to DB |
|
|
Anthony Minessale napsal(a):
Quote: | you could try sippcapdump but i hear it needs work but it snoops the
wire and tries to make individual files out of each call.
|
Thank you very much, Anthony, for the suggestion!
I will look at it.
Best regards,
kokoska.rokoska
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org |
|
Back to top |
|
|
jaugenstine at gmail.com Guest
|
Posted: Fri Feb 20, 2009 11:11 am Post subject: [Freeswitch-users] SIP dump to DB |
|
|
You can tcpdump and then use wireshark to graph the calls. When the dump is displayed in wireshark, select 'Statistics' -> VoIP Calls. You will see a display of all VoIP calls. Select the one you want graphed, or select them all and you will see REINVITE and REFER interaction as well as RTP streams.
Jonathan
On Fri, Feb 20, 2009 at 7:33 AM, Anthony Minessale <anthony.minessale@gmail.com (anthony.minessale@gmail.com)> wrote:
|
|
Back to top |
|
|
kokoska.rokoska at pos... Guest
|
Posted: Fri Feb 20, 2009 11:22 am Post subject: [Freeswitch-users] SIP dump to DB |
|
|
Shelby Ramsey napsal(a):
Quote: | Sorry .. I didn't give enough detail. My point was to dump it via NGREP
... parse it using something else to get it into a database where it
would be usable.
|
This is good point! Thank you very much, Shelby!
Quote: | Then you can match calls from the CDR (using the UUID)
to the database.
|
This is exactly what I try to accomplish
Quote: | The benefit is that you don't have to put the burden
on your FS boxes to do it ... Just monitor from another device and then
dump it into the database.
|
I think of 2 possibilities:
1. On router replicate whole traffic to dedicated machine and process
dump there (but it probably kills that machine, because it gets all RTP
traffic)
2. Modify FreeSWITCH/Sofia (I have no idea how hard it will be, or if it
is even possible) to duplicate all SIP messages to given URI - main
benefit of this scenario is that I have only SIP messages on logging
machine and that I can use SERlike proxy to parse messages and store
them to DB.
Quote: | Of course you better have a beast of
database if you want to do 10,000 writes per second or be running
something like NDB that scales well.
|
NDB is overkill for logging
What I think of is some kind of "caching" on FreeSWITCH side. I.e. store
mesasges to DB in bigger chukns (whole call etc.) It will significatnly
reduce DB utilization...
Quote: | There are other tools like scapy as well that can be quite useful in fact.
| I will look at it, thank you Shelby!
Best regards,
kokoska.roksoka
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org |
|
Back to top |
|
|
kokoska.rokoska at pos... Guest
|
Posted: Fri Feb 20, 2009 11:23 am Post subject: [Freeswitch-users] SIP dump to DB |
|
|
jonathan augenstine napsal(a):
Quote: | You can tcpdump and then use wireshark to graph the calls. When the
dump is displayed in wireshark, select 'Statistics' -> VoIP Calls. You
will see a display of all VoIP calls. Select the one you want graphed,
or select them all and you will see REINVITE and REFER interaction as
well as RTP streams.
|
Thank you very much, jonathan, for your interest!
I use ngrep+wireshark many times a day, but I'm affraid it is not
suitable for that amount of data.
Even with few hundreds MiBs of pcap file wireshark becoms very slow and
I can't imagine how to load 50-100 GiB file with milions of calls and
try to search for one of them
And, even worse, I should "rotate" the file and, don't end with call
divided to multiple files...
Best regards,
kokoska.rokoska
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org |
|
Back to top |
|
|
josephbajin at gmail.com Guest
|
Posted: Mon Feb 23, 2009 3:19 pm Post subject: [Freeswitch-users] SIP dump to DB |
|
|
Basically, you are trying to build what Empirix has with their Hammer tool.
You can create an application that is basically a mix of tshark and a database feeder.
You sniff with tshark and going to basically pipe it to another application that will read the pcap file, parse it, and load it into the db for you. There are plenty of modules out there that will read pcap for you.
On Fri, Feb 20, 2009 at 11:15 AM, kokoska rokoska <kokoska.rokoska@post.cz (kokoska.rokoska@post.cz)> wrote:
Quote: |
jonathan augenstine napsal(a):
Quote: | You can tcpdump and then use wireshark to graph the calls. When the
dump is displayed in wireshark, select 'Statistics' -> VoIP Calls. You
will see a display of all VoIP calls. Select the one you want graphed,
or select them all and you will see REINVITE and REFER interaction as
well as RTP streams.
|
Thank you very much, jonathan, for your interest!
I use ngrep+wireshark many times a day, but I'm affraid it is not
suitable for that amount of data.
Even with few hundreds MiBs of pcap file wireshark becoms very slow and
I can't imagine how to load 50-100 GiB file with milions of calls and
try to search for one of them
And, even worse, I should "rotate" the file and, don't end with call
divided to multiple files...
Best regards,
kokoska.rokoska
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org (Freeswitch-users@lists.freeswitch.org)
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org
|
|
|
Back to top |
|
|
kokoska.rokoska at pos... Guest
|
Posted: Mon Feb 23, 2009 5:43 pm Post subject: [Freeswitch-users] SIP dump to DB |
|
|
Joseph Bajin napsal(a):
Quote: | Basically, you are trying to build what Empirix has with their Hammer tool.
|
Thank you very much, Joseph, for your interest!
I have never heard about Empirix (I'll look at it), but what I'm trying
to build is something like SER/Kamailio/OpenSIPS sip_trace module.
Quote: | You can create an application that is basically a mix of tshark and a
database feeder.
You sniff with tshark and going to basically pipe it to another
application that will read the pcap file, parse it, and load it into the
db for you. There are plenty of modules out there that will read pcap
for you.
|
Thank you once more, Joseph, for suggestion!
I think about it - it will be challenge for me to write robust and still
fast enough (thousands messages per second) SIP parser + DB feeder
Best regards,
kokoska.rokoska
_______________________________________________
Freeswitch-users mailing list
Freeswitch-users@lists.freeswitch.org
http://lists.freeswitch.org/mailman/listinfo/freeswitch-users
UNSUBSCRIBE:http://lists.freeswitch.org/mailman/options/freeswitch-users
http://www.freeswitch.org |
|
Back to top |
|
|
|
|
|
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|