VoIP Mailing List Archives
Mailing list archives for the VoIP community |
|
| View previous topic :: View next topic |
| Author |
Message |
serov.d.p at gmail.com
Guest
|
 Posted: Tue Sep 06, 2016 8:33 am Post subject: [asterisk-users] Upgrading asterisk 13.7 to 13.11. Segfaults |
 |
|
Hello.
Several months server working on asterisk 13.7 and pjproject 2.5
(installed separately). Once a day the server crashes or hangs and is
familiar sores that written watchdogs.
Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5).
Solved all the problems with compilation I started asterisk several
times and each time after 5-7 seconds was seg fault.
So I didn't get to use the new version of asterisk. And I really wanted
to be able to find and remove the cause. I would be grateful for any help.
first segfault:
Program terminated with signal 11, Segmentation fault.
#0 0x00007fe19337357e in pj_atomic_dec_and_get (atomic_var=0x1a9a) at
../src/pj/os_core_unix.c:962
962 pj_mutex_lock( atomic_var->mutex );
backtrace: https://ruvoip.net/_other/voip/2016-09-05-1/backtrace-threads.txt
log (16 MB) : https://ruvoip.net/_other/voip/2016-09-05-1/full.txt
second segfault:
Program terminated with signal 11, Segmentation fault.
#0 0x00007f5777412788 in pjsip_auth_clt_reinit_req
(sess=0x7f56f4088a30, rdata=0x7f56ec2b0c98, old_request=0x7f56f47775a8,
new_request=0x7f56d5b38980) at ../src/pjsip/sip_auth_client.c:1144
1144 PJ_ASSERT_RETURN(old_request->msg->type == PJSIP_REQUEST_MSG,
backtrace: https://ruvoip.net/_other/voip/2016-09-05-2/backtrace-threads.txt
log (15 MB) : https://ruvoip.net/_other/voip/2016-09-05-2/full.txt
third segfault:
Program terminated with signal 11, Segmentation fault.
#0 0x00007f592bcad53d in pj_pool_alloc (pool=0x7f580000002c, size=80)
at ../include/pj/pool_i.h:60
60 void *ptr = pj_pool_alloc_from_block(pool->block_list.next, size);
backtrace: https://ruvoip.net/_other/voip/2016-09-05-3/backtrace-threads.txt
log (11 MB) : https://ruvoip.net/_other/voip/2016-09-05-3/full.txt
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
http://www.asterisk.org/community/astricon-user-conference
New to Asterisk? Start here:
https://wiki.asterisk.org/wiki/display/AST/Getting+Started
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users |
|
| Back to top |
|
 |
gjoseph at digium.com
Guest
|
| Posted: Tue Sep 06, 2016 8:43 am Post subject: [asterisk-users] Upgrading asterisk 13.7 to 13.11. Segfaults |
|
|
On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov <serov.d.p@gmail.com (serov.d.p@gmail.com)> wrote:
| Quote: | Hello.
Several months server working on asterisk 13.7 and pjproject 2.5 (installed separately). Once a day the server crashes or hangs and is familiar sores that written watchdogs.
Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5). Solved all the problems with compilation I started asterisk several times and each time after 5-7 seconds was seg fault.
So I didn't get to use the new version of asterisk. And I really wanted to be able to find and remove the cause. I would be grateful for any help.
|
All 3 of the backtraces are in different pjproject places which is weird. Makes me think there's still a library mismatch somewhere. Is the separately compiled pjproject still installed on both the build machine and the server? They should be ignored if --with-pjproject-bundled is specified but you might want to remove them and try again.
| Quote: |
first segfault:
Program terminated with signal 11, Segmentation fault.
#0 0x00007fe19337357e in pj_atomic_dec_and_get (atomic_var=0x1a9a) at ../src/pj/os_core_unix.c:962
962 pj_mutex_lock( atomic_var->mutex );
backtrace: https://ruvoip.net/_other/voip/2016-09-05-1/backtrace-threads.txt
log (16 MB) : https://ruvoip.net/_other/voip/2016-09-05-1/full.txt
second segfault:
Program terminated with signal 11, Segmentation fault.
#0 0x00007f5777412788 in pjsip_auth_clt_reinit_req (sess=0x7f56f4088a30, rdata=0x7f56ec2b0c98, old_request=0x7f56f47775a8, new_request=0x7f56d5b38980) at ../src/pjsip/sip_auth_client.c:1144
1144 PJ_ASSERT_RETURN(old_request->msg->type == PJSIP_REQUEST_MSG,
backtrace: https://ruvoip.net/_other/voip/2016-09-05-2/backtrace-threads.txt
log (15 MB) : https://ruvoip.net/_other/voip/2016-09-05-2/full.txt
third segfault:
Program terminated with signal 11, Segmentation fault.
#0 0x00007f592bcad53d in pj_pool_alloc (pool=0x7f580000002c, size=80) at ../include/pj/pool_i.h:60
60 void *ptr = pj_pool_alloc_from_block(pool->block_list.next, size);
backtrace: https://ruvoip.net/_other/voip/2016-09-05-3/backtrace-threads.txt
log (11 MB) : https://ruvoip.net/_other/voip/2016-09-05-3/full.txt
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
http://www.asterisk.org/community/astricon-user-conference
New to Asterisk? Start here:
https://wiki.asterisk.org/wiki/display/AST/Getting+Started
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
|
--
George JosephDigium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org |
|
| Back to top |
|
|
lardconcepts at gmail.com
Guest
|
| Posted: Tue Sep 06, 2016 8:52 am Post subject: [asterisk-users] Upgrading asterisk 13.7 to 13.11. Segfaults |
|
|
All your libraries, kernel, headers and build tools up to date?
The other thing that might be worth noting is the warning along the
lines of "contains modules that were not installed by this version of
Asterisk".
Might be worth deleting anything that appears there, and then starting Asterisk.
On 6 September 2016 at 14:42, George Joseph <gjoseph@digium.com> wrote:
| Quote: |
On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov <serov.d.p@gmail.com> wrote:
| Quote: |
Hello.
Several months server working on asterisk 13.7 and pjproject 2.5
(installed separately). Once a day the server crashes or hangs and is
familiar sores that written watchdogs.
Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5).
Solved all the problems with compilation I started asterisk several times
and each time after 5-7 seconds was seg fault.
So I didn't get to use the new version of asterisk. And I really wanted to
be able to find and remove the cause. I would be grateful for any help.
|
All 3 of the backtraces are in different pjproject places which is weird.
Makes me think there's still a library mismatch somewhere. Is the
separately compiled pjproject still installed on both the build machine and
the server? They should be ignored if --with-pjproject-bundled is specified
but you might want to remove them and try again.
| Quote: |
first segfault:
Program terminated with signal 11, Segmentation fault.
#0 0x00007fe19337357e in pj_atomic_dec_and_get (atomic_var=0x1a9a) at
../src/pj/os_core_unix.c:962
962 pj_mutex_lock( atomic_var->mutex );
backtrace:
https://ruvoip.net/_other/voip/2016-09-05-1/backtrace-threads.txt
log (16 MB) : https://ruvoip.net/_other/voip/2016-09-05-1/full.txt
second segfault:
Program terminated with signal 11, Segmentation fault.
#0 0x00007f5777412788 in pjsip_auth_clt_reinit_req (sess=0x7f56f4088a30,
rdata=0x7f56ec2b0c98, old_request=0x7f56f47775a8,
new_request=0x7f56d5b38980) at ../src/pjsip/sip_auth_client.c:1144
1144 PJ_ASSERT_RETURN(old_request->msg->type == PJSIP_REQUEST_MSG,
backtrace:
https://ruvoip.net/_other/voip/2016-09-05-2/backtrace-threads.txt
log (15 MB) : https://ruvoip.net/_other/voip/2016-09-05-2/full.txt
third segfault:
Program terminated with signal 11, Segmentation fault.
#0 0x00007f592bcad53d in pj_pool_alloc (pool=0x7f580000002c, size=80) at
../include/pj/pool_i.h:60
60 void *ptr = pj_pool_alloc_from_block(pool->block_list.next,
size);
backtrace:
https://ruvoip.net/_other/voip/2016-09-05-3/backtrace-threads.txt
log (11 MB) : https://ruvoip.net/_other/voip/2016-09-05-3/full.txt
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
http://www.asterisk.org/community/astricon-user-conference
New to Asterisk? Start here:
https://wiki.asterisk.org/wiki/display/AST/Getting+Started
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
|
--
George Joseph
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
http://www.asterisk.org/community/astricon-user-conference
New to Asterisk? Start here:
https://wiki.asterisk.org/wiki/display/AST/Getting+Started
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
|
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
http://www.asterisk.org/community/astricon-user-conference
New to Asterisk? Start here:
https://wiki.asterisk.org/wiki/display/AST/Getting+Started
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users |
|
| Back to top |
|
|
serov.d.p at gmail.com
Guest
|
| Posted: Tue Sep 06, 2016 8:59 am Post subject: [asterisk-users] Upgrading asterisk 13.7 to 13.11. Segfaults |
|
|
06.09.2016 16:42, George Joseph пишет:
| Quote: |
On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov <serov.d.p@gmail.com (serov.d.p@gmail.com)> wrote:
| Quote: | Hello.
Several months server working on asterisk 13.7 and pjproject 2.5 (installed separately). Once a day the server crashes or hangs and is familiar sores that written watchdogs.
Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5). Solved all the problems with compilation I started asterisk several times and each time after 5-7 seconds was seg fault.
So I didn't get to use the new version of asterisk. And I really wanted to be able to find and remove the cause. I would be grateful for any help.
|
All 3 of the backtraces are in different pjproject places which is weird. Makes me think there's still a library mismatch somewhere. Is the separately compiled pjproject still installed on both the build machine and the server? They should be ignored if --with-pjproject-bundled is specified but you might want to remove them and try again.
|
George, thank you that you responded. What I was hoping 
I was also very surprised that SF can happen so quickly and so consistently. With all this in completely different places.
When installed separately pjproject source code with bundled not even compiled. Of course, I uninstalled and cleaned pjproject according to this instructions: https://wiki.asterisk.org/wiki/display/AST/Building+and+Installing+pjproject
At the moment I have restore separately pjproject and asterisk 13.7. But there is a directory with configured and compiled asterisk 13.11. Happy to provide the contents of the autoconfiguration files. |
|
| Back to top |
|
|
gjoseph at digium.com
Guest
|
| Posted: Tue Sep 06, 2016 9:08 am Post subject: [asterisk-users] Upgrading asterisk 13.7 to 13.11. Segfaults |
|
|
On Tue, Sep 6, 2016 at 7:58 AM, Dmitriy Serov <serov.d.p@gmail.com (serov.d.p@gmail.com)> wrote:
| Quote: | 06.09.2016 16:42, George Joseph пишет:
| Quote: |
On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov <serov.d.p@gmail.com (serov.d.p@gmail.com)> wrote:
| Quote: | Hello.
Several months server working on asterisk 13.7 and pjproject 2.5 (installed separately). Once a day the server crashes or hangs and is familiar sores that written watchdogs.
Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5). Solved all the problems with compilation I started asterisk several times and each time after 5-7 seconds was seg fault.
So I didn't get to use the new version of asterisk. And I really wanted to be able to find and remove the cause. I would be grateful for any help.
|
All 3 of the backtraces are in different pjproject places which is weird. Makes me think there's still a library mismatch somewhere. Is the separately compiled pjproject still installed on both the build machine and the server? They should be ignored if --with-pjproject-bundled is specified but you might want to remove them and try again.
|
George, thank you that you responded. What I was hoping
I was also very surprised that SF can happen so quickly and so consistently. With all this in completely different places.
When installed separately pjproject source code with bundled not even compiled. Of course, I uninstalled and cleaned pjproject according to this instructions: https://wiki.asterisk.org/wiki/display/AST/Building+and+Installing+pjproject
At the moment I have restore separately pjproject and asterisk 13.7. But there is a directory with configured and compiled asterisk 13.11. Happy to provide the contents of the autoconfiguration files.
|
config.log and makeopts would be useful if you can zip them up. Also, what distribution and version are you running? As Jonathan suggested, could there have been any left over or third-party modules left in the asterisk modules directory?
|
|
| Back to top |
|
|
serov.d.p at gmail.com
Guest
|
| Posted: Tue Sep 06, 2016 9:17 am Post subject: [asterisk-users] Upgrading asterisk 13.7 to 13.11. Segfaults |
|
|
06.09.2016 16:51, Jonathan H пишет:
| Quote: | All your libraries, kernel, headers and build tools up to date?
|
I can't be sure because I do not know the required versions of these
libraries. When you configure and build asterisk, no error detected.
The server has not a new kernel 3.13.6 #4 SMP Wed Aug 20 17:52:10 MSK
2014 x86_64
| Quote: |
The other thing that might be worth noting is the warning along the
lines of "contains modules that were not installed by this version of
Asterisk".
Might be worth deleting anything that appears there, and then starting Asterisk.
|
I don't use the autoloading of modules. All the necessary modules
written manually in modules.conf.
At the end of the installation it reported the extra modules (thanks)
and they removed me as "garbage".
| Quote: | On 6 September 2016 at 14:42, George Joseph <gjoseph@digium.com> wrote:
| Quote: |
On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov <serov.d.p@gmail.com> wrote:
| Quote: | Hello.
Several months server working on asterisk 13.7 and pjproject 2.5
(installed separately). Once a day the server crashes or hangs and is
familiar sores that written watchdogs.
Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5).
Solved all the problems with compilation I started asterisk several times
and each time after 5-7 seconds was seg fault.
So I didn't get to use the new version of asterisk. And I really wanted to
be able to find and remove the cause. I would be grateful for any help.
|
All 3 of the backtraces are in different pjproject places which is weird.
Makes me think there's still a library mismatch somewhere. Is the
separately compiled pjproject still installed on both the build machine and
the server? They should be ignored if --with-pjproject-bundled is specified
but you might want to remove them and try again.
| Quote: |
first segfault:
Program terminated with signal 11, Segmentation fault.
#0 0x00007fe19337357e in pj_atomic_dec_and_get (atomic_var=0x1a9a) at
../src/pj/os_core_unix.c:962
962 pj_mutex_lock( atomic_var->mutex );
backtrace:
https://ruvoip.net/_other/voip/2016-09-05-1/backtrace-threads.txt
log (16 MB) : https://ruvoip.net/_other/voip/2016-09-05-1/full.txt
second segfault:
Program terminated with signal 11, Segmentation fault.
#0 0x00007f5777412788 in pjsip_auth_clt_reinit_req (sess=0x7f56f4088a30,
rdata=0x7f56ec2b0c98, old_request=0x7f56f47775a8,
new_request=0x7f56d5b38980) at ../src/pjsip/sip_auth_client.c:1144
1144 PJ_ASSERT_RETURN(old_request->msg->type == PJSIP_REQUEST_MSG,
backtrace:
https://ruvoip.net/_other/voip/2016-09-05-2/backtrace-threads.txt
log (15 MB) : https://ruvoip.net/_other/voip/2016-09-05-2/full.txt
third segfault:
Program terminated with signal 11, Segmentation fault.
#0 0x00007f592bcad53d in pj_pool_alloc (pool=0x7f580000002c, size=80) at
../include/pj/pool_i.h:60
60 void *ptr = pj_pool_alloc_from_block(pool->block_list.next,
size);
backtrace:
https://ruvoip.net/_other/voip/2016-09-05-3/backtrace-threads.txt
log (11 MB) : https://ruvoip.net/_other/voip/2016-09-05-3/full.txt
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
http://www.asterisk.org/community/astricon-user-conference
New to Asterisk? Start here:
https://wiki.asterisk.org/wiki/display/AST/Getting+Started
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
|
--
George Joseph
Digium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
http://www.asterisk.org/community/astricon-user-conference
New to Asterisk? Start here:
https://wiki.asterisk.org/wiki/display/AST/Getting+Started
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
|
|
--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
Join the Asterisk Community at the 13th AstriCon, September 27-29, 2016
http://www.asterisk.org/community/astricon-user-conference
New to Asterisk? Start here:
https://wiki.asterisk.org/wiki/display/AST/Getting+Started
asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users |
|
| Back to top |
|
|
serov.d.p at gmail.com
Guest
|
| Posted: Tue Sep 06, 2016 9:47 am Post subject: [asterisk-users] Upgrading asterisk 13.7 to 13.11. Segfaults |
|
|
06.09.2016 17:08, George Joseph пишет:
| Quote: |
On Tue, Sep 6, 2016 at 7:58 AM, Dmitriy Serov <serov.d.p@gmail.com (serov.d.p@gmail.com)> wrote:
| Quote: | 06.09.2016 16:42, George Joseph пишет:
| Quote: |
On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov <serov.d.p@gmail.com (serov.d.p@gmail.com)> wrote:
| Quote: | Hello.
Several months server working on asterisk 13.7 and pjproject 2.5 (installed separately). Once a day the server crashes or hangs and is familiar sores that written watchdogs.
Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5). Solved all the problems with compilation I started asterisk several times and each time after 5-7 seconds was seg fault.
So I didn't get to use the new version of asterisk. And I really wanted to be able to find and remove the cause. I would be grateful for any help.
|
All 3 of the backtraces are in different pjproject places which is weird. Makes me think there's still a library mismatch somewhere. Is the separately compiled pjproject still installed on both the build machine and the server? They should be ignored if --with-pjproject-bundled is specified but you might want to remove them and try again.
|
George, thank you that you responded. What I was hoping
I was also very surprised that SF can happen so quickly and so consistently. With all this in completely different places.
When installed separately pjproject source code with bundled not even compiled. Of course, I uninstalled and cleaned pjproject according to this instructions: https://wiki.asterisk.org/wiki/display/AST/Building+and+Installing+pjproject
At the moment I have restore separately pjproject and asterisk 13.7. But there is a directory with configured and compiled asterisk 13.11. Happy to provide the contents of the autoconfiguration files.
|
config.log and makeopts would be useful if you can zip them up. Also, what distribution and version are you running? As Jonathan suggested, could there have been any left over or third-party modules left in the asterisk modules directory?
|
https://ruvoip.net/_other/voip/2016-09-05/makeopts_configlogs.zip
Slackware 14.1 x64 with some packages upgraded and some of libraries compiled from source. |
|
| Back to top |
|
|
gjoseph at digium.com
Guest
|
| Posted: Wed Sep 07, 2016 8:14 am Post subject: [asterisk-users] Upgrading asterisk 13.7 to 13.11. Segfaults |
|
|
On Tue, Sep 6, 2016 at 8:40 AM, Dmitriy Serov <serov.d.p@gmail.com (serov.d.p@gmail.com)> wrote:
| Quote: | 06.09.2016 17:08, George Joseph пишет:
| Quote: |
On Tue, Sep 6, 2016 at 7:58 AM, Dmitriy Serov <serov.d.p@gmail.com (serov.d.p@gmail.com)> wrote:
| Quote: | 06.09.2016 16:42, George Joseph пишет:
| Quote: |
On Tue, Sep 6, 2016 at 7:32 AM, Dmitriy Serov <serov.d.p@gmail.com (serov.d.p@gmail.com)> wrote:
| Quote: | Hello.
Several months server working on asterisk 13.7 and pjproject 2.5 (installed separately). Once a day the server crashes or hangs and is familiar sores that written watchdogs.
Yesterday I decided to upgrade to 13.11 and bundled pjproject (2.5.5). Solved all the problems with compilation I started asterisk several times and each time after 5-7 seconds was seg fault.
So I didn't get to use the new version of asterisk. And I really wanted to be able to find and remove the cause. I would be grateful for any help.
|
All 3 of the backtraces are in different pjproject places which is weird. Makes me think there's still a library mismatch somewhere. Is the separately compiled pjproject still installed on both the build machine and the server? They should be ignored if --with-pjproject-bundled is specified but you might want to remove them and try again.
|
George, thank you that you responded. What I was hoping
I was also very surprised that SF can happen so quickly and so consistently. With all this in completely different places.
When installed separately pjproject source code with bundled not even compiled. Of course, I uninstalled and cleaned pjproject according to this instructions: https://wiki.asterisk.org/wiki/display/AST/Building+and+Installing+pjproject
At the moment I have restore separately pjproject and asterisk 13.7. But there is a directory with configured and compiled asterisk 13.11. Happy to provide the contents of the autoconfiguration files.
|
config.log and makeopts would be useful if you can zip them up. Also, what distribution and version are you running? As Jonathan suggested, could there have been any left over or third-party modules left in the asterisk modules directory?
|
https://ruvoip.net/_other/voip/2016-09-05/makeopts_configlogs.zip
|
Went through the logs and configs and unfortunately nothing stands out. The only things I can suggest at this point are to check that there really are no libpj* files left in /usr/lib, no pj* header files left in /usr/include, do a system update, make sure that all res_pj* modules are loaded in modules.conf (maybe try autoload), etc.
--
George JosephDigium, Inc. | Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org |
|
| Back to top |
|
|
|
|
|
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
|
Powered by phpBB © 2001, 2005 phpBB Group
|
Search
