Sponsor: VoiceMeUp - Corporate & Wholesale VoIP Services

VoIP Mailing List Archives
Mailing list archives for the VoIP community
 SearchSearch 

[asterisk-users] Strange Issue: asterisk deleted


 
Post new topic   Reply to topic    VoIP Mailing List Archives Forum Index -> Asterisk Users
View previous topic :: View next topic  
Author Message
aatef at rocketmail.com
Guest





PostPosted: Wed Nov 26, 2014 5:38 am    Post subject: [asterisk-users] Strange Issue: asterisk deleted Reply with quote

Hi,


I am struggling with a very strange issue I have been facing for the past week;
I have a fresh install of CENTOS 5.11 and I have installed asterisk 1.8.32 form sources.
The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the "Ready" line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found


I cleaned the source and re-installed asterisk and again the same thing happened again !!!
I downloaded asterisk versions 1.4, 11, 12 and compiled them from sources and installed them (make install) and amazingly, the same thing happened to all of them: I do a "make" then "make install" and as soon as I start asterisk the process is killed and the executable removed from /usr/sbin.


I tried to look a the asterisk log files but I cannot find a single error in them.
Also if it was really deleted how did bash know that asterisk is supposed to be located in /usr/sbin/asterisk ?


I tried to copy the executable myself after compilation (everything done as root) to the /usr/sbin and again if it runs then it is deleted.


If someone can explain to me this behavior or advise me on what to check to resolve this issue, then I would be grateful.


Thank you for your help.
Regards,
Antoine Megalla
Back to top
tg at ovm-group.com
Guest





PostPosted: Wed Nov 26, 2014 11:13 am    Post subject: [asterisk-users] Strange Issue: asterisk deleted Reply with quote

Am 26.11.2014 11:37, schrieb Antoine Megalla:

Quote:
Hi,


I am struggling with a very strange issue I have been facing for the past week;
I have a fresh install of CENTOS 5.11 and I have installed asterisk 1.8.32 form sources.
The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the "Ready" line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found


I cleaned the source and re-installed asterisk and again the same thing happened again !!!
I downloaded asterisk versions 1.4, 11, 12 and compiled them from sources and installed them (make install) and amazingly, the same thing happened to all of them: I do a "make" then "make install" and as soon as I start asterisk the process is killed and the executable removed from /usr/sbin.


I tried to look a the asterisk log files but I cannot find a single error in them.
Also if it was really deleted how did bash know that asterisk is supposed to be located in /usr/sbin/asterisk ?


I tried to copy the executable myself after compilation (everything done as root) to the /usr/sbin and again if it runs then it is deleted.


If someone can explain to me this behavior or advise me on what to check to resolve this issue, then I would be grateful.




Hi,

you write "Also if it was really deleted .." - did you looked at it via "ls /usr/sbin/asterisk"?

You compiled asterisk (make / make install) as root I think. Perhaps access rights are not set properly? root is owner but you try to start the daemon as "normal" user?

You write "the process is killed". Where do you now? Did you get a message on your terminal? Did you take a look at /var/log/syslog?

Best regards
-Thorsten-
Back to top
tzafrir.cohen at xorco...
Guest





PostPosted: Wed Nov 26, 2014 11:16 am    Post subject: [asterisk-users] Strange Issue: asterisk deleted Reply with quote

On Wed, Nov 26, 2014 at 10:37:49AM +0000, Antoine Megalla wrote:
Quote:
Hi,
I am struggling with  a very strange issue I have been facing for the past week;I have a fresh install of CENTOS 5.11 and I have installed asterisk 1.8.32 form sources.The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the "Ready" line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found
I cleaned the source and re-installed asterisk and again the same thing happened again !!!I downloaded asterisk versions 1.4, 11, 12 and compiled them from sources and installed them (make install) and amazingly, the same thing happened to all of them: I do a "make" then "make install" and as soon as I start asterisk the process is killed and the executable removed from /usr/sbin.

If you suspect that something is being run from the asterisk process or
one of its children, run it under 'strace -f' and look for hints (e.g.:
'unlink') in the generated log.

--
Tzafrir Cohen
icq#16849755 jabber:tzafrir.cohen@xorcom.com
+972-50-7952406 mailto:tzafrir.cohen@xorcom.com
http://www.xorcom.com

--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
New to Asterisk? Join us for a live introductory webinar every Thurs:
http://www.asterisk.org/hello

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
Back to top
aatef at rocketmail.com
Guest





PostPosted: Wed Nov 26, 2014 3:08 pm    Post subject: [asterisk-users] Strange Issue: asterisk deleted Reply with quote

Hi,


I looked for asterisk in /usr/sbin using the commands ls and find and whereis and it was not there.


I know that the process is killed because when I start asterisk using the command asterisk -vvvvc it starts and then it exits and the word killed is wrote on the console.


Ever time I copy a new executable to /usr/sbin either using cp command or make install it gets deleted too.


Now I used the strace command on asterisk and I can clearly see at the end of the strace the line : killed by SIGKILL
This means that something or someone is actually and purposely killing asterisk but I do not know what or who is doing that also I know that I am the only user on the system.


Again any indicators to solve this very weird issue are welcomed.


Regards,
Antoine Megalla

Sent from my iPhone

On Nov 26, 2014, at 6:12 PM, Thorsten Göllner <tg@ovm-group.com (tg@ovm-group.com)> wrote:


Quote:

Am 26.11.2014 11:37, schrieb Antoine Megalla:

Quote:
Hi,


I am struggling with a very strange issue I have been facing for the past week;
I have a fresh install of CENTOS 5.11 and I have installed asterisk 1.8.32 form sources.
The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the "Ready" line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found


I cleaned the source and re-installed asterisk and again the same thing happened again !!!
I downloaded asterisk versions 1.4, 11, 12 and compiled them from sources and installed them (make install) and amazingly, the same thing happened to all of them: I do a "make" then "make install" and as soon as I start asterisk the process is killed and the executable removed from /usr/sbin.


I tried to look a the asterisk log files but I cannot find a single error in them.
Also if it was really deleted how did bash know that asterisk is supposed to be located in /usr/sbin/asterisk ?


I tried to copy the executable myself after compilation (everything done as root) to the /usr/sbin and again if it runs then it is deleted.


If someone can explain to me this behavior or advise me on what to check to resolve this issue, then I would be grateful.




Hi,

you write "Also if it was really deleted .." - did you looked at it via "ls /usr/sbin/asterisk"?

You compiled asterisk (make / make install) as root I think. Perhaps access rights are not set properly? root is owner but you try to start the daemon as "normal" user?

You write "the process is killed". Where do you now? Did you get a message on your terminal? Did you take a look at /var/log/syslog?

Best regards
-Thorsten-

Back to top
cwallace at lodgingcom...
Guest





PostPosted: Wed Nov 26, 2014 5:54 pm    Post subject: [asterisk-users] Strange Issue: asterisk deleted Reply with quote

On Wed, 26 Nov 2014 22:08:05 +0200
Antoine Megalla <aatef@rocketmail.com> wrote:

Quote:
I looked for asterisk in /usr/sbin using the commands ls and find and
whereis and it was not there.

I know that the process is killed because when I start asterisk using
the command asterisk -vvvvc it starts and then it exits and the word
killed is wrote on the console.

Ever time I copy a new executable to /usr/sbin either using cp
command or make install it gets deleted too.

Now I used the strace command on asterisk and I can clearly see at
the end of the strace the line : killed by SIGKILL This means that
something or someone is actually and purposely killing asterisk but I
do not know what or who is doing that also I know that I am the only
user on the system.

I don't know if there's any way to see where the signal comes from.
But I think it would have to be another process. Is this a hosted
machine? Could it be that your hosting provider doesn't allow
asterisk? This would be a good way to enforce that rule. Otherwise,
it could be a root kit or a virus.

Or it could be that you (or someone else) wanted to make sure asterisk
wasn't running at some point and left "while true; do killall -9
asterisk; done" running in a shell, and forgot about it.

You can list all the processes with the command "ps -ef"

And to see if anyone else (or yourself) is logged in, run "w". That
will show every individual session and where they're connected from.


--

C. Chad Wallace, B.Sc.
The Lodging Company
http://www.lodgingcompany.com/
OpenPGP Public Key ID: 0x262208A0


--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
New to Asterisk? Join us for a live introductory webinar every Thurs:
http://www.asterisk.org/hello

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
Back to top
marie at vtl.ee
Guest





PostPosted: Wed Nov 26, 2014 11:18 pm    Post subject: [asterisk-users] Strange Issue: asterisk deleted Reply with quote

On 26.11.2014, at 22:08, Antoine Megalla <aatef@rocketmail.com> wrote:
Quote:
Quote:
The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the "Ready" line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found

I looked for asterisk in /usr/sbin using the commands ls and find and whereis and it was not there.

I know that the process is killed because when I start asterisk using the command asterisk -vvvvc it starts and then it exits and the word killed is wrote on the console.

Ever time I copy a new executable to /usr/sbin either using cp command or make install it gets deleted too.

Interesting problem, I'm quite curious what the cause is.

Are you 100% sure that the asterisk your are running is in /usr/sbin? Try 'which asterisk' to see what your shell is running and/or start asterisk with a full path as /usr/sbin/asterisk -vvvvc.

You could also try renaming the binary to find out if indeed something kills Asterisk by name.

There's a tool called SystemTap which could give you information which process sent the SIGKILL:
https://sourceware.org/systemtap/
http://www.percona.com/blog/2014/07/18/systemtap-solves-phantom-mysqld-sigterm-sigkill-issue/

--

marie


--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
New to Asterisk? Join us for a live introductory webinar every Thurs:
http://www.asterisk.org/hello

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
Back to top
tg at ovm-group.com
Guest





PostPosted: Thu Nov 27, 2014 4:09 am    Post subject: [asterisk-users] Strange Issue: asterisk deleted Reply with quote

Did you take a look at /var/log/syslog?

Am 26.11.2014 21:08, schrieb Antoine Megalla:

Quote:
Hi,


I looked for asterisk in /usr/sbin using the commands ls and find and whereis and it was not there.


I know that the process is killed because when I start asterisk using the command asterisk -vvvvc it starts and then it exits and the word killed is wrote on the console.


Ever time I copy a new executable to /usr/sbin either using cp command or make install it gets deleted too.


Now I used the strace command on asterisk and I can clearly see at the end of the strace the line : killed by SIGKILL 
This means that something or someone is actually and purposely killing asterisk but I do not know what or who is doing that also I know that I am the only user on the system.


Again any indicators to solve this very weird issue are welcomed.


Regards,
Antoine Megalla

Sent from my iPhone

On Nov 26, 2014, at 6:12 PM, Thorsten Göllner <tg@ovm-group.com (tg@ovm-group.com)> wrote:


Quote:

Am 26.11.2014 11:37, schrieb Antoine Megalla:

Quote:
Hi,


I am struggling with  a very strange issue I have been facing for the past week;
I have a fresh install of CENTOS 5.11 and I have installed asterisk 1.8.32 form sources.
The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the "Ready" line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found


I cleaned the source and re-installed asterisk and again the same thing happened again !!!
I downloaded asterisk versions 1.4, 11, 12 and compiled them from sources and installed them (make install) and amazingly, the same thing happened to all of them: I do a "make" then "make install" and as soon as I start asterisk the process is killed and the executable removed from /usr/sbin.


I tried to look a the asterisk log files but I cannot find a single error in them.
Also if it was really deleted how did bash know that asterisk is supposed to be located in /usr/sbin/asterisk ?


I tried to copy the executable myself after compilation (everything done as root) to the /usr/sbin and again if it runs then it is deleted.


If someone can explain to me this behavior or advise me on what to check to resolve this issue, then I would be grateful.




Hi,

you write "Also if it was really deleted .." - did you looked at it via "ls /usr/sbin/asterisk"?

You compiled asterisk (make / make install) as root I think. Perhaps access rights are not set properly? root is owner but you try to start the daemon as "normal" user?

You write "the process is killed". Where do you now? Did you get a message on your terminal? Did you take a look at /var/log/syslog?

Back to top
aatef at rocketmail.com
Guest





PostPosted: Thu Nov 27, 2014 4:12 am    Post subject: [asterisk-users] Strange Issue: asterisk deleted Reply with quote

Yes I did, and there is nothing about asterisk in the /var/log folder


I am starting to think that the server on compromised.


Sent from my iPhone

On Nov 27, 2014, at 11:09 AM, Thorsten Göllner <tg@ovm-group.com (tg@ovm-group.com)> wrote:


Quote:
Did you take a look at /var/log/syslog?

Am 26.11.2014 21:08, schrieb Antoine Megalla:

Quote:
Hi,


I looked for asterisk in /usr/sbin using the commands ls and find and whereis and it was not there.


I know that the process is killed because when I start asterisk using the command asterisk -vvvvc it starts and then it exits and the word killed is wrote on the console.


Ever time I copy a new executable to /usr/sbin either using cp command or make install it gets deleted too.


Now I used the strace command on asterisk and I can clearly see at the end of the strace the line : killed by SIGKILL
This means that something or someone is actually and purposely killing asterisk but I do not know what or who is doing that also I know that I am the only user on the system.


Again any indicators to solve this very weird issue are welcomed.


Regards,
Antoine Megalla

Sent from my iPhone

On Nov 26, 2014, at 6:12 PM, Thorsten Göllner <tg@ovm-group.com (tg@ovm-group.com)> wrote:


Quote:

Am 26.11.2014 11:37, schrieb Antoine Megalla:

Quote:
Hi,


I am struggling with a very strange issue I have been facing for the past week;
I have a fresh install of CENTOS 5.11 and I have installed asterisk 1.8.32 form sources.
The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the "Ready" line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found


I cleaned the source and re-installed asterisk and again the same thing happened again !!!
I downloaded asterisk versions 1.4, 11, 12 and compiled them from sources and installed them (make install) and amazingly, the same thing happened to all of them: I do a "make" then "make install" and as soon as I start asterisk the process is killed and the executable removed from /usr/sbin.


I tried to look a the asterisk log files but I cannot find a single error in them.
Also if it was really deleted how did bash know that asterisk is supposed to be located in /usr/sbin/asterisk ?


I tried to copy the executable myself after compilation (everything done as root) to the /usr/sbin and again if it runs then it is deleted.


If someone can explain to me this behavior or advise me on what to check to resolve this issue, then I would be grateful.




Hi,

you write "Also if it was really deleted .." - did you looked at it via "ls /usr/sbin/asterisk"?

You compiled asterisk (make / make install) as root I think. Perhaps access rights are not set properly? root is owner but you try to start the daemon as "normal" user?

You write "the process is killed". Where do you now? Did you get a message on your terminal? Did you take a look at /var/log/syslog?



Back to top
asterisk_list at earth...
Guest





PostPosted: Thu Nov 27, 2014 5:06 am    Post subject: [asterisk-users] Strange Issue: asterisk deleted Reply with quote

On Wednesday 26 Nov 2014, Antoine Megalla wrote:
Quote:
Hi,

I looked for asterisk in /usr/sbin using the commands ls and find and
whereis and it was not there.

I know that the process is killed because when I start asterisk using the
command asterisk -vvvvc it starts and then it exits and the word killed is
wrote on the console.

Ever time I copy a new executable to /usr/sbin either using cp command or
make install it gets deleted too.

Now I used the strace command on asterisk and I can clearly see at the end
of the strace the line : killed by SIGKILL This means that something or
someone is actually and purposely killing asterisk but I do not know what
or who is doing that also I know that I am the only user on the system.

Again any indicators to solve this very weird issue are welcomed.

It sounds as though your server might have been compromised.

Get another machine of the same bit architecture and perform a fresh install
of exactly the same OS as your Asterisk box on that. Install busybox too
(it's usually there anyway, as it's required for building the initial RAMdisks
used by most distros for booting). Using a USB stick (preferrably one that
can be set read-only), copy at least the `ls`, `ps`, `netstat`, `w`,
`lsattr`, `md5sum`, `cat`, `diff` and `busybox` binaries over (to somewhere
that isn't /usr/bin/). Use both the existing installed and the newly-copied
md5sum and diff to check each system binary against the known-good ones. You
can use busybox to replicate commands you haven't copied (but note that
busybox versions are rather cut-down as compared to the GNU tools you know and
love. Come to think of it, they're cut-down as compared to the BSD tools
everyone replaces with GNU versions once they have a C compiler up and
running).

Compare /etc/inittab between the two machines.

Many rootkits mess with ext[2-4]fs attributes, presumably to stop you
overwriting their overwritten system binaries; so use a known good lsattr to
check the attributes of everything in /bin/, /sbin/, /usr/bin/ and /usr/sbin/
-- watch out for anything set immutable.


Getting rid of the compromise fortunately is reasonably easy, especially if
your /home folder is on its own partition. Just ignore that partition during
reinstallation, edit your /etc/fstab afterwards and reboot -- your original
/home will be preserved intact. If not, use systemrescuecd or something
similar to boot a known-good system. Use mv to rename /home to a new name.
Shrink a disk partition and create a new small partition. Use that for your
/home during the reinstall. Then again edit /etc/fstab, unmount /home, mv
your old /home back to /home and reboot.

--
AJS

Note: Originating address only accepts e-mail from list! If replying off-
list, change address to asterisk1list at earthshod dot co dot uk .

--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
New to Asterisk? Join us for a live introductory webinar every Thurs:
http://www.asterisk.org/hello

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
Back to top
aatef at rocketmail.com
Guest





PostPosted: Thu Nov 27, 2014 1:30 pm    Post subject: [asterisk-users] Strange Issue: asterisk deleted Reply with quote

Hi

Thank you for your support.
The server is actually compromised, I discovered that after making a deep trace using the audit daemon and looking for the kill signal (SIGKILL) that terminates asterisk.
I discovered that there is an executable with a random name in the /boot folder that is killing and deleting asterisk !!!

This executable is launched by a service in /etc/rc.d/ with the same random name.
When I stopped this service, a new service was created with another different random name and it too is killing and deleting asterisk.
This was the evidence i needed to be convinced that the server has a virus and is compromised.

The good thing is that this is a fresh install and hence there are no sensitive data or a lot of work done on it so i will reinstall the OS and start over. The bad thing is that I spent more than 4 days trying to understand what was going on.

Again, thank you for your support.

Regards,
Antoine Megalla

Sent from my iPhone

On Nov 27, 2014, at 8:00 PM, asterisk-users-request@lists.digium.com wrote:

Quote:
Send asterisk-users mailing list submissions to
asterisk-users@lists.digium.com

To subscribe or unsubscribe via the World Wide Web, visit
http://lists.digium.com/mailman/listinfo/asterisk-users
or, via email, send a message with subject or body 'help' to
asterisk-users-request@lists.digium.com

You can reach the person managing the list at
asterisk-users-owner@lists.digium.com

When replying, please edit your Subject line so it is more specific
than "Re: Contents of asterisk-users digest..."


Today's Topics:

1. Re: Strange Issue: asterisk deleted (Antoine Megalla)
2. Re: High resident memory with 11.14.0 ? (James Lamanna)
3. Re: Strange Issue: asterisk deleted (Chad Wallace)
4. Re: Strange Issue: asterisk deleted (Marie Fischer)
5. Re: SIP call drops after 32 seconds, but only when....
(Marie Fischer)
6. Re: SIP call drops after 32 seconds, but only when....
(Amit Patkar)
7. Re: Strange Issue: asterisk deleted (Thorsten G?llner)
8. Re: Strange Issue: asterisk deleted (Antoine Megalla)
9. Re: Strange Issue: asterisk deleted (A J Stiles)


----------------------------------------------------------------------

Message: 1
Date: Wed, 26 Nov 2014 22:08:05 +0200
From: Antoine Megalla <aatef@rocketmail.com>
To: Thorsten G?llner <tg@ovm-group.com>
Cc: Asterisk Users Mailing List - Non-Commercial Discussion
<asterisk-users@lists.digium.com>
Subject: Re: [asterisk-users] Strange Issue: asterisk deleted
Message-ID: <7D5A57FB-657C-439B-9DCB-2790AE9C920D@rocketmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

I looked for asterisk in /usr/sbin using the commands ls and find and whereis and it was not there.

I know that the process is killed because when I start asterisk using the command asterisk -vvvvc it starts and then it exits and the word killed is wrote on the console.

Ever time I copy a new executable to /usr/sbin either using cp command or make install it gets deleted too.

Now I used the strace command on asterisk and I can clearly see at the end of the strace the line : killed by SIGKILL
This means that something or someone is actually and purposely killing asterisk but I do not know what or who is doing that also I know that I am the only user on the system.

Again any indicators to solve this very weird issue are welcomed.

Regards,
Antoine Megalla

Sent from my iPhone

On Nov 26, 2014, at 6:12 PM, Thorsten G?llner <tg@ovm-group.com> wrote:

Quote:

Am 26.11.2014 11:37, schrieb Antoine Megalla:
Quote:
Hi,

I am struggling with a very strange issue I have been facing for the past week;
I have a fresh install of CENTOS 5.11 and I have installed asterisk 1.8.32 form sources.
The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the "Ready" line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found

I cleaned the source and re-installed asterisk and again the same thing happened again !!!
I downloaded asterisk versions 1.4, 11, 12 and compiled them from sources and installed them (make install) and amazingly, the same thing happened to all of them: I do a "make" then "make install" and as soon as I start asterisk the process is killed and the executable removed from /usr/sbin.

I tried to look a the asterisk log files but I cannot find a single error in them.
Also if it was really deleted how did bash know that asterisk is supposed to be located in /usr/sbin/asterisk ?

I tried to copy the executable myself after compilation (everything done as root) to the /usr/sbin and again if it runs then it is deleted.

If someone can explain to me this behavior or advise me on what to check to resolve this issue, then I would be grateful.

Hi,

you write "Also if it was really deleted .." - did you looked at it via "ls /usr/sbin/asterisk"?

You compiled asterisk (make / make install) as root I think. Perhaps access rights are not set properly? root is owner but you try to start the daemon as "normal" user?

You write "the process is killed". Where do you now? Did you get a message on your terminal? Did you take a look at /var/log/syslog?

Best regards
-Thorsten-
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20141126/d64c9a5b/attachment-0001.html>

------------------------------

Message: 2
Date: Wed, 26 Nov 2014 15:20:06 -0500
From: James Lamanna <jlamanna@gmail.com>
To: Asterisk Users Mailing List - Non-Commercial Discussion
<asterisk-users@lists.digium.com>
Subject: Re: [asterisk-users] High resident memory with 11.14.0 ?
Message-ID:
<CADScKLzHeEiZL51Oi=6bc6VCgOoqeRnuOiriw10SP+YC5vFFrw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

On Tue, Nov 25, 2014 at 10:21 AM, James Lamanna <jlamanna@gmail.com> wrote:

Quote:

On Tue, Nov 25, 2014 at 8:14 AM, Matthew Jordan <mjordan@digium.com>
wrote:

Quote:
On Mon, Nov 24, 2014 at 2:12 PM, James Lamanna <jlamanna@gmail.com>
wrote:
Quote:
Also, how big does the cache in frame.c grow to?
I've recompiled with MALLOC_DEBUG on that server:

asterisk -rx "memory show summary"

....
1780466242 bytes (1780181594 cache) in 2352909 allocations in file
frame.c
...

Seems like a ridiculous cache.

I'm not going to respond to your new thread, since it is the same
discussion as this one.

The frame cache is a per-thread local cache of frames that prevents
having to re-allocate frames as they pass through Asterisk. Clearly,
something is abusing it.

I think you'll need to provide some more information on how you're
producing this situation. Specifically:
* Channel technologies involved, and the formats on the channels
* Dialplan that reproduces the problem

Are you using any non-core dialplan applications or channel drivers?
This PBX has about 100 registered SIP clients, along with 23 PRI channels,
2 inbound/outbound SIP trunks and around 100 IAXModems registered to it. It
primarily handles faxing.
I am not using any non-standard channel drivers. I am using the T.38
gateway funcionality.

The jist of the dialplan is this: (example of the PRI and a SIP trunk,
inbound)

[pri-in]
exten => _X.,1,Set(__FROM_DID=${EXTEN})
exten => _X.,n,Set(FAX_IDX=700)
exten => _X.,n,Set(MAX_IDX=719)
exten => _X.,n,Goto(dial-hylafax,s,1)

[sip-trunk-in]
exten => _X.,1(normal),Set(__FROM_DID=${EXTEN})
exten => _X.,n,Set(FAX_IDX=950)
exten => _X.,n,Set(MAX_IDX=959)
exten => _X.,n,Set(FAXOPT(gateway)=yes)
exten => _X.,n,Goto(dial-hylafax,s,1)

[dial-hylafax]
exten => s,1,GotoIf($["${FROM_DID:0:1}" = "1"]?prune:cont)
exten => s,n(prune),Set(__FROM_DID=${FROM_DID:1})
exten => s,n(cont),GotoIf($[${FAX_IDX} <= ${MAX_IDX}]?tryfax:nofax)
exten => s,n(tryfax),Set(STATE=${DEVICE_STATE(Custom:iaxmodem${FAX_IDX})})
exten => s,n,NoOp(${STATE})
exten => s,n,Set(DEVICE_STATE(Custom:iaxmodem${FAX_IDX})=INUSE)
exten => s,n,Dial(IAX2/iaxmodem${FAX_IDX}/${FROM_DID},60,g)
exten => s,n,Goto(s-${DIALSTATUS},1)
exten => s,n(nofax),Playtones(busy)
exten => s,n,NoOp(NO MODEMS AVAILABLE)
exten => s,n,Wait(20)
exten => s,n,Hangup()
exten => s-ANSWER,1,NoOp(IAXMODEM HANGUP)
exten => s-ANSWER,n,Set(DEVICE_STATE(Custom:iaxmodem${FAX_IDX})=NOT_INUSE)
exten => s-ANSWER,n,Hangup()
exten => _s-.,1,Set(FAX_IDX=${MATH(1+${FAX_IDX},i)})
exten => _s-.,n,Goto(s,1)
exten => h,1,Set(DEVICE_STATE(Custom:iaxmodem${FAX_IDX})=NOT_INUSE)

The current state requires me to restart Asterisk almost every day.
I'm also seeing this on a completely different machine after upgrading
from Asterisk10 to 11.
I'm wondering if this is a problem in the SLIN converter?
I do use SLIN with iaxmodem.

-- James
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20141126/9deca244/attachment-0001.html>

------------------------------

Message: 3
Date: Wed, 26 Nov 2014 14:54:27 -0800
From: Chad Wallace <cwallace@lodgingcompany.com>
To: asterisk-users@lists.digium.com
Subject: Re: [asterisk-users] Strange Issue: asterisk deleted
Message-ID: <20141126145427.4819c67b@ws78.int.tlc>
Content-Type: text/plain; charset=US-ASCII

On Wed, 26 Nov 2014 22:08:05 +0200
Antoine Megalla <aatef@rocketmail.com> wrote:

Quote:
I looked for asterisk in /usr/sbin using the commands ls and find and
whereis and it was not there.

I know that the process is killed because when I start asterisk using
the command asterisk -vvvvc it starts and then it exits and the word
killed is wrote on the console.

Ever time I copy a new executable to /usr/sbin either using cp
command or make install it gets deleted too.

Now I used the strace command on asterisk and I can clearly see at
the end of the strace the line : killed by SIGKILL This means that
something or someone is actually and purposely killing asterisk but I
do not know what or who is doing that also I know that I am the only
user on the system.

I don't know if there's any way to see where the signal comes from.
But I think it would have to be another process. Is this a hosted
machine? Could it be that your hosting provider doesn't allow
asterisk? This would be a good way to enforce that rule. Otherwise,
it could be a root kit or a virus.

Or it could be that you (or someone else) wanted to make sure asterisk
wasn't running at some point and left "while true; do killall -9
asterisk; done" running in a shell, and forgot about it.

You can list all the processes with the command "ps -ef"

And to see if anyone else (or yourself) is logged in, run "w". That
will show every individual session and where they're connected from.


--

C. Chad Wallace, B.Sc.
The Lodging Company
http://www.lodgingcompany.com/
OpenPGP Public Key ID: 0x262208A0




------------------------------

Message: 4
Date: Thu, 27 Nov 2014 06:18:19 +0200
From: Marie Fischer <marie@vtl.ee>
To: Asterisk Users Mailing List - Non-Commercial Discussion
<asterisk-users@lists.digium.com>
Subject: Re: [asterisk-users] Strange Issue: asterisk deleted
Message-ID: <7442CB28-9F60-480D-9E8F-D139727DBF76@vtl.ee>
Content-Type: text/plain; charset=us-ascii


On 26.11.2014, at 22:08, Antoine Megalla <aatef@rocketmail.com> wrote:
Quote:
Quote:
The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the "Ready" line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found
I looked for asterisk in /usr/sbin using the commands ls and find and whereis and it was not there.

I know that the process is killed because when I start asterisk using the command asterisk -vvvvc it starts and then it exits and the word killed is wrote on the console.

Ever time I copy a new executable to /usr/sbin either using cp command or make install it gets deleted too.

Interesting problem, I'm quite curious what the cause is.

Are you 100% sure that the asterisk your are running is in /usr/sbin? Try 'which asterisk' to see what your shell is running and/or start asterisk with a full path as /usr/sbin/asterisk -vvvvc.

You could also try renaming the binary to find out if indeed something kills Asterisk by name.

There's a tool called SystemTap which could give you information which process sent the SIGKILL:
https://sourceware.org/systemtap/
http://www.percona.com/blog/2014/07/18/systemtap-solves-phantom-mysqld-sigterm-sigkill-issue/

--

marie




------------------------------

Message: 5
Date: Thu, 27 Nov 2014 06:31:37 +0200
From: Marie Fischer <marie@vtl.ee>
To: Asterisk Users Mailing List - Non-Commercial Discussion
<asterisk-users@lists.digium.com>
Subject: Re: [asterisk-users] SIP call drops after 32 seconds, but
only when....
Message-ID: <CF4F37ED-8DDF-43DC-9E9C-79A292E86FAE@vtl.ee>
Content-Type: text/plain; charset=windows-1252

On 22.11.2014, at 13:40, Yves A. <yves030@gmx.de> wrote:
Quote:
I have a really strange problem which is driving me crazy for days now.

If I register my asterisk (tried all versions from 1.6 up to 13.x) with one sip registrar,
everything works... calls go out and call come in... no 32 seconds limit.

but as soon as I configure another sip registration on another server, outgoing
calls drop after 32 seconds.

Do a 'sip set debug on' and see what they (Asterisk and the registrar) are talking about just before the call drops.

--

marie




------------------------------

Message: 6
Date: Thu, 27 Nov 2014 10:49:23 +0530
From: Amit Patkar <amit@avhan.com>
To: asterisk-users@lists.digium.com
Subject: Re: [asterisk-users] SIP call drops after 32 seconds, but
only when....
Message-ID: <5476B45B.4020400@avhan.com>
Content-Type: text/plain; charset="iso-8859-1"; Format="flowed"

Call drop after 30+sec happens if RTP is not received by asterisk for 30
seconds (RTP Timeout).
You should look for media IP address in SDP. If there is firewall, apart
from port UDP/5060, you also need to open port UDP/10000-UDP/20000
(standard RTP ports)
You should try with RTP debug. It should show bidirectional traffic. If
not, you surely have an issue with media IP or ports.

*Thanks & Regards,*
Amit Patkar


On 11/27/2014 10:01 AM, Marie Fischer wrote:
Quote:
On 22.11.2014, at 13:40, Yves A. <yves030@gmx.de> wrote:
Quote:
I have a really strange problem which is driving me crazy for days now.

If I register my asterisk (tried all versions from 1.6 up to 13.x) with one sip registrar,
everything works... calls go out and call come in... no 32 seconds limit.

but as soon as I configure another sip registration on another server, outgoing
calls drop after 32 seconds.
Do a 'sip set debug on' and see what they (Asterisk and the registrar) are talking about just before the call drops.


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20141127/7b0ab3fa/attachment-0001.html>

------------------------------

Message: 7
Date: Thu, 27 Nov 2014 10:09:23 +0100
From: Thorsten G?llner <tg@ovm-group.com>
To: Antoine Megalla <aatef@rocketmail.com>
Cc: Asterisk Users Mailing List - Non-Commercial Discussion
<asterisk-users@lists.digium.com>
Subject: Re: [asterisk-users] Strange Issue: asterisk deleted
Message-ID: <5476EA43.1090008@ovm-group.com>
Content-Type: text/plain; charset="utf-8"

Did you take a look at /var/log/syslog?

Am 26.11.2014 21:08, schrieb Antoine Megalla:
Quote:
Hi,

I looked for asterisk in /usr/sbin using the commands ls and find and
whereis and it was not there.

I know that the process is killed because when I start asterisk using
the command asterisk -vvvvc it starts and then it exits and the word
killed is wrote on the console.

Ever time I copy a new executable to /usr/sbin either using cp command
or make install it gets deleted too.

Now I used the strace command on asterisk and I can clearly see at the
end of the strace the line : killed by SIGKILL
This means that something or someone is actually and purposely killing
asterisk but I do not know what or who is doing that also I know that
I am the only user on the system.

Again any indicators to solve this very weird issue are welcomed.

Regards,
Antoine Megalla

Sent from my iPhone

On Nov 26, 2014, at 6:12 PM, Thorsten G?llner <tg@ovm-group.com
<mailto:tg@ovm-group.com>> wrote:

Quote:

Am 26.11.2014 11:37, schrieb Antoine Megalla:
Quote:
Hi,

I am struggling with a very strange issue I have been facing for
the past week;
I have a fresh install of CENTOS 5.11 and I have installed asterisk
1.8.32 form sources.
The asterisk installation went fine but as soon as I start asterisk
executable it loads everything and then after the "Ready" line the
process gets killed and when I try to run it again i get:
/usr/sbin/asterisk : command not found

I cleaned the source and re-installed asterisk and again the same
thing happened again !!!
I downloaded asterisk versions 1.4, 11, 12 and compiled them from
sources and installed them (make install) and amazingly, the same
thing happened to all of them: I do a "make" then "make install" and
as soon as I start asterisk the process is killed and the executable
removed from /usr/sbin.

I tried to look a the asterisk log files but I cannot find a single
error in them.
Also if it was really deleted how did bash know that asterisk is
supposed to be located in /usr/sbin/asterisk ?

I tried to copy the executable myself after compilation (everything
done as root) to the /usr/sbin and again if it runs then it is deleted.

If someone can explain to me this behavior or advise me on what to
check to resolve this issue, then I would be grateful.

Hi,

you write "Also if it was really deleted .." - did you looked at it
via "ls /usr/sbin/asterisk"?

You compiled asterisk (make / make install) as root I think. Perhaps
access rights are not set properly? root is owner but you try to
start the daemon as "normal" user?

You write "the process is killed". Where do you now? Did you get a
message on your terminal? Did you take a look at /var/log/syslog?

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20141127/ddec7744/attachment-0001.html>

------------------------------

Message: 8
Date: Thu, 27 Nov 2014 11:11:36 +0200
From: Antoine Megalla <aatef@rocketmail.com>
To: Thorsten G?llner <tg@ovm-group.com>
Cc: Asterisk Users Mailing List - Non-Commercial Discussion
<asterisk-users@lists.digium.com>
Subject: Re: [asterisk-users] Strange Issue: asterisk deleted
Message-ID: <FF950549-B06C-4E2C-9413-AA8FAFFB2E6A@rocketmail.com>
Content-Type: text/plain; charset="utf-8"

Yes I did, and there is nothing about asterisk in the /var/log folder

I am starting to think that the server on compromised.


Sent from my iPhone

On Nov 27, 2014, at 11:09 AM, Thorsten G?llner <tg@ovm-group.com> wrote:

Quote:
Did you take a look at /var/log/syslog?

Am 26.11.2014 21:08, schrieb Antoine Megalla:
Quote:
Hi,

I looked for asterisk in /usr/sbin using the commands ls and find and whereis and it was not there.

I know that the process is killed because when I start asterisk using the command asterisk -vvvvc it starts and then it exits and the word killed is wrote on the console.

Ever time I copy a new executable to /usr/sbin either using cp command or make install it gets deleted too.

Now I used the strace command on asterisk and I can clearly see at the end of the strace the line : killed by SIGKILL
This means that something or someone is actually and purposely killing asterisk but I do not know what or who is doing that also I know that I am the only user on the system.

Again any indicators to solve this very weird issue are welcomed.

Regards,
Antoine Megalla

Sent from my iPhone

On Nov 26, 2014, at 6:12 PM, Thorsten G?llner <tg@ovm-group.com> wrote:

Quote:

Am 26.11.2014 11:37, schrieb Antoine Megalla:
Quote:
Hi,

I am struggling with a very strange issue I have been facing for the past week;
I have a fresh install of CENTOS 5.11 and I have installed asterisk 1.8.32 form sources.
The asterisk installation went fine but as soon as I start asterisk executable it loads everything and then after the "Ready" line the process gets killed and when I try to run it again i get: /usr/sbin/asterisk : command not found

I cleaned the source and re-installed asterisk and again the same thing happened again !!!
I downloaded asterisk versions 1.4, 11, 12 and compiled them from sources and installed them (make install) and amazingly, the same thing happened to all of them: I do a "make" then "make install" and as soon as I start asterisk the process is killed and the executable removed from /usr/sbin.

I tried to look a the asterisk log files but I cannot find a single error in them.
Also if it was really deleted how did bash know that asterisk is supposed to be located in /usr/sbin/asterisk ?

I tried to copy the executable myself after compilation (everything done as root) to the /usr/sbin and again if it runs then it is deleted.

If someone can explain to me this behavior or advise me on what to check to resolve this issue, then I would be grateful.

Hi,

you write "Also if it was really deleted .." - did you looked at it via "ls /usr/sbin/asterisk"?

You compiled asterisk (make / make install) as root I think. Perhaps access rights are not set properly? root is owner but you try to start the daemon as "normal" user?

You write "the process is killed". Where do you now? Did you get a message on your terminal? Did you take a look at /var/log/syslog?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-users/attachments/20141127/7903c187/attachment-0001.html>

------------------------------

Message: 9
Date: Thu, 27 Nov 2014 10:05:44 +0000
From: A J Stiles <asterisk_list@earthshod.co.uk>
To: "Asterisk Users Mailing List - Non-Commercial Discussion"
<asterisk-users@lists.digium.com>
Subject: Re: [asterisk-users] Strange Issue: asterisk deleted
Message-ID: <201411271005.44407.asterisk_list@earthshod.co.uk>
Content-Type: Text/Plain; charset="iso-8859-6"

On Wednesday 26 Nov 2014, Antoine Megalla wrote:
Quote:
Hi,

I looked for asterisk in /usr/sbin using the commands ls and find and
whereis and it was not there.

I know that the process is killed because when I start asterisk using the
command asterisk -vvvvc it starts and then it exits and the word killed is
wrote on the console.

Ever time I copy a new executable to /usr/sbin either using cp command or
make install it gets deleted too.

Now I used the strace command on asterisk and I can clearly see at the end
of the strace the line : killed by SIGKILL This means that something or
someone is actually and purposely killing asterisk but I do not know what
or who is doing that also I know that I am the only user on the system.

Again any indicators to solve this very weird issue are welcomed.

It sounds as though your server might have been compromised.

Get another machine of the same bit architecture and perform a fresh install
of exactly the same OS as your Asterisk box on that. Install busybox too
(it's usually there anyway, as it's required for building the initial RAMdisks
used by most distros for booting). Using a USB stick (preferrably one that
can be set read-only), copy at least the `ls`, `ps`, `netstat`, `w`,
`lsattr`, `md5sum`, `cat`, `diff` and `busybox` binaries over (to somewhere
that isn't /usr/bin/). Use both the existing installed and the newly-copied
md5sum and diff to check each system binary against the known-good ones. You
can use busybox to replicate commands you haven't copied (but note that
busybox versions are rather cut-down as compared to the GNU tools you know and
love. Come to think of it, they're cut-down as compared to the BSD tools
everyone replaces with GNU versions once they have a C compiler up and
running).

Compare /etc/inittab between the two machines.

Many rootkits mess with ext[2-4]fs attributes, presumably to stop you
overwriting their overwritten system binaries; so use a known good lsattr to
check the attributes of everything in /bin/, /sbin/, /usr/bin/ and /usr/sbin/
-- watch out for anything set immutable.


Getting rid of the compromise fortunately is reasonably easy, especially if
your /home folder is on its own partition. Just ignore that partition during
reinstallation, edit your /etc/fstab afterwards and reboot -- your original
/home will be preserved intact. If not, use systemrescuecd or something
similar to boot a known-good system. Use mv to rename /home to a new name.
Shrink a disk partition and create a new small partition. Use that for your
/home during the reinstall. Then again edit /etc/fstab, unmount /home, mv
your old /home back to /home and reboot.

--
AJS

Note: Originating address only accepts e-mail from list! If replying off-
list, change address to asterisk1list at earthshod dot co dot uk .



------------------------------

_______________________________________________
--Bandwidth and Colocation Provided by http://www.api-digital.com--

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users

End of asterisk-users Digest, Vol 124, Issue 29
***********************************************

--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
New to Asterisk? Join us for a live introductory webinar every Thurs:
http://www.asterisk.org/hello

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
Back to top
jnovack at stromberg-c...
Guest





PostPosted: Thu Nov 27, 2014 2:07 pm    Post subject: [asterisk-users] Strange Issue: asterisk deleted Reply with quote

Question remains, how was it compromised?
In the original install ?
A "fresh" install perhaps from another source?

Best you determine HOW before spending more time going down another rabbit hole!

John Novack

Antoine Megalla wrote:

Quote:
Quote:
Hi

Thank you for your support.
The server is actually compromised, I discovered that after making a deep trace using the audit daemon and looking for the kill signal (SIGKILL) that terminates asterisk.
<snipped to please the mailing list >
Quote:
--

Dog is my Co-pilot
Back to top
tnelson at rockbochs.com
Guest





PostPosted: Tue Dec 02, 2014 1:08 am    Post subject: [asterisk-users] Strange Issue: asterisk deleted Reply with quote

----- Original Message -----
Quote:
Hi

Thank you for your support.
The server is actually compromised, I discovered that after making a
deep trace using the audit daemon and looking for the kill signal
(SIGKILL) that terminates asterisk.
I discovered that there is an executable with a random name in the
/boot folder that is killing and deleting asterisk !!!

This executable is launched by a service in /etc/rc.d/ with the same
random name.
When I stopped this service, a new service was created with another
different random name and it too is killing and deleting asterisk.
This was the evidence i needed to be convinced that the server has a
virus and is compromised.

The good thing is that this is a fresh install and hence there are no
sensitive data or a lot of work done on it so i will reinstall the
OS and start over. The bad thing is that I spent more than 4 days
trying to understand what was going on.


Very interesting. Any ideas on how the system was compromised? Are any other daemons being actively replaced, or just Asterisk? I did hear of a similar issue to the one you describe (also on an Asterisk box) via a third party recently, but don't have any real specifics other than it being Asterisk 1.4.x on Debian (5 or 6), running on a local LAN, no outside access. Curious if there are any commonalities to the two compromised systems.

--Tim

--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
New to Asterisk? Join us for a live introductory webinar every Thurs:
http://www.asterisk.org/hello

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users
Back to top
Display posts from previous:   
Post new topic   Reply to topic    VoIP Mailing List Archives Forum Index -> Asterisk Users All times are GMT - 5 Hours
Page 1 of 1

 
Jump to:  
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum


Powered by phpBB © 2001, 2005 phpBB Group

VoiceMeUp - Corporate & Wholesale VoIP Services