VoIP Mailing List Archives :: View topic - [asterisk-users] 786 000 files limit Centos 7

Sponsor: VoiceMeUp - Corporate & Wholesale VoIP Services

VoIP Mailing List Archives
Mailing list archives for the VoIP community

[asterisk-users] 786 000 files limit Centos 7 - Asterisk

VoIP Mailing List Archives Forum Index -> Asterisk Users

View previous topic :: View next topic

Author

Message

viljoens at verishare....
Guest

Posted: Wed Aug 12, 2015 2:44 am Post subject: [asterisk-users] 786 000 files limit Centos 7 - Asterisk

Hi Markus

Thanks for the reply, I have set those files previously as well...

It seems the problem for me is on my Centos 7 box that the Asterisk binary
does not "know" that these are the limits, and imposes a 1024 open file
limit count for some reason.

It seems using prlimit to force the running binary to use a higher file
limit is working - but it has to be done manually each time the asterisk
process is started, the prlimit cronjob I set up to force the file limit
higher does not seem to work - you have to physically type in

/usr/bin/prlimit --pid `pidof asterisk` --nofile=786000:786000

and running that from a script (As root) doesn't seem to work.

Anyway, thanks for the advice and taking the time to reply.

Kind regards,

Message: 11
Date: Tue, 11 Aug 2015 18:29:01 +0200
From: Markus Weiler <markus_weiler@mailworks.org>
To: asterisk-users@lists.digium.com
Subject: Re: [asterisk-users] 786 000 files limit Centos 7 - Asterisk
keep complaining
Message-ID: <55CA22CD.4060400@mailworks.org>
Content-Type: text/plain; charset=windows-1252; format=flowed

Hi Stefan,

we ran into a similar problem using Debian.

There we are able to check the current limits using:

pidof asterisk -> 23351
cat /proc/23351/limits

Output:
Limit Soft Limit Hard Limit Units
Max open files 1024 1024 files

I think that in the end

/etc/security/limits.conf

* hard nofile 500000
* soft nofile 500000
root hard nofile 500000
root soft nofile 500000

did the trick. We also tried

vi /etc/sysctl.conf
fs.file-max = 500000

not sure what the solution in the end was. But I remember rebooting was
important.

Markus

Am 11.08.2015 um 11:00 schrieb Stefan Viljoen:

Quote:

Anybody else ran into this?

No, but I would ask myself why so many file descriptors are being used.
It sounds like you have a file descriptor leak (not being closed when
finished with).

Hi Tony

Thanks for replying.

I suspected something like that, though repeatedly running

lsof | wc -l

Always stays quite low - 100 000 open files, which is still 8 times less
than the system maximum as confirmed by running ulimit -n

I also note that this number will increase to about 125 000 but never go
higher than that, then, as calls hang up, decreate again - during times

when

Quote:

the CLI is spammed with 100s of "broken pipe" errors due to insuffiecient
file descriptors, this number never reaches beyond 125 000 out of the
available 800 000 open files.

Quote:

You might also want to look at the output of lsof (or at least some of

it)

Quote:

to see what all these file descriptors are pointing to, and whether it is
indeed Asterisk that is consuming them.

If I grep by asterisk on the output of lsof the few thousand lines I have
looked at all seem to indicate legitimate uses - there are at least two
files for each conversation in progress (I assume for inward and outward
RTP) plus one for each file being mixmonitored (which also seems logical)
and also number-of-active-calls connections to res_timing_dahdi - which

all

Quote:

looks correct...

Quote:

If it is Asterisk, it's quite possible, even probable, that such a leak
has been found and fixed, even in the 1.8 series. 1.8.11.0 is rather old

Quote:

the latest is 1.8.32.3, so it would be best to update to that version and
see if the problem persists.

Ok, I will have to consider that. The thing is the problem is not

consistent

Quote:

- I can (for example) run 60 calls, with no problems and no reported
failures in opening files, then calls will -decrease- to about 40 and then
later spike to 70, but around 50 calls I get the errors coming up

thousands

Quote:

of times in the CLI, then suddenly stop as the calls -increase- which
doesn't make sense. But this kind of behaviour does seem consistent with a
possible leak.

SOMETHING NEW

I have now ran

/usr/bin/prlimit --pid `pidof asterisk`

and I have noticed that even though I have 800 000 files specified, the
ACTUAL limit in place on Asterisk for numbers of files is only 1024?!

# prlimit --pid `pidof asterisk`
RESOURCE DESCRIPTION SOFT HARD UNITS
AS address space limit unlimited unlimited bytes
CORE max core file size unlimited unlimited blocks
CPU CPU time unlimited unlimited seconds
DATA max data size unlimited unlimited bytes
FSIZE max file size unlimited unlimited blocks
LOCKS max number of file locks held unlimited unlimited
MEMLOCK max locked-in-memory address space 65536 65536 bytes
MSGQUEUE max bytes in POSIX mqueues 819200 819200 bytes
NICE max nice prio allowed to raise 0 0
NOFILE max number of open files 1024 4096
NPROC max number of processes 30861 30861
RSS max resident set size unlimited unlimited pages
RTPRIO max real-time priority 0 0
RTTIME timeout for real-time tasks unlimited unlimited

microsecs

Quote:

SIGPENDING max number of pending signals 30861 30861
STACK max stack size 8388608 unlimited bytes

Accordingly I have put this into a cronjob ran each minute:

prlimit --pid `pidof asterisk` --nofile=786000:786000

to try and force the running binary to keep a high file limit (sources say
to keep it less than the ACTUAL system file limit, in my case 800 000

files)

Quote:

on the live Asterisk process.

I'll see if this maybe helps - the above runs via cron each minute.

So it appears for some reason somehow the live running asterisk process
"loses track" of how many open files it may have, or when it starts it
somehow does not start with the correct number of maximum open files, as

set

Quote:

in the system / kernel config?

Anyway, thank you for replying, I'll monitor this new "Cronjob fixup" I'm
trying and see if it helps.

No wonder it is complaining about running out of file handles if it

ACTUALLY

Quote:

was only using 1024!

Kind regards

Stefan

------------------------------

Message: 12
Date: Tue, 11 Aug 2015 16:50:29 +0000 (UTC)
From: tony@softins.co.uk (Tony Mountifield)
To: asterisk-users@lists.digium.com
Subject: Re: [asterisk-users] 786 000 files limit Centos 7 - Asterisk
keep complaining
Message-ID: <assp.06651285b6.mqd94l$qb1$1@softins.softins.co.uk>

In article <002b01d0d414$36af31b0$a40d9510$@verishare.co.za>,
Stefan Viljoen <viljoens@verishare.co.za> wrote:

Quote:

Anybody else ran into this?

Quote:

No, but I would ask myself why so many file descriptors are being used.
It sounds like you have a file descriptor leak (not being closed when
finished with).

Quote:

From what you said below, the above is probably not relevant...

Quote:

SOMETHING NEW

I have now ran

/usr/bin/prlimit --pid `pidof asterisk`

and I have noticed that even though I have 800 000 files specified, the
ACTUAL limit in place on Asterisk for numbers of files is only 1024?!

Yes, this is likely. Have a look in /usr/sbin/safe_asterisk, at the
commented-out settings for SYSMAXFILES and MAXFILES, and try setting those.

Cheers
Tony
--
Tony Mountifield
Work: tony@softins.co.uk - http://www.softins.co.uk
Play: tony@mountifield.org - http://tony.mountifield.org

------------------------------

_______________________________________________
--Bandwidth and Colocation Provided by http://www.api-digital.com--

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users

End of asterisk-users Digest, Vol 133, Issue 13
***********************************************

--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
New to Asterisk? Join us for a live introductory webinar every Thurs:
http://www.asterisk.org/hello

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users

viljoens at verishare....
Guest

Posted: Wed Aug 12, 2015 2:47 am Post subject: [asterisk-users] 786 000 files limit Centos 7 - Asterisk

Hi Tony

Thanks again for replying.

Ok, I've set the SYSMAXFILES and MAXFILES in the script, hopefully it will
have an effect.

Kind regards,

Message: 12
Date: Tue, 11 Aug 2015 16:50:29 +0000 (UTC)
From: tony@softins.co.uk (Tony Mountifield)
To: asterisk-users@lists.digium.com
Subject: Re: [asterisk-users] 786 000 files limit Centos 7 - Asterisk
keep complaining
Message-ID: <assp.06651285b6.mqd94l$qb1$1@softins.softins.co.uk>

In article <002b01d0d414$36af31b0$a40d9510$@verishare.co.za>,
Stefan Viljoen <viljoens@verishare.co.za> wrote:

Quote:

Anybody else ran into this?

Quote:

No, but I would ask myself why so many file descriptors are being used.
It sounds like you have a file descriptor leak (not being closed when
finished with).

Quote:

From what you said below, the above is probably not relevant...

Quote:

jcolp at digium.com
Guest

Posted: Wed Aug 12, 2015 6:34 am Post subject: [asterisk-users] 786 000 files limit Centos 7 - Asterisk

On Wed, Aug 12, 2015, at 04:43 AM, Stefan Viljoen wrote:

Quote:

<snip>

This is actually enforced by the system, not by Asterisk itself.

--
Joshua Colp
Digium, Inc. | Senior Software Developer
445 Jan Davis Drive NW - Huntsville, AL 35806 - US
Check us out at: www.digium.com & www.asterisk.org

--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by http://www.api-digital.com --
New to Asterisk? Join us for a live introductory webinar every Thurs:
http://www.asterisk.org/hello

asterisk-users mailing list
To UNSUBSCRIBE or update options visit:
http://lists.digium.com/mailman/listinfo/asterisk-users

Display posts from previous:

	VoIP Mailing List Archives Forum Index -> Asterisk Users	All times are GMT - 5 Hours
Page 1 of 1

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum