Fix Apache - No space left on device: Couldn't create accept lock or Cannot create SSLMutex

When dealing with mem-leaks in my mod_perl-apps I ran into a curious apache-problem. After a while apache could not be started but failed with strange errors like: [emerg] (28)No space left on device: Couldn't create accept lock or [crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed or [Wed Dec 07 00:00:09 2005] [error] (28)No space left on device: Cannot create SSLMutex There was definitely enough space on the device where the locks are stored (default /usr/local/apache2/logs/). I tried to explicetely different Lockfiles using the LockFile-directive but this did not help. I also tried a non-default AcceptMutex (flock) which then solved the acceptlock-issue and ended in the rewrite_log_lock-issue. Only reboot of the system helped out of my crisis. Solution: There were myriads of semaphore-arrays left, owned by my apache-user. ipcs -s | grep apache Removing this semaphores immediately solved the problem. ipcs -s | grep apache | perl -e 'while (<STDIN>) { @a=split(/\s+/); print `ipcrm sem $a[1]`}'

Comments

Finding used test equipment

Finding used test equipment for macs is pretty tough going to have to get a netbook

Some changes required to work on Ubuntu 8.04

Thanks a lot for this, here are a few changes I made to make this work on Ubuntu Hardy: ipcs -s | grep www-data | sudo perl -alne 'qx(ipcrm -s $F[1])' 1. The apache user is www-data instead of apache 2. ipcrm changed its parameters format from "sem " to "-s " 3. Used sudo for the ipcrm 3. Just for fun, I tweaked the perl command line parameters to use auto-split and auto-line ending

Suggested permanent fix

Thanks for the fix, had me stumped for a while. Very useful script. Suggested permanent fix is setting the SSLMutex directive to "sem" SSLMutex sem http://www.modssl.org/docs/2.1/ssl_reference.html HTH, and Thanks again...

thanks Solved

saved me from much down time. thanks lot chetanM

Thank you

Very usefull script

Shell version

Hi , for sh lovers i am putting this command ipcs -s | grep apache | awk ' { print $2 } ' | xargs ipcrm sem Thanks, Nightlife_king

Thanks

THanks, i had the same problem and because of your post could fix it in no time

thanks for your help

the following is doing the same if you don't want to use perl: ipcs -s | grep apache | gawk '{ print $2 }' | xargs -n 1 ipcrm sem

Re: Fix Apache - No space left on device: Couldn't create accept

[quote=carlos] Removing this semaphores immediately solved the problem. ipcs -s | grep apache | perl -e 'while (<STDIN>) { @a=split(/\s+/); print `ipcrm sem $a[1]`}' [/quote] Great. You saved my live!!! (Rebooting was not an option for me). Just my two cents. Next the equivalent with bash+textutils instead of perl (yes, I don't like perl, it's so ugly to read): for ipsemId in $(ipcs -s | grep apache | cut -f 2 -d ' '); do ipcrm $ipsemId; done Have a nice day

Fix Apache - No space left on device: Couldn't create accept loc

Killing the semaphores is a nice temporary solution to this problem, but I have been dealing with it for months and I was wondering if there is a more permanent solution to the problem of memory leaks here?

Thanks !!!

Hi, I've just found your webpage via Google's 2nd result on " [emerg] (28)No space left on device: Couldn't create accept lock " query. Your solution applied and Apache running again in 2 mn ! My panic is down, I promise I definitvly change my VPS webhosting. Many thanks for posting this dude ! See U.

correction

Very helpful information, BUT the command line to fix the problem is not correct. The while() shold actually be while(<STDIN>). So cut-and-paste this line:  ipcs -s | grep apache | perl -e 'while (<STDIN>) { @a=split(/\s+/); print `ipcrm sem $a[1]`}' ;-)

thanks fixed

ops didnt notice went i posted... looks like the <STDIN> got filtered.... thanks... (now updated)