You are here

Fix Apache - No space left on device: Couldn't create accept lock or Cannot create SSLMutex

When dealing with mem-leaks in my mod_perl-apps I ran into a curious apache-problem. After a while apache could not be started but failed with strange errors like:

[emerg] (28)No space left on device: Couldn't create accept lock

or

[crit] (28)No space left on device: mod_rewrite: could not create rewrite_log_lock Configuration Failed

or

[Wed Dec 07 00:00:09 2005] [error] (28)No space left on device: Cannot create SSLMutex

There was definitely enough space on the device where the locks are stored (default /usr/local/apache2/logs/). I tried to explicetely different Lockfiles using the LockFile-directive but this did not help. I also tried a non-default AcceptMutex (flock) which then solved the acceptlock-issue and ended in the rewrite_log_lock-issue.

Only reboot of the system helped out of my crisis.

Solution: There were myriads of semaphore-arrays left, owned by my apache-user.

ipcs -s | grep apache

Removing this semaphores immediately solved the problem.

ipcs -s | grep apache | perl -e 'while (<STDIN>) { @a=split(/\s+/); print `ipcrm sem $a[1]`}'

Tags: 

Comments

Thanks a lot for this, here are a few changes I made to make this work on Ubuntu Hardy:
ipcs -s | grep www-data | sudo perl -alne 'qx(ipcrm -s $F[1])'

1. The apache user is www-data instead of apache
2. ipcrm changed its parameters format from "sem " to "-s "
3. Used sudo for the ipcrm
3. Just for fun, I tweaked the perl command line parameters to use auto-split and auto-line ending

Thanks for the fix, had me stumped for a while. Very useful script.
Suggested permanent fix is setting the SSLMutex directive to "sem"
SSLMutex  sem

Hi ,
for sh lovers i am putting this command
ipcs -s | grep apache | awk ' { print $2 } ' | xargs ipcrm sem
Thanks,
Nightlife_king

Removing this semaphores immediately solved the problem.
ipcs -s | grep apache | perl -e 'while (<STDIN>) { @a=split(/\s+/); print `ipcrm sem $a[1]`}'

Great. You saved my live!!! (Rebooting was not an option for me).
Just my two cents. Next the equivalent with bash+textutils instead of perl (yes, I don't like perl, it's so ugly to read):
for ipsemId in $(ipcs -s | grep apache | cut -f 2 -d ' '); do ipcrm $ipsemId; done
Have a nice day

Hi, I've just found your webpage via Google's 2nd result on " [emerg] (28)No space left on device: Couldn't create accept lock " query.
Your solution applied and Apache running again in 2 mn !
My panic is down, I promise I definitvly change my VPS webhosting.
Many thanks for posting this dude !
See U.