Ubuntu: 12.04 LTS (Linux mysql02 3.2.0-40-generic #64-Ubuntu SMP Mon Mar 25 21:22:10 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux)

MySQL: Ubuntu distro 5.5.31

Apparmor: REMOVED!

Server has been running rock solid for over a year. Then this Monday MySQL began failing. An update has caused the problem and we cannot figure it out what it is. We have even tried to roll back to MySQL 5.5.30 but with no luck. We returned at 5.5.31.

MySQL error log entries:

130430  7:55:46 [ERROR] Error in accept: Too many open files

130430  7:55:46 [ERROR] /usr/sbin/mysqld: Can't open file: './eci_elite_test/fclvod.frm' (errno: 24)

130430  7:55:46 [ERROR] /usr/sbin/mysqld: Can't open file: './eci_elite_test/fcnote.frm' (errno: 24)

130430  7:55:47 [ERROR] /usr/sbin/mysqld: Can't open file: './eci_elite_test/ffcont.frm' (errno: 24)

130430  7:55:47 [ERROR] /usr/sbin/mysqld: Can't open file: './eci_elite_test/ffcontv.frm' (errno: 24)

130430  7:55:47 [ERROR] /usr/sbin/mysqld: Can't open file: './eci_elite_test/ffnote.frm' (errno: 24)

130430  7:55:47 [ERROR] /usr/sbin/mysqld: Can't open file: './eci_elite_test/frcfcl.frm' (errno: 24)

It appears we are running into ulimit problem. We have removed APPARMOR completely. We have increased the /etc/security/limits.conf and still no luck:

# Out of desperation....

* soft  nofile  49152

* hard  nofile  65536



# No effect!?!!?

#mysql  soft  nofile  49152

#mysql  hard  nofile  65536

And to show the limits.conf is working:

root@mysql02:/etc/security# ulimit -Sa | grep "open files"

open files                      (-n) 49152



root@mysql02:/etc/security# ulimit -Ha | grep "open files"

open files                      (-n) 65536

And here are the important entries in my.cnf

[mysqld_safe]

open_files_limit = 16384



[mysqld]

open_files_limit = 16384

However:

root@mysql02:/etc/mysql# mysqladmin -u root -pThePassword variables| grep open_files_limit

open_files_limit                                  | 1024

We are totally stumped and down. Any assistance would be greatly appreciated.