I am receiving the below error randomly from the php backend jobs and php web page logs. Have a app server which runs php backend jobs and php webservers. Both connect to the same database server. Using php mysqli object oriented library for connecting to the database. Have set max connections to 750 in my.cnf. Dont see that much connections is reached.
PHP Warning: mysqli::mysqli(): (HY000/2003): Can’t connect to MySQL server on ‘77.777.120.81’ (99) in /usr/local/dev/classes/Admin.php on line 15
Failed to connect to MySQL: Can’t connect to MySQL server on ‘77.777.120.81’ (99)
As described excellently in this Percona Database Performance Blog article, your problem is that your application cannot open another connection to MySQL server. You are running out of local TCP ports. As a solution i would propose to Tweak TCP parameter settings
- tcp_tw_reuse (Boolean; default: disabled; since Linux 2.4.19/2.6)
Allow to reuse TIME_WAIT sockets for new connections when it
is safe from protocol viewpoint. It should not be changed
without advice/request of technical experts.
It is possible to force the kernel to reuse a connection hanging in TIME_WAIT state by setting /proc/sys/net/ipv4/tcp_tw_reuse to 1. What happens in practice is that you’ll keep seeing the closed connections hanging in TIME_WAIT until either they expire or a new connection is requested. In the later case, the connection will be “relived”.
- tcp_tw_recycle (Boolean; default: disabled; since Linux 2.4)
Enable fast recycling of TIME_WAIT sockets. Enabling this
option is not recommended since this causes problems when
working with NAT (Network Address Translation).
When you enable /proc/sys/net/ipv4/tcp_tw_recycle closed connections will not show under TIME_WAIT anymore – they disappear from netstat altogether. But as soon as you open a new connection (within the 60 seconds mark) it will recycle one of those. But everyone writing about this alternative seems to advise against it’s use. Bottom line is: it’s preferable to reuse a connection than to recycle it.
- tcp_max_tw_buckets (integer; default: see below; since Linux 2.4)
The maximum number of sockets in TIME_WAIT state allowed in
the system. This limit exists only to prevent simple denial-
of-service attacks. The default value of NR_FILE*2 is
adjusted depending on the memory in the system. If this
number is exceeded, the socket is closed and a warning is
This parameter rules how many connections can remain in TIME_WAIT state concurrently: the kernel will
simply kill connections hanging in such state above that number. For example, in a scenario where the server has a TCP port range composed of only 6 ports, if
/proc/sys/net/ipv4/tcp_max_tw_bucketsis set to 5, then open 6 concurrent connections with MySQL and then immediately close all 6 you would find only 5 of them hanging in the TIME_WAIT state – as with
tcp_tw_recycle, one of them would simply disappear from
netstatoutput. This situation allows to immediately open a new connection without needing to wait for a minute*.
- When it comes to connecting to database servers, many applications chose to open a new connection for a single request only, closing it right after the request is processed. Even though the connection is closed by the client (application) the local port it was using is not immediately released by the OS to be reused by another connection: it will sit in a TIME_WAIT state for (usually) 60 seconds – this value cannot be easily changed as it is hard coded in the kernel.
However, a second connection won’t be able to open until one of the other 5 connections in TIME_WAIT expire and release the local port it was using. The secret here, then, is to find a compromise between the number of available network ports and the number of connections we allow to remain in TIME_WAIT state. The default value of this setting is 65536, which means by default the system allows all possible connections to go over the TIME_WAIT state when closed.
PS: There more possible solutions to your problem, read the full article for detailed description of the problem.
tcp_tw_reuse looks better solution. Here is described why:
mysql error (99) means that you are running out of the tcp ports.
Enabling tcp recycle should fix it.
echo 1 >/proc/sys/net/ipv4/tcp_tw_recycle
I am able to connect from local shell, so first thought was if something wrong with recent Zend Framework upgrade, but after a while I figured out that answer is very simple – SELinux was blocking remote connections from PHP scripts executed by Apache web server. The error code (13) at the end of error message means “permission denied”, so that’s the indication to see if you have similar issue or not.
Anyway, login as root and do
setsebool -P httpd_can_network_connect=1
to make it work.
Of course, think twice because you make web server a bit less secure, so don’t do that unless you are sure you need it.