我们以前用于在我们的数据库表 (Postgres RDS) 中存储用户 session
我们决定将用户 session 从数据库迁移到 Redis,并对我们的应用程序进行了更改
对于 Redis,我们决定使用具有 1 个分片、2 个节点(主 + 副本)和启用多可用区的 Elastic Cache 服务
在 Live 环境中,一切都很顺利,直到 session 数超过 50 万(下午 3 点左右)
此时Redis Node突然停止响应导致我们的Production环境彻底崩溃(等待连接的线程过多)
我们必须重启实例才能恢复服务
这在晚上 9 点左右再次发生
Java端(spring)产生的异常
2016/11/22 09:19:31.749 <a href="http-nio-8080-exec-780">http-nio-8080-exec-780</a> <a href="https://forums.aws.amazon.com/">ERROR</a> org.apache.catalina.core.ContainerBase.<a href="https://forums.aws.amazon.com/">Tomcat</a>.<a href="https://forums.aws.amazon.com/">localhost</a>.[/].<a href="https://forums.aws.amazon.com/">dispatcherServlet</a> - Servlet.service() for servlet <a href="https://forums.aws.amazon.com/">dispatcherServlet] in context with path [</a> threw exception
org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:140) ~<strike>spring-data-redis-1.4.2.RELEASE.jar!/:1.4.2.RELEASE</strike>
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:229) ~<strike>spring-data-redis-1.4.2.RELEASE.jar!/:1.4.2.RELEASE</strike>
....
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) <strike>na:1.7.0_72</strike>
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) <strike>tomcat-embed-core-8.0.20.jar!/:8.0.20</strike>
at java.lang.Thread.run(Thread.java:745) <strike>na:1.7.0_72</strike>
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool
at redis.clients.util.Pool.getResource(Pool.java:42) ~<strike>jedis-2.5.2.jar!/:na</strike>
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:84) ~<strike>jedis-2.5.2.jar!/:na</strike>
at redis.clients.jedis.JedisPool.getResource(JedisPool.java:10) ~<strike>jedis-2.5.2.jar!/:na</strike>
at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:133) ~<strike>spring-data-redis-1.4.2.RELEASE.jar!/:1.4.2.RELEASE</strike>
... 55 common frames omitted
Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: connect timed out
at redis.clients.jedis.Connection.connect(Connection.java:150) ~<strike>jedis-2.5.2.jar!/:na</strike>
at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:71) ~<strike>jedis-2.5.2.jar!/:na</strike>
at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1783) ~<strike>jedis-2.5.2.jar!/:na</strike>
at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:65) ~<strike>jedis-2.5.2.jar!/:na</strike>
at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:836) ~<strike>commons-pool2-2.2.jar!/:2.2</strike>
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:434) ~<strike>commons-pool2-2.2.jar!/:2.2</strike>
at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~<strike>commons-pool2-2.2.jar!/:2.2</strike>
at redis.clients.util.Pool.getResource(Pool.java:40) ~<strike>jedis-2.5.2.jar!/:na</strike>
... 58 common frames omitted
Caused by: java.net.SocketTimeoutException: connect timed out
at java.net.PlainSocketImpl.socketConnect(Native Method) ~<strike>na:1.7.0_72</strike>
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~<strike>na:1.7.0_72</strike>
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~<strike>na:1.7.0_72</strike>
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~<strike>na:1.7.0_72</strike>
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~<strike>na:1.7.0_72</strike>
at java.net.Socket.connect(Socket.java:579) ~<strike>na:1.7.0_72</strike>
at redis.clients.jedis.Connection.connect(Connection.java:144) ~<strike>jedis-2.5.2.jar!/:na</strike>
... 65 common frames omitted
我们还不知道这到底是什么原因?
有人可以为我们指出正确的方向并帮助我们确定问题的根本原因和解决方案吗?
最佳答案
在某些版本的 Spring 框架上,Spring 不会在事务完成后关闭 redis 连接,因此连接最终会耗尽。如果您使用 setEnableTransactionSupport(true) 初始化 Jedis 模板,这可能会触发错误。将其设置为 false 应该可以修复它。
如果您需要交易,还有其他解决方法。参见本文“Spring Data Redis中的一个事务陷阱”部分; http://www.javaworld.com/article/3062899/big-data/lightning-fast-nosql-with-spring-data-redis.html
关于java - Redis (AWS Elasticache) 在一段时间后停止工作,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/40749526/