python-3.x - ConnectionError : Error 104 while writing to socket. 连接被对等方重置

Question

ConnectionError:写入套接字时出现错误 104。对等方重置连接。

环境: Ubuntu的:16.04 python :3.6 PC总内存:32G

我已经安装了 redis '3.0.6'。

尝试插入500,000条数据时成功，但尝试插入4000万条大数据时失败。

尝试将Python数据框插入redis时，由于容量太大而失败。

数据插入成功:

 1. r = redis.StrictRedis(host='localhost', port=6379, db=0)
 2.log_df_50.shape
   -> (500000, 6)
 3.r.setex('log_df_50',100,log_df_50.to_json())
   -> True

数据插入失败:

 1.r = redis.StrictRedis(host='localhost', port=6379, db=0)
 2. log_df.shape
   -> (41757802, 6)
 3. r.setex('session',100,log_df.to_json())

---

ConnectionResetError Traceback (most recent call last) ~/anaconda3/envs/Colabo/lib/python3.6/site-packages/redis/connection.py in send_packed_command(self, command, check_health) 705 for item in command: --> 706 sendall(self._sock, item) 707 except socket.timeout:

~/anaconda3/envs/Colabo/lib/python3.6/site-packages/redis/_compat.py in sendall(sock, *args, **kwargs) 8 def sendall(sock, *args, **kwargs): ----> 9 return sock.sendall(*args, **kwargs) 10

ConnectionResetError: [Errno 104] Connection reset by peer

During handling of the above exception, another exception occurred:

ConnectionError Traceback (most recent call last) in ----> 1 r.setex('session',100,log_df.to_json())

~/anaconda3/envs/Colabo/lib/python3.6/site-packages/redis/client.py in setex(self, name, time, value) 1820 if isinstance(time, datetime.timedelta): 1821 time = int(time.total_seconds()) -> 1822 return self.execute_command('SETEX', name, time, value) 1823 1824 def setnx(self, name, value):

~/anaconda3/envs/Colabo/lib/python3.6/site-packages/redis/client.py in execute_command(self, *args, **options) 898 conn = self.connection or pool.get_connection(command_name, **options) 899 try: --> 900 conn.send_command(*args) 901 return self.parse_response(conn, command_name, **options) 902 except (ConnectionError, TimeoutError) as e:

~/anaconda3/envs/Colabo/lib/python3.6/site-packages/redis/connection.py in send_command(self, *args, **kwargs) 724 "Pack and send a command to the Redis server" 725 self.send_packed_command(self.pack_command(*args), --> 726 check_health=kwargs.get('check_health', True)) 727 728 def can_read(self, timeout=0):

~/anaconda3/envs/Colabo/lib/python3.6/site-packages/redis/connection.py in send_packed_command(self, command, check_health) 716 errmsg = e.args[1] 717 raise ConnectionError("Error %s while writing to socket. %s." % --> 718 (errno, errmsg)) 719 except BaseException: 720 self.disconnect()

ConnectionError: Error 104 while writing to socket. Connection reset by peer.

关于原因的任何提示？

如何将python大容量Dataframe插入redis？

我应该怎么做才能解决这个问题？

Answer 1

我认为问题在于 Redis 协议(protocol)中键/值的 512 MB 限制。
https://redis.io/topics/protocol
https://redis.io/topics/data-types-intro

现在你无能为力了。

您可能只需要将数据帧切成更小的部分。

更新:
您也可以使用较新的 Redis(最低 5.0)并在 redis.conf 中增加 proto-max-bulk-len。 https://github.com/redis/redis/issues/7354
我还不得不增加 client-query-buffer-limit。