python - 循环 web.client.getPage 函数时内存泄漏

Question

我有一个使用此脚本定期刷新的页面:

from twisted.web.client import getPage
from twisted.internet import reactor, task

def getData():
    dgp = getPage('http://www.google.com/')
    dgp.addCallback(dataLoadOK)
    dgp.addErrback(dataLoadError)

def dataLoadOK(value):
    print value

def dataLoadError(error):
    print error

loop = task.LoopingCall(getData)
loop.start(10, now=True)
reactor.run()

购买时使用这种方式我得到了内存泄漏。有没有人帮我找到它？

编辑: 我尝试使用 garbage collection python module ，然后把它放出来:

GARBAGE OBJECTS:
:: <HTTPClientFactory: http://www.google.com/>
        type: <type 'instance'>
referrers: 3
    is class: True
    module: <module 'twisted.web.client' from '/usr/lib/python2.7/site-packages/twisted/web/client.pyc'>

:: {'status': '200', 'cookies': {'PREF': 'ID=d894e510f2ebe263:FF=0:TM=1306053252:LM=1306053252:S=ebpb4ZebRUu_EhiI', 'NID': '47=LxM9fbBBN-bVIeuLPOfvO-fgXOKw1n2suyZ2...
        type: <type 'dict'>
referrers: 3
    is class: True
    module: None

:: InsensitiveDict({})
        type: <type 'instance'>
referrers: 3
    is class: True
    module: <module 'twisted.python.util' from '/usr/lib/python2.7/site-packages/twisted/python/util.pyc'>

:: {'preserve': 1, 'data': {}}
        type: <type 'dict'>
referrers: 3
    is class: True
    module: None

:: <Deferred at 0x29e2cf8 current result: None>
        type: <type 'instance'>
referrers: 3
    is class: True
    module: <module 'twisted.internet.defer' from '/usr/lib/python2.7/site-packages/twisted/internet/defer.pyc'>

:: {'_chainedTo': None, 'called': True, '_canceller': None, 'callbacks': [], 'result': None, '_runningCallbacks': False}
        type: <type 'dict'>
referrers: 3
    is class: True
    module: None

:: <<class 'twisted.internet.tcp.Client'> to ('www.google.com', 80) at 2445090>
        type: <class 'twisted.internet.tcp.Client'>
referrers: 3
    is class: True
    module: <module 'twisted.internet.tcp' from '/usr/lib/python2.7/site-packages/twisted/internet/tcp.pyc'>
    line num: 681
        line: class Client(BaseClient):
        line:     """A TCP client."""
        line: 
        line:     def __init__(self, host, port, bindAddress, connector, reactor=None):
        line:         # BaseClient.__init__ is invoked later
        line:         self.connector = connector
        line:         self.addr = (host, port)
        line: 
        line:         whenDone = self.resolveAddress
        line:         err = None
        line:         skt = None
        line: 
        line:         try:
        line:             skt = self.createInternetSocket()
        line:         except socket.error, se:
        line:             err = error.ConnectBindError(se[0], se[1])
        line:             whenDone = None
        line:         if whenDone and bindAddress is not None:
        line:             try:
        line:                 skt.bind(bindAddress)
        line:             except socket.error, se:
        line:                 err = error.ConnectBindError(se[0], se[1])
        line:                 whenDone = None
        line:         self._finishInit(whenDone, skt, err, reactor)
        line: 
        line:     def getHost(self):
        line:         """Returns an IPv4Address.
        line: 
        line:         This indicates the address from which I am connecting.
        line:         """
        line:         return address.IPv4Address('TCP', *(self.socket.getsockname() + ('INET',)))
        line: 
        line:     def getPeer(self):
        line:         """Returns an IPv4Address.
        line: 
        line:         This indicates the address that I am connected to.
        line:         """
        line:         return address.IPv4Address('TCP', *(self.realAddress + ('INET',)))
        line: 
        line:     def __repr__(self):
        line:         s = '<%s to %s at %x>' % (self.__class__, self.addr, unsignedID(self))
        line:         return s

:: {'_tempDataBuffer': [], 'disconnected': 1, 'dataBuffer': '', '_tempDataLen': 0, 'realAddress': ('74.125.225.81', 80), 'connector': <twisted.internet.tcp.Connect...
        type: <type 'dict'>
referrers: 3
    is class: True
    module: None

:: []
        type: <type 'list'>
referrers: 3
    is class: True
    module: None

:: {'x-xss-protection': ['1; mode=block'], 'set-cookie': ['PREF=ID=d894e510f2ebe263:FF=0:TM=1306053252:LM=1306053252:S=ebpb4ZebRUu_EhiI; expires=Tue, 21-May-2013 0...
        type: <type 'dict'>
referrers: 3
    is class: True
    module: None

:: ['-1']
        type: <type 'list'>
referrers: 3
    is class: True
    module: None

:: ['private, max-age=0']
        type: <type 'list'>
referrers: 3
    is class: True
    module: None

:: ['text/html; charset=ISO-8859-1']
        type: <type 'list'>
referrers: 3
    is class: True
    module: None

:: ['PREF=ID=d894e510f2ebe263:FF=0:TM=1306053252:LM=1306053252:S=ebpb4ZebRUu_EhiI; expires=Tue, 21-May-2013 08:34:12 GMT; path=/; domain=.google.com', 'NID=47=LxM9...
        type: <type 'list'>
referrers: 3
    is class: True
    module: None

:: ['gws']
        type: <type 'list'>
referrers: 3
    is class: True
    module: None

:: ['1; mode=block']
        type: <type 'list'>
referrers: 3
    is class: True
    module: None

:: []
        type: <type 'list'>
referrers: 3
    is class: True
    module: None

:: <twisted.internet.tcp.Connector instance at 0x29e2cb0>
        type: <type 'instance'>
referrers: 3
    is class: True
    module: <module 'twisted.internet.tcp' from '/usr/lib/python2.7/site-packages/twisted/internet/tcp.pyc'>

:: ['Sun, 22 May 2011 08:34:12 GMT']
        type: <type 'list'>
referrers: 3
    is class: True
    module: None

:: {'reactor': <twisted.internet.selectreactor.SelectReactor object at 0x288bd10>, 'state': 'disconnected', 'factoryStarted': 0, 'bindAddress': None, 'factory': <H...
        type: <type 'dict'>
referrers: 3
    is class: True
    module: None

所以我在 twisted 函数中看到了一些未关闭的引用，我该如何避免呢？

Answer 1

尝试在 related questions 中推荐的一些策略.但是，您可能没有内存泄漏，您只有 memory fragmentation。 .

看起来“Python 内存泄漏检测器”有一个非常严重的错误。它启用DEBUG_LEAK，防止收集所有循环。换句话说，它造成大量的大规模泄漏。如果您只是在示例中添加一些代码来报告 gc.garbage 的内容而没有启用 DEBUG_LEAK，那么它仍然是空的(gc.garbage如果任何对象实际上正在泄漏，将被填充，即使您没有启用任何 gc 调试标志)。