从另一个 python 脚本调用 python 脚本时，Python 日志记录挂起

Question

我在 python 日志类中观察到这个奇怪的问题，我有两个脚本，一个是从另一个脚本调用的。第一个脚本等待其他脚本结束，而其他脚本使用 logging.info

记录大量日志

这是代码片段

#!/usr/bin/env python

import subprocess
import time
import sys

chars = ["/","-","\\","|"]
i = 0
command = 'sudo python /home/tejto/test/writeIssue.py'
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
while process.poll() is None:
    print chars[i],
    sys.stdout.flush()
    time.sleep(.3)
    print "\b\b\b",
    sys.stdout.flush()
    i = (i + 1)%4
output = process.communicate()

另一个脚本是

#!/usr/bin/env python

import os
import logging as log_status

class upgradestatus():
    def __init__(self):
        if (os.path.exists("/tmp/updatestatus.txt")):
                os.remove("/tmp/updatestatus.txt")

        logFormatter = log_status.Formatter("%(asctime)s [%(levelname)-5.5s]  %(message)s")
        logger = log_status.getLogger()
        logger.setLevel(log_status.DEBUG)

        fileHandler = log_status.FileHandler('/tmp/updatestatus.txt', "a")
        fileHandler.setLevel(log_status.DEBUG)
        fileHandler.setFormatter(logFormatter)
        logger.addHandler(fileHandler)

        consoleHandler = log_status.StreamHandler()
        consoleHandler.setLevel(log_status.DEBUG)
        consoleHandler.setFormatter(logFormatter)
        logger.addHandler(consoleHandler)

    def status_change(self, status):
            log_status.info(str(status))

class upgradeThread ():
    def __init__(self, link):
        self.upgradethreadstatus = upgradestatus()
    self.upgradethreadstatus.status_change("Entered upgrade routine")
    procoutput = 'very huge logs, mine were 145091 characters'
    self.upgradethreadstatus.status_change(procoutput)
    self.upgradethreadstatus.status_change("Exiting upgrade routine")

if __name__ == '__main__':
    upgradeclass = upgradeThread(sys.argv[1:]

如果我运行第一个脚本，两个脚本都会挂起，问题似乎出在代码 while process.poll() is None ，如果我注释此代码，则一切正常。 (无法将其与我的问题联系起来!!)

PS 我也尝试调试 python 日志记录类，其中我发现该进程被 StreamHandler 类的 emit 函数卡住，其中它卡在 Stream.write 函数调用，在写入大量日志后没有出来，但是我的退出日志没有出现。

那么这些脚本中可能出现什么问题而导致死锁情况呢？

Edit 1 (code with threading)

脚本.py

#!/usr/bin/env python 

import subprocess
import time
import sys
import threading

def launch():
    command = ['python', 'script2.py']
    process = subprocess.Popen(command,stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)
    output = process.communicate()

t = threading.Thread(target=launch)
t.start()

chars = ["/","-","\\","|"]
i = 0

while t.is_alive:
    print chars[i],
    sys.stdout.flush()
    time.sleep(.3)
    print "\b\b\b",
    sys.stdout.flush()
    i = (i + 1)%4
t.join()

脚本2.py

#!/usr/bin/env python  
import os
import sys
import logging as log_status
import time

class upgradestatus():
    def __init__(self):
        if (os.path.exists("/tmp/updatestatus.txt")):
                        os.remove("/tmp/updatestatus.txt")

        logFormatter = log_status.Formatter("%(asctime)s [%(levelname)-5.5s]  %(message)s")
        logger = log_status.getLogger()
        logger.setLevel(log_status.DEBUG)

        fileHandler = log_status.FileHandler('/tmp/updatestatus.txt', "a")
        fileHandler.setLevel(log_status.DEBUG)
        fileHandler.setFormatter(logFormatter)
        logger.addHandler(fileHandler)

        consoleHandler = log_status.StreamHandler()
        consoleHandler.setLevel(log_status.DEBUG)
        consoleHandler.setFormatter(logFormatter)
        logger.addHandler(consoleHandler)

    def status_change(self, status):
        log_status.info(str(status))

class upgradeThread ():
    def __init__(self, link):
        self.upgradethreadstatus = upgradestatus()
        self.upgradethreadstatus.status_change("Entered upgrade routine")
        procoutput = "Please put logs of 145091 characters over here otherwise the situtation wouldn't remain same or run any command whose output is larger then 145091 characters"
    self.upgradethreadstatus.status_change(procoutput)
        time.sleep(1)
        self.upgradethreadstatus.status_change("Exiting upgrade routine")

if __name__ == '__main__':
    upgradeclass = upgradeThread(sys.argv[1:])

在这种情况下 t.is_alive 函数不会返回 false (不知道，但启动函数已经返回，所以理想情况下它应该返回 false!!:()

Answer 1

标准输出缓冲区阻塞。

process.communicate() 调用没有被执行，因为它是在 当 process.poll() 为 None 时:。因此，writeIssue.py 尝试向 stdout 写入太多字节，并且所有字节都在 subprocess.PIPE 中缓冲，并且不会被拉取离开 PIPE，直到调用 communicate。

缓冲区的大小是有限的。当缓冲区已满时，stream.write将阻塞 直到缓冲区有空间。如果缓冲区从未清空(如发生在 你的代码)，然后进程就会死锁。

修复方法是在缓冲区完全填满之前调用communicate()。您可以通过在线程中启动 writeIssue.py 并在 while-thread-is-alive 循环中并发调用 communicate() 来实现这一点在主线程中运行。

<小时/>

脚本.py:

import subprocess
import time
import sys
import threading

def launch():
    command = ['python', 'script2.py']
    process = subprocess.Popen(command)
    process.communicate()

t = threading.Thread(target=launch)
t.start()

chars = ["/","-","\\","|"]
i = 0
while t.is_alive():
    print chars[i],
    sys.stdout.flush()
    time.sleep(.3)
    print "\b\b\b",
    sys.stdout.flush()
    i = (i + 1)%4

t.join()

<小时/>

script2.py:

import sys
import logging
import time

class UpgradeStatus():
    def __init__(self):
        logFormatter = logging.Formatter("%(asctime)s [%(levelname)-5.5s]  %(message)s")
        self.logger = logging.getLogger()
        self.logger.setLevel(logging.DEBUG)

        consoleHandler = logging.StreamHandler()
        consoleHandler.setLevel(logging.DEBUG)
        consoleHandler.setFormatter(logFormatter)
        self.logger.addHandler(consoleHandler)

    def status_change(self, status):
        self.logger.info(str(status))

class UpgradeThread():
    def __init__(self, link):
        self.upgradethreadstatus = UpgradeStatus()
        self.upgradethreadstatus.status_change("Entered upgrade routine")
        for i in range(5):
            procoutput = 'very huge logs, mine were 145091 characters'
            self.upgradethreadstatus.status_change(procoutput)
            time.sleep(1)
        self.upgradethreadstatus.status_change("Exiting upgrade routine")

if __name__ == '__main__':
    upgradeclass = UpgradeThread(sys.argv[1:])

<小时/>

请注意，如果有两个线程同时写入 stdout，则输出 会出现乱码。如果您想避免这种情况，那么所有输出都应该由 单线程配备队列。所有其他希望的线程或进程 写输出应将字符串或日志记录推送到队列中 专用输出线程来处理。

然后，该输出线程可以使用 for 循环从队列中提取输出:

for message from iter(queue.get, None):
    print(message)