Python 记录子进程

Question

我想在 python 中使用并行的 joblib 为每个子进程创建一个日志文件。但是，当我尝试以下最简单的示例时，只创建了几个日志文件，并且所有日志消息都以随机顺序插入到文件中。我做错了什么吗？这是Python 3.7.1，joblib版本是0.13.2。

from joblib import Parallel, delayed
import os

def func(i):
    import logging
    logging.basicConfig(filename='logs/%d.txt' % i)
    logger = logging.getLogger()
    logger.warning('This is task %d' % i)

Parallel(n_jobs=4)(delayed(func)(i) for i in range(16))

for f in os.listdir('logs'):
    r = open('logs/' + f, 'r')
    print(f, '\n', r.read())
    r.close()

输出为:

1.txt 
 WARNING:root:This is task 1
WARNING:root:This is task 13
WARNING:root:This is task 15

0.txt 
 WARNING:root:This is task 0
WARNING:root:This is task 2
WARNING:root:This is task 3
WARNING:root:This is task 4
WARNING:root:This is task 5
WARNING:root:This is task 6
WARNING:root:This is task 7
WARNING:root:This is task 8
WARNING:root:This is task 10
WARNING:root:This is task 11
WARNING:root:This is task 12
WARNING:root:This is task 14

9.txt 
 WARNING:root:This is task 9

Answer 1

如果我理解日志记录文档正确，logging.basicConfig 仍然适用于“root”记录器，这就是问题所在。然后发生的事情取决于每个新进程在代码中的文件 0.txt 或 1.txt 中生成的位置和时间(如 @IronMan 所回答)。因此，您需要向您创建的每个新记录器添加一个 FileHandler。像这样的事情:

import logging
from joblib import Parallel, delayed
import os

def func(i):
    logger = logging.getLogger(f"logger_{i}")
    # change 'mode' if appropriate for your situation ('a' is for append)
    logger.addHandler(logging.FileHandler(f"logs/{i}.txt", mode='a')) 
    logger.warning(f'This is task {i}')               

Parallel(n_jobs=4)(delayed(func)(i) for i in range(16))

for f in os.listdir('logs'):
    r = open('logs/' + f, 'r')
    print(f, '\n', r.read())
    r.close()

有关更多信息，请查看官方文档中的操作方法:https://docs.python.org/3/howto/logging.html

希望它能起作用(晚回答总比不回答好)。