我想在 python 中创建一个 rdiff-backup 包装程序，用于将 windows 机器备份到 linux 服务器。
我想在包装程序中处理 rdiff-backup 的输出。但是当使用子进程模块执行 rdiff-backup 并将 stdout 和 stderr 输送到包装器时，stdout 总是在管道的末端结束。
在不通过管道传输 stdout 和 stderr 的情况下使用子进程时，会在控制台中以正确的顺序打印。
我还注意到，当在本地使用 rdiff-backup 而不是使用 ssh 管道时，stdout 和 stderr 的顺序是正确的。 Rdiff-backup 还使用 subprocess.popen 打开 ssh session 并将数据通过管道传输到服务器。我怀疑由于某种原因，stdout 被阻塞，直到 ssh session 关闭。
这是我的代码，这是真实程序使用线程来监听标准输出的简化版本:
import sys
import subprocess
class Rdiffbackup(object):
def __init__(self):
#self.io_q = Queue()
self.exe = 'F:\\workspace\\pysubprocess\\bin\\rdiff-backup\\rdiff-backup.exe'
self.verbosity = '-v5'
self.ssh_exe = './bin/openssh/bin/ssh'
self.quiet = '-q'
self.compression = '-C'
self.port = '-p 5555'
self.key = '-i ./keys/rdiffbackup'
self.options = '-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null'
self.remote_schema = self.ssh_exe + ' ' + self.quiet + ' ' + self.compression + ' ' + self.port + ' ' + self.key + ' ' + self.options + ' %s rdiff-backup --server'
def start(self,source,dest):
with subprocess.Popen([self.exe,self.verbosity,'--remote-schema',self.remote_schema,source,dest],stdout=subprocess.PIPE,stderr=subprocess.STDOUT) as self.proc:
for line in self.proc.stdout:
sys.stdout.write(line.decode("utf-8"))
if __name__ == '__main__':
rdb = Rdiffbackup()
source = "C:/Users/vdrmrt/Desktop/data"
dest = "vdrmrt@hostname::backup"
rdb.start(source,dest)
输出:
-----------------------------------------------------------------
Detected abilities for source (read only) file system:
Access control lists Off
Extended attributes Off
Windows access control lists On
Case sensitivity Off
Escape DOS devices Off
Escape trailing spaces Off
Mac OS X style resource forks Off
Mac OS X Finder information Off
-----------------------------------------------------------------
Unable to import win32security module. Windows ACLs
not supported by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
escape_dos_devices not required by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
Ownership changing Off
Hard linking On
fsync() directories On
Directory inc permissions On
High-bit permissions On
Symlink permissions Off
Extended filenames On
Windows reserved filenames Off
Access control lists On
Extended attributes On
Windows access control lists Off
Case sensitivity On
Escape DOS devices Off
Escape trailing spaces Off
Mac OS X style resource forks Off
Mac OS X Finder information Off
-----------------------------------------------------------------
Backup: must_escape_dos_devices = 0
Using rdiff-backup version 1.2.8
Executing ./bin/openssh/bin/ssh -q -C -p 5555 -i ./keys/rdiffbackup -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null vdrmrt@hostname rdiff-backup --server
Hardlinks disabled by default on Windows
Unable to import module xattr.
Extended attributes not supported on filesystem at C:/Users/vdrmrt/Desktop/data
Unable to import module posix1e from pylibacl package.
POSIX ACLs not supported on filesystem at C:/Users/vdrmrt/Desktop/data
escape_dos_devices not required by filesystem at C:/Users/vdrmrt/Desktop/data
Symbolic links excluded by default on Windows
Starting increment operation C:/Users/vdrmrt/Desktop/data to backup
正确的输出:
Using rdiff-backup version 1.2.8
Executing ./bin/openssh/bin/ssh -q -C -p 5555 -i ./keys/rdiffbackup -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null vdrmrt@hostname rdiff-backup --server
Hardlinks disabled by default on Windows
Unable to import module xattr.
Extended attributes not supported on filesystem at C:/Users/vdrmrt/Desktop/data
Unable to import module posix1e from pylibacl package.
POSIX ACLs not supported on filesystem at C:/Users/vdrmrt/Desktop/data
escape_dos_devices not required by filesystem at C:/Users/vdrmrt/Desktop/data
-----------------------------------------------------------------
Detected abilities for source (read only) file system:
Access control lists Off
Extended attributes Off
Windows access control lists On
Case sensitivity Off
Escape DOS devices Off
Escape trailing spaces Off
Mac OS X style resource forks Off
Mac OS X Finder information Off
-----------------------------------------------------------------
Unable to import win32security module. Windows ACLs
not supported by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
escape_dos_devices not required by filesystem at backup/rdiff-backup-data/rdiff-backup.tmp.0
-----------------------------------------------------------------
Detected abilities for destination (read/write) file system:
Ownership changing Off
Hard linking On
fsync() directories On
Directory inc permissions On
High-bit permissions On
Symlink permissions Off
Extended filenames On
Windows reserved filenames Off
Access control lists On
Extended attributes On
Windows access control lists Off
Case sensitivity On
Escape DOS devices Off
Escape trailing spaces Off
Mac OS X style resource forks Off
Mac OS X Finder information Off
-----------------------------------------------------------------
Backup: must_escape_dos_devices = 0
Symbolic links excluded by default on Windows
Starting increment operation C:/Users/vdrmrt/Desktop/data to backup
最佳答案
我终于解决了这个问题。
Rdiff-backup 在通过管道传输到其他程序时缓冲 stdout 和 stderr。
解决方案是使用额外的 py2exe 选项重建 rdiff-backup:'unbuffered': True
我还必须添加一个额外的选项以使其在使用 Windows 7 构建时能够正常工作。
Py2exe - win32api.pyc ImportError DLL load failed
这是来自 rdiff-backup 构建的最终 setup.py 的片段。
if '--single-file' in sys.argv[1:]:
sys.argv.remove('--single-file')
extra_options.update({
'options': {'py2exe': {'bundle_files': 1,
'unbuffered': True,
'dll_excludes': [ "mswsock.dll", "powrprof.dll" ]}},
'zipfile': None
})
关于Python subprocess.popen 和 rdiff-backup，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/18298044/