python - 在 Google DFS 文件夹中运行 jupyter 笔记本时出现 "Save and Checkpoint"错误

Question

我长期使用Jupyter Notebook在我的 Google 云端硬盘中，它运行得很好。最近我被迫切换到较新的版本Drive File Stream 。我制作了我的文件夹Available offline ，这应该意味着我将该文件夹保存在本地存储空间中。当我开始 Jupyter Notebook ，它运行良好，但试图 Save and Checkpoint给我一个错误 Checkpoint failed 。似乎它以某种方式正确保存了文件，但返回了错误。

我通过运行 Jupyter Notebook 启动了 jupyter 笔记本在Anaconda Prompt 。当我尝试Save and Checkpoint时出现以下错误。似乎它在提示保存的文件是相同的 - shutil.SameFileError这真的很奇怪，因为我确实通过额外的 Save Attempt x 改变了笔记本。帮助我判断它是否已保存。我还尝试删除文件夹.ipynb_checkpoints中相应的检查点并做Save and Checkpoint ，但我仍然遇到同样的错误...

我向 Jupyter Notebook 提交了问题单 github page ，似乎其他几个人也遇到了同样的问题，但没有得到任何帮助。

下面是我的日志:

[I 10:49:44.573 NotebookApp] Saving file at /Writing_MVUE_2d.ipynb [E 10:49:44.690 NotebookApp] Unhandled error in API request Traceback (most recent call last): File "C:\ProgramData\Anaconda3\lib\site-packages\notebook\base\handlers.py", line 516, in wrapper result = yield gen.maybe_future(method(self, *args, **kwargs)) File "C:\ProgramData\Anaconda3\lib\site-packages\tornado\gen.py", line 1015, in run value = future.result() File "C:\ProgramData\Anaconda3\lib\site-packages\tornado\concurrent.py", line 237, in result raise_exc_info(self._exc_info) File "<string>", line 3, in raise_exc_info File "C:\ProgramData\Anaconda3\lib\site-packages\tornado\gen.py", line 285, in wrapper yielded = next(result) File "C:\ProgramData\Anaconda3\lib\site-packages\notebook\services\contents\handlers.py", line 278, in post checkpoint = yield gen.maybe_future(cm.create_checkpoint(path)) File "C:\ProgramData\Anaconda3\lib\site-packages\notebook\services\contents\manager.py", line 468, in create_checkpoint return self.checkpoints.create_checkpoint(self, path) File "C:\ProgramData\Anaconda3\lib\site-packages\notebook\services\contents\filecheckpoints.py", line 56, in create_checkpoint self._copy(src_path, dest_path) File "C:\ProgramData\Anaconda3\lib\site-packages\notebook\services\contents\fileio.py", line 241, in _copy copy2_safe(src, dest, log=self.log) File "C:\ProgramData\Anaconda3\lib\site-packages\notebook\services\contents\fileio.py", line 51, in copy2_safe shutil.copyfile(src, dst) File "C:\ProgramData\Anaconda3\lib\shutil.py", line 98, in copyfile raise SameFileError("{!r} and {!r} are the same file".format(src, dst)) shutil.SameFileError: 'G:\\My Drive\\Research\\Camera_Project\\Writing_MVUE_2d.ipynb' and 'G:\\My Drive\\Research\\Camera_Project\\.ipynb_checkpoints\\Writing_MVUE_2d-checkpoint.ipynb' are the same file [E 10:49:44.698 NotebookApp] { "Cache-Control": "no-cache", "Cookie": "username-localhost-8891=\"2|1:0|10:1525749572|23:username-localhost-8891|44:ZGE5NjFkMTgwOTI3NDFjMWJhMzY1NDIwNjhlODk5ODA=|52cc6ad64c261ccd78d2f66a0c14a95edc8468e7e4ac9c27770ce2bbebd8cd55\"; _xsrf=2|f34c78bf|7c7913196386a495e7fb6a7c3c7bbf7e|1525206477; username-localhost-8890=\"2|1:0|10:1525723124|23:username-localhost-8890|44:MGFhNmE2ZjMxZmFmNDU0M2IyMTgzNWE2NDdmMmRhMDc=|73c556d569c5f55b60a18dd36f06850f90c89c2dc329e11a4041a6c5ede7cb38\"; username-localhost-8889=\"2|1:0|10:1526327000|23:username-localhost-8889|44:YjdmMTExODQyNjNmNGM5MWIzZmM0OWIzYjE3NDgzZmM=|43a55e40b88869d8e054546feca0fbf7efae9c993b450311f5046f4c0ba45a66\"; username-localhost-8888=\"2|1:0|10:1526395222|23:username-localhost-8888|44:ZDJmZWNjZWUzZDJjNDVmMjkwMTZhNWZmNTI1MjhhMTE=|9cd671557f9b67b203088e3852aaf37de56ad7c8ad45f26798bc3e0d47a49c43\"", "Referer": "http://localhost:8888/notebooks/Writing_MVUE_2d.ipynb", "X-Xsrftoken": "2|f34c78bf|7c7913196386a495e7fb6a7c3c7bbf7e|1525206477", "Origin": "http://localhost:8888", "Content-Length": "0", "Connection": "Keep-Alive", "Accept-Encoding": "gzip, deflate", "Accept": "application/json, text/javascript, */*; q=0.01", "Accept-Language": "en-US", "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.110 Safari/537.36 Edge/16.16299", "X-Requested-With": "XMLHttpRequest", "Host": "localhost:8888" } [E 10:49:44.698 NotebookApp] 500 POST /api/contents/Writing_MVUE_2d.ipynb/checkpoints (::1) 12.00ms referer=http://localhost:8888/notebooks/Writing_MVUE_2d.ipynb

非常感谢任何帮助!

Answer 1

deniz195 的一些出色的调试带来了一个很好的答案!这个临时解决方案适用于我和许多其他 github friend !

在这里引用他原来的答案:

I debugged the issue a little bit and found the following solution:

Part 1 - Minimal working example The issue seems to originate from shutil.copyfile, which checks if source and destination file are the same (using os.path.samefile).

It appears that the combination of windows and google drive file stream yields invalid results. Here is the minimum example (assuming G:\My Drive\foo.txt exists, using python 3.6.2):

>>> f1 = 'G:\\My Drive\\foo.txt'
>>> f2 = 'G:\\My Drive\\foo2.txt'
>>> import shutil
>>> shutil.copyfile(f1, f2)
>>> shutil.copyfile(f1, f2)

--> The last line throws the SameFileError although it clearly shouldnt!

Whereas:

>>> f1 = 'G:\\My Drive\\foo.txt'
>>> f3 = 'C:\\Scratch\\foo2.txt'
>>> import shutil
>>> shutil.copyfile(f1, f3)
>>> shutil.copyfile(f1, f3)

--> Throws no error (correct)!

Part 2 - Why? How does this happen?

It turns out that shutil.copyfile uses os.path.samefile to determine if a file is being copied on itself: (from https://github.com/python/cpython/blob/master/Lib/genericpath.py)

# Are two filenames really pointing to the same file?
def samefile(f1, f2):
    """Test whether two pathnames reference the same actual file"""
    s1 = os.stat(f1)
    s2 = os.stat(f2)
    return samestat(s1, s2)

# Are two stat buffers (obtained from stat, fstat or lstat)
# describing the same file?
def samestat(s1, s2):
    """Test whether two stat buffers reference the same file"""
    return (s1.st_ino == s2.st_ino and
            s1.st_dev == s2.st_dev)

Now unfortunately st_ino depends on windows on the file system (Meaning of ST_INO (os.stat() output) in Windows OS) and specifically is st_ino==0 for all files on Google Drive File Stream.

Part 3 - Where to go and who to blame? It seems that this error is an unlucky combination of a lazy filesystem (why not report some kind of uid as the inode?) and a naive os python library (Checking for file identity this way seems not to generalize well... why not check if st_ino==0?)

Part 4 - The dirty fix

1. Find genericpath.py of your python library:

>>> import os
>>> os.path.genericpath.__file__
'C:\\Program Files (x86)\\Microsoft Visual Studio\\Shared\\Anaconda3_64\\lib\\genericpath.py'

1. In this file, replace the samestat function with the following patch:

# Are two stat buffers (obtained from stat, fstat or lstat)
# describing the same file?
def samestat(s1, s2):
    """Test whether two stat buffers reference the same file"""
    return (s1.st_ino != 0 and
          s2.st_ino != 0 and
          s1.st_ino == s2.st_ino and
            s1.st_dev == s2.st_dev)

1. Save the file. Restart python (and/or jupyter).
2. Be happy and wait until either google or the python fixes this issue properly...