python - 无法加载 ipython magic 扩展 Sparkmagic : module not found

Question

我刚开始将自定义magics加载到ipython中。加载说明

sparkmagics

扩展包括已成功完成的以下步骤:

pip install sparkmagic
pip show sparkmagic
jupyter-kernelspec install sparkmagic/kernels/sparkkernel
jupyter-kernelspec install sparkmagic/kernels/pysparkkernel
jupyter-kernelspec install sparkmagic/kernels/pyspark3kernel
jupyter-kernelspec install sparkmagic/kernels/sparkrkernel

命令pip show Sparkmagic似乎很健康:

$pip show sparkmagic
Name: sparkmagic
Version: 0.10.1
Summary: SparkMagic: Spark execution via Livy
Home-page: https://github.com/jupyter-incubator/sparkmagic/sparkmagic
Author: Jupyter Development Team
Author-email: <a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="610b14111815041321060e0e060d0406130e1411124f0e1306" rel="noreferrer noopener nofollow">[email protected]</a>
License: BSD 3-clause
Location: /usr/local/lib/python2.7/site-packages
Requires: mock, requests, tornado, nose, notebook, ipywidgets, ipykernel, pandas, hdijupyterutils, autovizwidget, ipython, numpy

一些不起作用的东西 - 但列为可选的是;

$ jupyter serverextension enable --py sparkmagic

Traceback (most recent call last):
  File "/usr/local/bin/jupyter-serverextension", line 11, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.6/site-packages/jupyter_core/application.py", line 267, in launch_instance
    return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/traitlets/config/application.py", line 658, in launch_instance
    app.start()
  File "/usr/local/lib/python3.6/site-packages/notebook/serverextensions.py", line 300, in start
    super(ServerExtensionApp, self).start()
  File "/usr/local/lib/python3.6/site-packages/jupyter_core/application.py", line 256, in start
    self.subapp.start()
  File "/usr/local/lib/python3.6/site-packages/notebook/serverextensions.py", line 217, in start
    self.toggle_server_extension_python(arg)
  File "/usr/local/lib/python3.6/site-packages/notebook/serverextensions.py", line 206, in toggle_server_extension_python
    m, server_exts = _get_server_extension_metadata(package)
  File "/usr/local/lib/python3.6/site-packages/notebook/serverextensions.py", line 334, in _get_server_extension_metadata
    m = import_item(module)
  File "/usr/local/lib/python3.6/site-packages/traitlets/utils/importstring.py", line 42, in import_item
    return __import__(parts[0])
ModuleNotFoundError: No module named 'sparkmagic'

错误ModuleNotFoundError:没有名为“sparkmagic”的模块似乎是不祥的。既然 show Sparkmagic 很健康，那么为什么要在这里提示呢？ 要正确设置 sparkmagic 模块，还缺少什么？

无论如何，我也在ipython内部尝试过，并且可以预见它失败了:

%load_ext sparkmagic.magics

这是(很长)堆栈跟踪的第一部分。让我知道是否需要整个跟踪。

---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
<ipython-input-1-2b579b518c56> in <module>()
----> 1 get_ipython().magic(u'load_ext sparkmagic.magics')

/usr/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in magic(self, arg_s)
   2161         magic_name, _, magic_arg_s = arg_s.partition(' ')
   2162         magic_name = magic_name.lstrip(prefilter.ESC_MAGIC)
-> 2163         return self.run_line_magic(magic_name, magic_arg_s)
   2164
   2165     #-------------------------------------------------------------------------

/usr/local/lib/python2.7/site-packages/IPython/core/interactiveshell.pyc in run_line_magic(self, magic_name, line)
   2082                 kwargs['local_ns'] = sys._getframe(stack_depth).f_locals
   2083             with self.builtin_trap:
-> 2084                 result = fn(*args,**kwargs)
   2085             return result
   2086

<decorator-gen-64> in load_ext(self, module_str)

/usr/local/lib/python2.7/site-packages/IPython/core/magic.pyc in <lambda>(f, *a, **k)
    191     # but it's overkill for just that one bit of state.
    192     def magic_deco(arg):
--> 193         call = lambda f, *a, **k: f(*a, **k)
    194
    195         if callable(arg):

/usr/local/lib/python2.7/site-packages/IPython/core/magics/extension.pyc in load_ext(self, module_str)
     64         if not module_str:
     65             raise UsageError('Missing module name.')
---> 66         res = self.shell.extension_manager.load_extension(module_str)
     67
     68         if res == 'already loaded':

总而言之，安装/使用任何魔法可能通常需要一个步骤。感谢指点。

Answer 1

从我在错误和安装消息中看到的情况来看，您安装了 python2.7 的 Spark magic 并尝试在 python 3.6 中运行它。这很令人困惑，当我开始时，我也多次遇到同样的问题。我建议您通过在笔记本上运行 !python --version 来检查您正在使用的 python 版本。如果 python 版本正确，则使用 !pip install Sparkmagic 安装 Sparkmagic，并检查安装消息是否安装到正确的 python 版本。

您还可以设置一个环境，让生活变得更加轻松。更多这里https://www.youtube.com/watch?v=LjQlmee58hg