python - 将参数值从文件传递到 numba 类

Question

我正在用 python 构建一个 numba 模拟类，我想将早期模拟中存储在文件中的一些参数值传递给它。文件中保存了很多参数值，因此它们位于我使用 pandas 访问的文件中。

不幸的是，尝试在 numba 中使用 pandas 会引发错误。具体来说，numba 似乎对我在其中调用 pandas 并分配值感到不满，但不确定如何键入它们。 (注意:如果我将这些链接的 pandas 调用分成多行，它会将 read_feather 行识别为罪魁祸首):

from collections import OrderedDict
from numba import jitclass, float32, int32
from pandas import read_feather

import os
PARAM_FILE = SOME_PATH_ON_MY_MACHINE

import numba
print(numba.__version__)

@jitclass(
    spec=OrderedDict(
        a=float32,
        b=float32,
    )
)
class MyParameters:
    def __init__(self,) -> None:
        simulated_parameters = read_feather(PARAM_FILE).sample(1, axis=0).squeeze().to_dict()
        self.a = simulated_parameters["a"]
        self.b = simulated_parameters["a"]

demo_bug = MyParameters()

### OUTPUT
0.45.1
Traceback (most recent call last):
  File "mwe-numba.py", line 42, in <module>
    demo_bug = MyParameters()
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/jitclass/base.py", line 126, in __call__
    return cls._ctor(*bind.args[1:], **bind.kwargs)
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/dispatcher.py", line 376, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/dispatcher.py", line 343, in error_rewrite
    reraise(type(e), e, None)
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/six.py", line 658, in reraise
    raise value.with_traceback(tb)
numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
Untyped global name 'read_feather': cannot determine Numba type of <class 'function'>

File "mwe-numba.py", line 32:
    def __init__(self,) -> None:
        simulated_parameters = read_feather(PARAM_FILE).sample(1, axis=0).squeeze().to_dict()
        ^

[1] During: resolving callee type: jitclass.MyParameters#10116c320<a:float32,b:float32,c:float32,tstar:int32,cstar:float32,loc:float32,scale:float32,shape:float32>
[2] During: typing of call at <string> (3)

[3] During: resolving callee type: jitclass.MyParameters#10116c320<a:float32,b:float32,c:float32,tstar:int32,cstar:float32,loc:float32,scale:float32,shape:float32>
[4] During: typing of call at <string> (3)


File "<string>", line 3:
<source missing, REPL/exec in use?>

This is not usually a problem with Numba itself but instead often caused by
the use of unsupported features or an issue in resolving types.

To see Python/NumPy features supported by the latest release of Numba visit:
http://numba.pydata.org/numba-doc/latest/reference/pysupported.html
and
http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html

For more information about typing errors and how to debug them visit:
http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile

If you think your code should work with Numba, please report the error message
and traceback, along with a minimal reproducer at:
https://github.com/numba/numba/issues/new

为了变得聪明，我决定创建另一个类来保存参数信息，然后将 read_feather 放在那里。然而，这也不起作用

class MyParameters:
    def __init__(self) -> None:
        simulated_parameters = read_feather(PARAM_FILE).sample(1, axis=0).squeeze().to_dict()
        self.a = simulated_parameters["a"]
        self.b = simulated_parameters["a"]

@jitclass(
  OrderedDict(
    a=float32,
    b=float32,
  )
)
class MyModel:
  def __init__(self, param: MyParameters) -> None:
    self.a = param.a
    self.b = param.b

my_param = MyParameters()
print("That was no problem")
my_model = MyModel(param=my_param)

### OUTPUT
0.45.1
That was no problem
Traceback (most recent call last):
  File "also-error.py", line 40, in <module>
    my_model = MyModel(param=my_param)
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/jitclass/base.py", line 126, in __call__
    return cls._ctor(*bind.args[1:], **bind.kwargs)
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/dispatcher.py", line 376, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/dispatcher.py", line 343, in error_rewrite
    reraise(type(e), e, None)
  File "/usr/local/miniconda3/envs/deeprl/lib/python3.7/site-packages/numba/six.py", line 658, in reraise
    raise value.with_traceback(tb)
numba.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
non-precise type pyobject
[1] During: typing of argument at <string> (3)

File "<string>", line 3:
<source missing, REPL/exec in use?>

This error may have been caused by the following argument(s):
- argument 0: cannot determine Numba type of <class '__main__.MyParameters'>

This is not usually a problem with Numba itself but instead often caused by
the use of unsupported features or an issue in resolving types.

To see Python/NumPy features supported by the latest release of Numba visit:
http://numba.pydata.org/numba-doc/latest/reference/pysupported.html
and
http://numba.pydata.org/numba-doc/latest/reference/numpysupported.html

For more information about typing errors and how to debug them visit:
http://numba.pydata.org/numba-doc/latest/user/troubleshoot.html#my-code-doesn-t-compile

If you think your code should work with Numba, please report the error message
and traceback, along with a minimal reproducer at:
https://github.com/numba/numba/issues/new

我可以做些什么来将此代码放入 numba 中吗？我可以想到解决方法，但它们都有点笨拙。

Answer 1

这个post强调 numba 最适合标量和向量运算，而不是数据整理，因此不支持 pandas。

另一个问题是，您必须将 dict 本质上作为类 __init__ 中的 kwargs 传递，此 issue Github 上解释了为什么这是一个问题。这种通用参数处理需要在 nopython 模式函数内以 Python 模式完成(例如，从 kwargs 中弹出项目)，目前不存在此功能。

这意味着当您在 OrderedDict 中进行解析时，您本质上必须告诉 numba 期待一个 python dict ，这实际上不会使有道理，但是 numba 有 Dict functionality自 0.43 起。 SO post example .

但是，如果您尝试这样做，将会导致以下错误:

加载模块

from collections import OrderedDict
from numba import jitclass
from numba import types 
from numba.typed import Dict

@jitclass( 
    OrderedDict([ 
        ('params', Dict.empty( 
            key_type=types.unicode_type,
            value_type=types.float64,
        )) 
     ]) 
) 
class MyModel: 
    def __init__(self, params): 
        self.a = params["a"] 
        self.b = params["b"] 

TypeError: spec values should be Numba type instances, got DictType[unicode_type,float64]({})

本质上，您必须告诉 numba 期望什么 args，在您的情况下，我会执行以下操作:

假设您使用了 read_feather，生成以下 dict:

simulated_parameters = {"a" : 2.0, "b" : 5.0}

现在初始化 numba 类:

@jitclass( 
    OrderedDict([ 
        ('a', types.float32), 
        ('b', types.float32) 
    ]) 
)
class MyModel: 
    def __init__(self, a, b): 
        self.a = a 
        self.b = b

foo = MyModel(simulated_parameters["a"], simulated_parameters["b"])