python - 转换新数据集以在 Python 中进行预测

Question

我用 *pd.get_dummies* 之类的迭代训练模型(足够的 linear_model.LinearRegression)

我得到了新的数据结构 现在我使用一个新的数据集并想要进行预测。我不能使用 predict 因为结构不同。 *pd.get_dummies*新数据将为我们提供另一个列数

如何转换此数据集？ 通过追加到先前的数据集并使用新数据再次训练？ 或者我可以对新数据使用“转换”吗？

 import pandas as pd 
import numpy as np
from sklearn import linear_model
df1 = pd.DataFrame({ 'y' : np.array([1., 1., 2., 3., 1.] ,dtype='int32'),
   ....:                      'X' : np.array(["1","1","2","2", "1"])})
y = df1[df1.columns[0]]
X = pd.get_dummies(df1['X'])
lr = linear_model.LinearRegression()
lr = lr.fit(X, y)
lr.predict(X)

现在我有

df2 = pd.DataFrame({ 'y' : 'nan',
   ....:                      'X' : np.array(["3"])})
Xnew = pd.get_dummies(df2['X'])
lr.predict(Xnew)
ValueError: shapes (1,1) and (2,) not aligned: 1 (dim 1) != 2 (dim 0)

Answer 1

我是这样看的

import numpy as np
import pandas as pd
from sklearn import linear_model, metrics, pipeline, preprocessing
df = pd.DataFrame({'a':range(12), 'b':[1,2,3,1,2,3,1,2,3,3,1,2], 'c':['a', 'b', 'c']*4, 'd': ['m', 'f']*6})
y = df.a
num = df[['b']]
cat = df[['c', 'd']]
from sklearn.feature_extraction import DictVectorizer
enc = DictVectorizer(sparse = False)
enc_data = enc.fit_transform(cat .T.to_dict().values())
crat = pd.DataFrame(enc_data,  columns=enc.get_feature_names())
X = pd.concat([crat, num], axis=1)
cat_columns = ['c=a', 'c=b', 'c=c', 'd=f', 'd=m'] 
cat_indices = np.array([(column in cat_columns) for column in X.columns], dtype = bool)
numeric_col = ['b']
num_indices = np.array([(column in numeric_col) for column in X.columns], dtype = bool)
reg = linear_model.SGDRegressor()
estimator = pipeline.Pipeline(steps = [       
    ('feature_processing', pipeline.FeatureUnion(transformer_list = [        
            ('categorical', preprocessing.FunctionTransformer(lambda data: data[:, cat_indices])), 

            #numeric
            ('numeric', pipeline.Pipeline(steps = [
                ('select', preprocessing.FunctionTransformer(lambda data: data[:, num_indices])),
                ('scale', preprocessing.StandardScaler())            
                        ]))
        ])),
    ('model', reg)
    ]
)
estimator.fit(X, y)

现在我们使用一个新的数据集

test = pd.DataFrame({ 'b':[1], 'c':['a'], 'd': ['f']})
cat = test[['c', 'd']]
num = test[['b']]
enc_data = enc.transform(cat.T.to_dict().values())
crat = pd.DataFrame(enc_data,  columns=enc.get_feature_names())
test = pd.concat([crat, num], axis=1)
estimator.predict(test)