python - TypeError : Level type mismatch: 0. 2.将数据拆分为训练集、验证集和测试集时

Question

美好的一天

我尝试在不使用 scikit-learn 的情况下训练、验证和测试数据。 我希望将数据分成以下样本:

• 训练数据0.7(70%)
• 验证数据 0.2(20%)
• 测试数据0.1(10%)

但是，当我尝试拆分数据时，出现以下错误:

 TypeError: Level type mismatch: 6.0

我需要帮助来理解我在这里做错了什么。样本数据和目标数据分别是数据框的x_data和Pandas系列的y_data。这是我在下面尝试的代码:

def train_valid_test(x_data y_data, train_split, valid_split, test_split):

    """ Parameters
    x_data: the input data
    y_data: target values 
    train_split: the portion used for training data 
    valid_split: the portion used for validating data
    test_split: the portion used for testing data 
    """ 

    # setting sizes to split the data into training validating and testing samples accordingly 
    train_size = float(len(all_x)*0.7)
    valid_size = float(len(all_x)*0.2)
    test_size = float(len(x_prime)*0.1)

    # Creating Training and Validation sets
    x_train, x_prime = x_data[:valid_size], x_data[valid_size:]
    y_train, y_prime = y_data[:valid_size], y_data[valid_size:]

    # Creating test sets
    x_valid, x_test = x_prime[:test_size], x_prime[test_size:]
    y_valid, y_test = y_prime[:test_size], y_prime[test_size:]


    # Return the samples 
    return X_train, X_valid, X_test, y_train, y_valid, y_test

Answer 1

您正在尝试使用 float 对 pandas 数据帧进行切片，因为以下操作会生成训练、验证和测试数据大小的非整数值:

train_size = float(len(all_x)*0.7)
valid_size = float(len(all_x)*0.2)
test_size = float(len(x_prime)*0.1)

请注意，您的分割不正确；您的训练集包括验证集和测试集的所有数据点，而验证集则包含测试集的所有实例。此外，您永远不应该依赖不会打乱数据集的分割。下面的函数应该能帮到你。

import numpy as np
import pandas as pd

def train_valid_test(df, train_split=.7, valid_split=.2, seed=None):    
    np.random.seed(seed)
    perm = np.random.permutation(df.index)

    training_max_index = int(train_split * len(df.index))
    validate_max_index = int(valid_split * len(df.index)) + training_max_index

    training = df.ix[perm[:training_max_index]]
    validation = df.ix[perm[training_max_index:validate_max_index]]
    test = df.ix[perm[validate_max_index:]]

    return training, validation, test

如果您想分别传递从属变量 (y) 和独立变量 (x)，您可以使用以下函数:

import numpy as np
import pandas as pd

def train_valid_test(x_data, y_data, train_split=.7, valid_split=.2, seed=None):
    if len(x_data.index) != len(y_data.index):
        raise Exception('x_data and y_data must contain the same number of data points'

    np.random.seed(seed)
    perm = np.random.permutation(x_data.index)
    x_data = x_data.reindex(perm)
    y_data = y_data.reindex(perm)

    training_max_index = int(train_split * len(x_data.index))
    validate_max_index = int(valid_split * len(x_data.index)) + training_max_index

    X_train, y_train = x_data[:training_max_index], y_data[:training_max_index]
    X_valid, y_valid = x_data[:validate_max_index], y_data[:validate_max_index]
    X_test, y_test = x_data[validate_max_index:], y_data[validate_max_index:]

    return X_train, X_valid, X_test, y_train, y_valid, y_test