php - 如何使用 faker 在 Laravel 中更快地播种大型数据集

Question

我使用 laravel faker 和模型工厂将 25000 条记录植入到 1 个表中，需要近 30 分钟才能完成，这只会延迟我的工作效率。有没有一种方法可以在更短的时间内播种如此大的数据集。一分钟左右。

这是模型工厂

<?php

use App\Models\Client\Client;
use Faker\Generator;

$factory->define(Client::class, function (Generator $faker) {

    $employeecount  = [15,48,5,12,26,30];
    $numbers        = [0125554,0145755,0123456,8874545,4512156];
    $gross          = [25000,13500,10000,8500,5000,15000,30000];
    $gender         = ['Male','Female'];
    $marketing      = ['E-mail','SMS','Telephone'];
    $maritalstatus  = ['Single','Married','Divorced','Widow'];
    $maritaltype    = ['COP','ANC'];
    $addresstype    = ['Owner','Rent'];
    $serviceused    = ['Status Insight','Credit Union'];
    $riskstatus     = ['Low Risk','Medium Risk','High Risk'];
    $companytype    = ['Public Company','CC','Partnership','Sole Trader','PTY','TLD','Other'];



    return [
        'name'             =>$faker->firstName,
        'surname'          =>$faker->lastName,
        'idnumber'         =>$faker->unique()->ean8,
        'gender'           =>$gender[array_rand($gender)],
        'maritalstatus'    =>$maritalstatus[array_rand($maritalstatus)],
        'maritaltype'      =>$maritaltype[array_rand($maritaltype)],
        'maidenname'       =>$faker->lastName,
        'companytype'      =>$companytype[array_rand($companytype)],
        'companyname'      =>$faker->company,
        'businessnature'   =>$faker->catchPhrase,
        'vatnumber'        =>$numbers[array_rand($numbers)],
        'coregnumber'      =>$numbers[array_rand($numbers)],
        'tradingname'      =>$faker->bs,
        'employeecount'    =>$employeecount[array_rand($employeecount)],
        'employer'         =>$faker->company,
        'gross'            =>$gross[array_rand($gross)],
        'mobilenumber'     =>$numbers[array_rand($numbers)],
        'homenumber'       =>$numbers[array_rand($numbers)],
        'worknumber'       =>$numbers[array_rand($numbers)],
        'email'            =>$faker->email,
        'marketing'        =>$marketing[array_rand($marketing)],
        'physicaladd1'     =>$faker->streetAddress,
        'physicaladd2'     =>$faker->streetName,
        'physicalcity'     =>$faker->city,
        'physicalcode'     =>str_pad(rand(0, pow(10, 4) - 1), 4, '0', STR_PAD_LEFT),
        'yearsAtAddress'   =>str_pad(rand(0, pow(4, 2) - 1), 2),
        'monthsAtAddress'  =>str_pad(rand(0, pow(4, 2) - 1), 2),
        'addresstype'      =>$addresstype[array_rand($addresstype)],
        'postaladd1'       =>$faker->streetAddress,
        'postaladd2'       =>$faker->streetName,  
        'postalcity'       =>$faker->city,
        'postalcode'       =>str_pad(rand(0, pow(10, 4) - 1), 4, '0', STR_PAD_LEFT),
        'riskstatus'       =>$riskstatus[array_rand($riskstatus)],
        'noofjudgements'   =>str_pad(rand(0, pow(2, 2) - 1), 2),
        'noofdefaults'     =>str_pad(rand(0, pow(2, 2) - 1), 2),
        'dateofenquiry'    =>$faker->date($format = 'Y-m-d', $max = 'now'),
        'serviceused'      =>$serviceused[array_rand($serviceused)],

    ];

});

这是表格播种机

<?php

use App\Models\Client\Client;
use Illuminate\Database\Seeder;

class ClientTableSeeder extends Seeder
{
    /**
     * Run the database seeds.
     *
     * @return void
     */
    public function run()
    {
        factory(Client::class, 25000)->create();
    }
}

这是主播种机

<?php

use Illuminate\Database\Eloquent\Model;
use Illuminate\Database\Seeder;

/**
 * Class DatabaseSeeder.
 */

class DatabaseSeeder extends Seeder
{

    /**
     * Run the database seeds.
     *
     * @return void
     */
    public function run()
    {
        Model::unguard();

         $this->call(ClientTableSeeder::class);

        Model::reguard();
    }
}

谁有更好的解决方案？ 提前致谢。

Answer 1

根据我的长期经验，工厂不是这种情况下的最佳选择。
我发现 Seeder 是比 Factory 更快的解决方案。
我所做的是创建一个巨大的数据集合，例如 25000~50000 条记录。
然后使用 Collection::chunk 一次插入(而不是创建)许多(500 ~ 1000+)条记录。
最终减少查询数量和时间。

例如

        $collectionToInsert = collect([]);
    
        $numberOfEntries = 25000;

        for ($i = 1; $i <= $numberOfEntries; $i++) {
            $collectionToInsert->push([
                'name' => $faker->firstName,
                'surname' => $faker->lastName,
                'idnumber' => $faker->unique()->ean8,
                // ...
            ]);
        }
        
        $chunks = $collectionToInsert->chunk(500);
        
         foreach ($chunks as $chunk) {
             SomeModel::insert($chunk->toArray());
         }