php - 在 php 中 curl 5,000 多个 url 并将它们插入 mysql 数据库的最快方法？

Question

我正在使用 php curl 从 api 获取数据

第一个 url 返回 50 个 url，这 50 个 url 中的每一个返回 500 个结果和下一个 url，直到没有更多结果为止。

我目前使用的代码需要 5 个多小时才能完成，因为有超过 200 万条记录要从 5,000 多个 http 请求中插入到 mysql 中。

目前我正在使用ParallelCurl从 petewarden 上课 -> https://github.com/petewarden/ParallelCurl

这是我的完整代码:

<?php
ini_set('memory_limit', '3000M');
ini_set('max_execution_time', 15000);
require_once('parallelcurl.php');
$host = "localhost";
$user = "user";
$pass = "pass";
$dbname = "db";
try {
# MySQL with PDO_MYSQL
$DBH = new PDO("mysql:host=$host;dbname=$dbname", $user, $pass);
}
catch(PDOException $e) {
echo $e->getMessage();
}

$table="table";
$nextUrl = 0;

function httpGet($url)
{
$ch = curl_init();

curl_setopt($ch,CURLOPT_URL,$url);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,true);
$output=curl_exec($ch);

curl_close($ch);
return $output;
}

$html = httpGet("https://url1.com");
$arr = json_decode($html, true);
$curl_options = array(
    CURLOPT_RETURNTRANSFER => true,
);
$max_requests = 20;
$parallel_curl = new ParallelCurl($max_requests, $curl_options);
foreach ($arr['apiGroups'] as $key => $category) {

    $getUrl = $category['availableVariants']['get'];

        $parallel_curl->startRequest($getUrl, 'on_request_done');
            while ($nextUrl) {
                $parallel_curl->startRequest($nextUrl, 'on_request_done');
                }

    }
function placeholders($text, $count=0, $separator=","){
$result = array();
if($count > 0){
for($x=0; $x<$count; $x++){
$result[] = $text;
}
}
return implode($separator, $result);
}

// This function gets called back for each request that completes
function on_request_done($content, $url, $ch) {
global $DBH, $parallel_curl, $table, $nextUrl;

$arr = json_decode($content, true);

$j=0;
foreach ($arr['productInfoList'] as $key => $item) {

    $title = $item['productBaseInfo'];
    $url = $item['productUrl'];
    $img = $item['imageUrls']['275x275'];
    $price = $item['sellingPrice']['amount'];
    $pid = $item['productId'];

    $selarr[$j] = array('title' => $title, 'url' => $url, 'imgurl' => $img, 'price' => $price, 'productid' => $pid);
    $j++;
    }
        $datafields = array('title' => '', 'url' => '', 'imgurl' => '', 'price' => '', 'productid' => '' );

        $insert_values = array();
        foreach($selarr as $d){
        $question_marks[] = '('  . placeholders('?', sizeof($d)) . ')';
        $insert_values = array_merge($insert_values, array_values($d));
        }
        $DBH->beginTransaction(); // also helps speed up your inserts
        $sql = "INSERT INTO $table (" . implode(",", array_keys($datafields) ) . ") VALUES " . implode(',', $question_marks);
        $stmt = $DBH->prepare ($sql);
        try {
        $stmt->execute($insert_values);
        } catch (PDOException $e){
        echo $e->getMessage();
        }
        $DBH->commit();


if($arr['nextUrl']) {
        $nextUrl = $arr['nextUrl'];
    }
else {
    $nextUrl = 0;
}
}

$parallel_curl->finishAllRequests();
?>

Answer 1

生成同一 PHP 脚本的多个实例(从 CLI 运行它)并使用集中队列来管理不在 url 上重叠的作业...

我通常使用 redis 作为我的队列来执行此操作，您甚至可以考虑使用现成的库，例如 this .

应该很容易实现...

// pseudocode, in your jobs
$batch = $queue->getFirstAvailableBatchOfUrls();

// Do your stuff

// put again in the queue if errors occurs...
if($status == PROCESS_DONE_OK) {
   $batch->markAsDone()
} else {
   $batch->putAgainInQueue()
}