php - 使用 PHP 和 xPath 从 HTML 中提取数据

Question

我正在尝试从网页中提取数据并将其插入数据库。我感兴趣的数据位于具有 class="company"的 div 中。在一个网页上有 15 个或更少的这样的 div，并且我试图从很多页面中提取这些数据。为此，我试图找到一种自动数据提取解决方案。

带有 class="company"的 div 如下(一页上有 15 个或更少的这样的 div，具有不同的数据):

<div class="company" id="company-6666"> <!-- EXTRACT 'company-6666' from id="company-6666" -->

  <div class="top clearfix">
    <div class="name clearfix">
      <h2>
        <a href="/company-name">Company Name</a>&nbsp; <!-- EXTRACT 'Company Name' from contents of A element and EXTRACT '/company-name' from href attribute -->
        <a href="/branches-list-link?parent_id=6666" class="branches">Branches <span>(5)</span></a> <!-- EXTRACT '/branches-list-link?parent_id=6666' from href attribute -->               
      </h2>
    </div>
  </div>

  <div class="inner clearfix has-logo">

    <div class="clearfix">          
      <div class="logo">
        <a href="/company-name">
          <img src="/graphics/company/logo/listing/123456.jpg?_ts=1365390237" border="0" alt="" /> <!-- EXTRACT '/graphics/company/logo/listing/123456.jpg?_ts=1365390237' from src attribute -->
        </a>
      </div>
      <div class="info">
        <div class="address">StreetName 500, 7777 City, County</div> <!-- EXTRACT 'StreetName 500, 7777 City, County' from contents of class="address" div -->
        <div class="clearfix">
          <div class="slogan">Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi ac condimentum mi.</div> <!-- EXTRACT 'Lorem ipsum dolor sit amet, consectetur adipiscing elit. Morbi ac condimentum mi.' from contents of class="slogan" div -->
        </div>
      </div>
    </div>

    <div class="actions-bar clearfix">
      <ul>              
        <li><span class="phone-number">6666666</span></li> <!-- EXTRACT '6666666' from contents of class="phone-number" div -->
        <li><a href="mailto:<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="59343830351934383035773a3634" rel="noreferrer noopener nofollow">[email protected]</a>" target="_blank" title="<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="1e737f77725e737f7772307d7173" rel="noreferrer noopener nofollow">[email protected]</a>" class="email"><a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="ed808c8481ad808c8481c38e8280" rel="noreferrer noopener nofollow">[email protected]</a></a></li> <!-- EXTRACT '<a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="80ede1e9ecc0ede1e9ecaee3efed" rel="noreferrer noopener nofollow">[email protected]</a>' from contents of class="email" div -->
        <li><a href="http://www.webpage.com" target="_blank" title="www.webpage.com" class="redirect url">www.webpage.com</a></li> <!-- EXTRACT 'www.webpage.com' from contents of class="redirect url" div -->
      </ul>
    </div>

  </div>

</div>

到目前为止，我有以下 PHP 代码($output 具有网页的 HTML 代码):

<?php

$doc = new DomDocument();
@$doc->loadHTML($output);
$doc->preserveWhiteSpace = false; 

$xpath = new DomXPath($doc);

$elements = $xpath->query("//*[@class='company']");

if (!is_null($elements)) {
    foreach ($elements as $element) {
        echo $element->nodeValue;
    }
}

?>

它似乎获得了 class="company"的所有 15 个 div，但我不知道如何提取前面提到的(在 HTML 代码的注释中)各个值。

每个 div(我指的是 class="company"的 div)并不包含 HTML block 中写入的所有值。因此，我必须以某种方式查询公司 div 内的特定 div(我感兴趣的数据)是否存在，如果存在，我必须检查它是否不为空(是否包含标签之间的文本)。如果它存在并且不为空，我会将其添加到变量中。

提取值后，我想将它们分配给 PHP 变量，这样我就可以在以后使用它们。如果将提取的值放入数组中，如下所示，那就更好了:

$result = array(
    // 1'st div's data
    [0] =>  
        'company name' => 'company name',
        'company link' => 'company link',
        'company id' => 'company id',
        'company branches'  => 'branches link',
        'company logo'  => 'logo',
        'company address'  => 'address',
        'company slogan'  => 'slogan',
        'company webpage'  => 'webpage',
        'company email'  => 'email',
        'company phone'  => 'phone'

    // 2'nd div's data
    [1] =>  
        'company name' => 'company name',
        'company link' => 'company link',
        'company id' => 'company id',
        'company branches'  => 'branches link',
        'company logo'  => 'logo',
        'company address'  => 'address',
        'company slogan'  => 'slogan',
        'company webpage'  => 'webpage',
        'company email'  => 'email',
        'company phone'  => 'phone'
    ...
    )

Answer 1

每个公司都可以由上下文节点表示，同时每个属性由与其相关的 xpath-表达式表示:

Company company-6666:
 ->id ....... = "company-6666"    --    string(@id)
 ->name ..... = "Company Name"    --    .//a[1]/text()
 ->href ..... = "/company-name"    --    .//a[1]/@href
 ->img ...... = "/graphics/company/logo/listing/123456.jpg?_ts=1365390237"    --    .//img[1]/@src
 ->address .. = "StreetName 500, 7777 City, County"    --    .//*[@class="address"]/text()
 ...

如果将其包装到对象中，那么使用起来非常漂亮:

$doc = new DOMDocument();
$doc->loadHTML($html);

/* @var $companies DOMValueObject[] */
$companies = new Companies($doc);

foreach ($companies as $company) {
    printf("Company %s:\n", $company->id);
    foreach ($company->getObjectProperties() as $name => $value) {
        $expression = $company->getPropertyExpression($name);
        printf(" ->%'.-10s = \"%s\"    --    %s\n", $name.' ', $value, $expression);
    }
}

这适用于 DOMObjectCollection 和 DOMValueObject，定义您自己的类型:

class Companies extends DOMValueCollection
{
    public function __construct(DOMDocument $doc) {
        parent::__construct($doc, '//*[@class="company"]');
    }

    /**
     * @return DOMValueObject
     */
    public function current() {
        $object = parent::current();
        $object->defineProperty('id', 'string(@id)');
        $object->defineProperty('name', './/a[1]/text()');
        $object->defineProperty('href', './/a[1]/@href');
        $object->defineProperty('img', './/img[1]/@src');
        $object->defineProperty('address', './/*[@class="address"]/text()');
        # ... add your definitions
        return $object;
    }
}

为了满足您的数组要求，有一个 getArrayCopy() 方法:

echo "\nGet Array Copy:\n\n";

print_r($companies->getArrayCopy());

输出:

Get Array Copy:

Array
(
    [0] => Array
        (
            [id] => company-6666
            [name] => Company Name
            [href] => /company-name
            [img] => /graphics/company/logo/listing/123456.jpg?_ts=1365390237
            [address] => StreetName 500, 7777 City, County
        )

)