php - php 读取 1Gb 文件的内存限制应该是多少？

Question

我有一个必须读取的 1Gb 文件 1。我选择 php 来过滤和更改一些行，而不是使用这些更改创建另一个 file2。代码很好。如果我读取 50M 文件，效果很好。正如预期的那样，代码生成了一个包含所有更改的文件 2。但是当我尝试运行 1Gb 文件时，没有创建 file2 并且我从浏览器中收到如下错误消息:

The connection to localhost was interrupted.
Check your Internet connection
Check any cables and reboot any routers, modems, or other network devices you may be using.
Allow Chrome to access the network in your firewall or antivirus settings.
If it is already listed as a program allowed to access the network, try removing it from the list and adding it again.
If you use a proxy server...
Check your proxy settings or contact your network administrator to make sure the proxy server is working. If you don't believe you should be using a proxy server: Go to the Chrome menu > Settings > Show advanced settings... > Change proxy settings... > LAN Settings and deselect "Use a proxy server for your LAN".

如果我返回并运行小文件，它会再次正常运行。 我已经将 php memeory 设置为 2040M ini_set('memory_limit', '2048M') 但我不知道它是否足够或是否可能。

那么，对于这个问题应该怎样方便内存呢？

---

注意:服务器是apache, win7, i7 8cores 64bits, 16G RAM。 我认为代码并不重要，但有人要求查看它:

                       ini_set('memory_limit', '2048M');#Set the memory limit
                              $new_dbName="au_site";
                              $patern[0]="/di_site/";
                              $replace[0]=$new_dbName;
                              #xdebug_break();
                              $dirBase=dirname(__FILE__);
                              $dir2 = new DirectoryIterator(dirname(__FILE__));
                              #xdebug_break();
                              foreach ($dir2 as $fileinfo) {
                                     if (!$fileinfo->isDot() && $fileinfo->isFile()) {
                                          $str = $fileinfo->getFilename();
                                          if (preg_match("/\.sql/i", $str)) {
                                               #xdebug_break();
                                               $i=1;
                                               if(!($handle= fopen("$str", "r"))){

                                                die("Cannot open the file");
                                               }
                                               else{
                                                     while (!feof($handle)) {   
                                                             #xdebug_break();
                                                             $line=trim(fgets($handle), "\t\n\r\0\x0B");
                                                             $firstChar =  substr($line, 0,1) ;
                                                             $ord = ord($firstChar);
                                                             if(ord($firstChar)<>45){
                                                               if (preg_match("/di_site/", $line)) {
                                                                xdebug_break();
                                                                   $chn=preg_replace($patern, $replace, $line); 
                                                                   $line=$chn;
                                                               }
                                                               #echo $line."<br>";
                                                               $sql.=$line."\n";
                                                             }                                                            
                                                      }
                                                      xdebug_break();
                                                      $newDBsql=$dirBase."/".$new_dbName.".sql";
                                                      if(!$handle =  fopen($newDBsql,"w")){
                                                           die("Can not open the file");
                                                      }
                                                      else{
                                                       fwrite($handle, $sql);
                                                       fclose($handle);
                                                      }

                                               }
                                            } 
                                     }
                               } 

Answer 1

与其构建您要写入的整个文件内容(这需要大量内存)，不如考虑编写 stream filter相反。

流过滤器对来自底层流的单个缓冲读取进行操作，通常是大约 8kB 的数据。下面的示例代码定义了这样一个过滤器，它将每个桶分成单独的行并调用您的代码对其进行更改。

<?php

class myfilter extends \php_user_filter
{
  private $buffer; // internal buffer to create data buckets with

  private $pattern = ['/di_site/'];
  private $replace = ['au_site'];

  function filter($in, $out, &$consumed, $closing)
  {
    while ($bucket = stream_bucket_make_writeable($in)) {
      $parts = preg_split('/(\n|\r\n)/', $bucket->data, -1, PREG_SPLIT_DELIM_CAPTURE);
      $buffer = '';
      // each line spans two array elements
      for ($i = 0, $n = count($parts); $i + 1 < $n; $i += 2) {
        $line = $parts[$i] . $parts[$i + 1];
        $buffer .= $this->treat_line($line);
        $consumed += strlen($line);
      }
      stream_bucket_append($out, stream_bucket_new($this->stream, $buffer));
    }
    return PSFS_PASS_ON;
  }

  /** THIS IS YOUR CODE **/
  function treat_line($line)
  {
    $line = trim($line, "\t\n\r\0\x0B");
    $firstChar =  substr($line, 0,1) ;
    if (ord($firstChar)<>45) {
      if (preg_match("/di_site/", $line)) {
        $line = preg_replace($this->pattern, $this->replace, $line);
      }
    }
    return $line . "\n";
  }

  function onCreate()
  {
    $this->buffer = fopen('php://memory', 'r+');
  }

  function onClose()
  {
    fclose($this->buffer);
  }
}

stream_filter_register("myfilter", "myfilter");

// open input and attach filter
$in = fopen(__FILE__, 'r');
stream_filter_prepend($in, 'myfilter');
// open output stream and start copying
$out = fopen('php://stdout', 'w');
stream_copy_to_stream($in, $out);

fclose($out);
fclose($in);