C++程序大数据内存不足

Question

我正在尝试解决我编写的 C++ 程序中的问题。我基本上内存不足。该程序是一个缓存模拟器。有一个文件预先收集了内存地址，如下所示:

线程地址类型大小指令指针
0 0x7fff60000000 1 8 0x7f058c482af3

可以有 100-5000 亿这样的条目。首先，我试图读取所有这些条目并将其存储在一个 vector 中。同样在阅读时，我建立了一组这些地址(使用 map )，并存储特定地址的序列号。序列号只是表示地址条目在文件中的位置(一个地址可以多次看到)。对于大输入，程序在执行此操作时会失败，并在大约第 30 百万个条目处出现 bad_alloc 错误。我想我的内存用完了。请告诉我如何规避这个问题。有没有另一种方法来处理这种大数据。非常感谢!很抱歉这篇文章很长。我想提供一些上下文和我正在编写的实际代码。

下面是相关代码。 ParseTaceFile() 读取每一行并调用 StoreTokens()，它获取地址和大小，并调用 AddAddress()，它实际上将地址存储在 vector 和映射中。类声明也在下面给出。 AddAddress() 中的第一个 try block 实际上抛出了 bad_alloc 异常。

void AddressList::ParseTraceFile(const char* filename) {
  std::ifstream in_file;
  std::cerr << "Reading Address Trace File..." << std::endl;
  in_file.exceptions(std::ifstream::failbit | std::ifstream::badbit);
  char *contents = NULL;
  try {
    in_file.open(filename, std::ifstream::in | std::ifstream::binary);
    in_file.seekg(0, std::ifstream::end);
    std::streampos length(in_file.tellg());
    if (length < 0) {
      std::cerr << "Can not read input file length" << std::endl;
      throw ExitException(1);
    }
    contents = (new char[length]);
    in_file.seekg(0, std::ifstream::beg);
    in_file.read(contents, length);
    in_file.close();
    uint64_t linecount = 0, i = 0, lastline = 0, startline = 0;
    while (i < static_cast<uint64_t>(length)) {
      if ((contents[i] == '\n') or (contents[i] == EOF)) {
        contents[i] = '\0';
        lastline = startline;
        startline = i + 1;
        ++linecount;
        if (linecount > 1) {
          StoreTokens((contents + lastline), &linecount);
        }
      }
      ++i;
    }
  } catch (std::bad_alloc& e) {
    delete [] contents;
    std::cerr << "error allocating memory while parsing" << std::endl;
    throw;
  } catch (std::ifstream::failure &exc1) {
    if (!in_file.eof()) {
      delete[] contents;
      std::cerr << "error in reading address trace file" << exc1.what()
          << std::endl;
      throw ExitException(1);
    }
  }
  std::cerr << "Done" << std::endl;
}
//=========================================================    
void AddressList::StoreTokens(char* line, uint64_t * const linecount) {
  uint64_t address, size;
  char *token = strtok(line, " \t");
  uint8_t tokencount = 0;
  while (NULL != token) {
    ++tokencount;
    switch (tokencount) {
    case 1:
      break;
    case 2:
      address = strtoul(token, NULL, 16);
      break;
    case 3:
      break;
    case 4:
      size = strtoul(token, NULL, 0);
      break;
    case 5:
      break;
    default:
      break;
    }
    token = strtok(NULL, " \t");
  }
  AddAddress(address, size);
}
//================================================================
void AddressList::AddAddress(const uint64_t& byteaddr, const uint64_t& size) {

  //allocate memory for the address vector
  try {
    if ((sequence_no_ % kReserveCount) == 0) address_list_.reserve(kReserveCount);

  } catch (std::bad_alloc& e) {
    std::cerr
        << "error allocating memory for address trace vector, address count"
        << sequence_no_ << std::endl;
    throw;
  }
  uint64_t offset = byteaddr & (CacheParam::Instance()->LineSize() - 1);
  //lineaddress = byteaddr >> CacheParam::Instance()->BitsForLine();
  // this try block is for allocating memory for the address set and the queue it holds
  try {
    // splitter
    uint64_t templinesize = 0;
    do {
      Address temp_addr(byteaddr + templinesize);
      address_list_.push_back(temp_addr);
      address_set_[temp_addr.LineAddress()].push(sequence_no_++);
      templinesize = templinesize + CacheParam::Instance()->LineSize();
    } while (size + offset > templinesize);
  } catch (std::bad_alloc& e) {
    address_list_.pop_back();
    std::cerr
    << "error allocating memory for address trace set, address count"
    << sequence_no_ << std::endl;
    throw;
  }
 }

//======================================================
typedef std::queue<uint64_t> TimeStampQueue;
typedef std::map<uint64_t, TimeStampQueue> AddressSet;
class AddressList {
public:
  AddressList(const char* tracefilename);
  bool Simulate(uint64_t *hit_count, uint64_t* miss_count);
  ~AddressList();

private:
  void AddAddress(const uint64_t& byteaddr, const uint64_t& size);
  void ParseTraceFile(const char* filename);
  void StoreTokens(char* line, uint64_t * const linecount);

  std::vector<Address> address_list_;
  AddressSet address_set_;
  uint64_t sequence_no_;
  CacheMemory cache_;

  AddressList (const AddressList&);
  AddressList& operator=(const AddressList&);
};

输出是这样的:

正在读取缓存配置文件...

缓存参数读取...

正在读取地址跟踪文件...

为地址跟踪集分配内存时出错，地址计数 30000000

解析时分配内存错误

Answer 1

因为看起来你的数据集会比你的内存大得多，所以你必须在磁盘上写一个索引。将整个东西导入数据库并让它为您构建索引可能是最简单的。