我正在使用 VS 2008,需要读取具有 UTF-8 中文字符的文本文件。该文件的组织方式如下:每行包含一个文档,文档以制表符分隔(index 'tab' doc title 'tab' doc body)。所以我需要做的是分隔选项卡上的行,然后用空格分隔第三列(文档正文)并将每个单词存储在一个 vector 中。当文件是 ANSI 编码时,所有这些都可以正常工作。但是当它是 UTF-8 时,它会抛出断言失败(无符号)(c+1)<=256。我想保留当前的功能和流程,并尽可能少地使用第三方库。
我看过不同的方法(ustream、wstream 等),但我对如何实际使用它们有点困惑。
读取文件的方法如下:
bool TabDelimitedSource::setup_next_buff_reader() {
this->current_source_file += 1;
bool no_more_files = false; // assume we have no more files by default
/** If there are still files int the directory load the next file*/
if(current_source_file < (data_source_files.size())){
string file_path = (this->data_source_files[this->current_source_file]);
string full_path = data_source_dir + file_path ;
buff_reader->open((char*)full_path.c_str());
}
else{
no_more_files = true;
}
return no_more_files; // let the caller know whether there was another file or not
}
这是进行解析的方法:
vector<string> TabDelimitedSource::getNext() {
// Returns the next document (a given cell) from the file(s)
string row; // Return NULL if no more documents/rows
vector<string> document;
try{
//Read each line in the file, corresponding to and individual document
std::getline(*buff_reader,row,'\n');
}
catch (ifstream::failure e){
; // Ignore and fall through
}
if (row.size()>0){
this->current_row += 1;
vector<string> cells;
this->split(row, "\t", cells); // Split the row on tabs
try{
string original_document = cells[column_holding_doc];
try{
split(original_document," ",document);
}catch (std::out_of_range e){
throw std::out_of_range ("Out of Range"); // ignore and fall through
}
}
catch (std::out_of_range e){
throw std::out_of_range ("Out of Range");
}
}
else{
// We're at the end of the current file, try loading the next one
buff_reader->close();
bool no_more_files = this->setup_next_buff_reader();
// If there was another file to load, recurse to get its first document
if (!no_more_files){
return this->getNext();
}
}
// Return our arrayList as an array... there has to be a better way to do this
vector<string> return_val ;
if(document.size()>0){ // return NULL by default
for(int i=0; i<(int)document.size(); i++){
return_val.push_back(document[i]);
}
}
return return_val;
}
拆分方法:
void TabDelimitedSource::split(const string& str, const string& delim, vector<string>& result){
size_t start_pos = 0;
size_t match_pos;
size_t substr_length;
while((match_pos = str.find(delim, start_pos)) != string::npos){
substr_length = match_pos - start_pos;
if (substr_length > 0){
result.push_back(str.substr(start_pos, substr_length));
}
start_pos = match_pos + delim.length();
}
substr_length = str.length() - start_pos;
if (substr_length > 0){
result.push_back(str.substr(start_pos, substr_length));
}
}
提前致谢
戴夫
最佳答案
在进行任何解析之前,您需要将 UTF-8 文件转换为 UTF-16 (wstring)。
当您使用 Windows 时,您可以使用 MultiByteToWideChar 来完成此操作
http://msdn.microsoft.com/en-us/library/dd319072(v=VS.85).aspx
里面有一些指向源代码的链接。
关于c++ - 读取 UTF-8 文件,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/6626992/