我有一个内容聚合网站。我想处理帖子标题以生成最受欢迎的帖子主题列表。主题可以是“软件开发”，但重要的一点是“软件”和“开发”这两个词不必在帖子标题中直接相邻。
想法是生成一个可能的集合列表(一组基于同一主题的帖子)
我已经开始编写代码，到目前为止已经成功地生成了一个最常用词的列表。现在，我需要生成一个新列表，该列表按帖子标题中多个单词的出现次数排序，我希望它能为我提供最终的热门主题列表。
正在寻求帮助来完成这项工作。向我的代码添加注释以描述我到目前为止所做的事情。
另外，如果有人知道更好的方法，请告诉我!
/**
* Execute the console command.
*
* @return mixed
*/
public function fire()
{
$this->info('Starting...');
// get last 1000 posts
$posts = Post::orderBy('created_at', 'desc')->take(1000)->get();
// create an array of post titles
$titles_arr = array_map(function($n){
return $n['title'];
}, $posts->toArray());
// create a big string of all the post titles
$titles_str = implode(' ', $titles_arr);
// create an array of the above string
$words = explode(' ', strtolower($titles_str));
// words to ignore
$stopwords = array('a', 'about', 'above', 'across', 'after', 'afterwards', 'again', 'against', 'all', 'almost', 'alone', 'along', 'already', 'also','although','always','am','among', 'amongst', 'amoungst', 'amount', 'an', 'and', 'another', 'any','anyhow','anyone','anything','anyway', 'anywhere', 'are', 'around', 'as', 'at', 'back','be','became', 'because','become','becomes', 'becoming', 'been', 'before', 'beforehand', 'behind', 'being', 'below', 'beside', 'besides', 'between', 'beyond', 'bill', 'both', 'bottom','but', 'by', 'call', 'can', 'cannot', 'cant', 'co', 'con', 'could', 'couldnt', 'cry', 'de', 'describe', 'detail', 'do', 'done', 'down', 'due', 'during', 'each', 'eg', 'eight', 'either', 'eleven','else', 'elsewhere', 'empty', 'enough', 'etc', 'even', 'ever', 'every', 'everyone', 'everything', 'everywhere', 'except', 'few', 'fifteen', 'fify', 'fill', 'find', 'fire', 'first', 'five', 'for', 'former', 'formerly', 'forty', 'found', 'four', 'from', 'front', 'full', 'further', 'get', 'give', 'go', 'had', 'has', 'hasnt', 'have', 'he', 'hence', 'her', 'here', 'hereafter', 'hereby', 'herein', 'hereupon', 'hers', 'herself', 'him', 'himself', 'his', 'how', 'however', 'hundred', 'ie', 'if', 'in', 'inc', 'indeed', 'interest', 'into', 'is', 'it', 'its', 'itself', 'keep', 'last', 'latter', 'latterly', 'least', 'less', 'ltd', 'made', 'many', 'may', 'me', 'meanwhile', 'might', 'mill', 'mine', 'more', 'moreover', 'most', 'mostly', 'move', 'much', 'must', 'my', 'myself', 'name', 'namely', 'neither', 'never', 'nevertheless', 'next', 'nine', 'no', 'nobody', 'none', 'noone', 'nor', 'not', 'nothing', 'now', 'nowhere', 'of', 'off', 'often', 'on', 'once', 'one', 'only', 'onto', 'or', 'other', 'others', 'otherwise', 'our', 'ours', 'ourselves', 'out', 'over', 'own','part', 'per', 'perhaps', 'please', 'put', 'rather', 're', 'same', 'see', 'seem', 'seemed', 'seeming', 'seems', 'serious', 'several', 'she', 'should', 'show', 'side', 'since', 'sincere', 'six', 'sixty', 'so', 'some', 'somehow', 'someone', 'something', 'sometime', 'sometimes', 'somewhere', 'still', 'such', 'system', 'take', 'ten', 'than', 'that', 'the', 'their', 'them', 'themselves', 'then', 'thence', 'there', 'thereafter', 'thereby', 'therefore', 'therein', 'thereupon', 'these', 'they', 'thickv', 'thin', 'third', 'this', 'those', 'though', 'three', 'through', 'throughout', 'thru', 'thus', 'to', 'together', 'too', 'top', 'toward', 'towards', 'twelve', 'twenty', 'two', 'un', 'under', 'until', 'up', 'upon', 'us', 'very', 'via', 'was', 'we', 'well', 'were', 'what', 'whatever', 'when', 'whence', 'whenever', 'where', 'whereafter', 'whereas', 'whereby', 'wherein', 'whereupon', 'wherever', 'whether', 'which', 'while', 'whither', 'who', 'whoever', 'whole', 'whom', 'whose', 'why', 'will', 'with', 'within', 'without', 'would', 'yet', 'you', 'your', 'yours', 'yourself', 'yourselves', 'the');
// $words after stop words are filtered out
$final_words = array_filter($words, function($n) use($stopwords){
return (!in_array(strtolower($n), $stopwords));
});
// count occurence of array values
$reduce = array_count_values($final_words);
arsort($reduce); // sort
// take first 1000 popular words
$top_1000 = array_slice($reduce, 0, 1000);
// I have the top 1000 used words. Now need to find which ones are present together the most (ordered)
$matched_total = array();
// for each post title
foreach ($titles_arr as $title) {
// split it into array
$words = explode(' ', strtolower($title));
$matched = array();
echo $title . "\n";
foreach ($words as $word) {
// if in $top_1000 print and add to $matched array
if(array_key_exists($word, $top_1000)) {
echo $word . "\n";
$matched[] = $word;
}
}
// if contains popular words at to matched_total
if(count($matched) > 0)
$matched_total[] = $matched;
}
// have the words that match for each post, now need to get which ones appear together the most
$this->info('Finished.');
}
最佳答案
我觉得这不应该在 PHP 中完成，但无论如何还是要这样做。
Remove non-stop words
For each word keep track of which titles it appears in see title_keys in the debug below
For each word, count the common titles between it and other words and store this count in the associative array (see count_other_word_in_same_title in the debug output) and keep store all counts in a separate array so you know at the end of the loop how many times the top n phrase appears
Grab all phrases that appear >= the number of times the top n phrase appears and stop when you've grabbed n phrases
$titles = array(
'where in the world is waldo',
'software development for beginners',
'waldo is travelling the world',
'beginners learn to develop software',
'the big brown dog jumped over waldo',
);
$stopwords = array('a', 'about', 'above', 'across', 'after', 'afterwards', 'again', 'against', 'all', 'almost', 'alone', 'along', 'already', 'also','although','always','am','among', 'amongst', 'amoungst', 'amount', 'an', 'and', 'another', 'any','anyhow','anyone','anything','anyway', 'anywhere', 'are', 'around', 'as', 'at', 'back','be','became', 'because','become','becomes', 'becoming', 'been', 'before', 'beforehand', 'behind', 'being', 'below', 'beside', 'besides', 'between', 'beyond', 'bill', 'both', 'bottom','but', 'by', 'call', 'can', 'cannot', 'cant', 'co', 'con', 'could', 'couldnt', 'cry', 'de', 'describe', 'detail', 'do', 'done', 'down', 'due', 'during', 'each', 'eg', 'eight', 'either', 'eleven','else', 'elsewhere', 'empty', 'enough', 'etc', 'even', 'ever', 'every', 'everyone', 'everything', 'everywhere', 'except', 'few', 'fifteen', 'fify', 'fill', 'find', 'fire', 'first', 'five', 'for', 'former', 'formerly', 'forty', 'found', 'four', 'from', 'front', 'full', 'further', 'get', 'give', 'go', 'had', 'has', 'hasnt', 'have', 'he', 'hence', 'her', 'here', 'hereafter', 'hereby', 'herein', 'hereupon', 'hers', 'herself', 'him', 'himself', 'his', 'how', 'however', 'hundred', 'ie', 'if', 'in', 'inc', 'indeed', 'interest', 'into', 'is', 'it', 'its', 'itself', 'keep', 'last', 'latter', 'latterly', 'least', 'less', 'ltd', 'made', 'many', 'may', 'me', 'meanwhile', 'might', 'mill', 'mine', 'more', 'moreover', 'most', 'mostly', 'move', 'much', 'must', 'my', 'myself', 'name', 'namely', 'neither', 'never', 'nevertheless', 'next', 'nine', 'no', 'nobody', 'none', 'noone', 'nor', 'not', 'nothing', 'now', 'nowhere', 'of', 'off', 'often', 'on', 'once', 'one', 'only', 'onto', 'or', 'other', 'others', 'otherwise', 'our', 'ours', 'ourselves', 'out', 'over', 'own','part', 'per', 'perhaps', 'please', 'put', 'rather', 're', 'same', 'see', 'seem', 'seemed', 'seeming', 'seems', 'serious', 'several', 'she', 'should', 'show', 'side', 'since', 'sincere', 'six', 'sixty', 'so', 'some', 'somehow', 'someone', 'something', 'sometime', 'sometimes', 'somewhere', 'still', 'such', 'system', 'take', 'ten', 'than', 'that', 'the', 'their', 'them', 'themselves', 'then', 'thence', 'there', 'thereafter', 'thereby', 'therefore', 'therein', 'thereupon', 'these', 'they', 'thickv', 'thin', 'third', 'this', 'those', 'though', 'three', 'through', 'throughout', 'thru', 'thus', 'to', 'together', 'too', 'top', 'toward', 'towards', 'twelve', 'twenty', 'two', 'un', 'under', 'until', 'up', 'upon', 'us', 'very', 'via', 'was', 'we', 'well', 'were', 'what', 'whatever', 'when', 'whence', 'whenever', 'where', 'whereafter', 'whereas', 'whereby', 'wherein', 'whereupon', 'wherever', 'whether', 'which', 'while', 'whither', 'who', 'whoever', 'whole', 'whom', 'whose', 'why', 'will', 'with', 'within', 'without', 'would', 'yet', 'you', 'your', 'yours', 'yourself', 'yourselves', 'the');
$stopwords = array_flip($stopwords);
$allwords = array();
foreach($titles as $key => $title) {
$words = explode(' ', strtolower($title));
$words = array_unique($words); //optional
foreach($words as $word) {
if(isset($stopwords[$word]))
continue;
$allwords[$word]['title_keys'][$key] = $key;
}
}
print_r($allwords);
$copy_allwords = $allwords;
$counts = array();
//if you want to look for more than 2 word combos
//make this recursive
foreach($copy_allwords as $word => $stats) {
unset($copy_allwords[$word]);
foreach($copy_allwords as $word2 => $stats2) {
$intersect = array_intersect_key($stats['title_keys'],$stats2['title_keys']);
if(!empty($intersect)) {
$allwords[$word]['count_other_word_in_same_title'][$word2] = count($intersect);
$counts[] = count($intersect);
}
}
}
rsort($counts);
$count_phrases = 2; // setting to retrieve top n phrases
$phrases = array();
foreach($allwords as $word => $stats) {
if(!isset($stats['count_other_word_in_same_title']))
continue;
foreach($stats['count_other_word_in_same_title'] as $other_word => $count) {
if($count >= $counts[($count_phrases-1)]) {
$phrases[$word . ' ' . $other_word] = $count;
if(count($phrases) >= $count_phrases)
break 2;
}
}
}
arsort($phrases);
print_r($phrases);
前 2 个短语输出
Array
(
[software beginners] => 2 //2 = number of titles this phrase appears in
[world waldo] => 2
)
末尾$allwords的内容
Array
(
[world] => Array
(
[title_keys] => Array
(
[0] => 0
[2] => 2
)
[count_other_word_in_same_title] => Array
(
[waldo] => 2
[travelling] => 1
)
)
[waldo] => Array
(
[title_keys] => Array
(
[0] => 0
[2] => 2
[4] => 4
)
[count_other_word_in_same_title] => Array
(
[travelling] => 1
[big] => 1
[brown] => 1
[dog] => 1
[jumped] => 1
)
)
[software] => Array
(
[title_keys] => Array
(
[1] => 1
[3] => 3
)
[count_other_word_in_same_title] => Array
(
[development] => 1
[beginners] => 2
[learn] => 1
[develop] => 1
)
)
[development] => Array
(
[title_keys] => Array
(
[1] => 1
)
[count_other_word_in_same_title] => Array
(
[beginners] => 1
)
)
[beginners] => Array
(
[title_keys] => Array
(
[1] => 1
[3] => 3
)
[count_other_word_in_same_title] => Array
(
[learn] => 1
[develop] => 1
)
)
[travelling] => Array
(
[title_keys] => Array
(
[2] => 2
)
)
[learn] => Array
(
[title_keys] => Array
(
[3] => 3
)
[count_other_word_in_same_title] => Array
(
[develop] => 1
)
)
[develop] => Array
(
[title_keys] => Array
(
[3] => 3
)
)
[big] => Array
(
[title_keys] => Array
(
[4] => 4
)
[count_other_word_in_same_title] => Array
(
[brown] => 1
[dog] => 1
[jumped] => 1
)
)
[brown] => Array
(
[title_keys] => Array
(
[4] => 4
)
[count_other_word_in_same_title] => Array
(
[dog] => 1
[jumped] => 1
)
)
[dog] => Array
(
[title_keys] => Array
(
[4] => 4
)
[count_other_word_in_same_title] => Array
(
[jumped] => 1
)
)
[jumped] => Array
(
[title_keys] => Array
(
[4] => 4
)
)
)
关于php - 从帖子标题集合中生成热门主题，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/23944661/