我正在利用 solr ExtractingRequestHandler 来提取和索引 HTML 内容。我的问题涉及它生成的提取链接部分。返回的提取内容在 HTML 源代码中不存在的位置插入了“rect”。
我的 solrconfig 单元配置如下:
<requestHandler name="/upate/extract"
startup="lazy"
class="solr.extraction.ExtractingRequestHandler" >
<lst name="defaults">
<str name="lowernames">true</str>
<!-- capture link hrefs but ignore div attributes -->
<str name="captureAttr">true</str>
<str name="fmap.div">ignored_</str>
</lst>
我的 solr schema.xml 具有以下内容:
<field name="content_type" type="string" indexed="true" stored="true" multiValued="true"/>
<field name="links" type="string" indexed="true" stored="true" multiValued="true"/>
<field name="meta" type="string" indexed="true" stored="true" multiValued="true"/>
<field name="content_encoding" type="string" indexed="false" stored="true" multiValued="false"/>
<field name="content" type="text_general" indexed="false" stored="true" multiValued="true"/>
我将以下 HTML 发布到 sorl 单元格:
<!DOCTYPE html>
<html>
<body>
<h1>Heading1</h1><a href="http://www.google.com">Link to Google</a><a href=
"http://www.google.com">Link to Google2</a><a href="http://www.google.com">Link to
Google3</a><a href="http://www.google.com">Link to Google</a>
<p>Paragraph1</p>
</body>
</html>
Solr 有以下索引:
{
"meta": [
"Content-Encoding",
"ISO-8859-1",
"ignored_hbaseindexer_mime_type",
"text/html",
"Content-Type",
"text/html; charset=ISO-8859-1"
],
"links": [
"rect",
"http://www.google.com",
"rect",
"http://www.google.com",
"rect",
"http://www.google.com",
"rect",
"http://www.google.com"
],
"content_encoding": "ISO-8859-1",
"content_type": [
"text/html; charset=ISO-8859-1"
],
"content": [
" Heading1 Link to Google Link to Google2 Link to Google3 Link to Google Paragraph1 "
],
"id": "row69",
"_version_": 1461665607851180000
}
注意每个链接之间的“rect”。为什么 solr cell 或 tika 插入这些？我没有定义要使用的 tika 配置文件。我需要配置 tika 吗？
最佳答案
虽然是一个老问题，但我在通过 Solr 8.7.0 索引 HTML 文档时也遇到了这个问题。
<requestHandler name="/update/extract"
class="org.apache.solr.handler.extraction.ExtractingRequestHandler">
....
<p>My website is <a href="https://buriedtruth.com/">BuriedTruth.com</a>.</p>
My website is rect https://buriedtruth.com/ BuriedTruth.com .
solr restart; sleep 1; post -c gettingstarted /mnt/Vancouver/programming/datasci/solr/test/solr_test9.html; ]我搜索了 (ripgrep:
rg --color=always -w -e 'rect' . |less) 这个词的 Solr 代码，但什么也没找到，所以 rect http... 的来源在索引的 URL 中我不知道。我的解决方案是在我的
solrconfig.xml 中添加一个正则表达式处理器: <updateRequestProcessorChain name="add-unknown-fields-to-the-schema" default="${update.autoCreateFields:true}"
processor="uuid,remove-blank,field-name-mutating,parse-boolean,parse-long,parse-double,parse-date,add-schema-fields">
<processor class="solr.LogUpdateProcessorFactory"/>
<processor class="solr.DistributedUpdateProcessorFactory"/>
<!-- ======================================== -->
<!-- https://lucene.apache.org/solr/7_4_0/solr-core/org/apache/solr/update/processor/RegexReplaceProcessorFactory.html -->
<!-- Solr bug? URLs parse as "rect https..." Managed-schema (Admin UI): defined p as text_general -->
<!-- but did not parse. Looking at content | title: text_general copied to string, so added -->
<!-- copyfield of p (text_general) as p_str ... regex below now works! -->
<processor class="solr.RegexReplaceProcessorFactory">
<str name="fieldName">content</str>
<str name="fieldName">title</str>
<str name="fieldName">p</str>
<!-- Case-sensitive (and only one pattern:replacement allowed, so use as many copies): -->
<!-- of this processor as needed: -->
<str name="pattern">rect http</str>
<str name="replacement">http</str>
<bool name="literalReplacement">true</bool>
</processor>
<!-- ======================================== -->
<!-- This needs to be last (may need to clear documents and reindex to see changes, e.g. Solr Admin UI): -->
<processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>
<p /> - 将 HTML 内容格式化为 p字段( field: p | type: text_general )。该内容没有用
RegexReplaceProcessorFactory 解析处理器。在 Solr 管理 UI 中，我注意到
title和 content被复制为字符串(例如: field: content | type: text_general | | copied to: content_str )，所以我制作了解决正则表达式问题的复制字段( p >> p_str )。为了完整起见，这里是我的
solrconfig.xml 的相关部分与 HTML 文档索引相关， <lib dir="${solr.install.dir:../../..}/contrib/extraction/lib" regex=".*\.jar" />
<lib dir="${solr.install.dir:../../..}/dist/" regex="solr-cell-\d.*\.jar" />
<!-- https://lucene.472066.n3.nabble.com/Prons-an-Cons-of-Startup-Lazy-a-Handler-td4059111.html -->
<!-- startup="lazy" -->
<requestHandler name="/update/extract"
class="org.apache.solr.handler.extraction.ExtractingRequestHandler">
<lst name="defaults">
<str name="lowernames">true</str>
<str name="uprefix">ignored_</str>
<str name="capture">div</str>
<str name="fmap.div">div</str>
<str name="capture">p</str>
<str name="fmap.p">p</str>
</lst>
</requestHandler>
managed-schema 添加了字段通过 Solr 管理用户界面。结果:
My website is https://buriedtruth.com/ BuriedTruth.com .
<field name="p" type="text_general" uninvertible="true" indexed="true" stored="true"/>
<copyField source="p" dest="p_str"/>
<requestHandler name="/update/extract"... :updateRequestProcessorChain /> 切换时，我在这里的回答(处理与上述 managed-schema 相关的特性)到经典schema.xml关于Solr ExtractingRequestHandler 在链接中提取 "rect"，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/22178700/