java - 如何让 Apache Tika 在 .java 和 .xml(等)文件中查找文本

Question

好的。我想出了如何使用 Apache Tika 搜索它可以处理的一些文件类型，而无需我提供比 tika-example 中存在的代码更多的代码:

public class MyFirstTika {

  public static boolean contains(File file, String s) throws MalformedURLException, 
     IOException, MimeTypeException, SAXException, TikaException{

    ContentHandler handler = new BodyContentHandler();

    MimeTypes mimeRegistry = TikaConfig.getDefaultConfig().getMimeRepository();

    Detector mimeDetector = (Detector) mimeRegistry;

    LanguageIdentifier lang = new LanguageIdentifier(new LanguageProfile(FileUtils.readFileToString(file)));

    Parser parser = TikaConfig.getDefaultConfig().getParser(MediaType.parse(mimeRegistry.getMimeType(file).getName()));

    Metadata parsedMet = new Metadata();

    parser.parse(file.toURI().toURL().openStream(), handler,parsedMet, new ParseContext());

    return handler.toString().toLowerCase().contains(s.toLowerCase());
  }

  public static void main(String[] args) throws Exception 
  {
    String searchString = "champion";
    String filename = "schedule.pdf"; //test.docx";//"meds.xlsx";//Test2.Doc";
    File file = new File(filename);

    System.out.println(file + " contains " + searchString + ": " 
            + contains(file, searchString));
    }
}

以上可以判断以下文件类型是否包含单词或短语: .doc .docx .xlsx .pdf 。TXT .html

它不适用于 .java 文件或 .xml 文件。

(a) 我需要做什么我想查看扩展名为 .java 或 .xml 的文本文件是否包含单词或短语？

(b) 这些不是我经常创建或编辑的唯一类型的文件。有没有办法让 Apache Tika 在不指定扩展名的情况下检测文件是否为文本文件？

编辑 背景:我为 Windows 编写了一个比搜索命令更好用的搜索程序。现在，我正在尝试在模式匹配文件中添加对特定文本的搜索。

编辑

这是当我在 Copy.java 中搜索 void 时程序的输出(经过修改以提供以下信息):

Examining: [copy.java]
The MIME type (based on filename) is: [text/x-java-source]
The MIME type (based on MAGIC) is: [application/octet-stream
The MIME type (based on the Detector interface) is: [text/plain]
The language of this content is: [et]
Parsed Metadata: 

Parsed Text: 

copy.java contains void: false

那么为什么它没有找到 void？ (答案:因为它没有找到任何Parsed Metadata或Parsed Text，但为什么没有找到那些？它应该显示整个文件。

我将 copy.java 复制到 copy.txt。程序 DID 发现 void。当我将 build.xml 复制到 build.txt 时发生了同样的事情。

也许这个添加的信息有助于回答这个问题:“我如何处理 .java 和 .xml 以及其他文本文件，例如 .c 等等？”

请注意搜索 copy.TXT 的输出:

run:
Examining: [copy.TXT]
The MIME type (based on filename) is: [text/plain]
The MIME type (based on MAGIC) is: [application/octet-stream
The MIME type (based on the Detector interface) is: [text/plain]
The language of this content is: [et]

解析的元数据:

Content-Encoding=UTF-8 Content-Type=text/plain; charset=UTF-8 

解析的文本:

  public static void main(String[] args) throws IOException {
    EventQueue.invokeLater(new Runnable() 
        { @Override
          public void run() { 
               insUserIO = new UserIO(); 
          }
        }
    ); 
  }

copy.TXT contains void: true
BUILD SUCCESSFUL (total time: 1 second)

整个修改后的程序

    package org.apache.tika.example;

    import java.io.File;
    import java.io.IOException;
    import java.net.MalformedURLException;

    import org.apache.commons.io.FileUtils;
    import org.apache.tika.config.TikaConfig;
    import org.apache.tika.detect.Detector;
    import org.apache.tika.exception.TikaException;
    import org.apache.tika.language.LanguageIdentifier;
    import org.apache.tika.language.LanguageProfile;
    import org.apache.tika.metadata.Metadata;
    import org.apache.tika.mime.MediaType;
    import org.apache.tika.mime.MimeTypeException;
    import org.apache.tika.mime.MimeTypes;
    import org.apache.tika.parser.ParseContext;
    import org.apache.tika.parser.Parser;
    import org.apache.tika.sax.BodyContentHandler;
    import org.xml.sax.ContentHandler;
    import org.xml.sax.SAXException;

    public class MyFirstTika {

      static boolean debugging = true;

      public static boolean contains(File file, String s) throws MalformedURLException, IOException, MimeTypeException, SAXException, TikaException{

        ContentHandler handler = new BodyContentHandler();

            MimeTypes mimeRegistry = TikaConfig.getDefaultConfig()
                    .getMimeRepository();

            if(debugging) System.out.println("Examining: [" + file + "]");

            if(debugging) System.out.println("The MIME type (based on filename) is: ["
                    + mimeRegistry.getMimeType(file.toString()) + "]");

            if(debugging) System.out.println("The MIME type (based on MAGIC) is: ["
                    + mimeRegistry.getMimeType(file + "]"));

            Detector mimeDetector = (Detector) mimeRegistry;
            if(debugging) System.out
                    .println("The MIME type (based on the Detector interface) is: ["
                            + mimeDetector.detect(file.toURI().toURL()
                                    .openStream(), new Metadata()) + "]");

            LanguageIdentifier lang = new LanguageIdentifier(new LanguageProfile(
                    FileUtils.readFileToString(file)));

            if(debugging) System.out.println("The language of this content is: ["
                    + lang.getLanguage() + "]");

            Parser parser = TikaConfig.getDefaultConfig().getParser(
                    MediaType.parse(mimeRegistry.getMimeType(file).getName()));

        Metadata parsedMet = new Metadata();
            parser.parse(file.toURI().toURL().openStream(), handler,
                    parsedMet, new ParseContext());

            if(debugging) System.out.println("Parsed Metadata: ");
            if(debugging) System.out.println(parsedMet);
            if(debugging) System.out.println("Parsed Text: ");
            if(debugging) System.out.println(handler.toString());
        return handler.toString().toLowerCase().contains(s.toLowerCase());
      }

      public static void main(String[] args) throws Exception 
      {
        File file = new File(filename);

        System.out.println(file + " contains " + searchString + ": " 
                + contains(file, searchString));
        }

        static String searchString = "void";
        static String filename = "copy.TXT";
    }

Answer 1

非常感谢@Gagravarr 引导我使用AutoDetectParser!

下面的小程序在所有类型的“文本”(人类可读)(和其他——例如，.doc*)文件中找到了文本，包括 private.properties、ParsingExample.java(本身!)、test.doc、test.pdf(由 Word 制作)、gradlew。 bat 等等等等

package org.apache.tika.example;
import java.io.File;
import java.io.FileInputStream;
import java.io.IOException;
import java.io.InputStream;
import java.net.MalformedURLException;
import org.apache.tika.exception.TikaException;
import org.apache.tika.metadata.Metadata;
import org.apache.tika.mime.MimeTypeException;
import org.apache.tika.parser.AutoDetectParser;
import org.apache.tika.sax.BodyContentHandler;
import org.xml.sax.SAXException;

public class ParsingExample {

  public static boolean contains(File file, String s) throws MalformedURLException, 
                     IOException, MimeTypeException, SAXException, TikaException
  {
    InputStream         stream    = new FileInputStream(file);
    AutoDetectParser    parser    = new AutoDetectParser();
    BodyContentHandler  handler   = new BodyContentHandler(-1);
    Metadata            metadata  = new Metadata();
    try{
      parser.parse(stream, handler, metadata);
      return handler.toString().toLowerCase().contains(s.toLowerCase());
    }
    catch (IOException | SAXException | TikaException e){
      System.out.println(file + ": " + e + "\n");
      return false;
    }
  }
  public static void main(String[] args)
  {
      try {
        System.out.println("File " + filename + " contains <" + searchString + "> : " 
          + contains(new File(filename), searchString));
      } catch (IOException | SAXException | TikaException ex){
        System.out.println("Error: " + ex);
      }
  }   

  static String parseExample = ":(";
  static String searchString = "slom";
  static String filename = "C:\\Users\\Dov\\x.pdf"; 
}
    /**
     * Example of how to use Tika to parse a file when you do not know its file type
     * ahead of time.
     *
     * AutoDetectParser attempts to discover the file's type automatically, then call
     * the exact Parser built for that file type.
     *
     * The stream to be parsed by the Parser. In this case, we get a file from the
     * resources folder of this project.
     *
     * Handlers are used to get the exact information you want out of the host of
     * information gathered by Parsers. The body content handler, intuitively, extracts
     * everything that would go between HTML body tags.
     *
     * The Metadata object will be filled by the Parser with Metadata discovered about
     * the file being parsed.
     *
     * Note: This example will extract content from the outer document and all
     * embedded documents.  However, if you choose to use a {@link ParseContext},
     * make sure to set a {@link Parser} or else embedded content will not be
     * parsed.
     *
     * @return The content of a file.
   * I let Netbeans add next 3 lines: 
   * @throws java.io.IOException
   * @throws org.xml.sax.SAXException
   * @throws org.apache.tika.exception.TikaException
     */