java - Gnocchi爬虫-如何在爬行过程中添加要爬行的url(爬行整个网站)

Question

有人有 Niocchi 图书馆的经验吗？我开始使用域名 url 进行爬行。在Worker方法processResource()中，我解析我得到的资源，提取该页面中的所有内部链接，我需要将它们添加到爬行中。但我找不到如何做。我应该将其添加到 UrlPool、ResourcePool 还是其他地方？

谢谢!

Answer 1

您可以将它们添加到现有 URLPool 中。现有的 URLPool 实现不可扩展，因此您必须创建自己的可扩展 URLPool 类。我将我的类命名为 ExpandableURLPool。

URLPool.setProcessed 方法在处理完成后由框架调用，您可以在该方法中向 url 列表添加其他 URL。我将举一个例子，但首先，URLPool 文档指出:

setProcessed(Query) is called by the crawler to inform the URLPool when a Query has been crawled and its resource processed. This is typically used by the URLPool to check the crawl status and log the error in case of a failure or to get more URL to crawl in case of success. A typical example where getNextQuery() returns null but hasNextQuery() returns true is when the URLPool is waiting for some processed resources from which more URL to crawl have been extracted to come back. Check the urlpools package for examples of implementation.

这意味着 ExapndableURLPool 实现中的棘手部分是，如果正在处理一个未完成的查询，可能会导致新的 url 被添加到池中，则 hasNextQuery 方法应该返回 true。同样，如果存在尚未完成的未完成查询，getNextQuery 必须返回 null，并且可能会导致新的 url 添加到池中。 [我不喜欢尼奥基在这方面的组合方式]

这是我的 ExpandableURLPool 的初步版本:

class ExpandableURLPool implements URLPool {
List<String> urlList = new ArrayList<String>();
int cursor = 0;

int outstandingQueryies = 0;

public ExpandableURLPool(Collection<String> seedURLS) {
    urlList.addAll(seedURLS);
}

@Override
public boolean hasNextQuery() {
   return  cursor < urlList.size() || outstandingQueryies > 0;

}

@Override
public Query getNextQuery() throws URLPoolException {
    try {
        if (cursor >= urlList.size()) {
            return null;
        } else {
            outstandingQueryies++;
            return new Query( urlList.get(cursor++) ) ;
        }
    } catch (MalformedURLException e) {
        throw new URLPoolException( "invalid url", e ) ;
    }    
}

@Override
public void setProcessed(Query query) {
    outstandingQueryies--;


}

public void addURL(String url) {
    urlList.add(url);
}

}

我还创建了一个从 DiskSaveWorker 派生的 Worker 类来测试上述实现:

    class MyWorker extends org.niocchi.gc.DiskSaveWorker {

    Crawler mCrawler = null;
    ExpandableURLPool pool = null;

    int maxepansion = 10;

    public MyWorker(Crawler crawler, String savePath, ExpandableURLPool aPool) {
        super(crawler, savePath);
        mCrawler = crawler;
        pool = aPool;
    }

    @Override
    public void processResource(Query query) {
        super.processResource(query);
        // The following is a test
        if (--maxepansion >= 0  ) {
            pool.addURL("http://www.somewhere.com");
        }       

    }


}