ruby - 如何使用 SAX 使用 Nokogiri 抓取元素内容

Question

我想从一个网站解析几千个 XML 文件(我有权限) 并且必须使用 SAX 来避免将文件加载到内存中。然后将它们保存到 CSV 文件中。

xml 文件如下所示:

<?xml version="1.0" encoding="UTF-8"?><educationInfo xmlns="http://skolverket.se/education/info/1.2" xmlns:ct="http://skolverket.se/education/commontypes/1.2" xmlns:nya="http://vhs.se/NyA-emil-extensions" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" expires="2013-08-01" id="info.uh.su.HIA80D" lastEdited="2011-10-13T10:10:05" xsi:schemaLocation="http://skolverket.se/education/info/1.2 educationinfo.xsd">
  <titles>
    <title xml:lang="sv">Arkivvetenskap</title>
    <title xml:lang="en">Archival science</title>
  </titles>
  <identifier>HIA80D</identifier>
  <educationLevelDetails>
    <typeOfLevel>uoh</typeOfLevel>
    <typeOfResponsibleBody>statlig</typeOfResponsibleBody>
    <academic>
      <course>
        <type>avancerad</type>
      </course>
    </academic>
  </educationLevelDetails>
  <credits>
    <exact>60</exact>
  </credits>
  <degrees>
    <degree>Ingen examen</degree>
  </degrees>
  <prerequisites>
    <academic>uh</academic>
  </prerequisites>
  <subjects>
    <subject>
      <code source="vhs">10.300</code>
    </subject>
  </subjects>
  <descriptions>
    <ct:description xml:lang="sv">
      <ct:text>Arkivvetenskap rör villkoren för befintliga arkiv och modern arkivbildning med fokus på arkivarieyrkets arbetsuppgifter: bevara, tillgängliggöra och styra information. Under ett år behandlas bl a informations- och dokumenthantering, arkivredovisning, gallring, lagstiftning och arkivteori. I kursen ingår praktik, där man under handledning får arbeta med olika arkivarieuppgifter.</ct:text>
    </ct:description>
  </descriptions>
</educationInfo> 

我使用这个代码模板，检查我的评论是否有问题:

class InfoData  < Nokogiri::XML::SAX::Document

  def initialize
    # do one-time setup here, called as part of Class.new
    # But what should I use hashes or arrays?
  end

  def start_element(name, attributes = [])
  # check the element name here and create an active record object if appropriate
  # How do I grab specific element like: ct:text ?
  # how do I grab root-element?
  end

  def characters(s)
     # save the characters that appear here and possibly use them in the current tag object
  end

  def end_element(name)
     # check the tag name and possibly use the characters you've collected
     # and save your activerecord object now
  end

end

parser = Nokogiri::XML::SAX::Parser.new(InfoData.new)

# How do I parse every xml-link? 
parser.parse_file('')

我写了这个方法来获取链接，但不知道在类中的什么地方使用它或者我是否应该在那里使用它:

@items = Set.new 
def get_links(url)
  doc = Nokogiri::HTML(open(url))
  doc.xpath('//a/@href').each do |url|
  item = {}
  item[:url] = url.content
  items << item
end

Answer 1

require 'nokogiri'

class LinkGrabber < Nokogiri::XML::SAX::Document
  def start_element(name, attrs = [])
    if name == 'a'
      puts Hash[attrs]['href']
    end
  end
end

parser = Nokogiri::XML::SAX::Parser.new(LinkGrabber.new)
parser.parse(File.read(ARGV[0], 'rb'))

现在您可以在管道中使用它:

find . -name "*.xml" -print0 | xargs -P 20 -0 -L 1 ruby parse.rb > links

但这每次都会启动 ruby​​。所以你最好使用 jruby (无论如何都更快)和威胁。

require 'threach'
require 'find'
require 'nokogiri'

class LinkGrabber < Nokogiri::XML::SAX::Document
  def start_element(name, attrs = [])
    if name == 'a'
      puts Hash[attrs]['href']
    end
  end
end

# let's hope it's threadsave
parser = Nokogiri::XML::SAX::Parser.new(LinkGrabber.new)
Find.find(ARGV[0]).threach do |path|
  next unless File.file?(path)
  parser.parse(File.read(path))
end