ruby - 404 未找到，但可以从网络浏览器正常访问

Question

我在这方面尝试了很多 URL，在我遇到这个特定的之前，它们似乎都很好:

require 'rubygems'
require 'nokogiri'
require 'open-uri'

doc = Nokogiri::HTML(open("http://www.moxyst.com/fashion/men-clothing/underwear.html"))
puts doc

这是结果:

/Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:353:in `open_http': 404 Not Found (OpenURI::HTTPError)
    from /Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:709:in `buffer_open'
    from /Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:210:in `block in open_loop'
    from /Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:208:in `catch'
    from /Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:208:in `open_loop'
    from /Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:149:in `open_uri'
    from /Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:689:in `open'
    from /Users/macbookair/.rvm/rubies/ruby-2.0.0-p481/lib/ruby/2.0.0/open-uri.rb:34:in `open'
    from test.rb:5:in `<main>'  

我可以从网络浏览器访问它，但我根本不明白。

这是怎么回事，我该如何处理这种错误？我可以忽略它，让其他人做他们的工作吗？

Answer 1

您得到 404 Not Found (OpenURI::HTTPError)，因此，如果您想让您的代码继续运行，请挽救该异常。这样的事情应该有效:

require 'nokogiri'
require 'open-uri'

URLS = %w[
  http://www.moxyst.com/fashion/men-clothing/underwear.html
]

URLs.each do |url|
  begin
    doc = Nokogiri::HTML(open(url))
  rescue OpenURI::HTTPError => e
    puts "Can't access #{ url }"
    puts e.message
    puts
    next
  end
  puts doc.to_html
end

您可以使用更通用的异常，但随后您会遇到问题，得到奇怪的输出，或者可能以导致更多问题的方式处理不相关的问题，因此您需要确定所需的粒度。

如果您想要更多控制并想对 401 或 404 做一些不同的事情，您甚至可以嗅探 HTTPd header 、响应状态或查看异常消息。

I can access this from a web browser, I just don't get it at all.

好吧，这可能是服务器端发生的事情:也许他们不喜欢您发送的 UserAgent 字符串？ OpenURI documentation显示如何更改该 header :

Additional header fields can be specified by an optional hash argument.

open("http://www.ruby-lang.org/en/",
  "User-Agent" => "Ruby/#{RUBY_VERSION}",
  "From" => "foo@bar.invalid",
  "Referer" => "http://www.ruby-lang.org/") {|f|
  # ...
}