python - 使用BeautifulSoup

Question

我正在使用 BeautifulSoup 制作一个文本爬虫。但是当我运行此代码时，我收到错误代码:

Traceback (most recent call last):
  File "D:\Python27\Crawling.py", line 33, in <module>
    text = content.get_text()
AttributeError: 'NoneType' object has no attribute 'get_text'

如果您告诉我如何修复它，我将非常感激。

import urllib
from bs4 import BeautifulSoup
import xml.dom.minidom

keyWord = raw_input("Enter the key-word : ")
#Enter my Search KeyWord

address = "http://openapi.naver.com/search?key=8d4b5b7fef7a607863013302754262a3&query="                   + keyWord + "&display=5&start=1&target=kin&sort=sim"

search_result = urllib.urlopen(address)
raw_data = search_result.read()
parsed_result = xml.dom.minidom.parseString(raw_data)
links = parsed_result.getElementsByTagName('link')

source_URL = links[3].firstChild.nodeValue
#The number 3 has no meaning, it has 0 to 9 and I just chose 3
page = urllib.urlopen(source_URL).read()

#save as html file
g = open(keyWord + '.html', 'w')
g.write(page)
g.close()

#open html file
g = open(keyWord + '.html', 'r')
bs = BeautifulSoup(g)
g.close()


content = bs.find(id="end_content")
text = content.get_text()

#save as text file
h = codecs.open(keyWord + '.txt', 'w', 'utf-8')
h.write(keyWord + ' ')
h.write(text)

print "file created"

Answer 1

考虑到 @Hooked 和 @alecxe 的答案，使用 requests 执行此操作的方法如下。请注意，我将使用 handbag 关键字进行搜索查询。

import requests as rq
from bs4 import BeautifulSoup as bsoup
from xml.dom.minidom import parseString

url = "http://openapi.naver.com/search?key=8d4b5b7fef7a607863013302754262a3&query=handbag&display=100&start=1&target=kin&sort=sim"
result = rq.get(url)
parsed_result = parseString(result.content)
links = parsed_result.getElementsByTagName("link")

new_url = links[3].firstChild.nodeValue
new_result = rq.get(new_url).content

g = open("handbag.html", "w")
g.write(new_result)
g.close()

g = open("handbag.html", "r")
soup = bsoup(g)
g.close()

content = soup.find("div", class_="end_content")
text = content.get_text()

print text.encode("utf-8").strip()

.encode("utf-8")部分是处理韩文字符的输出。结果如下:

아디다스 그래픽핸드백
거의품절이던데............
어디파는데알수없을가요 ㅜ ㅜ ??!?!?
[Finished in 4.7s]

请告诉我们这是否有帮助。