python - 当我使用 selenium 爬取网站时出现 python UnicodeEncodeError

Question

我试图使用selenium来抓取这个网站上论文的标题:http://www.ncbi.nlm.nih.gov/pubmed?term=(%222013%22%5BDate%20-%20Publication%5D%20%3A%20%222013%22%5BDate%20-%20Publication%5D)

#coding="utf-8"

from selenium import webdriver
from selenium.webdriver.common.keys import Keys
import time

domain = "http://www.ncbi.nlm.nih.gov/"
url_tail = "pubmed?term=(%222013%22%5BDate%20-%20Publication%5D%20%3A%20%222013%22%5BDate%20-%20Publication%5D)"
url = domain + url_tail

browser = webdriver.Firefox()
browser.get(url)
time.sleep(5)

def extract_data(browser):
    titles = browser.find_elements_by_css_selector("div.rprt div.rslt p.title a")
    return [title.text for title in titles]

page_start = 1
page_end = 10

f = open('titles.txt', 'a')
for page in range(page_start, page_end):
    print "page %d" % page
    page_jump_box = browser.find_element_by_class_name("num").clear()
    page_jump_box_cleared = browser.find_element_by_class_name("num")
    page_jump_box_cleared.send_keys(str(page) + Keys.RETURN)

    time.sleep(15)

    f = open('titles.txt', 'a')
    for line in extract_data(browser):
        f.write(line + '\n')

f.close()

当我运行它时，我得到了这个:

gao@gao:~/crawler$ python crawler3.0.py 
page 1
page 2
page 3
page 4
Traceback (most recent call last):
  File "crawler3.0.py", line 33, in <module>
    f.write(line + '\n')
UnicodeEncodeError: 'ascii' codec can't encode character u'\u03b1' in position 36: ordinal not in range(128)

当我在Stackoverflow上搜索时，我发现了类似的问题:UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 20: ordinal not in range(128) 。 我了解到当你使用str()时，它会导致unicode问题。但是在我的代码中，我只使用str()来使page数字成为一个字符串。那么，如何更正代码。

这是另一个问题。我了解到，如果我想将 phantomjs 与 selenium 一起使用，我只需将 browser = webdriver.Firefox() 更改为 browser = webdriver .PhantomJS()，但是当我这样做时，我抓取的内容被重复(仅抓取了第1页的标题)。

我的母语不是英语，如果有任何语法错误或任何错误，请告诉我。

提前致谢。

Answer 1

在写入文件之前，您需要对该行进行编码:

for line in extract_data(browser):
    f.write(line.encode('utf-8') + '\n')

---

对于您的第二个问题，我建议进行以下改进(这将使其发挥作用):

• 使用Explicit Waits而不是 time.sleep() 调用 - 这也会显着提高性能
• 不要输入页码，而是点击“下一步”按钮
• 在循环之前以“追加”模式打开文件并使用 with context manager
• close() 完成后关闭浏览器

代码:

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

domain = "http://www.ncbi.nlm.nih.gov/"
url_tail = "pubmed?term=(%222013%22%5BDate%20-%20Publication%5D%20%3A%20%222013%22%5BDate%20-%20Publication%5D)"
url = domain + url_tail

browser = webdriver.PhantomJS()
browser.get(url)


def extract_data(browser):
    titles = browser.find_elements_by_css_selector("div.rprt div.rslt p.title a")
    return [title.text for title in titles]


page_start, page_end = 1, 10

with open('titles.txt', 'a') as f:
    for page in range(page_start, page_end):
        WebDriverWait(browser, 10).until(
            EC.presence_of_element_located((By.CSS_SELECTOR, "div.rprt p.title"))
        )

        for line in extract_data(browser):
            f.write(line.encode('utf-8') + '\n')

        print "page %d" % page

        browser.find_element_by_css_selector("div.pagination a.next").click()

browser.close()

这会生成 titles.txt，其中包含结果页 1-9 的标题:

Robotic-assisted tubal anastomosis with one-stitch technique.
Effectiveness of pictorial health warnings on cigarette packs among Lebanese school and university students.
...
Importance and globalization status of good manufacturing practice (GMP) requirements for pharmaceutical excipients.
Systemic review on drug related hospital admissions - A pubmed based search.