我正在尝试抓取此网址上的项目:
"https://www.bloomberg.com/news/articles/2019-05-30/tesla-dealt-another-blow-as-barclays-sees-it-as-niche-carmaker"
我只想获取标题和发布日期, 您可以给我任何示例代码,甚至是splash等
到目前为止我尝试过的是这个
def parse(self, response):
yield scrapy.Request('https://www.bloomberg.com/news/articles/2019-05-30/tesla-dealt-another-blow-as-barclays-sees-it-as-niche-carmaker -H x-crawlera-use-https:1',
headers={'X-Crawlera-Session': create,
'X-Crawlera-Timeout': 40000,
'X-Crawlera-Max-Retries': 5,
'X-Crawlera-Cookies': disable,
'X-Crawlera-Session': self.session_id
},
callback=self.parse_sub,
)
def parse_sub(self, response):
response.xpath("//h1[@class = 'lede-text-v2__hed']").extract_first()
response.xpath("//meta[@property = 'og:title']/@content").extract_first()
response.xpath("//time[@class = 'article-timestamp']/@datetime").extract_first()
print(response.text)
我也在使用crawlera,但它一直将我检测为机器人
最佳答案
仅使用 selenium提取标题,即特斯拉在巴克莱称其为“利基汽车制造商”时再次受到打击和发布日期,即5月30日, 2019 年下午 5:26 GMT+5:30 您必须为 visibility_of_element_ located()
引入 WebDriverWait,并且您可以使用以下解决方案:
代码块
from selenium import webdriver driver = webdriver.Firefox(executable_path=r'C:\Utility\BrowserDrivers\geckodriver.exe') driver.get('https://www.bloomberg.com/news/articles/2019-05-30/tesla-dealt-another-blow-as-barclays-sees-it-as-niche-carmaker') print(WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//div[text()='markets']//following:: h1[1]"))).get_attribute("innerHTML")) print(WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//div[text()='markets']//following:: h1[1]//following::div[@class='lede-text-v2__times']/time[@itemprop='datePublished']"))).get_attribute("innerHTML")) driver.quit()
控制台输出:
Tesla Dealt Another Blow When Barclays Calls It a ‘Niche Carmaker’ May 30, 2019, 5:26 PM GMT+5:30
注意:您必须添加以下导入:
from selenium import webdriver from selenium.webdriver.support.ui import WebDriverWait from selenium.webdriver.common.by import By from selenium.webdriver.support import expected_conditions as EC
关于python - 如何从彭博等安全网站提取数据,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/56537590/