python - 使用beautifulsoup抓取时如何在html日期选择器中输入日期范围?

原文 标签 python python-3.x web-scraping beautifulsoup

我正在尝试以html形式输入自定义日期范围,以抓取给定范围的数据。 HTML代码如下所示:

<div class="dateRange inlineblock datePickerBinder arial_11 lightgrayFont" 
     id="widgetFieldDateRange">03/19/2019 - 04/18/2019</div>
</div>
<input id="picker" type="hidden" value=" 03/19/2019 - 04/18/2019">


我尝试了以下方法:

import requests
import urllib.parse as urlParse

url = 'https://www.investing.com/funds/lansforsakringar-global-indexnara-historical-data'
values = {'start':'01/18/2019','end':'04/18/2019'}

# pretend to be a chrome 47 browser on a windows 10 machine
headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36",
    "Accept" : "text/plain, */*; q=0.01",
    "Content-Type" : "application/x-www-form-urlencoded",
    "X-Requested-With" : "XMLHttpRequest"
}

# encode values for the url
params = urlParse.urlencode(values).encode("utf-8")

# create the url
s = requests.Session()  
targetUrl = s.post(url=url, data=params, headers=headers)


# open the url
html = BeautifulSoup(targetUrl.content, "html.parser")

# read the response
print(html.prettify)


但是在打印响应时,我看到设置了默认日期范围,并且尚未应用我的自定义日期范围。我该如何解决这个问题?

还发现了这一点,我相信是发布日期的javascript

<script type="text/javascript">
            window.siteData = {
                htmlDirection: 'ltr',
                decimalPoint: '.' || '.',
                thousandSep: ',' || ',',
                isEu : false,
                userLoggedIn: false,
                userHasPhoneRegistered: false,
                currencyPosition: 'left',
                datepicker: {
                    applyButton: 'Apply',
                    format: 'm/d/Y',
                    formatShort: 'm/d/y',
                    formatLong: 'm/d/Y',
                    formatSend: 'yy-mm-dd',
                    firstDay: '1',
                    dayNames: ["Su","Mo","Tu","We","Th","Fr","Sa"],
                    monthNamesShort: ["Jan.", "Feb.", "Mar.", "Apr.", "May", "Jun.", "Jul.", "Aug.", "Sep.", "Oct.", "Nov.", "Dec."],
                    monthNames: ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"],
                    translations: {
                        custom: 'Custom dates',
                        start: 'Start Date',
                        end: 'End Date'
                    }

最佳答案

以下内容将帮助您单击日历菜单并使用Selenium输入值。该页面有一个Ajax POST,但我无法传递正确的cookie(我认为)

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

d = webdriver.Chrome()
d.get('https://www.investing.com/funds/lansforsakringar-global-indexnara-historical-data')
try:  #attempt to dismiss banners that could block later clicks
    WebDriverWait(d, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".closer"))).click()
    d.find_element_by_css_selector('.closer').click()
except:
    pass
d.find_element_by_id('widgetFieldDateRange').click() #show the date picker
sDate  = d.find_element_by_id('startDate') # set start date input element into variable
sDate.clear() #clear existing entry
sDate.send_keys('01/18/2019') #add custom entry
eDate = d.find_element_by_id('endDate') #repeat for end date
eDate.clear()
eDate.send_keys('04/18/2019')
d.find_element_by_id('applyBtn').click() #submit changes

关于python - 使用beautifulsoup抓取时如何在html日期选择器中输入日期范围?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/55750897/

相关文章:

Python async/await 执行时间的差异

python - 给定一个值,我如何知道它存在于哪些列中?

python - PyQt5 QWebEnginePage - 可以编辑 HTML 以打开下拉列表吗?

python - 尝试解压集合时的 ​​TypeError 与 ValueError

python - 在 Matplotlib 中使用 LineCollection 时是否可以设置透明度?

python - 动态创建类

python - 包含Django应用的url.py会产生404

python-3.x - 用组的现有值替换 NaN

python - 用xpath/lxml抓取文本

python - 无法使用beautifulsoup 3从json脚本中提取所有URL