python - 使用 beautifulsoup 抓取时如何在 html 日期选择器中输入日期范围？

Question

我正在尝试以 html 表单输入自定义日期范围来抓取给定范围的数据。 HTML 代码如下所示:

<div class="dateRange inlineblock datePickerBinder arial_11 lightgrayFont" 
     id="widgetFieldDateRange">03/19/2019 - 04/18/2019</div>
</div>
<input id="picker" type="hidden" value=" 03/19/2019 - 04/18/2019">

我尝试过以下方法:

import requests
import urllib.parse as urlParse

url = 'https://www.investing.com/funds/lansforsakringar-global-indexnara-historical-data'
values = {'start':'01/18/2019','end':'04/18/2019'}

# pretend to be a chrome 47 browser on a windows 10 machine
headers = {
    "User-Agent" : "Mozilla/5.0 (Windows NT 10.0; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/47.0.2526.106 Safari/537.36",
    "Accept" : "text/plain, */*; q=0.01",
    "Content-Type" : "application/x-www-form-urlencoded",
    "X-Requested-With" : "XMLHttpRequest"
}

# encode values for the url
params = urlParse.urlencode(values).encode("utf-8")

# create the url
s = requests.Session()  
targetUrl = s.post(url=url, data=params, headers=headers)


# open the url
html = BeautifulSoup(targetUrl.content, "html.parser")

# read the response
print(html.prettify)

但是在打印响应时，我看到默认日期范围已设置，并且我的自定义日期范围尚未应用。我该如何解决这个问题？

也发现了这个，我相信是发布日期的 javascript

<script type="text/javascript">
            window.siteData = {
                htmlDirection: 'ltr',
                decimalPoint: '.' || '.',
                thousandSep: ',' || ',',
                isEu : false,
                userLoggedIn: false,
                userHasPhoneRegistered: false,
                currencyPosition: 'left',
                datepicker: {
                    applyButton: 'Apply',
                    format: 'm/d/Y',
                    formatShort: 'm/d/y',
                    formatLong: 'm/d/Y',
                    formatSend: 'yy-mm-dd',
                    firstDay: '1',
                    dayNames: ["Su","Mo","Tu","We","Th","Fr","Sa"],
                    monthNamesShort: ["Jan.", "Feb.", "Mar.", "Apr.", "May", "Jun.", "Jul.", "Aug.", "Sep.", "Oct.", "Nov.", "Dec."],
                    monthNames: ["January", "February", "March", "April", "May", "June", "July", "August", "September", "October", "November", "December"],
                    translations: {
                        custom: 'Custom dates',
                        start: 'Start Date',
                        end: 'End Date'
                    }

Answer 1

以下内容应该可以帮助您单击日历菜单并使用 Selenium 输入值。该页面有一个 Ajax POST，但我无法传递正确的 cookie(我认为)

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

d = webdriver.Chrome()
d.get('https://www.investing.com/funds/lansforsakringar-global-indexnara-historical-data')
try:  #attempt to dismiss banners that could block later clicks
    WebDriverWait(d, 10).until(EC.element_to_be_clickable((By.CSS_SELECTOR, ".closer"))).click()
    d.find_element_by_css_selector('.closer').click()
except:
    pass
d.find_element_by_id('widgetFieldDateRange').click() #show the date picker
sDate  = d.find_element_by_id('startDate') # set start date input element into variable
sDate.clear() #clear existing entry
sDate.send_keys('01/18/2019') #add custom entry
eDate = d.find_element_by_id('endDate') #repeat for end date
eDate.clear()
eDate.send_keys('04/18/2019')
d.find_element_by_id('applyBtn').click() #submit changes