我是 python 和网络抓取的新手。我想这很简单，但我无法让它发挥作用。
我使用 Flask 创建了一个本地网页，其中包含一个表和一个填充该表的函数。 下一步是在其他计算机上获取这些数据。这是我尝试过的:
import requests
from bs4 import BeautifulSoup
requests.get('http://127.0.0.1:5000')
soup = BeautifulSoup(source_code)
这就是我得到的:
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="utf-8"/>
<link href="https://www.w3schools.com/w3css/4/w3.css" rel="stylesheet"/>
<link href="https://use.fontawesome.com/releases/v5.7.0/css/all.css" rel="stylesheet"/>
<title>Title</title>
</head>
<body>
<div class="w3-container w3-teal">
<h1>SpaceWire Devices</h1>
</div>
<div>
<table class="w3-table-all w3-large" id="device_table">
<colgroup>
<col/>
<col/>
<col/>
</colgroup>
<tr class="w3-blue">
<th>Device ID</th>
<th colspan="2">Channel 1 (Left)</th>
<th colspan="2">Channel 2 (Left)</th>
</tr>
</table>
</div>
<script>
//first add an event listener for page load
document.addEventListener( "DOMContentLoaded", get_json_data, false ); // get_json_data is the function name that will fire on page load
//this function is in the event listener and will execute on page load
function get_json_data(){
// Relative URL of external json file
var json_url = '/status';
//Build the XMLHttpRequest (aka AJAX Request)
xmlhttp = new XMLHttpRequest();
xmlhttp.onreadystatechange = function() {
if (this.readyState == 4 && this.status == 200) {//when a good response is given do this
var data = JSON.parse(this.responseText); // convert the response to a json object
append_json(data);// pass the json object to the append_json function
}
}
//set the request destination and type
xmlhttp.open("get", json_url, true);
//set required headers for the request
// xmlhttp.setRequestHeader("Content-type", "application/x-www-form-urlencoded");
// send the request
xmlhttp.send(); // when the request completes it will execute the code in onreadystatechange section
}
//this function appends the json data to the table 'gable'
function append_json(data){
var table = document.getElementById('device_table');
for (var device in data) {
var tr = document.createElement('tr');
tr.innerHTML = '<td>' + data[device].id + '</td>' +
'<td>' + data[device].channel_1 + '</td>' +
'<td>' + data[device].channel_1_port + '</td>' +
'<td>' + data[device].channel_2 + '</td>' +
'<td>' + data[device].channel_2_port + '</td>'
table.appendChild(tr);
};
}
</script>
</body>
</html>
我真正想要的是append_json()最终创建的数据。我该怎么做？
最佳答案
Jammy Dodger 关于 Selenium 的评论是正确的。 html是由js生成的。您的请求不允许该代码像在浏览器中一样执行。我会使用 selenium 打开页面，然后以这种方式检索 DOM。从那里您可以导航并抓取您想要的数据。它应该看起来像这样。
from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary
from selenium.webdriver.firefox.options import Options
options = Options()
options.add_argument("--headless")
binary = FirefoxBinary('/usr/lib/firefox/firefox')
browser = webdriver.Firefox(firefox_options=options, firefox_binary=binary)
url = f'https://www.rottentomatoes.com/search/?search={title}'
try:
browser.get(url)
# Give the js a little bit of time to generate the html
time.sleep(1)
html = browser.page_source
browser.quit()
soup = BeautifulSoup(html, 'lxml')
关于python - 我正在尝试网络抓取，但得到的是函数而不是实际数据，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/58383491/