javascript - 在 Puppeteer 中循环访问一组 url

Question

我如何使用 Puppeteer 从多个 URL 中抓取内容？

我已经创建了一个循环，但我只看到了第一个 url 的结果。

我怀疑这与我声明结果变量的位置有关，但我没有成功尝试，有人知道如何做到这一点吗？

const puppeteer = require('puppeteer');

function run() {
    return new Promise(async (resolve, reject) => {
        try {

            const browser = await puppeteer.launch();
            const page = await browser.newPage();
            const urls = ["https://www.marksandspencer.com/high-neck-long-sleeve-blouse/p/p60260040?image=SD_01_T43_5168_HD_X_EC_90&color=LIGHTDENIM&prevPage=plp", "https://www.marksandspencer.com/pure-cotton-printed-short-sleeve-t-shirt/p/p60263529?image=SD_01_T41_8030Z_Z4_X_EC_90&color=WHITEMIX&prevPage=plp"];
            

              for (let i = 0; i < urls.length; i++) {
                const url = urls[i];
                await page.goto(url);
                let products = await page.evaluate(() => {
                    let product = document.querySelector('h1[itemprop=name]').innerText;
                    let results = [];
                    let items = document.querySelectorAll('[data-ttip-id=sizeGridTooltip] tbody tr td label');
                    items.forEach((element) => {
                        let size = element.getAttribute('for');
                        let stockLevel = "";
                        let nearest_td = element.closest('td');
                        if (nearest_td.classList.contains('low-stock')) {
                            stockLevel = "Low stock"
                        } else if (nearest_td.classList.contains('out-of-stock')) {
                            stockLevel = "Out of stock"
                        } else {
                            stockLevel = "In stock"
                        }
                        results.push({
                            product: product,
                            size: size,
                            stock: stockLevel
                        })
                    });
                    return results
                })
                browser.close();
                return resolve(products);
            }
            
        } catch (e) {
            return reject(e);
        }
    })
}
run().then(console.log).catch(console.error);

Answer 1

这些行在你的 for 循环中:

                browser.close();
                return resolve(products);

因此，作为第一次迭代的一部分，您关闭浏览器并返回函数。您应该将其移出您的 for 循环并将 products 存储在这样的数组中:

              const urls = /* ... */;
              const productsList = [];

              for (let i = 0; i < urls.length; i++) {
                const url = urls[i];
                await page.goto(url);
                let products = await page.evaluate(/* ... */);
                productsList.push(products);
              }
              browser.close();
              return resolve(productsList); // resolve with an array containing the aggregated products

如果您正在寻找更优雅的解决方案(用于并行抓取页面)，您可能想看看库 puppeteer-cluster (免责声明:我是作者)。