python - 获取scrapy上不同部分的数据列表

Question

我需要在网站的不同部分中抓取数据。在第一部分中，我从客户那里获取数据和订单 ID。通过此 ID，我可以访问第二部分并从订单中获取商品详细信息。因此，我需要将字典“customer”的结果与“orders”列表和“itens”列表连接起来。

基本上，我的算法是:

def parse1(self, response):
    customer['data'] = response.xpath("path to customer data").extract()
    customer_orders = response.xpath("path to customer orders")
    for index, customer_order in enumarate(customer_orders):
         id = customer_order.xpath('path to order id').extract_first()
         customer['orders'].append({'id' : id})
         yield scrapy.FormRequest(url="www.url.com/orders"+id, callback=self.parse2, method='GET', meta= {'customer': customer})

def parse2(self, response):
    customer = response.meta['customer']
    customer['orders']['items'] = []  
    for index, order_item in response.xpath("path to order items"):
           customer['orders']['items'].append({"items_details": order_item.xpath("path to items details").extract_first()})
    yield customer

但我无法使用 Scrapy 异步架构编写此逻辑。更接近的是，我得到的结果是多次打印相同的客户指令。任何人都可以帮助做到这一点吗？

Answer 1

由于您对 1 个项目有 A 和 B 类型请求，因此您有两个链请求按顺序执行:首先抓取 A，然后抓取 B N 次:

customer -> N order pages -> 1 item

所以你的抓取逻辑是:

1. 获取客户数据
2. 获取订单 ID
   2.1 弹出订单id
   2.2 抓取订单id
   2.3 将订单​​详细信息附加到 #1 客户数据
3. 返回客户数据和订单数据

在 scrapy 中它看起来像这样:

def parse_customer(self, response):
    # find root customer data
    customer = {}
    # find order ids
    orders = [1,2,3]
    # schedule first order request and start order scraping loop
    first_order = order_url + orders.pop(0)
    yield Request(
        first_order, 
        self.parse_orders, 
        meta={'orders': orders, 'item': customer},
        )

def parse_orders(self, response):
    item = response.meta['item']
    remaining_orders = response.meta['orders']
    # first loop it's [1, 2] but finally it'll be []
    if not remaining_orders:  # all orders are scraped -> save item
        yield item
        return

    # attach found order details to root customer item we have
    found_orders = ...
    item['orders'].expand(found_orders)

    # scrape next order
    next_order = order_url + orders.pop(0),
    yield Request(
        next_order,
        self.parse_orders, 
        meta={'orders': orders, 'item': item},
        )