python - 如何将参数传递给 scrapy 管道对象

Question

用 Scrapy 蜘蛛抓取一些数据后:

class Test_Spider(Spider):

    name = "test"
    def start_requests(self):
        for i in range(900,902,1):
            ........
            yield item

我将数据传递给管道对象，使用 SQLAlchemy::将其写入 SQLite 表

class SQLlitePipeline(object):

    def __init__(self):
        _engine = create_engine("sqlite:///data.db")
        _connection = _engine.connect()
        _metadata = MetaData()
        _stack_items = Table("table1", _metadata,
                             Column("id", Integer, primary_key=True),
                             Column("detail_url", Text),
        _metadata.create_all(_engine)
        self.connection = _connection
        self.stack_items = _stack_items

    def process_item(self, item, spider):
        is_valid = True

我希望能够将表名设置为变量，而不是像现在这样硬编码“table1”。如何做到这一点？

Answer 1

假设您通过命令行传递此参数(例如 -s table="table1")，定义一个 from_crawler 方法。

@classmethod
def from_crawler(cls, crawler):
    # Here, you get whatever value was passed through the "table" parameter
    settings = crawler.settings
    table = settings.get('table')

    # Instantiate the pipeline with your table
    return cls(table)

def __init__(self, table):
    _engine = create_engine("sqlite:///data.db")
    _connection = _engine.connect()
    _metadata = MetaData()
    _stack_items = Table(table, _metadata,
                         Column("id", Integer, primary_key=True),
                         Column("detail_url", Text),
    _metadata.create_all(_engine)
    self.connection = _connection
    self.stack_items = _stack_items