python - 如何序列化作为 XML Exporter 中的项目列表的 Scrapy 字段

标签 python xml serialization scrapy exporter

我构建了复杂的项目，其中字段可能是其他项目类型的列表。当我使用默认 XmlItemExporter 导出它时子列表项的前缀为 <value>标签。我正在寻找如何将子项目标识符分配给这些值标签的示例。

文档的“项目导出器”页面解释了这句话:

Unless overridden in the serialize_field() method, multi-valued fields are exported by serializing each value inside a <value> element. This is for convenience, as multi-valued fields are very common.

文档页面还提供了在字段中声明序列化器和重写 Serialize_Field() 方法的简单示例，但两者都是针对单值字段，没有建议如何针对多值字段自定义它们。

我在网上搜索了如何完成此操作的示例，但没有找到任何示例。

这是我用于测试的示例项目树:

class Course(scrapy.Item):
    title = scrapy.Field()
    lessons = scrapy.Field()

class Lesson(scrapy.Item):
    session = scrapy.Field()
    topic = scrapy.Field()
    assignment = scrapy.Field()

class ReadingAssignment(scrapy.Item):
    textBook = scrapy.Field()
    pages = scrapy.Field()

course = Course()
course['title'] = 'Greatness'
course['lessons'] = []

lesson = Lesson()
lesson['session'] = 'Week 1'
lesson['topic'] = 'Think Great'
lesson['assignment'] = []

reading =  ReadingAssignment()
reading['textBook'] = 'Great Book 1'
reading['pages'] = '1-20'
lesson['assignment'].append(reading)
course['lessons'].append(lesson)

lesson = Lesson()
lesson['session'] = 'Week 2'
lesson['topic'] = 'Act Great'
lesson['assignment'] = []

reading =  ReadingAssignment()
reading['textBook'] = 'Great Book 2'
reading['pages'] = '21-40'
lesson['assignment'].append(reading)
course['lessons'].append(lesson)

lesson = Lesson()
lesson['session'] = 'Week 3'
lesson['topic'] = 'Look Great'
lesson['assignment'] = []

reading =  ReadingAssignment()
reading['textBook'] = 'Great Book 3'
reading['pages'] = '41-60'
lesson['assignment'].append(reading)
course['lessons'].append(lesson)

lesson = Lesson()
lesson['session'] = 'Week 4'
lesson['topic'] = 'Be Great'
lesson['assignment'] = []

reading =  ReadingAssignment()
reading['textBook'] = 'Great Book 4'
reading['pages'] = '61-80'
lesson['assignment'].append(reading)
course['lessons'].append(lesson)

输出:

>>> course
{'lessons': [{'assignment': [{'pages': '1-20', 'textBook': 'Great Book 1'}],
              'session': 'Week 1',
              'topic': 'Think Great'},
             {'assignment': [{'pages': '21-40', 'textBook': 'Great Book 2'}],
              'session': 'Week 2',
              'topic': 'Act Great'},
             {'assignment': [{'pages': '41-60', 'textBook': 'Great Book 3'}],
              'session': 'Week 3',
              'topic': 'Look Great'},
             {'assignment': [{'pages': '61-80', 'textBook': 'Great Book 4'}],
              'session': 'Week 4',
              'topic': 'Be Great'}],
 'title': 'Greatness'}

当我通过 XmlItemExporter 运行此命令时我得到:

<?xml version="1.0" encoding="utf-8"?>
<items>
  <course>
    <title>Greatness</title>
    <lessons>
      <value>
        <session>Week 1</session>
        <topic>Think Great</topic>
        <assignment>
          <value>
            <textBook>Great Book 1</textBook>
            <pages>1-20</pages>
          </value>
        </assignment>
      </value>
      <value>
        <session>Week 2</session>
        <topic>Act Great</topic>
        <assignment>
          <value>
            <textBook>Great Book 2</textBook>
            <pages>21-40</pages>
          </value>
        </assignment>
      </value>
      <value>
        <session>Week 3</session>
        <topic>Look Great</topic>
        <assignment>
          <value>
            <textBook>Great Book 3</textBook>
            <pages>41-60</pages>
          </value>
        </assignment>
      </value>
      <value>
        <session>Week 4</session>
        <topic>Be Great</topic>
        <assignment>
          <value>
            <textBook>Great Book 4</textBook>
            <pages>61-80</pages>
          </value>
        </assignment>
      </value>
    </lessons>
  </course>
</items>

我想做的是改变那些 <value>标签附加到列表中的项目名称。像这样:

<items>
  <course>
    <title>Greatness</title>
    <lessons>
      <lesson>
        <session>Week 1</session>
        <topic>Think Great</topic>
        <assignment>
          <reading>
            <textBook>Great Book 1</textBook>
            <pages>1-20</pages>
          </reading>
        </assignment>
      </lesson>
      <lesson>
        <session>Week 2</session>
        <topic>Act Great</topic>
        <assignment>
          <reading>
            <textBook>Great Book 2</textBook>
            <pages>21-40</pages>
          </reading>
        </assignment>
      </lesson>
      <lesson>
        <session>Week 3</session>
        <topic>Look Great</topic>
        <assignment>
          <reading>
            <textBook>Great Book 3</textBook>
            <pages>41-60</pages>
          </reading>
        </assignment>
      </lesson>
      <lesson>
        <session>Week 4</session>
        <topic>Be Great</topic>
        <assignment>
          <reading>
            <textBook>Great Book 4</textBook>
            <pages>61-80</pages>
          </reading>
        </assignment>
      </lesson>
    </lessons>
  </course>
</items>

最佳答案

这确实没有很好的记录，我们必须阅读 XmlItemExporter source code ，结果表明 <value>标签选择已硬编码在 XmlItemExporter._export_xml_field() method 中:

elif is_listlike(serialized_value):
    self._beautify_newline()
    for value in serialized_value:
        self._export_xml_field('value', value, depth=depth+1)
    self._beautify_indent(depth=depth)

幸运的是，有出路，就在前面的几行:

if hasattr(serialized_value, 'items'):
    self._beautify_newline()
    for subname, value in serialized_value.items():
        self._export_xml_field(subname, value, depth=depth+1)
    self._beautify_indent(depth=depth)

这意味着要处理字典，但实际上它会接受任何具有 .items() 的东西。返回字符串和项目元组的方法!

但是，导出器中缺少一个重要步骤:递归。基本上只能设置serializer顶级项目字段上的标志，任意 Field() Item 上的元素当前 Scrapy 实现完全忽略顶级项目之外的子类。每个导出商在如何驱动内部 BaseItemExporter._get_serialized_fields() method方面都有自己的特点。 ，因此我们不能预先处理递归，因为每个特定的导出器(JSON、XML 等)在需要字段序列化的方式上有所不同。我们可以使用 XmlItemExporter 的子类来解决这个问题类，更多内容见下文。

因此，这里的第一个技巧是创建一个具有 .items() 的专用对象。方法并为您提供 <container>标签。请注意，您必须自己处理序列化的递归! Scrapy 序列化器本身不处理嵌套结构的递归:

class CustomXMLValuesSerializer:
    @classmethod
    def serialize_as(cls, name):
        def serializer(items, serialize):
            return cls(name, items, serialize)
        return serializer

    def __init__(self, name, items, serialize=None):
        self._name = name
        self._items = items
        self._serialize = serialize if serialise is not None else lambda x: x

    def items(self):
        for item in self._items:
            yield (self._name, self._serialize(item))

然后使用 CustomXMLValuesSerializer.serialize_as()类方法为您的列表字段创建自定义序列化器:

class Course(scrapy.Item):
    title = scrapy.Field()
    lessons = scrapy.Field(
        serializer=CustomXMLValuesSerializer.serialize_as("lesson")
    )

class Lesson(scrapy.Item):
    session = scrapy.Field()
    topic = scrapy.Field()
    assignment = scrapy.Field(
        serializer=CustomXMLValuesSerializer.serialize_as("reading")
    )

class ReadingAssignment(scrapy.Item):
    textBook = scrapy.Field()
    pages = scrapy.Field()

最后，我们需要一个稍微定制的导出器，它实际上可以让我们递归地处理嵌套项目:

from functools import partial

class RecursingXmlItemExporter(XmlItemExporter):
    def _recursive_serialized_fields(self, item):
        if isinstance(item, scrapy.Item):
            return dict(self._get_serialized_fields(item, default_value=''))
        return item

    def serialize_field(self, field, name, value):
        serializer = field.get('serializer', lambda x: x)
        try:
            return serializer(value, self._recursive_serialized_fields)
        except TypeError:
            return serializer(value)

请注意，这会传入 default_value='' ，因为that's what the base XmlItemExporter.export_item() implementation uses .

确保使用此自定义导出器，因为它会传入序列化嵌套项所需的上下文:

exporter = RecursingXmlItemExporter(some_file, indent=2, item_element='course')
exporter.start_exporting()
exporter.export_item(course)
exporter.finish_exporting()

现在容器实际上是使用 name 导出的string 作为容器元素:

<?xml version="1.0" encoding="utf-8"?>
<items>
  <course>
    <title>Greatness</title>
    <lessons>
      <lesson>
        <session>Week 1</session>
        <topic>Think Great</topic>
        <assignment>
          <reading>
            <textBook>Great Book 1</textBook>
            <pages>1-20</pages>
          </reading>
        </assignment>
      </lesson>
      <lesson>
        <session>Week 2</session>
        <topic>Act Great</topic>
        <assignment>
          <reading>
            <textBook>Great Book 2</textBook>
            <pages>21-40</pages>
          </reading>
        </assignment>
      </lesson>
      <lesson>
        <session>Week 3</session>
        <topic>Look Great</topic>
        <assignment>
          <reading>
            <textBook>Great Book 3</textBook>
            <pages>41-60</pages>
          </reading>
        </assignment>
      </lesson>
      <lesson>
        <session>Week 4</session>
        <topic>Be Great</topic>
        <assignment>
          <reading>
            <textBook>Great Book 4</textBook>
            <pages>61-80</pages>
          </reading>
        </assignment>
      </lesson>
    </lessons>
  </course>
</items>

我字段issue #3888用Scrapy看看项目是否有兴趣支持嵌套Item结构更好。

另一种方法是通过单独调用 XmlItemExporter.export_item() 来导出嵌套项目。方法，但这要求导出器可以作为与序列化器相同的命名空间中的全局变量进行访问，或者您对导出器进行子类化并...将导出器传递给序列化器。然后你必须满足于 XmlItemExporter.export_item() 的事实对缩进进行硬编码。

关于python - 如何序列化作为 XML Exporter 中的项目列表的 Scrapy 字段，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/57110413/

上一篇：python - 基于两个 pandas 列创建一系列浮点值

下一篇：python - 如何将两个文本文件合并为一个？

相关文章：

python - 使用 nltk 改进人名的提取

python - 尝试 MNIST 数据集时，tensorflow 和 matplotlib 包出现问题

"normal" View 上的 Android 自定义属性？

c#反序列化json为List

python - 内爆列表以在 python MySQLDB IN 子句中使用

python - 如何找到赢家通吃的最高值(value)

java - xml java xslt 标签

python - 如何使用 Python 中的 ElementTree 从特定供应商的 XML 格式的 nmap 输出中获取 IP 地址

c++ - boost序列化异常: unregistered class,序列化多态基础问题

java - 如何在java中使用紧凑字节将对象序列化为字节数组

©2024 IT工具网  联系我们