Python 解析奇怪的 XML？

Question

我正在尝试解析这个奇怪的 XML，读完本文后，我仍然遇到问题。

我正在尝试解析 NIST CVE 数据库，它仅以 XML 形式提供。这是它的一个示例。

<?xml version='1.0' encoding='UTF-8'?>
<nvd xmlns:scap-core="http://scap.nist.gov/schema/scap-core/0.1" xmlns:cvss="http://scap.nist.gov/schema/cvss-v2/0.2" xmlns:vuln="http://scap.nist.gov/schema/vulnerability/0.4" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:patch="http://scap.nist.gov/schema/patch/0.1" xmlns="http://scap.nist.gov/schema/feed/vulnerability/2.0" xmlns:cpe-lang="http://cpe.mitre.org/language/2.0" nvd_xml_version="2.0" pub_date="2017-04-12T18:00:08" xsi:schemaLocation="http://scap.nist.gov/schema/patch/0.1 https://scap.nist.gov/schema/nvd/patch_0.1.xsd http://scap.nist.gov/schema/feed/vulnerability/2.0 https://scap.nist.gov/schema/nvd/nvd-cve-feed_2.0.xsd http://scap.nist.gov/schema/scap-core/0.1 https://scap.nist.gov/schema/nvd/scap-core_0.1.xsd">
  <entry id="CVE-2013-7450">
    <vuln:vulnerable-configuration id="http://nvd.nist.gov/">
      <cpe-lang:logical-test operator="OR" negate="false">
        <cpe-lang:fact-ref name="cpe:/a:pulp_project:pulp:2.2.1-1"/>
      </cpe-lang:logical-test>
    </vuln:vulnerable-configuration>
    <vuln:vulnerable-software-list>
      <vuln:product>cpe:/a:pulp_project:pulp:2.2.1-1</vuln:product>
    </vuln:vulnerable-software-list>
    <vuln:cve-id>CVE-2013-7450</vuln:cve-id>
    <vuln:published-datetime>2017-04-03T11:59:00.143-04:00</vuln:published-datetime>
    <vuln:last-modified-datetime>2017-04-11T10:01:04.323-04:00</vuln:last-modified-datetime>
    <vuln:cvss>
      <cvss:base_metrics>
        <cvss:score>5.0</cvss:score>
        <cvss:access-vector>NETWORK</cvss:access-vector>
        <cvss:access-complexity>LOW</cvss:access-complexity>
        <cvss:authentication>NONE</cvss:authentication>
        <cvss:confidentiality-impact>NONE</cvss:confidentiality-impact>
        <cvss:integrity-impact>PARTIAL</cvss:integrity-impact>
        <cvss:availability-impact>NONE</cvss:availability-impact>
        <cvss:source>http://nvd.nist.gov</cvss:source>
        <cvss:generated-on-datetime>2017-04-11T09:43:13.623-04:00</cvss:generated-on-datetime>
      </cvss:base_metrics>
    </vuln:cvss>
    <vuln:cwe id="CWE-295"/>
    <vuln:references xml:lang="en" reference_type="VENDOR_ADVISORY">
      <vuln:source>MLIST</vuln:source>
      <vuln:reference href="http://www.openwall.com/lists/oss-security/2016/04/18/11" xml:lang="en">[oss-security] 20160418 CVE-2013-7450: Pulp &lt; 2.3.0 distributed the same CA key to all users</vuln:reference>
    </vuln:references>
    <vuln:references xml:lang="en" reference_type="VENDOR_ADVISORY">
      <vuln:source>MLIST</vuln:source>
      <vuln:reference href="http://www.openwall.com/lists/oss-security/2016/04/18/5" xml:lang="en">[oss-security] 20160418 Re: CVE request - Pulp &lt; 2.3.0 shipped the same authentication CA key/cert to all users</vuln:reference>
    </vuln:references>
    <vuln:references xml:lang="en" reference_type="VENDOR_ADVISORY">
      <vuln:source>MLIST</vuln:source>
      <vuln:reference href="http://www.openwall.com/lists/oss-security/2016/05/20/1" xml:lang="en">[oss-security] 20160519 Pulp 2.8.3 Released to address multiple CVEs</vuln:reference>
    </vuln:references>
    <vuln:references xml:lang="en" reference_type="PATCH">
      <vuln:source>CONFIRM</vuln:source>
      <vuln:reference href="https://bugzilla.redhat.com/show_bug.cgi?id=1003326" xml:lang="en">https://bugzilla.redhat.com/show_bug.cgi?id=1003326</vuln:reference>
    </vuln:references>
    <vuln:references xml:lang="en" reference_type="PATCH">
      <vuln:source>CONFIRM</vuln:source>
      <vuln:reference href="https://bugzilla.redhat.com/show_bug.cgi?id=1328345" xml:lang="en">https://bugzilla.redhat.com/show_bug.cgi?id=1328345</vuln:reference>
    </vuln:references>
    <vuln:references xml:lang="en" reference_type="VENDOR_ADVISORY">
      <vuln:source>CONFIRM</vuln:source>
      <vuln:reference href="https://github.com/pulp/pulp/pull/627" xml:lang="en">https://github.com/pulp/pulp/pull/627</vuln:reference>
    </vuln:references>
    <vuln:summary>Pulp before 2.3.0 uses the same the same certificate authority key and certificate for all installations.</vuln:summary>
  </entry>
<nvd>

我试图用 ET 解析它，但我得到一些奇怪的输出......

例如，当我使用这个时，

with open('/tmp/nvdcve-2.0-modified 2.xml', 'rt') as f:
    tree = ElementTree.parse(f)
for child in root:
     print child.tag, child.attrib

我的输出看起来像这样...

{http://scap.nist.gov/schema/feed/vulnerability/2.0}entry {'id': 'CVE-2007-6759'}

令人困惑的是，如果我想迭代它，我似乎需要这样做..

for child in root.iter('{http://scap.nist.gov/schema/feed/vulnerability/2.0}entry'):

如果我这样做，我就不知道 child 的 child 是什么，或者什么。

例如，我试图提取 vuln:cve-id 和每个单独的 cvss:base_metrics (分数访问向量)、vuln:summary 和 vuln:product。

基本上，我试图每小时从 NIST 网站下载“xml 流”并将其更新到本地 mysql 数据库中，这样我就有了一个本地位置，在我的环境中执行漏洞评估时也可以进行查询。弄清楚如何迭代这些 XML 内容非常令人困惑。我想尝试将其转换为 JSON，但这似乎是一个不必要的额外步骤，可能会出现问题，因为没有 1:1 XML/JSON 转换。

Answer 1

是的，必须处理带有命名空间的 XML a little differently 。这是继续使用 ElementTree API 的另一个解决方案。

使用此库中的命名空间时，如果您看到 vuln:summary，则需要在根元素的 xmlns:vuln 中查找 vuln 命名空间code> 属性，然后将其引用为 {http://scap.nist.gov/schema/vulnerability/0.4}summary。

import xml.etree.ElementTree as ET
tree = ET.parse('nvdcve-2.0-Modified.xml')
root = tree.getroot()
# default namespace is given by xmlns attribute of root element, still must be provided
for entry in root.findall('{http://scap.nist.gov/schema/feed/vulnerability/2.0}entry'):
    product_list = []
    metric_list = []
    # just use the element's id attribute
    id = entry.get('id')

    summary = entry.find('{http://scap.nist.gov/schema/vulnerability/0.4}summary').text

    software = entry.find('{http://scap.nist.gov/schema/vulnerability/0.4}vulnerable-software-list')
    if software is not None:
        for sw in software.findall('{http://scap.nist.gov/schema/vulnerability/0.4}product'):
            product_list.append(sw.text)

    metrics = entry.find('{http://scap.nist.gov/schema/vulnerability/0.4}cvss')
    if metrics is not None:
        for metric in metrics.find('{http://scap.nist.gov/schema/cvss-v2/0.2}base_metrics').findall('*'):
            # we don't know the element name, but can get it with the tag property
            metric_list.append(metric.tag.replace('{http://scap.nist.gov/schema/cvss-v2/0.2}', '') + ': ' + metric.text)

    print(id, summary, product_list, metric_list)
    #save to database!