将csv读取到dataframe,然后使用lxml库将其转换为xml
这是我第一次处理xml,看来部分成功。任何帮助将不胜感激。
用于创建数据框的CSV文件:
Parent,Element,Text,Attribute
,TXLife,"
",{'Version': '2.25.00'}
TXLife,UserAuthRequest,"
",{}
UserAuthRequest,UserLoginName,*****,{}
UserAuthRequest,UserPswd,"
",{}
UserPswd,CryptType,None,{}
UserPswd,Pswd,****,{}
TXLife,TXLifeRequest,"
",{'PrimaryObjectID': 'Policy_1'}
TXLifeRequest,TransRefGUID,706D67C1-CC4D-11CF-91FB444554540000,{}
TXLifeRequest,TransType,Holding Change,{'tc': '502'}
TXLifeRequest,TransExeDate,2006-11-19,{}
TXLifeRequest,TransExeTime,13:15:33-07:00,{}
TXLifeRequest,ChangeSubType,"
",{}
ChangeSubType,ChangeTC,Change Participant,{'tc': '9'}
TXLifeRequest,OLifE,"
",{}
OLifE,Holding,"
",{'id': 'Policy_1'}
Holding,HoldingTypeCode,Policy,{'tc': '2'}
Holding,Policy,"
",{}
Policy,PolNumber,1234567,{}
Policy,LineOfBusiness,Annuity,{'tc': '2'}
Policy,Annuity,,{}
OLifE,Party,"
",{'id': 'Beneficiary_1'}
Party,PartyTypeCode,Organization,{'tc': '2'}
Party,FullName,The Smith Trust,{}
Party,Organization,"
",{}
Organization,OrgForm,Trust,{'tc': '16'}
Organization,DBA,The Smith Trust,{}
OLifE,Relation,"
","{'id': 'Relation_1', 'OriginatingObjectID': 'Policy_1', 'RelatedObjectID': 'Beneficiary_1'}"
Relation,OriginatingObjectType,Holding,{'tc': '4'}
Relation,RelatedObjectType,Party,{'tc': '6'}
Relation,RelationRoleCode,Primary Beneficiary,{'tc': '34'}
Relation,BeneficiaryDesignation,Named,{'tc': '1'}
import lxml.etree as etree
import pandas as pd
import json
# Read the csv file
dfc = pd.read_csv('test_data_txlife.csv') .fillna('NA')
# # Remove rows with comments
# dfc = dfc[~dfc['Element'].str.contains("<cyfunction")].fillna('')
dfc['Attribute'] = dfc['Attribute'].apply(lambda x: x.replace("'", '"'))
# Add the root element for xml
root = etree.Element(dfc['Element'][0])
tree = root.getroottree()
for prnt, elem, txt, attr in dfc[['Parent', 'Element', 'Text', 'Attribute']][1:].values:
# Convert attributes to json (dictionary)
attrib = json.loads(attr)
# list(root) = root.getchildren()
children = [item for item in str(list(root)).split(' ')]
rootstring = str(root).split(' ')[1]
# If the parent is root then add the element as child (appaers to work?)
if prnt == str(root).split(' ')[1]:
parent = etree.SubElement(root, elem)
# If the parent is not root but is one of its children then add the elements to the parent
elif not prnt == rootstring and prnt in children:
child = etree.SubElement(parent, elem, attrib).text = txt
# # If the parent is not in root's descendents then add the childern to the parents
elif not prnt in [str(item).split(' ') for item in root.iterdescendants()]:
child = etree.SubElement(parent, elem, attrib).text = txt
print(etree.tostring(tree, pretty_print=True).decode())
实际结果:
<TXLife>
<UserAuthRequest>
<UserLoginName>*****</UserLoginName>
<UserPswd>
</UserPswd>
<CryptType>None</CryptType>
<Pswd>xxxxxx</Pswd>
</UserAuthRequest>
<TXLifeRequest>
<TransRefGUID>706D67C1-CC4D-11CF-91FB444554540000</TransRefGUID>
<TransType tc="502">Holding Change</TransType>
<TransExeDate>11/19/2006</TransExeDate>
<TransExeTime>13:15:33-07:00</TransExeTime>
<ChangeSubType>
</ChangeSubType>
<ChangeTC tc="9">Change Participant</ChangeTC>
<OLifE>
</OLifE>
<Holding id="Policy_1">
</Holding>
<HoldingTypeCode tc="2">Policy</HoldingTypeCode>
<Policy>
</Policy>
<PolNumber>1234567</PolNumber>
<LineOfBusiness tc="2">Annuity</LineOfBusiness>
<Annuity>NA</Annuity>
<Party id="Beneficiary_1">
</Party>
<PartyTypeCode tc="2">Organization</PartyTypeCode>
<FullName>The Smith Trust</FullName>
<Organization>
</Organization>
<OrgForm tc="16">Trust</OrgForm>
<DBA>The Smith Trust</DBA>
<Relation OriginatingObjectID="Policy_1" RelatedObjectID="Beneficiary_1" id="Relation_1">
</Relation>
<OriginatingObjectType tc="4">Holding</OriginatingObjectType>
<RelatedObjectType tc="6">Party</RelatedObjectType>
<RelationRoleCode tc="34">Primary Beneficiary</RelationRoleCode>
<BeneficiaryDesignation tc="1">Named</BeneficiaryDesignation>
</TXLifeRequest>
</TXLife>
所需结果:
<TXLife Version="2.25.00">
<UserAuthRequest>
<UserLoginName>*****</UserLoginName>
<UserPswd>
<CryptType>None</CryptType>
<Pswd>****</Pswd>
</UserPswd>
</UserAuthRequest>
<TXLifeRequest PrimaryObjectID="Policy_1">
<TransRefGUID>706D67C1-CC4D-11CF-91FB444554540000</TransRefGUID>
<TransType tc="502">Holding Change</TransType>
<TransExeDate>2006-11-19</TransExeDate>
<TransExeTime>13:15:33-07:00</TransExeTime>
<ChangeSubType>
<ChangeTC tc="9">Change Participant</ChangeTC>
</ChangeSubType>
<OLifE>
<Holding id="Policy_1">
<HoldingTypeCode tc="2">Policy</HoldingTypeCode>
<Policy>
<PolNumber>1234567</PolNumber>
<LineOfBusiness tc="2">Annuity</LineOfBusiness>
<Annuity></Annuity>
</Policy>
</Holding>
<Party id="Beneficiary_1">
<PartyTypeCode tc="2">Organization</PartyTypeCode>
<FullName>The Smith Trust</FullName>
<Organization>
<OrgForm tc="16">Trust</OrgForm>
<DBA>The Smith Trust</DBA>
</Organization>
</Party>
<Relation id="Relation_1" OriginatingObjectID="Policy_1" RelatedObjectID="Beneficiary_1">
<OriginatingObjectType tc="4">Holding</OriginatingObjectType>
<RelatedObjectType tc="6">Party</RelatedObjectType>
<RelationRoleCode tc="34">Primary Beneficiary</RelationRoleCode>
<BeneficiaryDesignation tc="1">Named</BeneficiaryDesignation>
</Relation>
</OLifE>
</TXLifeRequest>
</TXLife>
如上所示,如何获得分层结果?
最佳答案
您已经有了一个不错的开始!认为逐位检查代码并解释需要调整的地方,并提出一些改进建议是最容易的:
读取和清理数据
# Read the csv file
dfc = pd.read_csv('test_data_txlife.csv').fillna('NA')
# # Remove rows with comments
# dfc = dfc[~dfc['Element'].str.contains("<cyfunction")].fillna('')
dfc['Attribute'] = dfc['Attribute'].apply(lambda x: x.replace("'", '"'))
.apply
可以正常工作,但是还有一个.str.replace()
方法可以使用,它会更加整洁和清晰(.str
可让您将列的值视为字符串类型并相应地对其进行操作)。添加根
# Add the root element for xml
root = etree.Element(dfc['Element'][0])
tree = root.getroottree()
一切都很好!
遍历行
for prnt, elem, txt, attr in dfc[['Parent', 'Element', 'Text', 'Attribute']][1:].values:
由于无论如何都检索所有列,因此无需索引到
dfc
即可选择它们,因此可以删除该部分:for prnt, elem, txt, attr in dfc[1:].values:
这很好用,但是有内置的方法可以遍历DataFrame中的项目,我们可以使用
itertuples()
。这会为每行返回一个NamedTuple
,其中将索引(基本上是行号)作为元组的第一项,因此我们需要对此进行调整:for idx, prnt, elem, txt, attr in dfc[1:].itertuples():
设置变量
# Convert attributes to json (dictionary)
attrib = json.loads(attr)
# list(root) = root.getchildren()
children = [item for item in str(list(root)).split(' ')]
rootstring = str(root).split(' ')[1][1:].values:
这是一个很好的技巧,可以早些时候用双引号替换单引号,以便我们可以使用
json
将属性转换成字典。每个
Element
都有一个.tag
属性,我们可以使用该属性来获取名称,这就是我们想要的名称:children = [item.tag for item in root]
rootstring = root.tag
list(root)
或root.getchildren()
都可以为我们提供root
子元素的列表,但是我们也可以像这样使用for ... in
和root
遍历它们。将元素添加到树中
# If the parent is root then add the element as child (appaers to work?)
if prnt == str(root).split(' ')[1]:
parent = etree.SubElement(root, elem)
# If the parent is not root but is one of its children then add the elements to the parent
elif not prnt == rootstring and prnt in children:
child = etree.SubElement(parent, elem, attrib).text = txt
# # If the parent is not in root's descendents then add the childern to the parents
elif not prnt in [str(item).split(' ') for item in root.iterdescendants()]:
child = etree.SubElement(parent, elem, attrib).text = txt
str(root).split(' ')[1]
正是我们将rootstring
设置为上方的内容,因此我们可以改用它由于我们已经在第一个
prnt == rootstring
语句中检查了if
,因此,如果我们到达了第一个elif
,我们知道它不可能相等,因此我们无需再次检查它当我们创建孩子时,我们一次有两个任务……以某种方式成功地创建了带有文本的孩子(!),但这意味着
child
被设置为text
而不是新的SubElement
。最好分两个步骤执行此操作。当寻找父项时,当前正在创建一个列表列表(
split()
返回一个列表),因此它将无法正常工作。我们需要item标签。进行所有这些更改将使我们:
# If the parent is root then add the element as child (appaers to work?)
if prnt == rootstring:
parent = etree.SubElement(root, elem)
# If the parent is not root but is one of its children then add the elements to the parent
elif prnt in children:
child = etree.SubElement(parent, elem, attrib)
child.text = txt
# # If the parent is not in root's descendents then add the childern to the parents
elif not prnt in [item.tag for item in root.iterdescendants()]:
child = etree.SubElement(parent, elem, attrib)
child.text = txt
但是这里有两个问题。
第一部分(
if
语句)可以。在第二部分(第一个
elif
语句)中,我们检查新元素的父级是否是root的子级之一。如果是,我们将新元素添加为parent
的子元素。 parent
绝对是root的孩子之一,但实际上我们尚未检查它是否正确。这只是我们添加到root
的最后一件事。幸运的是,由于我们的CSV具有按顺序排列的所有元素,因此这是正确的元素,但是最好对此做得更明确。在第三部分(第二个
elif
)中,最好检查树后面的prnt
是否已经存在。但是目前,如果prnt
不存在,我们将在jusr中将元素添加到parent
中,这不是它的实际父级!如果确实存在prnt
,那么我们根本就不会添加该元素(因此在这里我们需要一个else
子句)。解
值得庆幸的是,有一个简单的解决方法:我们可以使用
.find()
在树中的任意位置找到prnt
元素,然后在其中添加新元素。这也使整个过程变短了!for idx, prnt, elem, txt, attr in dfc[1:].itertuples():
# Convert attributes to json (dictionary)
attrib = json.loads(attr)
# Find parent element
if prnt == root.tag:
parent = root
else:
parent = root.find(".//" + prnt)
child = etree.SubElement(parent, elem, attrib)
child.text = txt
.//
中的root.find(".//" + prnt)
表示它将在树中的任何位置搜索匹配的元素标签(在此处了解更多信息:https://lxml.de/tutorial.html#elementpath)。最终脚本
import lxml.etree as etree
import pandas as pd
import json
# Read the csv file
dfc = pd.read_csv('test_data_txlife.csv').fillna("NA")
dfc['Attribute'] = dfc['Attribute'].str.replace("'", '"').apply(lambda s: json.loads(s))
# Add the root element for xml
root = etree.Element(dfc['Element'][0], dfc['Attribute'][0])
for idx, prnt, elem, txt, attr in dfc[1:].itertuples():
# Fix text
text = txt.strip()
if not text:
text = None
# Find parent element
if prnt == root.tag:
parent = root
else:
parent = root.find(".//" + prnt)
# Create element
child = etree.SubElement(parent, elem, attr)
child.text = text
xml_string = etree.tostring(root, pretty_print=True).decode().replace(">NA<", "><")
print(xml_string)
我做了几处更改:
在更改引号时,我将属性字典的
json.loads
位上移了,最后使用apply
将其添加到了末尾。我们在那里需要它,以便在创建根元素时可以使用字典。要使漂亮的打印正常工作会有一些问题,这就是“修复文本”部分的目的(有关我遇到的问题,请参见this Stack Overflow question)。
拥有
.fillna("")
(用空字符串填充)是最明智的选择,但是如果这样做,我们最终会以</Annuity>
而不是<Annuity></Annuity>
(这是合法的XML-如果您的元素没有文本或子元素,您可以只做结束标记)。但是要使它如我们所愿地发布,我们需要它具有一些“内容”,以便创建开始标签。因此,我将其保留为.fillna("NA")
,然后将其保留在末尾,手动将其替换为输出字符串。还很高兴知道此脚本(至少)对输入数据做出了四个假设:
父元素是在其任何子元素之前创建的(即,它们在CSV文件中更远的位置出现)
元素名称是唯一的(或者至少任何重复的名称没有任何子代,因此我们永远不会在可能存在多个匹配项的情况下执行
.find()
; .find()
始终返回第一个匹配项)在最终的XML中没有要保留的'NA'文本值(当我们从
Annuity
元素中删除虚假的'NA'文本时,它们也会被删除)仅包含空格的文本无需保留
关于python - 数据框到分层xml,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/57682123/