我正在使用 Scrapy 抓取网站,我需要抓取的其中一个链接似乎是由页面中的一小段 Javascript 代码生成的,如下所示:
<!--
var prefix = 'mailto:';
var suffix = '';
var attribs = '';
var path = 'hr' + 'ef' + '=';
var addy59933 = 'HR-Cologne' + '@';
addy59933 = addy59933 + 'scor' + '.' + 'com';
var addy_text59933 = 'Submit your application';
document.write( '<a ' + path + '\'' + prefix + addy59933 + suffix + '\'' + attribs + '>' );
document.write( addy_text59933 );
document.write( '<\/a>' );
//-->
除非您从浏览器查看该页面,否则该链接不会显示,但无论如何我都需要我的蜘蛛能够抓取它。由于代码嵌入在页面中,我想从那里获取然后重新组合链接 url,但文本的格式我不熟悉。
有更好的方法吗?
编辑:刚刚发现那些是 HTML 字符实体。我仍然想知道是否有更好的方法来克服这种混淆。
最佳答案
这是一个使用 js2xml 的解决方案:
>>> import js2xml
>>> import pprint
>>> jscode = r"""
... var prefix = 'mailto:';
... var suffix = '';
... var attribs = '';
... var path = 'hr' + 'ef' + '=';
... var addy59933 = 'HR-Cologne' + '@';
... addy59933 = addy59933 + 'scor' + '.' + 'com';
... var addy_text59933 = 'Submit your application';
... document.write( '<a ' + path + '\'' + prefix + addy59933 + suffix + '\'' + attribs + '>' );
... document.write( addy_text59933 );
... document.write( '<\/a>' );
>>> js = js2xml.parse(jscode)
变量声明由var_decl
元素表示,它们的名称在identifier
节点中,它们的值在这里是字符串,带有+
运算符,所以让我们在 string/text()
元素上使用 "".join()
从它们中创建一个 dict
:
>>> # variables
... variables = dict([(var.xpath('string(./identifier)'), u"".join(var.xpath('.//string/text()')))
... for var in js.xpath('.//var_decl')])
>>> pprint.pprint(variables)
{'addy59933': u'HR-Cologne@',
'addy_text59933': u'Submit your application',
'attribs': u'',
'path': u'href=',
'prefix': u'mailto:',
'suffix': u''}
然后赋值改变一些变量的值,混合使用字符串和变量。连接变量标识符的 %(identifidername)s
和字符串的字符串值
>>> # identifiers are assigned other string values
... assigns = {}
>>> for assign in js.xpath('.//assign'):
... value = u"".join(['%%(%s)s' % el.text if el.tag=='identifier' else el.text
... for el in assign.xpath('./right//*[self::string or self::identifier]')])
... key = assign.xpath('string(left/identifier)')
... assigns[key] = value
...
>>> pprint.pprint(assigns)
{'addy59933': u'%(addy59933)sscor.com'}
“应用”赋值更新变量字典
>>> # update variables dict with new values
... for key, val in assigns.items():
... variables[key] = val % variables
...
>>> pprint.pprint(variables)
{'addy59933': u'HR-Cologne@scor.com',
'addy_text59933': u'Submit your application',
'attribs': u'',
'path': u'href=',
'prefix': u'mailto:',
'suffix': u''}
>>>
函数参数在arguments
节点下(XPath .//arguments/*
):
>>> # interpret arguments of document.write()
... arguments = [u"".join(['%%(%s)s' % el.text if el.tag=='identifier' else el.text
... for el in arg.xpath('./descendant-or-self::*[self::string or self::identifier]')])
... for arg in js.xpath('.//arguments/*')]
>>>
>>> pprint.pprint(arguments)
[u"<a %(path)s'%(prefix)s%(addy59933)s%(suffix)s'%(attribs)s>",
u'%(addy_text59933)s',
u'</a>']
>>>
如果你替换那里的标识符,你会得到
>>> # apply string formatting replacing identifiers
... arguments = [arg % variables for arg in arguments]
>>>
>>> pprint.pprint(arguments)
[u"<a href='mailto:HR-Cologne@scor.com'>",
u'Submit your application',
u'</a>']
>>>
现在看起来很有趣,让我们通过 lxml.html
运行它以摆脱数字字符引用:
>>> import lxml.html
>>> import lxml.etree
>>>
>>> doc = lxml.html.fromstring("".join(arguments))
>>> print lxml.etree.tostring(doc)
<a href="mailto:HR-Cologne@scor.com">Submit your application</a>
>>>
使用 Scrapy 选择器
:
>>> from scrapy.selector import Selector
>>> selector = Selector(text="".join(arguments), type="html")
>>> selector.xpath('.//a/@href').extract()
[u'mailto:HR-Cologne@scor.com']
>>>
关于python-2.7 - 抓取由javascript生成的链接,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/23771983/