python - Linkedin 抓取工具提取技能

Question

我正在尝试抓取人们的公开资料，以获取某些角色的最常见技能。我能够提取电子邮件、公司、姓名、职位等，但我无法获得技能。 我正在使用 parsel 中的选择器。我尝试了很多方法，但显然我的目标是错误的类别，我可能应该循环遍历技能。这是到目前为止我的代码:

def linkedin_scrape(linkedin_urls):

profiles = []

for url in linkedin_urls:

    _DRIVER_CHROME.get(url)
    sleep(5)

    selector = Selector(text=_DRIVER_CHROME.page_source)

    # Use xpath to extract the exact class containing the profile name
    name = selector.xpath('//*[starts-with(@class, "inline")]/text()').extract_first()
    if name:
        name = name.strip()

    # Use xpath to extract the exact class containing the profile position
    position = selector.xpath('//*[starts-with(@class, "mt1")]/text()').extract_first()

    if position:
        position = position.strip()
        position = position[0:position.find(' at ')]

    # Use xpath to extract the exact class containing the profile company
    company = selector.xpath('//*[starts-with(@class, "text-align-left")]/text()').extract_first()

    if company:
        company = company.strip()

    # Use xpath to extract skills

    skills = selector.xpath('//*[starts-with(@class, "pv-skill")]/text()').extract_first()

    if skills:
        skills = skills.strip()


    profiles.append([name, position, company, url])
    print(f'{len(profiles)}: {name}, {position}, {company}, {url}, {skills}')

return profiles

Answer 1

为了捕获所有技能，您需要首先展开技能部分，以便它显示所有技能，然后定位名称以“pv-skill-category-entity__name-text”开头的类别。

直到今天这对我都有效。

#locate link to expand skills
show_more_skills_button = driver.find_element_by_class_name("pv-skills-section__chevron-icon")
#expand
show_more_skills_button.click()

skills = driver.find_elements_by_xpath("//*[starts-with(@class,'pv-skill-category-entity__name-text')]")

#create skills set
skill_set = []
for skill in skills:
    skill_set.append(skill.text)