python - 如何使用具有多个参数的函数运行多处理 python 请求

Question

我正在尝试使用 python requests 库从单个 url 收集数据。

我想运行多重处理来加速数据收集，但是当我在 Pool 内传递函数的参数时出现错误。

请注意，我已经阅读了以下之前的问题:

a link 和 a link 然而这些都没有回答我的问题。

如何同时运行这些 get 请求并传递 3 个强制参数？

这是我的代码:

from multiprocessing import Pool
import requests
url = 'http://icanhazip.com'
url_two = 'https://httpbin.org/ip'
url_three = 'https://httpbin.org/get'
start_point = 'a'
start_point_two = 'b'
start_point_three ='c'
ending_point = 'c'
ending_point_two = 'z'
ending_point_three = 'x'


def get_info(url,start_point,ending_point):
    r = requests.get(url)
    html = r.text
    if start_point in html:
        print('Do Something')
    elif ending_point in html:
        print('Do Something else')
   else:
        pass

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.map(get_info, [[url,start_point,ending_point]]))

这是我得到的错误:

TypeError: get_info() missing 2 required positional arguments: 'start_point' and 'ending_point'

Answer 1

要将多个参数传递给目标函数 - 使用 Pool.starmap特点:

在您的情况下，它看起来如下:

if __name__ == '__main__':
    with Pool(5) as p:
        print(p.starmap(get_info, [(url, start_point, ending_point),
                               (url_two, start_point_two, ending_point_two),
                               (url_three, start_point_three, ending_point_three),]