我正在从 s3 存储桶读取数据,并使用 aws lambda 将该数据插入到 aws elasticsearch 中。
如果我使用request.post,那么它就可以工作。但是我需要管道作为参数。
from elasticsearch import Elasticsearch, RequestsHttpConnection
import boto3
import re
import requests
from requests_aws4auth import AWS4Auth
region = 'us-east-2' # e.g. us-west-1
service = 'es'
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key,region, service, session_token=credentials.token)
host = 'https://search-internship6-aqt7s3tuokjcvx7um3lmm7wvbe.us- east-2.es.amazonaws.com/' # the Amazon ES domain, including https://
index = 'lambda-s3-index11'
type1 = 'lambda-type10'
url1 = host + '/' + index + '/' + type1
headers = {"Content-Type": "application/json"}
s3 = boto3.client('s3')
#r=requests.get(url=url1,auth=awsauth)
#print("request ",r)
#es = Elasticsearch([ 'https://search-internship6-aqt7s3tuokjcvx7um3lmm7wvbe.us-east-2.es.amazonaws.com'])
print("Hello1")
es = Elasticsearch(
'https://search-internship6-aqt7s3tuokjcvx7um3lmm7wvbe.us-east-2.es.amazonaws.com',
http_auth = awsauth,
use_ssl = True,
verify_certs = True,
connection_class = RequestsHttpConnection
)
# Lambda execution starts here
def handler(event, context):
#r=requests.get(url= host ,auth=awsauth,headers=headers)
#print("request :",r)
print("es :",es.info())
for record in event['Records']:
# Get the bucket name and key for the new file
bucket = record['s3']['bucket']['name']
key = record['s3']['object']['key']
# Get, read, and split the file into lines
obj = s3.get_object(Bucket=bucket, Key=key)
body = obj['Body'].read()
lines = body.splitlines()
# Match the regular expressions to each line and index the JSON
for line in lines:
# ip = ip_pattern.search(line).group(1)
# timestamp = time_pattern.search(line).group(1)
# message = message_pattern.search(line).group(1)
print(line)
# document = { "ip": ip, "timestamp": timestamp, "message": message }
#r = requests.post(url1, data=line, auth=awsauth, headers=headers)
es.index(index='internship11', doc_type='packets11', body=line, pipeline='epoch-to-format')
print("hello2")
#r = requests.post(url1, data=line, auth=awsauth, headers=headers)
#print(es.info())
我收到此错误。
ConnectionError(HTTPSConnectionPool(host='https', port=443): Max retries exceeded with url: //search-internship6-aqt7s3tuokjcvx7um3lmm7wvbe.us-east-2.es.amazonaws.com/:443/ (Caused by NewConnectionError (': Failed to establish a new connection: [Errno -2] Name or service not known',))) caused by: ConnectionError(HTTPSConnectionPool (host='https', port=443): Max retries exceeded with url: //search-internship6-aqt7s3tuokjcvx7um3lmm7wvbe.us-east-2.es.amazonaws.com/:443/ (Caused by NewConnectionError (': Failed to establish a new connection: [Errno -2] Name or service not known',))): ConnectionError
最佳答案
尝试从主机名中删除“https://”
并使用以下代码
host = 'search-internship6-aqt7s3tuokjcvx7um3lmm7wvbe.us-east-2.es.amazonaws.com'
port = 80
auth = AWSRequestsAuth(
aws_access_key=credentials.access_key,
aws_secret_access_key=credentials.secret_key,
aws_region=region,
aws_host=host,
aws_service="es"
)
# Use the requests connection_class and pass in our custom auth class
es = Elasticsearch(
host=host,
port=port,
connection_class=RequestsHttpConnection,
http_auth=auth
)
关于python - 为什么会出现超出最大尝试次数的错误?,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/56888358/