我想调用 HDFS REST api 以使用 httplib
上传文件。
我的程序创建了文件,但其中没有内容。
============================================= ======
这是我的代码:
import httplib
conn=httplib.HTTPConnection("localhost:50070")
conn.request("PUT","/webhdfs/v1/levi/4?op=CREATE")
res=conn.getresponse()
print res.status,res.reason
conn.close()
conn=httplib.HTTPConnection("localhost:50075")
conn.connect()
conn.putrequest("PUT","/webhdfs/v1/levi/4?op=CREATE&user.name=levi")
conn.endheaders()
a_file=open("/home/levi/4","rb")
a_file.seek(0)
data=a_file.read()
conn.send(data)
res=conn.getresponse()
print res.status,res.reason
conn.close()
============================================= ===
这是返回:
307 TEMPORARY_REDIRECT 201 Created
============================================= ==========
OK,文件已创建,但没有内容发送。
当我评论#conn.send(data)
时,结果是一样的,仍然没有内容。
可能读取的文件或发送的文件有误,不确定。
你知道这是怎么发生的吗?
最佳答案
看起来您的代码没有在第二个 PUT 请求中使用 307 中的“位置” header 。
我一直在研究可能有用的 python WebHDFS 包装器的分支,您可以在此处查看完整代码:https://github.com/carlosmarin/webhdfs-py/blob/master/webhdfs/webhdfs.py
您感兴趣的方法是:
def copyfromlocal(self, source_path, target_path, replication=1, overwrite=True):
url_path = WEBHDFS_CONTEXT_ROOT + target_path + '?op=CREATE&overwrite=' + 'true' if overwrite else 'false'
with _NameNodeHTTPClient('PUT', url_path, self.namenode_host, self.namenode_port, self.username) as response:
logger.debug("HTTP Response: %d, %s" % (response.status, response.reason))
redirect_location = response.msg["location"]
logger.debug("HTTP Location: %s" % redirect_location)
(redirect_host, redirect_port, redirect_path, query) = self.parse_url(redirect_location)
# Bug in WebHDFS 0.20.205 => requires param otherwise a NullPointerException is thrown
redirect_path = redirect_path + "?" + query + "&replication=" + str(replication)
logger.debug("Redirect: host: %s, port: %s, path: %s " % (redirect_host, redirect_port, redirect_path))
fileUploadClient = HTTPConnection(redirect_host, redirect_port, timeout=600)
# This requires currently Python 2.6 or higher
fileUploadClient.request('PUT', redirect_path, open(source_path, "r").read(), headers={})
response = fileUploadClient.getresponse()
logger.debug("HTTP Response: %d, %s" % (response.status, response.reason))
fileUploadClient.close()
return json.loads(response.read())
关于python - 我想调用 HDFS REST api 来上传文件,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/15870381/