我通过brew install hadoop安装hadoop然后使用 pip install pyarrow作为客户端
client = pa.hdfs.connect('localhost', 9000, driver='libhdfs')
for x in client.ls('/'):
print(x)
出现错误
Traceback (most recent call last):
File "/Users/wyx/project/py3.7aio/hdfs/list_dir.py", line 13, in <module>
client = pa.hdfs.connect('localhost', 9000, )
File "/Users/wyx/project/py3.7aio/.env/lib/python3.6/site-packages/pyarrow/hdfs.py", line 207, in connect
extra_conf=extra_conf)
File "/Users/wyx/project/py3.7aio/.env/lib/python3.6/site-packages/pyarrow/hdfs.py", line 38, in __init__
self._connect(host, port, user, kerb_ticket, driver, extra_conf)
File "pyarrow/io-hdfs.pxi", line 89, in pyarrow.lib.HadoopFileSystem._connect
File "pyarrow/error.pxi", line 83, in pyarrow.lib.check_status
pyarrow.lib.ArrowIOError: Unable to load libhdfs
如何获取libhdfs或libhdfs3？
我认为这可能与本地库有关，因为
➜ hadoop-rel-release-3.1.1 hadoop checknative -a
2019-02-21 15:46:20,805 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Native library checking:
hadoop: false
zlib: false
zstd : false
snappy: false
lz4: false
bzip2: false
openssl: false
ISA-L: false
2019-02-21 15:46:20,934 INFO util.ExitUtil: Exiting with status 1: ExitException
之后我brew install gcc autoconf automake libtool cmake snappy gzip bzip2 <a href="https://stackoverflow.com/cdn-cgi/l/email-protection" class="__cf_email__" data-cfemail="5f2f2d302b303d2a391f6d716a" rel="noreferrer noopener nofollow">[email protected]</a> zlib openssl
下载hadoop-rel-release-3.1.1构建。
[INFO] Compiling 3 source files to /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/classes
[INFO]
[INFO] --- hadoop-maven-plugins:3.1.1:cmake-compile (cmake-compile) @ hadoop-yarn-server-nodemanager ---
[DEBUG] Configuring mojo org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile from plugin realm ClassRealm[plugin>org.apache.hadoop:hadoop-maven-plugins:3.1.1, parent: sun.misc.Launcher$AppClassLoader@55f96302]
[DEBUG] Configuring mojo 'org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile' with basic configurator -->
[DEBUG] (f) env = {CFLAGS=null}
[DEBUG] (f) output = /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native
[DEBUG] (f) source = /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src
[DEBUG] (f) vars = {HADOOP_CONF_DIR=../etc/hadoop, JVM_ARCH_DATA_MODEL=64}
[DEBUG] -- end configuration --
[INFO] Running cmake /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src -DHADOOP_CONF_DIR=../etc/hadoop -DJVM_ARCH_DATA_MODEL=64 -G Unix Makefiles
[INFO] with extra environment variables {
CFLAGS = ''
}
[INFO] Running make -j 8 VERBOSE=1
[WARNING] /usr/local/Cellar/cmake/3.13.4/bin/cmake -S/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src -B/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native --check-build-system CMakeFiles/Makefile.cmake 0
[WARNING] /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_progress_start /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/progress.marks
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/Makefile2 all
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/container.dir/build.make CMakeFiles/container.dir/depend
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/gtest.dir/build.make CMakeFiles/gtest.dir/depend
[WARNING] cd /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native && /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/gtest.dir/DependInfo.cmake --color=
[WARNING] cd /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native && /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/container.dir/DependInfo.cmake --color=
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/gtest.dir/build.make CMakeFiles/gtest.dir/build
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/container.dir/build.make CMakeFiles/container.dir/build
[WARNING] make[2]: Nothing to be done for `CMakeFiles/gtest.dir/build'.
[WARNING] make[2]: Nothing to be done for `CMakeFiles/container.dir/build'.
[WARNING] [ 6%] Built target gtest
[WARNING] [ 46%] Built target container
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/cetest.dir/build.make CMakeFiles/cetest.dir/depend
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/test-container-executor.dir/build.make CMakeFiles/test-container-executor.dir/depend
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/container-executor.dir/build.make CMakeFiles/container-executor.dir/depend
[WARNING] cd /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native && /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/test-container-executor.dir/DependInfo.cmake --color=
[WARNING] cd /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native && /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/container-executor.dir/DependInfo.cmake --color=
[WARNING] cd /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native && /usr/local/Cellar/cmake/3.13.4/bin/cmake -E cmake_depends "Unix Makefiles" /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/CMakeFiles/cetest.dir/DependInfo.cmake --color=
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/container-executor.dir/build.make CMakeFiles/container-executor.dir/build
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/test-container-executor.dir/build.make CMakeFiles/test-container-executor.dir/build
[WARNING] /Applications/Xcode.app/Contents/Developer/usr/bin/make -f CMakeFiles/cetest.dir/build.make CMakeFiles/cetest.dir/build
[WARNING] make[2]: Nothing to be done for `CMakeFiles/container-executor.dir/build'.
[WARNING] make[2]: Nothing to be done for `CMakeFiles/test-container-executor.dir/build'.
[WARNING] [ 60%] Built target container-executor
[WARNING] [ 60%] Built target test-container-executor
[WARNING] [ 63%] Building CXX object CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o
[WARNING] /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -I/System/Library/Frameworks -I/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src -I/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native -I/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor -I/Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl -isystem /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/../../../../../hadoop-common-project/hadoop-common/src/main/native/gtest/include -g -O2 -Wall -pthread -D_FILE_OFFSET_BITS=64 -isysroot /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk -o CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o -c /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc
[WARNING] In file included from /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:56:13: warning: comparison of array 'args->data' equal to a null pointer is always false [-Wtautological-pointer-compare]
[WARNING] if (args->data == NULL || args->length >= DOCKER_ARG_MAX) {
[WARNING] ~~~~~~^~~~ ~~~~
[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:63:13: warning: comparison of array 'args->data' not equal to a null pointer is always true [-Wtautological-pointer-compare]
[WARNING] if (args->data != NULL) {
[WARNING] ~~~~~~^~~~ ~~~~
[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1227:12: error: no matching function for call to 'getgrouplist'
[WARNING] int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
[WARNING] ^~~~~~~~~~~~
[WARNING] /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/unistd.h:653:6: note: candidate function not viable: no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd argument
[WARNING] int getgrouplist(const char *, int, int *, int *);
[WARNING] ^
[WARNING] In file included from /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1234:9: error: no matching function for call to 'getgrouplist'
[WARNING] if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
[WARNING] ^~~~~~~~~~~~
[WARNING] /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/unistd.h:653:6: note: candidate function not viable: no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd argument
[WARNING] int getgrouplist(const char *, int, int *, int *);
[WARNING] ^
[WARNING] 2 warnings and 2 errors generated.
[WARNING] make[2]: *** [CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o] Error 1
[WARNING] make[1]: *** [CMakeFiles/cetest.dir/all] Error 2
[WARNING] make: *** [all] Error 2
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary for Apache Hadoop Main 3.1.1:
[INFO]
[INFO] Apache Hadoop Main ................................. SUCCESS [ 1.115 s]
[INFO] Apache Hadoop Build Tools .......................... SUCCESS [ 1.784 s]
[INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.991 s]
[INFO] Apache Hadoop Annotations .......................... SUCCESS [ 2.055 s]
[INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.579 s]
[INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.898 s]
[INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 3.680 s]
[INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 1.903 s]
[INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.898 s]
[INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 2.586 s]
[INFO] Apache Hadoop Common ............................... SUCCESS [ 47.036 s]
[INFO] Apache Hadoop NFS .................................. SUCCESS [ 4.246 s]
[INFO] Apache Hadoop KMS .................................. SUCCESS [ 5.686 s]
[INFO] Apache Hadoop Common Project ....................... SUCCESS [ 0.385 s]
[INFO] Apache Hadoop HDFS Client .......................... SUCCESS [ 16.175 s]
[INFO] Apache Hadoop HDFS ................................. SUCCESS [ 36.668 s]
[INFO] Apache Hadoop HDFS Native Client ................... SUCCESS [ 3.188 s]
[INFO] Apache Hadoop HttpFS ............................... SUCCESS [ 6.474 s]
[INFO] Apache Hadoop HDFS-NFS ............................. SUCCESS [ 2.781 s]
[INFO] Apache Hadoop HDFS-RBF ............................. SUCCESS [ 13.786 s]
[INFO] Apache Hadoop HDFS Project ......................... SUCCESS [ 0.317 s]
[INFO] Apache Hadoop YARN ................................. SUCCESS [ 0.291 s]
[INFO] Apache Hadoop YARN API ............................. SUCCESS [ 8.560 s]
[INFO] Apache Hadoop YARN Common .......................... SUCCESS [ 29.751 s]
[INFO] Apache Hadoop YARN Registry ........................ SUCCESS [ 4.885 s]
[INFO] Apache Hadoop YARN Server .......................... SUCCESS [ 0.354 s]
[INFO] Apache Hadoop YARN Server Common ................... SUCCESS [ 8.905 s]
[INFO] Apache Hadoop YARN NodeManager ..................... FAILURE [ 2.680 s]
[INFO] Apache Hadoop YARN Web Proxy ....................... SKIPPED
[INFO] Apache Hadoop YARN ApplicationHistoryService ....... SKIPPED
[INFO] Apache Hadoop YARN Timeline Service ................ SKIPPED
[INFO] Apache Hadoop YARN ResourceManager ................. SKIPPED
[INFO] Apache Hadoop YARN Server Tests .................... SKIPPED
[INFO] Apache Hadoop YARN Client .......................... SKIPPED
[INFO] Apache Hadoop YARN SharedCacheManager .............. SKIPPED
[INFO] Apache Hadoop YARN Timeline Plugin Storage ......... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Backend ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Common .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Client .... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Servers ... SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase Server 1.2 SKIPPED
[INFO] Apache Hadoop YARN TimelineService HBase tests ..... SKIPPED
[INFO] Apache Hadoop YARN Router .......................... SKIPPED
[INFO] Apache Hadoop YARN Applications .................... SKIPPED
[INFO] Apache Hadoop YARN DistributedShell ................ SKIPPED
[INFO] Apache Hadoop YARN Unmanaged Am Launcher ........... SKIPPED
[INFO] Apache Hadoop MapReduce Client ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Core ....................... SKIPPED
[INFO] Apache Hadoop MapReduce Common ..................... SKIPPED
[INFO] Apache Hadoop MapReduce Shuffle .................... SKIPPED
[INFO] Apache Hadoop MapReduce App ........................ SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer .............. SKIPPED
[INFO] Apache Hadoop MapReduce JobClient .................. SKIPPED
[INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
[INFO] Apache Hadoop YARN Services ........................ SKIPPED
[INFO] Apache Hadoop YARN Services Core ................... SKIPPED
[INFO] Apache Hadoop YARN Services API .................... SKIPPED
[INFO] Apache Hadoop YARN Site ............................ SKIPPED
[INFO] Apache Hadoop YARN UI .............................. SKIPPED
[INFO] Apache Hadoop YARN Project ......................... SKIPPED
[INFO] Apache Hadoop MapReduce HistoryServer Plugins ...... SKIPPED
[INFO] Apache Hadoop MapReduce NativeTask ................. SKIPPED
[INFO] Apache Hadoop MapReduce Uploader ................... SKIPPED
[INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
[INFO] Apache Hadoop MapReduce ............................ SKIPPED
[INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
[INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
[INFO] Apache Hadoop Archives ............................. SKIPPED
[INFO] Apache Hadoop Archive Logs ......................... SKIPPED
[INFO] Apache Hadoop Rumen ................................ SKIPPED
[INFO] Apache Hadoop Gridmix .............................. SKIPPED
[INFO] Apache Hadoop Data Join ............................ SKIPPED
[INFO] Apache Hadoop Extras ............................... SKIPPED
[INFO] Apache Hadoop Pipes ................................ SKIPPED
[INFO] Apache Hadoop OpenStack support .................... SKIPPED
[INFO] Apache Hadoop Amazon Web Services support .......... SKIPPED
[INFO] Apache Hadoop Kafka Library support ................ SKIPPED
[INFO] Apache Hadoop Azure support ........................ SKIPPED
[INFO] Apache Hadoop Aliyun OSS support ................... SKIPPED
[INFO] Apache Hadoop Client Aggregator .................... SKIPPED
[INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
[INFO] Apache Hadoop Resource Estimator Service ........... SKIPPED
[INFO] Apache Hadoop Azure Data Lake support .............. SKIPPED
[INFO] Apache Hadoop Image Generation Tool ................ SKIPPED
[INFO] Apache Hadoop Tools Dist ........................... SKIPPED
[INFO] Apache Hadoop Tools ................................ SKIPPED
[INFO] Apache Hadoop Client API ........................... SKIPPED
[INFO] Apache Hadoop Client Runtime ....................... SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants .......... SKIPPED
[INFO] Apache Hadoop Client Test Minicluster .............. SKIPPED
[INFO] Apache Hadoop Client Packaging Invariants for Test . SKIPPED
[INFO] Apache Hadoop Client Packaging Integration Tests ... SKIPPED
[INFO] Apache Hadoop Distribution ......................... SKIPPED
[INFO] Apache Hadoop Client Modules ....................... SKIPPED
[INFO] Apache Hadoop Cloud Storage ........................ SKIPPED
[INFO] Apache Hadoop Cloud Storage Project ................ SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 03:35 min
[INFO] Finished at: 2019-02-21T15:31:33+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile (cmake-compile) on project hadoop-yarn-server-nodemanager: make failed with error code 2 -> [Help 1]
org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute goal org.apache.hadoop:hadoop-maven-plugins:3.1.1:cmake-compile (cmake-compile) on project hadoop-yarn-server-nodemanager: make failed with error code 2
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:215)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
Caused by: org.apache.maven.plugin.MojoExecutionException: make failed with error code 2
at org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.runMake (CompileMojo.java:229)
at org.apache.hadoop.maven.plugin.cmakebuilder.CompileMojo.execute (CompileMojo.java:98)
at org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo (DefaultBuildPluginManager.java:137)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:210)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:156)
at org.apache.maven.lifecycle.internal.MojoExecutor.execute (MojoExecutor.java:148)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:117)
at org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject (LifecycleModuleBuilder.java:81)
at org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build (SingleThreadedBuilder.java:56)
at org.apache.maven.lifecycle.internal.LifecycleStarter.execute (LifecycleStarter.java:128)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:305)
at org.apache.maven.DefaultMaven.doExecute (DefaultMaven.java:192)
at org.apache.maven.DefaultMaven.execute (DefaultMaven.java:105)
at org.apache.maven.cli.MavenCli.execute (MavenCli.java:956)
at org.apache.maven.cli.MavenCli.doMain (MavenCli.java:288)
at org.apache.maven.cli.MavenCli.main (MavenCli.java:192)
at sun.reflect.NativeMethodAccessorImpl.invoke0 (Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke (NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke (DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke (Method.java:498)
at org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced (Launcher.java:289)
at org.codehaus.plexus.classworlds.launcher.Launcher.launch (Launcher.java:229)
at org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode (Launcher.java:415)
at org.codehaus.plexus.classworlds.launcher.Launcher.main (Launcher.java:356)
[ERROR]
[ERROR]
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR]
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR] mvn <goals> -rf :hadoop-yarn-server-nodemanager
最佳答案
同样的问题，如果你仔细阅读编译日志，你会发现
[WARNING] /Users/wyx/Downloads/hadoop-rel-release-3.1.1/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:1227:12: error: no matching function for call to 'getgrouplist'
[WARNING] int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
[WARNING] ^~~~~~~~~~~~
[WARNING] /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.14.sdk/usr/include/unistd.h:653:6: note: candidate function not viable: no known conversion from 'gid_t *' (aka 'unsigned int *') to 'int *' for 3rd argument
[WARNING] int getgrouplist(const char *, int, int *, int *);
谷歌搜索并发现一个问题https://issues.apache.org/jira/browse/YARN-8622这里
以下更改对我来说效果很好:
source file: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c
line 1227:
- int rc = getgrouplist(user, pw->pw_gid, groups, &ngroups);
+ int rc = getgrouplist(user, pw->pw_gid, (int *)groups, &ngroups);
line 1234:
- if (getgrouplist(user, pw->pw_gid, groups, &ngroups) == -1) {
+ if (getgrouplist(user, pw->pw_gid, (int *)groups, &ngroups) == -1) {
关于hadoop - 在osx中构建hadoop 3.1.1以获取 native 库，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/54801924/