hadoop - Lein 在 Clojars 和依赖项上的部署

标签 hadoop clojure hbase leiningen clojars

我正在开发一个使用 Apache Hbase 作为其数据存储的应用程序。我围绕一些常见的 hbase 操作编写了一个 clojure 包装器,

https://github.com/mobiusinversion/hbase

并将其推送到 clojars。

https://clojars.org/hbase

在我的 hbase 包装器中,我导入了 Bytes 类:

(ns hbase.table
    (:gen-class)
    (:refer-clojure :exclude [get])
    (:import [clojure.lang PersistentVector PersistentArrayMap]
             [org.apache.hadoop.hbase.util Bytes]
             [org.apache.hadoop.hbase.client Put Get HTable Scan]))

然后在另一个名为“wtf”的项目中,我将包装器声明为依赖项,这很有效,它只是将 jar 从 clojars 中拉下来。

MacBook-Pro-2:wtf $ lein do clean, deps
... blah blah
Retrieving org/mortbay/jetty/jsp-2.1/6.1.14/jsp-2.1-6.1.14.jar from central
Retrieving ant/ant/1.6.5/ant-1.6.5.jar from central
Retrieving commons-el/commons-el/1.0/commons-el-1.0.jar from central
Retrieving net/java/dev/jets3t/jets3t/0.6.1/jets3t-0.6.1.jar from central
Retrieving hsqldb/hsqldb/1.8.0.10/hsqldb-1.8.0.10.jar from central
Retrieving oro/oro/2.0.8/oro-2.0.8.jar from central
Retrieving org/eclipse/jdt/core/3.1.1/core-3.1.1.jar from central
Retrieving org/codehaus/jackson/jackson-mapper-asl/1.8.8/jackson-mapper-asl-1.8.8.jar from central
Retrieving hbase/hbase/0.1.1/hbase-0.1.1.jar from clojars
MacBook-Pro-2:wtf $

但是在新项目中找不到 Bytes 类(以及所有其他 Hadoop 类):

$ lein repl
nREPL server started on port 58693
REPL-y 0.1.10
Clojure 1.5.1
    Exit: Control+D or (exit) or (quit)
Commands: (user/help)
    Docs: (doc function-name-here)
          (find-doc "part-of-name-here")
  Source: (source function-name-here)
          (user/sourcery function-name-here)
 Javadoc: (javadoc java-object-or-class-here)
Examples from clojuredocs.org: [clojuredocs or cdoc]
          (user/clojuredocs name-here)
          (user/clojuredocs "ns-here" "name-here")

user=> (use 'hbase.schema)
ClassNotFoundException org.apache.hadoop.hbase.HTableDescriptor java.net.URLClassLoader$1.run (URLClassLoader.java:202)

user=> (use 'hbase.config)
ClassNotFoundException org.apache.hadoop.hbase.HBaseConfiguration  java.net.URLClassLoader$1.run (URLClassLoader.java:202)

user=> (use 'hbase.table)
ClassNotFoundException org.apache.hadoop.hbase.util.Bytes  java.net.URLClassLoader$1.run (URLClassLoader.java:202)

user=>

我不知道为什么 hadoop 类既没有被 Leiningen 作为依赖项引入,也没有打包为我的 clojars 部署的一部分。我怎样才能让这些类可见???

最佳答案

事实证明这与使用限定符声明依赖项的顺序有关:

这里没有注册HBase依赖

(defproject hbase "0.1.0"
:description "HBase Access in Clojure"
:license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"}
:url "https://github.com/mobiusinversion/hbase"
:dependencies [
    [org.clojure/clojure "1.5.1"]
        [org.apache.hadoop/hadoop-core "1.2.0"]
    [org.apache.hadoop/hadoop-test "1.2.0" :scope "test"]
    [org.apache.hbase/hbase "0.94.6.1" :classifier "tests" :scope "test"]
    [org.apache.hbase/hbase "0.94.6.1"]]
:plugins [[lein-marginalia "0.7.1"]])

然而,这确实:

(defproject hbase "0.1.3"
:description "HBase Access in Clojure"
:license {:name "Eclipse Public License" :url "http://www.eclipse.org/legal/epl-v10.html"}
:url "https://github.com/mobiusinversion/hbase"
:dependencies [
    [org.clojure/clojure "1.5.1"]
    [org.apache.hadoop/hadoop-core "1.2.0"]
    [org.apache.hbase/hbase "0.94.6.1"]
    [org.apache.hadoop/hadoop-test "1.2.0"]
    [org.apache.hbase/hbase "0.94.6.1" :classifier "tests" :scope "test"]]
:plugins [[lein-marginalia "0.7.1"]])

关于hadoop - Lein 在 Clojars 和依赖项上的部署,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/17536693/

相关文章:

hadoop - HDFS:使用 HDFS API 附加到 SequenceFile

java - 如何让Hadoop将jar部署到集群?

Clojure:连续重复项的数量

clojure - 从 Compojure 提供静态文件

maven - hbase 和 osgi - 找不到 hbase-default.xml

java - 合并大量小文件以进行 mapreduce 输入

functional-programming - Clojure 中的可插入向量处理单元

hadoop - Hbase- 即使删除列族后 Hadoop DFS 大小也没有减少

hadoop - HBase - WAL 和 MemStore 有什么区别?

java - Hadoop 中的 NoClassDefFoundError 错误