mysql - sqoop 导出给出了带有表(~120 列)的 java.nio.BufferOverflowException

Question

我正在尝试将表导出到 maria db；然而，大多数表都工作正常，但有一个大约有 120 列的表出现了缓冲区溢出错误。

2017-05-11 13:33:36,674 INFO [main] org.apache.hive.hcatalog.mapreduce.InternalUtil: Initializing org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe with properties {name=default.temp_searched_sp, numFiles=4, columns.types=bigint,varchar(254),timestamp,timestamp,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,string,bigint,string,timestamp,timestamp,string,string,string,double,string,bigint,string,string,timestamp,timestamp,string,string,timestamp,timestamp,bigint,bigint,string,string,string,boolean,string,string,timestamp,timestamp,string, serialization.format=1, columns=sequence_id,id,est_received_at,received_at,advertisers_on_page,context_accept,context_accept_charset,context_accept_encoding,context_accept_language,context_akamai_origin_hop,context_akamai_reputation,context_alexatoolbar_alx_ns_ph,context_authorization,context_cache_control,context_cdma1989,context_client_ip,context_connection,context_content_length,context_content_type,context_cookie,context_d_token,context_dnt,context_el_auth_param,context_fooheader,context_gateway_ip,context_giga_transport,context_host,context_iorad_extension,context_iv_user,context_library_name,context_library_version,context_myrefer,context_oppo_request_type,context_orig_host,context_origin,context_pragma,context_prefer,context_q_token,context_ra_sid,context_ra_ver,context_referer,context_s_token,context_save_data,context_sm_user,context_surrogate_capability,context_transaction_id,context_true_client_ip,context_ua_cpu,context_up_recursive_request,context_user_agent,context_usertoken,context_via,context_wk_utd_ip,context_ws_grp,context_x_akamai_config_log_detail,context_x_angi_applicationversion,context_x_angi_featureflags,context_x_angi_proxyversion,context_x_angi_requestid,context_x_angi_sourceapplication,context_x_att_deviceid,context_x_bluecoat_via,context_x_browser_session,context_x_clickoncesupport,context_x_client_id,context_x_csix_custid,context_x_csix_custkey,context_x_cw_pageurl,context_x_elastica_gw,context_x_forwarded_for,context_x_forwarded_port,context_x_forwarded_proto,context_x_icm,context_x_imforwards,context_x_int,context_x_iws_via,context_x_mwg_via,context_x_newrelic_id,context_x_paas_uid,context_x_personasinteractive_addon,context_x_psa_client_features,context_x_psa_client_options,context_x_real_ip,context_x_request_id,context_x_requested_with,context_x_rl_forwarded_for,context_x_sfs_embed,context_x_sharepath_rum_enabled,context_x_target_proxy,context_x_wap_profile,context_xroxy_connection,event,event_text,location_info_advertising_zone,location_info_search_zip_code,est_original_timestamp,original_timestamp,results,search_for,search_params_filters_categories,search_params_filters_distance_from_provider,search_params_filters_first_name,search_params_page,search_params_query,search_params_type,est_sent_at,sent_at,sort_sort_by,sort_sort_field,est_e_timestamp,e_timestamp,total_pages,total_results,user_id,query,search_type,test_user,category_id,edh_raw_file_name,est_load_timestamp,utc_load_timestamp,edh_bus_month, rawDataSize=112904662, columns.comments=nullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnullnull, numRows=155793, serialization.lib=org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe, COLUMN_STATS_ACCURATE={"BASIC_STATS":"true"}, totalSize=113060455, serialization.null.format=\N, transient_lastDdlTime=1494509566}
2017-05-11 13:33:37,971 ERROR [Thread-13] org.apache.hadoop.yarn.YarnUncaughtExceptionHandler: Thread Thread[Thread-13,5,main] threw an Exception.
java.nio.BufferOverflowException
    at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:189)
    at java.nio.ByteBuffer.put(ByteBuffer.java:859)
    at org.mariadb.jdbc.internal.packet.send.SendExecutePrepareStatementPacket.send(SendExecutePrepareStatementPacket.java:105)
    at org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executePreparedQuery(AbstractQueryProtocol.java:578)
    at org.mariadb.jdbc.MariaDbServerPreparedStatement.executeInternal(MariaDbServerPreparedStatement.java:279)
    at org.mariadb.jdbc.MariaDbServerPreparedStatement.execute(MariaDbServerPreparedStatement.java:369)
    at org.apache.sqoop.mapreduce.AsyncSqlOutputFormat$AsyncSqlExecThread.run(AsyncSqlOutputFormat.java:233)

我在 aws emr 上运行 sqoop。

我正在运行的 Sqoop 命令是:

sqoop export --connect jdbc:mysql://xxx.xxx.xxx.xxx:3306/Web --username stage --password stage --table  searched_sp_UAT --staging-table searched_sp_UAT_stg --clear-staging-table --hcatalog-table temp_searched_sp

如果有人能给我一些建议，请告诉我；或者遇到过类似的情况。

Answer 1

看起来非常像这个错误:CONJ-270

1. 您正在使用 MariaDB JDBC 连接器，而不是常规的 MySQL 连接器(参见您的堆栈跟踪)
2. Sqoop 使用带有大量参数的 PreparedStatement(参见您的堆栈跟踪)
3. 在 V4.1 之前，MariaDB 连接器在包含大量参数的 PreparedStatement 上存在错误

因此，我强烈建议您改用广泛使用且经过彻底测试的常规 MySQL JDBC 驱动程序。
不要使用外来软件，除非您愿意知道你在做什么。