Hadoop如何加载第三方包

飞熊 • 2023年4月17日上午11:04 • 后端漫谈 • 阅读 217

报错背景：

本地idea编写Hadoop程序，对nginx日志进行分析，引入nl.bitwalker.useragentutils.UserAgent包解析nginx的浏览器信息获取浏览器系统名称和浏览器名称，

本地测试可以完全ok，然后打包到Hadoop集群运行，报错找不到nl.bitwalker.useragentutils.UserAgent包

java.lang.Exception: java.lang.NoClassDefFoundError: nl/bitwalker/useragentutils/UserAgent
	at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:492)
	at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:552)
Caused by: java.lang.NoClassDefFoundError: nl/bitwalker/useragentutils/UserAgent
	at cn.awz.log.LogMapper.getOperatingSystem(LogMapper.java:84)
	at cn.awz.log.LogMapper.getBrowser(LogMapper.java:93)
	at cn.awz.log.LogMapper.parse(LogMapper.java:49)
	at cn.awz.log.LogMapper.map(LogMapper.java:25)
	at cn.awz.log.LogMapper.map(LogMapper.java:16)
	at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146)
	at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
	at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
	at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:271)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.ClassNotFoundException: nl.bitwalker.useragentutils.UserAgent
	at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
	... 14 more

原因Hadoop默认不支持第三方包，

查了好多文档和博客依然没解决，无意间看见一篇博客（后来找不到啦）叫我在环境变量里面配置
HADOOP_CLASSPATH完美解决

解决在maven仓库中找到该包，

上传到主节点所在的Linux中，配置环境变量，

export HADOOP_CLASSPATH=~/UserAgentUtils-1.2.4.jar

激活环境变量

source /etc/profile

运行架包

hadoop jar code-1.0-SNAPSHOT.jar cn/awz/log/LogDriver /flume/upload/20220317/17 /nginxLogs

成功

文章由极客之音整理，本文链接：https://www.bmabk.com/index.php/post/133806.html