flume打包遇到的一些問題

1.ipc兼容性問題，線上使用2.3.0的hdfs，但是打包時默認為1.2.1的

1
2
3
4
08?Apr?2015?19:38:25,122?WARN??[SinkRunner-PollingRunner-DefaultSinkProcessor]?(org.apache.flume.sink.hdfs.HDFSEventSink.process:455)??-?HDFS?IO?error
org.apache.hadoop.ipc.RemoteException:?Server?IPC?version?9?cannot?communicate?with?client?version?4
????????at?org.apache.hadoop.ipc.Client.call(Client.java:1113)
????????at?org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:229)

拷貝hadoop-core包即可,或者更改pom.xml文件

1
2
3
4
5
6
??????<properties>
????????<hadoop.version>1.2.1</hadoop.version>?//hadoop-core-1.2.1.jar??改為hadoop-core-2.3.0-mr1-cdh5.1.0.jar
????????<hbase.version>0.92.1</hbase.version>
????????<hadoop.common.artifact.id>hadoop-core</hadoop.common.artifact.id>
????????<thrift.version>0.7.0</thrift.version>
??????</properties>

或者打包時指定：

1
mvn?clean?install?-Phadoop-2

2.沒有跳過test時

1
org.apache.flume.auth.TestFlumeAuthenticator:?org/apache/commons/io/Charsets?(no?class?error)

根目錄下的pom.xml文件中更改為:

1
2
3
4
5
<dependency>
???<groupId>commons-io</groupId>
???<artifactId>commons-io</artifactId>
???<version>2.4</version>?//2.1的包沒有Charsets這個類
</dependency>

如果手動下載jar包可以使用如下命令導入：

1
mvn?install:install-file?-DgroupId=commons-io?-DartifactId=commons-io?-Dversion=2.0.1?-Dpackaging=jar?-Dfile=commons-io-2.4.jar

3. 程序語言問題，因為在系統中使用了中文環境導致編譯不通過

1）

1
2
3
4
5
6
@Test
??public?void?shouldUseUtcAsBasisForDateFormat()?{
????assertEquals("Coordinated?Universal?Time",
????????factory.fastDateFormat.getTimeZone().getDisplayName());?//請求英文，返回中文
??}
...

2）

1
2
3
4
5
6
Running?org.apache.flume.source.twitter.TestTwitterSource
Tests?run:?2,?Failures:?0,?Errors:?1,?Skipped:?1,?Time?elapsed:?0.247?sec?<<<?FAILURE!
testCarrotDateFormatBug(org.apache.flume.source.twitter.TestTwitterSource)??Time?elapsed:?21?sec??<<<?ERROR!
java.text.ParseException:?Unparseable?date:?"Fri?Oct?26?22:53:55?+0000?2012"
???at?java.text.DateFormat.parse(DateFormat.java:357)
???at?org.apache.flume.source.twitter.TestTwitterSource.testCarrotDateFormatBug(TestTwitterSource.java:109)

代碼：

1
2
3
4
5
@Test
??public?void?testCarrotDateFormatBug()?throws?Exception?{
????SimpleDateFormat?formatterFrom?=?new?SimpleDateFormat("EEE?MMM?dd?HH:mm:ss?Z?yyyy");?//更改為new?SimpleDateFormat("EEE?MMM?dd?HH:mm:ss?Z?yyyy",Locale.US);
????formatterFrom.parse("Fri?Oct?26?22:53:55?+0000?2012");
??}


或者簡單點，更改語言設置或者直接skip test

1
mvn?clean?install?-Phadoop-2?-DskipTests


4.json問題

替換單個jar包時，因為依賴問題，包class not found:

1
2
3
4
5
Exception?in?thread?"PollableSourceRunner-KafkaSource-kafka1"?java.lang.NoClassDefFoundError:?net/sf/ezmorph/Morpher
????????at?org.apache.flume.source.kafka.KafkaSourceUtil.getDateMessage(KafkaSourceUtil.java:117)
????????at?org.apache.flume.source.kafka.KafkaSource.process(KafkaSource.java:123)
????????at?org.apache.flume.source.PollableSourceRunner$PollingRunner.run(PollableSourceRunner.java:139)
????????at?java.lang.Thread.run(Thread.java:745)

可以打成ensmbly包

1
2
3
4
5
6
7
flume-1.6.0/flume-ng-sources/flume-kafka-source/pom.xml
<dependency>
???<groupId>net.sf.json-lib</groupId>
???<artifactId>json-lib</artifactId>
???<version>2.4</version>
???<classifier>jdk15</classifier>
</dependency>

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
<plugin>
????<artifactId>maven-assembly-plugin</artifactId>
????<version>2.4</version>
????<configuration>
??????<descriptorRefs>
????????<descriptorRef>jar-with-dependencies</descriptorRef>
??????</descriptorRefs>
????</configuration>
????<executions>
??????<execution>
????????<id>make-assembly</id>
????????<phase>package</phase>
????????<goals>
??????????<goal>single</goal>
????????</goals>
??????</execution>
????</executions>
??</plugin>

或者拷貝jar包：

1
2
3
4
5
6
/Users/nizengguang/.m2/repository/net/sf/ezmorph/ezmorph/1.0.6/ezmorph-1.0.6.jar
/Users/nizengguang/.m2/repository/net/sf/json-lib/json-lib/2.4/json-lib-2.4-jdk15.jar
/Users/nizengguang/.m2/repository/commons-beanutils/commons-beanutils/1.8.0/commons-beanutils-1.8.0.jar
/Users/nizengguang/.m2/repository/commons-collections/commons-collections/3.2.1/commons-collections-3.2.1.jar
/Users/nizengguang/.m2/repository/commons-logging/commons-logging/1.1.1/commons-logging-1.1.1.jar
/Users/nizengguang/.m2/repository/commons-lang/commons-lang/2.5/commons-lang-2.5.jar