?/*
? ? ?* Mapreduce步驟
? ? ?* 1、map：獲取一行
? ? ?* 2、按空格分隔
? ? ?* 3、每個單詞統計，<key,1> 輸出到 reducer
? ? ?* 4、reducer <key,{1,1,1,1,1}> 匯總
? ? ?* 5、輸出結果
? ? ?*/

1、聲明一個集合，模仿要統計的文檔內容，一個逗號分隔的集合元素相當于一行

scala> val lines = List("hello tom hello jerry", "hello jerry", "hello kitty","beijing shanghai tianjing","xiaoming xiaohua xiaoli xiaozhang")

2、?map：獲取一行，按空格分隔

scala> val line2=lines.flatMap(_.split(" "))
line2: List[String] = List(hello, tom, hello, jerry, hello, jerry, hello, kitty, beijing, shanghai, tianjing, xiaoming, xiaohua, xiaoli, xiaozhang)

3、每個單詞統計，<key,1> 輸出到 reducer

scala> val line3=line2.map((_,1))
line3: List[(String, Int)] = List((hello,1), (tom,1), (hello,1), (jerry,1), (hello,1), (jerry,1), (hello,1), (kitty,1), (beijing,1), (shanghai,1), (tianjing,1), (xiaoming,1), (xiaohua,1), (xiaoli,1), (xiaozhang,1))

4、reducer <key,{1,1,1,1,1}> 匯總

scala> val line4=line3.groupBy(_._1)
line4: scala.collection.immutable.Map[String,List[(String, Int)]] = Map(kitty -> List((kitty,1)), xiaoming -> List((xiaoming,1)), tianjing -> List((tianjing,1)), shanghai -> List((shanghai,1)), xiaohua -> List((xiaohua,1)), tom -> List((tom,1)), beijing -> List((beijing,1)), hello -> List((hello,1), (hello,1), (hello,1), (hello,1)), jerry -> List((jerry,1), (jerry,1)), xiaozhang -> List((xiaozhang,1)), xiaoli -> List((xiaoli,1)))

scala> val line5=line4.map(t=>(t._1,t._2.size))
line5: scala.collection.immutable.Map[String,Int] = Map(kitty -> 1, xiaoming -> 1, tianjing -> 1, shanghai -> 1, xiaohua -> 1, tom -> 1, beijing -> 1, hello -> 4, jerry -> 2, xiaozhang -> 1, xiaoli -> 1)

5、輸出結果

scala> val line6=line5.toList
line6: List[(String, Int)] = List((kitty,1), (xiaoming,1), (tianjing,1), (shanghai,1), (xiaohua,1), (tom,1), (beijing,1), (hello,4), (jerry,2), (xiaozhang,1), (xiaoli,1))

scala> val line7=line6.sortBy(_._2).reverse
line7: List[(String, Int)] = List((hello,4), (jerry,2), (xiaoli,1), (xiaozhang,1), (beijing,1), (tom,1), (xiaohua,1), (shanghai,1), (tianjing,1), (xiaoming,1), (kitty,1))

-------------

更多的Java，Android，大數據，J2EE，Python，數據庫，Linux，Java架構師，教程，視頻請訪問：

http://www.cnblogs.com/zengmiaogen/p/7083694.html