hadoop2.7 偽分布

hadoop 2.7.3偽分布式環境運行官方wordcount

hadoop 2.7.3偽分布式模式運行wordcount

基本環境：
系統：win7
虛機環境：virtualBox
虛機：centos 7
hadoop版本：2.7.3

本次以偽分布式模式來運行wordcount。

參考：

  • hadoop docs

?

1 hadoop環境

偽分布式就是將多個hadoop組件部署在一臺機器上。因此涉及到各組件的配置，以及機器信任關系。

### 準備一個全新的環境
# cd /home/jungle/hadoop
# tar -zxvf hadoop-2.7.3.tar.gz
# mv hadoop-2.7.3 hadoop-daemon
# cd /home/jungle/hadoop/hadoop-daemon/

1.1 修改hadoop配置

  • core-site.xml
# vi etc/hadoop/core-site.xml
<configuration><property><name>fs.defaultFS</name><value>hdfs://localhost:9000</value></property>
</configuration>
  • hdfs-site.xml
# vi etc/hadoop/hdfs-site.xml
<configuration><property><name>dfs.replication</name><value>1</value></property>
</configuration>

1.2 信任關系

# ssh-keygen -t rsa
# cat ~/.ssh/id_rsa.pub >> ~/.ssh/authorized_keys# ps
# ssh localhost
### 登陸本機# ps
### 確認兩次ps顯示的終端是不同的tty，則成功了

1.3 格式化hdfs

# hadoop fs -ls /
Found 20 items
-rw-r--r--   1 root root          0 2016-12-30 12:26 /1
dr-xr-xr-x   - root root      45056 2016-12-30 13:06 /bin
dr-xr-xr-x   - root root       4096 2016-12-29 20:09 /boot
drwxr-xr-x   - root root       3120 2017-01-06 18:31 /dev
drwxr-xr-x   - root root       8192 2017-01-06 18:32 /etc
# ... 是linux文件系統 # hdfs namenode -format
17/01/06 19:29:51 INFO namenode.NameNode: STARTUP_MSG: 
/************************************************************
STARTUP_MSG: Starting NameNode
STARTUP_MSG:   host = localhost/127.0.0.1
STARTUP_MSG:   args = [-format]
STARTUP_MSG:   version = 2.7.3
#...STARTUP_MSG:   java = 1.8.0_111
************************************************************/
17/01/06 19:29:51 INFO namenode.NameNode: registered UNIX signal handlers for [TERM, HUP, INT]
17/01/06 19:29:51 INFO namenode.NameNode: createNameNode [-format]
Formatting using clusterid: CID-ee109ab5-d5f1-4919-a1c6-5ff4de21a03f
17/01/06 19:29:52 INFO namenode.FSNamesystem: No KeyProvider found.
17/01/06 19:29:52 INFO namenode.FSNamesystem: fsLock is fair:true
17/01/06 19:29:52 INFO blockmanagement.DatanodeManager: dfs.block.invalidate.limit=1000
17/01/06 19:29:52 INFO blockmanagement.DatanodeManager: dfs.namenode.datanode.registration.ip-hostname-check=true
17/01/06 19:29:52 INFO blockmanagement.BlockManager: dfs.namenode.startup.delay.block.deletion.sec is set to 000:00:00:00.000
17/01/06 19:29:52 INFO blockmanagement.BlockManager: The block deletion will start around 2017 Jan 06 19:29:52
17/01/06 19:29:52 INFO util.GSet: Computing capacity for map BlocksMap
17/01/06 19:29:52 INFO util.GSet: VM type       = 64-bit
17/01/06 19:29:52 INFO util.GSet: 2.0% max memory 966.7 MB = 19.3 MB
17/01/06 19:29:52 INFO util.GSet: capacity      = 2^21 = 2097152 entries
17/01/06 19:29:52 INFO blockmanagement.BlockManager: dfs.block.access.token.enable=false
17/01/06 19:29:52 INFO blockmanagement.BlockManager: defaultReplication         = 3
17/01/06 19:29:52 INFO blockmanagement.BlockManager: maxReplication             = 512
17/01/06 19:29:52 INFO blockmanagement.BlockManager: minReplication             = 1
17/01/06 19:29:52 INFO blockmanagement.BlockManager: maxReplicationStreams      = 2
17/01/06 19:29:52 INFO blockmanagement.BlockManager: replicationRecheckInterval = 3000
17/01/06 19:29:52 INFO blockmanagement.BlockManager: encryptDataTransfer        = false
17/01/06 19:29:52 INFO blockmanagement.BlockManager: maxNumBlocksToLog          = 1000
17/01/06 19:29:52 INFO namenode.FSNamesystem: fsOwner             = jungle (auth:SIMPLE)
17/01/06 19:29:52 INFO namenode.FSNamesystem: supergroup          = supergroup
17/01/06 19:29:52 INFO namenode.FSNamesystem: isPermissionEnabled = true
17/01/06 19:29:52 INFO namenode.FSNamesystem: HA Enabled: false
17/01/06 19:29:52 INFO namenode.FSNamesystem: Append Enabled: true
17/01/06 19:29:52 INFO util.GSet: Computing capacity for map INodeMap
17/01/06 19:29:52 INFO util.GSet: VM type       = 64-bit
17/01/06 19:29:52 INFO util.GSet: 1.0% max memory 966.7 MB = 9.7 MB
17/01/06 19:29:52 INFO util.GSet: capacity      = 2^20 = 1048576 entries
17/01/06 19:29:52 INFO namenode.FSDirectory: ACLs enabled? false
17/01/06 19:29:52 INFO namenode.FSDirectory: XAttrs enabled? true
17/01/06 19:29:52 INFO namenode.FSDirectory: Maximum size of an xattr: 16384
17/01/06 19:29:52 INFO namenode.NameNode: Caching file names occuring more than 10 times
17/01/06 19:29:52 INFO util.GSet: Computing capacity for map cachedBlocks
17/01/06 19:29:52 INFO util.GSet: VM type       = 64-bit
17/01/06 19:29:52 INFO util.GSet: 0.25% max memory 966.7 MB = 2.4 MB
17/01/06 19:29:52 INFO util.GSet: capacity      = 2^18 = 262144 entries
17/01/06 19:29:52 INFO namenode.FSNamesystem: dfs.namenode.safemode.threshold-pct = 0.9990000128746033
17/01/06 19:29:52 INFO namenode.FSNamesystem: dfs.namenode.safemode.min.datanodes = 0
17/01/06 19:29:52 INFO namenode.FSNamesystem: dfs.namenode.safemode.extension     = 30000
17/01/06 19:29:52 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.window.num.buckets = 10
17/01/06 19:29:52 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.num.users = 10
17/01/06 19:29:52 INFO metrics.TopMetrics: NNTop conf: dfs.namenode.top.windows.minutes = 1,5,25
17/01/06 19:29:52 INFO namenode.FSNamesystem: Retry cache on namenode is enabled
17/01/06 19:29:53 INFO namenode.FSNamesystem: Retry cache will use 0.03 of total heap and retry cache entry expiry time is 600000 millis
17/01/06 19:29:53 INFO util.GSet: Computing capacity for map NameNodeRetryCache
17/01/06 19:29:53 INFO util.GSet: VM type       = 64-bit
17/01/06 19:29:53 INFO util.GSet: 0.029999999329447746% max memory 966.7 MB = 297.0 KB
17/01/06 19:29:53 INFO util.GSet: capacity      = 2^15 = 32768 entries
17/01/06 19:29:53 INFO namenode.FSImage: Allocated new BlockPoolId: BP-1788036100-127.0.0.1-1483702193052
17/01/06 19:29:53 INFO common.Storage: Storage directory /tmp/hadoop-jungle/dfs/name has been successfully formatted.
17/01/06 19:29:53 INFO namenode.FSImageFormatProtobuf: Saving image file /tmp/hadoop-jungle/dfs/name/current/fsimage.ckpt_0000000000000000000 using no compression
17/01/06 19:29:53 INFO namenode.FSImageFormatProtobuf: Image file /tmp/hadoop-jungle/dfs/name/current/fsimage.ckpt_0000000000000000000 of size 353 bytes saved in 0 seconds.
17/01/06 19:29:53 INFO namenode.NNStorageRetentionManager: Going to retain 1 images with txid >= 0
17/01/06 19:29:53 INFO util.ExitUtil: Exiting with status 0
17/01/06 19:29:53 INFO namenode.NameNode: SHUTDOWN_MSG: 
/************************************************************
SHUTDOWN_MSG: Shutting down NameNode at localhost/127.0.0.1
************************************************************/

列出上面的日志，可以看到操作結果。其中最重要的應該就是基于linux文件系統存儲的hdfs：

# ls -l /tmp/hadoop-jungle/dfs/name/current/
total 16
-rw-rw-r--. 1 jungle jungle 353 Jan  6 19:29 fsimage_0000000000000000000
-rw-rw-r--. 1 jungle jungle  62 Jan  6 19:29 fsimage_0000000000000000000.md5
-rw-rw-r--. 1 jungle jungle   2 Jan  6 19:29 seen_txid
-rw-rw-r--. 1 jungle jungle 201 Jan  6 19:29 VERSION

1.4 安裝jps

如上篇中只安裝了java。還需要安裝jps等工具

# yum install java-1.8.0-openjdk-devel#jps
4497 Jps

2 啟動hadoop

2.1 啟動hdfs

# sbin/start-dfs.sh 
Starting namenodes on [localhost]
localhost: starting namenode, logging to /home/jungle/hadoop/hadoop-daemon/logs/hadoop-jungle-namenode-localhost.out
localhost: starting datanode, logging to /home/jungle/hadoop/hadoop-daemon/logs/hadoop-jungle-datanode-localhost.out
Starting secondary namenodes [0.0.0.0]
The authenticity of host '0.0.0.0 (0.0.0.0)' can't be established.
ECDSA key fingerprint is 6a:67:9f:8b:84:64:db:19:1a:ba:86:4f:f1:9a:1c:82.
Are you sure you want to continue connecting (yes/no)? yes
0.0.0.0: Warning: Permanently added '0.0.0.0' (ECDSA) to the list of known hosts.
0.0.0.0: starting secondarynamenode, logging to /home/jungle/hadoop/hadoop-daemon/logs/hadoop-jungle-secondarynamenode-localhost.out# echo $?
0# ls -ltr logs/ 
total 96
-rw-rw-r--. 1 jungle jungle     0 Jan  6 20:17 SecurityAuth-jungle.audit
-rw-rw-r--. 1 jungle jungle   716 Jan  6 20:17 hadoop-jungle-namenode-localhost.out
-rw-rw-r--. 1 jungle jungle   716 Jan  6 20:17 hadoop-jungle-datanode-localhost.out
-rw-rw-r--. 1 jungle jungle 29280 Jan  6 20:17 hadoop-jungle-namenode-localhost.log
-rw-rw-r--. 1 jungle jungle 25370 Jan  6 20:17 hadoop-jungle-datanode-localhost.log
-rw-rw-r--. 1 jungle jungle   716 Jan  6 20:17 hadoop-jungle-secondarynamenode-localhost.out
-rw-rw-r--. 1 jungle jungle 22386 Jan  6 20:17 hadoop-jungle-secondarynamenode-localhost.log# jps
4977 SecondaryNameNode
4802 DataNode
4660 NameNode
5095 Jps

如上可以看到，已經啟動了NameNode及SecondaryNameNode。以及DataNode。相應的，日志文件下也有對應的out和log文件。


# ls -l /tmp/hadoop-jungle/dfs/name/current/
total 3036
-rw-rw-r--. 1 jungle jungle      42 Jan  6 20:18 edits_0000000000000000001-0000000000000000002
-rw-rw-r--. 1 jungle jungle 1048576 Jan  6 20:18 edits_0000000000000000003-0000000000000000003
-rw-rw-r--. 1 jungle jungle 1048576 Jan  8 14:56 edits_inprogress_0000000000000000004
-rw-rw-r--. 1 jungle jungle     353 Jan  6 20:18 fsimage_0000000000000000002
-rw-rw-r--. 1 jungle jungle      62 Jan  6 20:18 fsimage_0000000000000000002.md5
-rw-rw-r--. 1 jungle jungle     353 Jan  8 14:56 fsimage_0000000000000000003
-rw-rw-r--. 1 jungle jungle      62 Jan  8 14:56 fsimage_0000000000000000003.md5
-rw-rw-r--. 1 jungle jungle       2 Jan  8 14:56 seen_txid
-rw-rw-r--. 1 jungle jungle     201 Jan  8 14:56 VERSION### pid 
# ls -l /tmp/hadoop-jungle-*
-rw-rw-r--. 1 jungle jungle 5 Jan  8 14:56 /tmp/hadoop-jungle-datanode.pid
-rw-rw-r--. 1 jungle jungle 5 Jan  8 14:56 /tmp/hadoop-jungle-namenode.pid
-rw-rw-r--. 1 jungle jungle 5 Jan  8 14:56 /tmp/hadoop-jungle-secondarynamenode.pid

2.2 檢查頁面

先關閉防火墻。

 
# systemctl status firewalld.service
● firewalld.service - firewalld - dynamic firewall daemon
   Loaded: loaded (/usr/lib/systemd/system/firewalld.service; enabled; vendor prest: enabled)
   Active: inactive (dead) since Sun 2017-01-08 15:12:58 CST; 8s ago
     Docs: man:firewalld(1)
  Process: 681 ExecStart=/usr/sbin/firewalld --nofork --nopid $FIREWALLD_ARGS (cod=exited, status=0/SUCCESS)Main PID: 681 (code=exited, status=0/SUCCESS)# systemctl disable firewalld.service
Removed symlink /etc/systemd/system/basic.target.wants/firewalld.service.
Removed symlink /etc/systemd/system/dbus-org.fedoraproject.FirewallD1.service.

2.3 訪問NameNode

鏈接：

http://192.168.1.111:50070/

修改環境變量。因為之前在單機模式下有另一個測試目錄。將其從hadoop-local改回hadoop-daemon

# vi ~/.bashrc
### export HADOOP_INSTALL=/home/jungle/hadoop/hadoop-local
export HADOOP_INSTALL=/home/jungle/hadoop/hadoop-daemon# source ~/.bashrc

操作hdfs

# hadoop fs -ls /
### 輸出為空，根目錄下沒有任何內容# hdfs dfs -mkdir /user
# hadoop fs -ls /
Found 1 items
drwxr-xr-x   - jungle supergroup          0 2017-01-08 15:57 /user# hdfs dfs -mkdir /user/test
# hadoop fs -ls /user/
Found 1 items
drwxr-xr-x   - jungle supergroup          0 2017-01-08 15:57 /user/test# hadoop fs -put ../hadoop-local/dataLocal/input/ /user/test# hadoop fs -ls /user/test
Found 1 items
drwxr-xr-x   - jungle supergroup          0 2017-01-08 16:02 /user/test/input# hadoop fs -ls /user/test/input
Found 2 items
-rw-r--r--   1 jungle supergroup         37 2017-01-08 16:02 /user/test/input/file1.txt
-rw-r--r--   1 jungle supergroup         70 2017-01-08 16:02 /user/test/input/file2.txt

訪問：

  • http://192.168.1.111:50070/explorer.html#
  • http://192.168.1.111:50070/explorer.html#/user/test/input

3 wordcount

# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /user/test/input/ /user/test/output# bin/hadoop fs -ls /user/test/output
Found 2 items
-rw-r--r--   1 jungle supergroup          0 2017-01-08 16:11 /user/test/output/_SUCCESS
-rw-r--r--   1 jungle supergroup         82 2017-01-08 16:11 /user/test/output/part-r-00000bin/hadoop fs -cat /user/test/output/part-r-00000
I   1
am  1
bye 2
great   1
hadoop. 3
hello   3
is  1
jungle. 2
software    1
the 1
world.  2

4 使用yarn

啟動yarn

# jps
4803 DataNode
4979 SecondaryNameNode
4661 NameNode
6309 Jps# sbin/start-yarn.sh 
starting yarn daemons
starting resourcemanager, logging to /home/jungle/hadoop/hadoop-daemon/logs/yarn-jungle-resourcemanager-localhost.localdomain.out
localhost: starting nodemanager, logging to /home/jungle/hadoop/hadoop-daemon/logs/yarn-jungle-nodemanager-localhost.localdomain.out# jps
4803 DataNode
4979 SecondaryNameNode
6355 ResourceManager
4661 NameNode
6477 NodeManager
6750 Jps# hadoop fs -ls /user/test/
Found 2 items
drwxr-xr-x   - jungle supergroup          0 2017-01-08 16:02 /user/test/input
drwxr-xr-x   - jungle supergroup          0 2017-01-08 16:11 /user/test/output# bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.3.jar wordcount /user/test/input/ /user/test/output2# hadoop fs -ls /user/test/
Found 3 items
drwxr-xr-x   - jungle supergroup          0 2017-01-08 16:02 /user/test/input
drwxr-xr-x   - jungle supergroup          0 2017-01-08 16:11 /user/test/output
drwxr-xr-x   - jungle supergroup          0 2017-01-08 16:25 /user/test/output2# hadoop fs -ls /user/test/output2
Found 2 items
-rw-r--r--   1 jungle supergroup          0 2017-01-08 16:25 /user/test/output2/_SUCCESS
-rw-r--r--   1 jungle supergroup         82 2017-01-08 16:25 /user/test/output2/part-r-00000# hadoop fs -cat /user/test/output2/part-r-00000
I   1
am  1
bye 2
great   1
hadoop. 3
hello   3
is  1
jungle. 2
software    1
the 1
world.  2

執行日志：

17/01/08 16:25:32 INFO Configuration.deprecation: session.id is deprecated. Instead, use dfs.metrics.session-id
17/01/08 16:25:32 INFO jvm.JvmMetrics: Initializing JVM Metrics with processName=JobTracker, sessionId=
17/01/08 16:25:32 INFO input.FileInputFormat: Total input paths to process : 2
17/01/08 16:25:33 INFO mapreduce.JobSubmitter: number of splits:2
17/01/08 16:25:33 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_local247232145_0001
17/01/08 16:25:33 INFO mapreduce.Job: The url to track the job: http://localhost:8080/
17/01/08 16:25:33 INFO mapreduce.Job: Running job: job_local247232145_0001
17/01/08 16:25:33 INFO mapred.LocalJobRunner: OutputCommitter set in config null
17/01/08 16:25:33 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/01/08 16:25:33 INFO mapred.LocalJobRunner: OutputCommitter is org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
17/01/08 16:25:33 INFO mapred.LocalJobRunner: Waiting for map tasks
17/01/08 16:25:33 INFO mapred.LocalJobRunner: Starting task: attempt_local247232145_0001_m_000000_0
17/01/08 16:25:33 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/01/08 16:25:33 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
17/01/08 16:25:33 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/test/input/file2.txt:0+70
17/01/08 16:25:35 INFO mapreduce.Job: Job job_local247232145_0001 running in uber mode : false
17/01/08 16:25:35 INFO mapreduce.Job:  map 0% reduce 0%
17/01/08 16:25:35 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/01/08 16:25:35 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/01/08 16:25:35 INFO mapred.MapTask: soft limit at 83886080
17/01/08 16:25:35 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/01/08 16:25:35 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/01/08 16:25:37 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/01/08 16:25:37 INFO mapred.LocalJobRunner: 
17/01/08 16:25:37 INFO mapred.MapTask: Starting flush of map output
17/01/08 16:25:37 INFO mapred.MapTask: Spilling map output
17/01/08 16:25:37 INFO mapred.MapTask: bufstart = 0; bufend = 114; bufvoid = 104857600
17/01/08 16:25:37 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214356(104857424); length = 41/6553600
17/01/08 16:25:38 INFO mapred.MapTask: Finished spill 0
17/01/08 16:25:38 INFO mapred.Task: Task:attempt_local247232145_0001_m_000000_0 is done. And is in the process of committing
17/01/08 16:25:38 INFO mapred.LocalJobRunner: map
17/01/08 16:25:38 INFO mapred.Task: Task 'attempt_local247232145_0001_m_000000_0' done.
17/01/08 16:25:38 INFO mapred.LocalJobRunner: Finishing task: attempt_local247232145_0001_m_000000_0
17/01/08 16:25:38 INFO mapred.LocalJobRunner: Starting task: attempt_local247232145_0001_m_000001_0
17/01/08 16:25:38 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/01/08 16:25:38 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
17/01/08 16:25:38 INFO mapred.MapTask: Processing split: hdfs://localhost:9000/user/test/input/file1.txt:0+37
17/01/08 16:25:38 INFO mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
17/01/08 16:25:38 INFO mapred.MapTask: mapreduce.task.io.sort.mb: 100
17/01/08 16:25:38 INFO mapred.MapTask: soft limit at 83886080
17/01/08 16:25:38 INFO mapred.MapTask: bufstart = 0; bufvoid = 104857600
17/01/08 16:25:38 INFO mapred.MapTask: kvstart = 26214396; length = 6553600
17/01/08 16:25:38 INFO mapred.MapTask: Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer
17/01/08 16:25:38 INFO mapred.LocalJobRunner: 
17/01/08 16:25:38 INFO mapred.MapTask: Starting flush of map output
17/01/08 16:25:38 INFO mapred.MapTask: Spilling map output
17/01/08 16:25:38 INFO mapred.MapTask: bufstart = 0; bufend = 65; bufvoid = 104857600
17/01/08 16:25:38 INFO mapred.MapTask: kvstart = 26214396(104857584); kvend = 26214372(104857488); length = 25/6553600
17/01/08 16:25:38 INFO mapred.MapTask: Finished spill 0
17/01/08 16:25:38 INFO mapred.Task: Task:attempt_local247232145_0001_m_000001_0 is done. And is in the process of committing
17/01/08 16:25:38 INFO mapred.LocalJobRunner: map
17/01/08 16:25:38 INFO mapred.Task: Task 'attempt_local247232145_0001_m_000001_0' done.
17/01/08 16:25:38 INFO mapred.LocalJobRunner: Finishing task: attempt_local247232145_0001_m_000001_0
17/01/08 16:25:38 INFO mapred.LocalJobRunner: map task executor complete.
17/01/08 16:25:38 INFO mapreduce.Job:  map 100% reduce 0%
17/01/08 16:25:39 INFO mapred.LocalJobRunner: Waiting for reduce tasks
17/01/08 16:25:39 INFO mapred.LocalJobRunner: Starting task: attempt_local247232145_0001_r_000000_0
17/01/08 16:25:39 INFO output.FileOutputCommitter: File Output Committer Algorithm version is 1
17/01/08 16:25:39 INFO mapred.Task:  Using ResourceCalculatorProcessTree : [ ]
17/01/08 16:25:39 INFO mapred.ReduceTask: Using ShuffleConsumerPlugin: org.apache.hadoop.mapreduce.task.reduce.Shuffle@6bef8a0a
17/01/08 16:25:39 INFO reduce.MergeManagerImpl: MergerManager: memoryLimit=363285696, maxSingleShuffleLimit=90821424, mergeThreshold=239768576, ioSortFactor=10, memToMemMergeOutputsThreshold=10
17/01/08 16:25:39 INFO reduce.EventFetcher: attempt_local247232145_0001_r_000000_0 Thread started: EventFetcher for fetching Map Completion Events
17/01/08 16:25:39 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local247232145_0001_m_000000_0 decomp: 98 len: 102 to MEMORY
17/01/08 16:25:39 INFO reduce.InMemoryMapOutput: Read 98 bytes from map-output for attempt_local247232145_0001_m_000000_0
17/01/08 16:25:39 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 98, inMemoryMapOutputs.size() -> 1, commitMemory -> 0, usedMemory ->98
17/01/08 16:25:39 INFO reduce.LocalFetcher: localfetcher#1 about to shuffle output of map attempt_local247232145_0001_m_000001_0 decomp: 68 len: 72 to MEMORY
17/01/08 16:25:39 INFO reduce.InMemoryMapOutput: Read 68 bytes from map-output for attempt_local247232145_0001_m_000001_0
17/01/08 16:25:39 INFO reduce.MergeManagerImpl: closeInMemoryFile -> map-output of size: 68, inMemoryMapOutputs.size() -> 2, commitMemory -> 98, usedMemory ->166
17/01/08 16:25:39 INFO reduce.EventFetcher: EventFetcher is interrupted.. Returning
17/01/08 16:25:39 INFO mapred.LocalJobRunner: 2 / 2 copied.
17/01/08 16:25:39 INFO reduce.MergeManagerImpl: finalMerge called with 2 in-memory map-outputs and 0 on-disk map-outputs
17/01/08 16:25:40 WARN io.ReadaheadPool: Failed readahead on ifile
EBADF: Bad file descriptorat org.apache.hadoop.io.nativeio.NativeIO$POSIX.posix_fadvise(Native Method)at org.apache.hadoop.io.nativeio.NativeIO$POSIX.posixFadviseIfPossible(NativeIO.java:267)at org.apache.hadoop.io.nativeio.NativeIO$POSIX$CacheManipulator.posixFadviseIfPossible(NativeIO.java:146)at org.apache.hadoop.io.ReadaheadPool$ReadaheadRequestImpl.run(ReadaheadPool.java:206)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)at java.lang.Thread.run(Thread.java:745)
17/01/08 16:25:40 INFO mapred.Merger: Merging 2 sorted segments
17/01/08 16:25:40 INFO mapred.Merger: Down to the last merge-pass, with 2 segments left of total size: 156 bytes
17/01/08 16:25:40 INFO reduce.MergeManagerImpl: Merged 2 segments, 166 bytes to disk to satisfy reduce memory limit
17/01/08 16:25:40 INFO reduce.MergeManagerImpl: Merging 1 files, 168 bytes from disk
17/01/08 16:25:40 INFO reduce.MergeManagerImpl: Merging 0 segments, 0 bytes from memory into reduce
17/01/08 16:25:40 INFO mapred.Merger: Merging 1 sorted segments
17/01/08 16:25:40 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of total size: 160 bytes
17/01/08 16:25:40 INFO mapred.LocalJobRunner: 2 / 2 copied.
17/01/08 16:25:40 INFO Configuration.deprecation: mapred.skip.on is deprecated. Instead, use mapreduce.job.skiprecords
17/01/08 16:25:40 INFO mapred.Task: Task:attempt_local247232145_0001_r_000000_0 is done. And is in the process of committing
17/01/08 16:25:40 INFO mapred.LocalJobRunner: 2 / 2 copied.
17/01/08 16:25:40 INFO mapred.Task: Task attempt_local247232145_0001_r_000000_0 is allowed to commit now
17/01/08 16:25:40 INFO output.FileOutputCommitter: Saved output of task 'attempt_local247232145_0001_r_000000_0' to hdfs://localhost:9000/user/test/output2/_temporary/0/task_local247232145_0001_r_000000
17/01/08 16:25:40 INFO mapred.LocalJobRunner: reduce > reduce
17/01/08 16:25:40 INFO mapred.Task: Task 'attempt_local247232145_0001_r_000000_0' done.
17/01/08 16:25:40 INFO mapred.LocalJobRunner: Finishing task: attempt_local247232145_0001_r_000000_0
17/01/08 16:25:40 INFO mapred.LocalJobRunner: reduce task executor complete.
17/01/08 16:25:40 INFO mapreduce.Job:  map 100% reduce 100%
17/01/08 16:25:40 INFO mapreduce.Job: Job job_local247232145_0001 completed successfully
17/01/08 16:25:41 INFO mapreduce.Job: Counters: 35File System CountersFILE: Number of bytes read=889201FILE: Number of bytes written=1745401FILE: Number of read operations=0FILE: Number of large read operations=0FILE: Number of write operations=0HDFS: Number of bytes read=284HDFS: Number of bytes written=82HDFS: Number of read operations=22HDFS: Number of large read operations=0HDFS: Number of write operations=5Map-Reduce FrameworkMap input records=3Map output records=18Map output bytes=179Map output materialized bytes=174Input split bytes=224Combine input records=18Combine output records=14Reduce input groups=11Reduce shuffle bytes=174Reduce input records=14Reduce output records=11Spilled Records=28Shuffled Maps =2Failed Shuffles=0Merged Map outputs=2GC time elapsed (ms)=117Total committed heap usage (bytes)=457912320Shuffle ErrorsBAD_ID=0CONNECTION=0IO_ERROR=0WRONG_LENGTH=0WRONG_MAP=0WRONG_REDUCE=0File Input Format Counters Bytes Read=107File Output Format Counters Bytes Written=82

?

本文來自互聯網用戶投稿，該文觀點僅代表作者本人，不代表本站立場。本站僅提供信息存儲空間服務，不擁有所有權，不承擔相關法律責任。
如若轉載，請注明出處：http://www.pswp.cn/news/387768.shtml
繁體地址，請注明出處：http://hk.pswp.cn/news/387768.shtml
英文地址，請注明出處：http://en.pswp.cn/news/387768.shtml

如若內容造成侵權/違法違規/事實不符，請聯系多彩編程網進行投訴反饋email:809451989@qq.com，一經查實，立即刪除！

相關文章

iPhone手機屏幕尺寸（分辨率）

iPhone手機屏幕尺寸（分辨率）

第一代iPhone2G屏幕為3.5英吋&#xff0c;分辨率為320*480像素&#xff0c;比例為3:2。 第二代iPhone3G屏幕為3.5英吋&#xff0c;分辨率為320*480像素&#xff0c;比例為3:2。 第三代iPhone3GS屏幕為3.5英吋&#xff0c;分辨率為320*480像素&#xff0c;比例為3:2。 第四代iPh…
閱讀更多...
[Java in NetBeans] Lesson 06. Custom classes

[Java in NetBeans] Lesson 06. Custom classes

這個課程的參考視頻和圖片來自youtube。 主要學到的知識點有&#xff1a; Constructors: A special method called when an object of the class is createdproperty pattern and encapsulation(封裝): hide the implementation details from the user, so when the class is b…
閱讀更多...
UDP打洞NAT大致分為下面四類 P2P

UDP打洞NAT大致分為下面四類 P2P

NAT大致分為下面四類 1) Full Cone 這種NAT內部的機器A連接過外網機器C后,NAT會打開一個端口.然后外網的任何發到這個打開的端口的UDP數據報都可以到達A.不管是不是C發過來的. 例如 A:192.168.8.100 NAT:202.100.100.100 C:292.88.88.88 A(192.168.8.100:5000) -> NAT(202.1…
閱讀更多...
讓內核突破512字節的限制

讓內核突破512字節的限制

轉載于:https://www.cnblogs.com/ZHONGZHENHUA/p/10124237.html
閱讀更多...
高頻算法面試題（字符串） 242. 有效的字母異位詞

高頻算法面試題（字符串） 242. 有效的字母異位詞

leetcode 242. 有效的字母異位詞 給定兩個字符串 s 和 t &#xff0c;編寫一個函數來判斷 t 是否是 s 的一個字母異位詞。示例 1: 輸入: s "anagram", t "nagaram" 輸出: true 復制代碼示例 2: 輸入: s "rat", t "car" 輸出: fals…
閱讀更多...
struts2的漏洞

struts2的漏洞

文章前半部分來自團隊小伙伴阿德馬的總結&#xff0c;后半部分的Poc和Exp是小編匆忙之際借鑒而來&#xff0c;感謝寫Poc和Exp的伙伴~ 安恒給官方上報的&#xff0c;然后官方選擇了1個對國內來說比較敏感的時期發了公告出來&#xff0c;好蛋疼。 該漏洞的CVE編號是CVE-2017-56…
閱讀更多...
Java Statement PK PrepareStatement

Java Statement PK PrepareStatement

PreparedStatement是用來執行SQL查詢語句的API之一&#xff0c;Java提供了 Statement、PreparedStatement 和 CallableStatement三種方式來執行查詢語句&#xff0c;其中 Statement 用于通用查詢&#xff0c; PreparedStatement 用于執行參數化查詢&#xff0c;而 CallableStat…
閱讀更多...
mysql在linux 下安裝

mysql在linux 下安裝

安裝環境&#xff1a;系統是 centos6.5 1、下載 下載地址&#xff1a;http://dev.mysql.com/downloads/mysql/5.6.html#downloads 下載版本&#xff1a;我這里選擇的5.6.33&#xff0c;通用版&#xff0c;linux下64位 也可以直接復制64位的下載地址&#xff0c;通過命令下載&a…
閱讀更多...
Leetcode PHP題解--D47 868. Binary Gap

Leetcode PHP題解--D47 868. Binary Gap

2019獨角獸企業重金招聘Python工程師標準>>> D47 868. Binary Gap 題目鏈接 868. Binary Gap 題目分析 給定一個數字&#xff0c;計算其二進制表示中&#xff0c;出現的兩個1最大距離。 思路 當然是先轉換成二進制了。再進行遍歷。 當只有一個1時&#xff0c;返回0。…
閱讀更多...
[洛谷P5048][Ynoi2019模擬賽]Yuno loves sqrt technology III

[洛谷P5048][Ynoi2019模擬賽]Yuno loves sqrt technology III

題目大意&#xff1a;有$n(n\leqslant5\times10^5)$個數&#xff0c;$m(m\leqslant5\times10^5)$個詢問&#xff0c;每個詢問問區間$[l,r]$中眾數的出現次數 題解&#xff1a;分塊&#xff0c;設塊大小為$S$&#xff0c;先可以預處理出兩兩塊之間的眾數出現次數&#xff0c;復雜…
閱讀更多...
C#接口實現多態

C#接口實現多態

我比較喜歡對感興趣的理論進行反復的理解甚至理解背誦下來&#xff0c;接下來再復習一下什么叫多態&#xff08;哈哈哈&#xff09; 多態&#xff1a;在同一粒度視圖下對相同類型的事物不做區別的統一處理 接下來看一下接口和引擎類是如何實現多態的&#xff1a; 一、 1、創建了…
閱讀更多...
docker 網絡 不好用 docker: Error response from daemon: failed to create endpoint jovial_wing on network b

docker 網絡 不好用 docker: Error response from daemon: failed to create endpoint jovial_wing on network b

啟動容器時&#xff0c;有可能會遇到如下問題&#xff0c;比如啟動redis容器&#xff1a; sudo docker run -d -p 6379:6379 --name redis redis:latest Linux代碼docker: Error response from daemon: failed to create endpoint redis on network bridge: iptables failed: …
閱讀更多...
hadoop-hdfs-存儲模型-架構模型-角色介紹

hadoop-hdfs-存儲模型-架構模型-角色介紹

轉載于:https://www.cnblogs.com/LXL616/p/10803978.html
閱讀更多...
docker 鏡像 導入導出

docker 鏡像 導入導出

很喜歡玩docker&#xff0c;但最新遇到一個問題&#xff0c;公司給的新機器的dns有問題&#xff0c;導致pull不下來鏡像。 沒辦法了&#xff0c;沒有鏡像什么神馬都干不了&#xff0c;又不能花很多時間去搭建私有的鏡像庫&#xff0c;只有另尋辦法了。 廢話少說&#xff0c;經…
閱讀更多...
使用Nginx+uWSGI部署Django項目

使用Nginx+uWSGI部署Django項目

1.linux安裝python3環境 參考鏈接&#xff1a;https://www.cnblogs.com/zzqit/p/10087680.html 2.安裝uwsgi pip3 install uwsgiln -s /usr/local/python3/bin/uwsgi /usr/local/bin/uwsgi #建立軟鏈接uwsgi --version #檢查安裝成功 3.基于uwsgidjango項目部署 django項目目…
閱讀更多...
Nagios使用check_mysql_health插件監控Mysql主機

Nagios使用check_mysql_health插件監控Mysql主機

基本信息 Nagios&#xff1a;Nagios core 4.4.3Nagios Plugins&#xff1a;check_mysql_health 2.2.2Mysql-server: 192.168.0.91db user&#xff1a;db操作流程&#xff1a;下載插件->安裝插件->配置command->添加主機->添加服務安裝插件 下載 wget https://labs.…
閱讀更多...
lsof使用

lsof使用

簡介 lsof(list open files)是一個列出當前系統打開文件的工具。在linux環境下&#xff0c;任何事物都以文件的形式存在&#xff0c;通過文件不僅僅可以訪問常規數據&#xff0c;還可以訪問網絡連接和硬件。所以如傳輸控制協議 (TCP) 和用戶數據報協議 (UDP) 套接字等&#xf…
閱讀更多...
解題：2017清華集訓 無限之環

解題：2017清華集訓 無限之環

題面 費用流 把每種水管再拆出來四個方向的接頭&#xff0c;然后根據水管的形狀連出旋轉時的代價。最后黑白染色成二分圖&#xff0c;然后白點對應的接頭向黑點對應的接頭連邊&#xff0c;源點向白點自己連邊&#xff0c;黑點自己向匯點連邊。 怎么連邊&#xff1f;我是大力討論…
閱讀更多...
Node.js學習之（第二章：exports和module.exports）

Node.js學習之（第二章：exports和module.exports）

前言 Node中&#xff0c;每個模塊都有一個exports接口對象&#xff0c;我們需要把公共的方法或者字符串掛載在這個接口對象中&#xff0c;其他的模塊才可以使用。 Node.js中只有模塊作用域&#xff0c;默認兩個模塊之間的變量&#xff0c;方法互不沖突&#xff0c;互不影響&…
閱讀更多...
docker命令及掛載

docker命令及掛載

常用命令所有鏡像:docker images當前執行:docker ps提交保存docker容器: docker commit進入到對應服務:docker attach <container id>已經執行帶容器:docker ps -l根據名稱啟動通過8081端口察看docker容器里的8080:docker run -i -t -d -p 8081:8080 -p23:22 ubuntu:ubun…
閱讀更多...
最新文章

Copyright @ 2022~2023