设为首页 加入收藏

TOP

Hadoop 文件操作和代码运行测试(一)
2014-11-24 02:50:47 来源: 作者: 【 】 浏览:3
Tags:Hadoop 文件 操作 代码 运行 测试

[linuxidc@hadoop02 hadoop-1.1.2]$ ./bin/hadoop fs -mkdir TEST
[linuxidc@hadoop02 hadoop-1.1.2]$ ./bin/hadoop fs -lsr
drwxr-xr-x - linuxidc supergroup 0 2013-09-10 14:08 /user/linuxidc/TEST
3. 上传文件到HDFS目录中
本地建立一个文件,内容如下:
[linuxidc@hadoop02 hadoop-1.1.2]$ touch test.txt
[linuxidc@hadoop02 hadoop-1.1.2]$ vim test.txt
[linuxidc@hadoop02 hadoop-1.1.2]$ cat test.txt
Hello, Hadoop !
你好, Hadoop !
上传到HDFS的TEST目录中,可以使用-copyFromLocal参数,也可以使用-moveFromLocal,分别是从本地拷贝到和从本地剪切到HDFS目录。
[linuxidc@hadoop02 hadoop-1.1.2]$ ./bin/hadoop fs -moveFromLocal test.txt TEST
[linuxidc@hadoop02 hadoop-1.1.2]$ ./bin/hadoop fs -ls
Found 1 items
drwxr-xr-x - linuxidc supergroup 0 2013-09-10 14:14 /user/linuxidc/TEST
[linuxidc@hadoop02 hadoop-1.1.2]$
4. 查看HDFS文件内容
[linuxidc@hadoop02 hadoop-1.1.2]$ ./bin/hadoop fs -cat TEST/test.txt
Hello, Hadoop !
你好, Hadoop !
[linuxidc@hadoop02 hadoop-1.1.2]$
5. 运行jar范例,统计文本单词词频
[linuxidc@hadoop02 hadoop-1.1.2]$ ./bin/hadoop jar hadoop-examples-1.1.2.jar
wordcount TEST out
13/09/10 14:20:50 INFO input.FileInputFormat: Total input paths to process : 1
13/09/10 14:20:50 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/09/10 14:20:50 WARN snappy.LoadSnappy: Snappy native library not loaded
13/09/10 14:20:51 INFO mapred.JobClient: Running job: job_201309082325_0001
13/09/10 14:20:52 INFO mapred.JobClient: map 0% reduce 0%
13/09/10 14:21:03 INFO mapred.JobClient: map 100% reduce 0%
13/09/10 14:21:11 INFO mapred.JobClient: map 100% reduce 33%
13/09/10 14:21:13 INFO mapred.JobClient: map 100% reduce 100%
13/09/10 14:21:14 INFO mapred.JobClient: Job complete: job_201309082325_0001
13/09/10 14:21:14 INFO mapred.JobClient: Counters: 29
13/09/10 14:21:14 INFO mapred.JobClient: Job Counters
13/09/10 14:21:14 INFO mapred.JobClient: Launched reduce tasks=1
13/09/10 14:21:14 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=10619
13/09/10 14:21:14 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0
13/09/10 14:21:14 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0
13/09/10 14:21:14 INFO mapred.JobClient: Launched map tasks=1
13/09/10 14:21:14 INFO mapred.JobClient: Data-local map tasks=1
13/09/10 14:21:14 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=9864
13/09/10 14:21:14 INFO mapred.JobClient: File Output Format Counters
13/09/10 14:21:14 INFO mapred.JobClient: Bytes Written=38
13/09/10 14:21:14 INFO mapred.JobClient: FileSystemCounters
13/09/10 14:21:14 INFO mapred.JobClient: FILE_BYTES_READ=64
13/09/10 14:21:14 INFO mapred.JobClient: HDFS_BYTES_READ=146
13/09/10 14:21:14 INFO mapred.JobClient: FILE_BYTES_WRITTEN=109756
13/09/10 14:21:14 INFO mapred.JobClient: HDFS_BYTES_WRITTEN=38
13/09/10 14:21:14 INFO mapred.JobClient: File Input Format Counters
13/09/10 14:21:14 INFO mapred.JobClient: Bytes Read=35
13/09/10 14:21:14 INFO mapred.JobClient: Map-Reduce Framework
13/09/10 14:21:14 INFO mapred.JobClient: Map output materialized bytes=64
13/09/10 14:21:14 INFO mapred.JobClient: Map input records=2
13/09/10 14:21:14 INFO mapred.JobClient: Reduce shuffle bytes=64
13/09/10 14:21:14 INFO mapred.JobClient: Spilled Records=10
13/09/10 14:21:14 INFO mapred.JobClient: Map output bytes=59
13/09/10 14:21:14 INFO mapred.JobClient: Total committed heap usage (bytes)=189464576
13/09/10 14:21:14 INFO mapred.JobClient: CPU time spent

首页 上一页 1 2 下一页 尾页 1/2/2
】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
分享到: 
上一篇HTML5移动Web开发笔记 下一篇Bash强制脚本使用某用户执行

评论

帐  号: 密码: (新用户注册)
验 证 码:
表  情:
内  容: