设为首页 加入收藏

TOP

Hadoop实现共同出现的单词(Word co-occurrence)(一)
2014-11-24 03:08:19 来源: 作者: 【 】 浏览:1
Tags:Hadoop 实现 共同 出现 单词 Word co-occurrence

Sample Input:


a b cc, c d d c
I Love U.
dd ee f g s sa dew ad da
So shaken as we are, so wan with care.
Find we a time for frighted peace to pant.
And breathe short-winded accents of new broil.
To be commenced in strands afar remote.
I Love U U love i.
i i i i


Sample Output:


a:b 1
a:time 1
a:we 1
accents:of 1
accents:short-winded 1
ad:da 1
ad:dew 1
afar:remote 1
afar:strands 1
and:breathe 1
are:so 1
are:we 1
as:shaken 1
as:we 1
b:cc 1
be:commenced 1
be:to 1
breathe:short-winded 1
broil:new 1
c:cc 1
c:d 2
care:with 1
commenced:in 1
d:d 1
dd:ee 1
dew:sa 1
ee:f 1
f:g 1
find:we 1
for:frighted 1
for:time 1
frighted:peace 1
g:s 1
i:i 3
i:love 3
in:strands 1
love:u 3
new:of 1
pant:to 1
peace:to 1
s:sa 1
shaken:so 1
so:wan 1
u:u 1
wan:with 1


Code:


import java.io.DataInput;
import java.io.DataOutput;
import java.io.IOException;
import java.util.StringTokenizer;


import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.RawComparator;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.io.WritableComparable;
import org.apache.hadoop.io.WritableComparator;
import org.apache.hadoop.io.WritableUtils;
import org.apache.hadoop.mapred.Reporter;
import org.apache.hadoop.mapreduce.lib.input.FileInputFormat;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.Mapper;
import org.apache.hadoop.mapreduce.Partitioner;
import org.apache.hadoop.mapreduce.Reducer;
import org.apache.hadoop.util.GenericOptionsParser;


public class CoOccurrence {



public static class TextPair implements WritableComparable {
private Text first;
private Text second;

public TextPair(){
set(new Text(), new Text());
}
public TextPair(String left, String right) {
set(new Text(left), new Text(right));
}
public TextPair(Text left, Text right) {
set(left, right);
}

public void set(Text left, Text right){
String l = left.toString();
String r = right.toString();
int cmp = l.compareTo(r);
if(cmp <= 0){
this.first = left;
this.second = right;
}else{
this.first = right;
this.second = left;
}
}

public Text getFirst() {
return first;
}
public Text getSecond() {
return second;
}


@Override
public void readFields(DataInput in) throws IOException {
first.readFields(in);
second.readFields(in);
}
@Override
public void write(DataOutput out) throws IOException {
first.write(out);
second.write(out);
}
@Override
public int hashCode() {
return first.hashCode() * 163 + second.hashCode();//May be some trouble here. why 163 sometimes 157
}
@Override
public boolean equals(Object o) {
if (o instanceof TextPair) {
TextPair tp = (TextPair) o;
return first.equals(tp.first) && second.equals(tp.second);
}
return false;
}
@Override
public String toString(){
return first + ":" + second;
}
@Override
public int compareTo(TextPair tp) {
int cmp = first.compareTo(tp.first);
if(cmp != 0)
return cmp;
return second.compareTo(tp.second);
}


// A Comparator that com.pares serialized StringPair.
public static class Comparator extends WritableComparator {
private static final Text.Comparator TEXT_COMPARATOR = new Text.Comparator();
public Comparator() {

首页 上一页 1 2 3 下一页 尾页 1/3/3
】【打印繁体】【投稿】【收藏】 【推荐】【举报】【评论】 【关闭】 【返回顶部
分享到: 
上一篇Linux 下smi/mdio总线通信 下一篇Android开发技巧:动态创建UI界面

评论

帐  号: 密码: (新用户注册)
验 证 码:
表  情:
内  容:

·Announcing October (2025-12-24 15:18:16)
·MySQL有什么推荐的学 (2025-12-24 15:18:13)
·到底应该用MySQL还是 (2025-12-24 15:18:11)
·进入Linux世界大门的 (2025-12-24 14:51:47)
·Download Linux | Li (2025-12-24 14:51:44)