java - How to Sort Reducer Output? -
i want sort output of reducer. sample of reducer output shown below:
0,0 2.5 0,1 3.0 1,0 4.0 1,1 1.5
the reducer output sorted first element of key. wanted sort second element of key output this:
0,0 2.5 1,0 4.0 0,1 3.0 1,1 1.5
any way can this?
please help!
this reducer:
import java.io.bufferedreader; import java.io.ioexception; import java.io.inputstreamreader; import java.util.hashmap; import org.apache.hadoop.fs.filesystem; import org.apache.hadoop.fs.path; import org.apache.hadoop.io.text; import org.apache.hadoop.mapreduce.reducer; public class recreduce extends reducer<text, text, text, text> { public static int n=0; @override public void setup(context context) throws ioexception, interruptedexception{ filesystem hdfs= filesystem.get(context.getconfiguration()); bufferedreader br = new bufferedreader(new inputstreamreader(hdfs.open(new path(context.getconfiguration().get("outfile"))))); string line=null; while((line=br.readline())!=null){ n=integer.parseint(line); break; } br.close(); hdfs.close(); } public void reduce(text key, iterable<text> values, context context) throws ioexception, interruptedexception { string[] value; hashmap<integer, float> hasha = new hashmap<integer, float>(); hashmap<integer, float> hashb = new hashmap<integer, float>(); (text val : values) { value = val.tostring().split(","); if (value[0].equals("a")) { for(int z=1;z<=n;z++){ hasha.put(z, float.parsefloat(value[z]));} } else{ for(int a=1;a<=n;a++){ hashb.put(a, float.parsefloat(value[a]));} } } float result = 0.0f; float a_ij; float b_jk; (int j=1;j<=n;j++) { a_ij = hasha.containskey(j) ? hasha.get(j) : 0.0f; b_jk = hashb.containskey(j) ? hashb.get(j) : 0.0f; result +=a_ij*b_jk; } context.write(null, new text(key.tostring() + "," + float.tostring(result))); } }
you can use composite key , composite key comparator
create class e.g.
class pair(){ string key string value; }
and use in reducer output this
context.write(new pair(key.tostring(), float.tostring(result)), null);
then create comparator
public class paircomparator extends writablecomparator { protected paircomparator() { super(pair.class, true); } @override public int compare(writablecomparable w1, writablecomparable w2) { pair k1 = (pair)w1; pair k2 = (pair)w2; return k1.getvalue().compareto(k2.getvalue()); } }
and use comparator in job definition
job.setsortcomparatorclass(paircomparator.class);
i didn't check code above. idea.
i hope
Comments
Post a Comment