Tuesday, November 16, 2010

How to read a file from HDFS in Hadoop API classes in Java

The following java program is an example of how you can programmtically read a file in HDFS using HDFS API's bundled with Hadoop
1. Open File Cat.java and paste the following code

package org.myorg;
import java.io.*;
import java.util.*;
import java.net.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.conf.*;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.*;

public class Cat{
public static void main (String [] args) throws Exception{
Path pt=new Path("hdfs://jp.seka.com:9000/user/john/abc.txt");
FileSystem fs = FileSystem.get(new Configuration());
BufferedReader br=new BufferedReader(new InputStreamReader(fs.open(pt)));
String line;
while (line != null){
}catch(Exception e){

2. Compile the code

javac -classpath hadoop-0.20.1-dev-core.jar -d Cat/ Cat.java

3. Create jar

jar -cvf Cat.jar -C Cat/ .

4. Run

hadoop jar Cat.jar org.myorg.Cat

No comments:

Post a Comment