org.apache.cassandra.hadoop.cql3

Class CqlPagingInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,Y>

org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat<java.util.Map<java.lang.String,java.nio.ByteBuffer>,java.util.Map<java.lang.String,java.nio.ByteBuffer>>

org.apache.cassandra.hadoop.cql3.CqlPagingInputFormat

All Implemented Interfaces:

org.apache.hadoop.mapred.InputFormat<java.util.Map<java.lang.String,java.nio.ByteBuffer>,java.util.Map<java.lang.String,java.nio.ByteBuffer>>

public class CqlPagingInputFormat
extends AbstractColumnFamilyInputFormat<java.util.Map<java.lang.String,java.nio.ByteBuffer>,java.util.Map<java.lang.String,java.nio.ByteBuffer>>

Hadoop InputFormat allowing map/reduce against Cassandra rows within one ColumnFamily. At minimum, you need to set the KS and CF in your Hadoop job Configuration. The ConfigHelper class is provided to make this simple: ConfigHelper.setInputColumnFamily You can also configure the number of rows per InputSplit with ConfigHelper.setInputSplitSize. The default split size is 64k rows. the number of CQL rows per page the number of CQL rows per page CQLConfigHelper.setInputCQLPageRowSize. The default page row size is 1000. You should set it to "as big as possible, but no bigger." It set the LIMIT for the CQL query, so you need set it big enough to minimize the network overhead, and also not too big to avoid out of memory issue. the column names of the select CQL query. The default is all columns CQLConfigHelper.setInputColumns the user defined the where clause CQLConfigHelper.setInputWhereClauses. The default is no user defined where clause

Field Summary

Fields inherited from class org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat

CASSANDRA_HADOOP_MAX_KEY_SIZE, CASSANDRA_HADOOP_MAX_KEY_SIZE_DEFAULT, MAPRED_TASK_ID

Constructor Summary

Constructors

Constructor and Description
CqlPagingInputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.RecordReader<java.util.Map<java.lang.String,java.nio.ByteBuffer>,java.util.Map<java.lang.String,java.nio.ByteBuffer>>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit arg0, org.apache.hadoop.mapreduce.TaskAttemptContext arg1)
org.apache.hadoop.mapred.RecordReader<java.util.Map<java.lang.String,java.nio.ByteBuffer>,java.util.Map<java.lang.String,java.nio.ByteBuffer>>	getRecordReader(org.apache.hadoop.mapred.InputSplit split, org.apache.hadoop.mapred.JobConf jobConf, org.apache.hadoop.mapred.Reporter reporter)

Methods inherited from class org.apache.cassandra.hadoop.AbstractColumnFamilyInputFormat

createAuthenticatedClient, getSplits, getSplits, validateConfiguration

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

CqlPagingInputFormat

public CqlPagingInputFormat()

Method Detail

getRecordReader

public org.apache.hadoop.mapred.RecordReader<java.util.Map<java.lang.String,java.nio.ByteBuffer>,java.util.Map<java.lang.String,java.nio.ByteBuffer>> getRecordReader(org.apache.hadoop.mapred.InputSplit split,
                                                                                                                                                             org.apache.hadoop.mapred.JobConf jobConf,
                                                                                                                                                             org.apache.hadoop.mapred.Reporter reporter)
                                                                                                                                                               throws java.io.IOException

Throws:

java.io.IOException

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<java.util.Map<java.lang.String,java.nio.ByteBuffer>,java.util.Map<java.lang.String,java.nio.ByteBuffer>> createRecordReader(org.apache.hadoop.mapreduce.InputSplit arg0,
                                                                                                                                                                   org.apache.hadoop.mapreduce.TaskAttemptContext arg1)
                                                                                                                                                                     throws java.io.IOException,
                                                                                                                                                                            java.lang.InterruptedException

Specified by:

createRecordReader in class org.apache.hadoop.mapreduce.InputFormat<java.util.Map<java.lang.String,java.nio.ByteBuffer>,java.util.Map<java.lang.String,java.nio.ByteBuffer>>

Throws:

java.io.IOException

java.lang.InterruptedException