org.hedera.io.etl

Class RevisionLinkInputFormat

java.lang.Object

org.apache.hadoop.mapreduce.InputFormat<K,V>

org.apache.hadoop.mapreduce.lib.input.FileInputFormat<KEYIN,VALUEIN>

org.hedera.io.input.WikiRevisionInputFormat<org.apache.hadoop.io.LongWritable,LinkProfile>

org.hedera.io.etl.RevisionLinkInputFormat

public class RevisionLinkInputFormat
extends WikiRevisionInputFormat<org.apache.hadoop.io.LongWritable,LinkProfile>

The input format that supports ETL reading and extract link structures from each revision on the go

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
static class	RevisionLinkInputFormat.LinkExtractor
static class	RevisionLinkInputFormat.RevisionLinkReader

Field Summary

Fields inherited from class org.hedera.io.input.WikiRevisionInputFormat

compressionCodecs, END_ID, END_NAMESPACE, END_PAGE, END_PAGE_TAG, END_PARENT_ID, END_REDIRECT, END_REVISION, END_TEXT, END_TIMESTAMP, END_TIMESTAMP_TAG, END_TITLE, KEY_SKIP_FACTOR, LOG, MINOR_TAG, REVISION_BEGIN_TIME, REVISION_END_TIME, SEED_FILE, SKIP_NON_ARTICLES, SKIP_REDIRECT, SPLIT_INDEX_OPTION, SPLIT_MAPFILE_LOC, START_ID, START_NAMESPACE, START_PAGE, START_PAGE_TAG, START_PARENT_ID, START_REDIRECT, START_REVISION, START_TEXT, START_TIMESTAMP, START_TIMESTAMP_TAG, START_TITLE, THRESHOLD, TIME_FORMAT

Fields inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

INPUT_DIR, NUM_INPUT_FILES, PATHFILTER_CLASS, SPLIT_MAXSIZE, SPLIT_MINSIZE

Constructor Summary

Constructors

Constructor and Description
RevisionLinkInputFormat()

Method Summary

Methods

Modifier and Type	Method and Description
org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.LongWritable,LinkProfile>	createRecordReader(org.apache.hadoop.mapreduce.InputSplit input, org.apache.hadoop.mapreduce.TaskAttemptContext context)

Methods inherited from class org.hedera.io.input.WikiRevisionInputFormat

configure, getSplits, getSplits, isSplitable, makeSplit

Methods inherited from class org.apache.hadoop.mapreduce.lib.input.FileInputFormat

addInputPath, addInputPaths, computeSplitSize, getBlockIndex, getFormatMinSplitSize, getInputPathFilter, getInputPaths, getMaxSplitSize, getMinSplitSize, listStatus, setInputPathFilter, setInputPaths, setInputPaths, setMaxInputSplitSize, setMinInputSplitSize

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RevisionLinkInputFormat

public RevisionLinkInputFormat()

Method Detail

createRecordReader

public org.apache.hadoop.mapreduce.RecordReader<org.apache.hadoop.io.LongWritable,LinkProfile> createRecordReader(org.apache.hadoop.mapreduce.InputSplit input,
                                                                                                         org.apache.hadoop.mapreduce.TaskAttemptContext context)
                                                                                                           throws IOException,
                                                                                                                  InterruptedException

Specified by:

createRecordReader in class WikiRevisionInputFormat<org.apache.hadoop.io.LongWritable,LinkProfile>

Throws:

IOException

InterruptedException