org.hedera.io.etl

Class IntervalRevisionETLReader<KEYIN,VALUEIN>

java.lang.Object

org.apache.hadoop.mapreduce.RecordReader<KEYIN,VALUEIN>

org.hedera.io.etl.RevisionETLReader<KEYIN,VALUEIN,RevisionHeader>

org.hedera.io.etl.DefaultRevisionETLReader<KEYIN,VALUEIN>

org.hedera.io.etl.IntervalRevisionETLReader<KEYIN,VALUEIN>

All Implemented Interfaces:

Closeable, AutoCloseable

Direct Known Subclasses:

RevisionBOWInputFormat.RevisionBOWReader, RevisionConcatInputFormat.IntervalRevisionConcatTextReader

public abstract class IntervalRevisionETLReader<KEYIN,VALUEIN>
extends DefaultRevisionETLReader<KEYIN,VALUEIN>

A WikiRevsionETLReader that skips all revisions out of a specific range

Author:

tuan

Nested Class Summary

Nested classes/interfaces inherited from class org.hedera.io.etl.RevisionETLReader

RevisionETLReader.Ack

Field Summary

Fields

Modifier and Type	Field and Description
static String	DAY_SCALE_OPT
static String	END_TIME_OPT
static String	HOUR_SCALE_OPT
static String	MONTH_SCALE_OPT
static String	SCALE_OPT
static String	START_TIME_OPT
static String	WEEK_SCALE_OPT

Fields inherited from class org.hedera.io.etl.DefaultRevisionETLReader

skipNonArticles, skipRedirect

Fields inherited from class org.hedera.io.etl.RevisionETLReader

DEFAULT_MAX_BLOCK_SIZE, extractor

Constructor Summary

Constructors

Constructor and Description
IntervalRevisionETLReader()

Method Summary

Methods

Modifier and Type	Method and Description
void	initialize(org.apache.hadoop.mapreduce.InputSplit input, org.apache.hadoop.mapreduce.TaskAttemptContext tac)
protected void	processMetaData(org.apache.hadoop.io.DataOutputBuffer buffer, RevisionHeader meta) This method processes after caching the currently visited revision.
protected RevisionETLReader.Ack	readToNextRevision(org.apache.hadoop.io.DataOutputBuffer buffer, RevisionHeader meta) This method reads bytes inside the input stream into the buffer until reaching EOF or the revision close tag.

Methods inherited from class org.hedera.io.etl.DefaultRevisionETLReader

initializeMeta, readToPageHeader

Methods inherited from class org.hedera.io.etl.RevisionETLReader

clearRevisions, close, fetchMore, freeKey, freeValue, getCurrentKey, getCurrentValue, getProgress, getTaskAttemptContext, hasData, initializeExtractor, initializeKey, initializeValue, nextByte, nextKeyValue, setBlockSize, updateRevision

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

START_TIME_OPT

public static final String START_TIME_OPT

See Also:

Constant Field Values

END_TIME_OPT

public static final String END_TIME_OPT

See Also:

Constant Field Values

SCALE_OPT

public static final String SCALE_OPT

See Also:

Constant Field Values

HOUR_SCALE_OPT

public static final String HOUR_SCALE_OPT

See Also:

Constant Field Values

DAY_SCALE_OPT

public static final String DAY_SCALE_OPT

See Also:

Constant Field Values

WEEK_SCALE_OPT

public static final String WEEK_SCALE_OPT

See Also:

Constant Field Values

MONTH_SCALE_OPT

public static final String MONTH_SCALE_OPT

See Also:

Constant Field Values

Constructor Detail

IntervalRevisionETLReader

public IntervalRevisionETLReader()

Method Detail

initialize

public void initialize(org.apache.hadoop.mapreduce.InputSplit input,
              org.apache.hadoop.mapreduce.TaskAttemptContext tac)
                throws IOException,
                       InterruptedException

Overrides:

initialize in class DefaultRevisionETLReader<KEYIN,VALUEIN>

Throws:

IOException

InterruptedException

readToNextRevision

protected RevisionETLReader.Ack readToNextRevision(org.apache.hadoop.io.DataOutputBuffer buffer,
                                       RevisionHeader meta)
                                            throws IOException

Description copied from class: RevisionETLReader

This method reads bytes inside the input stream into the buffer until reaching EOF or the revision close tag. In case of success, it extracts the meta-data into the meta form

Specified by:

readToNextRevision in class RevisionETLReader<KEYIN,VALUEIN,RevisionHeader>

Throws:

IOException

processMetaData

protected void processMetaData(org.apache.hadoop.io.DataOutputBuffer buffer,
                   RevisionHeader meta)

This method processes after caching the currently visited revision. It performs the meta-data quick updates before the actual extraction (in WikiRevisionETLReader's code)

Parameters:

buffer -

meta -