pignlproc.markup

Class AnnotatingMarkupParser

java.lang.Object

pignlproc.markup.AnnotatingMarkupParser

All Implemented Interfaces:

info.bliki.wiki.filter.ITextConverter

public class AnnotatingMarkupParser
extends Object
implements info.bliki.wiki.filter.ITextConverter

Parse mediawiki markup to strip the formatting info and extract a simple text version suitable for NLP along with header, paragraph and link position annotations. Use the #convert(String) and #getWikiLinks() methods. Due to the constraints imposed by the ITextConverter / WikiModel API, this class is not thread safe: only one instance should be run by thread.

Nested Class Summary

Nested Classes

Modifier and Type	Class and Description
class	AnnotatingMarkupParser.CountingAppendable

Field Summary

Fields

Modifier and Type	Field and Description
protected List<Annotation>	headers
static Set<String>	HEADING_TAGS
static String	HREF_ATTR_KEY
static Pattern	INTERWIKI_PATTERN
protected String	languageCode
protected info.bliki.wiki.model.WikiModel	model
static Set<String>	PARAGRAPH_TAGS
protected List<Annotation>	paragraphs
static int	RECURSION_LIMIT
protected String	redirect
protected static Map<String,Pattern>	REDIRECT_PATTERNS
protected Pattern	redirectPattern
protected String	text
static String	WIKILINK_TARGET_ATTR_KEY
static String	WIKILINK_TITLE_ATTR_KEY
protected List<Annotation>	wikilinks
static String	WIKIOBJECT_ATTR_KEY

Constructor Summary

Constructors

Constructor and Description
AnnotatingMarkupParser()
AnnotatingMarkupParser(String languageCode)

Method Summary

Methods

Modifier and Type	Method and Description
List<Annotation>	getHeaderAnnotations()
List<String>	getHeaders()
List<Annotation>	getParagraphAnnotations()
List<String>	getParagraphs()
String	getRedirect()
List<Annotation>	getWikiLinkAnnotations()
void	imageNodeToText(info.bliki.htmlcleaner.TagNode tagNode, info.bliki.wiki.model.ImageFormat imageFormat, Appendable buffer, info.bliki.wiki.model.IWikiModel model)
info.bliki.wiki.model.WikiModel	makeWikiModel(String languageCode)
void	nodesToText(List<? extends Object> nodes, Appendable buffer, info.bliki.wiki.model.IWikiModel model)
boolean	noLinks()
String	parse(String rawWikiMarkup) Convert WikiMarkup to a simple text representation suitable for NLP analysis.
static String	titleToUri(String title, String languageCode)

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

HREF_ATTR_KEY

public static final String HREF_ATTR_KEY

See Also:

Constant Field Values

WIKILINK_TITLE_ATTR_KEY

public static final String WIKILINK_TITLE_ATTR_KEY

See Also:

Constant Field Values

WIKILINK_TARGET_ATTR_KEY

public static final String WIKILINK_TARGET_ATTR_KEY

See Also:

Constant Field Values

WIKIOBJECT_ATTR_KEY

public static final String WIKIOBJECT_ATTR_KEY

See Also:

Constant Field Values

RECURSION_LIMIT

public static final int RECURSION_LIMIT

See Also:

Constant Field Values

PARAGRAPH_TAGS

public static final Set<String> PARAGRAPH_TAGS

HEADING_TAGS

public static final Set<String> HEADING_TAGS

INTERWIKI_PATTERN

public static final Pattern INTERWIKI_PATTERN

wikilinks

protected final List<Annotation> wikilinks

headers

protected final List<Annotation> headers

paragraphs

protected final List<Annotation> paragraphs

languageCode

protected String languageCode

model

protected final info.bliki.wiki.model.WikiModel model

redirect

protected String redirect

text

protected String text

REDIRECT_PATTERNS

protected static final Map<String,Pattern> REDIRECT_PATTERNS

redirectPattern

protected Pattern redirectPattern

Constructor Detail

AnnotatingMarkupParser

public AnnotatingMarkupParser()

AnnotatingMarkupParser

public AnnotatingMarkupParser(String languageCode)

Method Detail

makeWikiModel

public info.bliki.wiki.model.WikiModel makeWikiModel(String languageCode)

parse

public String parse(String rawWikiMarkup)

Convert WikiMarkup to a simple text representation suitable for NLP analysis. The links encountered during the extraction are then available by calling #getWikiLinks().

Parameters:

rawWikiMarkup -

Returns:

the simple text without the markup

titleToUri

public static String titleToUri(String title,
                String languageCode)

nodesToText

public void nodesToText(List<? extends Object> nodes,
               Appendable buffer,
               info.bliki.wiki.model.IWikiModel model)
                 throws IOException

Specified by:

nodesToText in interface info.bliki.wiki.filter.ITextConverter

Throws:

IOException

imageNodeToText

public void imageNodeToText(info.bliki.htmlcleaner.TagNode tagNode,
                   info.bliki.wiki.model.ImageFormat imageFormat,
                   Appendable buffer,
                   info.bliki.wiki.model.IWikiModel model)
                     throws IOException

Specified by:

imageNodeToText in interface info.bliki.wiki.filter.ITextConverter

Throws:

IOException

noLinks

public boolean noLinks()

Specified by:

noLinks in interface info.bliki.wiki.filter.ITextConverter

getWikiLinkAnnotations

public List<Annotation> getWikiLinkAnnotations()

getHeaderAnnotations

public List<Annotation> getHeaderAnnotations()

getParagraphAnnotations

public List<Annotation> getParagraphAnnotations()

getParagraphs

public List<String> getParagraphs()

getHeaders

public List<String> getHeaders()

getRedirect

public String getRedirect()