srchilite::WordTokenizer Class Reference

Tokenizes a paragraph separating words from spaces. More...

#include <wordtokenizer.h>

Public Types
typedef std::list< std::pair< std::string, std::string > >	WordTokenizerResults
	Results of the tokenizer; each element is a pair where the first string represents a possible space and the second string a possible word.

Static Public Member Functions
static void	tokenize (const std::string &s, WordTokenizerResults &results)
	Tokenizes the passed string and stores the results.

Detailed Description

Tokenizes a paragraph separating words from spaces.

Member Typedef Documentation

WordTokenizerResults

typedef std::list<std::pair<std::string, std::string> > srchilite::WordTokenizer::WordTokenizerResults

Results of the tokenizer; each element is a pair where the first string represents a possible space and the second string a possible word.

The two elements are mutually exclusive

Member Function Documentation

tokenize()

void srchilite::WordTokenizer::tokenize ( const std::string & s, WordTokenizerResults & results )

static

Tokenizes the passed string and stores the results.

Parameters

s	the string to tokenize
results	where to store the results

---

The documentation for this class was generated from the following files:

• /cygdrive/d/a/scallywag/source-highlight/source-highlight-3.1.9-1.x86_64/src/source-highlight-3.1.9/lib/srchilite/wordtokenizer.h
• /cygdrive/d/a/scallywag/source-highlight/source-highlight-3.1.9-1.x86_64/src/source-highlight-3.1.9/lib/srchilite/wordtokenizer.cpp