Parser::Tokenizer Class Reference
[Dependency Parser]

Read a plain text file, split into tokens. More...

#include <Tokenizer.h>

Inheritance diagram for Parser::Tokenizer:

List of all members.

Public Member Functions
	Tokenizer (std::istream *is, char const *lang=0)
	Creates a new Tokenizer.
Scanner::Token const *	Current ()
	Returns the current token.
bool	MoveNext ()
	Advance to the next token and return true if there is one available.
void	Reset ()
	Restart.

---

Detailed Description

Read a plain text file, split into tokens.

Provides an Enumerator interface, since it needs to look at next token to determine if there are more. An Iterator interface would be more cumbersome to implement.

---

Constructor & Destructor Documentation

Parser::Tokenizer::Tokenizer	(	std::istream *	is,
		char const *	lang = 0
	)

Creates a new Tokenizer.

Parameters:

is

the stream containing the sentence to read.

---

The documentation for this class was generated from the following files:

• parse/desr/src/Tokenizer.h
• parse/desr/src/Tokenizer.cpp