dictionary Class Reference

#include <dictionary.h>

Collaboration diagram for dictionary:

List of all members.

Public Member Functions

int dub ()

int dub (int value)

const char * OOV ()

const char * BoS ()

const char * EoS ()

int oovcode (int v=-1)

int incflag ()

int incflag (int v)

int getword (fstream &inp, char *buffer)

int isprintable (char *w)

void genoovcode ()

int setoovrate (double oovrate)

long long incfreq (int code, long long value)

long long multfreq (int code, double value)

long freq (int code, long long value=-1)

long long totfreq ()

void grow ()

void sort ()

dictionary (char *filename, int size=DICT_INITSIZE)

dictionary (dictionary *d, bool sortflag=true)

~dictionary ()

void generate (char *filename)

void load (char *filename)

void save (char *filename, int freqflag=0)

void load (std::istream &fd)

void save (std::ostream &fd)

int size ()

int getcode (const char *w)

int encode (const char *w)

const char * decode (int c)

void stat ()

void print_curve (int curvesize, float *testOOV=NULL)

float * test (int curvesize, const char *filename, int listflag=0)

void cleanfreq ()

Public Attributes

dictionary * oovlex

Friends

class dictionary_iter

Detailed Description

Definition at line 67 of file dictionary.h.

Constructor & Destructor Documentation

dictionary::dictionary	(	char *	filename,
		int	size = `DICT_INITSIZE`
	)

Definition at line 36 of file dictionary.cpp.

References mfstream::close(), freq(), generate(), load(), LOAD_FACTOR, and NULL.

Here is the call graph for this function:

dictionary::dictionary	(	dictionary *	d,
		bool	sortflag = `true`
	)

Definition at line 343 of file dictionary.cpp.

References cmpdictentry(), dict_entry::code, dict_entry::freq, HT_ENTER, LOAD_FACTOR, strstack::push(), htable::search(), and dict_entry::word.

Here is the call graph for this function:

dictionary::~dictionary ( )

Definition at line 404 of file dictionary.cpp.

Member Function Documentation

const char* dictionary::BoS ( ) [inline]

Definition at line 89 of file dictionary.h.

References BOS_.

Referenced by interplm::gencorrcounts(), ngramtable::generate(), interplm::interplm(), main(), test(), interplm::test_ngt(), interplm::test_txt(), mshiftbeta::train(), and shiftbeta::train().

Here is the caller graph for this function:

void dictionary::cleanfreq ( ) [inline]

Definition at line 158 of file dictionary.h.

References dict_entry::freq.

Referenced by main().

Here is the caller graph for this function:

const char * dictionary::decode ( int c )

Definition at line 511 of file dictionary.cpp.

References OOV(), and dict_entry::word.

Referenced by ngramtable::augment(), lmtable::cpsublm(), lmmacro::cutLex(), mshiftbeta::discount(), ngramtable::generate(), interplm::interplm(), lmmacro::loadLexicalClasses(), ngramtable::loadtxt(), main(), lmmacro::Micro2MacroMapping(), lmmacro::One2OneMapping(), operator<<(), mdiadaptlm::saveARPA(), mdiadaptlm::saveASR(), and ngram::trans().

Here is the call graph for this function:

Here is the caller graph for this function:

int dictionary::dub ( int value ) [inline]

Definition at line 86 of file dictionary.h.

int dictionary::dub ( ) [inline]

Definition at line 85 of file dictionary.h.

Referenced by mixture::discount(), mixture::dub(), interplm::dub(), interplm::test_ngt(), interplm::test_txt(), mixture::train(), interplm::trainunigr(), and mdiadaptlm::txclprob().

Here is the caller graph for this function:

int dictionary::encode ( const char * w )

Definition at line 476 of file dictionary.cpp.

References dict_entry::code, dict_entry::freq, grow(), HT_ENTER, HT_FIND, NULL, OOV(), strstack::push(), htable::search(), and dict_entry::word.

Referenced by ngramtable::augment(), ngram::containsWord(), interplm::gencorrcounts(), ngramtable::generate(), generate(), genoovcode(), Moses::LanguageModelIRST::GetLmID(), interplm::interplm(), lmmacro::loadLexicalClasses(), ngramtable::loadtxt(), main(), lmmacro::Micro2MacroMapping(), mixture::mixture(), ngramtable::ngramtable(), operator>>(), ngram::pushw(), lmtable::reload(), mdiadaptlm::saveMT(), setoovrate(), interplm::test_ngt(), interplm::test_txt(), mixture::train(), interplm::trainunigr(), and ngram::trans().

Here is the call graph for this function:

Here is the caller graph for this function:

const char* dictionary::EoS ( ) [inline]

Definition at line 90 of file dictionary.h.

References EOS_.

Referenced by main(), mdiadaptlm::saveARPA(), mshiftbeta::train(), and shiftbeta::train().

Here is the caller graph for this function:

long dictionary::freq	(	int	code,
		long long	value = `-1`
	)			`[inline]`

Definition at line 126 of file dictionary.h.

References dict_entry::freq.

Referenced by ngramtable::augment(), dictionary(), grow(), interplm::interplm(), main(), multfreq(), print_curve(), save(), mdiadaptlm::saveMT(), setoovrate(), test(), interplm::trainunigr(), and interplm::unigr().

Here is the caller graph for this function:

void dictionary::generate ( char * filename )

Definition at line 107 of file dictionary.cpp.

References mfstream::close(), encode(), getword(), incfreq(), and MAX_WORD.

Referenced by dictionary().

Here is the call graph for this function:

Here is the caller graph for this function:

void dictionary::genoovcode ( ) [inline]

Definition at line 104 of file dictionary.h.

References encode(), OOV(), and oovcode().

Referenced by lmtable::cpsublm(), and main().

Here is the call graph for this function:

Here is the caller graph for this function:

int dictionary::getcode ( const char * w )

Definition at line 470 of file dictionary.cpp.

References dict_entry::code, HT_FIND, NULL, and htable::search().

Referenced by ngramtable::augment(), main(), test(), and interplm::trainunigr().

Here is the call graph for this function:

Here is the caller graph for this function:

int dictionary::getword	(	fstream &	inp,
		char *	buffer
	)

Definition at line 80 of file dictionary.cpp.

References MAX_WORD.

Referenced by generate(), load(), and test().

Here is the caller graph for this function:

void dictionary::grow ( )

Definition at line 419 of file dictionary.cpp.

References freq(), GROWTH_STEP, HT_ENTER, LOAD_FACTOR, htable::search(), and dict_entry::word.

Referenced by encode(), and load().

Here is the call graph for this function:

Here is the caller graph for this function:

int dictionary::incflag ( int v ) [inline]

Definition at line 95 of file dictionary.h.

int dictionary::incflag ( ) [inline]

Definition at line 94 of file dictionary.h.

Referenced by ngramtable::augment(), ngramtable::generate(), ngramtable::generate_dstco(), ngramtable::generate_hmask(), Moses::LanguageModelIRST::GetLmID(), Moses::LanguageModelIRST::Load(), lmmacro::loadLexicalClasses(), ngramtable::loadtxt(), main(), lmtable::reload(), and mdiadaptlm::saveMT().

Here is the caller graph for this function:

long long dictionary::incfreq	(	int	code,
		long long	value
	)			`[inline]`

Definition at line 119 of file dictionary.h.

References dict_entry::freq.

Referenced by ngramtable::augment(), ngramtable::generate(), generate(), ngramtable::generate_dstco(), ngramtable::generate_hmask(), main(), and interplm::trainunigr().

Here is the caller graph for this function:

int dictionary::isprintable ( char * w ) [inline]

Definition at line 98 of file dictionary.h.

References MAX_WORD.

void dictionary::load ( std::istream & fd )

Definition at line 294 of file dictionary.cpp.

References dict_entry::code, dict_entry::freq, grow(), HT_ENTER, MAX_WORD, OOV(), strstack::push(), htable::search(), size(), and dict_entry::word.

Here is the call graph for this function:

void dictionary::load ( char * filename )

Definition at line 239 of file dictionary.cpp.

References mfstream::close(), dict_entry::code, dict_entry::freq, getword(), grow(), HT_ENTER, MAX_WORD, OOV(), strstack::push(), htable::search(), and dict_entry::word.

Referenced by dictionary(), Moses::LanguageModelIRST::Load(), ngramtable::loadbin(), and ngramtable::loadtxt().

Here is the call graph for this function:

Here is the caller graph for this function:

long long dictionary::multfreq	(	int	code,
		double	value
	)			`[inline]`

Definition at line 121 of file dictionary.h.

References dict_entry::freq, and freq().

Here is the call graph for this function:

const char* dictionary::OOV ( ) [inline]

Definition at line 88 of file dictionary.h.

References OOV_.

Referenced by ngramtable::augment(), decode(), encode(), genoovcode(), load(), main(), mixture::mixture(), parseline(), mdiadaptlm::saveARPA(), mdiadaptlm::saveASR(), mdiadaptlm::saveMT(), setoovrate(), mshiftbeta::train(), shiftbeta::train(), mixture::train(), and interplm::trainunigr().

Here is the caller graph for this function:

int dictionary::oovcode ( int v = -1 ) [inline]

Definition at line 92 of file dictionary.h.

Referenced by ngramtable::augment(), ngram::ckhisto(), lmtable::cpsublm(), mshiftbeta::discount(), shiftbeta::discount(), shiftone::discount(), mixture::discount(), linearwb::discount(), interplm::gencorrcounts(), ngramtable::generate(), genoovcode(), interplm::interplm(), Moses::LanguageModelIRST::Load(), ngramtable::loadtxt(), lmtable::lprob(), main(), lmmacro::One2OneMapping(), mdiadaptlm::saveARPA(), mdiadaptlm::saveASR(), ngramtable::savebin(), mdiadaptlm::saveMT(), mdiadaptlm::scalefact(), setoovrate(), interplm::test_ngt(), interplm::test_txt(), mixture::train(), interplm::trainunigr(), mdiadaptlm::txclprob(), and interplm::zerofreq().

Here is the caller graph for this function:

void dictionary::print_curve	(	int	curvesize,
		float *	testOOV = `NULL`
	)

Definition at line 141 of file dictionary.cpp.

References dict_entry::freq, freq(), and NULL.

Referenced by main().

Here is the call graph for this function:

Here is the caller graph for this function:

void dictionary::save ( std::ostream & fd )

Definition at line 327 of file dictionary.cpp.

References freq().

Here is the call graph for this function:

void dictionary::save	(	char *	filename,
		int	freqflag = `0`
	)

Definition at line 444 of file dictionary.cpp.

References dict_entry::freq, freq(), and dict_entry::word.

Referenced by main(), ngramtable::savebin(), lmtable::savebin(), mdiadaptlm::saveMT(), and ngramtable::savetxt().

Here is the call graph for this function:

Here is the caller graph for this function:

int dictionary::setoovrate ( double oovrate ) [inline]

Definition at line 111 of file dictionary.h.

References encode(), freq(), OOV(), oovcode(), and totfreq().

Referenced by main().

Here is the call graph for this function:

Here is the caller graph for this function:

int dictionary::size ( ) [inline]

Definition at line 149 of file dictionary.h.

Referenced by mdiadaptlm::adapt(), ngramtable::augment(), lmtable::cpsublm(), mixture::discount(), interplm::gencorrcounts(), ngramtable::generate(), ngramtable::generate_dstco(), ngramtable::generate_hmask(), load(), lmmacro::loadLexicalClasses(), main(), mdiadaptlm::netsize(), normcache::normcache(), interplm::obswrd(), mdiadaptlm::saveARPA(), mdiadaptlm::saveASR(), mdiadaptlm::scalefact(), lmtable::setlogOOVpenalty(), interplm::test_ngt(), interplm::test_txt(), mixture::train(), interplm::trainunigr(), mdiadaptlm::txclprob(), and interplm::unigr().

Here is the caller graph for this function:

void dictionary::sort ( )

Definition at line 384 of file dictionary.cpp.

References cmpdictentry(), dict_entry::code, HT_ENTER, LOAD_FACTOR, NULL, and htable::search().

Here is the call graph for this function:

void dictionary::stat ( )

Definition at line 410 of file dictionary.cpp.

References strstack::used(), and htable::used().

Referenced by ngramtable::stat(), and lmtable::stat().

Here is the call graph for this function:

Here is the caller graph for this function:

float * dictionary::test	(	int	curvesize,
		const char *	filename,
		int	listflag = `0`
	)

Definition at line 184 of file dictionary.cpp.

References BoS(), mfstream::close(), dict_entry::freq, freq(), getcode(), getword(), k, MAX_WORD, and NULL.

Referenced by main().

Here is the call graph for this function:

Here is the caller graph for this function:

long long dictionary::totfreq ( ) [inline]

Definition at line 134 of file dictionary.h.

Referenced by main(), setoovrate(), interplm::trainunigr(), and interplm::unigr().

Here is the caller graph for this function:

Friends And Related Function Documentation

friend class dictionary_iter [friend]

Definition at line 81 of file dictionary.h.

Member Data Documentation

dictionary* dictionary::oovlex

Definition at line 83 of file dictionary.h.

The documentation for this class was generated from the following files:

irstlm/src/dictionary.h
irstlm/src/dictionary.cpp


Public Member Functions
int	dub ()
int	dub (int value)
const char *	OOV ()
const char *	BoS ()
const char *	EoS ()
int	oovcode (int v=-1)
int	incflag ()
int	incflag (int v)
int	getword (fstream &inp, char *buffer)
int	isprintable (char *w)
void	genoovcode ()
int	setoovrate (double oovrate)
long long	incfreq (int code, long long value)
long long	multfreq (int code, double value)
long	freq (int code, long long value=-1)
long long	totfreq ()
void	grow ()
void	sort ()
	dictionary (char *filename, int size=DICT_INITSIZE)
	dictionary (dictionary *d, bool sortflag=true)
	~dictionary ()
void	generate (char *filename)
void	load (char *filename)
void	save (char *filename, int freqflag=0)
void	load (std::istream &fd)
void	save (std::ostream &fd)
int	size ()
int	getcode (const char *w)
int	encode (const char *w)
const char *	decode (int c)
void	stat ()
void	print_curve (int curvesize, float *testOOV=NULL)
float *	test (int curvesize, const char *filename, int listflag=0)
void	cleanfreq ()
Public Attributes
dictionary *	oovlex
Friends
class	dictionary_iter