ugdiss::TokenIndex Class Reference

#include <tpt_tokenindex.h>

Collaboration diagram for ugdiss::TokenIndex:

Collaboration graph
[legend]

List of all members.

Classes

class  CompFunc
class  Entry

Public Member Functions

 TokenIndex (std::string unkToken="UNK")
void open (std::string fname, std::string unkToken="UNK", bool dyna=false)
void close ()
id_type operator[] (char const *w) const
id_type operator[] (std::string const &w) const
char const *const operator[] (id_type id) const
char const *const operator[] (id_type id)
std::vector< char const * > reverseIndex () const
std::string toString (std::vector< id_type > const &v)
std::string toString (std::vector< id_type > const &v) const
std::string toString (id_type const *start, id_type const *const stop)
std::string toString (id_type const *start, id_type const *const stop) const
std::vector< id_typetoIdSeq (std::string const &line) const
bool fillIdSeq (std::string const &line, std::vector< id_type > &v) const
void iniReverseIndex ()
id_type getNumTokens () const
id_type getUnkId () const
id_type knownVocabSize () const
id_type totalVocabSize () const
id_type ksize () const
id_type tsize () const
char const *const getUnkToken () const
void write (std::string fname)
bool isDynamic () const
bool setDynamic (bool onoff)
void setUnkLabel (std::string unk)

Public Attributes

bio::mapped_file_source file
Entry const * startIdx
Entry const * endIdx
CompFunc comp


Detailed Description

Definition at line 28 of file tpt_tokenindex.h.


Constructor & Destructor Documentation

ugdiss::TokenIndex::TokenIndex ( std::string  unkToken = "UNK"  ) 


Member Function Documentation

void ugdiss::TokenIndex::close (  ) 

Definition at line 76 of file tpt_tokenindex.cc.

References file.

Referenced by write_tokenindex().

Here is the caller graph for this function:

bool ugdiss::TokenIndex::fillIdSeq ( std::string const &  line,
std::vector< id_type > &  v 
) const

Referenced by IBM1::align(), and main().

Here is the caller graph for this function:

id_type ugdiss::TokenIndex::getNumTokens (  )  const

Definition at line 293 of file tpt_tokenindex.cc.

id_type ugdiss::TokenIndex::getUnkId (  )  const

Definition at line 300 of file tpt_tokenindex.cc.

char const *const ugdiss::TokenIndex::getUnkToken (  )  const

Definition at line 307 of file tpt_tokenindex.cc.

void ugdiss::TokenIndex::iniReverseIndex (  ) 

Definition at line 173 of file tpt_tokenindex.cc.

References reverseIndex().

Referenced by main().

Here is the call graph for this function:

Here is the caller graph for this function:

bool ugdiss::TokenIndex::isDynamic (  )  const

Definition at line 387 of file tpt_tokenindex.cc.

id_type ugdiss::TokenIndex::knownVocabSize (  )  const

Definition at line 315 of file tpt_tokenindex.cc.

Referenced by conservative_sort(), main(), and save_vocabs().

Here is the caller graph for this function:

id_type ugdiss::TokenIndex::ksize (  )  const

Definition at line 322 of file tpt_tokenindex.cc.

Referenced by main(), Counter::operator()(), writeTable(), and writeTableHeader().

Here is the caller graph for this function:

void ugdiss::TokenIndex::open ( std::string  fname,
std::string  unkToken = "UNK",
bool  dyna = false 
)

Referenced by main(), IBM1::open(), open_bitext(), and open_vocab().

Here is the caller graph for this function:

char const *const ugdiss::TokenIndex::operator[] ( id_type  id  ) 

Definition at line 185 of file tpt_tokenindex.cc.

References reverseIndex().

Here is the call graph for this function:

char const *const ugdiss::TokenIndex::operator[] ( id_type  id  )  const

Definition at line 153 of file tpt_tokenindex.cc.

References reverseIndex().

Here is the call graph for this function:

id_type ugdiss::TokenIndex::operator[] ( std::string const &  w  )  const

id_type ugdiss::TokenIndex::operator[] ( char const *  w  )  const

vector< char const * > ugdiss::TokenIndex::reverseIndex (  )  const

Definition at line 133 of file tpt_tokenindex.cc.

References ugdiss::TokenIndex::CompFunc::base, comp, endIdx, NULL, and startIdx.

Referenced by iniReverseIndex(), main(), operator[](), and toString().

Here is the caller graph for this function:

bool ugdiss::TokenIndex::setDynamic ( bool  onoff  ) 

Definition at line 394 of file tpt_tokenindex.cc.

References NULL.

Referenced by main(), open_bitext(), and open_vocab().

Here is the caller graph for this function:

void ugdiss::TokenIndex::setUnkLabel ( std::string  unk  ) 

Referenced by open_vocab().

Here is the caller graph for this function:

std::vector<id_type> ugdiss::TokenIndex::toIdSeq ( std::string const &  line  )  const

string ugdiss::TokenIndex::toString ( id_type const *  start,
id_type const *const   stop 
) const

Definition at line 249 of file tpt_tokenindex.cc.

References reverseIndex().

Here is the call graph for this function:

string ugdiss::TokenIndex::toString ( id_type const *  start,
id_type const *const   stop 
)

Definition at line 232 of file tpt_tokenindex.cc.

References reverseIndex().

Here is the call graph for this function:

std::string ugdiss::TokenIndex::toString ( std::vector< id_type > const &  v  )  const

std::string ugdiss::TokenIndex::toString ( std::vector< id_type > const &  v  ) 

id_type ugdiss::TokenIndex::totalVocabSize (  )  const

Definition at line 329 of file tpt_tokenindex.cc.

References tsize().

Referenced by conservative_sort(), ini_cnt_vec(), main(), remap(), and save_vocabs().

Here is the call graph for this function:

Here is the caller graph for this function:

id_type ugdiss::TokenIndex::tsize (  )  const

Definition at line 334 of file tpt_tokenindex.cc.

References NULL.

Referenced by Moses::bitext::snt_adder< L2R_Token< SimpleWordId > >::operator()(), and totalVocabSize().

Here is the caller graph for this function:

void ugdiss::TokenIndex::write ( std::string  fname  ) 


Member Data Documentation

Definition at line 71 of file tpt_tokenindex.h.

Referenced by operator[](), and reverseIndex().

Definition at line 70 of file tpt_tokenindex.h.

Referenced by operator[](), and reverseIndex().

bio::mapped_file_source ugdiss::TokenIndex::file

Definition at line 68 of file tpt_tokenindex.h.

Referenced by close().

Definition at line 69 of file tpt_tokenindex.h.

Referenced by operator[](), and reverseIndex().


The documentation for this class was generated from the following files:

Generated on Sun Aug 9 00:34:28 2015 for Moses by  doxygen 1.5.9