Moses::LanguageModelIRST Class Reference
#include <IRST.h>
Public Member Functions | |
| LanguageModelIRST (int dub) | |
| ~LanguageModelIRST () | |
| bool | Load (const std::string &filePath, FactorType factorType, size_t nGramOrder) |
| virtual LMResult | GetValue (const std::vector< const Word * > &contextFactor, State *finalState=NULL) const |
| void | CleanUpAfterSentenceProcessing () |
| void | InitializeBeforeSentenceProcessing () |
| overrideable funtions for IRST LM to cleanup. Maybe something to do with on demand/cache loading/unloading | |
| void | set_dictionary_upperbound (int dub) |
Protected Member Functions | |
| void | CreateFactors (FactorCollection &factorCollection) |
| int | GetLmID (const std::string &str) const |
| int | GetLmID (const Factor *factor) const |
Protected Attributes | |
| std::vector< int > | m_lmIdLookup |
| lmContainer * | m_lmtb |
| int | m_unknownId |
| int | m_empty |
| int | m_lmtb_sentenceStart |
| int | m_lmtb_sentenceEnd |
| int | m_lmtb_size |
| int | m_lmtb_dub |
| std::string | m_mapFilePath |
| dictionary * | d |
Detailed Description
Implementation of single factor LM using IRST's code. This is the default LM for Moses and is available from the same sourceforge repositoryDefinition at line 43 of file IRST.h.
Constructor & Destructor Documentation
| Moses::LanguageModelIRST::~LanguageModelIRST | ( | ) |
Member Function Documentation
| void Moses::LanguageModelIRST::CleanUpAfterSentenceProcessing | ( | ) | [virtual] |
Reimplemented from Moses::LanguageModelImplementation.
Definition at line 253 of file IRST.cpp.
References Moses::StaticData::GetLMCacheCleanupThreshold(), Moses::StaticData::Instance(), Moses::LMCacheCleanup(), m_lmtb, and TRACE_ERR.
| void Moses::LanguageModelIRST::CreateFactors | ( | FactorCollection & | factorCollection | ) | [protected] |
Definition at line 105 of file IRST.cpp.
References Moses::FactorCollection::AddFactor(), BOS_, d, EOS_, Moses::Factor::GetId(), GetLmID(), lmIdMap, m_empty, Moses::LanguageModelSingleFactor::m_factorType, m_lmIdLookup, m_lmtb_sentenceEnd, m_lmtb_sentenceStart, Moses::LanguageModelSingleFactor::m_sentenceEnd, Moses::LanguageModelImplementation::m_sentenceEndArray, Moses::LanguageModelSingleFactor::m_sentenceStart, Moses::LanguageModelImplementation::m_sentenceStartArray, NULL, and Moses::Output.
Referenced by Load().
| int Moses::LanguageModelIRST::GetLmID | ( | const Factor * | factor | ) | const [protected] |
poiche' non c'e' distinzione tra i factorIDs delle parole sorgenti e delle parole target in Moses, puo' accadere che una parola target di cui non sia stato ancora calcolato il suo codice target abbia comunque un factorID noto (e quindi minore di m_lmIdLookup.size()) E' necessario dunque identificare questi casi di indeterminatezza del codice target. Attualamente, questo controllo e' stato implementato impostando a m_empty tutti i termini che non hanno ancora
OLD PROBLEM - SOLVED
IL PPROBLEMA ERA QUI m_lmIdLookup.push_back(code); PERCHE' USANDO PUSH_BACK IN REALTA' INSEREVIVAMO L'ELEMENTO NUOVO IN POSIZIONE (factorID-1) invece che in posizione factrID dove dopo andiamo a leggerlo (vedi caso C Cosi' funziona .... ho un dubbio su cosa c'e' nelle prime posizioni di m_lmIdLookup quindi e scopro che rimane vuota una entry ogni due perche' factorID cresce di due in due (perche' codifica sia source che target) "vuota" la posizione (factorID-1) non da problemi di correttezza, ma solo di "spreco" di memoria potremmo sostituirerendere m_lmIdLookup una std:map invece che un std::vector, ma si perde in efficienza nell'accesso perche' non e' piu' possibile quello random dei vettori a te la scelta!!!!
Definition at line 150 of file IRST.cpp.
References d, Moses::Factor::GetId(), Moses::Factor::GetString(), m_empty, m_lmIdLookup, and m_unknownId.
| int Moses::LanguageModelIRST::GetLmID | ( | const std::string & | str | ) | const [protected] |
Definition at line 145 of file IRST.cpp.
References d.
Referenced by CreateFactors().
| virtual LMResult Moses::LanguageModelIRST::GetValue | ( | const std::vector< const Word * > & | contextFactor, | |
| State * | finalState = NULL | |||
| ) | const [virtual] |
Implements Moses::LanguageModelPointerState.
| void Moses::LanguageModelIRST::InitializeBeforeSentenceProcessing | ( | ) | [virtual] |
overrideable funtions for IRST LM to cleanup. Maybe something to do with on demand/cache loading/unloading
Reimplemented from Moses::LanguageModelImplementation.
Definition at line 267 of file IRST.cpp.
References m_lmtb.
| bool Moses::LanguageModelIRST::Load | ( | const std::string & | filePath, | |
| FactorType | factorType, | |||
| size_t | nGramOrder | |||
| ) | [virtual] |
Implements Moses::LanguageModelSingleFactor.
Definition at line 60 of file IRST.cpp.
References Moses::UserMessage::Add(), CreateFactors(), d, Moses::FactorCollection::Instance(), Moses::StaticData::Instance(), m_empty, Moses::LanguageModelSingleFactor::m_factorType, Moses::LanguageModelImplementation::m_filePath, m_lmtb, m_lmtb_dub, m_lmtb_size, Moses::LanguageModelImplementation::m_nGramOrder, m_unknownId, Moses::StaticData::ThreadCount(), and VERBOSE.
| void Moses::LanguageModelIRST::set_dictionary_upperbound | ( | int | dub | ) | [inline] |
Member Data Documentation
dictionary* Moses::LanguageModelIRST::d [protected] |
int Moses::LanguageModelIRST::m_empty [protected] |
std::vector<int> Moses::LanguageModelIRST::m_lmIdLookup [mutable, protected] |
lmContainer* Moses::LanguageModelIRST::m_lmtb [protected] |
Definition at line 47 of file IRST.h.
Referenced by CleanUpAfterSentenceProcessing(), InitializeBeforeSentenceProcessing(), Load(), and ~LanguageModelIRST().
int Moses::LanguageModelIRST::m_lmtb_dub [protected] |
int Moses::LanguageModelIRST::m_lmtb_sentenceEnd [protected] |
int Moses::LanguageModelIRST::m_lmtb_sentenceStart [protected] |
int Moses::LanguageModelIRST::m_lmtb_size [protected] |
std::string Moses::LanguageModelIRST::m_mapFilePath [protected] |
int Moses::LanguageModelIRST::m_unknownId [protected] |
The documentation for this class was generated from the following files:
