Home » Uncategories » Moses-support Digest, Vol 93, Issue 11

Moses-support Digest, Vol 93, Issue 11

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Filtering Issue (Judah Schvimer)
2. Re: Filtering Issue (Philipp Koehn)
3. Can't build moses server (Peter Kleiweg)
4. Re: How to use the NeuralLM (wangrui.nlp@gmail.com)


----------------------------------------------------------------------

Message: 1
Date: Wed, 9 Jul 2014 13:20:56 -0400
From: Judah Schvimer <judah.schvimer@mongodb.com>
Subject: Re: [Moses-support] Filtering Issue
To: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Cc: moses-support <moses-support@mit.edu>, "user-irstlm@list.fbk.eu"
<user-irstlm@list.fbk.eu>
Message-ID:
<CALF9aB46-N77nR9Rj+7F-9vrh56M5f33OaSjiyy1NAaVK8bHHA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

I did, I ended up just removing the filtering since it didn't seem so
necessary. binarizing normally worked fine


On Wed, Jul 9, 2014 at 12:02 PM, Hieu Hoang <Hieu.Hoang@ed.ac.uk> wrote:

> Sorry for late reply.
>
> Did you clean the data before you created the language model? Or you can
> try binarizing with KenLM instead
>
>
> On 3 July 2014 14:50, Judah Schvimer <judah.schvimer@mongodb.com> wrote:
>
>> Hi,
>>
>> I have a script that basically does all of the training steps for me and
>> it's worked many times before, but suddenly I got this error with a
>> different corpus. It doesn't occur when I use the moses.ini file in
>> train/model/ or in mert-work/, just when I use the one in filtered/. Any
>> idea what's wrong? Something appears to be weird around the 39-grams step.
>>
>> Defined parameters (per moses.ini or switch):
>> config: /home/judah/archive8/3/working/filtered/moses.ini
>> distortion-limit: 6
>> feature: UnknownWordPenalty WordPenalty PhrasePenalty
>> PhraseDictionaryBinary name=TranslationModel0 table-limit=20 num-features=4
>> path=/home/judah/archive8/3/working/filtered/phrase-table.0-0.1.1
>> input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0
>> num-features=6 type=hier-msd-bidirectional-fe-allff input-factor=0
>> output-factor=0
>> path=/home/judah/archive8/3/working/filtered/reordering-table.hier-msd-bidirectional-fe
>> Distortion IRSTLM name=LM0 factor=0 path=/home/judah/archive8/3/lm/
>> train.en-es.blm.es order=3
>> input-factors: 0
>> mapping: 0 T 0
>> weight: LexicalReordering0= 0.389589 0.0418995 0.0286706 0.0187875
>> 0.0287628 0.00958344 Distortion0= 0.0583275 LM0= -0.00740405 WordPenalty0=
>> 0.0123212 PhrasePenalty0= 0.0413057 TranslationModel0= 0.0360237 0.040722
>> 0.266723 0.0198795 UnknownWordPenalty0= 1
>> /home/judah/mosesdecoder/bin
>> line=UnknownWordPenalty
>> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
>> line=WordPenalty
>> FeatureFunction: WordPenalty0 start: 1 end: 1
>> line=PhrasePenalty
>> FeatureFunction: PhrasePenalty0 start: 2 end: 2
>> line=PhraseDictionaryBinary name=TranslationModel0 table-limit=20
>> num-features=4
>> path=/home/judah/archive8/3/working/filtered/phrase-table.0-0.1.1
>> input-factor=0 output-factor=0
>> FeatureFunction: TranslationModel0 start: 3 end: 6
>> line=LexicalReordering name=LexicalReordering0 num-features=6
>> type=hier-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>> path=/home/judah/archive8/3/working/filtered/reordering-table.hier-msd-bidirectional-fe
>> FeatureFunction: LexicalReordering0 start: 7 end: 12
>> Initializing LexicalReordering..
>> line=Distortion
>> FeatureFunction: Distortion0 start: 13 end: 13
>> line=IRSTLM name=LM0 factor=0 path=/home/judah/archive8/3/lm/
>> train.en-es.blm.es order=3
>> FeatureFunction: LM0 start: 14 end: 14
>> Loading UnknownWordPenalty0
>> Loading WordPenalty0
>> Loading PhrasePenalty0
>> Loading LexicalReordering0
>> binary file loaded, default OFF_T: -1
>> Loading Distortion0
>> Loading LM0
>> In LanguageModelIRST::Load: nGramOrder = 3
>> Language Model Type of /home/judah/archive8/3/lm/train.en-es.blm.es is 1
>> Language Model Type is 1
>> mmap
>> loadtxt_ram()
>> 3-grams: reading 0 entries
>> done level 3
>> 1-grams: reading 0 entries
>> done level 1
>> 8-grams: reading 0 entries
>> done level 8
>> 2-grams: reading 0 entries
>> done level 2
>> 39-grams: reading 3991252117 entries
>> moses: util.cpp:289: int parseline(std::istream&, int, ngram&, float&,
>> float&): Assertion `howmany == (Order+ 1) || howmany == (Order + 2)' failed.
>> [1] 5062 abort (core dumped) /home/judah/mosesdecoder/bin/moses -f <
>> >
>>
>>
>> Thanks,
>> Judah
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140709/3db7dc1e/attachment-0001.htm

------------------------------

Message: 2
Date: Wed, 9 Jul 2014 14:12:46 -0400
From: Philipp Koehn <pkoehn@inf.ed.ac.uk>
Subject: Re: [Moses-support] Filtering Issue
To: Judah Schvimer <judah.schvimer@mongodb.com>
Cc: Hieu Hoang <hieu.hoang@ed.ac.uk>, moses-support
<moses-support@mit.edu>, "user-irstlm@list.fbk.eu"
<user-irstlm@list.fbk.eu>
Message-ID:
<CAAFADDB+RwQAZ4JbirL-QCamUt6-Cm2FkcMqYnwu=JPa-WhCug@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Hi,

I am not an expert on the logging of the LM loading, but the following
looks pretty wacky:

3-grams: reading 0 entries
done level 3
1-grams: reading 0 entries
done level 1
8-grams: reading 0 entries
done level 8
2-grams: reading 0 entries
done level 2
39-grams: reading 3991252117 entries

No 1-3 grams, but gazillion 39-grams? Wait, 39-grams?

-phi


On Wed, Jul 9, 2014 at 1:20 PM, Judah Schvimer
<judah.schvimer@mongodb.com> wrote:
> I did, I ended up just removing the filtering since it didn't seem so
> necessary. binarizing normally worked fine
>
>
> On Wed, Jul 9, 2014 at 12:02 PM, Hieu Hoang <Hieu.Hoang@ed.ac.uk> wrote:
>>
>> Sorry for late reply.
>>
>> Did you clean the data before you created the language model? Or you can
>> try binarizing with KenLM instead
>>
>>
>> On 3 July 2014 14:50, Judah Schvimer <judah.schvimer@mongodb.com> wrote:
>>>
>>> Hi,
>>>
>>> I have a script that basically does all of the training steps for me and
>>> it's worked many times before, but suddenly I got this error with a
>>> different corpus. It doesn't occur when I use the moses.ini file in
>>> train/model/ or in mert-work/, just when I use the one in filtered/. Any
>>> idea what's wrong? Something appears to be weird around the 39-grams step.
>>>
>>> Defined parameters (per moses.ini or switch):
>>> config: /home/judah/archive8/3/working/filtered/moses.ini
>>> distortion-limit: 6
>>> feature: UnknownWordPenalty WordPenalty PhrasePenalty
>>> PhraseDictionaryBinary name=TranslationModel0 table-limit=20 num-features=4
>>> path=/home/judah/archive8/3/working/filtered/phrase-table.0-0.1.1
>>> input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0
>>> num-features=6 type=hier-msd-bidirectional-fe-allff input-factor=0
>>> output-factor=0
>>> path=/home/judah/archive8/3/working/filtered/reordering-table.hier-msd-bidirectional-fe
>>> Distortion IRSTLM name=LM0 factor=0
>>> path=/home/judah/archive8/3/lm/train.en-es.blm.es order=3
>>> input-factors: 0
>>> mapping: 0 T 0
>>> weight: LexicalReordering0= 0.389589 0.0418995 0.0286706 0.0187875
>>> 0.0287628 0.00958344 Distortion0= 0.0583275 LM0= -0.00740405 WordPenalty0=
>>> 0.0123212 PhrasePenalty0= 0.0413057 TranslationModel0= 0.0360237 0.040722
>>> 0.266723 0.0198795 UnknownWordPenalty0= 1
>>> /home/judah/mosesdecoder/bin
>>> line=UnknownWordPenalty
>>> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
>>> line=WordPenalty
>>> FeatureFunction: WordPenalty0 start: 1 end: 1
>>> line=PhrasePenalty
>>> FeatureFunction: PhrasePenalty0 start: 2 end: 2
>>> line=PhraseDictionaryBinary name=TranslationModel0 table-limit=20
>>> num-features=4
>>> path=/home/judah/archive8/3/working/filtered/phrase-table.0-0.1.1
>>> input-factor=0 output-factor=0
>>> FeatureFunction: TranslationModel0 start: 3 end: 6
>>> line=LexicalReordering name=LexicalReordering0 num-features=6
>>> type=hier-msd-bidirectional-fe-allff input-factor=0 output-factor=0
>>> path=/home/judah/archive8/3/working/filtered/reordering-table.hier-msd-bidirectional-fe
>>> FeatureFunction: LexicalReordering0 start: 7 end: 12
>>> Initializing LexicalReordering..
>>> line=Distortion
>>> FeatureFunction: Distortion0 start: 13 end: 13
>>> line=IRSTLM name=LM0 factor=0
>>> path=/home/judah/archive8/3/lm/train.en-es.blm.es order=3
>>> FeatureFunction: LM0 start: 14 end: 14
>>> Loading UnknownWordPenalty0
>>> Loading WordPenalty0
>>> Loading PhrasePenalty0
>>> Loading LexicalReordering0
>>> binary file loaded, default OFF_T: -1
>>> Loading Distortion0
>>> Loading LM0
>>> In LanguageModelIRST::Load: nGramOrder = 3
>>> Language Model Type of /home/judah/archive8/3/lm/train.en-es.blm.es is 1
>>> Language Model Type is 1
>>> mmap
>>> loadtxt_ram()
>>> 3-grams: reading 0 entries
>>> done level 3
>>> 1-grams: reading 0 entries
>>> done level 1
>>> 8-grams: reading 0 entries
>>> done level 8
>>> 2-grams: reading 0 entries
>>> done level 2
>>> 39-grams: reading 3991252117 entries
>>> moses: util.cpp:289: int parseline(std::istream&, int, ngram&, float&,
>>> float&): Assertion `howmany == (Order+ 1) || howmany == (Order + 2)' failed.
>>> [1] 5062 abort (core dumped) /home/judah/mosesdecoder/bin/moses -f <
>>> >
>>>
>>>
>>> Thanks,
>>> Judah
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>
>>
>>
>> --
>> Hieu Hoang
>> Research Associate
>> University of Edinburgh
>> http://www.hoang.co.uk/hieu
>>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


------------------------------

Message: 3
Date: Thu, 10 Jul 2014 01:02:37 +0200
From: Peter Kleiweg <p.c.j.kleiweg@rug.nl>
Subject: [Moses-support] Can't build moses server
To: moses-support@mit.edu
Message-ID: <b558b64aa8f3125538239d75c7bd85b8@smtp-cloud6.xs4all.net>
Content-Type: text/plain; charset="us-ascii"


Hi,

I can't build the moses server. I have tried with boost versions
1.42.0, 1.47.0 and 1.55.0, all give the same result.


--
Peter Kleiweg
http://pkleiweg.home.xs4all.nl/
-------------- next part --------------
A non-text attachment was scrubbed...
Name: build.log.gz
Type: application/octet-stream
Size: 1958 bytes
Desc:
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140710/4a9f3fca/attachment-0001.obj

------------------------------

Message: 4
Date: Thu, 10 Jul 2014 11:38:25 +0200
From: "wangrui.nlp@gmail.com" <wangrui.nlp@gmail.com>
Subject: Re: [Moses-support] How to use the NeuralLM
To: moses <moses@kheafield.com>
Cc: moses-support <moses-support@mit.edu>, moses-support-request
<moses-support-request@mit.edu>
Message-ID: <201407101138243539374@gmail.com>
Content-Type: text/plain; charset="utf-8"

I have implement the NPLM tookit, both the original one and the thread safe one. When I integrate NPLM into latest version Moses, using "./bjam ... --with-nplm=path to nplm, "I found many errors.

Firstly, the "lm/wrappers/nplm.cc and lm/wrappers/nplm.hh" is needed. I found it is not in Moses, but in Kenlm toolkit, so I download from https://github.com/kpu/kenlm/tree/master/lm/wrappers .

Then when I try it again, the error become:

------------------------------------------------
lm/wrappers/nplm.cc: In constructor ?lm::np::Model::Model(const std::string&, size_t)?:
lm/wrappers/nplm.cc:36: error: no matching function for call to ?nplm::neuralLM::neuralLM(const std::basic_string<char, std::char_traits<char>, std::allocator<char> >&)?
/panfs/panmt/users/rwang/nplm-0.1/src/neuralLM.h:49: note: candidates are: nplm::neuralLM::neuralLM()
/panfs/panmt/users/rwang/nplm-0.1/src/neuralLM.h:26: note: nplm::neuralLM::neuralLM(const nplm::neuralLM&)
lm/wrappers/nplm.cc: In member function ?lm::FullScoreReturn lm::np::Model::FullScore(const lm::np::State&, lm::WordIndex, lm::np::State&) const?:
lm/wrappers/nplm.cc:60: error: ?class nplm::neuralLM? has no member named ?staging_ngram?
lm/wrappers/nplm.cc:62: error: ?class nplm::neuralLM? has no member named ?staging_ngram?
lm/wrappers/nplm.cc:63: error: ?class nplm::neuralLM? has no member named ?lookup_from_staging?
In file included from /panfs/panmt/users/rwang/nplm-0.1/src/neuralLM.h:16,
from lm/wrappers/nplm.cc:8:
------------------------------------------------

I tried mon both the original one and the thread safe one of NPLM.




--
------------------

Yours sincerely,

WANG Rui

-------------------------------------------------------------
Joint Ph.D. Candidate
Institute of Cognitive Science
National Center for Scientific Research (CNRS, France)


Ph.D. Candidate
Department of Computer Science and Engineering,
Shanghai Jiao Tong University (SJTU, China)

Homepage : http://bcmi.sjtu.edu.cn/~wangrui/

Tel: +33-(0)-6 78 41 93 69 (Lyon, France)
+86-18818272601?Shanghai, China?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140710/b5b803fb/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 93, Issue 11
*********************************************

0 Response to "Moses-support Digest, Vol 93, Issue 11"

Post a Comment

Subscribe to: Post Comments (Atom)