Home » Uncategories » Moses-support Digest, Vol 89, Issue 71

Moses-support Digest, Vol 89, Issue 71

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Is it possible to use -xml-input flag with the recaser ?
(Hieu Hoang)
2. Re: Mosesserver fails with "core dumped" (Hieu Hoang)
3. Re: cube pruning for phrase-based decoding (Hieu Hoang)
4. Re: Mosesserver fails with "core dumped"
(Ertu?rul YILMAZ (B?LGEM))


----------------------------------------------------------------------

Message: 1
Date: Thu, 27 Mar 2014 11:46:54 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Is it possible to use -xml-input flag
with the recaser ?
To: kamel nebhi <k.nebhi@sheffield.ac.uk>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbhAT+_sP2TtVpa81gQOTFYdBeffcXYoQwa9_ZtH-d5oSA@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

try adding
[xml-input]
whatever

to the recaser's moses.ini file. The recaser is just a monotone
phrase-based model


On 23 March 2014 20:53, kamel nebhi <k.nebhi@sheffield.ac.uk> wrote:

> Hello,
>
> I'm actually use the recaser provided by Moses. I want to ignore some part
> of the corpus and i'm looking for a solution such as the -xml-input flag
> (see at http://www.statmt.org/moses/?n=Moses.AdvancedFeatures#ntoc9).
>
> Is it possible to use this feature with the recaser ? If no, how can i
> ignore some part of the corpus ?
>
> Best
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140327/436a9f69/attachment-0001.htm

------------------------------

Message: 2
Date: Thu, 27 Mar 2014 11:55:53 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Mosesserver fails with "core dumped"
To: Ertu?rul YILMAZ (B?LGEM) <yilmaz.ertugrul@tubitak.gov.tr>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbi+o9jG1HHsBo7Ht7BiXpxXKrH2RDSB+yOVuHOyqpnaLg@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Does the ini file work when you use the command line moses?

How many scores are there for each rule in the translation table? There
should be 4 but if you had trained it with Moses v.1 then there will be 5.

In that case, just delete the last score, which is just a constant 2.718


On 26 March 2014 10:11, Ertu?rul YILMAZ (B?LGEM) <
yilmaz.ertugrul@tubitak.gov.tr> wrote:

> Hi All,
>
> I am working on running a phrase-based MT system with mosesserver
> resulting in the attached failure.
> I was able to run the same system with binarized phrase and reordering
> tables, but it fails to run them in memory.
>
> Did any of you run into a similar problem or have any ideas on the
> resolution for this? Attaching my moses.ini also.
>
> Thanks,
>
> ertugrul@pala01$ /usr/local/smt/moses_2013_11/bin/mosesserver -threads 8
> -dl 5 -minimum-bayes-risk -v 0 -f moses.tuned.ini.5.mod
> /usr/local/smt/moses_2013_11/bin
> line=UnknownWordPenalty
> line=WordPenalty
> line=PhrasePenalty
> line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20
> num-features=4
> path=/media/pala12/burak/aren/ems2/model/phrase-table-sigtest-filter.5.gz
> input-factor=0 output-factor=0
> line=LexicalReordering name=LexicalReordering0 num-features=6
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/media/pala12/burak/aren/ems2/model/reordering-table-sigtest-filter.5.wbe-msd-bidirectional-fe.gz
> Initializing LexicalReordering..
> line=Distortion
> line=KENLM lazyken=0 name=LM0 factor=0
> path=/media/pala12/burak/aren/ems2/lm/nist12.binlm.1 order=4
> line=KENLM lazyken=0 name=LM1 factor=0
> path=/media/pala12/burak/aren/ems2/lm/isi.binlm.1 order=4
> line=KENLM lazyken=0 name=LM2 factor=0
> path=/media/pala12/burak/aren/ems2/lm/un.binlm.1 order=4
> line=KENLM lazyken=0 name=LM3 factor=0
> path=/media/pala06/ilknur/pala05/gigaword_lm/gigaword5gr.tcafteraps3.irstlm.bin
> order=5
> Loading table into memory...done.
> Start loading text SCFG phrase table. Moses format : [233] seconds
> Check nextPos != string::npos failed in moses/Phrase.cpp:214
> Aborted (core dumped)
>
>
>
> *----------------------------------------------------------*
> *Ertu?rul YILMAZ*
> Senior Researcher
>
> Speech and Natural Language Processing Group
> T?B?TAK B?LGEM
> 41470 GEBZE / KOCAEL?, TURKEY
> T +90 262 648 1000 <callto:262%20648%201000> - 2216
> F +90 262 648 1100 <callto:262%20648%201100>
> <http://www.bilgem.tubitak.gov.tr/>www.bilgem.tubitak.gov.tr
> yilmaz.ertugrul@tubitak.gov.tr <http://www.bilgem.tubitak.gov.tr/>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140327/c8d4ccb3/attachment-0001.htm

------------------------------

Message: 3
Date: Thu, 27 Mar 2014 12:00:18 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] cube pruning for phrase-based decoding
To: Michael me <therestlesstimer@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbgmaekWeGBgp4drtAiyDA82Y6R21g+DgeQaCp89GaON7g@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

yes


On 26 March 2014 11:11, Michael me <therestlesstimer@gmail.com> wrote:

> Hi All,
>
> looking into the code, it appears to me that with cube pruning, the PB
> decoder processes the stacks one by one until all stacks are processed,
> which is the same procedure without cube pruning. The difference is that
> you put all hypotheses through a priority queue before you drop them into
> stacks further down.
>
> But looking at this paper (Figure 5):
>
> http://acl.ldc.upenn.edu/P/P07/P07-1019.pdf
>
> it seems that with cube pruning a PB decoder will process multiple stacks
> together. e.g. if we are currently at stack #5, then all hypotheses from
> previous 4 stacks are expanded into stack #5 through a priority queue? Is
> this how cube pruning for the PB decoder is implemented in Moses, or is it
> not?
>
> Eric
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140327/38277796/attachment-0001.htm

------------------------------

Message: 4
Date: Thu, 27 Mar 2014 15:25:29 +0200 (EET)
From: Ertu?rul YILMAZ (B?LGEM) <yilmaz.ertugrul@tubitak.gov.tr>
Subject: Re: [Moses-support] Mosesserver fails with "core dumped"
To: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<377564082.104671114.1395926729289.JavaMail.zimbra@tubitak.gov.tr>
Content-Type: text/plain; charset="utf-8"

Hi,

Well, I get the same failure with command line moses too :(
The interesting thing is that, I could use the exact same moses.ini with the only difference of using a smaller phrase table filtered for an evaluation set of ours (only changing the line for PhraseDictionaryMemory) and both moses and mosesserver works fine.
The one that works is about 12MB compressed and the one that fails is about 596MB compressed, would there be some sort of a setting to play with to work with larger phrase tables?

I do have four scores in the translation table (system was trained with one of latest versions of Moses).

Thanks,

---
/usr/local/smt/moses_2013_11/bin/moses -threads 8 -dl 5 -minimum-bayes-risk -v 0 -f moses.tuned.ini.5.mod
/usr/local/smt/moses_2013_11/bin
line=UnknownWordPenalty
line=WordPenalty
line=PhrasePenalty
line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4 path=/media/pala12/burak/aren/ems2/model/phrase-table-sigtest-filter.5.gz input-factor=0 output-factor=0
line=LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/media/pala12/burak/aren/ems2/model/reordering-table-sigtest-filter.5.wbe-msd-bidirectional-fe.gz
Initializing LexicalReordering..
line=Distortion
line=KENLM lazyken=0 name=LM0 factor=0 path=/media/pala12/burak/aren/ems2/lm/nist12.binlm.1 order=4
line=KENLM lazyken=0 name=LM1 factor=0 path=/media/pala12/burak/aren/ems2/lm/isi.binlm.1 order=4
line=KENLM lazyken=0 name=LM2 factor=0 path=/media/pala12/burak/aren/ems2/lm/un.binlm.1 order=4
line=KENLM lazyken=0 name=LM3 factor=0 path=/media/pala06/ilknur/pala05/gigaword_lm/gigaword5gr.tcafteraps3.irstlm.bin order=5
Loading table into memory...done.
Start loading text SCFG phrase table. Moses format : [235.000] seconds
Check nextPos != string::npos failed in moses/Phrase.cpp:214
Aborted (core dumped)



----- Orijinal Mesaj -----

Kimden: "Hieu Hoang" <Hieu.Hoang@ed.ac.uk>
Kime: "Ertu?rul YILMAZ (B?LGEM)" <yilmaz.ertugrul@tubitak.gov.tr>
Kk: "moses-support" <moses-support@mit.edu>
G?nderilenler: 27 Mart Per?embe 2014 13:55:53
Konu: Re: [Moses-support] Mosesserver fails with "core dumped"

Does the ini file work when you use the command line moses?

How many scores are there for each rule in the translation table? There should be 4 but if you had trained it with Moses v.1 then there will be 5.

In that case, just delete the last score, which is just a constant 2.718


On 26 March 2014 10:11, Ertu?rul YILMAZ (B?LGEM) < yilmaz.ertugrul@tubitak.gov.tr > wrote:



Hi All,

I am working on running a phrase-based MT system with mosesserver resulting in the attached failure.
I was able to run the same system with binarized phrase and reordering tables, but it fails to run them in memory.

Did any of you run into a similar problem or have any ideas on the resolution for this? Attaching my moses.ini also.

Thanks,

ertugrul@pala01$ /usr/local/smt/moses_2013_11/bin/mosesserver -threads 8 -dl 5 -minimum-bayes-risk -v 0 -f moses.tuned.ini.5.mod
/usr/local/smt/moses_2013_11/bin
line=UnknownWordPenalty
line=WordPenalty
line=PhrasePenalty
line=PhraseDictionaryMemory name=TranslationModel0 table-limit=20 num-features=4 path=/media/pala12/burak/aren/ems2/model/phrase-table-sigtest-filter.5.gz input-factor=0 output-factor=0
line=LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/media/pala12/burak/aren/ems2/model/reordering-table-sigtest-filter.5.wbe-msd-bidirectional-fe.gz
Initializing LexicalReordering..
line=Distortion
line=KENLM lazyken=0 name=LM0 factor=0 path=/media/pala12/burak/aren/ems2/lm/nist12.binlm.1 order=4
line=KENLM lazyken=0 name=LM1 factor=0 path=/media/pala12/burak/aren/ems2/lm/isi.binlm.1 order=4
line=KENLM lazyken=0 name=LM2 factor=0 path=/media/pala12/burak/aren/ems2/lm/un.binlm.1 order=4
line=KENLM lazyken=0 name=LM3 factor=0 path=/media/pala06/ilknur/pala05/gigaword_lm/gigaword5gr.tcafteraps3.irstlm.bin order=5
Loading table into memory...done.
Start loading text SCFG phrase table. Moses format : [233] seconds
Check nextPos != string::npos failed in moses/Phrase.cpp:214
Aborted (core dumped)


----------------------------------------------------------
Ertu?rul YILMAZ
Senior Researcher

Speech and Natural Language Processing Group
T?B?TAK B?LGEM
41470 GEBZE / KOCAEL?, TURKEY
T +90 262 648 1000 - 2216
F +90 262 648 1100
www.bilgem.tubitak.gov.tr
yilmaz.ertugrul@tubitak.gov.tr



_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support







--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu


-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140327/8ff70dbd/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 89, Issue 71
*********************************************

0 Response to "Moses-support Digest, Vol 89, Issue 71"

Post a Comment

Subscribe to: Post Comments (Atom)