Moses-support Digest, Vol 86, Issue 19

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Specifying LM type in moses build (Jeffrey Micher)
2. Re: error during testing (amir haghighi)
3. Re: Specifying LM type in moses build (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Fri, 6 Dec 2013 17:50:05 -0500
From: "Jeffrey Micher" <jmicher@cs.cmu.edu>
Subject: [Moses-support] Specifying LM type in moses build
To: <moses-support@mit.edu>
Message-ID: <001501cef2d5$81f906e0$85eb14a0$@cs.cmu.edu>
Content-Type: text/plain; charset="us-ascii"

Hi,



I'm trying to use KenLM instead of SRILM. I initially built moses with
SRILM support.



The documentation for using the various LMs has changed today. Now it's not
clear what I should do.



I seem to remember reading that I should use 8 for KENLM, 0 for SRILM when
I'm training, so I used to use:



-lm 0:3:path_to_srilm (3 gram)



I changed it to:



-lm 8:5:path_to_kenlm (5 gram)



When I did this, my moses.ini file had something like: SRILM name=LM0
factor=8 path=/home/jmicher/experiments/exp13/data/exp13.lm.bin order=5



I went to check the documentation again in two places: first in the Language
Modeling page, where I finally figured out that the documentation had been
changed earlier today. so I can edit my moses.ini file now to have KENLM
name=LM0 order=5 .etc



But how do I specify this when I'm training? I went to the Training
Reference page and found:



--lm -- language model: <factor>:<order>:<filename> (option can be repeated)



On the Build Baseline page, the training parameter for specifying lm info is
given as:



-lm 0:3:$HOME/lm/news-commentary-v8.fr-en.blm.en:8



Why is there an 8 at the end of this string?



And how should I now specify that I want to use KenLM when run the training
script?



Thanks.





-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131206/008ee919/attachment-0001.htm

------------------------------

Message: 2
Date: Sat, 7 Dec 2013 17:33:17 +0330
From: amir haghighi <amir.haghighi.64@gmail.com>
Subject: Re: [Moses-support] error during testing
To: moses-support@mit.edu
Message-ID:
<CA+UVbEh75Qiakuu+G=9yUh8bJFs0VOBAwBUUW8YMH9Pkzqnegg@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi,

I have also the same problem in evaluation step with EMS and I would be
thankful if you could help me.
the lexical reordering file is emtpy and the log of the output in
evaluation_test_filter.2.stderr is:

Using SCRIPTS_ROOTDIR:
/opt/tools/mosesdecoder-RELEASE-1.0/mosesdecoder-RELEASE-1.0/scripts
(9) create moses.ini @ Sat Dec 7 04:50:15 PST 2013
Executing: mkdir -p /opt/tools/workingEms/evaluation/test.filtered.2
Considering factor 0
Considering factor 0
filtering /opt/tools/workingEms/model/phrase-table.2 ->
/opt/tools/workingEms/evaluation/test.filtered.2/phrase-table.0-0.1.1...
0 of 2197240 phrases pairs used (0.00%) - note: max length 10
binarizing...cat
/opt/tools/workingEms/evaluation/test.filtered.2/phrase-table.0-0.1.1 |
LC_ALL=C sort -T /opt/tools/workingEms/evaluation/test.filtered.2 |
/opt/tools/mosesdecoder-RELEASE-1.0/mosesdecoder-RELEASE-1.0/bin/processPhraseTable
-ttable 0 0 - -nscores 5 -out
/opt/tools/workingEms/evaluation/test.filtered.2/phrase-table.0-0.1.1
processing ptree for stdin
Segmentation fault (core dumped)
filtering
/opt/tools/workingEms/model/reordering-table.2.wbe-msd-bidirectional-fe.gz
->
/opt/tools/workingEms/evaluation/test.filtered.2/reordering-table.2.wbe-msd-bidirectional-fe...
0 of 2197240 phrases pairs used (0.00%) - note: max length 10
binarizing.../opt/tools/mosesdecoder-RELEASE-1.0/mosesdecoder-RELEASE-1.0/bin/processLexicalTable
-in
/opt/tools/workingEms/evaluation/test.filtered.2/reordering-table.2.wbe-msd-bidirectional-fe
-out
/opt/tools/workingEms/evaluation/test.filtered.2/reordering-table.2.wbe-msd-bidirectional-fe
processLexicalTable v0.1 by Konrad Rawlik
processing
/opt/tools/workingEms/evaluation/test.filtered.2/reordering-table.2.wbe-msd-bidirectional-fe
to
/opt/tools/workingEms/evaluation/test.filtered.2/reordering-table.2.wbe-msd-bidirectional-fe.*
ERROR: empty lexicalised reordering file



Barry Haddow <bhaddow@...> writes:

>
> Hi Irene
>
> > But the output is empty. And the errors are 1. segmentation fault
> 2. error: empty lexicalized
> > reordering file
>
> Is this lexicalised reordering file empty then?
>
> It would be helpful if you could post the full log of the output when
> your run the filter command,
>
> cheers - Barry
>
> On 26/10/12 17:59, Irene Huang wrote:
> > Hi, I have trained and tuned the model, now I am using
> >
> > ~/mosesdecoder/scripts/training/filter-model-given-input.pl
> > <http://filter-model-given-input.pl> filtered-newstest2011
> > mert-work/moses.ini ~/corpus/newstest2011.true.fr
> > <http://newstest2011.true.fr> \
> > -Binarizer ~/mosesdecoder/bin/processPhraseTable
> >
> > to filter the phrase table.
> >
> > But the output is empty. And the errors are 1. segmentation fault
> > 2. error: empty lexicalized reordering file
> >
> > So does this mean it's out of memory error?
> >
> > Thanks
> >
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@...
> > http://mailman.mit.edu/mailman/listinfo/moses-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131207/5ec84057/attachment-0001.htm

------------------------------

Message: 3
Date: Sat, 7 Dec 2013 15:47:44 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Specifying LM type in moses build
To: Jeffrey Micher <jmicher@cs.cmu.edu>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbhBO5pBPq2cZyt28uWX1LAcBWkyF16WfWEzc6Lrnd1bWQ@mail.gmail.com>
Content-Type: text/plain; charset="windows-1252"

On 6 December 2013 22:50, Jeffrey Micher <jmicher@cs.cmu.edu> wrote:

> Hi,
>
>
>
> I?m trying to use KenLM instead of SRILM. I initially built moses with
> SRILM support.
>
>
>
> The documentation for using the various LMs has changed today. Now it?s
> not clear what I should do.
>

The documentation has been changed to reflect the new moses.ini file
format. The new format is used in the current github code, and in the
forthcoming RELEASE-2.0. The old format was used in RELEASE-1.0

>
>
> I seem to remember reading that I should use 8 for KENLM, 0 for SRILM when
> I?m training, so I used to use:
>
>
>
> -lm 0:3:path_to_srilm (3 gram)
>
>
>
> I changed it to:
>
>
>
> -lm 8:5:path_to_kenlm (5 gram)
>

to create a moses.ini file that says
KENLM ....
the train-model.perl argument is
-lm 0:5:path_to_kenlm:8


>
> When I did this, my moses.ini file had something like: SRILM name=LM0
> factor=8 path=/home/jmicher/experiments/exp13/data/exp13.lm.bin order=5
>
>
>
> I went to check the documentation again in two places: first in the
> Language Modeling page, where I finally figured out that the documentation
> had been changed earlier today? so I can edit my moses.ini file now to have
> KENLM name=LM0 order=5 ?etc
>
>
>
> But how do I specify this when I?m training? I went to the Training
> Reference page and found:
>
>
>
> --lm -- language model: <factor>:<order>:<filename> (option can be
> repeated)
>
>
>
> On the Build Baseline page, the training parameter for specifying lm info
> is given as:
>
>
>
> -lm 0:3:$HOME/lm/news-commentary-v8.fr-en.blm.en:8
>
>
>
> Why is there an 8 at the end of this string?
>
>
>
> And how should I now specify that I want to use KenLM when run the
> training script?
>
>
>
> Thanks.
>
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131207/398e7d04/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 86, Issue 19
*********************************************

Tweet