Moses-support Digest, Vol 111, Issue 21

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Tiny sample data uses outdated file format (Lane Schwartz)
2. Error when running moses server with sample code (Lane Schwartz)
3. Re: Which symal? (Ergun Bicici)


----------------------------------------------------------------------

Message: 1
Date: Mon, 11 Jan 2016 15:33:03 -0600
From: Lane Schwartz <dowobeha@gmail.com>
Subject: [Moses-support] Tiny sample data uses outdated file format
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CABv3vZk-vfVGxnMJw9cDKXd+87L-ov-K_1spMUE96pf=84XU1Q@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

http://www.statmt.org/moses/sample-data/complete/tiny.zip

The config file included in the above sample doesn't work anymore,
presumably due to a change in the ini format.

$ ~/mosesdecoder.git/bin/moses -f moses.ini < in.txt
>
> Defined parameters (per moses.ini or switch):
> config: moses.ini
>
> description: French English Tiny test data used for debugging
>
> distortion-limit: 4
>
> generation-file: 0 1 2 generation0-1.txt
>
> input-factors: 0 1
>
> lmodel-file: 0 0 3 europarl.en.srilm 0 1 3 europarl.pos.en.srilm
>
> load-dir: .
>
> mapping: T 0 G 0 T 1
>
> ttable-file: 0 0 5 trans0-0.txt 1 1 5 trans1-1.txt
>
> ttable-limit: 20 20
>
> weight-d: 0.2
>
> weight-generation: -110.232 -110.3
>
> weight-l: 0.5 0.5
>
> weight-t: 0.2 0.2 0.2 0.2 0.2 0.1 0.1 0.1 0.1 0.1
>
> weight-w: -1
>
> Phrase table specification in old 4-field format. No longer
> supportedUnknown parameter load-dirUnknown parameter ttable-limit
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160111/49773958/attachment-0001.html

------------------------------

Message: 2
Date: Mon, 11 Jan 2016 15:43:58 -0600
From: Lane Schwartz <dowobeha@gmail.com>
Subject: [Moses-support] Error when running moses server with sample
code
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CABv3vZnJxncZbf74GjGxMv=gKHndBeEz_mJNQ0Dw_5fkhBa2SA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

I'm trying out mosesserver for the first time. I have a config file for
fr-en that uses a smallish TM and LM, that work fine when run with Moses.
When I try running the same config using mosesserver, and then use the
sample Perl or Python code in contrib/server, mosesserver dies with the
following error:

Defined parameters (per moses.ini or switch):
> config: moses.tuned.ini.2.probing.noLexRO
> cube-pruning-pop-limit: 400
> distortion-limit: 6
> feature: UnknownWordPenalty WordPenalty PhrasePenalty ProbingPT
> name=TranslationModel0 num-features=4
> path=phrase-table.2.pruned100.probing/ input-factor=0 output-factor=0
> table-limit=20 Distortion KENLM lazyken=1 name=LM0 factor=0
> path=europarl.kenlm order=5
> input-factors: 0
> mapping: 0 T 0
> max-phrase-length: 20
> n-best-list: nbest.txt 111
> output-hypo-score: 1
> search-algorithm: 0
> server:
> threads: 1
> weight: Distortion0= 0.0222366 LM0= 0.0834208 WordPenalty0= -0.0654626
> PhrasePenalty0= 0.0220686 TranslationModel0= 0.0520176 0.0415173 0.124293
> 0.027126 UnknownWordPenalty0= 1
>
> line=UnknownWordPenalty
> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
> line=WordPenalty
> FeatureFunction: WordPenalty0 start: 1 end: 1
> line=PhrasePenalty
> FeatureFunction: PhrasePenalty0 start: 2 end: 2
> line=ProbingPT name=TranslationModel0 num-features=4
> path=phrase-table.2.pruned100.probing/ input-factor=0 output-factor=0
> table-limit=20
> FeatureFunction: TranslationModel0 start: 3 end: 6
> line=Distortion
> FeatureFunction: Distortion0 start: 7 end: 7
> line=KENLM lazyken=1 name=LM0 factor=0 path=europarl.kenlm order=5
> FeatureFunction: LM0 start: 8 end: 8
> Loading UnknownWordPenalty0
> Loading WordPenalty0
> Loading PhrasePenalty0
> Loading Distortion0
> Loading LM0
> Loading TranslationModel0
> Initialized successfully!
>
> RUN SERVER at pid 1733327039
> [moses/server/Server.cpp:49] Listening on port 8080
> [moses/server/TranslationRequest.cpp:281] Input: il a souhait? que la
> pr?sidence trace ? nice le chemin pour l' avenir .
> Translating: il a souhait? que la pr?sidence trace ? nice le chemin pour
> l' avenir .
>
> Line 0: Collecting options took 0.038 seconds at moses/Manager.cpp Line 141
> Line 0: Search took 0.674 seconds
> terminate called after throwing an instance of 'util::Exception'
> what(): No factor 1 at position 0


Any idea what's going on here? I'm using a basic single-factor model, so I
don't get why it would be complaining about factors. I'm using the latest
moses from git.

Thanks,
Lane
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160111/759a38de/attachment-0001.html

------------------------------

Message: 3
Date: Mon, 11 Jan 2016 23:02:24 +0100
From: Ergun Bicici <ergun.bicici@dfki.de>
Subject: Re: [Moses-support] Which symal?
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAB2pGnfjw+PtY+QP6PXkE_k6YMfeceXwenzvk0YhSxLr1ZTDog@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Warnings from bjam could be nice. Maybe from a previous installation
instruction set. If it is not used by bjam, makes sense to remove remaining
references:
https://github.com/moses-smt/mosesdecoder/blob/master/cruise-control/test_all_new_commits.sh


*Best Regards,*
Ergun

Ergun Bi?ici
DFKI Projektb?ro Berlin


On Mon, Jan 11, 2016 at 9:56 PM, Hieu Hoang <hieuhoang@gmail.com> wrote:

> I'm not sure if the bjam argument
> --with-giza
> is actually used during compilation. Where did you see this mention? The
> bad thing about bjam is it doesn't tell you if an argument is invalid, it
> simply ignores it.
>
> It would be nice to have 1 directory for all your MT tools. If you wanna
> make it happen, be my guest.
>
> I suppose we should be mindful of people who use mgiza but don't use
> Moses, they'll still need symal so we can't just delete it from mgiza.
>
> On 11/01/16 18:04, Ergun Bicici wrote:
>
>
> ?Hi Hieu?,
>
> Since
> ?
> the path to mgiza is provided
> ? during compilation, could be nice that moses knows where to look for
> mgiza afterwards without additional path directives and environment
> variable settings (?e.g. ?export BINDIR=~/workspace/bin/training-tools?
> in <http://www.statmt.org/moses/?n=Moses.ExternalTools#ntoc3>
> http://www.statmt.org/moses/?n=Moses.ExternalTools#ntoc3 instead,
> I am using the following for external-bin-dir in EMS: external-bin-dir =
> $moses-install-dir/bin).
>
> This also did not appear to work for tools in opt/ compiled with make -f
> contrib/Makefiles/install-dependencies.gmake. I still added opt/lib and
> opt/bin to LD_LIBRARY_PATH and PATH respectively.
>
> Having all binaries and libraries in the same place may decrease confusion
> as well and can prevent further confusion if a file with the same name
> appears in both paths. A compilation procedure that puts these directives
> together (dependency compilation input to bjam) while reducing path
> directives may help simplify installation.
>
> *Best Regards,*
> Ergun
>
> Ergun Bi?ici
> DFKI Projektb?ro Berlin
>
>
> On Mon, Jan 11, 2016 at 12:02 PM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>
>> you shouldn't copy anything into the moses/bin directory.
>>
>> The mgiza files should have its own directory. When you run Moses'
>> train-model.perl you can refer to that directory using
>> .../train-model.perl external-bin-dir=[directory with mgiza]
>>
>>
>> On 10/01/16 17:20, Ergun Bicici wrote:
>>
>>
>> Hi Hieu,
>>
>> First a compile:
>> ./bjam --max-kenlm-order=10 --git --prefix=/path/moses/mosesdecoder/
>> --with-giza=/path/mgiza/mgizapp/inst/ --with-xmlrpc-c=/path/moses/mosesdecoder/opt/
>> --with-boost=/path/moses/mosesdecoder/opt/ --with-cmph=/path/moses/mosesdecoder/opt/
>> -j 20
>>
>> then, a copy:
>> cp mgiza/mgizapp/inst/bin/* moses/mosesdecoder/instdir/bin/
>> cp mgiza/mgizapp/inst/lib/* moses/mosesdecoder/instdir/lib/
>> cp mgiza/mgizapp/inst/scripts/* moses/mosesdecoder/instdir/bin/
>>
>> With which another copy appears to be needed to use Moses' symal:
>> cp
>> moses/mosesdecoder/symal/bin/gcc-4.8/release/link-static/threading-multi/symal
>> moses/mosesdecoder/bin/symal
>>
>> Therefore, even
>> ? ?
>> if the path to mgiza is provided (--with-giza=/path/mgiza/mgizapp/inst/),
>> some copying and updated appear to be needed (see also
>> ? ?
>> <http://www.statmt.org/moses/?n=Moses.ExternalTools#ntoc3>
>> http://www.statmt.org/moses/?n=Moses.ExternalTools#ntoc3).
>>
>>
>> *Best Regards,*
>> Ergun
>>
>> Ergun Bi?ici
>> DFKI Projektb?ro Berlin
>>
>>
>> On Sun, Jan 10, 2016 at 3:34 PM, Hieu Hoang < <hieuhoang@gmail.com>
>> hieuhoang@gmail.com> wrote:
>>
>>> What the exact commands u used to compile moses and mgiza? I'm pretty
>>> sure they don't overwrite each other unless you ask them too. They're
>>> independent projects
>>> On 10 Jan 2016 14:07, "Ergun Bicici" < <ergun.bicici@dfki.de>
>>> ergun.bicici@dfki.de> wrote:
>>>
>>>>
>>>> Hi,
>>>>
>>>> I compiled another Moses instance and symal appears to be copied from
>>>> mgiza still to mosesdecoder/bin/. ?
>>>>
>>>>
>>>> *Best Regards,*
>>>> Ergun
>>>>
>>>> Ergun Bi?ici
>>>> DFKI Projektb?ro Berlin
>>>>
>>>>
>>>> On Sun, May 17, 2015 at 2:47 PM, Ergun Bicici <
>>>> <Ergun.Bicici@computing.dcu.ie>Ergun.Bicici@computing.dcu.ie> wrote:
>>>>
>>>>>
>>>>> Moses' symal:
>>>>> <http://article.gmane.org/gmane.comp.nlp.moses.user/11544>
>>>>> http://article.gmane.org/gmane.comp.nlp.moses.user/11544
>>>>>
>>>>>
>>>>> Best Regards,
>>>>> Ergun
>>>>>
>>>>> Ergun Bi?ici, CNGL, School of Computing, DCU, <http://www.cngl.ie>
>>>>> www.cngl.ie
>>>>> <http://www.computing.dcu.ie/%7Eebicici/>
>>>>> http://www.computing.dcu.ie/~ebicici/
>>>>>
>>>>>
>>>>> On Sun, May 17, 2015 at 1:15 PM, Jeroen Vermeulen <
>>>>> <jtv@precisiontranslationtools.com>jtv@precisiontranslationtools.com>
>>>>> wrote:
>>>>>
>>>>>> The symal source code is duplicated between the moses-smt and mgiza
>>>>>> repositories. Does it make sense to have both? They're quietly
>>>>>> diverging, which is probably a bad thing.
>>>>>>
>>>>>> Here's the differences that I can see:
>>>>>> * I modernized the code in moses-smt. Big diff, no functional
>>>>>> change.
>>>>>> * The moses-smt version supports longer source and target strings.
>>>>>> * The mgiza version has what looks like some extra debug output.
>>>>>> * The moses-smt version avoids non-portable use of /dev/stdout.
>>>>>> * One builds through bjam, the other through cmake.
>>>>>>
>>>>>> Could we perhaps just delete the mgiza one, and tell people to use the
>>>>>> one from moses instead?
>>>>>>
>>>>>>
>>>>>> Jeroen
>>>>>> _______________________________________________
>>>>>> Moses-support mailing list
>>>>>> <Moses-support@mit.edu>Moses-support@mit.edu
>>>>>> <http://mailman.mit.edu/mailman/listinfo/moses-support>
>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>>
>>
>> --
>> Hieu Hoanghttp://www.hoang.co.uk/hieu
>>
>>
>
> --
> Hieu Hoanghttp://www.hoang.co.uk/hieu
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20160111/e69e245c/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 111, Issue 21
**********************************************

Tweet