Moses-support Digest, Vol 126, Issue 33

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Did anyone tried Edinburgh English-German syntax system
for WMT 2015? (Rico Sennrich)
2. Re: Discount fallback in KenLM on super small file (liling tan)
3. EMS with multiple language models (K Pavan)
4. Re: Did anyone tried Edinburgh English-German syntax system
for WMT 2015? (Rico Sennrich)


----------------------------------------------------------------------

Message: 1
Date: Mon, 24 Apr 2017 08:53:50 +0100
From: Rico Sennrich <rico.sennrich@gmx.ch>
Subject: Re: [Moses-support] Did anyone tried Edinburgh English-German
syntax system for WMT 2015?
To: moses-support@mit.edu
Message-ID: <9cdab0ac-42a4-773b-db1d-2a5e37df38d5@gmx.ch>
Content-Type: text/plain; charset="windows-1252"

Hello Xin,

what is the error message for the filtering step? Look at the STDERR and
STDOUT files that are being produced in the step to see if there's an
error message in there.

best wishes,
Rico

On 20/04/17 12:41, dai xin wrote:
> Hi,
>
> Did anyone have experience of Edinburgh English-German syntax system
> for WMT 2015 ?
>
> The scripts, configs and data can be accessed in
> https://github.com/rsennrich/wmt2014-scripts .
>
> I am doing experiments using the configs described there. But after
> trying for many times(using both toy-data and my own data), I always
> got stuck at 'flitering' step.
>
> Does anyone have luck with that? Or give me some instructions how to
> do tuning munually? From EMS I already trained language and
> translation models, the only steps left and got stuck is tuning.
>
> Thanks in advance and hoping for reply.
>
> Best regards,
>
> Xin
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170424/e564d3e8/attachment-0001.html

------------------------------

Message: 2
Date: Mon, 24 Apr 2017 16:18:25 +0800
From: liling tan <alvations@gmail.com>
Subject: Re: [Moses-support] Discount fallback in KenLM on super small
file
To: Kenneth Heafield <me@kheafield.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAKzPaJ+YRhRSGCD8QkAOovUAZe+9Hx_PW31W3omxcEJdzYbUWQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Thanks Kenneth for the answer!!

On Mon, Apr 24, 2017 at 2:48 PM, Kenneth Heafield <me@kheafield.com> wrote:

> Yes. Though formally I would say in range.
>
> On April 24, 2017 4:18:41 AM GMT+01:00, liling tan <alvations@gmail.com>
> wrote:
>>
>> Dear Moses community,
>>
>> Is it correct that when using --discount_fallback, if discount is
>> computable from Kneyser-Ney, the fallback will not be used?
>>
>>
>> Regards,
>> Liling
>>
>> For context:
>>
>>
>>
>> $ cat test.zh
>>
>> ? ? ? ?
>>
>>
>> ~$ time ~/mosesdecoder/bin/lmplz -o 3 --discount_fallback=0 < test.zh >
>> test.zh.arpa
>>
>>
>>
>> === 1/5 Counting and sorting n-grams === Reading /home/ltan/test.zh
>>
>> ----5---10---15---20---25---30---35---40---45---50---55---
>> 60---65---70---75---80---85---90---95--100
>>
>> ************************************************************
>> ****************************************
>>
>> Unigram tokens 4 types 7
>>
>>
>>
>> === 2/5 Calculating and sorting adjusted counts
>>
>>
>>
>> === Chain sizes: 1:84 2:75238187008 3:141071597568
>>
>>
>>
>> Substituting fallback discounts for order 0: D1=0 D2=0 D3+=0
>>
>> Substituting fallback discounts for order 1: D1=0 D2=0 D3+=0
>>
>> Substituting fallback discounts for order 2: D1=0 D2=0 D3+=0
>>
>> Statistics:
>>
>> 1 7 D1=0 D2=0 D3+=0
>>
>> 2 5 D1=0 D2=0 D3+=0
>>
>> 3 4 D1=0 D2=0 D3+=0
>>
>> Memory estimate for binary LM:
>>
>> type B probing 376 assuming -p 1.5 probing 436 assuming -r models
>> -p 1.5 trie 298 without quantization trie 3328 assuming -q 8 -b 8
>> quantization trie 321 assuming -a 22 array pointer compression trie
>> 3351 assuming -a 22 -q 8 -b 8 array pointer compression and quantization
>>
>>
>>
>> === 3/5 Calculating and sorting initial probabilities ===
>>
>> Chain sizes: 1:84 2:80 3:80
>>
>> ----5---10---15---20---25---30---35---40---45---50---55---
>> 60---65---70---75---80---85---90---95--100
>>
>> ############################################################
>> ########################################
>>
>>
>>
>> === 4/5 Calculating and writing order-interpolated probabilities ===
>>
>> Chain sizes: 1:84 2:80 3:80 ----5---10---15---20---25---
>> 30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>> ############################################################
>> ########################################
>>
>>
>>
>> === 5/5 Writing ARPA model ===
>>
>> Name:lmplz VmPeak:211512300 kB VmRSS:4740 kB
>> RSSMax:48754048 kB user:2.64 sys:159.492
>> CPU:162.132 real:162.148
>>
>>
>>
>> real 2m42.152s
>>
>> user 0m2.644s
>>
>> sys 2m39.492s
>>
>
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170424/b060d782/attachment-0001.html

------------------------------

Message: 3
Date: Mon, 24 Apr 2017 15:55:04 +0530
From: K Pavan <kosuru.pavan@gmail.com>
Subject: [Moses-support] EMS with multiple language models
To: moses-support@mit.edu
Message-ID:
<CAAQYKLLXV9fEtW8EcCnt3g-uw2OOQE5TWuTTM25rAy22n=ma1w@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi ,

I was experiment with EMS (experiment.perl). I have language model built on
input corpus also language model with all english corpus (say europal
corpora). So ideally I want my moes config should have LM0 and LM1 with
equal weight. I tried to add this as part of config in EMS, failing to do
so. Can somebody help me out here.


My training command looks like below if I do it manually

moses/scripts/training/train-model.perl -root-dir $WORKDIR \
--sort-buffer-size=4G \
-corpus $CORPUS \
-mgiza root/usr/bin/mgiza -mgiza-cpus 8 \
-f fr -e en -alignment grow-diag-final-and \
-reordering msd-bidirectional-fe \
* -lm 0:4:/lm/big-4gram.2012-02-02.en.lm.kenlm.trie.q8:8 *
*-lm 0:4:moses-fr-en_europal/lm/fr_en_europal.lm.kenlm.trie.q8* \
-score-options='--GoodTuring' \
--dont-zip \
--parallel"


Thanks,
Pavan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170424/427b8b63/attachment-0001.html

------------------------------

Message: 4
Date: Mon, 24 Apr 2017 15:33:44 +0100
From: Rico Sennrich <rico.sennrich@gmx.ch>
Subject: Re: [Moses-support] Did anyone tried Edinburgh English-German
syntax system for WMT 2015?
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <58fad0ea-18ac-f1e0-732d-22e485dfe485@gmx.ch>
Content-Type: text/plain; charset="utf-8"

Hello Xin,

I've fixed the problem with train-custom-syntax (commit 61f5b4) , and
may have fixed the problem with filtering (commit b99af3) - you should
pull the latest version of Moses, then try again.

There may be more regressions that have happened since 2015, and testing
takes a while.

best wishes,
Rico

On 24/04/17 12:30, dai xin wrote:
> Hi Rico,
>
> thanks again for the reply.
>
> In attachment, you can find my change to config file to do my
> experiments.
>
> Also, there are error messages:
>
> filtering error: can't find a file in tuning directory. I think it
> should be a file get from parser. But I have no idea why there is no
> file in this directory.
>
> RDLM:train-custom-syntax crashed.
>
> Is the problem could be lack of any libraries, or the version of Moses?
>
> Thanks in advance and hoping for reply.
>
> Best regards,
>
> Xin
>
> On Mon, Apr 24, 2017 at 9:53 AM, Rico Sennrich <rico.sennrich@gmx.ch
> <mailto:rico.sennrich@gmx.ch>> wrote:
>
> Hello Xin,
>
> what is the error message for the filtering step? Look at the
> STDERR and STDOUT files that are being produced in the step to see
> if there's an error message in there.
>
> best wishes,
> Rico
>
>
> On 20/04/17 12:41, dai xin wrote:
>> Hi,
>>
>> Did anyone have experience of Edinburgh English-German syntax
>> system for WMT 2015 ?
>>
>> The scripts, configs and data can be accessed in
>> https://github.com/rsennrich/wmt2014-scripts
>> <https://github.com/rsennrich/wmt2014-scripts> .
>>
>> I am doing experiments using the configs described there. But
>> after trying for many times(using both toy-data and my own data),
>> I always got stuck at 'flitering' step.
>>
>> Does anyone have luck with that? Or give me some instructions
>> how to do tuning munually? From EMS I already trained language
>> and translation models, the only steps left and got stuck is tuning.
>>
>> Thanks in advance and hoping for reply.
>>
>> Best regards,
>>
>> Xin
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>> <http://mailman.mit.edu/mailman/listinfo/moses-support>
> _______________________________________________ Moses-support
> mailing list Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
> <http://mailman.mit.edu/mailman/listinfo/moses-support>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170424/90d9396d/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 126, Issue 33
**********************************************

Tweet