Moses-support Digest, Vol 153, Issue 12

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: ERROR: Wrong format of --lm. Expected: --lm
factor:order:filename //train-model.perl line 605. (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Tue, 30 Jul 2019 11:58:14 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] ERROR: Wrong format of --lm. Expected:
--lm factor:order:filename //train-model.perl line 605.
To: Aaron Poet <hanlifengaaron@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbjaF7Hj9wXK5Z3ZByqvYS7S48NdaK6Cyo0bULLm-Ake=Q@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

you're gonna have to do some debugging yourself. Perhaps look inside
train_Moses.job and training.out

Hieu Hoang
http://statmt.org/hieu


On Tue, 30 Jul 2019 at 11:48, Aaron Poet <hanlifengaaron@gmail.com> wrote:

> Hi Hieu
> Thanks for the reply. But, after I added the colon there, it still have
> some problem in running.
> *When I tried to run directly on the note of server it shows below:*
>
> (tensorflow) aaronhan@grove2:~/Moses/working$ nohup nice
> /home/aaronhan/Moses/mosesdecoder/scripts/training/train-model.perl
> -root-dir train -corpus
> /home/aaronhan/NMT/THUMT/corpora/preprocessed_wmt18en5maa.true -f zh -e en
> -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm
> 0:3:$HOME/Moses/lm/wmt18_5maaZhEn.blm.en:8 -external-bin-dir
> /home/aaronhan/Moses/mosesdecoder/tools/ >& training.out &
>
> [1] 26177
>
> (tensorflow) aaronhan@grove2:~/Moses/working$
>
> It produced a folder 'train' and sub-folder 'corpus' and the empty files "en.vcb.classes
> en.vcb.classes.cats zh.vcb.classes zh.vcb.classes.cats"
>
> (tensorflow) aaronhan@grove2:~/Moses/working$ ls
>
> slurm-23199.out slurm-23201.out *train* *train_Moses.job*
>
> slurm-23200.out slurm-23202.out training.out
>
> (tensorflow) aaronhan@grove2:~/Moses/working$ cd train/
>
> (tensorflow) aaronhan@grove2:~/Moses/working/train$ ls
>
> *corpus*
>
> (tensorflow) aaronhan@grove2:~/Moses/working/train$ ls corpus/
>
> en.vcb.classes en.vcb.classes.cats zh.vcb.classes zh.vcb.classes.cats
>
> (tensorflow) aaronhan@grove2:~/Moses/working/train$
>
>
> *When I submitted this command as job file to server, it does not appear
> in the job list/ doesn't to look like running.*
>
> Do you have any clue for this, please?
>
> Kind regards,
>
>
>
> On Tue, Jul 30, 2019 at 8:45 AM Hieu Hoang <hieuhoang@gmail.com> wrote:
>
>>
>>
>> On Mon, 29 Jul 2019, 19:58 Aaron Poet, <hanlifengaaron@gmail.com> wrote:
>>
>>> Hi Moses-Support
>>>
>>> I firstly build the lm as below:
>>>
>>> (base) aaronhan@grove2:~/Moses$ head -20
>>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.arpa.en
>>>
>>> \data\
>>>
>>> ngram 1=553697
>>>
>>> ngram 2=8790963
>>>
>>> ngram 3=28838697
>>>
>>>
>>> \1-grams:
>>>
>>> -6.993945 <unk> 0
>>>
>>> 0 <s> -1.6929523
>>>
>>> -2.0611665 </s> 0
>>>
>>> -2.3942525 the -0.9461817
>>>
>>> -3.6470015 show -0.6162386
>>>
>>> -4.1808434 stars -0.5792577
>>>
>>> -4.1823444 X -0.53191173
>>>
>>> -5.0213814 Girls -0.33374956
>>>
>>> -2.5064495 - -0.6326866
>>>
>>> -2.5754578 a -0.9741243
>>>
>>> -5.1265607 troupe -0.3527482
>>>
>>> -2.2843323 of -0.91076595
>>>
>>> -4.663717 talented -0.3983
>>>
>>> -5.2061567 topless -0.27003714
>>>
>>> Then, I binary it as bellow:
>>>
>>> /home/aaronhan/Moses/mosesdecoder/bin/build_binary
>>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.arpa.en
>>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>>>
>>> (base) aaronhan@grove2:~/Moses$ ls -lh
>>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>>>
>>> -rw-r--r-- 1 aaronhan users 715M Jul 29 18:38
>>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>>>
>>> when I test it, it looks working out:
>>>
>>> (tensorflow) aaronhan@grove2:~/Moses/working$ echo "is this an English
>>> sentence ?" \
>>>
>>> > | /home/aaronhan/Moses/mosesdecoder/bin/query
>>> /home/aaronhan/Moses/lm/wmt18_5maaZhEn.blm.en
>>>
>>> is=23 2 -2.5766807 this=321 3 -1.0483611 an=108 3 -2.1032705
>>> English=1731 2 -2.8970845 sentence=3266 3 -2.8317716 ?=815 2
>>> -2.0824053 </s>=2 3 -0.035550244 Total: -13.575124 OOV: 0
>>>
>>> Perplexity including OOVs: 86.9567697150713
>>>
>>> Perplexity excluding OOVs: 86.9567697150713
>>>
>>> OOVs: 0
>>>
>>> Tokens: 7
>>>
>>> Name:query VmPeak:748300 kB VmRSS:4052 kB RSSMax:503644
>>> kB user:0.008 sys:0.652 CPU:0.66 real:61.9797
>>>
>>> However, when I run the overall Moses training, it produced error as
>>> below: *Wrong format of --lm. Expected: --lm factor:order:filename*
>>>
>>> (tensorflow) aaronhan@grove2:~/Moses/working$ nohup nice
>>> /home/aaronhan/Moses/mosesdecoder/scripts/training/train-model.perl
>>> -root-dir train -corpus
>>> /home/aaronhan/NMT/THUMT/corpora/preprocessed_wmt18en5maa.true -f zh -e en
>>> -alignment grow-diag-final-and -reordering msd-bidirectional-fe -lm
>>> 0:3$HOME/Moses/lm/wmt18_5maaZhEn.blm.en:8 -external-bin-dir
>>> /home/aaronhan/Moses/mosesdecoder/tools/ >& training.out &
>>>
>>> [1] 23346
>>>
>>> You forgot a colon between 3 and $HOME
>>
>>>
>>> GNU nano 2.7.4
>>> File: training.out
>>>
>>>
>>>
>>> nohup: ignoring input
>>>
>>> Using SCRIPTS_ROOTDIR: /home/aaronhan/Moses/mosesdecoder/scripts
>>>
>>> Using single-thread GIZA
>>>
>>> using gzip
>>>
>>> ERROR: Wrong format of --lm. Expected: --lm factor:order:filename at
>>> /home/aaronhan/Moses/mosesdecoder/scripts/training/train-model.perl line
>>> 605.
>>>
>>>
>>>
>>> Can you let me know how to solve this, please?
>>>
>>> Kind regards,
>>> Aaron
>>>
>>> --
>>>
>>> PhD student, Msc. (excellent), Bsc. (Maths)
>>>
>>> Google-Scholar
>>> <https://scholar.google.nl/citations?user=_vf3E2QAAAAJ&hl=en> ,
>>> Presentation <https://www.slideshare.net/AaronHanLiFeng>-slides,
>>> Research-Gate <https://www.researchgate.net/profile/Aaron_L-F_Han>
>>>
>>> Google-site <https://sites.google.com/view/poetgarden/home> Linkedin
>>> <https://www.linkedin.com/in/aaronhan/>,
>>>
>>> ADAPT <https://www.adaptcentre.ie/> ResearchCentre &
>>>
>>> School of Computing, DCU,
>>>
>>> Dublin, Ireland
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>
>
> --
>
> PhD student, Msc. (excellent), Bsc. (Maths)
>
> Google-Scholar
> <https://scholar.google.nl/citations?user=_vf3E2QAAAAJ&hl=en> ,
> Presentation <https://www.slideshare.net/AaronHanLiFeng>-slides,
> Research-Gate <https://www.researchgate.net/profile/Aaron_L-F_Han>
>
> Google-site <https://sites.google.com/view/poetgarden/home> Linkedin
> <https://www.linkedin.com/in/aaronhan/>,
>
> ADAPT <https://www.adaptcentre.ie/> ResearchCentre &
>
> School of Computing, DCU,
>
> Dublin, Ireland
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20190730/5d499d26/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 153, Issue 12
**********************************************

Tweet