Moses-support Digest, Vol 170, Issue 3

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: kenlm vs srilm (Philipp Koehn)
2. Re: hierarchical pbmt issue (Hieu Hoang)
3. Re: hierarchical pbmt issue (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Wed, 16 Dec 2020 13:50:24 -0500
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] kenlm vs srilm
To: Y-Anees <yusra.anees96@gmail.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDB13tXEs-a+p7QGSeE-WhGURaQCnxuBZJzvhNHER3+awA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Hi,

it is always better to use kenlm instead of srilm.

-phi

On Wed, Dec 16, 2020 at 4:29 AM Y-Anees <yusra.anees96@gmail.com> wrote:

>
> * External Email - Use Caution *
>
>
>
> Hi there,
> I am using Moses' baseline system. I have installed it. I installed it and
> it is working fine. moses default language model is KENLM?
> I want to know about which language model I should install. how to check
> its language model. ? Should I install SRILM. I am working on a small
> dataset, not a huge dataset. around 80ksentence in a parallel file.
>
> Thank you..
> Yusra
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20201216/4ec8a69d/attachment-0001.html

------------------------------

Message: 2
Date: Thu, 17 Dec 2020 02:31:25 -0800
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] hierarchical pbmt issue
To: ram anirudh cherukupally <ramanirudh28@gmail.com>,
moses-support@mit.edu
Message-ID: <3704c226-bc48-67a1-4f06-9d9d8d41d6d0@gmail.com>
Content-Type: text/plain; charset="utf-8"

its likely you have run out of disk space during the extraction of the
hierarchical phrase table. it requires about 10 times more disk space to
train than the standard phrase-nased models

On Thu Dec 17,20 2:10 AM, ram anirudh cherukupally wrote:
Hi Everyone

I have been trying to train a? hierarchical PBMT (without linguistic
annotation) system using moses, with the switch --hierarchical to
train-model.perl (as specified in the manual)

I see that rule-table is not being generated during the training. Phrase
table is being generated without any problem, but rule table is empty
(rule-table.gz) turns out to be empty.

Apparently, gzip is failing in Step 5. But I don't know why, because it
appears to work fine with vanilla PBMT. I am sending the training log
(stdout), Kindly help me.

--
CH Ram Anirudh

--
Hieu Hoang
http://statmt.org/hieu

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20201217/a58e345b/attachment-0001.html

------------------------------

Message: 3
Date: Thu, 17 Dec 2020 02:40:11 -0800
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] hierarchical pbmt issue
To: ram anirudh cherukupally <ramanirudh28@gmail.com>, moses-support
<moses-support@mit.edu>
Message-ID:
<CAEKMkbizpMPCUeM-pA692fGuRW4tak3WLacz_S2gxnXf75n41g@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

also, the lack of disk space may be in /tmp, not just your output directory.

i'm not sure exactly how you run the training but in the command
extract-parallel.perl, the 'sort' should use a temp directory that has lots
space. So change your command
...extract-parallel.perl 40 split "sort "....
to
...extract-parallel.perl 40 split "sort -T [directory with lots of
space]"....

Hieu Hoang
http://statmt.org/hieu


On Thu, 17 Dec 2020 at 02:31, Hieu Hoang <hieuhoang@gmail.com> wrote:

> its likely you have run out of disk space during the extraction of the
> hierarchical phrase table. it requires about 10 times more disk space to
> train than the standard phrase-nased models
> On Thu Dec 17,20 2:10 AM, ram anirudh cherukupally wrote:
> Hi Everyone
>
> I have been trying to train a hierarchical PBMT (without linguistic
> annotation) system using moses, with the switch --hierarchical to
> train-model.perl (as specified in the manual)
>
> I see that rule-table is not being generated during the training. Phrase
> table is being generated without any problem, but rule table is empty
> (rule-table.gz) turns out to be empty.
>
> Apparently, gzip is failing in Step 5. But I don't know why, because it
> appears to work fine with vanilla PBMT. I am sending the training log
> (stdout), Kindly help me.
>
> --
> CH Ram Anirudh
>
> --
> Hieu Hoanghttp://statmt.org/hieu
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20201217/0a7275d3/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 170, Issue 3
*********************************************

Tweet