Moses-support Digest, Vol 86, Issue 13

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Warning: Too many arguments while IRSTLM language model
Training (Prashant Mathur)
2. Re: Word-to-word alignment when moses-parallel.pl is used
(Hieu Hoang)
3. Re: Word-to-word alignment when moses-parallel.pl is used
(Thomas Meyer)


----------------------------------------------------------------------

Message: 1
Date: Thu, 05 Dec 2013 12:21:40 +0100
From: Prashant Mathur <prashant@fbk.eu>
Subject: Re: [Moses-support] Warning: Too many arguments while IRSTLM
language model Training
To: renubalyan <renubalyan@cdac.in>
Cc: moses-support@mit.edu
Message-ID: <52A061C4.6040509@fbk.eu>
Content-Type: text/plain; charset="iso-8859-1"

The command is correct, so it should work. You are using compile-lm to
convert from iARPA to ARPA format.
Try updating irstlm toolkit to latest version.

--
Prashant

On 12/04/2013 04:58 PM, renubalyan wrote:
> Hi,
> I am building the baseline system based on Moses manual instructions.
> I have installed Moses, GIZA++ and IRSTLM as mentioned in the manual.
> The corpus preparation (tokenization, ...cleaning) steps also goes well.
> However when I move to Language Model Training: I have some problems
> I am following these steps:
> 1. mkdir ~/lm
>
> 2. cd ~/lm
>
> 3. /home/renu/Desktop/irstlm/bin/add-start-end.sh <
> /home/renu/Desktop/corpus/news-commentary-v8.fr-en.true.en>
> news-commentary-v8.fr-en.sb.en
>
> 4. export IRSTLM=/home/renu/Desktop/irstlm;
> /home/renu/Desktop/irstlm/bin/build-lm.sh -i
> news-commentary-v8.fr-en.sb.en -t ./tmp -p -s improved-kneser-ney -o
> news-commentary-v8.fr-en.lm.en
>
> 5. /home/renu/Desktop/irstlm/bin/compile-lm --text yes
> news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
> Steps 1-4 work well but step 5 gives me -------(Warning:Too many
> parameters)
>
> I have searched the web for any possible solution but could not find any.
> I am not able to move ahead, kindly help.
> Thanks
> Renu
>
> -------------------------------------------------------------------------------------------------------------------------------
>
> This e-mail is for the sole use of the intended recipient(s) and may
> contain confidential and privileged information. If you are not the
> intended recipient, please contact the sender by reply e-mail and destroy
> all copies and the original message. Any unauthorized review, use,
> disclosure, dissemination, forwarding, printing or copying of this email
> is strictly prohibited and appropriate legal action will be taken.
> -------------------------------------------------------------------------------------------------------------------------------
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131205/fc07a35e/attachment-0001.htm

------------------------------

Message: 2
Date: Thu, 5 Dec 2013 11:30:25 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Word-to-word alignment when
moses-parallel.pl is used
To: Dimitris Mavroeidis <dmavroeidis@csri.gr>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbgSc3W=4pB_hVTcJ4W23y2ZZJsL+CizrNtBh1_e1rwHcA@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-7"

as far as I know, it should work with the latest version. What is the
error? I can help you fix it if it's broken, but i can't do it myself


On 2 December 2013 09:10, Dimitris Mavroeidis <dmavroeidis@csri.gr> wrote:

> Hi all,
>
> We still use SGE for moses, since we have a cluster of low-end machines.
> It is very convenient for us.
>
> It would be nice if we could use SGE with the _latest_ version of moses.
> So, Thomas, feel free to share your modifications with the rest of the list.
>
> Best regards,
> Dimitris
>
>
>
> On 29/11/2013 08:52 ??, Thomas Meyer wrote:
>
> Hi all,
>
> I 'still' do and find it quite convenient. At least with release 1.0 and
> some modifications to moses-parallel.pl
>
> Thomas
>
>
> *From: *CHATZITHEODOROU Konstantinos
> *Sent: *Friday, 29 November 2013 7:48 PM
> *To: *Hieu Hoang
> *Cc: *moses-support@mit.edu
> *Subject: *Re: [Moses-support] Word-to-word alignment when
> moses-parallel.pl
> is used
>
> Hi Hieu,
>
> Thanks for your reply. I have a question. Why not many people still use
> SGE to run Moses?
>
> Thank you in advance.
>
> Best,
> Konstantinos
>
>
> 2013/11/29 Hieu Hoang <Hieu.Hoang@ed.ac.uk>
>
>> I'm not sure if many people still uses Sun Grid Engine to run Moses. If
>> not, scripts like moses-parallel.pl may become outdated.
>>
>> I know at Edinburgh, we no longer use SGE. Instead we run our experiment
>> on 1 large multi-core machine. Most of moses has been re-engineered to be
>> multithreaded to take advantage of multicores
>>
>>
>> On 28 November 2013 10:25, CHATZITHEODOROU Konstantinos <
>> dinosaxon@gmail.com> wrote:
>>
>>> Hi Philipp,
>>>
>>> Thank you your your prompt reply. I have tried the option
>>> "alignment-output-file" for moses-parallel.pl but didn't use it as part
>>> of the decoder setting. But it didn't work. Each individual sub-job
>>> finished with an error status stating something the sub-job is
>>> untranslated.
>>>
>>> Thank you for your time and consideration.
>>>
>>> Regards,
>>> Konstantinos
>>>
>>>
>>>
>>> 2013/11/27 Philipp Koehn <pkoehn@inf.ed.ac.uk>
>>>
>>>> Hi,
>>>>
>>>> the script has an option "alignment-output-file" - did you try this?
>>>>
>>>> -phi
>>>>
>>>> On Tue, Nov 26, 2013 at 8:13 PM, CHATZITHEODOROU Konstantinos
>>>> <dinosaxon@gmail.com> wrote:
>>>> > Hi all,
>>>> >
>>>> > Could you please give an example on how to output the word-to-word
>>>> alignment
>>>> > during decoding when moses-parallel.pl is used?
>>>> >
>>>> > Thank you in advance for your help.
>>>> >
>>>> > Regards,
>>>> > Konstantinos
>>>> >
>>>> >
>>>> > _______________________________________________
>>>> > Moses-support mailing list
>>>> > Moses-support@mit.edu
>>>> > http://mailman.mit.edu/mailman/listinfo/moses-support
>>>> >
>>>>
>>>
>>>
>>>
>>> --
>>> CHATZITHEODOROU Konstantinos
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>
>>
>> --
>> Hieu Hoang
>> Research Associate
>> University of Edinburgh
>> http://www.hoang.co.uk/hieu
>>
>>
>
>
> --
> CHATZITHEODOROU Konstantinos
>
>
>
>
> _______________________________________________
> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131205/8f982248/attachment-0001.htm

------------------------------

Message: 3
Date: Thu, 05 Dec 2013 12:54:23 +0100
From: Thomas Meyer <Thomas.Meyer@idiap.ch>
Subject: Re: [Moses-support] Word-to-word alignment when
moses-parallel.pl is used
To: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <52A0696F.1050705@idiap.ch>
Content-Type: text/plain; charset="iso-8859-1"

Hi all,

Merged and tested. All works for me. I attach the latest github
moses-parallel.pl wherein my changes are commented out and to be found
under #SGE-make-it-work:

Hope it helps,
Thomas

On 05/12/13 12:05, Hieu Hoang wrote:
> The script seems to have been forked just before the last changes
>
> https://github.com/moses-smt/mosesdecoder/commit/93433cf0157ecadd19162929a077aa6179dc0d8b
>
> If someone who uses SGE can do a merge with the latest version in
> github and test that it works, I can check it in
>
>
>
> On 2 December 2013 09:03, Thomas Meyer <Thomas.Meyer@idiap.ch
> <mailto:Thomas.Meyer@idiap.ch>> wrote:
>
> Hi all,
>
> Attached is moses-parallel.pl <http://moses-parallel.pl> with some
> modifications (marked #thm:) to make it work with the SGE
> architecture we have here. I just ran a test whether it outputs
> the word alignment and all works normally.
>
> Here is the command I used:
>
> ./moses-parallel.pl <http://moses-parallel.pl> -jobs 2
> -alignment-output-file alignment.txt -input-file
> /temp/tmeyer/test.en -decoder /temp/tmeyer/mosesdecoder/bin/moses
> -config /temp/tmeyer/en_de/test/moses.tuned.ini > test.txt
>
> which outputs, in alignment.txt:
>
> 0-0 1-1 2-2 3-3 4-4 5-5 6-6 7-7 8-8 9-9 10-10 11-11 12-12 13-13
> 14-14 15-15 16-16 17-17 18-18 19-19 20-20 21-21 22-22 23-23 24-24
> 0-0 1-1 2-2 3-3 4-4 5-4 6-5 7-6 8-7 9-8 10-9 11-10 12-11 13-12
> 13-13 14-14 16-15 17-16 18-17 20-18 21-19 22-20 23-21 24-22 25-23
> 26-24 27-25 28-26 29-27 30-28 31-28 32-29 33-30 34-31 36-32 37-33
> 38-34 39-35
>
> etc.
>
> Hope it will work for you, too, Konstantinos.
>
> Best,
> Thomas
>
>
>
> On 29/11/13 21:59, Philipp Koehn wrote:
>> Hi Thomas,
>>
>> in that case - can you share your modifications if they are of general
>> interest and check if there is a problem with the word alignment reporting?
>>
>> -phi
>>
>> On Fri, Nov 29, 2013 at 6:52 PM, Thomas Meyer <Thomas.Meyer@idiap.ch> <mailto:Thomas.Meyer@idiap.ch> wrote:
>>> Hi all,
>>>
>>> I 'still' do and find it quite convenient. At least with release 1.0 and
>>> some modifications to moses-parallel.pl <http://moses-parallel.pl>
>>>
>>> Thomas
>>>
>>>
>>> From: CHATZITHEODOROU Konstantinos
>>> Sent: Friday, 29 November 2013 7:48 PM
>>> To: Hieu Hoang
>>> Cc: moses-support@mit.edu <mailto:moses-support@mit.edu>
>>> Subject: Re: [Moses-support] Word-to-word alignment when moses-parallel.pl <http://moses-parallel.pl>
>>> is used
>>>
>>> Hi Hieu,
>>>
>>> Thanks for your reply. I have a question. Why not many people still use SGE
>>> to run Moses?
>>>
>>> Thank you in advance.
>>>
>>> Best,
>>> Konstantinos
>>>
>>>
>>> 2013/11/29 Hieu Hoang <Hieu.Hoang@ed.ac.uk> <mailto:Hieu.Hoang@ed.ac.uk>
>>>> I'm not sure if many people still uses Sun Grid Engine to run Moses. If
>>>> not, scripts like moses-parallel.pl <http://moses-parallel.pl> may become outdated.
>>>>
>>>> I know at Edinburgh, we no longer use SGE. Instead we run our experiment
>>>> on 1 large multi-core machine. Most of moses has been re-engineered to be
>>>> multithreaded to take advantage of multicores
>>>>
>>>>
>>>> On 28 November 2013 10:25, CHATZITHEODOROU Konstantinos
>>>> <dinosaxon@gmail.com> <mailto:dinosaxon@gmail.com> wrote:
>>>>> Hi Philipp,
>>>>>
>>>>> Thank you your your prompt reply. I have tried the option
>>>>> "alignment-output-file" for moses-parallel.pl <http://moses-parallel.pl> but didn't use it as part of
>>>>> the decoder setting. But it didn't work. Each individual sub-job finished
>>>>> with an error status stating something the sub-job is untranslated.
>>>>>
>>>>> Thank you for your time and consideration.
>>>>>
>>>>> Regards,
>>>>> Konstantinos
>>>>>
>>>>>
>>>>>
>>>>> 2013/11/27 Philipp Koehn <pkoehn@inf.ed.ac.uk> <mailto:pkoehn@inf.ed.ac.uk>
>>>>>> Hi,
>>>>>>
>>>>>> the script has an option "alignment-output-file" - did you try this?
>>>>>>
>>>>>> -phi
>>>>>>
>>>>>> On Tue, Nov 26, 2013 at 8:13 PM, CHATZITHEODOROU Konstantinos
>>>>>> <dinosaxon@gmail.com> <mailto:dinosaxon@gmail.com> wrote:
>>>>>>> Hi all,
>>>>>>>
>>>>>>> Could you please give an example on how to output the word-to-word
>>>>>>> alignment
>>>>>>> during decoding when moses-parallel.pl <http://moses-parallel.pl> is used?
>>>>>>>
>>>>>>> Thank you in advance for your help.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Konstantinos
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Moses-support mailing list
>>>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>>>
>>>>>
>>>>> --
>>>>> CHATZITHEODOROU Konstantinos
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Moses-support mailing list
>>>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>>
>>>> --
>>>> Hieu Hoang
>>>> Research Associate
>>>> University of Edinburgh
>>>> http://www.hoang.co.uk/hieu
>>>>
>>> --
>>> CHATZITHEODOROU Konstantinos
>>>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131205/4fba4fee/attachment.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: moses-parallel.pl
Type: application/x-perl
Size: 31012 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20131205/4fba4fee/attachment.bin

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 86, Issue 13
*********************************************

Tweet