Moses-support Digest, Vol 86, Issue 15

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Word-to-word alignment when moses-parallel.pl is used
(CHATZITHEODOROU Konstantinos)
2. Re: Warning: Too many arguments while IRSTLM language model
Training (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Thu, 5 Dec 2013 16:21:08 +0000
From: CHATZITHEODOROU Konstantinos <dinosaxon@gmail.com>
Subject: Re: [Moses-support] Word-to-word alignment when
moses-parallel.pl is used
To: Thomas Meyer <Thomas.Meyer@idiap.ch>
Cc: Hieu Hoang <hieu.hoang@ed.ac.uk>, "moses-support@mit.edu"
<moses-support@mit.edu>
Message-ID:
<CAB+50nMmcwCZ+a3VbgSyRk79raj=W2JkuSMuy+M+CD=F0kNasA@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Hi everybody,

Thank you so much for your replies to my question. Unfortunately, I didn't
have time to test the modified script. I hope to test it in the next few
days and will come back with my answer asap.

Regards
Konstantinos


2013/12/5 Thomas Meyer <Thomas.Meyer@idiap.ch>

> Hi all,
>
> Merged and tested. All works for me. I attach the latest github
> moses-parallel.pl wherein my changes are commented out and to be found
> under #SGE-make-it-work:
>
> Hope it helps,
> Thomas
>
> On 05/12/13 12:05, Hieu Hoang wrote:
>
> The script seems to have been forked just before the last changes
>
> https://github.com/moses-smt/mosesdecoder/commit/93433cf0157ecadd19162929a077aa6179dc0d8b
>
> If someone who uses SGE can do a merge with the latest version in github
> and test that it works, I can check it in
>
>
>
> On 2 December 2013 09:03, Thomas Meyer <Thomas.Meyer@idiap.ch> wrote:
>
>> Hi all,
>>
>> Attached is moses-parallel.pl with some modifications (marked #thm:) to
>> make it work with the SGE architecture we have here. I just ran a test
>> whether it outputs the word alignment and all works normally.
>>
>> Here is the command I used:
>>
>> ./moses-parallel.pl -jobs 2 -alignment-output-file alignment.txt
>> -input-file /temp/tmeyer/test.en -decoder
>> /temp/tmeyer/mosesdecoder/bin/moses -config
>> /temp/tmeyer/en_de/test/moses.tuned.ini > test.txt
>>
>> which outputs, in alignment.txt:
>>
>> 0-0 1-1 2-2 3-3 4-4 5-5 6-6 7-7 8-8 9-9 10-10 11-11 12-12 13-13 14-14
>> 15-15 16-16 17-17 18-18 19-19 20-20 21-21 22-22 23-23 24-24
>> 0-0 1-1 2-2 3-3 4-4 5-4 6-5 7-6 8-7 9-8 10-9 11-10 12-11 13-12 13-13
>> 14-14 16-15 17-16 18-17 20-18 21-19 22-20 23-21 24-22 25-23 26-24 27-25
>> 28-26 29-27 30-28 31-28 32-29 33-30 34-31 36-32 37-33 38-34 39-35
>>
>> etc.
>>
>> Hope it will work for you, too, Konstantinos.
>>
>> Best,
>> Thomas
>>
>>
>>
>> On 29/11/13 21:59, Philipp Koehn wrote:
>>
>> Hi Thomas,
>>
>> in that case - can you share your modifications if they are of general
>> interest and check if there is a problem with the word alignment reporting?
>>
>> -phi
>>
>> On Fri, Nov 29, 2013 at 6:52 PM, Thomas Meyer <Thomas.Meyer@idiap.ch> <Thomas.Meyer@idiap.ch> wrote:
>>
>> Hi all,
>>
>> I 'still' do and find it quite convenient. At least with release 1.0 and
>> some modifications to moses-parallel.pl
>>
>> Thomas
>>
>>
>> From: CHATZITHEODOROU Konstantinos
>> Sent: Friday, 29 November 2013 7:48 PM
>> To: Hieu Hoang
>> Cc: moses-support@mit.edu
>> Subject: Re: [Moses-support] Word-to-word alignment when moses-parallel.pl
>> is used
>>
>> Hi Hieu,
>>
>> Thanks for your reply. I have a question. Why not many people still use SGE
>> to run Moses?
>>
>> Thank you in advance.
>>
>> Best,
>> Konstantinos
>>
>>
>> 2013/11/29 Hieu Hoang <Hieu.Hoang@ed.ac.uk> <Hieu.Hoang@ed.ac.uk>
>>
>> I'm not sure if many people still uses Sun Grid Engine to run Moses. If
>> not, scripts like moses-parallel.pl may become outdated.
>>
>> I know at Edinburgh, we no longer use SGE. Instead we run our experiment
>> on 1 large multi-core machine. Most of moses has been re-engineered to be
>> multithreaded to take advantage of multicores
>>
>>
>> On 28 November 2013 10:25, CHATZITHEODOROU Konstantinos<dinosaxon@gmail.com> <dinosaxon@gmail.com> wrote:
>>
>> Hi Philipp,
>>
>> Thank you your your prompt reply. I have tried the option
>> "alignment-output-file" for moses-parallel.pl but didn't use it as part of
>> the decoder setting. But it didn't work. Each individual sub-job finished
>> with an error status stating something the sub-job is untranslated.
>>
>> Thank you for your time and consideration.
>>
>> Regards,
>> Konstantinos
>>
>>
>>
>> 2013/11/27 Philipp Koehn <pkoehn@inf.ed.ac.uk> <pkoehn@inf.ed.ac.uk>
>>
>> Hi,
>>
>> the script has an option "alignment-output-file" - did you try this?
>>
>> -phi
>>
>> On Tue, Nov 26, 2013 at 8:13 PM, CHATZITHEODOROU Konstantinos<dinosaxon@gmail.com> <dinosaxon@gmail.com> wrote:
>>
>> Hi all,
>>
>> Could you please give an example on how to output the word-to-word
>> alignment
>> during decoding when moses-parallel.pl is used?
>>
>> Thank you in advance for your help.
>>
>> Regards,
>> Konstantinos
>>
>>
>> _______________________________________________
>> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>> --
>> CHATZITHEODOROU Konstantinos
>>
>>
>> _______________________________________________
>> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>
>> --
>> Hieu Hoang
>> Research Associate
>> University of Edinburghhttp://www.hoang.co.uk/hieu
>>
>> --
>> CHATZITHEODOROU Konstantinos
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>
>> _______________________________________________
>> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>


--
CHATZITHEODOROU Konstantinos
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131205/59cd3c8c/attachment-0001.htm

------------------------------

Message: 2
Date: Thu, 5 Dec 2013 16:30:41 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Warning: Too many arguments while IRSTLM
language model Training
To: renubalyan <renubalyan@cdac.in>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbjEDE3WCC++fN+ySKoXsix+vNcN_hFXK+t8mBJPP4_ZpQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Sorry, I was wrong and Prashant was correct.
./compile-lm --text
creates the ARPA file.

Perhaps an easier way to create a LM using IRSTLM is to use the Moses
wrapper script
scripts/generic/trainlm-irst2.perl

This does steps 1 to 5 for you. Here is an example of how to run it

/home/s0565741/workspace/github/hh/scripts/generic/trainlm-irst2.perl
-cores 4 -irst-dir /home/s0565741/workspace/bin/irstlm/bin -p 0 -order 5
-text
/home/s0565741/workspace/experiment/europarl/en-es/lm/europarl.lowercased.1
-lm /home/s0565741/workspace/experiment/europarl/en-es/lm/europarl.lm.1





On 5 December 2013 15:12, renubalyan <renubalyan@cdac.in> wrote:

> Hi,
>
> Thanks for the response.
>
> I tried this option too, if I run the command without '--text yes' option
> then the command runs fine, However I wanted to ask one thing does this
> give me an arpa file or a binarized one? Because when I run the next
> command mentioned in the manual:
>
> 6. /home/renu/Desktop/mosesdecoder/bin/build_binary
> news-commentary-v8.fr-en.arpa.en news-commentary-v8.fr-en.blm.en
>
> *I get the following output:*
>
> Reading news-commentary-v8.fr-en.arpa.en
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>
> ****************************************************************************************************
>
> lm/read_arpa.cc:63 in void lm::ReadARPACounts(util::FilePiece&,
> std::vector<long long unsigned int>&) threw FormatLoadException because
> `line.size() >= 4 && StringPiece(line.data(), 4) == "blmt"'.
> *This looks like an IRSTLM binary file. Did you forget to pass --text yes
> to compile-lm? Byte: 40 File: news-commentary-v8.fr-en.arpa.en*
> ERROR
>
> The last second line put in bold indicates that the one I am using is a
> binary file.
> Does that mean I already have a binary file and I do not need to use
> step 6 mentioned above (which infact is for converting from arpa to binary
> file)
>
>
> Thanks
> Renu
>
>
>
>
>
> On December 5, 2013 at 4:19 PM Hieu Hoang <Hieu.Hoang@ed.ac.uk> wrote:
>
> I'm not sure what is
> --text yes
> this is how the EMS runs IRSTLM compile-lm:
> .../compile-lm .../europarl_pos.lm.4 .../europarl_pos.binlm.4
>
>
>
> On 4 December 2013 15:58, renubalyan <renubalyan@cdac.in> wrote:
>
> Hi,
>
> I am building the baseline system based on Moses manual instructions.
>
> I have installed Moses, GIZA++ and IRSTLM as mentioned in the manual.
> The corpus preparation (tokenization, ...cleaning) steps also goes well.
>
> However when I move to Language Model Training: I have some problems
>
> I am following these steps:
>
> 1. mkdir ~/lm
>
> 2. cd ~/lm
>
> 3. /home/renu/Desktop/irstlm/bin/add-start-end.sh <
> /home/renu/Desktop/corpus/news-commentary-v8.fr-en.true.en>
> news-commentary-v8.fr-en.sb.en
>
> 4. export IRSTLM=/home/renu/Desktop/irstlm;
> /home/renu/Desktop/irstlm/bin/build-lm.sh -i news-commentary-v8.fr-en.sb.en
> -t ./tmp -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en
>
> 5. /home/renu/Desktop/irstlm/bin/compile-lm --text yes
> news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
>
> Steps 1-4 work well but step 5 gives me -------(Warning:Too many
> parameters)
>
> I have searched the web for any possible solution but could not find any.
>
> I am not able to move ahead, kindly help.
>
> Thanks
> Renu
>
> -------------------------------------------------------------------------------------------------------------------------------
>
> This e-mail is for the sole use of the intended recipient(s) and may
> contain confidential and privileged information. If you are not the
> intended recipient, please contact the sender by reply e-mail and destroy
> all copies and the original message. Any unauthorized review, use,
> disclosure, dissemination, forwarding, printing or copying of this email
> is strictly prohibited and appropriate legal action will be taken.
> -------------------------------------------------------------------------------------------------------------------------------
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>
>
>
> -------------------------------------------------------------------------------------------------------------------------------
>
> This e-mail is for the sole use of the intended recipient(s) and may
> contain confidential and privileged information. If you are not the
> intended recipient, please contact the sender by reply e-mail and destroy
> all copies and the original message. Any unauthorized review, use,
> disclosure, dissemination, forwarding, printing or copying of this email
> is strictly prohibited and appropriate legal action will be taken.
> -------------------------------------------------------------------------------------------------------------------------------
>
>



--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131205/de6c3f66/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 86, Issue 15
*********************************************

Tweet