Moses-support Digest, Vol 86, Issue 12

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Warning: Too many arguments while IRSTLM language model
Training (Hieu Hoang)
2. Re: Viewer for parallel texts (Hieu Hoang)
3. Re: Viewer for parallel texts (Yu Tracy Chen)
4. Re: Word-to-word alignment when moses-parallel.pl is used
(Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Thu, 5 Dec 2013 10:49:08 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Warning: Too many arguments while IRSTLM
language model Training
To: renubalyan <renubalyan@cdac.in>, moses-support
<moses-support@mit.edu>
Message-ID:
<CAEKMkbiAf-77XXgr7B2J_pBg6UTinF64Aod3qNWj17tccBKGiA@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

I'm not sure what is
--text yes
this is how the EMS runs IRSTLM compile-lm:
.../compile-lm .../europarl_pos.lm.4 .../europarl_pos.binlm.4



On 4 December 2013 15:58, renubalyan <renubalyan@cdac.in> wrote:

> Hi,
>
> I am building the baseline system based on Moses manual instructions.
>
> I have installed Moses, GIZA++ and IRSTLM as mentioned in the manual.
> The corpus preparation (tokenization, ...cleaning) steps also goes well.
>
> However when I move to Language Model Training: I have some problems
>
> I am following these steps:
>
> 1. mkdir ~/lm
>
> 2. cd ~/lm
>
> 3. /home/renu/Desktop/irstlm/bin/add-start-end.sh <
> /home/renu/Desktop/corpus/news-commentary-v8.fr-en.true.en>
> news-commentary-v8.fr-en.sb.en
>
> 4. export IRSTLM=/home/renu/Desktop/irstlm;
> /home/renu/Desktop/irstlm/bin/build-lm.sh -i news-commentary-v8.fr-en.sb.en
> -t ./tmp -p -s improved-kneser-ney -o news-commentary-v8.fr-en.lm.en
>
> 5. /home/renu/Desktop/irstlm/bin/compile-lm --text yes
> news-commentary-v8.fr-en.lm.en.gz news-commentary-v8.fr-en.arpa.en
>
> Steps 1-4 work well but step 5 gives me -------(Warning:Too many
> parameters)
>
>
> I have searched the web for any possible solution but could not find any.
>
> I am not able to move ahead, kindly help.
>
> Thanks
> Renu
>
> -------------------------------------------------------------------------------------------------------------------------------
>
> This e-mail is for the sole use of the intended recipient(s) and may
> contain confidential and privileged information. If you are not the
> intended recipient, please contact the sender by reply e-mail and destroy
> all copies and the original message. Any unauthorized review, use,
> disclosure, dissemination, forwarding, printing or copying of this email
> is strictly prohibited and appropriate legal action will be taken.
> -------------------------------------------------------------------------------------------------------------------------------
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131205/b44be770/attachment-0001.htm

------------------------------

Message: 2
Date: Thu, 5 Dec 2013 10:52:21 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Viewer for parallel texts
To: Eleftherios Avramidis <eleftherios.avramidis@dfki.de>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbg7eQN4b8_WOPAbJ5vDXr6XuNuaT-A+Gv1XEec6PZv_hg@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

you mean a gui diff? kdiff, meld?

or something that shows you the word alignments too? I don't know of any
tools, if you find 1 let me know


On 4 December 2013 17:44, Eleftherios Avramidis <
eleftherios.avramidis@dfki.de> wrote:

> Hi,
>
> I need a viewer which would display the two sides of a parallel corpus
> side-by-side, so that I can check for sanity etc. I remember I used to
> have one of them, but it has been some time, so I cannot find it right
> now. Could somebody give me a quick suggestion?
>
> best
> Lefteris
>
> --
> MSc. Inf. Eleftherios Avramidis
> DFKI GmbH, Alt-Moabit 91c, 10559 Berlin
> Tel. +49-30 238 95-1806
>
> Fax. +49-30 238 95-1810
>
>
> -------------------------------------------------------------------------------------------
> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>
> Geschaeftsfuehrung:
> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
> Dr. Walter Olthoff
>
> Vorsitzender des Aufsichtsrats:
> Prof. Dr. h.c. Hans A. Aukes
>
> Amtsgericht Kaiserslautern, HRB 2313
>
> -------------------------------------------------------------------------------------------
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>



--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131205/11af5228/attachment-0001.htm

------------------------------

Message: 3
Date: Thu, 5 Dec 2013 11:59:31 +0100
From: Yu Tracy Chen <yuchen@coli.uni-sb.de>
Subject: Re: [Moses-support] Viewer for parallel texts
To: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Cc: Eleftherios Avramidis <eleftherios.avramidis@dfki.de>,
moses-support <moses-support@mit.edu>
Message-ID:
<CA+zXwvg49kvCKySmDwNBFYUe2oRs9Lo0KFkgBDqC=fvuEc0f9Q@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

Try this?

http://wanthalf.saga.cz/intertext


On Thu, Dec 5, 2013 at 11:52 AM, Hieu Hoang <Hieu.Hoang@ed.ac.uk> wrote:

> you mean a gui diff? kdiff, meld?
>
> or something that shows you the word alignments too? I don't know of any
> tools, if you find 1 let me know
>
>
> On 4 December 2013 17:44, Eleftherios Avramidis <
> eleftherios.avramidis@dfki.de> wrote:
>
>> Hi,
>>
>> I need a viewer which would display the two sides of a parallel corpus
>> side-by-side, so that I can check for sanity etc. I remember I used to
>> have one of them, but it has been some time, so I cannot find it right
>> now. Could somebody give me a quick suggestion?
>>
>> best
>> Lefteris
>>
>> --
>> MSc. Inf. Eleftherios Avramidis
>> DFKI GmbH, Alt-Moabit 91c, 10559 Berlin
>> Tel. +49-30 238 95-1806
>>
>> Fax. +49-30 238 95-1810
>>
>>
>> -------------------------------------------------------------------------------------------
>> Deutsches Forschungszentrum fuer Kuenstliche Intelligenz GmbH
>> Firmensitz: Trippstadter Strasse 122, D-67663 Kaiserslautern
>>
>> Geschaeftsfuehrung:
>> Prof. Dr. Dr. h.c. mult. Wolfgang Wahlster (Vorsitzender)
>> Dr. Walter Olthoff
>>
>> Vorsitzender des Aufsichtsrats:
>> Prof. Dr. h.c. Hans A. Aukes
>>
>> Amtsgericht Kaiserslautern, HRB 2313
>>
>> -------------------------------------------------------------------------------------------
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburgh
> http://www.hoang.co.uk/hieu
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131205/3d05382a/attachment-0001.htm

------------------------------

Message: 4
Date: Thu, 5 Dec 2013 11:05:58 +0000
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] Word-to-word alignment when
moses-parallel.pl is used
To: Thomas Meyer <Thomas.Meyer@idiap.ch>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAEKMkbhTWLfduQNRtCGSsucqJby6nzq_0TkbSqZHk53jz6RmXQ@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"

The script seems to have been forked just before the last changes

https://github.com/moses-smt/mosesdecoder/commit/93433cf0157ecadd19162929a077aa6179dc0d8b

If someone who uses SGE can do a merge with the latest version in github
and test that it works, I can check it in



On 2 December 2013 09:03, Thomas Meyer <Thomas.Meyer@idiap.ch> wrote:

> Hi all,
>
> Attached is moses-parallel.pl with some modifications (marked #thm:) to
> make it work with the SGE architecture we have here. I just ran a test
> whether it outputs the word alignment and all works normally.
>
> Here is the command I used:
>
> ./moses-parallel.pl -jobs 2 -alignment-output-file alignment.txt
> -input-file /temp/tmeyer/test.en -decoder
> /temp/tmeyer/mosesdecoder/bin/moses -config
> /temp/tmeyer/en_de/test/moses.tuned.ini > test.txt
>
> which outputs, in alignment.txt:
>
> 0-0 1-1 2-2 3-3 4-4 5-5 6-6 7-7 8-8 9-9 10-10 11-11 12-12 13-13 14-14
> 15-15 16-16 17-17 18-18 19-19 20-20 21-21 22-22 23-23 24-24
> 0-0 1-1 2-2 3-3 4-4 5-4 6-5 7-6 8-7 9-8 10-9 11-10 12-11 13-12 13-13 14-14
> 16-15 17-16 18-17 20-18 21-19 22-20 23-21 24-22 25-23 26-24 27-25 28-26
> 29-27 30-28 31-28 32-29 33-30 34-31 36-32 37-33 38-34 39-35
>
> etc.
>
> Hope it will work for you, too, Konstantinos.
>
> Best,
> Thomas
>
>
>
> On 29/11/13 21:59, Philipp Koehn wrote:
>
> Hi Thomas,
>
> in that case - can you share your modifications if they are of general
> interest and check if there is a problem with the word alignment reporting?
>
> -phi
>
> On Fri, Nov 29, 2013 at 6:52 PM, Thomas Meyer <Thomas.Meyer@idiap.ch> <Thomas.Meyer@idiap.ch> wrote:
>
> Hi all,
>
> I 'still' do and find it quite convenient. At least with release 1.0 and
> some modifications to moses-parallel.pl
>
> Thomas
>
>
> From: CHATZITHEODOROU Konstantinos
> Sent: Friday, 29 November 2013 7:48 PM
> To: Hieu Hoang
> Cc: moses-support@mit.edu
> Subject: Re: [Moses-support] Word-to-word alignment when moses-parallel.pl
> is used
>
> Hi Hieu,
>
> Thanks for your reply. I have a question. Why not many people still use SGE
> to run Moses?
>
> Thank you in advance.
>
> Best,
> Konstantinos
>
>
> 2013/11/29 Hieu Hoang <Hieu.Hoang@ed.ac.uk> <Hieu.Hoang@ed.ac.uk>
>
> I'm not sure if many people still uses Sun Grid Engine to run Moses. If
> not, scripts like moses-parallel.pl may become outdated.
>
> I know at Edinburgh, we no longer use SGE. Instead we run our experiment
> on 1 large multi-core machine. Most of moses has been re-engineered to be
> multithreaded to take advantage of multicores
>
>
> On 28 November 2013 10:25, CHATZITHEODOROU Konstantinos<dinosaxon@gmail.com> <dinosaxon@gmail.com> wrote:
>
> Hi Philipp,
>
> Thank you your your prompt reply. I have tried the option
> "alignment-output-file" for moses-parallel.pl but didn't use it as part of
> the decoder setting. But it didn't work. Each individual sub-job finished
> with an error status stating something the sub-job is untranslated.
>
> Thank you for your time and consideration.
>
> Regards,
> Konstantinos
>
>
>
> 2013/11/27 Philipp Koehn <pkoehn@inf.ed.ac.uk> <pkoehn@inf.ed.ac.uk>
>
> Hi,
>
> the script has an option "alignment-output-file" - did you try this?
>
> -phi
>
> On Tue, Nov 26, 2013 at 8:13 PM, CHATZITHEODOROU Konstantinos<dinosaxon@gmail.com> <dinosaxon@gmail.com> wrote:
>
> Hi all,
>
> Could you please give an example on how to output the word-to-word
> alignment
> during decoding when moses-parallel.pl is used?
>
> Thank you in advance for your help.
>
> Regards,
> Konstantinos
>
>
> _______________________________________________
> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> --
> CHATZITHEODOROU Konstantinos
>
>
> _______________________________________________
> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
> --
> Hieu Hoang
> Research Associate
> University of Edinburghhttp://www.hoang.co.uk/hieu
>
>
>
> --
> CHATZITHEODOROU Konstantinos
>
>
>
> _______________________________________________
> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
> _______________________________________________
> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131205/542d8b22/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 86, Issue 12
*********************************************

Tweet