Moses-support Digest, Vol 94, Issue 7

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. how to generate an n-best list with a larger set of
reorderings (Saeed Farzi)
2. Re: how to generate an n-best list with a larger set of
reorderings (Barry Haddow)
3. moses version 2.1 very slow loading and decoding (Marie Sayadchi)
4. Re: moses version 2.1 very slow loading and decoding (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Tue, 5 Aug 2014 12:21:54 +0430
From: Saeed Farzi <saeedfarzi@gmail.com>
Subject: [Moses-support] how to generate an n-best list with a larger
set of reorderings
To: moses-support <moses-support@mit.edu>, "corpora@uib.no"
<corpora@uib.no>
Message-ID:
<CAOhrzg60Dwo7iMzj5ernK6LQYjNhffdwk-zyQKp7VQ76LzRbZA@mail.gmail.com>
Content-Type: text/plain; charset=ISO-8859-1

Dear all,

As you know, in most cases, translations in an n-best list differ only
in a few words and their word orders are basically the same. I need an
n-best list which contains verity of reorderings for each input
sentence. I wonder if anybody tell me how to force Moses to generate
such a n-best list.

Cheers,

--
S.Farzi, Ph.D. Student
Natural Language Processing Lab,
School of Electrical and Computer Eng.,
Tehran University
Tel: +9821-6111-9719


------------------------------

Message: 2
Date: Tue, 05 Aug 2014 10:05:16 +0100
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] how to generate an n-best list with a
larger set of reorderings
To: Saeed Farzi <saeedfarzi@gmail.com>, moses-support
<moses-support@mit.edu>
Message-ID: <53E09E4C.1000807@staffmail.ed.ac.uk>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Hi Saeed

Usually if you need more diversity than an n-best list, you should use a
lattice. You can obtain this by using one of the search graph options
(-output-search-graph , -output-search-graph-slf or
-output-search-graph-hypergraph)

Alternatively, you could try generating samples from the lattice using
-lattice-samples. This outputs in the same format as the nbest list. It
was claimed
(http://eprints.pascal-network.org/archive/00007186/01/MERT-sampling.pdf) that
this helps mert, but I have not found this to be the case,

cheers - Barry

On 05/08/14 08:51, Saeed Farzi wrote:
> Dear all,
>
> As you know, in most cases, translations in an n-best list differ only
> in a few words and their word orders are basically the same. I need an
> n-best list which contains verity of reorderings for each input
> sentence. I wonder if anybody tell me how to force Moses to generate
> such a n-best list.
>
> Cheers,
>


--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



------------------------------

Message: 3
Date: Tue, 5 Aug 2014 14:33:17 +0200
From: Marie Sayadchi <marie.sayadchi@gmail.com>
Subject: [Moses-support] moses version 2.1 very slow loading and
decoding
To: moses-support@mit.edu
Message-ID:
<CAJ8Ehf52p7jwaOV4FKVgG7Tue_VN0xzEub3r_BgVUtYaT+LHHQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Dear all

I am using moses RELEASE-2.1 binaries.
my training data is very small corpus of 170k sentences.
I following instructions from "http://www.statmt.org/moses/?n=Moses.Baseline"
for training and testing.
the problem is that when I test the system, loading time is very long and
after 30min the phrase-table is still loading .. the models are small and
it should not take that long, also translation itself is very slow.
same model loads relatively fast and works fine with my older version of
moses(before 2012). both were using only one cpu and almost same amount of
memory.
my question is :
how can I get the same decoding speed as before; with the moses 2.1 , and
what is wrong with my current settings?


binarising the models helped with initial loading, but the translation is
still much faster with my old moses.
please see my moses.ini and translation log files in the attachment.
the command that I am using:
training:
nohup nice
/home/lingua/smt/moses-binaries/www.statmt.org_64bit/moses/RELEASE-2.1/binaries/linux-64bit/scripts/training/train-model.perl
-root-dir new/ -corpus ./all -f de -e en -max-phrase-length 5 -alignment
grow-diag-final-and -reordering msd-bidirectional-fe -lm
0:3:/home/lingua/smt/Multisensor/news-models/news.en.truecase.irstlm.blm
-external-bin-dir
/home/lingua/smt/moses-binaries/www.statmt.org_64bit/moses/RELEASE-2.1/binaries/linux-64bit/training-tools
-mgiza mgiza -snt2cooc snt2cooc >& ./new/training.out

binarising:
/home/lingua/smt/moses-binaries/www.statmt.org_64bit/moses/RELEASE-2.1/binaries/linux-64bit/bin/processLexicalTable
-in ./new/model/reordering-table.wbe-msd-bidirectional-fe.gz -out
./new/binariesed/reordering-table

/home/lingua/smt/moses-binaries/www.statmt.org_64bit/moses/RELEASE-2.1/binaries/linux-64bit/bin/processLexicalTable
-in ./new/model/reordering-table.wbe-msd-bidirectional-fe.gz -out
./new/binariesed/reordering-table

testing:
nohup nice
/home/lingua/smt/moses-binaries/www.statmt.org_64bit/moses/RELEASE-2.1/binaries/linux-64bit/bin/moses
-f ./new/model/moses-bin.ini <./newstest2013.tok.norm.de > out 2> log

Best,
Marieh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140805/16798ae2/attachment-0001.htm
-------------- next part --------------
A non-text attachment was scrubbed...
Name: moses.ini
Type: application/octet-stream
Size: 990 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140805/16798ae2/attachment-0003.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: moses-bin.ini
Type: application/octet-stream
Size: 969 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140805/16798ae2/attachment-0004.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: log
Type: application/octet-stream
Size: 5851 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20140805/16798ae2/attachment-0005.obj

------------------------------

Message: 4
Date: Tue, 5 Aug 2014 13:45:48 +0100
From: Hieu Hoang <Hieu.Hoang@ed.ac.uk>
Subject: Re: [Moses-support] moses version 2.1 very slow loading and
decoding
To: Marie Sayadchi <marie.sayadchi@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbjz=BPRg=JpQeTJjNiU_BfBy_ryN5ECvzqNYsQLEzBN1g@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

from the log file, the loading seemed to have taken 1 second, which is fast
and normal.

the translation of each sentence took 30+ sec each. Are the binary files in
filesystem cache? what happens if you fun the decoding again with same
sentence and moses-bin.ini ?




On 5 August 2014 13:33, Marie Sayadchi <marie.sayadchi@gmail.com> wrote:

> Dear all
>
> I am using moses RELEASE-2.1 binaries.
> my training data is very small corpus of 170k sentences.
> I following instructions from "
> http://www.statmt.org/moses/?n=Moses.Baseline" for training and testing.
> the problem is that when I test the system, loading time is very long and
> after 30min the phrase-table is still loading .. the models are small and
> it should not take that long, also translation itself is very slow.
> same model loads relatively fast and works fine with my older version of
> moses(before 2012). both were using only one cpu and almost same amount of
> memory.
> my question is :
> how can I get the same decoding speed as before; with the moses 2.1 , and
> what is wrong with my current settings?
>
>
> binarising the models helped with initial loading, but the translation is
> still much faster with my old moses.
> please see my moses.ini and translation log files in the attachment.
> the command that I am using:
> training:
> nohup nice
> /home/lingua/smt/moses-binaries/www.statmt.org_64bit/moses/RELEASE-2.1/binaries/linux-64bit/scripts/training/train-model.perl
> -root-dir new/ -corpus ./all -f de -e en -max-phrase-length 5 -alignment
> grow-diag-final-and -reordering msd-bidirectional-fe -lm
> 0:3:/home/lingua/smt/Multisensor/news-models/news.en.truecase.irstlm.blm
> -external-bin-dir
> /home/lingua/smt/moses-binaries/www.statmt.org_64bit/moses/RELEASE-2.1/binaries/linux-64bit/training-tools
> -mgiza mgiza -snt2cooc snt2cooc >& ./new/training.out
>
> binarising:
> /home/lingua/smt/moses-binaries/www.statmt.org_64bit/moses/RELEASE-2.1/binaries/linux-64bit/bin/processLexicalTable
> -in ./new/model/reordering-table.wbe-msd-bidirectional-fe.gz -out
> ./new/binariesed/reordering-table
>
> /home/lingua/smt/moses-binaries/www.statmt.org_64bit/moses/RELEASE-2.1/binaries/linux-64bit/bin/processLexicalTable
> -in ./new/model/reordering-table.wbe-msd-bidirectional-fe.gz -out
> ./new/binariesed/reordering-table
>
> testing:
> nohup nice
> /home/lingua/smt/moses-binaries/www.statmt.org_64bit/moses/RELEASE-2.1/binaries/linux-64bit/bin/moses
> -f ./new/model/moses-bin.ini <./newstest2013.tok.norm.de > out 2> log
>
> Best,
> Marieh
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Hieu Hoang
Research Associate
University of Edinburgh
http://www.hoang.co.uk/hieu
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20140805/4b8b1e21/attachment.htm

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 94, Issue 7
********************************************

Tweet