Moses-support Digest, Vol 130, Issue 11

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Meaning of probabilities in Word lattices and how Moses
decodes them. (Hieu Hoang)
2. Re: Error while Training at Step 5 (Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Sun, 20 Aug 2017 22:19:40 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Meaning of probabilities in Word lattices
and how Moses decodes them.
To: Sanket Gandhare <sanketgandhare1991.sg@gmail.com>,
moses-support@mit.edu
Message-ID: <15cddf4a-6e38-d344-611c-adcd8f32a3bf@gmail.com>
Content-Type: text/plain; charset="utf-8"

the probabilities are calculated by the external tool that creates the
lattice, eg. the segmentation tool may give probabilities for the
different segmentations it makes. The probabilities are used as a
feature in the log-linear model in the same way as probabilities from
the phrase-table or language model etc

On 20/08/2017 13:39, Sanket Gandhare wrote:
> Hello all,
>
> ??? I want to give input to moses in lattice format but I am not sure
> what the probabilities in the lattice mean. I went through this link :
>
> http://www.statmt.org/moses/?n=Moses.WordLattices
>
> Also, how these probabilities calculated. What algorithm Moses uses to
> decode these word lattices?
>
> Please give me idea of all this.
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

--
Hieu Hoang
http://moses-smt.org/

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170820/e711737a/attachment-0001.html

------------------------------

Message: 2
Date: Mon, 21 Aug 2017 08:23:01 +0100
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Error while Training at Step 5
To: Aileen Joan Vicente <aovicente@up.edu.ph>, moses-support
<moses-support@mit.edu>
Message-ID:
<CAEKMkbgZ-H0VtNWt6cgcv2ig-vfHde-nL7NNccMcd=aao2Bg=A@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

in the directory
/Users/dcs/Moses/tools/mosesdecoder/bin
do you see a file 'extract'. If not, you must make sure moses compiles
successfully. Follow these instructions
http://www.statmt.org/moses/?n=Development.GetStarted

Hieu Hoang
http://moses-smt.org/


On 21 August 2017 at 01:52, Aileen Joan Vicente <aovicente@up.edu.ph> wrote:

> Thank you for your advice.
>
> I installed coreutils via macports but still get the same result:
>
> (5) extract phrases @ Mon Aug 21 08:45:08 PHT 2017
>
> MAX 7 0 0
>
> /Users/dcs/Moses/tools/mosesdecoder/scripts/generic/extract-parallel.perl
> 4 gsplit "gsort " /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract
> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.en
> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.cbk
> /Users/dcs/Desktop/Corpus/training/model/aligned.grow-diag-final
> /Users/dcs/Desktop/Corpus/training/model/extract 7 --GZOutput
>
> Executing: /Users/dcs/Moses/tools/mosesdecoder/scripts/generic/extract-parallel.perl
> 4 gsplit "gsort " /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract
> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.en
> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.cbk
> /Users/dcs/Desktop/Corpus/training/model/aligned.grow-diag-final
> /Users/dcs/Desktop/Corpus/training/model/extract 7 --GZOutput
>
> Started Mon Aug 21 08:45:08 2017
>
> using gzip
>
> isBSDSplit=0
>
> Executing: mkdir -p /Users/dcs/Desktop/Corpus/training/model/tmp.24478;
> ls -l /Users/dcs/Desktop/Corpus/training/model/tmp.24478
>
> total=2942 line-per-split=736
>
> gsplit -d -l 736 -a 7 /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.en
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/target.gsplit -d -l
> 736 -a 7 /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.cbk
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/source.gsplit -d -l
> 736 -a 7 /Users/dcs/Desktop/Corpus/training/model/aligned.grow-diag-final
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/align.sh:
> /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such file
> or directory
>
> sh: /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such
> file or directory
>
> sh: /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such
> file or directory
>
> sh: /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such
> file or directory
>
> merging extract / extract.inv
>
> gunzip -c /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000000.gz
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000001.gz
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000002.gz
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000003.gz |
> LC_ALL=C gsort -T /Users/dcs/Desktop/Corpus/training/model/tmp.24478
> 2>> /dev/stderr | gzip -c > /Users/dcs/Desktop/Corpus/
> training/model/extract.sorted.gz 2>> /dev/stderr
>
> gunzip -c /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000000.inv.gz
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000001.inv.gz
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000002.inv.gz
> /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000003.inv.gz
> | LC_ALL=C gsort -T /Users/dcs/Desktop/Corpus/training/model/tmp.24478
> 2>> /dev/stderr | gzip -c > /Users/dcs/Desktop/Corpus/
> training/model/extract.inv.sorted.gz 2>> /dev/stderr
>
> gunzip: can't stat: /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000000.inv.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000000.inv.gz.gz):
> No such file or directory
>
> gunzip: can't stat: /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000001.inv.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000001.inv.gz.gz):
> No such file or directory
>
> gunzip: can't stat: /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000002.inv.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000002.inv.gz.gz):
> No such file or directory
>
> gunzip: can't stat: /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000003.inv.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000003.inv.gz.gz):
> No such file or directory
>
> gunzip: can't stat: /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000000.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000000.gz.gz):
> No such file or directory
>
> gunzip: can't stat: /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000001.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000001.gz.gz):
> No such file or directory
>
> gunzip: can't stat: /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000002.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000002.gz.gz):
> No such file or directory
>
> gunzip: can't stat: /Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000003.gz
> (/Users/dcs/Desktop/Corpus/training/model/tmp.24478/extract.0000003.gz.gz):
> No such file or directory
>
> Finished Mon Aug 21 08:45:09 2017
>
>
> i noted that BSD is no longer used but isn't working still.
>
> Thank you in advance for your reply.
>
> On Mon, Aug 21, 2017 at 3:59 AM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>
>> it looks like you're using Mac OSX. You must make sure that the GNU
>> version of sort and split, not the BSD version. Moses has been tested on
>> OSX using MacPorts to install sort and split so you should install MacPorts
>>
>> https://www.macports.org/
>>
>>
>>
>> On 20/08/2017 13:22, Aileen Joan Vicente wrote:
>>
>>
>>
>> Hi! I ran training.perl, succeeded but found an empty phrase table. I
>> googled around to look for answer and some guy advised to check the logs. I
>> wonder if the problem is caused by this error in Step 5. Please help.
>>
>> Using SCRIPTS_ROOTDIR: /Users/dcs/Moses/tools/mosesdecoder/scripts
>> using gzip
>> (5) extract phrases @ Sun Aug 20 19:26:48 PHT 2017
>> MAX 7 0 0
>> /Users/dcs/Moses/tools/mosesdecoder/scripts/generic/extract-parallel.perl
>> 4 split "sort " /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract
>> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.en
>> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.cbk
>> /Users/dcs/Desktop/Corpus/training/model/aligned.grow-diag-final
>> /Users/dcs/Desktop/Corpus/training/model/extract 7 --GZOutput
>> Executing: /Users/dcs/Moses/tools/mosesdecoder/scripts/generic/extract-parallel.perl
>> 4 split "sort " /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract
>> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.en
>> /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.cbk
>> /Users/dcs/Desktop/Corpus/training/model/aligned.grow-diag-final
>> /Users/dcs/Desktop/Corpus/training/model/extract 7 --GZOutput
>> Started Sun Aug 20 19:26:48 2017
>> using gzip
>> split: illegal option -- -
>> usage: split [-a sufflen] [-b byte_count] [-l line_count] [-p pattern]
>> [file [prefix]]
>> isBSDSplit=1
>> Executing: mkdir -p /Users/dcs/Desktop/Corpus/training/model/tmp.798; ls
>> -l /Users/dcs/Desktop/Corpus/training/model/tmp.798
>> total=2942 line-per-split=736
>> split -l 736 -a 7 /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.en
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/target.split -l 736 -a
>> 7 /Users/dcs/Desktop/Corpus/training/corpus/mycorpus.cbk
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/source.split -l 736 -a
>> 7 /Users/dcs/Desktop/Corpus/training/model/aligned.grow-diag-final
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/align.sh:
>> /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such file
>> or directory
>> sh: /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such
>> file or directory
>> sh: /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such
>> file or directory
>> sh: /Users/dcs/Moses/tools/mosesdecoder/scripts/../bin/extract: No such
>> file or directory
>> merging extract / extract.inv
>> gunzip -c /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaaaaaa.gz
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaaaaab.gz
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaaaaac.gz
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaaaaad.gz |
>> LC_ALL=C sort -T /Users/dcs/Desktop/Corpus/training/model/tmp.798
>> 2>> /dev/stderr | gzip -c > /Users/dcs/Desktop/Corpus/training/model/extract.sorted.gz
>> 2>> /dev/stderr
>> gunzip -c /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaaaaaa.inv.gz
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaaaaab.inv.gz
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaaaaac.inv.gz
>> /Users/dcs/Desktop/Corpus/training/model/tmp.798/extract.aaaaaad.inv.gz
>> | LC_ALL=C sort -T /Users/dcs/Desktop/Corpus/training/model/tmp.798
>> 2>> /dev/stderr | gzip -c > /Users/dcs/Desktop/Corpus/trai
>> ning/model/extract.inv.sorted.gz 2>> /dev/stderr
>> gunzip: can't stat: /Users/dcs/Desktop/Corpus/trai
>> ning/model/tmp.798/extract.aaaaaaa.inv.gz (/Users/dcs/Desktop/Corpus/tra
>> ining/model/tmp.798/extract.aaaaaaa.inv.gz.gz): No such file or directory
>> gunzip: can't stat: /Users/dcs/Desktop/Corpus/trai
>> ning/model/tmp.798/extract.aaaaaab.inv.gz (/Users/dcs/Desktop/Corpus/tra
>> ining/model/tmp.798/extract.aaaaaab.inv.gz.gz): No such file or directory
>> gunzip: can't stat: /Users/dcs/Desktop/Corpus/trai
>> ning/model/tmp.798/extract.aaaaaac.inv.gz (/Users/dcs/Desktop/Corpus/tra
>> ining/model/tmp.798/extract.aaaaaac.inv.gz.gz): No such file or directory
>> gunzip: can't stat: /Users/dcs/Desktop/Corpus/trai
>> ning/model/tmp.798/extract.aaaaaad.inv.gz (/Users/dcs/Desktop/Corpus/tra
>> ining/model/tmp.798/extract.aaaaaad.inv.gz.gz): No such file or directory
>> gunzip: can't stat: /Users/dcs/Desktop/Corpus/trai
>> ning/model/tmp.798/extract.aaaaaaa.gz (/Users/dcs/Desktop/Corpus/tra
>> ining/model/tmp.798/extract.aaaaaaa.gz.gz): No such file or directory
>> gunzip: can't stat: /Users/dcs/Desktop/Corpus/trai
>> ning/model/tmp.798/extract.aaaaaab.gz (/Users/dcs/Desktop/Corpus/tra
>> ining/model/tmp.798/extract.aaaaaab.gz.gz): No such file or directory
>> gunzip: can't stat: /Users/dcs/Desktop/Corpus/trai
>> ning/model/tmp.798/extract.aaaaaac.gz (/Users/dcs/Desktop/Corpus/tra
>> ining/model/tmp.798/extract.aaaaaac.gz.gz): No such file or directory
>> gunzip: can't stat: /Users/dcs/Desktop/Corpus/trai
>> ning/model/tmp.798/extract.aaaaaad.gz (/Users/dcs/Desktop/Corpus/tra
>> ining/model/tmp.798/extract.aaaaaad.gz.gz): No such file or directory
>> Finished Sun Aug 20 19:26:48 2017
>>
>>
>>
>>
>>
>> _______________________________________________
>> Moses-support mailing listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>> --
>> Hieu Hoanghttp://moses-smt.org/
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170821/dbc292c3/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 130, Issue 11
**********************************************

Tweet