Home » Uncategories » Moses-support Digest, Vol 100, Issue 75

Moses-support Digest, Vol 100, Issue 75

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: SRI/KenLM kneser ney implementation (koormoosh)
2. Re: Moses-support Digest, Vol 100, Issue 73 (Kenneth Heafield)
3. Re: Moses with SRILM Compile Error (Barry Haddow)
4. Re: Baseline System (Barry Haddow)
5. Re: Moses with SRILM Compile Error (Kenneth Heafield)


----------------------------------------------------------------------

Message: 1
Date: Mon, 23 Feb 2015 07:42:21 +1100
From: koormoosh <koormoosh@gmail.com>
Subject: Re: [Moses-support] SRI/KenLM kneser ney implementation
To: moses-support@mit.edu
Message-ID:
<CAN3_CDhuJW76mOALqg7V_ZQEHkKu9P6xPB+eOaCe9BbH7zmFFw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Thanks Kenneth. I expected that passing

-gt1min 0 -gt2min 0 -gt3min 0 -gt4min 0 ....

to srilm prevents that pruning you mentioned. But it doesn't. Is this what
you meant by "pruning", or there exists a separate unavoidable pruning that
srilm does?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150223/7f914d5b/attachment-0001.htm

------------------------------

Message: 2
Date: Sun, 22 Feb 2015 15:46:51 -0500
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] Moses-support Digest, Vol 100, Issue 73
To: moses-support@mit.edu
Message-ID: <54EA403B.7000109@kheafield.com>
Content-Type: text/plain; charset=windows-1252

Great question! Let me know what you find from srilm-user.

On 02/22/2015 03:39 PM, koormoosh wrote:
> Thanks Kenneth. I expected that passing
>
> -gt1min 0 -gt2min 0 -gt3min 0 -gt4min 0 ....
>
> to srilm prevents that pruning you mentioned. But it doesn't. Is this
> what you meant by "pruning", or there exists a separate unavoidable
> pruning that srilm does?
>
> On 23/02/2015 4:28 AM, <moses-support-request@mit.edu
> <mailto:moses-support-request@mit.edu>> wrote:
>
> Send Moses-support mailing list submissions to
> moses-support@mit.edu <mailto:moses-support@mit.edu>
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mailman.mit.edu/mailman/listinfo/moses-support
> or, via email, send a message with subject or body 'help' to
> moses-support-request@mit.edu
> <mailto:moses-support-request@mit.edu>
>
> You can reach the person managing the list at
> moses-support-owner@mit.edu <mailto:moses-support-owner@mit.edu>
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Moses-support digest..."
>
>
> Today's Topics:
>
> 1. Re: SRI/KenLM kneser ney implementation (Kenneth Heafield)
> 2. Re: Fwd: Fwd: SGM files (Barry Haddow)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Sun, 22 Feb 2015 07:00:05 -0500
> From: Kenneth Heafield <moses@kheafield.com
> <mailto:moses@kheafield.com>>
> Subject: Re: [Moses-support] SRI/KenLM kneser ney implementation
> To: moses-support@mit.edu <mailto:moses-support@mit.edu>
> Message-ID: <54E9C4C5.7020702@kheafield.com
> <mailto:54E9C4C5.7020702@kheafield.com>>
> Content-Type: text/plain; charset=windows-1252
>
> SRILM prunes singletons for trigrams and above by default. You're
> likely to get better answers to SRILM-specific questions on srilm-user.
>
> On 02/22/2015 06:28 AM, koormoosh wrote:
> > Hi,
> >
> > I wonder if SRI does any sort of implicit pruning or refinement?
> To be more
> > precise, is there any way to force SRI not to prune anything (removing
> > singletons, etc). I thought that my way of calling it does what I
> want (not
> > pruning), but then I don't know how to explain getting different
> results.
> > This is how I call SRI:
> >
> >
> -----------------------------------------------------------------------------------------------------
> > ./ngram-count -order 3 -text training.txt -write training.ngrams
> >
> > ./ngram-count -order 3 -read training.ngrams -lm training.binary
> > -interpolate -ukndiscount -gt1min 0 -gt2min 0 -gt3min 0
> -write-binary-lm
> >
> > ./ngram -order 3 -lm training.binary -ppl test.txt -debug 2
> >
> > am I missing/misusing something?
> >
> >
> --------------------------------------------------------------------------------------------------------
> > An example to show this problem:
> > (Example-1):
> > Test: "13 13 13"
> > Training: "13 13 13 13 17"
> > perplexity *matches* SRI: "2.79327"
> >
> > (Example-2):
> > Test: "13 13 13"
> > Training "13 13 13 13 13 13 17 17 17 17 17 14 14 15 15 15 16 16 16 16"
> > perplexity *doesn't match* SRI: "4.51546" and what SRI returns us
> "4.242".
> >
> -------------------------------------------------------------------------------------------------------
> >
> > Thanks in advance,
> > Koorm
> >
> >
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
>
>
> ------------------------------
>
> Message: 2
> Date: Sun, 22 Feb 2015 13:03:33 +0000
> From: Barry Haddow <bhaddow@staffmail.ed.ac.uk
> <mailto:bhaddow@staffmail.ed.ac.uk>>
> Subject: Re: [Moses-support] Fwd: Fwd: SGM files
> To: Ricardo Cabello S?nchez
> <ricardo.cabello.sanchez@googlemail.com
> <mailto:ricardo.cabello.sanchez@googlemail.com>>
> Cc: moses-support@mit.edu <mailto:moses-support@mit.edu>
> Message-ID: <20150222130333.52775wu0xcldgp44@www.staffmail.ed.ac.uk
> <mailto:20150222130333.52775wu0xcldgp44@www.staffmail.ed.ac.uk>>
> Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes";
> format="flowed"
>
> Hi Ricardo
>
> The only documentation for EMS is on the Moses website, see
> http://www.statmt.org/moses/?n=FactoredTraining.EMS plus the examples
> in the Moses distribution and the one for the baseline system
> (http://www.statmt.org/moses/uploads/Moses/config). The file
> experiment.meta shows the templates that EMS uses to execute an
> experiment, but is quite hard to read if you are new to Moses.
>
> For evaluating with text files rather than sgm, look at the EVALUATION
> section of the baseline system config file. This should take care of
> the errors you reported below,
>
> cheers - Barry
>
> Quoting Ricardo Cabello S?nchez
> <ricardo.cabello.sanchez@googlemail.com
> <mailto:ricardo.cabello.sanchez@googlemail.com>> on Sun, 22 Feb 2015
> 12:52:34
> +0100:
>
> > Hi,
> >
> > I get problems also at
> > stepEVALUATION:test:reference-from-sgm crashed
> > step EVALUATION:test:analysis-coverage crashed
> >
> > Coudl you please advice? Please find attached log files.
> >
> > Thanks
> >
> > Ricardo
> >
> > 2015-02-22 12:25 GMT+01:00 Ricardo Cabello S?nchez <
> > ricardo.cabello.sanchez@googlemail.com
> <mailto:ricardo.cabello.sanchez@googlemail.com>>:
> >
> >> Hi Barry,
> >>
> >> Thank you very much for your quick response. It helped a lot. Now
> I can
> >> train my system much better. I do not have much expertise in
> scripts and
> >> config files but I am doing my best. Do you know where I can find
> a quick
> >> guidance to know how to configure config file?
> >>
> >> Experiment crashed at test filter step. Could you please advice?
> Please
> >> find attached log file.
> >>
> >> Thank you and best regards,
> >>
> >> Ricardo
> >>
> >> 2015-02-22 11:55 GMT+01:00 Barry Haddow
> <bhaddow@staffmail.ed.ac.uk <mailto:bhaddow@staffmail.ed.ac.uk>>:
> >>
> >>> Hi Ricardo
> >>>
> >>> You only need sgm files if you want to evaluate with the nist bleu
> >>> script. If you want to use bleu to evaluate with txt files, just use
> >>> multi-bleu.perl (in scripts/generic)
> >>>
> >>> cheers - Barry
> >>>
> >>>
> >>> Quoting Ricardo Cabello S?nchez
> <ricardo.cabello.sanchez@googlemail.com
> <mailto:ricardo.cabello.sanchez@googlemail.com>>
> >>> on Sun, 22 Feb 2015 11:44:51 +0100:
> >>>
> >>> Hi all,
> >>>>
> >>>> I do not know if my message reached you.
> >>>>
> >>>> Thank you
> >>>>
> >>>> Ricardo
> >>>>
> >>>> ---------- Forwarded message ----------
> >>>> From: Ricardo Cabello S?nchez
> <ricardo.cabello.sanchez@googlemail.com
> <mailto:ricardo.cabello.sanchez@googlemail.com>>
> >>>> Date: 2015-02-21 19:51 GMT+01:00
> >>>> Subject: SGM files
> >>>> To: moses-support@mit.edu <mailto:moses-support@mit.edu>
> >>>>
> >>>>
> >>>> Hi,
> >>>>
> >>>> I would like to ask you how I can get sgm files for reference.
> I already
> >>>> have txt files in ENG and in PTG correctly aligned. Is there
> any way to
> >>>> run
> >>>> moses without these sgm files?
> >>>>
> >>>> Thank you and best regards,
> >>>>
> >>>> Ricardo
> >>>>
> >>>>
> >>>
> >>>
> >>> --
> >>> The University of Edinburgh is a charitable body, registered in
> >>> Scotland, with registration number SC005336.
> >>>
> >>>
> >>>
> >>
> >
>
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>
>
>
>
> ------------------------------
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> End of Moses-support Digest, Vol 100, Issue 73
> **********************************************
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


------------------------------

Message: 3
Date: Sun, 22 Feb 2015 21:32:46 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] Moses with SRILM Compile Error
To: TinTin Kalaw <tintin.kalaw@gmail.com>
Cc: moses-support@mit.edu
Message-ID: <20150222213246.118252ll22unccw8@www.staffmail.ed.ac.uk>
Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes";
format="flowed"

Hi TinTin

First, make sure you do a clean build, and that you're absolutely sure
Moses is linking against SRILM V1.6. If the build still fails, then
post your log to the list,

cheers - Barry

Quoting TinTin Kalaw <tintin.kalaw@gmail.com> on Sun, 22 Feb 2015
22:48:03 +0800:

> It is because this other tool that I need (a thesis project of an
> upperclassmen) was made with SRILM. To run their project, SRILM must work.
>
> I used an older version of SRILM (v1.6.0). The build still failed.
>
> Thank you for your fast reply.
>
> Regards,
>
>
>
> *Kristine Ma. Dominique F. KalawContact No.: 0927.854.4201Email:
> tintin.kalaw@gmail.com <tintin.kalaw@gmail.com>*
>
> On Sun, Feb 22, 2015 at 5:30 PM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>
>> use a older version of SRILM.
>>
>> Can I ask why you use SRILM in more detail? Do you use it to create
>> language models, or within the decoder to look up LM scores? In both cases,
>> there are now better tools to use than SRILM
>>
>>
>> On 22/02/15 07:22, TinTin Kalaw wrote:
>>
>> Good day!
>>
>> Whenever I try to compile Moses with the *--with-srilm=/my/path/to/srilm*,
>> I get a compilation error. If I compile it with the
>> *--with-boost=/my/path/to/boost* or with just *./bjam*, it is a success.
>> Unfortunately I cannot use an alternative to SRILM because this other tool
>> that I am using makes use of SRILM and Moses.
>>
>> I have already successfully installed/compiled the other
>> tools/packages/dependencies that Moses needs. My machine is running on a
>> dual-boot OS of *Windows 8.1* and *Ubuntu 14.04 LTS 64-bit*. I am
>> currently using Ubuntu. I used *Giza v1.0.7*, *SRILM v1.7.1*, *Boost
>> 1_57_0*, and the version of *Moses* as of Feb 7 2015.
>>
>> Attached here is the *build.log.gz* of the command *./bjam
>> --with-srilm=$SRILM*.
>>
>> I am hoping for your swift response regarding this issue. Thank you.
>>
>> --
>> Regards,
>>
>>
>>
>> *Kristine Ma. Dominique F. Kalaw Contact No.: 0927.854.4201 Email:
>> tintin.kalaw@gmail.com <tintin.kalaw@gmail.com>*
>>
>>
>> _______________________________________________
>> Moses-support mailing
>> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>> --
>> Hieu Hoang
>> Research Associate (until March 2015)
>> ** searching for interesting commercial MT position **
>> University of Edinburghhttp://www.hoang.co.uk/hieu
>>
>>
>



--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.




------------------------------

Message: 4
Date: Sun, 22 Feb 2015 21:36:17 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] Baseline System
To: Hamed Soleimani <hamed1soleimani@gmail.com>
Cc: moses-support@mit.edu
Message-ID: <20150222213617.17523sycjxg5zrok@www.staffmail.ed.ac.uk>
Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes";
format="flowed"

Hi Hamed

The behaviour with nohup is expected - the command runs in the background.
See http://en.wikipedia.org/wiki/Job_control_%28Unix%29

If your translation just echoes back the input, then check carefully
that you gave the correct corpus files to Moses, and that there were
no errors during the training process.

cheers - Barry

Quoting Hamed Soleimani <hamed1soleimani@gmail.com> on Fri, 20 Feb
2015 17:21:20 +0000:

> Hi
> I'm following the baseline system tutorial.
> As i reach to this command i got some strange result.
>
> nohup nice ~/mosesdecoder/scripts/training/train-model.perl -root-dir train \
> -corpus ~/corpus/news-commentary-v8.fr-en.clean
> \
> -f fr -e en -alignment grow-diag-final-and -reordering
> msd-bidirectional-fe \
> -lm 0:3:$HOME/lm/news-commentary-v8.fr-en.blm.en:8
> \
> -external-bin-dir ~/mosesdecoder/tools >& training.out &
>
> after less than a second in the the terminal it get me:
> [1] (some_number)
> but after i removed "nohup nice" and ">& training.out &" from this command
> it worked. this strange result also appeared in the "mert-moses.pl" command
> at tuning part. also i removed "nohup nice" and "&> mert.out &" from it and
> worked. Now when i want to test the translator it don't translate anything
> and just give the input as translation. Also it takes lots of memory spacse
> and my OS use has to swap space to handle it.
> "i'm using fedora 21 in a (core-i7 6GB RAM) laptop"
> sorry for bad English :(
> Thanks in advance
> Hamed Soleimani
>



--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.




------------------------------

Message: 5
Date: Sun, 22 Feb 2015 16:39:28 -0500
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] Moses with SRILM Compile Error
To: moses-support@mit.edu
Message-ID: <54EA4C90.106@kheafield.com>
Content-Type: text/plain; charset=windows-1252

SRILM appears to be linking against openmp so gcc needs -fopenmp in
Moses. Try this:

bjam linkflags=-fopenmp cxxflags=-fopenmp --with-srilm=...

On 02/22/2015 04:32 PM, Barry Haddow wrote:
> Hi TinTin
>
> First, make sure you do a clean build, and that you're absolutely sure
> Moses is linking against SRILM V1.6. If the build still fails, then
> post your log to the list,
>
> cheers - Barry
>
> Quoting TinTin Kalaw <tintin.kalaw@gmail.com> on Sun, 22 Feb 2015
> 22:48:03 +0800:
>
>> It is because this other tool that I need (a thesis project of an
>> upperclassmen) was made with SRILM. To run their project, SRILM must work.
>>
>> I used an older version of SRILM (v1.6.0). The build still failed.
>>
>> Thank you for your fast reply.
>>
>> Regards,
>>
>>
>>
>> *Kristine Ma. Dominique F. KalawContact No.: 0927.854.4201Email:
>> tintin.kalaw@gmail.com <tintin.kalaw@gmail.com>*
>>
>> On Sun, Feb 22, 2015 at 5:30 PM, Hieu Hoang <hieuhoang@gmail.com> wrote:
>>
>>> use a older version of SRILM.
>>>
>>> Can I ask why you use SRILM in more detail? Do you use it to create
>>> language models, or within the decoder to look up LM scores? In both cases,
>>> there are now better tools to use than SRILM
>>>
>>>
>>> On 22/02/15 07:22, TinTin Kalaw wrote:
>>>
>>> Good day!
>>>
>>> Whenever I try to compile Moses with the *--with-srilm=/my/path/to/srilm*,
>>> I get a compilation error. If I compile it with the
>>> *--with-boost=/my/path/to/boost* or with just *./bjam*, it is a success.
>>> Unfortunately I cannot use an alternative to SRILM because this other tool
>>> that I am using makes use of SRILM and Moses.
>>>
>>> I have already successfully installed/compiled the other
>>> tools/packages/dependencies that Moses needs. My machine is running on a
>>> dual-boot OS of *Windows 8.1* and *Ubuntu 14.04 LTS 64-bit*. I am
>>> currently using Ubuntu. I used *Giza v1.0.7*, *SRILM v1.7.1*, *Boost
>>> 1_57_0*, and the version of *Moses* as of Feb 7 2015.
>>>
>>> Attached here is the *build.log.gz* of the command *./bjam
>>> --with-srilm=$SRILM*.
>>>
>>> I am hoping for your swift response regarding this issue. Thank you.
>>>
>>> --
>>> Regards,
>>>
>>>
>>>
>>> *Kristine Ma. Dominique F. Kalaw Contact No.: 0927.854.4201 Email:
>>> tintin.kalaw@gmail.com <tintin.kalaw@gmail.com>*
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing
>>> listMoses-support@mit.eduhttp://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>>
>>> --
>>> Hieu Hoang
>>> Research Associate (until March 2015)
>>> ** searching for interesting commercial MT position **
>>> University of Edinburghhttp://www.hoang.co.uk/hieu
>>>
>>>
>>
>
>
>


------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 100, Issue 75
**********************************************

0 Response to "Moses-support Digest, Vol 100, Issue 75"

Post a Comment

Subscribe to: Post Comments (Atom)