Moses-support Digest, Vol 100, Issue 73

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: SRI/KenLM kneser ney implementation (Kenneth Heafield)
2. Re: Fwd: Fwd: SGM files (Barry Haddow)


----------------------------------------------------------------------

Message: 1
Date: Sun, 22 Feb 2015 07:00:05 -0500
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] SRI/KenLM kneser ney implementation
To: moses-support@mit.edu
Message-ID: <54E9C4C5.7020702@kheafield.com>
Content-Type: text/plain; charset=windows-1252

SRILM prunes singletons for trigrams and above by default. You're
likely to get better answers to SRILM-specific questions on srilm-user.

On 02/22/2015 06:28 AM, koormoosh wrote:
> Hi,
>
> I wonder if SRI does any sort of implicit pruning or refinement? To be more
> precise, is there any way to force SRI not to prune anything (removing
> singletons, etc). I thought that my way of calling it does what I want (not
> pruning), but then I don't know how to explain getting different results.
> This is how I call SRI:
>
> -----------------------------------------------------------------------------------------------------
> ./ngram-count -order 3 -text training.txt -write training.ngrams
>
> ./ngram-count -order 3 -read training.ngrams -lm training.binary
> -interpolate -ukndiscount -gt1min 0 -gt2min 0 -gt3min 0 -write-binary-lm
>
> ./ngram -order 3 -lm training.binary -ppl test.txt -debug 2
>
> am I missing/misusing something?
>
> --------------------------------------------------------------------------------------------------------
> An example to show this problem:
> (Example-1):
> Test: "13 13 13"
> Training: "13 13 13 13 17"
> perplexity *matches* SRI: "2.79327"
>
> (Example-2):
> Test: "13 13 13"
> Training "13 13 13 13 13 13 17 17 17 17 17 14 14 15 15 15 16 16 16 16"
> perplexity *doesn't match* SRI: "4.51546" and what SRI returns us "4.242".
> -------------------------------------------------------------------------------------------------------
>
> Thanks in advance,
> Koorm
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


------------------------------

Message: 2
Date: Sun, 22 Feb 2015 13:03:33 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] Fwd: Fwd: SGM files
To: Ricardo Cabello S?nchez <ricardo.cabello.sanchez@googlemail.com>
Cc: moses-support@mit.edu
Message-ID: <20150222130333.52775wu0xcldgp44@www.staffmail.ed.ac.uk>
Content-Type: text/plain; charset=ISO-8859-1; DelSp="Yes";
format="flowed"

Hi Ricardo

The only documentation for EMS is on the Moses website, see
http://www.statmt.org/moses/?n=FactoredTraining.EMS plus the examples
in the Moses distribution and the one for the baseline system
(http://www.statmt.org/moses/uploads/Moses/config). The file
experiment.meta shows the templates that EMS uses to execute an
experiment, but is quite hard to read if you are new to Moses.

For evaluating with text files rather than sgm, look at the EVALUATION
section of the baseline system config file. This should take care of
the errors you reported below,

cheers - Barry

Quoting Ricardo Cabello S?nchez
<ricardo.cabello.sanchez@googlemail.com> on Sun, 22 Feb 2015 12:52:34
+0100:

> Hi,
>
> I get problems also at
> stepEVALUATION:test:reference-from-sgm crashed
> step EVALUATION:test:analysis-coverage crashed
>
> Coudl you please advice? Please find attached log files.
>
> Thanks
>
> Ricardo
>
> 2015-02-22 12:25 GMT+01:00 Ricardo Cabello S?nchez <
> ricardo.cabello.sanchez@googlemail.com>:
>
>> Hi Barry,
>>
>> Thank you very much for your quick response. It helped a lot. Now I can
>> train my system much better. I do not have much expertise in scripts and
>> config files but I am doing my best. Do you know where I can find a quick
>> guidance to know how to configure config file?
>>
>> Experiment crashed at test filter step. Could you please advice? Please
>> find attached log file.
>>
>> Thank you and best regards,
>>
>> Ricardo
>>
>> 2015-02-22 11:55 GMT+01:00 Barry Haddow <bhaddow@staffmail.ed.ac.uk>:
>>
>>> Hi Ricardo
>>>
>>> You only need sgm files if you want to evaluate with the nist bleu
>>> script. If you want to use bleu to evaluate with txt files, just use
>>> multi-bleu.perl (in scripts/generic)
>>>
>>> cheers - Barry
>>>
>>>
>>> Quoting Ricardo Cabello S?nchez <ricardo.cabello.sanchez@googlemail.com>
>>> on Sun, 22 Feb 2015 11:44:51 +0100:
>>>
>>> Hi all,
>>>>
>>>> I do not know if my message reached you.
>>>>
>>>> Thank you
>>>>
>>>> Ricardo
>>>>
>>>> ---------- Forwarded message ----------
>>>> From: Ricardo Cabello S?nchez <ricardo.cabello.sanchez@googlemail.com>
>>>> Date: 2015-02-21 19:51 GMT+01:00
>>>> Subject: SGM files
>>>> To: moses-support@mit.edu
>>>>
>>>>
>>>> Hi,
>>>>
>>>> I would like to ask you how I can get sgm files for reference. I already
>>>> have txt files in ENG and in PTG correctly aligned. Is there any way to
>>>> run
>>>> moses without these sgm files?
>>>>
>>>> Thank you and best regards,
>>>>
>>>> Ricardo
>>>>
>>>>
>>>
>>>
>>> --
>>> The University of Edinburgh is a charitable body, registered in
>>> Scotland, with registration number SC005336.
>>>
>>>
>>>
>>
>



--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.




------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 100, Issue 73
**********************************************

Tweet