Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Single score in phrase table (Matthias Huck)
2. Re: Single score in phrase table (Marcin Junczys-Dowmunt)
3. Re: Single score in phrase table (Hieu Hoang)
4. Re: Single score in phrase table (Marcin Junczys-Dowmunt)
5. Fwd: Re: BadDiscountException (Kenneth Heafield)
----------------------------------------------------------------------
Message: 1
Date: Tue, 24 Feb 2015 23:06:34 +0000
From: Matthias Huck <mhuck@inf.ed.ac.uk>
Subject: Re: [Moses-support] Single score in phrase table
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <1424819194.2192.538.camel@portedgar>
Content-Type: text/plain; charset="UTF-8"
Set a higher weight for UnknownWordPenalty? Maybe the default is not
adequate if you do strange things like this.
On Tue, 2015-02-24 at 23:49 +0100, Marcin Junczys-Dowmunt wrote:
> Hi,
> I have a problem with a single score phrase table. All scores have been
> combined into one score as a linear combination of scores and weights.
> However, for both, my compact phrase table the the in memory phrase
> table, all input result in UNK for all input tokens. The phrases are
> correctly found and returned by both phrase tables (including future
> score calculation), so this happens somewhere later. Any ideas?
>
> Best,
> Marcin
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.
------------------------------
Message: 2
Date: Wed, 25 Feb 2015 00:11:29 +0100
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Single score in phrase table
To: Matthias Huck <mhuck@inf.ed.ac.uk>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <54ED0521.60907@amu.edu.pl>
Content-Type: text/plain; charset=utf-8; format=flowed
Doesn't work, set it to 100, still the same effect.
And it's not that strange a thing: for an in-production phrase table
that will never be used for tuning this saves more than 30% size of the
translation model while not changing translation output (if it works).
W dniu 25.02.2015 o 00:06, Matthias Huck pisze:
> Set a higher weight for UnknownWordPenalty? Maybe the default is not
> adequate if you do strange things like this.
>
>
> On Tue, 2015-02-24 at 23:49 +0100, Marcin Junczys-Dowmunt wrote:
>> Hi,
>> I have a problem with a single score phrase table. All scores have been
>> combined into one score as a linear combination of scores and weights.
>> However, for both, my compact phrase table the the in memory phrase
>> table, all input result in UNK for all input tokens. The phrases are
>> correctly found and returned by both phrase tables (including future
>> score calculation), so this happens somewhere later. Any ideas?
>>
>> Best,
>> Marcin
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
>
------------------------------
Message: 3
Date: Wed, 25 Feb 2015 00:02:22 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Single score in phrase table
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <54ED110E.6050608@gmail.com>
Content-Type: text/plain; charset=windows-1252; format=flowed
odd. it should work. Are you sure the [mapping] section is in the ini file?
[mapping]
0 T 0
On 24/02/15 23:11, Marcin Junczys-Dowmunt wrote:
> Doesn't work, set it to 100, still the same effect.
>
> And it's not that strange a thing: for an in-production phrase table
> that will never be used for tuning this saves more than 30% size of the
> translation model while not changing translation output (if it works).
>
> W dniu 25.02.2015 o 00:06, Matthias Huck pisze:
>> Set a higher weight for UnknownWordPenalty? Maybe the default is not
>> adequate if you do strange things like this.
>>
>>
>> On Tue, 2015-02-24 at 23:49 +0100, Marcin Junczys-Dowmunt wrote:
>>> Hi,
>>> I have a problem with a single score phrase table. All scores have been
>>> combined into one score as a linear combination of scores and weights.
>>> However, for both, my compact phrase table the the in memory phrase
>>> table, all input result in UNK for all input tokens. The phrases are
>>> correctly found and returned by both phrase tables (including future
>>> score calculation), so this happens somewhere later. Any ideas?
>>>
>>> Best,
>>> Marcin
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>
>>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
--
Hieu Hoang
Research Associate (until March 2015)
** searching for interesting commercial MT position **
University of Edinburgh
http://www.hoang.co.uk/hieu
------------------------------
Message: 4
Date: Wed, 25 Feb 2015 01:19:18 +0100
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Single score in phrase table
To: Hieu Hoang <hieuhoang@gmail.com>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <54ED1506.7080903@amu.edu.pl>
Content-Type: text/plain; charset=windows-1252; format=flowed
Got it. Mentioning the mapping helped.
It's actually a different bug: turns out Moses does not complain during
loading about missing generation models (referenced by a wrong path in
ini-file) and just goes on removing all translation options while decoding.
W dniu 25.02.2015 o 01:02, Hieu Hoang pisze:
> odd. it should work. Are you sure the [mapping] section is in the ini
> file?
> [mapping]
> 0 T 0
>
> On 24/02/15 23:11, Marcin Junczys-Dowmunt wrote:
>> Doesn't work, set it to 100, still the same effect.
>>
>> And it's not that strange a thing: for an in-production phrase table
>> that will never be used for tuning this saves more than 30% size of the
>> translation model while not changing translation output (if it works).
>>
>> W dniu 25.02.2015 o 00:06, Matthias Huck pisze:
>>> Set a higher weight for UnknownWordPenalty? Maybe the default is not
>>> adequate if you do strange things like this.
>>>
>>>
>>> On Tue, 2015-02-24 at 23:49 +0100, Marcin Junczys-Dowmunt wrote:
>>>> Hi,
>>>> I have a problem with a single score phrase table. All scores have
>>>> been
>>>> combined into one score as a linear combination of scores and weights.
>>>> However, for both, my compact phrase table the the in memory phrase
>>>> table, all input result in UNK for all input tokens. The phrases are
>>>> correctly found and returned by both phrase tables (including future
>>>> score calculation), so this happens somewhere later. Any ideas?
>>>>
>>>> Best,
>>>> Marcin
>>>> _______________________________________________
>>>> Moses-support mailing list
>>>> Moses-support@mit.edu
>>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>>>
>>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>
------------------------------
Message: 5
Date: Tue, 24 Feb 2015 20:39:46 -0500
From: Kenneth Heafield <moses@kheafield.com>
Subject: [Moses-support] Fwd: Re: BadDiscountException
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <54ED27E2.50205@kheafield.com>
Content-Type: text/plain; charset="windows-1252"
Try removing this bit of text and just calling the lmplz binary
directly. It's not clear to me why that wrapper script still exists.
$moses-script-dir/ems/support/lmplz-wrapper.perl -bin
-------- Forwarded Message --------
Subject: Re: [Moses-support] BadDiscountException
Date: Tue, 24 Feb 2015 06:16:43 -0800
From: fatma elzahraa Eltaher <fatmaeltaher@gmail.com>
To: Kenneth Heafield <moses@kheafield.com>
I use kenlm model and when try to add --discount_fallback=1 for setting
I get this error Unknown option: discount_fallback.
I attached config.toy where must I change to solve this problem ?
thank you,
Fatma El-Zahraa El -Taher
Teaching Assistant at Computer & System department
Faculty of Engineering, Azhar University
Email : fatmaeltaher@gmail.com <mailto:fatmaeltaher@gmail.com>
mobile: +201141600434
On Tue, Feb 24, 2015 at 5:22 AM, Kenneth Heafield <moses@kheafield.com
<mailto:moses@kheafield.com>> wrote:
The closed-form estimates for Kneser-Ney are not well-defined on toy or
class-based data. I recommend using more training data. If this is a
class-based model, pass --discount_fallback.
Kenneth
On 02/24/2015 08:04 AM, fatma elzahraa Eltaher wrote:
> Dears,
> I get the following error in LM_toy_train.65.STDERR:
> Unigram tokens 25188 types 39
> === 2/5 Calculating and sorting adjusted counts ===
> Chain sizes: 1:468 2:322921696 3:605478272 4:968765120 5:1412782592
> /home/fatma/Desktop/Folder/mosesdecoder/lm/builder/adjust_counts.cc:50
> in void
> lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const
> lm::builder::DiscountConfig&) threw BadDiscountException because
`s.n[j]
> == 0'.
> Could not calculate Kneser-Ney discounts for 1-grams with adjusted
count
> 4 because we didn't observe any 1-grams with adjusted count 3; Is this
> small or artificial data?
> How do I fix it?
>
>
> thank you,
>
>
>
> Fatma El-Zahraa El -Taher
>
> Teaching Assistant at Computer & System department
>
> Faculty of Engineering, Azhar University
>
> Email : fatmaeltaher@gmail.com <mailto:fatmaeltaher@gmail.com>
<mailto:fatmaeltaher@gmail.com <mailto:fatmaeltaher@gmail.com>>
> mobile: +201141600434
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu <mailto:Moses-support@mit.edu>
http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config.toy
Type: application/octet-stream
Size: 18851 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20150224/904facae/attachment.obj
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 100, Issue 86
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 100, Issue 86"
Post a Comment