Home » Uncategories » Moses-support Digest, Vol 100, Issue 92

Moses-support Digest, Vol 100, Issue 92

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Moses killed (mhmd hassnen)
2. Re: Moses killed (Barry Haddow)
3. Re: My phrase-table.tgz is 20-bytes long (Barry Haddow)
4. Re: Fwd: Re: BadDiscountException (Philipp Koehn)


----------------------------------------------------------------------

Message: 1
Date: Thu, 26 Feb 2015 20:45:29 +0200
From: mhmd hassnen <mhmd_hasnen@yahoo.com>
Subject: Re: [Moses-support] Moses killed
To: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Cc: Moses-support Support <moses-support@mit.edu>
Message-ID: <60739EFE-CC87-42DB-AD03-4E3053DE2431@yahoo.com>
Content-Type: text/plain; charset=us-ascii

Hi Barry
Thank you for your response
My machine ram is 8 GB what can i do

Sent from my iPhone

> On Feb 26, 2015, at 4:54 PM, Barry Haddow <bhaddow@staffmail.ed.ac.uk> wrote:
>
> Hi Mohamed
>
> The most likely explanation is that your machine ran out of memory,
>
> cheers
> Barry
>
>> On 26/02/15 13:04, mohamed hasanien wrote:
>> Hi All
>>
>> i try to run these tow command
>> ~/mosesdecoder/bin/moses -f ~/working/train/model/moses.ini Comment alle
>> ~/mosesdecoder/bin/moses -f ~/working/mert-work/moses.ini Comment alle
>> i alwayes get killd error
>> the output
>> --------------------------------------------------------
>> Defined parameters (per moses.ini or switch):
>> config: /mhmd/working/train/model/moses.ini Comment allez-vous
>> distortion-limit: 6
>> feature: UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryMemory name=TranslationModel0 num-features=4 path=/mhmd/working/train/model/phrase-table.gz input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/mhmd/working/train/model/reordering-table.wbe-msd-bidirectional-fe.gz Distortion KENLM lazyken=0 name=LM0 factor=0 path=/mhmd/lm/news-commentary-v8.fr-en.blm.en order=3
>> input-factors: 0
>> mapping: 0 T 0
>> weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5
>> line=UnknownWordPenalty
>> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
>> line=WordPenalty
>> FeatureFunction: WordPenalty0 start: 1 end: 1
>> line=PhrasePenalty
>> FeatureFunction: PhrasePenalty0 start: 2 end: 2
>> line=PhraseDictionaryMemory name=TranslationModel0 num-features=4 path=/mhmd/working/train/model/phrase-table.gz input-factor=0 output-factor=0
>> FeatureFunction: TranslationModel0 start: 3 end: 6
>> line=LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/mhmd/working/train/model/reordering-table.wbe-msd-bidirectional-fe.gz
>> FeatureFunction: LexicalReordering0 start: 7 end: 12
>> Initializing Lexical Reordering Feature..
>> line=Distortion
>> FeatureFunction: Distortion0 start: 13 end: 13
>> line=KENLM lazyken=0 name=LM0 factor=0 path=/mhmd/lm/news-commentary-v8.fr-en.blm.en order=3
>> FeatureFunction: LM0 start: 14 end: 14
>> Loading UnknownWordPenalty0
>> Loading WordPenalty0
>> Loading PhrasePenalty0
>> Loading LexicalReordering0
>> Loading table into memory...done.
>> Loading Distortion0
>> Loading LM0
>> Loading TranslationModel0
>> Start loading text phrase table. Moses format : [226.350] seconds
>> Reading /mhmd/working/train/model/phrase-table.gz
>> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>> ********************************************************************************Killed
>>
>> mohammed hassanien Mohammed
>> Egyption Programmers Vice-captain
>> 01000121556
>> Egyption Programmers Syndicate
>> <http://www.egprogrammers.org/>
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> --
> The University of Edinburgh is a charitable body, registered in
> Scotland, with registration number SC005336.
>



------------------------------

Message: 2
Date: Thu, 26 Feb 2015 19:28:59 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] Moses killed
To: mhmd hassnen <mhmd_hasnen@yahoo.com>
Cc: Moses-support Support <moses-support@mit.edu>
Message-ID: <54EF73FB.1020905@staffmail.ed.ac.uk>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Use one of the binarised phrase tables.

http://www.statmt.org/moses/?n=Advanced.RuleTables

On 26/02/15 18:45, mhmd hassnen wrote:
> Hi Barry
> Thank you for your response
> My machine ram is 8 GB what can i do
>
> Sent from my iPhone
>
>> On Feb 26, 2015, at 4:54 PM, Barry Haddow <bhaddow@staffmail.ed.ac.uk> wrote:
>>
>> Hi Mohamed
>>
>> The most likely explanation is that your machine ran out of memory,
>>
>> cheers
>> Barry
>>
>>> On 26/02/15 13:04, mohamed hasanien wrote:
>>> Hi All
>>>
>>> i try to run these tow command
>>> ~/mosesdecoder/bin/moses -f ~/working/train/model/moses.ini Comment alle
>>> ~/mosesdecoder/bin/moses -f ~/working/mert-work/moses.ini Comment alle
>>> i alwayes get killd error
>>> the output
>>> --------------------------------------------------------
>>> Defined parameters (per moses.ini or switch):
>>> config: /mhmd/working/train/model/moses.ini Comment allez-vous
>>> distortion-limit: 6
>>> feature: UnknownWordPenalty WordPenalty PhrasePenalty PhraseDictionaryMemory name=TranslationModel0 num-features=4 path=/mhmd/working/train/model/phrase-table.gz input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/mhmd/working/train/model/reordering-table.wbe-msd-bidirectional-fe.gz Distortion KENLM lazyken=0 name=LM0 factor=0 path=/mhmd/lm/news-commentary-v8.fr-en.blm.en order=3
>>> input-factors: 0
>>> mapping: 0 T 0
>>> weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2 TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3 0.3 Distortion0= 0.3 LM0= 0.5
>>> line=UnknownWordPenalty
>>> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
>>> line=WordPenalty
>>> FeatureFunction: WordPenalty0 start: 1 end: 1
>>> line=PhrasePenalty
>>> FeatureFunction: PhrasePenalty0 start: 2 end: 2
>>> line=PhraseDictionaryMemory name=TranslationModel0 num-features=4 path=/mhmd/working/train/model/phrase-table.gz input-factor=0 output-factor=0
>>> FeatureFunction: TranslationModel0 start: 3 end: 6
>>> line=LexicalReordering name=LexicalReordering0 num-features=6 type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0 path=/mhmd/working/train/model/reordering-table.wbe-msd-bidirectional-fe.gz
>>> FeatureFunction: LexicalReordering0 start: 7 end: 12
>>> Initializing Lexical Reordering Feature..
>>> line=Distortion
>>> FeatureFunction: Distortion0 start: 13 end: 13
>>> line=KENLM lazyken=0 name=LM0 factor=0 path=/mhmd/lm/news-commentary-v8.fr-en.blm.en order=3
>>> FeatureFunction: LM0 start: 14 end: 14
>>> Loading UnknownWordPenalty0
>>> Loading WordPenalty0
>>> Loading PhrasePenalty0
>>> Loading LexicalReordering0
>>> Loading table into memory...done.
>>> Loading Distortion0
>>> Loading LM0
>>> Loading TranslationModel0
>>> Start loading text phrase table. Moses format : [226.350] seconds
>>> Reading /mhmd/working/train/model/phrase-table.gz
>>> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
>>> ********************************************************************************Killed
>>>
>>> mohammed hassanien Mohammed
>>> Egyption Programmers Vice-captain
>>> 01000121556
>>> Egyption Programmers Syndicate
>>> <http://www.egprogrammers.org/>
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>> --
>> The University of Edinburgh is a charitable body, registered in
>> Scotland, with registration number SC005336.
>>



------------------------------

Message: 3
Date: Thu, 26 Feb 2015 21:28:51 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] My phrase-table.tgz is 20-bytes long
To: ????????? ??????? <deadyaga@gmail.com>
Cc: moses-support@mit.edu
Message-ID: <54EF9013.2050206@staffmail.ed.ac.uk>
Content-Type: text/plain; charset="utf-8"

Hi Alexander

From the error logs, it looks as though alignment went fine, the
training pipeline reports 24860460 lines of aligned bitext. Since the
extract files were empty, I'd suggest that extraction crashed, and the
most likely is that it ran out of disk. I'm not sure what happened to
the error messages.

For 25M sentence pairs, the final phrase table could easily be 30G and
the intermediate files are larger. You probably need more like 500G to
be safe.

I would follow Tom's advice and start with a much smaller corpus to see
how the process works. Also, for the full corpus, you could look in to
fast_align (https://github.com/clab/fast_align) for alignment as it is
much faster than mgiza (e.g. 2 days versus 2 weeks), and use EMS for
large jobs since it's much easier to restart a failed step.

cheers - Barry

On 26/02/15 15:06, ????????? ??????? wrote:
>
> Hi Barry!
>
> Here you can download training.out
> https://www.dropbox.com/s/d0f0n99x4wbw3mo/training.out.gz?dl=1
>
> I have about 50 Gb of free space in working dir.
>
>
> 2015-02-25 17:19 GMT+07:00 Barry Haddow <bhaddow@staffmail.ed.ac.uk
> <mailto:bhaddow@staffmail.ed.ac.uk>>:
>
> Hi Alexander,
>
> It looks like something went wrong at the extract stage. If you
> could make your training.out available then we can look for clues.
>
> Could the system have run out of disk space, either in the working
> directory or in /tmp? A lot of space is required to build the
> extract files and phrase tables.
>
> cheers - Barry
>
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150226/effaf3fb/attachment-0001.htm

------------------------------

Message: 4
Date: Thu, 26 Feb 2015 16:44:54 -0500
From: Philipp Koehn <phi@jhu.edu>
Subject: Re: [Moses-support] Fwd: Re: BadDiscountException
To: Kenneth Heafield <moses@kheafield.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAAFADDBoh8c8PyC1N2nDTNzGgZzbWOXdGvPXcMF0s9cLmsGOig@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8

Hui,

the wrapper script really just exists, because SRILM (and the wrapper)
sets the name of the produced LM file with "-lm" and lmplz sets it
with "-arpa". If you allow as an alternative name for the switch
"-lm", I'll remove it.

-phi

On Tue, Feb 24, 2015 at 8:39 PM, Kenneth Heafield <moses@kheafield.com> wrote:
> Try removing this bit of text and just calling the lmplz binary
> directly. It's not clear to me why that wrapper script still exists.
>
> $moses-script-dir/ems/support/lmplz-wrapper.perl -bin
>
>
> -------- Forwarded Message --------
> Subject: Re: [Moses-support] BadDiscountException
> Date: Tue, 24 Feb 2015 06:16:43 -0800
> From: fatma elzahraa Eltaher <fatmaeltaher@gmail.com>
> To: Kenneth Heafield <moses@kheafield.com>
>
>
>
> I use kenlm model and when try to add --discount_fallback=1 for setting
> I get this error Unknown option: discount_fallback.
> I attached config.toy where must I change to solve this problem ?
>
>
> thank you,
>
>
>
> Fatma El-Zahraa El -Taher
>
> Teaching Assistant at Computer & System department
>
> Faculty of Engineering, Azhar University
>
> Email : fatmaeltaher@gmail.com <mailto:fatmaeltaher@gmail.com>
> mobile: +201141600434
>
>
> On Tue, Feb 24, 2015 at 5:22 AM, Kenneth Heafield <moses@kheafield.com
> <mailto:moses@kheafield.com>> wrote:
>
> The closed-form estimates for Kneser-Ney are not well-defined on toy or
> class-based data. I recommend using more training data. If this is a
> class-based model, pass --discount_fallback.
>
> Kenneth
>
> On 02/24/2015 08:04 AM, fatma elzahraa Eltaher wrote:
> > Dears,
> > I get the following error in LM_toy_train.65.STDERR:
> > Unigram tokens 25188 types 39
> > === 2/5 Calculating and sorting adjusted counts ===
> > Chain sizes: 1:468 2:322921696 3:605478272 4:968765120 5:1412782592
> > /home/fatma/Desktop/Folder/mosesdecoder/lm/builder/adjust_counts.cc:50
> > in void
> > lm::builder::{anonymous}::StatCollector::CalculateDiscounts(const
> > lm::builder::DiscountConfig&) threw BadDiscountException because
> `s.n[j]
> > == 0'.
> > Could not calculate Kneser-Ney discounts for 1-grams with adjusted
> count
> > 4 because we didn't observe any 1-grams with adjusted count 3; Is this
> > small or artificial data?
> > How do I fix it?
> >
> >
> > thank you,
> >
> >
> >
> > Fatma El-Zahraa El -Taher
> >
> > Teaching Assistant at Computer & System department
> >
> > Faculty of Engineering, Azhar University
> >
> > Email : fatmaeltaher@gmail.com <mailto:fatmaeltaher@gmail.com>
> <mailto:fatmaeltaher@gmail.com <mailto:fatmaeltaher@gmail.com>>
> > mobile: +201141600434
> >
> >
> >
> > _______________________________________________
> > Moses-support mailing list
> > Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> > http://mailman.mit.edu/mailman/listinfo/moses-support
> >
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 100, Issue 92
**********************************************

0 Response to "Moses-support Digest, Vol 100, Issue 92"

Post a Comment

Subscribe to: Post Comments (Atom)