Moses-support Digest, Vol 106, Issue 39

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: sigtest filtering reordering (Vincent Nguyen)


----------------------------------------------------------------------

Message: 1
Date: Wed, 19 Aug 2015 14:32:33 +0200
From: Vincent Nguyen <vnguyen@neuf.fr>
Subject: Re: [Moses-support] sigtest filtering reordering
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Cc: moses-support <moses-support@mit.edu>
Message-ID: <55D47761.40404@neuf.fr>
Content-Type: text/plain; charset="utf-8"

ok I am confused

I had a step 1 run done with no sigtest filtering.
My step 2 is just adding the sigtest filtering (then tune, evaluation, ...)

During step 2, sigtest-filter-suffix-array went through ok and the 2
processes sigtest-filter-ttable and reordering are running simultaneously.
so when you ask if I already have a filtered phrase table, you are
mentioning the one being build buy this step 2 sigtest-filter-ttable ?

thanks for your help

NB: I am doing this to evaluate the BLEU impact but I just saw a post
form Phil saying sigtest decreases Belu scores most if the time ....


Le 19/08/2015 14:10, Marcin Junczys-Dowmunt a ?crit :
>
> Hi,
>
> I guess that was the operation system killing the process due to lack
> of memory. Do you have a filtered phrase-table already? If yes, you
> can just remove the spurious reordering entries with the script
> remove-orphaned-reordering-entries.perl (someting like that, I am
> writing this from memory.). You give the pruned phrase-table and the
> unpruned reordering model to the script, and the script takes care
> that the contents match. The good thing is, is hardly requires any RAM.
>
> Best,
>
> Marcin
>
> W dniu 2015-08-19 13:44, Vincent Nguyen napisa?(a):
>
>> Hi,
>>
>> it crashed (whereas the sigtest filetring ttable continues ...) and
>> no message for disk space nor out of memory.
>>
>> just a simple "killed" at the end of the stderr, any clue ?
>>
>>
>> -l = a+e
>> P(f|e) filter limit: 50
>> Loading Vocabulary...
>> Loading existing vocabulary file:
>> /home/moses/working/training/corpus.2.en.id_voc
>> Total 1891716 word types loaded
>> Max VocID=1891716
>> Vocabulary loaded in 25 seconds.
>> Loading corpus...
>> Corpus loaded in 3 seconds.
>> Loading suffix...
>> Initialize level-1 buckets...
>> Suffix loaded in 54 seconds.
>> Loading offset...
>> Offset loaded in 2 seconds.
>> Total: 23400935 sentences loaded.
>> Loading Vocabulary...
>> Loading existing vocabulary file:
>> /home/moses/working/training/corpus.2.fr.id_voc
>> Total 1952991 word types loaded
>> Max VocID=1952991
>> Vocabulary loaded in 26 seconds.
>> Loading corpus...
>> Corpus loaded in 14 seconds.
>> Loading suffix...
>> Initialize level-1 buckets...
>> Suffix loaded in 97 seconds.
>> Loading offset...
>> Offset loaded in 2 seconds.
>> Total: 23400935 sentences loaded.
>> Training corpus: 23400935 lines
>> \alpha = 16.9683
>> Sig filter threshold is = 16.9693
>> ..................................................[n:500000]
>> ..................................................[n:1000000]
>> ..................................................[n:1500000]
>> ..................................................[n:2000000]
>> ..................................................[n:2500000]
>> ..................................................[n:3000000]
>> ..................................................[n:3500000]
>> ..................................................[n:4000000]
>> ..................................................[n:4500000]
>> ..................................................[n:5000000]
>> ..................................................[n:5500000]
>> ..................................................[n:6000000]
>> ..................................................[n:6500000]
>> ..................................................[n:7000000]
>> ..................................................[n:7500000]
>> ..................................................[n:8000000]
>> ..................................................[n:8500000]
>> ..................................................[n:9000000]
>> ..................................................[n:9500000]
>> ..................................................[n:10000000]
>> ------------------------------------------------------
>> unfiltered phrases pairs: 10000000
>> P(f|e) filter [first]: 1749725 (17.4972%)
>> significance filter: 6426892 (64.2689%)
>> TOTAL FILTERED: 8176617 (81.7662%)
>> FILTERED phrase pairs: 1823383 (18.2338%)
>> ------------------------------------------------------
>> ..................................................[n:10500000]
>> ..................................................[n:11000000]
>> ..................................................[n:11500000]
>> ..................................................[n:12000000]
>> ..................................................[n:12500000]
>> ..................................................[n:13000000]
>> ..................................................[n:13500000]
>> ..................................................[n:14000000]
>> ..................................................[n:14500000]
>> ..................................................[n:15000000]
>> ..................................................[n:15500000]
>> ..................................................[n:16000000]
>> ..................................................[n:16500000]
>> ..................................................[n:17000000]
>> ..................................................[n:17500000]
>> ..................................................[n:18000000]
>> ..................................................[n:18500000]
>> ..................................................[n:19000000]
>> ..................................................[n:19500000]
>> ..................................................[n:20000000]
>> ------------------------------------------------------
>> unfiltered phrases pairs: 20000000
>> P(f|e) filter [first]: 3905428 (19.5271%)
>> significance filter: 12645364 (63.2268%)
>> TOTAL FILTERED: 16550792 (82.754%)
>> FILTERED phrase pairs: 3449208 (17.246%)
>> ------------------------------------------------------
>> ..................................................[n:20500000]
>> ..................................................[n:21000000]
>> ..................................................[n:21500000]
>> ..................................................[n:22000000]
>> ..................................................[n:22500000]
>> ..................................................[n:23000000]
>> ..................................................[n:23500000]
>> ..................................................[n:24000000]
>> ..................................................[n:24500000]
>> ..................................................[n:25000000]
>> ..................................................[n:25500000]
>> ..................................................[n:26000000]
>> ..................................................[n:26500000]
>> ..................................................[n:27000000]
>> ..................................................[n:27500000]
>> ..................................................[n:28000000]
>> ..................................................[n:28500000]
>> ..................................................[n:29000000]
>> ..................................................[n:29500000]
>> ..................................................[n:30000000]
>> ------------------------------------------------------
>> unfiltered phrases pairs: 30000000
>> P(f|e) filter [first]: 6941355 (23.1378%)
>> significance filter: 18163654 (60.5455%)
>> TOTAL FILTERED: 25105009 (83.6834%)
>> FILTERED phrase pairs: 4894991 (16.3166%)
>> ------------------------------------------------------
>> ..................................................[n:30500000]
>> ..................................................[n:31000000]
>> ..................................................[n:31500000]
>> ..................................................[n:32000000]
>> ..................................................[n:32500000]
>> ..................................................[n:33000000]
>> ..................................................[n:33500000]
>> ..................................................[n:34000000]
>> ..................................................[n:34500000]
>> ..................................................[n:35000000]
>> ..................................................[n:35500000]
>> ..................................................[n:36000000]
>> ..................................................[n:36500000]
>> ..................................................[n:37000000]
>> ..................................................[n:37500000]
>> ..................................................[n:38000000]
>> ..................................................[n:38500000]
>> ..................................................[n:39000000]
>> ..................................................[n:39500000]
>> ..................................................[n:40000000]
>> ------------------------------------------------------
>> unfiltered phrases pairs: 40000000
>> P(f|e) filter [first]: 10132773 (25.3319%)
>> significance filter: 23513247 (58.7831%)
>> TOTAL FILTERED: 33646020 (84.1151%)
>> FILTERED phrase pairs: 6353980 (15.8849%)
>> ------------------------------------------------------
>> ..................................................[n:40500000]
>> ..................................................[n:41000000]
>> ..................................................[n:41500000]
>> ..................................................[n:42000000]
>> ..................................................[n:42500000]
>> ..................................................[n:43000000]
>> ..................................................[n:43500000]
>> ..................................................[n:44000000]
>> ..................................................[n:44500000]
>> ..................................................[n:45000000]
>> ..................................................[n:45500000]
>> ..................................................[n:46000000]
>> ..................................................[n:46500000]
>> ..................................................[n:47000000]
>> ..................................................[n:47500000]
>> ..................................................[n:48000000]
>> ..................................................[n:48500000]
>> .....................................Killed
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150819/9a27cc93/attachment-0001.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 106, Issue 39
**********************************************

Tweet