Home » Uncategories » Moses-support Digest, Vol 112, Issue 9

Moses-support Digest, Vol 112, Issue 9

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Problem with processPhraseTableMin (Jeremy Gwinnup)
2. Re: Problem with processPhraseTableMin (Marcin Junczys-Dowmunt)
3. Re: Problem with processPhraseTableMin (Kenneth Heafield)
4. Re: Problem with processPhraseTableMin (Marcin Junczys-Dowmunt)


----------------------------------------------------------------------

Message: 1
Date: Tue, 2 Feb 2016 10:21:45 -0500
From: Jeremy Gwinnup <jeremy@gwinnup.org>
Subject: [Moses-support] Problem with processPhraseTableMin
To: moses-support@mit.edu
Message-ID: <3E96F6B1-9361-42B0-867B-4E055C7015A5@gwinnup.org>
Content-Type: text/plain; charset=utf-8

Hi,

I?m having a problem using processPhraseTableMin to compress a phrase table with 7 scores - the program consistently coredumps at step 3 - command and relevant output below. Is there anything I?m doing glaringly wrong?

Thanks!
-Jeremy

Command:

1tqoct1:model> $MOSES/bin/processPhraseTableMin -in phrase-table.1.gz -out phrase-table.1 -threads all -nscores 7

Once we get to step 3:

Intermezzo: Calculating Huffman code sets
Creating Huffman codes for 471366 target phrase symbols
tcmalloc: large alloc 13983629312 bytes == 0xb14ce000 @
tcmalloc: large alloc 27967250432 bytes == 0x3f3ca4000 @
tcmalloc: large alloc 15681406635450368 bytes == (nil) @
terminate called after throwing an instance of 'std::bad_alloc'
what(): std::bad_alloc

Top looked like this when the program ran into trouble:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
27416 jgwinnup 20 0 45.9g 30g 4.0g R 10.6 3.0 1589:17 processPhraseTa


------------------------------

Message: 2
Date: Tue, 02 Feb 2016 16:53:24 +0100
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Problem with processPhraseTableMin
To: moses-support@mit.edu
Message-ID: <56B0D0F4.8090400@amu.edu.pl>
Content-Type: text/plain; charset=UTF-8; format=flowed

I think it fills up your temporary folder, try "-T ." to specify thew
local folder for temporary files.

On 02.02.2016 16:21, Jeremy Gwinnup wrote:
> Hi,
>
> I?m having a problem using processPhraseTableMin to compress a phrase table with 7 scores - the program consistently coredumps at step 3 - command and relevant output below. Is there anything I?m doing glaringly wrong?
>
> Thanks!
> -Jeremy
>
> Command:
>
> 1tqoct1:model> $MOSES/bin/processPhraseTableMin -in phrase-table.1.gz -out phrase-table.1 -threads all -nscores 7
>
> Once we get to step 3:
>
> Intermezzo: Calculating Huffman code sets
> Creating Huffman codes for 471366 target phrase symbols
> tcmalloc: large alloc 13983629312 bytes == 0xb14ce000 @
> tcmalloc: large alloc 27967250432 bytes == 0x3f3ca4000 @
> tcmalloc: large alloc 15681406635450368 bytes == (nil) @
> terminate called after throwing an instance of 'std::bad_alloc'
> what(): std::bad_alloc
>
> Top looked like this when the program ran into trouble:
> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
> 27416 jgwinnup 20 0 45.9g 30g 4.0g R 10.6 3.0 1589:17 processPhraseTa
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support




------------------------------

Message: 3
Date: Tue, 2 Feb 2016 15:58:26 +0000
From: Kenneth Heafield <moses@kheafield.com>
Subject: Re: [Moses-support] Problem with processPhraseTableMin
To: moses-support@mit.edu
Message-ID: <56B0D222.4010600@kheafield.com>
Content-Type: text/plain; charset=utf-8

That typically causes a bus error. Why is there an overly huge malloc?

On 02/02/2016 03:53 PM, Marcin Junczys-Dowmunt wrote:
> I think it fills up your temporary folder, try "-T ." to specify thew
> local folder for temporary files.
>
> On 02.02.2016 16:21, Jeremy Gwinnup wrote:
>> Hi,
>>
>> I?m having a problem using processPhraseTableMin to compress a phrase table with 7 scores - the program consistently coredumps at step 3 - command and relevant output below. Is there anything I?m doing glaringly wrong?
>>
>> Thanks!
>> -Jeremy
>>
>> Command:
>>
>> 1tqoct1:model> $MOSES/bin/processPhraseTableMin -in phrase-table.1.gz -out phrase-table.1 -threads all -nscores 7
>>
>> Once we get to step 3:
>>
>> Intermezzo: Calculating Huffman code sets
>> Creating Huffman codes for 471366 target phrase symbols
>> tcmalloc: large alloc 13983629312 bytes == 0xb14ce000 @
>> tcmalloc: large alloc 27967250432 bytes == 0x3f3ca4000 @
>> tcmalloc: large alloc 15681406635450368 bytes == (nil) @
>> terminate called after throwing an instance of 'std::bad_alloc'
>> what(): std::bad_alloc
>>
>> Top looked like this when the program ran into trouble:
>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>> 27416 jgwinnup 20 0 45.9g 30g 4.0g R 10.6 3.0 1589:17 processPhraseTa
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>


------------------------------

Message: 4
Date: Tue, 02 Feb 2016 17:10:25 +0100
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Problem with processPhraseTableMin
To: moses-support@mit.edu
Message-ID: <56B0D4F1.8080705@amu.edu.pl>
Content-Type: text/plain; charset=UTF-8; format=flowed

Ooh right, didn't look at that. That's weird.

Is there something special about that phrase table other than the 7
scores? It seems to build a huge huffman tree for the the first set of
scores. Based on the reported memory allocation it seems to be quite
big, but not big enough that it should cause any problems.

On 02.02.2016 16:58, Kenneth Heafield wrote:
> That typically causes a bus error. Why is there an overly huge malloc?
>
> On 02/02/2016 03:53 PM, Marcin Junczys-Dowmunt wrote:
>> I think it fills up your temporary folder, try "-T ." to specify thew
>> local folder for temporary files.
>>
>> On 02.02.2016 16:21, Jeremy Gwinnup wrote:
>>> Hi,
>>>
>>> I?m having a problem using processPhraseTableMin to compress a phrase table with 7 scores - the program consistently coredumps at step 3 - command and relevant output below. Is there anything I?m doing glaringly wrong?
>>>
>>> Thanks!
>>> -Jeremy
>>>
>>> Command:
>>>
>>> 1tqoct1:model> $MOSES/bin/processPhraseTableMin -in phrase-table.1.gz -out phrase-table.1 -threads all -nscores 7
>>>
>>> Once we get to step 3:
>>>
>>> Intermezzo: Calculating Huffman code sets
>>> Creating Huffman codes for 471366 target phrase symbols
>>> tcmalloc: large alloc 13983629312 bytes == 0xb14ce000 @
>>> tcmalloc: large alloc 27967250432 bytes == 0x3f3ca4000 @
>>> tcmalloc: large alloc 15681406635450368 bytes == (nil) @
>>> terminate called after throwing an instance of 'std::bad_alloc'
>>> what(): std::bad_alloc
>>>
>>> Top looked like this when the program ran into trouble:
>>> PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
>>> 27416 jgwinnup 20 0 45.9g 30g 4.0g R 10.6 3.0 1589:17 processPhraseTa
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support




------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 112, Issue 9
*********************************************

0 Response to "Moses-support Digest, Vol 112, Issue 9"

Post a Comment

Subscribe to: Post Comments (Atom)