Moses-support Digest, Vol 158, Issue 4

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Segmentation fault (core dumped) while translating
(Hieu Hoang)


----------------------------------------------------------------------

Message: 1
Date: Tue, 10 Dec 2019 20:49:42 -0800
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Segmentation fault (core dumped) while
translating
To: Claudia Matos Veliz <claudia.matosveliz@ugent.be>, moses-support
<moses-support@mit.edu>
Message-ID: <ba6fcf73-13ae-634b-af64-6a1011d198a1@gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed

hi there

the input sentence can't contain <> characters.

If you had tokenized the input with the moses script

??? /scripts/tokenizer/tokenizer.perl
then those characters would have been converted to &lt; &gt;

If you used your own tokenizer, be sure to

?? /scripts/tokenizer/escape-special-chars.perl

before feeding it to the moses decoder.

And of course, you should process your training data the same way you
process your test and tuning data

On 12/10/2019 2:13 AM, Claudia Matos Veliz wrote:
> Hello everyone
>
> I?m getting a very estrange error while translating using Moses. I?m
> doing 10-CV experiments and in one of the folds I get a Segmentation
> Fault error. For the rest everything finished without problem. I hope
> you can help me with that. Bellow you can see the log. I get the error
> in the first line of the file I want to translate. I?ve tried removing
> the line and I got the same error.
>
> $ /opt/mosesdecoder-4.0-prebuilt/moses/bin/moses -config
> ../model/moses.ini -input-file token.all.fold2.test.clean.ori >
> token.all.fold2.test.norm
> Defined parameters (per moses.ini or switch):
> config: ../model/moses.ini
> distortion-limit: 6
> feature: UnknownWordPenalty WordPenalty PhrasePenalty
> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
> path=/home/claudia/SMT_ext_data/data/mixed/amica/1folds/2/model/phrase-table.table
> input-factor=0 output-factor=0 LexicalReordering
> name=LexicalReordering0 num-features=6
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/home/claudia/SMT_ext_data/data/mixed/amica/1folds/2/model/reordering-table.wbe-msd-bidirectional-fe.gz
> Distortion KENLM name=LM0 factor=0
> path=/home/claudia/SMT_ext_data/data/lm/lm/sms.cgn.token.lm order=5
> input-factors: 0
> input-file: token.all.fold2.test.clean.ori
> mapping: 0 T 0
> weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2
> TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3
> 0.3 0.3 Distortion0= 0.3 LM0= 0.5
> line=UnknownWordPenalty
> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
> line=WordPenalty
> FeatureFunction: WordPenalty0 start: 1 end: 1
> line=PhrasePenalty
> FeatureFunction: PhrasePenalty0 start: 2 end: 2
> line=PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
> path=/home/claudia/SMT_ext_data/data/mixed/amica/1folds/2/model/phrase-table.table
> input-factor=0 output-factor=0
> FeatureFunction: TranslationModel0 start: 3 end: 6
> line=LexicalReordering name=LexicalReordering0 num-features=6
> type=wbe-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/home/claudia/SMT_ext_data/data/mixed/amica/1folds/2/model/reordering-table.wbe-msd-bidirectional-fe.gz
> Initializing Lexical Reordering Feature..
> FeatureFunction: LexicalReordering0 start: 7 end: 12
> line=Distortion
> FeatureFunction: Distortion0 start: 13 end: 13
> line=KENLM name=LM0 factor=0
> path=/home/claudia/SMT_ext_data/data/lm/lm/sms.cgn.token.lm order=5
> Loading the LM will be faster if you build a binary file.
> Reading /home/claudia/SMT_ext_data/data/lm/lm/sms.cgn.token.lm
> ----5---10---15---20---25---30---35---40---45---50---55---60---65---70---75---80---85---90---95--100
> ****************************************************************************************************
> FeatureFunction: LM0 start: 14 end: 14
> Loading UnknownWordPenalty0
> Loading WordPenalty0
> Loading PhrasePenalty0
> Loading LexicalReordering0
> Loading table into memory...done.
> Loading Distortion0
> Loading LM0
> Loading TranslationModel0
> Created input-output object : [2.910] seconds
> Translating: <emoji> ik vind die zo mooii . ik hou van je <emoji>
> Segmentation fault (core dumped)
>
>
>
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

--
Hieu Hoang
http://statmt.org/hieu



------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 158, Issue 4
*********************************************

Tweet