Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Unicode Issues when Using Compact Phrase Table, Binaries
vs. Own Build (????????? ????? (Ventsislav Zhechev))
2. Segmentation fault when loading a generation model in the
decoder (Miguel Gra?a)
3. Requirement of sorting tables before compacting: unclear
documentation (Eleftherios Avramidis)
----------------------------------------------------------------------
Message: 1
Date: Tue, 31 Mar 2015 11:34:12 +0200
From: "????????? ????? (Ventsislav Zhechev)"
<contact@VentsislavZhechev.eu>
Subject: Re: [Moses-support] Unicode Issues when Using Compact Phrase
Table, Binaries vs. Own Build
To: moses-support@mit.edu
Message-ID:
<98E4B5EA-4F44-42D4-982A-7286739584B6@VentsislavZhechev.eu>
Content-Type: text/plain; charset=utf-8
Hi,
Any clue what systems could be messed up? On Ubuntu I complied boost 1.57, cmph and Moses right out of the box, so I don?t see what I could have done wrong there.
I just checked and the gzip phrase tables are proper UTF-8. I even ran the processPhraseTableMin binary from the website on the Ubuntu machine and still got the same results. That is, if I query the compact phrase table with the queryPhraseTableMin binary from the website, UTF-8 is recognised and I get results; if I use queryPhraseTableMin that I complied on the same system, UTF-8 is not recognised and I get no results.
Does anyone have an idea what could influence the compilation of Moses in a way that would prevent it from properly reading UTF-8?
Especially given that the Moses binaries for MacOS X from the website don?t seem to read UTF-8 properly (at least on my machine), and I didn?t compile those.
Cheers,
Ventzi
> 30.03.2015 ?., ? 11:08, moses-support-request@mit.edu ???????(?):
>
> Date: Mon, 30 Mar 2015 11:08:13 +0200
> From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
> Subject: Re: [Moses-support] Unicode Issues when Using Compact Phrase
> Table, Binaries vs. Own Build
> To: moses-support@mit.edu
> Message-ID: <5519127D.7080206@amu.edu.pl>
> Content-Type: text/plain; charset="utf-8"
>
> Hi,
> the phrase-table and as far as I know Moses in general are
> unicode-agnostic, as long as you use utf-8. Input is handled as raw byte
> sequences, most of the time there are numeric identifiers only.
> Sounds more like a couple of messed up systems on your side, especially
> the part where self-compiled systems work or don't work. Cannot give you
> much more insight, unfortunately.
> Best,
> Marcin
------------------------------
Message: 2
Date: Tue, 31 Mar 2015 12:34:36 +0200
From: Miguel Gra?a <miguelrgraca@gmail.com>
Subject: [Moses-support] Segmentation fault when loading a generation
model in the decoder
To: moses-support@mit.edu
Message-ID:
<CANBbWMPsjPs6vzK-pcO02q8YN8LOvjzNzAdv5DtJmURG=eNsAA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hello,
I am trying to configure Moses to use a language model over word classes.
To do this, I've added a generation model, which is basically just a
mapping from a word to a word class and a generation step after the
translation step.
The issue is that while loading the generation model I get a segmentation
fault. I am using a custom word mapping file, so there might be issues with
the structure of the file. I have tried to follow the layout as shown in
your factored training tutorial. A line in the mapping file is split into 4
fields: <target_word> <word_class> 1 0
The output of the command "./mosesdecoder/bin/moses -i
/work/smt2/graca/iwslt/2013/de-en/translation/pbt.all.wclm.hrm.moses.0/f-demo
-f moses.ini -v 3 > out" is given below:
I have also tried not including the generation model and the language model
over word classes and in that case I do not obtain a segmentation fault.
Regards,
Miguel
--------------------------------------
Defined parameters (per moses.ini or switch):
config: moses.ini
distortion-limit: 6
feature: UnknownWordPenalty WordPenalty PhrasePenalty
PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
path=/work/smt2/graca/iwslt/2013/de-en/training/pbt.moses/model/filtered/phrase-table.bin
input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0
num-features=6 type=hier-msd-bidirectional-fe-allff input-factor=0
output-factor=0
path=/work/smt2/graca/iwslt/2013/de-en/training/pbt.moses/model/filtered/reordering-table.hier-msd-bidirectional-fe
Distortion KENLM name=LM0 factor=0 path=lm order=4 KENLM name=LM1 factor=1
path=wclm order=7 Generation name=GEN path=e.classes.ext input-factor=0
output-factor=1
input-factors: 0
input-file:
/work/smt2/graca/iwslt/2013/de-en/translation/pbt.all.wclm.hrm.moses.0/f-demo
mapping: 0 T 0 0 G 1
verbose: 3
weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2
TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3
0.3 Distortion0= 0.3 LM0= 0.5 LM1= 0.5 GEN= 1
input type is: text input
line=UnknownWordPenalty
FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
line=WordPenalty
FeatureFunction: WordPenalty0 start: 1 end: 1
line=PhrasePenalty
FeatureFunction: PhrasePenalty0 start: 2 end: 2
line=PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
path=/work/smt2/graca/iwslt/2013/de-en/training/pbt.moses/model/filtered/phrase-table.bin
input-factor=0 output-factor=0
FeatureFunction: TranslationModel0 start: 3 end: 6
DecodeFeature:
line=LexicalReordering name=LexicalReordering0 num-features=6
type=hier-msd-bidirectional-fe-allff input-factor=0 output-factor=0
path=/work/smt2/graca/iwslt/2013/de-en/training/pbt.moses/model/filtered/reordering-table.hier-msd-bidirectional-fe
FeatureFunction: LexicalReordering0 start: 7 end: 12
Initializing Lexical Reordering Feature..
line=Distortion
FeatureFunction: Distortion0 start: 13 end: 13
line=KENLM name=LM0 factor=0 path=lm order=4
FeatureFunction: LM0 start: 14 end: 14
line=KENLM name=LM1 factor=1 path=wclm order=7
FeatureFunction: LM1 start: 15 end: 15
line=Generation name=GEN path=e.classes.ext input-factor=0 output-factor=1
FeatureFunction: GEN start: 16 end: 16
DecodeFeature:
Loading UnknownWordPenalty0
Loading WordPenalty0
Loading PhrasePenalty0
Loading LexicalReordering0
Loading table into memory...done.
Loading Distortion0
Loading LM0
Loading LM1
Loading GEN
./queueTrans.sh: line 1: 13276 Segmentation fault (core dumped)
./mosesdecoder/bin/moses -i
/work/smt2/graca/iwslt/2013/de-en/translation/pbt.all.wclm.hrm.moses.0/f-demo
-f moses.ini -v 3 > out
------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150331/134215e7/attachment-0001.htm
------------------------------
Message: 3
Date: Tue, 31 Mar 2015 16:37:58 +0200
From: Eleftherios Avramidis <eleftherios.avramidis@dfki.de>
Subject: [Moses-support] Requirement of sorting tables before
compacting: unclear documentation
To: moses-support@mit.edu
Message-ID: <551AB146.2030602@dfki.de>
Content-Type: text/plain; charset=utf-8; format=flowed
Hi,
there is an unclear piece of documentation concerning the required
"sorting" of compact phrase and reordering tables
http://www.statmt.org/moses/?n=Advanced.RuleTables
"Phrase tables are required to be sorted as above." but there is no
sorting mentioned "above". It appears twice. Can somebody clarify this
to me, so that I can proceed with table processing; and could you also
possibly fix the documentation?
best
Lefteris
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 101, Issue 84
**********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 101, Issue 84"
Post a Comment