Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Segmentation fault when loading a generation model in the
decoder (Nadir Durrani)
2. ProbingPT tests not building (Jeroen Vermeulen)
3. Re: Binarize phrase table with domain feature throws "Already
saved" exception (Hieu Hoang)
4. Re: ProbingPT tests not building (Nikolay Bogoychev)
----------------------------------------------------------------------
Message: 1
Date: Wed, 1 Apr 2015 13:32:45 +0300
From: Nadir Durrani <nadir.durrani@nu.edu.pk>
Subject: Re: [Moses-support] Segmentation fault when loading a
generation model in the decoder
To: moses-support@mit.edu
Message-ID:
<CAFDj2Q0Wem40nxJc3LknCB8n-MwBtJXZY2svW0LVNPfjcQH4vA@mail.gmail.com>
Content-Type: text/plain; charset=UTF-8
Hi Miguel,
If you just want to use language model over word-classes, here's a set
of instructions to do that
http://www.statmt.org/moses/?n=Advanced.Models#ntoc4
Cheers,
Nadir
On Tue, Mar 31, 2015 at 7:25 PM, <moses-support-request@mit.edu> wrote:
> Send Moses-support mailing list submissions to
> moses-support@mit.edu
>
> To subscribe or unsubscribe via the World Wide Web, visit
> http://mailman.mit.edu/mailman/listinfo/moses-support
> or, via email, send a message with subject or body 'help' to
> moses-support-request@mit.edu
>
> You can reach the person managing the list at
> moses-support-owner@mit.edu
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Moses-support digest..."
>
>
> Today's Topics:
>
> 1. Re: Unicode Issues when Using Compact Phrase Table, Binaries
> vs. Own Build (????????? ????? (Ventsislav Zhechev))
> 2. Segmentation fault when loading a generation model in the
> decoder (Miguel Gra?a)
> 3. Requirement of sorting tables before compacting: unclear
> documentation (Eleftherios Avramidis)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Tue, 31 Mar 2015 11:34:12 +0200
> From: "????????? ????? (Ventsislav Zhechev)"
> <contact@VentsislavZhechev.eu>
> Subject: Re: [Moses-support] Unicode Issues when Using Compact Phrase
> Table, Binaries vs. Own Build
> To: moses-support@mit.edu
> Message-ID:
> <98E4B5EA-4F44-42D4-982A-7286739584B6@VentsislavZhechev.eu>
> Content-Type: text/plain; charset=utf-8
>
> Hi,
>
> Any clue what systems could be messed up? On Ubuntu I complied boost 1.57, cmph and Moses right out of the box, so I don?t see what I could have done wrong there.
>
> I just checked and the gzip phrase tables are proper UTF-8. I even ran the processPhraseTableMin binary from the website on the Ubuntu machine and still got the same results. That is, if I query the compact phrase table with the queryPhraseTableMin binary from the website, UTF-8 is recognised and I get results; if I use queryPhraseTableMin that I complied on the same system, UTF-8 is not recognised and I get no results.
>
> Does anyone have an idea what could influence the compilation of Moses in a way that would prevent it from properly reading UTF-8?
> Especially given that the Moses binaries for MacOS X from the website don?t seem to read UTF-8 properly (at least on my machine), and I didn?t compile those.
>
>
> Cheers,
>
> Ventzi
>
>> 30.03.2015 ?., ? 11:08, moses-support-request@mit.edu ???????(?):
>>
>> Date: Mon, 30 Mar 2015 11:08:13 +0200
>> From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
>> Subject: Re: [Moses-support] Unicode Issues when Using Compact Phrase
>> Table, Binaries vs. Own Build
>> To: moses-support@mit.edu
>> Message-ID: <5519127D.7080206@amu.edu.pl>
>> Content-Type: text/plain; charset="utf-8"
>>
>> Hi,
>> the phrase-table and as far as I know Moses in general are
>> unicode-agnostic, as long as you use utf-8. Input is handled as raw byte
>> sequences, most of the time there are numeric identifiers only.
>> Sounds more like a couple of messed up systems on your side, especially
>> the part where self-compiled systems work or don't work. Cannot give you
>> much more insight, unfortunately.
>> Best,
>> Marcin
>
>
>
>
> ------------------------------
>
> Message: 2
> Date: Tue, 31 Mar 2015 12:34:36 +0200
> From: Miguel Gra?a <miguelrgraca@gmail.com>
> Subject: [Moses-support] Segmentation fault when loading a generation
> model in the decoder
> To: moses-support@mit.edu
> Message-ID:
> <CANBbWMPsjPs6vzK-pcO02q8YN8LOvjzNzAdv5DtJmURG=eNsAA@mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Hello,
>
> I am trying to configure Moses to use a language model over word classes.
> To do this, I've added a generation model, which is basically just a
> mapping from a word to a word class and a generation step after the
> translation step.
>
> The issue is that while loading the generation model I get a segmentation
> fault. I am using a custom word mapping file, so there might be issues with
> the structure of the file. I have tried to follow the layout as shown in
> your factored training tutorial. A line in the mapping file is split into 4
> fields: <target_word> <word_class> 1 0
>
> The output of the command "./mosesdecoder/bin/moses -i
> /work/smt2/graca/iwslt/2013/de-en/translation/pbt.all.wclm.hrm.moses.0/f-demo
> -f moses.ini -v 3 > out" is given below:
>
> I have also tried not including the generation model and the language model
> over word classes and in that case I do not obtain a segmentation fault.
>
> Regards,
> Miguel
> --------------------------------------
>
> Defined parameters (per moses.ini or switch):
> config: moses.ini
> distortion-limit: 6
> feature: UnknownWordPenalty WordPenalty PhrasePenalty
> PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
> path=/work/smt2/graca/iwslt/2013/de-en/training/pbt.moses/model/filtered/phrase-table.bin
> input-factor=0 output-factor=0 LexicalReordering name=LexicalReordering0
> num-features=6 type=hier-msd-bidirectional-fe-allff input-factor=0
> output-factor=0
> path=/work/smt2/graca/iwslt/2013/de-en/training/pbt.moses/model/filtered/reordering-table.hier-msd-bidirectional-fe
> Distortion KENLM name=LM0 factor=0 path=lm order=4 KENLM name=LM1 factor=1
> path=wclm order=7 Generation name=GEN path=e.classes.ext input-factor=0
> output-factor=1
> input-factors: 0
> input-file:
> /work/smt2/graca/iwslt/2013/de-en/translation/pbt.all.wclm.hrm.moses.0/f-demo
> mapping: 0 T 0 0 G 1
> verbose: 3
> weight: UnknownWordPenalty0= 1 WordPenalty0= -1 PhrasePenalty0= 0.2
> TranslationModel0= 0.2 0.2 0.2 0.2 LexicalReordering0= 0.3 0.3 0.3 0.3 0.3
> 0.3 Distortion0= 0.3 LM0= 0.5 LM1= 0.5 GEN= 1
> input type is: text input
> line=UnknownWordPenalty
> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
> line=WordPenalty
> FeatureFunction: WordPenalty0 start: 1 end: 1
> line=PhrasePenalty
> FeatureFunction: PhrasePenalty0 start: 2 end: 2
> line=PhraseDictionaryOnDisk name=TranslationModel0 num-features=4
> path=/work/smt2/graca/iwslt/2013/de-en/training/pbt.moses/model/filtered/phrase-table.bin
> input-factor=0 output-factor=0
> FeatureFunction: TranslationModel0 start: 3 end: 6
> DecodeFeature:
> line=LexicalReordering name=LexicalReordering0 num-features=6
> type=hier-msd-bidirectional-fe-allff input-factor=0 output-factor=0
> path=/work/smt2/graca/iwslt/2013/de-en/training/pbt.moses/model/filtered/reordering-table.hier-msd-bidirectional-fe
> FeatureFunction: LexicalReordering0 start: 7 end: 12
> Initializing Lexical Reordering Feature..
> line=Distortion
> FeatureFunction: Distortion0 start: 13 end: 13
> line=KENLM name=LM0 factor=0 path=lm order=4
> FeatureFunction: LM0 start: 14 end: 14
> line=KENLM name=LM1 factor=1 path=wclm order=7
> FeatureFunction: LM1 start: 15 end: 15
> line=Generation name=GEN path=e.classes.ext input-factor=0 output-factor=1
> FeatureFunction: GEN start: 16 end: 16
> DecodeFeature:
> Loading UnknownWordPenalty0
> Loading WordPenalty0
> Loading PhrasePenalty0
> Loading LexicalReordering0
> Loading table into memory...done.
> Loading Distortion0
> Loading LM0
> Loading LM1
> Loading GEN
> ./queueTrans.sh: line 1: 13276 Segmentation fault (core dumped)
> ./mosesdecoder/bin/moses -i
> /work/smt2/graca/iwslt/2013/de-en/translation/pbt.all.wclm.hrm.moses.0/f-demo
> -f moses.ini -v 3 > out
> ------------------------------------------
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150331/134215e7/attachment-0001.htm
>
> ------------------------------
>
> Message: 3
> Date: Tue, 31 Mar 2015 16:37:58 +0200
> From: Eleftherios Avramidis <eleftherios.avramidis@dfki.de>
> Subject: [Moses-support] Requirement of sorting tables before
> compacting: unclear documentation
> To: moses-support@mit.edu
> Message-ID: <551AB146.2030602@dfki.de>
> Content-Type: text/plain; charset=utf-8; format=flowed
>
> Hi,
>
> there is an unclear piece of documentation concerning the required
> "sorting" of compact phrase and reordering tables
> http://www.statmt.org/moses/?n=Advanced.RuleTables
>
> "Phrase tables are required to be sorted as above." but there is no
> sorting mentioned "above". It appears twice. Can somebody clarify this
> to me, so that I can proceed with table processing; and could you also
> possibly fix the documentation?
>
> best
> Lefteris
>
>
> ------------------------------
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
> End of Moses-support Digest, Vol 101, Issue 84
> **********************************************
------------------------------
Message: 2
Date: Wed, 01 Apr 2015 20:39:56 +0700
From: Jeroen Vermeulen <jtv@precisiontranslationtools.com>
Subject: [Moses-support] ProbingPT tests not building
To: moses-support@mit.edu
Message-ID: <551BF52C.3070705@precisiontranslationtools.com>
Content-Type: text/plain; charset=utf-8
Here's another case where I may be breaking things because I'm building
manually, or something might be genuinely wrong. For me,
moses/TranslationModel/ProbingPT/tests/vocabid_test.cpp doesn't build.
The problem is that main() calls getStringFromID(), which is not defined
anywhere in the codebase that I can see. Obsolete test?
Jeroen
------------------------------
Message: 3
Date: Wed, 1 Apr 2015 19:20:04 +0400
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Binarize phrase table with domain feature
throws "Already saved" exception
To: jian zhang <zhangj@computing.dcu.ie>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAEKMkbi9_VHQDi-Kv8iQNLX5FUxDwR=vmQEJKKd-uMCC1UB+0w@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
strange. is it possible you can make the text phrase-table available for
download
Hieu Hoang
Research Associate
New York University, Abu Dhabi
http://www.hoang.co.uk/hieu
On 1 April 2015 at 14:27, jian zhang <zhangj@computing.dcu.ie> wrote:
> Hi,
>
> When I am binarizing phrase table with domain sparse features, I got
> exception:
>
> Executing: /system/mosesdecoder-master/bin/CreateOnDiskPt 1 1 4 100 2
> /experiments/de-en/domains-30-03-2015/domain-indicator/tuning/filtered.1/phrase-table.0-0.1.1
> /experiments/de-en/domains-30-03-2015/domain-indicator/tuning/filtered.1/phrase-table.0-0.1.1.bin
> terminate called after throwing an instance of 'util::Exception'
> what(): OnDiskPt/PhraseNode.cpp:97 in void
> OnDiskPt::PhraseNode::Save(OnDiskPt::OnDiskWrapper&, size_t, size_t) threw
> util::Exception because `m_saved'.
> Already saved
>
> I am using ems on moses version 3.0, and the configuration for domain
> feature is
> domain-features = "subset"
>
> The phrase table entries look like:
> ! ! ! ! ! ! ! ||| ! ! ! ! ! ! ! ||| 0.41129 0.0415342 0.344595 9.9589e-05
> 2.718 1 1 ||| 0-0 1-1 2-2 3-3 4-4 5-5 6-6 ||| 124 148 51 ||| |||
> ! ! ! ! ! ! ! ||| ! ! ! ! ! ! ||| 0.00574516 0.0415342 0.00590043
> 0.000371446 2.718 1 1 ||| 1-0 0-1 2-1 3-2 4-3 5-4 6-5 ||| 152 148 2 ||| |||
> ! ! ! ! ! ! ! ||| ! ! ! ! ! ||| 0.000320174 0.0415342 0.00044132
> 0.00138541 2.718 1 1 ||| 0-0 1-0 1-1 2-1 3-1 2-2 4-2 3-3 5-3 4-4 6-4 |||
> 204 148 1 ||| |||
>
> Did I miss anything?
>
> Thanks,
>
> Jian
>
> --
> Jian Zhang
> Centre for Next Generation Localisation (CNGL)
> <http://www.cngl.ie/index.html>
> Dublin City University <http://www.dcu.ie/>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150401/0989fb79/attachment-0001.htm
------------------------------
Message: 4
Date: Wed, 1 Apr 2015 16:29:17 +0100
From: Nikolay Bogoychev <nheart@gmail.com>
Subject: Re: [Moses-support] ProbingPT tests not building
To: Jeroen Vermeulen <jtv@precisiontranslationtools.com>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAJzPUEyV42Ahc9M8Pif=AXE_jCdHL6n39Xtoh_+DHceLdgKUAA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"
Hey Jeroen,
Those tests are indeed obsolete, I used them to test some behavior when I
was building probingPT but the function in question became part of the
HuffmanDecoder class as getTargetWordFromID
You don't need to build or worry about those tests (there isn't a Jamfile
in that directory) I still don't have __proper__ tests for probingPT.
Cheers,
Nick
On Wed, Apr 1, 2015 at 2:39 PM, Jeroen Vermeulen <
jtv@precisiontranslationtools.com> wrote:
> Here's another case where I may be breaking things because I'm building
> manually, or something might be genuinely wrong. For me,
> moses/TranslationModel/ProbingPT/tests/vocabid_test.cpp doesn't build.
>
> The problem is that main() calls getStringFromID(), which is not defined
> anywhere in the codebase that I can see. Obsolete test?
>
>
> Jeroen
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150401/aa00dde6/attachment.htm
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 102, Issue 2
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 102, Issue 2"
Post a Comment