Moses-support Digest, Vol 107, Issue 42

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Oldest version of boost to work with --with-mm
(Ulrich Germann)
2. Re: Help on pipeline .... (Vincent Nguyen)
3. Re: (no subject) (Rico Sennrich)


----------------------------------------------------------------------

Message: 1
Date: Wed, 16 Sep 2015 18:56:49 +0100
From: Ulrich Germann <ulrich.germann@gmail.com>
Subject: Re: [Moses-support] Oldest version of boost to work with
--with-mm
To: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Cc: moses-support <moses-support@mit.edu>
Message-ID:
<CAHQSRUobUS0FD-NNMnhMODZ=PxMJ+Gp1P4uTNiZRHwG=BL7EAQ@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

1.55 should work. That's what we use at UEdin.

- Uli

On Wed, Sep 16, 2015 at 3:11 PM, Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
wrote:

> Hi,
> what's the currently oldest version of boost for moses with the
> --with-mm option? It seems boost 1.54 is not supported any more although
> that is still standard for the current Ubuntu LTS? It works with a
> by-hand installation of 1.59, I haven't tried any in-betweeners.
> Best,
> Marcin
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>


--
Ulrich Germann
Senior Researcher
School of Informatics
University of Edinburgh
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150916/7ef0ede0/attachment-0001.html

------------------------------

Message: 2
Date: Thu, 17 Sep 2015 09:35:54 +0200
From: Vincent Nguyen <vnguyen@neuf.fr>
Subject: Re: [Moses-support] Help on pipeline ....
To: moses-support@mit.edu
Message-ID: <55FA6D5A.5080606@neuf.fr>
Content-Type: text/plain; charset="windows-1252"


Well in fact, it comes to how to handle xml mark up in text to translate
and obviously this is not trivial and no good solution.

If I am not mistaken this issue has been tackled in various circumstances :
- casmacat project / matecat_util project
- web translation in the contrib / web translate.cgi script (which
handles html tags)

I found old posts from Philipp saying the exact location of mark ups
could be determined with phrase and word alignment .....

well if I am not mistaken :

let's take this example :
<g id="1">Les banques de la zone <g id="2">euro</g> sont soumises :</g>

only the word "euro" is tagged g2

if "zone euro" is phrase translated to "euro zone" then Moses will only
send back the phrase alignment. So we lose the information
and we cannot tag g2 the word "euro".

I guess to prevent this one did try to get the Word alignment using
ibm1align.py
in this
https://github.com/casmacat/moses-mt-server/tree/master/code/tags4moses

BUT it requires the lex vocabulary files which is to me not a good
solution since it must temper a lot the performance.

In the end, I don't really see any good solution without making Moses
xml aware .....

big debate ?




Le 16/09/2015 17:30, Vincent Nguyen a ?crit :
>
> I am struggling with a pipeline .....
>
> Here is the text1.txt file I would like to translate from FR to EN
> <g id="1">Les banques de la zone euro sont soumises :</g>
> <g id="1">au ratio de capital li? ? la d?tention d?actifs risqu?s
> (nous nous int?ressons ici au cr?dit) ;</g>
> <g id="1">au ratio de levier, qui d?termine le capital r?glementaire ?
> partir de la taille du bilan de la banque ;</g>
> <g id="1">au ratio de liquidit?, qui impose aux banques de d?tenir en
> particulier des portefeuilles importants de titres publics.</g>
>
> I am running the following properly :
>
> /home/moses/mosesdecoder/scripts/tokenizer/normalize-punctuation.perl
> fr < text1.txt > text2.txt
> /home/moses/matecat/matecat_util/code/tokenizer/deescape-special-chars.perl
> < text2.txt > text3.txt
> /home/moses/matecat/matecat_util/code/tokenizer/tokenizer.perl -X -a
> -l fr < text3.txt > text4.txt
> /home/moses/mosesdecoder/scripts/recaser/truecase.perl --model
> /home/moses/working/truecaser/truecase-model.1.fr < text4.txt > text5.txt
> /home/moses/mosesdecoder/bin/moses -f
> /home/moses/working/tuning/moses.tuned.ini.1 < text5.txt > text6.txt
>
> then in my text6.txt I have
>
> <g id="1"> banks in the euro zone are subject :</g>
> <g id="1"> ratio of capital linked to the detention of risky assets (
> we are here to credit ;</g> )
> <g id="1"> the leverage ratio , which determines the regulatory
> capital from the size of the balance sheet of the bank ;</g>
> <g id="1"> ratio of liquidity , which requires banks to hold
> especially important portfolios of securities .</g> public
>
> but then neither the detokenizer nor the detruecaser will give me the
> correct output.
> "banks" will not get the uppercase B
>
>
> I also tried to look at this
> https://github.com/christianbuck/matecat_util/tree/master/python_server
> or this
> https://github.com/christianbuck/matecat_util/tree/master/code/tags4moses
>
> but no luck.
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150917/69b41ad2/attachment-0001.html

------------------------------

Message: 3
Date: Thu, 17 Sep 2015 09:24:58 +0100
From: Rico Sennrich <rico.sennrich@gmx.ch>
Subject: Re: [Moses-support] (no subject)
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <55FA78DA.9000604@gmx.ch>
Content-Type: text/plain; charset="utf-8"

Hi Sanjanasri,

if you first compiled moses without the option '--with-nplm', and then
add the option later, the build system isn't smart enough to know which
files it needs to recompile. if you change one of the compile options,
use the option '-a' to force recompilation from scratch.

best wishes,
Rico



On 16/09/15 06:30, Sanjanashree Palanivel wrote:
> Dear Rico,
>
>
> I did the following steps
>
>
> 1. Installed NPLM and trained a language model
> 2. I compiled it with Moses with the command ./bjam
> --with-nplm=path/to/nplm
>
> ./bjam --with-nplm=/home/sanjana/Documents/SMT/NPLM/nplm
> Tip: install tcmalloc for faster threading. See
> BUILD-INSTRUCTIONS.txt for more information.
> warning: No toolsets are configured.
> warning: Configuring default toolset "gcc".
> warning: If the default is wrong, your build may not work correctly.
> warning: Use the "toolset=xxxxx" option to override our guess.
> warning: For more configuration options, please consult
> warning:
> http://boost.org/boost-build2/doc/html/bbv2/advanced/configuration.html
> NOT BUILDING MOSES SERVER!
> Performing configuration checks
>
> - Shared Boost : yes (cached)
> - Static Boost : yes (cached)
> ...patience...
> ...patience...
> ...found 4823 targets...
> SUCCESS
>
> 3. I added the the following lines to the moses.ini file
>
> NeuralLM factor=0 name=LM1 order=5 path=/path/to/nplmmodel
> LM1= 0.5
>
> Then i did testing. and end up with the error
>
>
> On Tue, Sep 15, 2015 at 8:43 PM, Rico Sennrich <rico.sennrich@gmx.ch
> <mailto:rico.sennrich@gmx.ch>> wrote:
>
> Hi Sanjanasri,
>
> this error occurs when Moses was compiled without the option
> '--with-nplm'.
>
> best wishes,
> Rico
>
>
>
> On 15.09.2015 15 <tel:15.09.2015%2015>:08, Sanjanashree Palanivel
> wrote:
>> Dear Rico,
>>
>> I updated moses and NPLM has been compiled
>> succesfully with moses. However, when I perform decoding I am
>> getting an error.
>>
>> Defined parameters (per moses.ini or switch):
>> config:
>> /home/sanjana/Documents/SMT/ICON15/Health/BL/Ta_H/model/moses.ini
>>
>> distortion-limit: 6
>> feature: UnknownWordPenalty WordPenalty PhrasePenalty
>> PhraseDictionaryMemory name=TranslationModel0 num-features=4
>> path=/home/sanjana/Documents/SMT/ICON15/Health/BL/Ta_H/model/phrase-table.gz
>> input-factor=0 output-factor=0 Distortion KENLM lazyken=0
>> name=LM0 factor=0
>> path=/home/sanjana/Documents/SMT/LM/Hindi/monolin80k.hi1.bin
>> order=3 NeuralLM factor=0 name=LM1 order=3
>> path=/home/sanjana/Documents/SMT/LM/Hindi/hin_out.txt
>> input-factors: 0
>> mapping: 0 T 0
>> weight: Distortion0= 0.136328 LM0= 0.135599 LM1= 0.5
>> WordPenalty0= -0.488892 PhrasePenalty0= 0.0826147
>> TranslationModel0= 0.0104273 0.0663914 0.0254094 0.0543384
>> UnknownWordPenalty0= 1
>> line=UnknownWordPenalty
>> FeatureFunction: UnknownWordPenalty0 start: 0 end: 0
>> line=WordPenalty
>> FeatureFunction: WordPenalty0 start: 1 end: 1
>> line=PhrasePenalty
>> FeatureFunction: PhrasePenalty0 start: 2 end: 2
>> line=PhraseDictionaryMemory name=TranslationModel0
>> num-features=4
>> path=/home/sanjana/Documents/SMT/ICON15/Health/BL/Ta_H/model/phrase-table.gz
>> input-factor=0 output-factor=0
>> FeatureFunction: TranslationModel0 start: 3 end: 6
>> line=Distortion
>> FeatureFunction: Distortion0 start: 7 end: 7
>> line=KENLM lazyken=0 name=LM0 factor=0
>> path=/home/sanjana/Documents/SMT/LM/Hindi/monolin80k.hi1.bin
>> order=3
>> FeatureFunction: LM0 start: 8 end: 8
>> line=NeuralLM factor=0 name=LM1 order=3
>> path=/home/sanjana/Documents/SMT/LM/Hindi/hin_out.txt
>> Exception: moses/FF/Factory.cpp:349 in void
>> Moses::FeatureRegistry::Construct(const string&, const
>> string&) threw UnknownFeatureException because `i ==
>> registry_.end()'.
>> Feature name NeuralLM is not registered.
>>
>>
>> I added following 2 lines in my moses file
>>
>> NeuralLM factor=0 name=LM1 order=5 path=/path/to/nplmmodel
>> LM1= 0.5
>>
>>
>>
>> On Tue, Sep 15, 2015 at 5:06 PM, Sanjanashree Palanivel
>> <sanjanashree@gmail.com <mailto:sanjanashree@gmail.com>> wrote:
>>
>> Thank you for your earnest response. I will update moses and
>> I will try
>>
>> On Tue, Sep 15, 2015 at 4:22 PM, Rico Sennrich
>> <rico.sennrich@gmx.ch <mailto:rico.sennrich@gmx.ch>> wrote:
>>
>> Hello Sanjanasri,
>>
>> this looks like a version mismatch between Moses and
>> NPLM. Specifically, you're using an older Moses commit
>> that is only compatible with nplm 0.2 (or specifically,
>> Kenneth's fork at https://github.com/kpu/nplm ).
>>
>> If you use the latest Moses version from
>> https://github.com/moses-smt/mosesdecoder , and the
>> latest nplm version from
>> https://github.com/moses-smt/nplm , it should work.
>>
>> best wishes,
>> Rico
>>
>>
>> On 15.09.2015 08 <tel:15.09.2015%2008>:24, Sanjanashree
>> Palanivel wrote:
>>>
>>> Dear all,
>>>
>>> I tried building language model using
>>> NPLM. Llanguage model was build succesfully, but, when I
>>> tried to compile NPLM with Moses using "./bjam
>>> --with-nplm=path/to/nplm" I am getting an error. I am
>>> using boost 1.55. I am attaching the log file for
>>> reference. I dont know where I went wrong. Any help
>>> would be appreciated.
>>>
>>>
>>> --
>>> Thanks and regards,
>>>
>>> Sanjanasri J.P
>>>
>>>
>>> _______________________________________________
>>> Moses-support mailing list
>>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>>
>>
>> --
>> Thanks and regards,
>>
>> Sanjanasri J.P
>>
>>
>>
>>
>> --
>> Thanks and regards,
>>
>> Sanjanasri J.P
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu <mailto:Moses-support@mit.edu>
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
> --
> Thanks and regards,
>
> Sanjanasri J.P

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20150917/2fbf8560/attachment.html

------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 107, Issue 42
**********************************************

Tweet