Home » Uncategories » Moses-support Digest, Vol 97, Issue 46

Moses-support Digest, Vol 97, Issue 46

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Re: Moses Build Error: Failed gcc.link (Rajen Chatterjee)
2. Positive log probability (emna hkiri)
3. Re: Positive log probability (Barry Haddow)
4. preprocessing data and BLEU scores (Stefy D.)
5. Re: Training Progress indicator (Matthias Huck)


----------------------------------------------------------------------

Message: 1
Date: Mon, 17 Nov 2014 13:34:28 +0100
From: Rajen Chatterjee <rajen.k.chatterjee@gmail.com>
Subject: Re: [Moses-support] Moses Build Error: Failed gcc.link
To: Yu Tracy Chen <yuchen@coli.uni-sb.de>
Cc: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<CAC4-+NwrJ2N+WULFi93a0yG10ziUgJWVSf61CHd4DVtS4kh+9w@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

ok thanks

On Mon, Nov 17, 2014 at 11:04 AM, Yu Tracy Chen <yuchen@coli.uni-sb.de>
wrote:

> Hi,
>
> You should probably try to recompile an earlier version of SRILM, earlier
> than 1.7.1.
>
> Cheers,
> Yu
>
> On Mon, Nov 17, 2014 at 10:48 AM, Rajen Chatterjee <
> rajen.k.chatterjee@gmail.com> wrote:
>
>> Hello Moses Support Group can you help me to fix this problem?
>>
>> On Fri, Nov 14, 2014 at 2:30 PM, Rajen Chatterjee <
>> rajen.k.chatterjee@gmail.com> wrote:
>>
>>> Hi Everyone,
>>>
>>> When I build moses with the following command it works:
>>> ./bjam --with-boost=/home/chatterjee/Public/SMT/boost_1_55_0 -j4
>>>
>>>
>>> but when I try to build with SRILM with the following command it shows
>>> error "failed gcc.link":(PFA log file)
>>> ./bjam --with-boost=/home/chatterjee/Public/SMT/boost_1_55_0
>>> --with-srilm=/home/chatterjee/Public/SMT/srilm-1.7.1 -j4
>>>
>>> Did anyone face similar problem and any solution to it?
>>>
>>>
>>> PS: SRILM is installed successfully and all test cases produced
>>> identical result. So I guess there is no problem with SRILM installation.
>>>
>>> --
>>> -Regards,
>>> Rajen Chatterjee.
>>>
>>
>>
>>
>> --
>> -Regards,
>> Rajen Chatterjee.
>>
>> _______________________________________________
>> Moses-support mailing list
>> Moses-support@mit.edu
>> http://mailman.mit.edu/mailman/listinfo/moses-support
>>
>>
>


--
-Regards,
Rajen Chatterjee.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141117/0b163f47/attachment-0001.htm

------------------------------

Message: 2
Date: Mon, 17 Nov 2014 14:29:31 +0100
From: emna hkiri <emna.hkiri@gmail.com>
Subject: [Moses-support] Positive log probability
To: moses-support@mit.edu
Message-ID:
<CAAp-nZ3Q6aLo4Do-8-Wgv-QWF9DOjwNrHWCiJhC5TXqVg=_AQw@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Dear Friends
i'm trying to build an arabic language model.
the error is in the binarise of the .arpa file command (
/home/emna/emna/mosesdecoder/
bin/build_binary 2000.arpa.ar \
2000.blm.ar)

i have this error would you help me please/

lm/read_arpa.cc:151 in void lm::PositiveProbWarn::Warn(float) threw
FormatLoadException'.
Positive log probability 2.40965e-07 in the model. This is a bug in
IRSTLM; you can set config.positive_log_probability = SILENT or pass -i to
build_binary to substitute 0.0 for the log probability. Error in the
3-gram at byte 195895800 Byte: 195895800 File: 2000.arpa.ar
ERROR


the code is as follows:
/home/emna/emna/irstlm/bin/add-start-end.sh \
< /home/emna/emna/corpus/2000.true.ar \
> 2000.sb.ar
export IRSTLM=$HOME/emna/irstlm; /home/emna/emna/irstlm/bin/build-lm.sh \
-i 2000.sb.ar \
-t ./tmp -p -s improved-kneser-ney -o 2000.lm.ar
/home/emna/emna/irstlm/bin/compile-lm --text 2000.lm.ar.gz \
2000.arpa.ar
/home/emna/emna/mosesdecoder/bin/build_binary 2000.arpa.ar \
2000.blm.ar
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141117/cd90eed3/attachment-0001.htm

------------------------------

Message: 3
Date: Mon, 17 Nov 2014 14:02:06 +0000
From: Barry Haddow <bhaddow@staffmail.ed.ac.uk>
Subject: Re: [Moses-support] Positive log probability
To: emna hkiri <emna.hkiri@gmail.com>, moses-support@mit.edu
Message-ID: <5469FFDE.1070808@staffmail.ed.ac.uk>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed

Hi Emna

Did you try the workaround suggested in the error message?

> pass -i to build_binary to substitute 0.0 for the log probability.

cheers - Barry

On 17/11/14 13:29, emna hkiri wrote:
> Dear Friends
> i'm trying to build an arabic language model.
> the error is in the binarise of the .arpa file command (
> /home/emna/emna/mosesdecoder/
> bin/build_binary 2000.arpa.ar <http://2000.arpa.ar> \
> 2000.blm.ar <http://2000.blm.ar>)
>
> i have this error would you help me please/
>
> lm/read_arpa.cc:151 in void lm::PositiveProbWarn::Warn(float) threw
> FormatLoadException'.
> Positive log probability 2.40965e-07 in the model. This is a bug in
> IRSTLM; you can set config.positive_log_probability = SILENT or pass
> -i to build_binary to substitute 0.0 for the log probability. Error
> in the 3-gram at byte 195895800 Byte: 195895800 File: 2000.arpa.ar
> <http://2000.arpa.ar>
> ERROR
>
>
> the code is as follows:
> /home/emna/emna/irstlm/bin/add-start-end.sh \
> < /home/emna/emna/corpus/2000.true.ar <http://2000.true.ar> \
> > 2000.sb.ar <http://2000.sb.ar>
> export IRSTLM=$HOME/emna/irstlm; /home/emna/emna/irstlm/bin/build-lm.sh \
> -i 2000.sb.ar <http://2000.sb.ar> \
> -t ./tmp -p -s improved-kneser-ney -o 2000.lm.ar <http://2000.lm.ar>
> /home/emna/emna/irstlm/bin/compile-lm --text 2000.lm.ar.gz \
> 2000.arpa.ar <http://2000.arpa.ar>
> /home/emna/emna/mosesdecoder/bin/build_binary 2000.arpa.ar
> <http://2000.arpa.ar> \
> 2000.blm.ar <http://2000.blm.ar>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support


--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



------------------------------

Message: 4
Date: Mon, 17 Nov 2014 14:22:57 +0000 (UTC)
From: "Stefy D." <tsuki_stefy@yahoo.com>
Subject: [Moses-support] preprocessing data and BLEU scores
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID:
<910552725.996597.1416234177275.JavaMail.yahoo@jws100211.mail.ne1.yahoo.com>

Content-Type: text/plain; charset="utf-8"

Hello,
I have a question regarding the preprocessing of data. On the current version of the moses-baseline page the necessary steps for preprocessing data are tokenization, truecasing and then cleaning. Some time ago lowercasing was done instead of truecasing. Could someone please tell me if the multi-bleu.perl script is used with the same arguments no matter if the MT output is lowercased or truecased? I read on another post that in the case of trucasing, the cased BLEU should be used but I am not sure what argument should I use and if this refers to the "lc" argument. No matter what output one has (truecased or lowercased), the "lc" argument should always be used because lowercasing is needed when computing the BLEU score right?

I have many experiments that were ran using tok+lowercased+cleaned data and now I am not sure if I should redo the experiments using truecasing instead of lowercasing.

Another question is if I have many experiments that were trained using data that was tok+lowercased+cleaned,? is it ok to use a recaser on lowercased output, detokenizer and SGML format to evaluate using the mteval-v13a.pl script? (When training using lowercasing for preprocessing data, i used multi-bleu.perl on the lowercased output to get the BLEU score so I always had BLEU on lowercased outputs).
Thank you very much for your time.Respectfully,Stefania

-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20141117/6e6d53e4/attachment-0001.htm

------------------------------

Message: 5
Date: Mon, 17 Nov 2014 14:42:31 +0000
From: Matthias Huck <mhuck@inf.ed.ac.uk>
Subject: Re: [Moses-support] Training Progress indicator
To: i.ramadan@saudisoft.com
Cc: moses-support@mit.edu
Message-ID: <1416235351.21013.178.camel@portedgar>
Content-Type: text/plain; charset="UTF-8"

Hi,

EMS generates the picture of a graph which shows the whole pipeline
you're about to run (with individual steps for the preparation of the
training data, word alignment training, building language models, phrase
extraction, tuning, testing, scoring etc.). It's updated whenever a step
in the pipeline is started, completed, or has crashed.

That graph might be useful to you. You find it in
steps/*/graph.*.{ps,png,dot} in your working directory.

If you use the EMS web interface, then you will be able to see an
estimate of how long the current step is going to take (in the column
"end" of your task page). It might for instance tell you something like
"TRAINING run-giza-inverse m4:it1 44% 136h left". I'm not sure whether
this estimate is particularly accurate, though.

Experiment Management System (EMS):
http://www.statmt.org/moses/?n=FactoredTraining.EMS

Cheers,
Matthias


On Mon, 2014-11-17 at 13:31 +0200, Ihab Ramadan wrote:
> Thanks Raj,
>
> I will try this solution
>
>
>
> From: Raj Dabre [mailto:prajdabre@gmail.com]
> Sent: Monday, November 17, 2014 12:15 PM
> To: i.ramadan@saudisoft.com
> Cc: moses-support@mit.edu
> Subject: Re: [Moses-support] Training Progress indicator
>
>
>
> Ihab,
>
>
> Im afraid that there is no easy way to do that.
>
>
> The only thing you can do is detect which stage of the training you
> are on.
>
>
> For example: There are a total of 7 steps in training: You can simply
> grep the .out file and find out which stage of the training you are
> on.
>
>
> Regards.
>
>
>
>
> On Mon, Nov 17, 2014 at 6:55 PM, Ihab Ramadan
> <i.ramadan@saudisoft.com> wrote:
>
> Thanks Raj for your reply
>
> I already can get the output from the training process I just want to
> get the progress in a percentage format ( 10% -- 20% -- 30% ) or find
> some way to calculate it
>
>
>
>
>
> From: Raj Dabre [mailto:prajdabre@gmail.com]
> Sent: Monday, November 17, 2014 11:31 AM
> To: i.ramadan@saudisoft.com
> Cc: moses-support@mit.edu
> Subject: Re: [Moses-support] Training Progress indicator
>
>
>
> Are you using: nohup nice -n 19 (training script line) >
> training.out& ????
>
>
> if so keep looking at the tail of training.out (or whichever .out file
> you are using) to check for progress.
>
>
>
>
> On Mon, Nov 17, 2014 at 6:10 PM, Ihab Ramadan
> <i.ramadan@saudisoft.com> wrote:
>
> Dear All,
>
> Is there any way to know the progress of the training process or any
> other process ?
>
>
>
> Note: lately I do not receive reply for my questions , how to make
> sure that I have no problem in receiving emails from your side?
>
> Thanks
>
>
>
> Best Regards
>
> Ihab Ramadan| Senior Developer| Saudisoft - Egypt | Tel +2 02 330 320
> 37 Ext- 0 | Mob+201007570826 | Fax+20233032036 | Follow us on
> linked | ZA102637861 | ZA102637858
>
>
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
>
>
>
>
> --
>
> Raj Dabre.
> Research Student,
>
>
> Graduate School of Informatics,
> Kyoto University.
>
> CSE MTech, IITB., 2011-2014
>
>
>
>
>
> --
>
> Raj Dabre.
> Research Student,
>
>
> Graduate School of Informatics,
> Kyoto University.
>
> CSE MTech, IITB., 2011-2014
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support



--
The University of Edinburgh is a charitable body, registered in
Scotland, with registration number SC005336.



------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 97, Issue 46
*********************************************

0 Response to "Moses-support Digest, Vol 97, Issue 46"

Post a Comment

Subscribe to: Post Comments (Atom)