Send Moses-support mailing list submissions to
moses-support@mit.edu
To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu
You can reach the person managing the list at
moses-support-owner@mit.edu
When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."
Today's Topics:
1. Re: Do lattice decoding and lexicalized reordering tables
play nicely? (Hieu Hoang)
2. Re: a few supposedly simple questions.. (Hieu Hoang)
3. Running moses fills up hard disk space.. (Andrew Shin)
4. getting WER metrics (Andrew Shin)
5. Big problem for the incremental training (=?GB2312?B?zO/Kog==?=)
----------------------------------------------------------------------
Message: 1
Date: Sun, 27 Oct 2013 19:44:30 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] Do lattice decoding and lexicalized
reordering tables play nicely?
To: moses-support@MIT.EDU
Message-ID: <526D6D1E.9050700@gmail.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
On 24/10/2013 17:59, jeremy@gwinnup.org wrote:
> Hi,
>
> I'm attempting to decode lattices while using a lexicalized reordering
> table using Moses 1.0 - Other than the normal lattice settings
> (inputtype=2, weight-i, max-phrase-len set to ridiculously large value)
> are there any other gotchas that I'm not aware of?
i think that's the main gotchas. If you're using the github source,
instead of release-1.0, a large max phrase length also increases memory
usage a lot
>
> Thanks!
> -Jeremy
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
>
------------------------------
Message: 2
Date: Sun, 27 Oct 2013 19:53:01 +0000
From: Hieu Hoang <hieuhoang@gmail.com>
Subject: Re: [Moses-support] a few supposedly simple questions..
To: moses-support@mit.edu
Message-ID: <526D6F1D.4050909@gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
On 26/10/2013 06:55, Andrew Shin wrote:
> Dear support team,
>
> thank you for your previous reply which worked out for me.
> I have a few questions which I think should be simple but couldn't
> find relevant information on the website.
>
> 1) When you run Moses and type in a sentence, is there any way you
> could have the translation with
> the corresponding probability?
run moses with the argument
-output-hypo-score
>
> 2) Also when you run and type in a sentence, is there a way to have
> not just one translation,
> but N-best candidates? (preferably with corresponding probabilities,
> which was the first question..)
run moses with the arguments
-n-best-list nbest.txt 100
>
> 3) I've done getting BLEU score using moses, but is there a way to
> also get word error rate to a reference?
>
> 4) After cleaning process, moses shows the number of lines in input
> and output text files,
> but I noticed that number of lines in output file decreased about
> 5%,resulting in non-matching number of lines
> for input and output.
> Looking at the translation results, it seems like it worked fine
> somehow, but it gets me concerned.
> Why is it, and does it affect the line-match of input-output and the
> training process?
How exactly do you clean your your data?
Non-matching number of lines should never happen. The script
clean-corpus-n.perl
takes in both the source and target language corpus files. If it discard
a sentence in 1 language, it should also discard it in the other
language. If it behaves any differently, please let us know
>
>
> I truly appreciate your help in advance.
>
> best,
> Andrew
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131027/d9e872d3/attachment-0001.htm
------------------------------
Message: 3
Date: Mon, 28 Oct 2013 12:09:02 +0900
From: Andrew Shin <ravenyj@hotmail.com>
Subject: [Moses-support] Running moses fills up hard disk space..
To: "moses-support@mit.edu" <moses-support@MIT.EDU>
Message-ID: <BLU171-W4DC5CB446A6D79FC19B7BB2080@phx.gbl>
Content-Type: text/plain; charset="iso-2022-jp"
Hello,
I'm running Moses after binarising it,and while it's reading from phrase table, it rapidly fills up hard disk space until there is no more space..I had 21GB and it wasn't enough..Once I quit terminal, the space is freed again..I successfully ran Moses before, so I don't understand why this happens..Is this natural? If so, how much reserve space do I need?and If not, what could have gone wrong?
Thank you very much for your help in advance.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131028/4130e385/attachment-0001.htm
------------------------------
Message: 4
Date: Mon, 28 Oct 2013 12:11:12 +0900
From: Andrew Shin <ravenyj@hotmail.com>
Subject: [Moses-support] getting WER metrics
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <BLU171-W1373AC85C5A83E122FD6111B2080@phx.gbl>
Content-Type: text/plain; charset="iso-2022-jp"
Hello,sorry to ask another question..
I've done getting BLEU score in the past following the baseline tutorial,but is there a way to also get WER given a reference text?
Thank you very much for your help.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131028/8c648cc6/attachment-0001.htm
------------------------------
Message: 5
Date: Mon, 28 Oct 2013 13:57:45 +0800
From: =?GB2312?B?zO/Kog==?= <kasim1891@gmail.com>
Subject: [Moses-support] Big problem for the incremental training
To: moses-support@mit.edu
Message-ID:
<CA+XoTiDu5Ph_Lx8fytf4r0Yx0b3EKsCPgNdxNq2270RVQMe86A@mail.gmail.com>
Content-Type: text/plain; charset="iso-8859-1"
Hi,
These days i am trying the incremental training on moses. But i meet with a
big problem i can't handle. I would appreciate it If some instruction or
advice can be given.
Problem:
i followed the tutorial of the incremental training, trained the data with
inc-giza-pp, got the moses.ini file and modified it as the tutorial says.
Then i start up the moseserver and it can work. Then i tried to add new
data to mosesserver by XMLrpc. It was strange that sometimes it could
continue to work and sometimes it stopped. In the case of success, if i
restart up the mosesserver after close the process, the error are shown in
attachment error.txt.
The attachment config is the EMS config i used. And The language model i
used is the irstlm.
My computer info is as follows:
Memory:8G
disk:100G
Processor: Intel Core i7-3770
System type 64-bit
Operating system:Ubuntu
Look forward to your reply.
Thanks.
Kasim
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20131028/5c5aa109/attachment.htm
-------------- next part --------------
Running single-threaded server
Defined parameters (per moses.ini or switch):
config: /home/kasim/working/fr-en/moses.ini
distortion-file: 0-0 wbe-msd-bidirectional-fe-allff 6 /home/kasim/working/fr-en/evaluation/newstest2011.filtered.1/reordering-table.1.wbe-msd-bidirectional-fe
distortion-limit: 6
input-factors: 0
lmodel-file: 8 0 3 /home/kasim/working/fr-en/lm/project-syndicate.binlm.1
mapping: 0 T 0
ttable-file: 8 0 0 3 /home/kasim/moses-data/fr-en/tm/corpus.1.fr /home/kasim/moses-data/fr-en/tm/corpus.1.en /home/kasim/moses-data/fr-en/tm/aligned.1.grow-diag-final-and
ttable-limit: 20
weight-d: 0.061213 0.0900232 0.0228936 0.0704107 0.0786754 0.00104701 0.0836828
weight-l: 0.0802438
weight-t: 0.0359469 0.0291794 0.103474 0.0713217 0.194258
weight-w: -0.0776305
/home/kasim/mosesdecoder/bin
Loading lexical distortion models...have 1 models
Creating lexical reordering...
weights: 0.0900232 0.0228936 0.0704107 0.0786754 0.00104701 0.0836828
binary file loaded, default OFF_T: -1
Start loading LanguageModel /home/kasim/working/fr-en/lm/project-syndicate.binlm.1 : [0.0249781] seconds
Finished loading LanguageModels : [0.0432379] seconds
Start loading PhraseTable /home/kasim/moses-data/fr-en/tm/corpus.1.fr : [0.0433415] seconds
filePath: /home/kasim/moses-data/fr-en/tm/corpus.1.fr
Finished loading phrase tables : [0.0541465] seconds
Start loading phrase table from /home/kasim/moses-data/fr-en/tm/corpus.1.fr : [0.0542286] seconds
Loading source corpus...
Loading target corpus...
Building Source Suffix Array...
DYNAMIC SUFFIX ARRAY CLASS INSTANTIATED WITH SIZE 320605
Building Target Suffix Array...
(Skipped. Not used)
Loading Alignment File...
10000Building frequent word cache...
*** Error in `/home/kasim/mosesdecoder/bin/mosesserver': free(): invalid next size (normal): 0x0000000002f0f6e0 ***
======= Backtrace: =========
/lib/x86_64-linux-gnu/libc.so.6(+0x80a46)[0x7fd2d73d1a46]
/home/kasim/mosesdecoder/bin/mosesserver[0x500275]
/home/kasim/mosesdecoder/bin/mosesserver[0x5009c6]
/home/kasim/mosesdecoder/bin/mosesserver[0x5056d6]
/home/kasim/mosesdecoder/bin/mosesserver[0x4ebf62]
/home/kasim/mosesdecoder/bin/mosesserver[0x418d65]
/home/kasim/mosesdecoder/bin/mosesserver[0x41911b]
/home/kasim/mosesdecoder/bin/mosesserver[0x4b8264]
/home/kasim/mosesdecoder/bin/mosesserver[0x4915ed]
/home/kasim/mosesdecoder/bin/mosesserver[0x40c863]
/lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xf5)[0x7fd2d7372ea5]
/home/kasim/mosesdecoder/bin/mosesserver[0x40fca1]
======= Memory map: ========
00400000-00661000 r-xp 00000000 08:01 4070057 /home/kasim/mosesdecoder/bin/mosesserver
00861000-00862000 r--p 00261000 08:01 4070057 /home/kasim/mosesdecoder/bin/mosesserver
00862000-00863000 rw-p 00262000 08:01 4070057 /home/kasim/mosesdecoder/bin/mosesserver
00863000-00866000 rw-p 00000000 00:00 0
0226e000-0355b000 rw-p 00000000 00:00 0 [heap]
7fd2d5038000-7fd2d5574000 rw-p 00000000 00:00 0
7fd2d5737000-7fd2d5e0b000 r--s 00000000 08:01 6029559 /home/kasim/working/fr-en/lm/project-syndicate.binlm.1
7fd2d5e0b000-7fd2d64ec000 r--p 00000000 08:01 3283233 /usr/lib/locale/locale-archive
7fd2d64ec000-7fd2d6505000 r-xp 00000000 08:01 3286070 /usr/lib/libxmlrpc_xmltok.so.3.16
7fd2d6505000-7fd2d6704000 ---p 00019000 08:01 3286070 /usr/lib/libxmlrpc_xmltok.so.3.16
7fd2d6704000-7fd2d6706000 r--p 00018000 08:01 3286070 /usr/lib/libxmlrpc_xmltok.so.3.16
7fd2d6706000-7fd2d6707000 rw-p 0001a000 08:01 3286070 /usr/lib/libxmlrpc_xmltok.so.3.16
7fd2d6707000-7fd2d6715000 r-xp 00000000 08:01 3286071 /usr/lib/libxmlrpc_xmlparse.so.3.16
7fd2d6715000-7fd2d6914000 ---p 0000e000 08:01 3286071 /usr/lib/libxmlrpc_xmlparse.so.3.16
7fd2d6914000-7fd2d6915000 r--p 0000d000 08:01 3286071 /usr/lib/libxmlrpc_xmlparse.so.3.16
7fd2d6915000-7fd2d6916000 rw-p 0000e000 08:01 3286071 /usr/lib/libxmlrpc_xmlparse.so.3.16
7fd2d6916000-7fd2d6929000 r-xp 00000000 08:01 3286064 /usr/lib/libxmlrpc.so.3.16
7fd2d6929000-7fd2d6b28000 ---p 00013000 08:01 3286064 /usr/lib/libxmlrpc.so.3.16
7fd2d6b28000-7fd2d6b29000 r--p 00012000 08:01 3286064 /usr/lib/libxmlrpc.so.3.16
7fd2d6b29000-7fd2d6b2a000 rw-p 00013000 08:01 3286064 /usr/lib/libxmlrpc.so.3.16
7fd2d6b2a000-7fd2d6b2f000 r-xp 00000000 08:01 3286066 /usr/lib/libxmlrpc_server.so.3.16
7fd2d6b2f000-7fd2d6d2e000 ---p 00005000 08:01 3286066 /usr/lib/libxmlrpc_server.so.3.16
7fd2d6d2e000-7fd2d6d2f000 r--p 00004000 08:01 3286066 /usr/lib/libxmlrpc_server.so.3.16
7fd2d6d2f000-7fd2d6d30000 rw-p 00005000 08:01 3286066 /usr/lib/libxmlrpc_server.so.3.16
7fd2d6d30000-7fd2d6d34000 r-xp 00000000 08:01 3286069 /usr/lib/libxmlrpc_util.so.3.16
7fd2d6d34000-7fd2d6f33000 ---p 00004000 08:01 3286069 /usr/lib/libxmlrpc_util.so.3.16
7fd2d6f33000-7fd2d6f34000 r--p 00003000 08:01 3286069 /usr/lib/libxmlrpc_util.so.3.16
7fd2d6f34000-7fd2d6f35000 rw-p 00004000 08:01 3286069 /usr/lib/libxmlrpc_util.so.3.16
7fd2d6f35000-7fd2d6f49000 r-xp 00000000 08:01 3286063 /usr/lib/libxmlrpc_abyss.so.3.16
7fd2d6f49000-7fd2d7148000 ---p 00014000 08:01 3286063 /usr/lib/libxmlrpc_abyss.so.3.16
7fd2d7148000-7fd2d7149000 r--p 00013000 08:01 3286063 /usr/lib/libxmlrpc_abyss.so.3.16
7fd2d7149000-7fd2d714a000 rw-p 00014000 08:01 3286063 /usr/lib/libxmlrpc_abyss.so.3.16
7fd2d714a000-7fd2d714f000 r-xp 00000000 08:01 3286067 /usr/lib/libxmlrpc_server_abyss.so.3.16
7fd2d714f000-7fd2d734f000 ---p 00005000 08:01 3286067 /usr/lib/libxmlrpc_server_abyss.so.3.16
7fd2d734f000-7fd2d7350000 r--p 00005000 08:01 3286067 /usr/lib/libxmlrpc_server_abyss.so.3.16
7fd2d7350000-7fd2d7351000 rw-p 00006000 08:01 3286067 /usr/lib/libxmlrpc_server_abyss.so.3.16
7fd2d7351000-7fd2d750f000 r-xp 00000000 08:01 1052320 /lib/x86_64-linux-gnu/libc-2.17.so
7fd2d750f000-7fd2d770e000 ---p 001be000 08:01 1052320 /lib/x86_64-linux-gnu/libc-2.17.so
7fd2d770e000-7fd2d7712000 r--p 001bd000 08:01 1052320 /lib/x86_64-linux-gnu/libc-2.17.so
7fd2d7712000-7fd2d7714000 rw-p 001c1000 08:01 1052320 /lib/x86_64-linux-gnu/libc-2.17.so
7fd2d7714000-7fd2d7719000 rw-p 00000000 00:00 0
7fd2d7719000-7fd2d772d000 r-xp 00000000 08:01 1052345 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd2d772d000-7fd2d792d000 ---p 00014000 08:01 1052345 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd2d792d000-7fd2d792e000 r--p 00014000 08:01 1052345 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd2d792e000-7fd2d792f000 rw-p 00015000 08:01 1052345 /lib/x86_64-linux-gnu/libgcc_s.so.1
7fd2d792f000-7fd2d7a32000 r-xp 00000000 08:01 1052368 /lib/x86_64-linux-gnu/libm-2.17.so
7fd2d7a32000-7fd2d7c32000 ---p 00103000 08:01 1052368 /lib/x86_64-linux-gnu/libm-2.17.so
7fd2d7c32000-7fd2d7c33000 r--p 00103000 08:01 1052368 /lib/x86_64-linux-gnu/libm-2.17.so
7fd2d7c33000-7fd2d7c34000 rw-p 00104000 08:01 1052368 /lib/x86_64-linux-gnu/libm-2.17.so
7fd2d7c34000-7fd2d7d19000 r-xp 00000000 08:01 3284940 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17
7fd2d7d19000-7fd2d7f18000 ---p 000e5000 08:01 3284940 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17
7fd2d7f18000-7fd2d7f20000 r--p 000e4000 08:01 3284940 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17
7fd2d7f20000-7fd2d7f22000 rw-p 000ec000 08:01 3284940 /usr/lib/x86_64-linux-gnu/libstdc++.so.6.0.17
7fd2d7f22000-7fd2d7f37000 rw-p 00000000 00:00 0
7fd2d7f37000-7fd2d7f58000 r-xp 00000000 08:01 3286087 /usr/lib/libxmlrpc++.so.4.16
7fd2d7f58000-7fd2d8157000 ---p 00021000 08:01 3286087 /usr/lib/libxmlrpc++.so.4.16
7fd2d8157000-7fd2d8158000 r--p 00020000 08:01 3286087 /usr/lib/libxmlrpc++.so.4.16
7fd2d8158000-7fd2d8159000 rw-p 00021000 08:01 3286087 /usr/lib/libxmlrpc++.so.4.16
7fd2d8159000-7fd2d8171000 r-xp 00000000 08:01 1052432 /lib/x86_64-linux-gnu/libpthread-2.17.so
7fd2d8171000-7fd2d8370000 ---p 00018000 08:01 1052432 /lib/x86_64-linux-gnu/libpthread-2.17.so
7fd2d8370000-7fd2d8371000 r--p 00017000 08:01 1052432 /lib/x86_64-linux-gnu/libpthread-2.17.so
7fd2d8371000-7fd2d8372000 rw-p 00018000 08:01 1052432 /lib/x86_64-linux-gnu/libpthread-2.17.so
7fd2d8372000-7fd2d8376000 rw-p 00000000 00:00 0
7fd2d8376000-7fd2d837f000 r-xp 00000000 08:01 3286083 /usr/lib/libxmlrpc_server++.so.4.16
7fd2d837f000-7fd2d857f000 ---p 00009000 08:01 3286083 /usr/lib/libxmlrpc_server++.so.4.16
7fd2d857f000-7fd2d8580000 r--p 00009000 08:01 3286083 /usr/lib/libxmlrpc_server++.so.4.16
7fd2d8580000-7fd2d8581000 rw-p 0000a000 08:01 3286083 /usr/lib/libxmlrpc_server++.so.4.16
7fd2d8581000-7fd2d8587000 r-xp 00000000 08:01 3286084 /usr/lib/libxmlrpc_server_abyss++.so.4.16
7fd2d8587000-7fd2d8786000 ---p 00006000 08:01 3286084 /usr/lib/libxmlrpc_server_abyss++.so.4.16
7fd2d8786000-7fd2d8787000 r--p 00005000 08:01 3286084 /usr/lib/libxmlrpc_server_abyss++.so.4.16
7fd2d8787000-7fd2d8788000 rw-p 00006000 08:01 3286084 /usr/lib/libxmlrpc_server_abyss++.so.4.16
7fd2d8788000-7fd2d878f000 r-xp 00000000 08:01 1052440 /lib/x86_64-linux-gnu/librt-2.17.so
7fd2d878f000-7fd2d898e000 ---p 00007000 08:01 1052440 /lib/x86_64-linux-gnu/librt-2.17.so
7fd2d898e000-7fd2d898f000 r--p 00006000 08:01 1052440 /lib/x86_64-linux-gnu/librt-2.17.so
7fd2d898f000-7fd2d8990000 rw-p 00007000 08:01 1052440 /lib/x86_64-linux-gnu/librt-2.17.so
7fd2d8990000-7fd2d89b3000 r-xp 00000000 08:01 1052296 /lib/x86_64-linux-gnu/ld-2.17.so
7fd2d8a0a000-7fd2d8b9b000 rw-p 00000000 00:00 0
7fd2d8ba8000-7fd2d8ba9000 rw-p 00000000 00:00 0
7fd2d8ba9000-7fd2d8bb0000 r--s 00000000 08:01 3541845 /usr/lib/x86_64-linux-gnu/gconv/gconv-modules.cache
7fd2d8bb0000-7fd2d8bb2000 rw-p 00000000 00:00 0
7fd2d8bb2000-7fd2d8bb3000 r--p 00022000 08:01 1052296 /lib/x86_64-linux-gnu/ld-2.17.so
7fd2d8bb3000-7fd2d8bb5000 rw-p 00023000 08:01 1052296 /lib/x86_64-linux-gnu/ld-2.17.so
7fffbaf7a000-7fffbaf9b000 rw-p 00000000 00:00 0 [stack]
7fffbaffe000-7fffbb000000 r-xp 00000000 00:00 0 [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0 [vsyscall]
-------------- next part --------------
A non-text attachment was scrubbed...
Name: config
Type: application/octet-stream
Size: 5257 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20131028/5c5aa109/attachment.obj
-------------- next part --------------
A non-text attachment was scrubbed...
Name: moses.ini
Type: application/octet-stream
Size: 1596 bytes
Desc: not available
Url : http://mailman.mit.edu/mailman/private/moses-support/attachments/20131028/5c5aa109/attachment-0001.obj
------------------------------
_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support
End of Moses-support Digest, Vol 84, Issue 39
*********************************************
Subscribe to:
Post Comments (Atom)
0 Response to "Moses-support Digest, Vol 84, Issue 39"
Post a Comment