Moses-support Digest, Vol 128, Issue 14

Send Moses-support mailing list submissions to
moses-support@mit.edu

To subscribe or unsubscribe via the World Wide Web, visit
http://mailman.mit.edu/mailman/listinfo/moses-support
or, via email, send a message with subject or body 'help' to
moses-support-request@mit.edu

You can reach the person managing the list at
moses-support-owner@mit.edu

When replying, please edit your Subject line so it is more specific
than "Re: Contents of Moses-support digest..."


Today's Topics:

1. Filtering? (Mike Ladwig)
2. Re: Filtering? (Marcin Junczys-Dowmunt)
3. problem installing Moses (Jan Tore L?nning)


----------------------------------------------------------------------

Message: 1
Date: Fri, 9 Jun 2017 12:14:24 -0400
From: Mike Ladwig <mdladwig@gmail.com>
Subject: [Moses-support] Filtering?
To: Moses Support <moses-support@mit.edu>
Message-ID:
<CAB3VaD1moy1OmSww3JOd1q5z5M-gOH6MBOq8RENo=gQKorWjbA@mail.gmail.com>
Content-Type: text/plain; charset="utf-8"

Anyone working with the Johnson pruning scripts?

I filtered my phrase table with:

zcat /home/mike/stelae-projects/fr-en/phrasemodel/model/phrase-table.gz |
/home/mike/stelae5/mosesdecoder/contrib/sigtest-filter/filter-pt -e
/home/mike/stelae-projects/fr-en/data/data.tok.en -f
/home/mike/stelae-projects/fr-en/data/data.tok.fr -l a+e -n 30 | gzip - >
/home/mike/stelae-projects/fr-en/phrasemodel/phrasefilter//phrase-table.1.gz

and all seemed to go well, finishing up with:

unfiltered phrases pairs: 482680813

P(f|e) filter [first]: 51170825 (10.6014%)
significance filter: 359148261 (74.407%)
TOTAL FILTERED: 410319086 (85.0084%)

FILTERED phrase pairs: 72361727 (14.9916%)

But I was unable to successfully reduce the reordering model:

zcat
/home/mike/stelae-projects/fr-en/phrasemodel/model/reordering-table.wbe-msd-bidirectional-fe.gz
|
/home/mike/stelae5/mosesdecoder/scripts/training/remove-orphan-phrase-pairs-from-reordering-table.perl
/home/mike/stelae-projects/fr-en/phrasemodel/phrasefilter//phrase-table.1 |
gzip - >
/home/mike/stelae-projects/fr-en/phrasemodel/phrasefilter//reordering-table.1.gz
ERROR: trailing ttable lines -> could not find ! ||| ! ||| 0.808885
0.840558 0.606903 0.639666 ||| 0-0 ||| 11412 15210 9231 ||| |||

Any ideas?

Thanks,
mike.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://mailman.mit.edu/mailman/private/moses-support/attachments/20170609/249af8c0/attachment-0001.html

------------------------------

Message: 2
Date: Fri, 9 Jun 2017 18:23:53 +0200
From: Marcin Junczys-Dowmunt <junczys@amu.edu.pl>
Subject: Re: [Moses-support] Filtering?
To: moses-support@mit.edu
Message-ID: <1cf61911-3f4c-f562-976f-80dd6ce52774@amu.edu.pl>
Content-Type: text/plain; charset=utf-8; format=flowed

Hi,
I think a

LC_ALL=C sort

on the filtered phrase-table might help?

W dniu 09.06.2017 o 18:14, Mike Ladwig pisze:
> Anyone working with the Johnson pruning scripts?
>
> I filtered my phrase table with:
>
> zcat
> /home/mike/stelae-projects/fr-en/phrasemodel/model/phrase-table.gz |
> /home/mike/stelae5/mosesdecoder/contrib/sigtest-filter/filter-pt -e
> /home/mike/stelae-projects/fr-en/data/data.tok.en -f
> /home/mike/stelae-projects/fr-en/data/data.tok.fr <http://data.tok.fr>
> -l a+e -n 30 | gzip - >
> /home/mike/stelae-projects/fr-en/phrasemodel/phrasefilter//phrase-table.1.gz
>
> and all seemed to go well, finishing up with:
>
> unfiltered phrases pairs: 482680813
>
> P(f|e) filter [first]: 51170825 (10.6014%)
> significance filter: 359148261 (74.407%)
> TOTAL FILTERED: 410319086 (85.0084%)
>
> FILTERED phrase pairs: 72361727 (14.9916%)
>
> But I was unable to successfully reduce the reordering model:
>
> zcat
> /home/mike/stelae-projects/fr-en/phrasemodel/model/reordering-table.wbe-msd-bidirectional-fe.gz
> |
> /home/mike/stelae5/mosesdecoder/scripts/training/remove-orphan-phrase-pairs-from-reordering-table.perl
> /home/mike/stelae-projects/fr-en/phrasemodel/phrasefilter//phrase-table.1
> | gzip - >
> /home/mike/stelae-projects/fr-en/phrasemodel/phrasefilter//reordering-table.1.gz
> ERROR: trailing ttable lines -> could not find ! ||| ! ||| 0.808885
> 0.840558 0.606903 0.639666 ||| 0-0 ||| 11412 15210 9231 ||| |||
>
> Any ideas?
>
> Thanks,
> mike.
>
>
>
> _______________________________________________
> Moses-support mailing list
> Moses-support@mit.edu
> http://mailman.mit.edu/mailman/listinfo/moses-support




------------------------------

Message: 3
Date: Sat, 10 Jun 2017 15:51:17 +0000
From: Jan Tore L?nning <jtl@ifi.uio.no>
Subject: [Moses-support] problem installing Moses
To: "moses-support@mit.edu" <moses-support@mit.edu>
Message-ID: <cf897cee33f547519f2aedce68331b45@mail-ex13.exprod.uio.no>
Content-Type: text/plain; charset="iso-8859-1"

Hi

I try to install Moses, but have problems with the boost.
I tried to follow the instructions at
http://www.statmt.org/moses/?n=Development.GetStarted

I followed the recipe for manually installing boost, installing it to ~/boost_1_55_0
Then I tried to install moses from within ~/mosesdecoder with the command

bash-4.1$ ./bjam --with-boost=../boost_1_55_0 -j8

This failed.
The log-file says

Unable to load Boost.Build: could not find "boost-build.jam"
---------------------------------------------------------------
Attempted search from /cluster/home/jtl/mosesdecoder up to the root
at /cluster/home/jtl/mosesdecoder/share/boost-build
and in these directories from BOOST_BUILD_PATH and BOOST_ROOT: /usr/share/boost-build.
++++++++++

Boost-build.jam is in boost_1_55_0. The problem seems to be to locate boost_1_55_0

Kind regards, Jan Tore



------------------------------

_______________________________________________
Moses-support mailing list
Moses-support@mit.edu
http://mailman.mit.edu/mailman/listinfo/moses-support


End of Moses-support Digest, Vol 128, Issue 14
**********************************************

Tweet