Python - pyparsing unicode characters

Question

:) I tried using w = Word(printables), but it isn't working. How should I give the spec for this. 'w' is meant to process Hindi characters (UTF-8)

The code specifies the grammar and parses accordingly.

671.assess  :: अहसास  ::2
x=number + "." + src + "::" + w + "::" + number + "." + number

If there is only english characters it is working so the code is correct for the ascii format but the code is not working for the unicode format.

I mean that the code works when we have something of the form 671.assess :: ahsaas ::2

i.e. it parses words in the english format, but I am not sure how to parse and then print characters in the unicode format. I need this for English Hindi word alignment for purpose.

The python code looks like this:

# -*- coding: utf-8 -*-
from pyparsing import Literal, Word, Optional, nums, alphas, ZeroOrMore, printables , Group , alphas8bit , 
# grammar 
src = Word(printables)
trans =  Word(printables)
number = Word(nums)
x=number + "." + src + "::" + trans + "::" + number + "." + number
#parsing for eng-dict
efiledata = open('b1aop_or_not_word.txt').read()
eresults = x.parseString(efiledata)
edict1 = {}
edict2 = {}
counter=0
xx=list()
for result in eresults:
  trans=""#translation string
  ew=""#english word
  xx=result[0]
  ew=xx[2]
  trans=xx[4]   
  edict1 = { ew:trans }
  edict2.update(edict1)
print len(edict2) #no of entries in the english dictionary
print "edict2 has been created"
print "english dictionary" , edict2 

#parsing for hin-dict
hfiledata = open('b1aop_or_not_word.txt').read()
hresults = x.scanString(hfiledata)
hdict1 = {}
hdict2 = {}
counter=0
for result in hresults:
  trans=""#translation string
  hw=""#hin word
  xx=result[0]  
  hw=xx[2]
  trans=xx[4]
  #print trans
  hdict1 = { trans:hw }
  hdict2.update(hdict1)

print len(hdict2) #no of entries in the hindi dictionary
print"hdict2 has been created"
print "hindi dictionary" , hdict2
'''
#######################################################################################################################

def translate(d, ow, hinlist):
   if ow in d.keys():#ow=old word d=dict
    print ow , "exists in the dictionary keys"
        transes = d[ow]
    transes = transes.split()
        print "possible transes for" , ow , " = ", transes
        for word in transes:
            if word in hinlist:
        print "trans for" , ow , " = ", word
                return word
        return None
   else:
        print ow , "absent"
        return None

f = open('bidir','w')
#lines = ["'\
#5# 10 # and better performance in business in turn benefits consumers .  # 0 0 0 0 0 0 0 0 0 0 \
#5# 11 # vHyaapaar mEmn bEhtr kaam upbhOkHtaaomn kE lIe laabhpHrdd hOtaa hAI .  # 0 0 0 0 0 0 0 0 0 0 0 \
#'"]
data=open('bi_full_2','rb').read()
lines = data.split('!@#$%')
loc=0
for line in lines:
    eng, hin = [subline.split(' # ')
                for subline in line.strip('\n').split('\n')]

    for transdict, source, dest in [(edict2, eng, hin),
                                    (hdict2, hin, eng)]:
        sourcethings = source[2].split()
        for word in source[1].split():
            tl = dest[1].split()
            otherword = translate(transdict, word, tl)
            loc = source[1].split().index(word)
            if otherword is not None:
                otherword = otherword.strip()
                print word, ' <-> ', otherword, 'meaning=good'
                if otherword in dest[1].split():
                    print word, ' <-> ', otherword, 'trans=good'
                    sourcethings[loc] = str(
                        dest[1].split().index(otherword) + 1)

        source[2] = ' '.join(sourcethings)

    eng = ' # '.join(eng)
    hin = ' # '.join(hin)
    f.write(eng+'\n'+hin+'\n\n\n')
f.close()
'''

if an example input sentence for the source file is:

1# 5 # modern markets : confident consumers  # 0 0 0 0 0 
1# 6 # AddhUnIk baajaar : AshHvsHt upbhOkHtaa .  # 0 0 0 0 0 0 
!@#$%

the ouptut would look like this :-

1# 5 # modern markets : confident consumers  # 1 2 3 4 5 
1# 6 # AddhUnIk baajaar : AshHvsHt upbhOkHtaa .  # 1 2 3 4 5 0 
!@#$%

Output Explanation:- This achieves bidirectional alignment. It means the first word of english 'modern' maps to the first word of hindi 'AddhUnIk' and vice versa. Here even characters are take as words as they also are an integral part of bidirectional mapping. Thus if you observe the hindi WORD '.' has a null alignment and it maps to nothing with respect to the English sentence as it doesn't have a full stop. The 3rd line int the output basically represents a delimiter when we are working for a number of sentences for which your trying to achieve bidirectional mapping.

What modification should i make for it to work if the I have the hindi sentences in Unicode(UTF-8) format.

Please edit this question and make use of proper formatting so that the question is readable. — Ignacio Vazquez-Abrams, Feb 26 '10 at 05:12

PaulMcG · Answer 1 · 2018-11-17T02:27:46.287

27

Pyparsing's printables only deals with strings in the ASCII range of characters. You want printables in the full Unicode range, like this:

unicodePrintables = u''.join(unichr(c) for c in xrange(sys.maxunicode) 
                                        if not unichr(c).isspace())

Now you can define trans using this more complete set of non-space characters:

trans = Word(unicodePrintables)

I was unable to test against your Hindi test string, but I think this will do the trick.

(If you are using Python 3, then there is no separate unichr function, and no xrange generator, just use:

unicodePrintables = ''.join(chr(c) for c in range(sys.maxunicode) 
                                        if not chr(c).isspace())

EDIT:

With the recent release of pyparsing 2.3.0, new namespace classes have been defined to give printables, alphas, nums, and alphanums for various Unicode language ranges.

import pyparsing as pp
pp.Word(pp.pyparsing_unicode.printables)
pp.Word(pp.pyparsing_unicode.Devanagari.printables)
pp.Word(pp.pyparsing_unicode.देवनागरी.printables)

edited Nov 17 '18 at 02:27

answered Feb 26 '10 at 09:43

PaulMcG

56,194
14
81
122

Thank you for your answer sir..:) – boddhisattva Mar 11 '10 at 05:25
this answer is long since obsolete: unicode no longer is 16 bit, and looping everything isn’t performant at all. – flying sheep Sep 15 '15 at 12:33
2

@flyingsheep - good tip, updated to use `sys.maxunicode` instead of a hard-coded constant, so it will track with Python's `sys` module. As for looping everything, this bit only runs once, when initially defining a parser, and when used to create a pyparsing `Word`, gets stored as a set(), so parse-time performance is still quite good. – PaulMcG Jul 21 '16 at 15:20
This would include a lot of unprintable characters, such as `u'\0'`. – user2357112 supports Monica Nov 16 '18 at 19:39
The latest version of pyparsing includes a number of Unicode ranges. All control characters below `u'\20'` are not included. – PaulMcG Nov 16 '18 at 22:42
Could you update your answer to show the new options? – user2357112 supports Monica Nov 17 '18 at 00:05
@PaulMcG Wouldn't the `Regex` class be better suited for constructing Unicode ranges? Your first Word constructor (in EDIT) takes 4s (in user-space) on my laptop (Intel Kaby Lake i5), while `quotedString` as it's defined in `core.py` matches almost all Unicode characters and is created almost instantly. – woky May 15 '20 at 21:43
A fair point. The 4s delay is probably due to the lazy init of the complete set of unicode printables - would be interesting to separate the timing by assigning `pp.pyparsing_unicode.printables` to a string (which would do the time-consuming lazy init) and then build a Word with it. A regex using `Regex(r"\S+")` would not have to incur this penalty. The language subset ranges would be more effort, since you can't just use `\S`. New `Word` regex code in 3.0.0 generates more performant regexen. – PaulMcG May 16 '20 at 03:49

score 7 · Accepted Answer · answered Feb 26 '10 at 06:08

7

As a general rule, do not process encoded bytestrings: make them into proper unicode strings (by calling their .decode method) as soon as possible, do all of your processing always on unicode strings, then, if you have to for I/O purposes, .encode them back into whatever bytestring encoding you require.

If you're talking about literals, as it seems you are in your code, the "as soon as possible" is at once: use u'...' to express your literals. In a more general case, where you're forced to do I/O in encoded form, it's immediately after input (just as it's immediately before output if you need to perform output in a specific encoded form).

answered Feb 26 '10 at 06:08

Alex Martelli

762,786
156
1,160
1,345

Hello Sir..:) Thank you for your answer.. whatever you have said in the 2nd para is exactly applicable to my case.. I tried this thing in the following line of the code: trans = u'Word(printables)' and I couldn achieve the desired output. Could you please correct me if I have made the modification in the wrong line, as after making this change the error is coming ' Expecting printables at that position ' with respect to the lines which defines the grammmar. – boddhisattva Feb 26 '10 at 09:06
@mgj, don't assign a unicode string literal to `trans`, that makes no sense. Just ensure `printables` is a unicode object (**not** a utf8-encoded byte string! -- nor a byte string with any other encoding!), and use `trans = Word(printables)`. If your _file_ is utf-8 encoded, or encoded with any other encoding, decode it by using `codecs.open` from the `codecs` module, _not_ the built-in `open` as you're doing, so that each `line` is a unicode object, not a byte string (in whatever encoding). – Alex Martelli Feb 26 '10 at 15:11

score 1 · Answer 3 · answered Nov 03 '19 at 22:45

I Was searching about french unicode chars and fall on this question. If you search french or other latin accents, with pyparsing 2.3.0 you can use:

>>> pp.pyparsing_unicode.Latin1.alphas
'ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyzªµºÀÁÂÃÄÅÆÇÈÉÊËÌÍÎÏÐÑÒÓÔÕÖØÙÚÛÜÝÞßàáâãäåæçèéêëìíîïðñòóôõöøùúûüýþÿ'

Python - pyparsing unicode characters

3 Answers3

Linked