Iteratively parsing HTML (with lxml?)

Question

I'm currently trying to iteratively parse a very large HTML document (I know.. yuck) to reduce the amount of memory used. The problem I'm having is that I'm getting XML syntax errors such as:

lxml.etree.XMLSyntaxError: Attribute name redefined, line 134, column 59

This then causes everything to stop.

Is there a way to iteratively parse HTML without choking on syntax errors?

At the moment I'm extracting the line number from the XML syntax error exception, removing that line from the document, and then restarting the process. Seems like a pretty disgusting solution. Is there a better way?

Edit:

This is what I'm currently doing:

context = etree.iterparse(tfile, events=('start', 'end'), html=True)
in_table = False
header_row = True
while context:
    try:
        event, el = context.next()

        # do something

        # remove old elements
        while el.getprevious() is not None:
            del el.getparent()[0]

    except etree.XMLSyntaxError, e:
        print e.msg
        lineno = int(re.search(r'line (\d+),', e.msg).group(1))
        remove_line(tfilename, lineno)
        tfile = open(tfilename)
        context = etree.iterparse(tfile, events=('start', 'end'), html=True)
    except KeyError:
        print 'oops keyerror'

score 8 · Accepted Answer · answered Dec 13 '11 at 04:18

The perfect solution ended up being Python's very own HTMLParser ^[docs].

This is the (pretty bad) code I ended up using:

class MyParser(HTMLParser):
    def __init__(self):
        self.finished = False
        self.in_table = False
        self.in_row = False
        self.in_cell = False
        self.current_row = []
        self.current_cell = ''
        HTMLParser.__init__(self)

    def handle_starttag(self, tag, attrs):
        attrs = dict(attrs)
        if not self.in_table:
            if tag == 'table':
                if ('id' in attrs) and (attrs['id'] == 'dgResult'):
                    self.in_table = True
        else:
            if tag == 'tr':
                self.in_row = True
            elif tag == 'td':
                self.in_cell = True
            elif (tag == 'a') and (len(self.current_row) == 7):
                url = attrs['href']
                self.current_cell = url


    def handle_endtag(self, tag):
        if tag == 'tr':
            if self.in_table:
                if self.in_row:
                    self.in_row = False
                    print self.current_row
                    self.current_row = []
        elif tag == 'td':
            if self.in_table:
                if self.in_cell:
                    self.in_cell = False
                    self.current_row.append(self.current_cell.strip())
                    self.current_cell = ''

        elif (tag == 'table') and self.in_table:
            self.finished = True

    def handle_data(self, data):
        if not len(self.current_row) == 7:
            if self.in_cell:
                self.current_cell += data

With that code I could then do this:

parser = MyParser()
for line in myfile:
    parser.feed(line)

score 5 · Answer 2 · answered Aug 17 '15 at 11:52

5

At the moment lxml etree.iterparse supports keyword argument recover=True, so that instead of writing custom subclass of HTMLParser fixing broken html you can just pass this argument to iterparse.

To properly parse huge and broken html you only need to do following:

etree.iterparse(tfile, events=('start', 'end'), html=True, recover=True)

answered Aug 17 '15 at 11:52

Pawel Miech

7,002
3
31
52

This is the best answer for me. – Zhang Yiwei Feb 17 '17 at 03:41

score 1 · Answer 3 · answered Dec 12 '11 at 17:09

1

Use True for iterparse's arguments html and huge_tree.

answered Dec 12 '11 at 17:09

Kabie

9,620
1
33
44

I am currently using `html=True`, it still raises XML syntax errors. I'll take a look at the `huge_tree` parameter. – Acorn Dec 12 '11 at 17:15
`huge_tree` doesn't seem relevant: "huge_tree: disable security restrictions and support very deep trees". My tree isn't deep, just long. – Acorn Dec 12 '11 at 17:22

score -1 · Answer 4 · answered Dec 12 '11 at 16:51

-1

Try parsing your HTML document with lxml.html:

Since version 2.0, lxml comes with a dedicated Python package for dealing with HTML: lxml.html. It is based on lxml's HTML parser, but provides a special Element API for HTML elements, as well as a number of utilities for common HTML processing tasks.

answered Dec 12 '11 at 16:51

Raymond Yee

529
5
12

I'm trying to iteratively parse the document due to its large size. lxml.html does not have an iterparse function as far as I can tell. – Acorn Dec 12 '11 at 16:53
I suggested lxml.html because in the OP there was no mention of trying lxml.html. I think that down-voting my answer is rather misguided. – Raymond Yee Dec 12 '11 at 17:30

Iteratively parsing HTML (with lxml?)

4 Answers4

Linked