2

I am currently working on a Firefox add-on, that is supposed to interact with a local native python program. So there is a JS file called Background.js that sends a json message to the native app.

The native python app is supposed to recieve the message and store it in a local file. So far so good, the problems arise, when I try to save the incoming json packages in a continuing csv file.

the JS file sends Messages like this:

      {"messageId": countGlobalId(), "url": e.target.href,
      "innerHTML": e.target.innerHTML,
      "outerHTML": e.target.outerHTML,
      "tagName": e.target.tagName});
  }

The basic functionality is given. In the Python file the following function recieves the message and returns a jsonobject?!:

def getMessage():
    rawLength = sys.stdin.buffer.read(4)
    if len(rawLength) == 0:
        sys.exit(0)
    messageLength = struct.unpack('@I', rawLength)[0]
    message = sys.stdin.buffer.read(messageLength)#.decode('utf-8')
    jsonMessage= json.loads(message)
    return jsonMessage

Here comes the problem; the following loop is supposed to save the incoming data to a specified csv file:

while True:
    receivedMessage = getMessage()
    if receivedMessage:
        messageString=json.dumps(receivedMessage)
        with open("ts.csv",'wb') as csvfile:
            writer= csv.DictWriter(csvfile,
                fieldnames=('messageId', 'url', 'innerHTML',
                'outerHTML', 'tagName'), extrasection='ignore') 
            #writer.writerows(receivedMessage)
            writer.writerows(messageString)
            # just trying alot of combinations above

sendmessage...

I know that my problem lies in the last while loop, but can't find out where. I tried a simpler version, just to test the file writing mechanics, and it worked just fine:

    while True:
        receivedMessage = getMessage()
        if receivedMessage:
            file=open("ts.csv,'a')
            file.write(json.dumps(receivedMessage))
            file.close()
sendmessage...

Apart from that, my only way of debugging this project is running snippets in seperate pyrthon testfiles, since the native app is called by the add-on and no console.

So, how the bloody hell could I get my json data to fit neatly into a csv dict?

It is the goal to do something like this:

import csv
toCSV = [{'name':'bob','age':25,'weight':200},
         {'name':'jim','age':31,'weight':180}]
keys = toCSV[0].keys()
with open('people.csv', 'wb') as output_file:
    dict_writer = csv.DictWriter(output_file, keys)
    dict_writer.writeheader()
    dict_writer.writerows(toCSV)

How do I convert this list of dictionaries to a csv file?

This works fine in a closed environment.

But when i adjust my code to:

while True:
    receivedMessage = getMessage()
    if receivedMessage:
        keys = receivedMessage[0].keys()
        with open('testfile.csv', 'wb') as output_file:
            dict_writer = csv.DictWriter(output_file, keys)
            dict_writer.writeheader()
            dict_writer.writerows(receivedMessage)

the script stops.

EDIT:::::::::::: The main issue was that my originial message sometimes contained empty entries. to solve that, i added a few lines to the creation JS:

if (e.target.href)
  {urlVar= e.target.href;}
  else {urlVar= "empty";}
if (e.target.innerHTML)
  {innerHTMLVar= e.target.innerHTML;}
  else {innerHTMLVar= "empty";}
if (e.target.outerHTML)
  {outerHTMLVar= e.target.outerHTML;}
  else {outerHTMLVar= "empty";}
if (e.target.tagName)
  {tagNameVar= e.target.tagName;}
  else {tagNameVar= "empty";}

That seemed to do the trick and now the python native app can use the data as follows:

....
jsonMessage= json.loads(message)
...
    with open('testfile.csv', 'a', newline='') as csvfile:
        fieldnames = ['messageId', 'url', 'innerHTML', 'outerHTML', 'tagName']
        writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
        writer.writerow(jsonMessage)
jmunsch
  • 16,405
  • 6
  • 74
  • 87

0 Answers0