I am currently working on a Firefox add-on, that is supposed to interact with a local native python program. So there is a JS file called Background.js that sends a json message to the native app.
The native python app is supposed to recieve the message and store it in a local file. So far so good, the problems arise, when I try to save the incoming json packages in a continuing csv file.
the JS file sends Messages like this:
{"messageId": countGlobalId(), "url": e.target.href,
"innerHTML": e.target.innerHTML,
"outerHTML": e.target.outerHTML,
"tagName": e.target.tagName});
}
The basic functionality is given. In the Python file the following function recieves the message and returns a jsonobject?!:
def getMessage():
rawLength = sys.stdin.buffer.read(4)
if len(rawLength) == 0:
sys.exit(0)
messageLength = struct.unpack('@I', rawLength)[0]
message = sys.stdin.buffer.read(messageLength)#.decode('utf-8')
jsonMessage= json.loads(message)
return jsonMessage
Here comes the problem; the following loop is supposed to save the incoming data to a specified csv file:
while True:
receivedMessage = getMessage()
if receivedMessage:
messageString=json.dumps(receivedMessage)
with open("ts.csv",'wb') as csvfile:
writer= csv.DictWriter(csvfile,
fieldnames=('messageId', 'url', 'innerHTML',
'outerHTML', 'tagName'), extrasection='ignore')
#writer.writerows(receivedMessage)
writer.writerows(messageString)
# just trying alot of combinations above
sendmessage...
I know that my problem lies in the last while loop, but can't find out where. I tried a simpler version, just to test the file writing mechanics, and it worked just fine:
while True:
receivedMessage = getMessage()
if receivedMessage:
file=open("ts.csv,'a')
file.write(json.dumps(receivedMessage))
file.close()
sendmessage...
Apart from that, my only way of debugging this project is running snippets in seperate pyrthon testfiles, since the native app is called by the add-on and no console.
So, how the bloody hell could I get my json data to fit neatly into a csv dict?
It is the goal to do something like this:
import csv
toCSV = [{'name':'bob','age':25,'weight':200},
{'name':'jim','age':31,'weight':180}]
keys = toCSV[0].keys()
with open('people.csv', 'wb') as output_file:
dict_writer = csv.DictWriter(output_file, keys)
dict_writer.writeheader()
dict_writer.writerows(toCSV)
How do I convert this list of dictionaries to a csv file?
This works fine in a closed environment.
But when i adjust my code to:
while True:
receivedMessage = getMessage()
if receivedMessage:
keys = receivedMessage[0].keys()
with open('testfile.csv', 'wb') as output_file:
dict_writer = csv.DictWriter(output_file, keys)
dict_writer.writeheader()
dict_writer.writerows(receivedMessage)
the script stops.
EDIT:::::::::::: The main issue was that my originial message sometimes contained empty entries. to solve that, i added a few lines to the creation JS:
if (e.target.href)
{urlVar= e.target.href;}
else {urlVar= "empty";}
if (e.target.innerHTML)
{innerHTMLVar= e.target.innerHTML;}
else {innerHTMLVar= "empty";}
if (e.target.outerHTML)
{outerHTMLVar= e.target.outerHTML;}
else {outerHTMLVar= "empty";}
if (e.target.tagName)
{tagNameVar= e.target.tagName;}
else {tagNameVar= "empty";}
That seemed to do the trick and now the python native app can use the data as follows:
....
jsonMessage= json.loads(message)
...
with open('testfile.csv', 'a', newline='') as csvfile:
fieldnames = ['messageId', 'url', 'innerHTML', 'outerHTML', 'tagName']
writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
writer.writerow(jsonMessage)