Headless script crashes after a few runs

Question

I have a script using a headless browser which I'm running using crontab -e. It runs fine the first few times and then crashes with the following Traceback:

Traceback (most recent call last):
  File "/home/clint-selenium-firefox.py", line 83, in <module>
    driver.get(url)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 248, in get
    self.execute(Command.GET, {'url': url})
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/webdriver.py", line 236, in execute
    self.error_handler.check_response(response)
  File "/usr/local/lib/python2.7/dist-packages/selenium/webdriver/remote/errorhandler.py", line 192, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: Failed to decode response from marionette

My crontab line is:

*/10 * * * * export DISPLAY=:0 && python /home/clint-selenium-firefox.py >> /home/error.log 2>&1

I don't want to overload this with the python script so I've pulled out what I think are the relevant bits.

from pyvirtualdisplay import Display

display = Display(visible=0, size=(800, 600))
display.start()
...
driver = webdriver.Firefox()
driver.get(url)
...
driver.quit()
...
display.stop()

Your help is much appreciated.

EDIT

Versions: Firefox 49.0.2; Selenium : 3.0.1; geckodriver: geckodriver-v0.11.1-linux64.tar.gz

Code around error (failing on driver.get(url)):

driver = webdriver.Firefox()
if DEBUG: print "Opened Firefox"

for u in urls:
    list_of_rows = []
    list_of_old_rows = []

    # get the old version of the site data
    mycsvfile = u[1]
    try:
        with open(mycsvfile, 'r') as csvfile:
            old_data = csv.reader(csvfile, delimiter=' ', quotechar='|')
            for o in old_data:
                list_of_old_rows.append(o)
    except: pass

    # get the new data
    url = u[0]
    if DEBUG: print url    

    driver.get(url)
    if DEBUG: print driver.title
    time.sleep(1)
    page_source = driver.page_source
    soup = bs4.BeautifulSoup(page_source,'html.parser')

I've been doing alot or reading about this. Do I need to use Marionette? — HenryM, Nov 16 '16 at 08:09
what are firefox, selenium and geckodriver versions that you are using? — Naveen Kumar R B, Nov 17 '16 at 14:53
I got the point but I need version names for confirmation :). Also, share the python code of the method which contains line 83 in clint-selenium-firefox.py — Naveen Kumar R B, Nov 17 '16 at 16:27
ok. try the answer here http://stackoverflow.com/a/40208762. — Naveen Kumar R B, Nov 17 '16 at 16:58
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/128375/discussion-between-naveen-and-henrym). — Naveen Kumar R B, Nov 17 '16 at 17:07
Firefox isn't headless. Are you using [Xvfb](http://tobyho.com/2015/01/09/headless-browser-testing-xvfb/)? — MikeJRamsey56, Nov 21 '16 at 23:23
Sorry, no, I'm using `pyvirtualdisplay` which I understand mimics it — HenryM, Nov 22 '16 at 16:47

MikeJRamsey56 · Accepted Answer · 2016-11-22T20:13:07.843

6

From Multiple Firefox instances failing with NS_ERROR_SOCKET_ADDRESS_IN_USE #99 This is because no --marionette-port option is passed to geckodriver - which means all instances of geckodriver launch firefox passing the same desired default port (2828). The first firefox instance binds to that port, future instances can't and all the geckodriver instances end up connecting to the first firefox instance - which produces all sorts of unpredictable behavior.

Followed by: I think a reasonable short-term solution is to do what the other drivers are doing and ask Marionette to bind to a randomised, free port generated by geckodriver. Currently it uses 2828 as the default for all instances it spawns of Firefox. Since Marionette unfortunately does not yet have an out-of-band way of communicating the port back to the client (geckodriver), this is inherently racy but we can improve the situation in the future with one of the proposals from bug 1240830.

This change was made in

Selenium 3.0.0.b2
* Updated Marionette port argument to match other drivers.

I guess random only works for so long. Raise an issue. A code fix may be required for the versions of selenium, firefox and geckodriver that you have. You could drop back to using Selenium 2.53.0 and firefox esr 38.8 until this is fixed. Your call.

UPDATE: Try

from selenium import webdriver
from selenium.webdriver.firefox.firefox_binary import FirefoxBinary

binary = FirefoxBinary('path/to/binary')
driver = webdriver.Firefox(firefox_binary=binary)

edited Nov 22 '16 at 20:13

answered Nov 21 '16 at 23:07

MikeJRamsey56

2,534
1
16
32

I'm trying to go back to earlier versions. Installing Selenium 2.53.0 was easy but I'm struggling to get firefox esr 38.8. I'm trying `sudo apt-get install iceweasel` and it is simply selecting and installing the latest version of firefox. Could you advice how to force the older version. – HenryM Nov 22 '16 at 16:40
I've managed to go back to the versions you've proposed but I now get `OSError: [Errno 13] Permission denied` – HenryM Nov 22 '16 at 16:59
On linux, I opened my google chrome browser and typed https://download.mozilla.org/?product=firefox-38.8.0esr-SSL&os=linux&lang=ach – MikeJRamsey56 Nov 22 '16 at 18:03
You get the error doing what? Please provide some context. – MikeJRamsey56 Nov 22 '16 at 18:04
You need to make sure that you completely uninstall the old firefox before installing the esr release. I run two different firefoxes myself but you have to be careful how you do it. – MikeJRamsey56 Nov 22 '16 at 18:31
I've installed firefox in a separate directory and I'm calling it using `driver=webdriver.Firefox(firefox_binary=path/to/binary)` and when I run my script I'm getting the OSError stated above – HenryM Nov 22 '16 at 18:40
sudo chown -R $USER /path/to/binary – MikeJRamsey56 Nov 22 '16 at 19:21
Change that to sudo chown 775 $USER /path/to/binary – MikeJRamsey56 Nov 22 '16 at 19:30
See the update I made above. Firefox wants to pull the path to the binary from your PATH. Worse case, change your PATH environment variable to point to the new binary's directory path. – MikeJRamsey56 Nov 22 '16 at 20:16
Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/128755/discussion-between-mikejramsey56-and-henrym). – MikeJRamsey56 Nov 22 '16 at 21:08
Got it, I was just directing to the directory rather than the binary. Sorted now and working. Thank you – HenryM Nov 23 '16 at 12:50

Headless script crashes after a few runs

1 Answers1

Linked