6

I have a user who I have added to the www-data group. I have a service setup to run an API that launches a Selenium instance. I am able to launch the code as my user, but the web app fails to launch. I originally had problems with Xvfb but after adding the path to the environment of the service file it was resolved. I am now having a similar issue with google-chrome not executing.

I've tried adding path to environment, and changing permissions on files and folders.

nginx conf

server {
        listen 443 ssl;

        root /usr/share/nginx/html;
        index index.html index.htm;

        server_name private;
        location / {
               proxy_pass http://127.0.0.1:8888;
               proxy_set_header Host $host;
               proxy_set_header X-Real-IP $remote_addr;
               proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        }

    ssl_certificate /etc/letsencrypt/live/private/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/private/privkey.pem; # managed by Certbot
}
server {
    if ($host = private) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        listen 80 default_server;
        listen [::]:80 default_server;

        server_name private;
    return 404; # managed by Certbot
}

[Unit]
Description=Gunicorn serving hrapi
After=network.target

[Service]
User=artem
Group=www-data
WorkingDirectory=/home/artem/ucs
Environment="PATH=/home/artem/env/bin:/usr/bin"
ExecStart=/home/artem/env/bin/gunicorn --workers 3 --bind 127.0.0.1:8888 wsgi:app

[Install]
WantedBy=multi-user.target

curl -H "Accept: application/json" -H "Content-type: application/json" -X POST -d '{"fname": "ijay", "lname": "private", "addr": "private", "city": "scottsdale", "state":"AZ", "postal":"private", "ssn":"private"}' https://hrapi.domain.com/getreport
Error:b'Traceback (most recent call last):
  File "/home/artem/ucs/get_report.py", line 158, in <module>
    b = UcsBot(sys.argv[1], sys.argv[2], sys.argv[3], sys.argv[4], sys.argv[5], sys.argv[6], sys.argv[7])
  File "/home/artem/ucs/get_report.py", line 40, in __init__
    self.driver = webdriver.Chrome(\'/home/artem/ucs/chromedriver\', chrome_options=options)
  File "/home/artem/env/lib/python3.6/site-packages/selenium/webdriver/chrome/webdriver.py", line 81, in __init__
    desired_capabilities=desired_capabilities)
  File "/home/artem/env/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 157, in __init__
    self.start_session(capabilities, browser_profile)
  File "/home/artem/env/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 252, in start_session
    response = self.execute(Command.NEW_SESSION, parameters)
  File "/home/artem/env/lib/python3.6/site-packages/selenium/webdriver/remote/webdriver.py", line 321, in execute
    self.error_handler.check_response(response)
  File "/home/artem/env/lib/python3.6/site-packages/selenium/webdriver/remote/errorhandler.py", line 242, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: unknown error: Chrome failed to start: exited abnormally
  (unknown error: DevToolsActivePort file doesn\'t exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)

Selenium is supposed to launch, and it does launch normally when I run the script. The script can't be launched from the API though.

Also, I have ensured that my chrome and chromedriver are correct versions. The script executes fine by itself from user as I said before just not when triggered via the API.

From the verbose log so it seems the problem is still permissions as suspected. j

[1563464973.354][INFO]: Launching chrome: /usr/bin/google-chrome --disable-background-networking --disable-client-side-phishing-detection --disable-default-apps --disable-dev-shm-usage --disable-hang-monitor --disable-popup-blocking --disable-prompt-on-repost --disable-sync --disable-web-resources --enable-automation --enable-blink-features=ShadowDOMV0 --enable-logging --force-fieldtrials=SiteIsolationExtensions/Control --headless --ignore-certificate-errors --load-extension=/tmp/.com.google.Chrome.rpiwaU/internal --log-level=0 --no-first-run --no-sandbox --password-store=basic --remote-debugging-port=0 --test-type=webdriver --use-mock-keychain --user-data-dir=/tmp/.com.google.Chrome.aQmopJ data:,
/usr/bin/google-chrome: line 8: readlink: command not found
/usr/bin/google-chrome: line 24: mkdir: command not found
/usr/bin/google-chrome: line 45: exec: cat: not found
/usr/bin/google-chrome: line 46: exec: cat: not found
[1563464973.406][INFO]: [60f2bc8b762ce68091b9abc2d8476bf5] RESPONSE InitSession ERROR unknown error: Chrome failed to start: exited abnormally
  (unknown error: DevToolsActivePort file doesn't exist)
  (The process started from chrome location /usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed.)
[1563464973.406][DEBUG]: Log type 'driver' lost 0 entries on destruction
[1563464973.406][DEBUG]: Log type 'browser' lost 0 entries on destruction

After this I did

sudo chown artem:www-data /usr/bin/google-chrome
sudo chown artem:www-data /usr/bin/chromedriver

Still same problem.

# wsgi.py

from api import app
if __name__=="__main__":
    app.run()
ThePyGuy
  • 907
  • 1
  • 3
  • 14
  • How are you launching XVFB? Also you want to run `driver = webdriver.Chrome(executable_path="D:\\chromedriver.exe", service_args=["--verbose", "--log-path=D:\\qc1.log"])`, to enable verbose logging to a file and see what you get in the log file for chrome – Tarun Lalwani Jul 18 '19 at 03:16
  • I am familiar with nginx, flask and gunicorn, not so much with Selenium and Xvfb. In order to help you with this expertise, maybe you could (1) elaborate a bit on the overall architecture and goal and (2) maybe attach some log-files from nginx, flask and gunicorn if you have any. (3) How does your nginx configuration look like? – B--rian Jul 18 '19 at 12:45
  • Xvfb is launched in the script with pyvirtualdisplay. There is no problem with Xvfb at least I don't think. The problem with Xvfb not being found was resolved by adding the /usr/bin path to my flask service file. – ThePyGuy Jul 18 '19 at 16:04
  • PS: I don't even need pyvirtualdisplay as headless mode actually works on this server. So the problem is just launching selenium for some reason it will not launch from the service. I thought perhaps running Selenium server could eliminate this problem.... – ThePyGuy Jul 18 '19 at 16:06

2 Answers2

3

The chrome startup script /usr/bin/google-chrome has a problem, as indicated by the /usr/bin/google-chrome: line 24: XXX: command not found:

[1563464973.354][INFO]: Launching chrome: /usr/bin/google-chrome [...]
/usr/bin/google-chrome: line 8: readlink: command not found
/usr/bin/google-chrome: line 24: mkdir: command not found
/usr/bin/google-chrome: line 45: exec: cat: not found
/usr/bin/google-chrome: line 46: exec: cat: not found

It might be a simple $PATH problem, meaning that the uWSGI server does not find all the mentioned commands readlink etc. because of it does not have these commands within the $PATH variable.

Warning: I would not allow www-data to access all the command XXX by changing their permissions, these would be a major security issue on a production server!

Instead, I suggest to look at the following SO question: How to fix "usr/bin/google-chrome is no longer running, so ChromeDriver is assuming that Chrome has crashed" error in Linux?

Alternative: Can you somehow solve your job without using Chrome? What are your reasons for starting an Chrome as a service on a webserver?

B--rian
  • 4,650
  • 8
  • 25
  • 63
  • It's part of automating an hiring process for a company. They have an onboarding process and this automates the process through a chain of webhooks they have setup which requires selenium. If it was doable without selenium I wouldnt be using it. Chrome is required for the website to function. I thought a workaround would be to launch Selenium server from my regular user and then access Chrome that way... – ThePyGuy Jul 23 '19 at 18:03
  • Also if you notice in my service file I am setting the path. This allowed my script to use Xvfb even though I found out it didn't need it. Still can't launch chrome. I agree that the highlighted lines are the problem but how do I fix them? Google-chrome is present in /usr/bin More over why would they be that way when my user owns the file. Ahh perhaps there are dependency files that are not owned or set appropriate perms. – ThePyGuy Jul 24 '19 at 09:44
  • The idea is to find out the **real** location of google-chrome (since `/usr/bin/chrome` is apparently only the startup wrapper script). Once you have done that, you have to ensure that specific directory (say `/opt/MyChromeInstanceForTheServer/` can be accessed by `www-data`. I suggest to use `/etc/sudoers` rather than permissions. – B--rian Jul 24 '19 at 10:32
  • Selenium is not my expertise, but if I understand https://stackoverflow.com/questions/22130109/ correctly, you do not need the full chrome to be running, only chrome-driver or something. Please tell me whether that helps you. – B--rian Jul 24 '19 at 10:36
  • I thank you for tryign to help me and you did know the right answer I knew that it was a path problem. I wish I could honestly award 50/50 to both of you. I thank you for your patience with me. I awarded him the answer because he gave the solution short and concise where as you had an idea but not the full answer yet. – ThePyGuy Jul 24 '19 at 23:20
3

Add :/bin to your PATH as shown below:

Environment="PATH=/home/artem/env/bin:/usr/bin:/bin"

mkdir, readlink etc are in /bin path which is not in the modified PATH

Shinto C V
  • 462
  • 1
  • 4
  • 13
  • This is the answer but as the other gentleman suggested its probably not good to let this service have free reign in these paths? So what would be the best way of isolating this behavior to only the files we need or is this okay by just letting it use the path /bin and /usr/bin ? – ThePyGuy Jul 25 '19 at 00:18