Using SSH (Scripts, Plugins, etc) to start processes

Question

I'm trying to finish a remote deployment by restarting the two processes that make my Python App work. Like so

process-one &
process-two &

I've tried to "Execute a Shell Script" by doing this

ssh -i ~/.ssh/id_... user@xxx.xxx ./startup.sh

I've tried using the Jekins SSH Plugin and the Publish Over SSH Plugin and doing the same thing. All of the previous steps, stopping the processes, restarting other services, pulling in new code work fine. But when I get to the part where I start the services. It executes those two lines, and none of the Plugins or the Default Script execution can get off of the server. They all either hang until I restart Jekins or time out int he case of the Publish Over SSH plugin. So my build either requires a restart of Jenkins, or is marked unstable.

Has anyone had any success doing something similar? I've tried

nohup process-one &

But the same thing has happened. It's not that the services are messing up either, because they actually start properly, it's just that Jenkins doesn't seem to understand that.

Any help would be greatly appreciated. Thank you.

score 5 · Accepted Answer · edited May 23 '17 at 11:46

5

What probably happens in that the process when spawned (even with the &) is consuming the same input and output as your ssh connection. Jenkins is waiting for these pipes to be emptied before the job closes, thus waits for the processes to exit. You could verify that by killing your processes and you will see that the jenkins job terminates.

Dissociating outputs and starting the process remotely

There are multiple solutions to your problem:

(preferred) use proper daemon control tools. Your target platform probably has a standard way to manage those services, e.g. init.d scripts. Note, when writing init.d scripts, make sure you detach the process in the background AND ensure the input/output of the daemon are detached from the shell that starts them. There are several techniques, like like http://www.unix.com/man-page/Linux/8/start-stop-daemon/ tools, daemonize, daemontools or something like the shell script described under https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins+as+a+Unix+daemon (take note of the su -s bin/sh jenkins -c "YOUR COMMAND; ...disown" etc). I also list some python specific techniques under.
ssh server 'program < /dev/null > /dev/null 2>&1 &'

or

ssh server 'program < /dev/null >> logfile.log 2>&1 &' if you want to have a crude output management (no log rotation, etc...)
potentially using setsid (I haven't tried) https://superuser.com/questions/172043/how-do-i-fork-a-process-that-doesnt-die-when-shell-exits . In my quick tests I wasn't able to get it to work though...

Python daemons

The initial question was focused on SSH, so I didn't fully described how to run the python process as daemon. This is mostly covered in other techniques:

with start-stop-daemon: start-stop-daemon and python
with upstart on ubuntu: Run python script as daemon at boot time (Ubuntu)
some more python oriented approaches:
- How to make a Python script run like a service or daemon in Linux
- Can I run a Python script as a service?

edited May 23 '17 at 11:46

Community

1
1

answered Aug 14 '13 at 16:26

coffeebreaks

3,587
1
24
25

Alright I set up a couple of init.d/xxx scripts that work on my local machine. But again, they just hang over SSH in Jenkins. – Squab Aug 14 '13 at 21:36
What happens if you run using your init.d scripts using ssh from a remote server. E.g. if you need to use /etc/init.d/yourapp restart to restart your server locally, what does ssh yourserver '/etc/init.d/yourapp restart' do ? – coffeebreaks Aug 14 '13 at 22:55
Interesting....it restarts the services, but kind of hangs there. I can't input another command until i CTRL-C. Any way I can fix that? When I stop the services it works properly. Just not when starting/restarting. I mean all my init.d script does is start the app in the same way. – Squab Aug 15 '13 at 14:24
Here is my init.d script, for reference http://pastie.org/private/edu3hvuoebfjva80ll6fkg – Squab Aug 15 '13 at 14:34
Here's my latest solution. I have Publish over ssh execute the following command ./startup.sh backup.log 2>&1 & Where startup.sh just starts the services with service .. start This actually works. – Squab Aug 15 '13 at 20:02
@Squab your init.d script wouldn't work as it still has the same issue with input/output redirection. Your solution is same as what I had in #2 (apart from a lack of explicit output redirection to a file). I updated my answer for options #1 and #2. Note that I use >> to avoid losing previous run output. I still suggest to find a proper daemon wrapper to manage your remote apps and get solution #1 to work. – coffeebreaks Aug 15 '13 at 20:23
What I managed to do was start the services as daemons like this: start-stop-daemon --start --background --quiet --exec /usr/local/bin/name however when I stop them like this start-stop-daemon --stop --exec /usr/bin/python /usr/local/bin/name I have to put the /usr/bin/python in front of it to stop it, otherwise it doesn't stop it. But I'm afraid that's going to kill all python processes. Which happens because it kills the other service. – Squab Aug 16 '13 at 20:43
@Squab if you use the --pidfile /the/path/loop.pid option of the start-stop-daemon to store the process id of your running service, then you can kill that unique process in the stop portion of your init.d script. – coffeebreaks Aug 17 '13 at 04:27
1

I just wanted to thank you, I've managed to do everything I needed you were a huge help :) – Squab Aug 20 '13 at 18:42

Using SSH (Scripts, Plugins, etc) to start processes

1 Answers1