2

All of a sudden (by 03.03.2018 at around 3pm), our Grafana Docker instances stopped working in Swisscom Application Cloud (cloudfoundry). Have there been any changes recently on the application cloud? Any hints on where to look at?

  • index: 0
  • exit_status: undefined
  • exit_description: APP/PROC/WEB:
  • Exited with status 1
  • reason: CRASHED

App Log:

2018-04-04 13:08:43 [CELL/0] OUT Cell b8093908-296c-4050-b61b-763989f2e453 creating container for instance 53de901b-e6c2-40d7-6071-e673

2018-04-04 13:09:03 [CELL/0] OUT Cell b8093908-296c-4050-b61b-763989f2e453 successfully created container for instance 53de901b-e6c2-40d7-6071-e673

2018-04-04 13:09:03 [CELL/0] OUT Starting health monitoring of container

2018-04-04 13:09:04 [APP/PROC/WEB/0] OUT installing grafana-clock-panel @ 0.0.9

2018-04-04 13:09:04 [APP/PROC/WEB/0] OUT from url: https://grafana.com/api/plugins/grafana-clock-panel/versions/0.0.9/download

2018-04-04 13:09:04 [APP/PROC/WEB/0] OUT into: /var/lib/grafana/plugins

2018-04-04 13:09:06 [APP/PROC/WEB/0] OUT ✔ Installed grafana-clock-panel successfully

2018-04-04 13:09:06 [APP/PROC/WEB/0] OUT Restart grafana after installing plugins . <service grafana-server restart>

2018-04-04 13:09:06 [APP/PROC/WEB/0] OUT installing grafana-piechart-panel @ 1.2.0

2018-04-04 13:09:06 [APP/PROC/WEB/0] OUT from url: https://grafana.com/api/plugins/grafana-piechart-panel/versions/1.2.0/download

2018-04-04 13:09:06 [APP/PROC/WEB/0] OUT into: /var/lib/grafana/plugins

2018-04-04 13:09:08 [APP/PROC/WEB/0] OUT ✔ Installed grafana-piechart-panel successfully

2018-04-04 13:09:08 [APP/PROC/WEB/0] OUT Restart grafana after installing plugins . <service grafana-server restart>

2018-04-04 13:09:09 [APP/PROC/WEB/0] OUT installing vonage-status-panel @ 1.0.7

2018-04-04 13:09:09 [APP/PROC/WEB/0] OUT from url: https://grafana.com/api/plugins/vonage-status-panel/versions/1.0.7/download

2018-04-04 13:09:09 [APP/PROC/WEB/0] OUT into: /var/lib/grafana/plugins

2018-04-04 13:09:10 [APP/PROC/WEB/0] OUT ✔ Installed vonage-status-panel successfully

2018-04-04 13:09:10 [APP/PROC/WEB/0] OUT Restart grafana after installing plugins . <service grafana-server restart>

**2018-04-04 13:09:10 [APP/PROC/WEB/0] OUT t=2018-04-04T11:09:10+0000 lvl=crit msg="Failed to parse /etc/grafana/grafana.ini, open /etc/grafana/grafana.ini: no such file or directory%!(EXTRA []interface {}=[])"**

2018-04-04 13:09:10 [APP/PROC/WEB/0] OUT Exit status 1

2018-04-04 13:09:10 [CELL/SSHD/0] OUT Exit status 0

2018-04-04 13:09:11 [CELL/0] OUT Cell b8093908-296c-4050-b61b-763989f2e453 
stopping instance 53de901b-e6c2-40d7-6071-e673

2018-04-04 13:09:11 [CELL/0] OUT Cell b8093908-296c-4050-b61b-763989f2e453 destroying container for instance 53de901b-e6c2-40d7-6071-e673

2018-04-04 13:09:11 [API/3] OUT Process has crashed with type: "web"

**2018-04-04 13:09:11 [API/3] OUT App instance exited with guid 4a8f31f8-19a4-4f2c-8429-57ca5d2340f8 payload: {"instance"=>"53de901b-e6c2-40d7-6071-e673", "index"=>0, "reason"=>"CRASHED", "exit_description"=>"APP/PROC/WEB: Exited with status 1", "crash_count"=>9, "crash_timestamp"=>1522840150921536309, "version"=>"a25f82a6-7e9b-4e23-b4e6-3498eb58f072"}**

2018-04-04 13:09:11 [CELL/0] OUT Cell b8093908-296c-4050-b61b-763989f2e453 successfully destroyed container for instance 53de901b-e6c2-40d7-6071-e673
Ivanov
  • 2,271
  • 2
  • 19
  • 34

1 Answers1

5

At this time we updated cloudfoundry/diego to use garden-runc grootfs and to replace garden-sched.
The grafana/grafana docker image (latest/5.0.4) includes a volume directive which is handled incorrectly on garden-runc v1.12.1 (latest). The necessary grafana files are missing in the garden container. See garden-runc issue for more details.

Workaround

The volume directive can be removed from the Dockerfile. docker build and docker push to another repository.

Remove this line from Dockerfile

VOLUME ["/var/lib/grafana", "/var/log/grafana", "/etc/grafana"]
Community
  • 1
  • 1
lerel
  • 51
  • 3