cannot deploy docker image form AWS private registry

Question

I am trying to push an app from a docker image hosted in the AWS Elastic Container Registry and am getting 500 error codes from the cloudfoundry API when trying to push. Am i doing something wrong or is there just an issue with the API currently? Any help is appreciated.

push command used (replaced real route, app and image name):

cf push dockerized-app --docker-image 300401118676.dkr.ecr.eu-central-1.amazonaws.com/my/image:latest --docker-username AWS --hostname my-dockerized-app -i 1 -m 1024M -k 1024M

cf-cli version: cf version 6.34.1+bbdf81482.2018-01-17

This ist the standard log output i get:

Using docker repository password from environment variable CF_DOCKER_PASSWORD.
Pushing app dockerized-app to org ORG / space SPACE as someone@somewhere.ch...
Getting app info...
Creating app with these attributes...
+ name:              dockerized-app
+ docker image:      300401118676.dkr.ecr.eu-central-1.amazonaws.com/my/image:latest
+ docker username:   AWS
+ disk quota:        1G
+ instances:         1
+ memory:            1G
  routes:
+   my-dockerized-app.scapp.io

Creating app dockerized-app...
Unexpected Response
Response code: 500
CC code:       0
CC error code: 
Request ID:    f0789965-19b1-4178-5cce-e42ff671a99b::6eb55c40-70de-4011-ad30-ee60aab54d82
Description:   {
  "error_code": "UnknownError",
  "description": "An unknown error occurred.",
  "code": 10001
}

FAILED

Here is the relevant log output with the -v flag set

Creating app with these attributes...
+ name:              dockerized-app
+ docker image:      300401118676.dkr.ecr.eu-central-1.amazonaws.com/my/image:latest
+ docker username:   AWS
+ disk quota:        1G
+ instances:         1
+ memory:            1G
  routes:
+   my-dockerized-app.scapp.io

Creating app dockerized-app...
REQUEST: [2018-02-27T18:39:28+01:00]
POST /v2/apps HTTP/1.1
Host: api.lyra-836.appcloud.swisscom.com
Accept: application/json
Authorization: [PRIVATE DATA HIDDEN]
Content-Type: application/json
User-Agent: cf/6.34.1+bbdf81482.2018-01-17 (go1.9.2; amd64 darwin)
{
  "disk_quota": 1024,
  "docker_credentials": {
    "password": "[PRIVATE DATA HIDDEN]",
    "username": "AWS"
  },
  "docker_image": "300401118676.dkr.ecr.eu-central-1.amazonaws.com/my/image:latest",
  "instances": 1,
  "memory": 1024,
  "name": "dockerized-app",
  "space_guid": "07cead83-7db5-477e-83ca-f7bbee10e557"
}


RESPONSE: [2018-02-27T18:39:28+01:00]
HTTP/1.1 500 Internal Server Error
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Content-Length: 99
Content-Type: application/json;charset=utf-8
Date: Tue, 27 Feb 2018 17:39:28 GMT
Expires: 0
Pragma: no-cache
Server: nginx
Strict-Transport-Security: max-age=16070400; includeSubDomains
X-Content-Type-Options: nosniff
X-Frame-Options: DENY
X-Vcap-Request-Id: 6c6acb3a-4ead-4f88-5d2c-e7d7f846b2af::0e919224-e372-46f1-8d70-19bf30f85145
X-Xss-Protection: 1; mode=block
{
  "code": 10001,
  "description": "An unknown error occurred.",
  "error_code": "UnknownError"
}


Unexpected Response
Response code: 500
CC code:       0
CC error code: 
Request ID:    6c6acb3a-4ead-4f88-5d2c-e7d7f846b2af::0e919224-e372-46f1-8d70-19bf30f85145
Description:   {
  "error_code": "UnknownError",
  "description": "An unknown error occurred.",
  "code": 10001
}

Seems to me like the docker registry username and password get picked up just fine (and yes they work).

A 500 error like that means something went wrong on the Cloud Controller. You would need to talk with your operator and have him or her check the Cloud Controller logs for more details. — Daniel Mikusa, Feb 28 '18 at 12:57
@DanielMikusa thanks for your reply! I was kind of expecting something of that sort. Since StackOverflow is the preferred support channel for the Swisscom AppCloud i posted my issue here. Hope someone from Swisscom gets around to looking at this. — Patrick Suter, Feb 28 '18 at 15:56

score 3 · Accepted Answer · answered May 15 '18 at 12:28

From an operator perspective, it looks like you're hitting CloudFoundry's password limit of 1000 characters by using the Amazon Elastic Container Registry signed tokens (which are around 2000 chars):

/var/vcap/sys/log/cloud_controller_ng/cloud_controller_ng.log.5.gz: 
{"timestamp":1526311559.8367982,"message":"Request failed: 500: 
{\"error_code\"=>\"UnknownError\", \"description\"=>\"An unknown 
error occurred.\", \"code\"=>10001, \"test_mode_info\"=> 
{\"description\"=>\"docker_password can be up to 1,000 characters\", 
...

We filed the issue with the CC team: https://github.com/cloudfoundry/cloud_controller_ng/issues/1141

score 1 · Answer 2 · answered Mar 19 '18 at 17:53

I'm not sure what version of Cloud Foundry your provider is running right now, but support for private docker registries (i.e. registries using HTTPS & basic auth) requires a fairly recent version of Cloud Foundry.

It definitely works in API versions 2.103 and later, as that's what we're running at Meshcloud right now and we have customer successfully using private registries ;-)

$ cf api
api endpoint:   https://api.cf.eu-de-netde.msh.host
api version:    2.103.0

Disclaimer: I'm a co-founder at Meshcloud.

thx for your reply, the current version is 2.106.0 and the issues is still present unfortunately — Patrick Suter, May 14 '18 at 11:25

cannot deploy docker image form AWS private registry

2 Answers2