How to copy a couchdb in dokku

Question

I want to take a snapshot of one database running in app www and put it into app staging. When I do that with clone or create/import none of the data is available.

How am I meant to do it?

matt@server:~$ dokku run www curl http://www:password@dokku-couchdb-www:5555/www

{"db_name":"www","doc_count":4966,"doc_del_count":232,"update_seq":46475,"purge_seq":0,"compact_running":false,"disk_size":3071180923,"data_size":334987077,"instance_start_time":"1500006610823893","disk_format_version":6,"committed_update_seq":46475}

So from that you can see there are 4966 documents.

matt@server:~$ dokku couchdb:clone www staging_www
-----> Starting container
       Waiting for container to be ready
=====> CouchDB container created: staging_www
       DSN: http://staging_www:password@dokku-couchdb-staging-www:5555/staging_www
-----> Copying data from www to staging_www
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                                                                                                Dload  Upload   Total   Spent    Left  Speed
100 1110M    0 1110M    0     0  30.4M      0 --:--:--  0:00:36 --:--:-- 31.9M
                                                                              -----> Done

So there are no errors in the clone. Then I run

dokku couchdb:link staging_www staging
dokku couchdb:promote staging_www staging

And there are no errors, but if I check the DB:

matt@server:~$ dokku run staging curl http://staging_www:password@dokku-couchdb-staging-www:5555/staging_www
{"db_name":"staging_www","doc_count":1,"doc_del_count":0,"update_seq":1,"purge_seq":0,"compact_running":false,"disk_size":4188,"data_size":342,"instance_start_time":"1509536857606369","disk_format_version":6,"committed_update_seq":1}

The doc count is 1 and I can't access any of the data in the staging app.

Equally I have tried

dokku couchdb:export www > www.couch
dokku couchdb:create staging_www
dokku couchdb:import staging_www < www.couch
dokku couchdb:link staging_www staging
dokku couchdb:promote staging_www staging

There are no errors, but again I end up with 1 doc in the database.

What am I meant to do?

Not able to set dokku on Vagrant VM Ubuntu 16.04 for reason. I am not able to deploy app to test your scenario. The git push just hangs — Tarun Lalwani, Nov 06 '17 at 22:12

svujic · Answer 1 · 2017-11-18T04:23:26.023

With dokku 0.9.4 and 'dokku couchdb service plugin' 1.0.0

Solution is quite simple. After the first attempt

root@dokku01:~# dokku couchdb:clone www staging_www

and fail to clone db. You need to destroy staging_www

root@dokku01:~# dokku couchdb:destroy staging_www

and do cloning again.

root@dokku01:~# dokku couchdb:clone www staging_www

Now it will works as expected. You can check new db with

root@dokku01:~# curl -X GET 'http://staging_www:password@dokku-couchdb-staging-www:5555/staging_www/_all_docs?include_docs=true&attachments=true'

Exporting from www and then importing a dump in newly created staging_www also works.

It's a bug in CouchDB plugin and it will be quite fun to find what is the root cause of it.

UPDATE

The root cause of this bug is bash script for backup and restore 'couchdb-backup'. In some cases a 'curl calls' inside the script fail to reach the db and consequently a restore operation doesn't work.

Cloning db is: create new instance, export (backup) old data and then import (restore) that backup in the new db.

Script below is updated 'clone' script from couchdb service plugin. It's made more resilient to 'couchdb-backup' import quirks.

#!/usr/bin/env bash
source "$(dirname "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)")/config"
set -eo pipefail; [[ $DOKKU_TRACE ]] && set -x
source "$PLUGIN_BASE_PATH/common/functions"
source "$(dirname "$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)")/functions"

service-clone-cmd() {
  #E you can clone an existing service to a new one
  #E dokku $PLUGIN_COMMAND_PREFIX:clone lolipop lolipop-2
  #A service, service to run command against
  #A new-service, name of new service
  declare desc="create container <new-name> then copy data from <name> into <new-name>"
  local cmd="$PLUGIN_COMMAND_PREFIX:clone" argv=("$@"); [[ ${argv[0]} == "$cmd" ]] && shift 1
  declare SERVICE="$1" NEW_SERVICE="$2" CLONE_FLAGS_LIST="${@:3}"

  [[ -z "$SERVICE" ]] && dokku_log_fail "Please specify a name for the service"
  [[ -z "$NEW_SERVICE" ]] && dokku_log_fail "Please specify a name for the new service"
  verify_service_name "$SERVICE"

  PLUGIN_IMAGE=$(service_version "$SERVICE" |  grep -o "^.*:" | sed -r "s/://g")
  PLUGIN_IMAGE_VERSION=$(service_version "$SERVICE" |  grep -o ":.*$" | sed -r "s/://g")

  service_create "$NEW_SERVICE" "${@:3}"
  dokku_log_info1 "Copying data from $SERVICE to $NEW_SERVICE"

attempts=5
attemptcount=0
R=2
succ_str=' Successfully.'
until [[ $R = 0 || $R = 1 ]]; do
  attemptcount=$((attemptcount+1))
  STDOUT1=$(service_export "$SERVICE" | service_import "$NEW_SERVICE" 2>&1) || true
  if [[ ! "$STDOUT1" = *"${succ_str}" ]]; then
    if [ $attemptcount = $attempts ]; then
      R=1
      echo -e "\nERROR: CouchDB Import failed - Stopping\n"
    else
      echo -e "\nWARN: CouchDB Import Reported an error - Attempt ${attemptcount}/${attempts} - Retrying...\n"
      sleep 1
    fi
  else
    R=0
  fi
done
dokku_log_info1 "Done"
exit $R
}

service-clone-cmd "$@"

Old version at /var/lib/dokku/plugins/available/couchdb/subcommands/clone should be replaced with this new one.

This did not work. Did you clone between dokku apps as in my example? — Matt Ellen, Nov 14 '17 at 18:13
I described the steps I take. I repeated these steps to be sure it works and it works for me every time. My tests are done Digital Ocean 512Mb node. Our environments are different and it seems is the cause of malfunction. — svujic, Nov 15 '17 at 17:22
If you clone a DB, it is linked to the original app. Did you link the DB to a different app? — Matt Ellen, Nov 15 '17 at 20:03
Linked to www app or standing alone it is the same. Cloning for second time is success. Maybe size of db is important, my www db is just 5 records long. — svujic, Nov 16 '17 at 11:57
For me it is not the same. In the original app I can access the data, but if I link it to a different app I cannot. — Matt Ellen, Nov 16 '17 at 12:04
@MattEllen you are right, there is problem. I'm now capable to provoke error if I run 'clone, destroy' in loop many times. Sometimes second attempt doesn't solve problem. I will dig in source and I'll keep you informed. — svujic, Nov 17 '17 at 17:41
Hi svujic. Thanks for the update. I will have a look when I get time. — Matt Ellen, Nov 20 '17 at 12:49

score -1 · Answer 2 · answered Nov 04 '17 at 13:07

-1

I usually just use the couchdb dokku plugin found here

Curl can have a tendency to timeout with multiple files, so I use the plugin to stage. Easy peasy

answered Nov 04 '17 at 13:07

jtillman

339
1
3

And how do you know that this is not the same plugin that @MattEllen is using? – Tarun Lalwani Nov 05 '17 at 08:58
Thanks, jtillman. That is the plugin I'm using. It's not a timeout issue, everything is completing correctly. Something else is going wrong. – Matt Ellen Nov 06 '17 at 00:24

How to copy a couchdb in dokku

2 Answers2