12

I have a specific use case in which we want to ask Keycloak for all the users and the groups and roles for each user, on a daily basis. For reconciliation purposes with other internal systems.

Currently we are using the provided Keycloak endpoints in the UsersResource for this. But we see that performance slows down after each call to a point we can't use this solution anymore. There are more then 30K users in the realm.

We've also seen that Keycloak can export the database, but only on system boot (I guess for migration purposes). Given that we want to extract all the users on a daily basis we cannot use this.

Are there some known functionalities or workarounds?

Michel
  • 8,310
  • 13
  • 41
  • 56
  • Are you using the KC provided database or some other external DB? – Xtreme Biker Feb 23 '18 at 07:51
  • external postgresql . So querying directly is possible but our last option ;-) – Michel Feb 23 '18 at 08:50
  • If it is for internal use I guess you could use some direct query... Still, for the performance problem, have you considered a more powerful machine? Or even upgrading the KC version? Are you using some kind of pagination to list them? – Xtreme Biker Feb 23 '18 at 09:31
  • Yes we do. 25 user takes 2 seconds and 50 takes 4 seconds. Upgrading the server may help but I guess the API isn't made for this kind of request. Direct queries seems to be the solution for now – Michel Feb 24 '18 at 16:58
  • @Michel did you find a solution to this? When I export realm, users are not exported. – tryingToLearn Oct 04 '18 at 09:01
  • Direct queries on your source DB seems to be the solution for now – Michel Oct 04 '18 at 11:47
  • I am trying to export users from keycloak database. I am not using a separate source db. – tryingToLearn Oct 05 '18 at 09:10

2 Answers2

9

I have done it with an parallel starting container via docker, which connects to the existing keycloak db.

Please use the same Version of the container keycloak as the real keacloak has. Because of db schema differences between versions.

EXPORT

docker run --rm\
    --name keycloak_exporter\
    -v /tmp:/tmp/keycloak-export:Z\
    -e POSTGRES_DATABASE=keycloak\
    -e POSTGRES_PASSWORD=PASSOWRD_PLEASE\
    -e POSTGRES_USER=keycloak\
    -e DB_VENDOR=POSTGRES\
    -e POSTGRES_PORT_5432_TCP_ADDR=postgresql.local\
    jboss/keycloak:3.4.3.Final\
    -Dkeycloak.migration.action=export\
    -Dkeycloak.migration.provider=dir\
    -Dkeycloak.migration.dir=/tmp/keycloak-export\
    -Dkeycloak.migration.usersExportStrategy=SAME_FILE\
    -Dkeycloak.migration.realmName=therealm

IMPORT

docker run --rm\
    --name keycloak_importer\
    -v /tmp:/tmp/keycloak-import:Z\
    -e POSTGRES_DATABASE=keycloak_dest\
    -e POSTGRES_PASSWORD=PASSOWRD_DEST_PLEASE\
    -e POSTGRES_USER=keycloak\
    -e DB_VENDOR=POSTGRES\
    -e POSTGRES_PORT_5432_TCP_ADDR=postgresql2.local\
    jboss/keycloak:3.4.3.Final\
    -Dkeycloak.migration.action=import\
    -Dkeycloak.migration.provider=dir\
    -Dkeycloak.migration.dir=/tmp/keycloak-import\
    -Dkeycloak.migration.strategy=IGNORE_EXISTING\
    -Dkeycloak.migration.usersExportStrategy=SAME_FILE\
    -Dkeycloak.migration.realmName=therealm

Possible config options: https://github.com/keycloak/keycloak-documentation/blob/master/server_admin/topics/export-import.adoc

Gernot Grames
  • 91
  • 1
  • 3
3

you need in your docker-compose-yml to bind your folder, not just the realm json file, like this:

keycloak:
    image: jboss/keycloak:8.0.1
    container_name: "keycloak"
     volumes:
      - ./realms/:/tmp/
    environment:
      - KEYCLOAK_USER=admin
      - KEYCLOAK_PASSWORD=admin
      - KEYCLOAK_IMPORT=/tmp/realm-export.json -Dkeycloak.profile.feature.upload_scripts=enabled

where realms is your folder beside the yaml file. At this point you can run docker-compose up -d with your basic realm-export.json as always, go in your admin panel, adding users with credentials and roles, and then with this command you will able to export the entire configuration:

docker exec -it keycloak /opt/jboss/keycloak/bin/standalone.sh -Djboss.socket.binding.port-offset=100 -Dkeycloak.migration.action=export -Dkeycloak.migration.provider=singleFile -Dkeycloak.migration.realmName=ed-realm -Dkeycloak.migration.usersExportStrategy=REALM_FILE -Dkeycloak.migration.file=/tmp/export.json

You will see in your realms folder that a new file will be created, and it will contain the entire configuration, so you can run docker-compose down, replace your old file with this new and run again docker-compose up as many time you want, and redoing the process when you will change your realm again.

Nather Webber
  • 493
  • 5
  • 11