Unfortunately, limiting the size of popup is not supported: the Region that's responsible for showing the MenuItems is ContextMenuContent and implements its computeMaxHeight to return the screenHeight. That container is created by ContextMenuSkin and stored into a private final field, so there's no way to replace it with a custom implementation with a more intelligent implementation.
What we can do, though, is to access that region and set its maxHeight to the same value as the ContextMenu. To remain off the evil illegal reflective access to the private field, we can register a handler for the Menu.ON_SHOWING event and update the size as needed [*].
Something like
public class MaxSizedContextMenu extends ContextMenu {
public MaxSizedContextMenu() {
addEventHandler(Menu.ON_SHOWING, e -> {
Node content = getSkin().getNode();
if (content instanceof Region) {
((Region) content).setMaxHeight(getMaxHeight());
}
});
}
}
[*] update: to make this work, the ContextMenu must have a reasonable maxHeight (default is Double.MAX_VALUE), that is it must be set manually after instantiation. Furthermore, we have to use the ContextMenu's maxHeight in the eventHandler (vs. f.i. an arbitrary constant), otherwise vertical location of the popup is broken - the layout code still thinking, that it's filling the entire screen height.
ContextMenu menu = new MaxSizedContextMenu();
menu.setMaxHeight(200);