Doesn't seem it's related to TOS. For me error looked like
{"error":{"message":"Permissions error","type":"OAuthException","code":200,"error_subcode":1870047,"is_transient":false,"error_user_title":"Audience Size too Low","error_user_msg":"You cannot remove users from this audience because it will result in a low audience size.","fbtrace_id":"xxx"}}
Here you can find an official answer from FB team
https://developers.facebook.com/support/bugs/2093728693993530
Thanks for writing in. I can confirm that this is a valid error message. Custom Audiences must maintain a certain minimum size, though the specific number is not actually documented. I believe the required minimum is approximately 100 unique users. If you need to remove so many users that you're going under this limit, I'd recommend deleting the custom audience entirely.
Also why it's Permissions error
...