We have cookies .example.com, so there will be sent also in the
requests to subdomain. Is it possible for us to point A record to
reverse proxy where we strip the cookies and send the request to third
party provider without them?
Great idea, you could do this yes, however you will also need to set the HttpOnly flag, otherwise they would be able to retrieve them with JavaScript.
Is it possible to set document.domain to example.com and do
XMLHttpRequest to the example.com?
No, subdomains for Ajax are treated as a different Origin. See this answer.
I guess that it would be no problem because of the same origin policy.
Subdomain is treated as separate domain?
JavaScript code could interact with each other subdomains - but only with the cooperation of your site. You would also need to also set document.domain = 'example.com';
If you do not do this, you are secure against this threat.
See here:
When using document.domain to allow a subdomain to access its parent
securely, you need to set document.domain to the same value in both
the parent domain and the subdomain. This is necessary even if doing
so is simply setting the parent domain back to its original value.
Failure to do this may result in permission errors.
Any other security issues?
You need to be aware of cookie poisoning. If evil.example.com
sets a non host-only cookie at .example.com
that your domain believes it has set itself, then the evil cookie may be used for your site.
For example, if you display the contents of the cookie as HTML, then this may introduce XSS. Also, if you're using the double submit cookies CSRF prevention method an evil domain may be able to set their own cookie value to achieve CSRF. See this answer.