It depends.
IdentityServer will provide you with OAuth 2.0 and OpenID Connect implementation, and it will handle all details for you (providing you endpoints, token management, scopes, grants and so on). It runs independently so you can use it for multiple clients (SPA, mobile, web apps) and it is nicely isolated from rest of your app. If you wish so, you can use it together with ASP.NET Core Identity.
If you don't use IdentityServer, you will have to write some of these things yourself because ASP.NET Core Identity is a membership system and it does not provide any ready to use endpoints and neither token management or support for different ways how to authorize.
You need to evaluate whether it is better for you to write these things yourself but have a more straightforward setup because you probably don't need everything IdentityServer provides although it might limit you in future.
You can also have a look at OpenIddict that is less complicated than IdentityServer.