The address you specify isn't the client's IP address, it's the address of the interface on your own host that it's going to listen on.
So, to listen only on localhost, you specify 127.0.0.1. To listen on the external interface that the outside world can see, you can specify that address.
Or, you can specify an address of 0.0.0.0
, so a client can connect via any available interface.
Note that although it may initially seem a little crazy, it can actually make perfect sense to listen only on localhost. This lets you "cheat" a little bit on security. Instead of trying (and most likely failing) to do security on your own, an outside user forms an SSH tunnel to your computer, then has their client connect to your server through that tunnel. Your server only has to do its thing, and leaves all the authentication, privacy, etc., to SSH.