As per my understanding from Apple docs, The size of App will be reduce when you compile it for specific architecture and it's the correct way of doing it.
Apple docs :
Target Fewer CPUs
By default, Xcode projects are configured to generate optimized copies
of your app in CPU-specific "slices" of your executable. Different
hardware will run a different slice of the executable. This enables
powerful optimizations that are only possible on some devices.
However, this can substantially increase the size of your app's
executable.
To target only specific CPUs, change the Architectures build setting
from Standard $(ARCHS_STANDARD) to a list of the names of specific
CPUs you want to support. Valid CPU names are listed in the Valid
Architectures (VALID_ARCHS) build setting. Do not change the Valid
Architectures setting. It is vestigial, and best managed by Xcode.
For more info refer this