How to change the compact size class programatically?

Question

I have added size classes on a Label for both compact and regular types.

The compact, however, affects all iPhone devices, but not all iPhone screen sizes are equal, hence I would like to have a bit more granular control and reduce the size for iPhone 5/SE even further.

I haven't been able to find an official way for this so I thought I do it directly in code.

There is the DeviceKit pod that helps to find the iPhone type:

import DeviceKit

override func viewDidLoad() {
    super.viewDidLoad()
    let groupOfAllowedDevices: [Device] = [.iPhoneSE, .iPhone5s, .iPhone5, .iPhone5c, .simulator(.iPhoneSE), .simulator(.iPhone5s), .simulator(.iPhone5), .simulator(.iPhone5c)]
    let device = Device()
    if device.isOneOf(groupOfAllowedDevices) {
        myLabel.font = UIFont.boldSystemFont(ofSize: 10.0)
    }
}

However, this doesn't work. This would only work if I removed both Size Classes in IB.

Is there a way to manipulate the Compact size class of a specific label? That way I could leave the default compact size for all devices other than iPhone SE/iPhone 5, and only decrease it further for the given two devices.

If there is a better way, I'm happy to know about it. Thanks