Strongbox is NOT compatible with the KeeChallenge plugin. You cannot use a KeeChallenge protected database with Strongbox and you cannot use a YubiKey protected database created by Strongbox with the KeeChallenge plugin.
Strongbox uses the KeePassXC paradigm for Challenge Response via YubiKey. There are couple of technical reasons for this design choice which means that YubiKey works better in the mobile context particularly. KeeChallenge has not been updated since 2016 and we are not sure about what kind of support is offered. We feel it is better to not require or use an auxilliary XML challenge file, which can get out of sync with your database or lost entirely. The KeePassXC mode of YubiKey support does not require this and feel it is more convenient and less prone to error/loss.
If you would like to be able to use YubiKey to protect your database with Strongbox and also on other platforms, we highly recommend you use KeePassXC on those platforms. To convert to the Strongbox and KeePassXC method, you must first remove YubiKey protection from your KeeChallenge based database and then add YubiKey as a factor via Strongbox or KeePassXC.