There are a couple of apps already on the App Store that use XTLS/xray-core, such as "V2Box - V2ray Client", "Fair VPN" and "FoXray". That might be why it is being rejected. Although you can change your library, you cannot modify the source code of xray, as it is compiled and included in your app's binary.