Additionally, in case needed as well, here is my conversion script: import torch import coremltools as ct import numpy as np import logging from ball_tracker_model import BallTrackerNet def convert_to_coreml(model_path): logging.basicConfig(level=logging.DEBUG) model = BallTrackerNet() model.load_state_dict(torch.load(model_path, map_location='cpu')) model.eval() example_input = torch.rand(1, 9, 360, 640) # Trace the model to verify shapes traced_model = torch.jit.trace(model, example_input) model_coreml = ct.convert( traced_model, inputs=[ ct.TensorType( name="input_frames", shape=(1, 9, 360, 640), dtype=np.float32, ) ], convert_to="mlprogram", minimum_deployment_target=ct.target.iOS15, ) model_coreml.save("BallTracker.mlpackage") return model_coreml # Run conversion try: model = convert_to_coreml("balltrackerbest.pt") print("Conversion successful!") except Exception as e: print(f"Conversion error: {str(e)}") Thanks again!

Yes, this is what I am seeing in Xcode. xcrun coremlcompiler metadata path/to/model.mlpackage says the following: [ { "metadataOutputVersion" : "3.0", "storagePrecision" : "Float16", "outputSchema" : [ { "hasShapeFlexibility" : "0", "isOptional" : "0", "dataType" : "Float32", "formattedType" : "MultiArray (Float32 1 × 256 × 230400)", "shortDescription" : "", "shape" : "[1, 256, 230400]", "name" : "var_462", "type" : "MultiArray" } ], "modelParameters" : [ ], "specificationVersion" : 6, "mlProgramOperationTypeHistogram" : { "Cast" : 2, "Conv" : 18, "Relu" : 18, "BatchNorm" : 18, "Reshape" : 1, "UpsampleNearestNeighbor" : 3, "MaxPool" : 3 }, "computePrecision" : "Mixed (Float16, Float32, Int32)", "isUpdatable" : "0", "availability" : { "macOS" : "12.0", "tvOS" : "15.0", "visionOS" : "1.0", "watchOS" : "8.0", "iOS" : "15.0", "macCatalyst" : "15.0" }, "modelType" : { "name" : "MLModelType_mlProgram" }, "userDefinedMetadata" : { "com.github.apple.coremltools.source_dialect" : "TorchScript", "com.github.apple.coremltools.source" : "torch==2.5.1", "com.github.apple.coremltools.version" : "8.1" }, "inputSchema" : [ { "hasShapeFlexibility" : "0", "isOptional" : "0", "dataType" : "Float32", "formattedType" : "MultiArray (Float32 1 × 9 × 360 × 640)", "shortDescription" : "", "shape" : "[1, 9, 360, 640]", "name" : "input_frames", "type" : "MultiArray" } ], "generatedClassName" : "BallTracker", "method" : "predict" } ]

Yes this only happens with BallTracker.mlpackage. Yes, I have filed a feedback assistant report and attached the model.

Yes, this is the link: https://feedbackassistant.apple.com/feedback/16382305 and the feedback number I believe is FB16382305 Thanks!

Oh, that is interesting. The BallTracker.mlpackage file I attached to the feedback assistant was the one I ran the xcrun coremlcompiler metadata path/to/model.mlpackage command on. If this produces: [ { "metadataOutputVersion" : "3.0", "storagePrecision" : "Float16", "outputSchema" : [ { "hasShapeFlexibility" : "0", "isOptional" : "0", "dataType" : "Float32", "formattedType" : "MultiArray (Float32 1 × 256 × 230400)", "shortDescription" : "", "shape" : "[1, 256, 230400]", "name" : "var_462", "type" : "MultiArray" } ], "modelParameters" : [ ], "specificationVersion" : 6, "mlProgramOperationTypeHistogram" : { "Cast" : 2, "Conv" : 18, "Relu" : 18, "BatchNorm" : 18, "Reshape" : 1, "UpsampleNearestNeighbor" : 3, "MaxPool" : 3 }, "computePrecision" : "Mixed (Float16, Float32, Int32)", "isUpdatable" : "0", "availability" : { "macOS" : "12.0", "tvOS" : "15.0", "visionOS" : "1.0", "watchOS" : "8.0", "iOS" : "15.0", "macCatalyst" : "15.0" }, "modelType" : { "name" : "MLModelType_mlProgram" }, "userDefinedMetadata" : { "com.github.apple.coremltools.source_dialect" : "TorchScript", "com.github.apple.coremltools.source" : "torch==2.5.1", "com.github.apple.coremltools.version" : "8.1" }, "inputSchema" : [ { "hasShapeFlexibility" : "0", "isOptional" : "0", "dataType" : "Float32", "formattedType" : "MultiArray (Float32 1 × 9 × 360 × 640)", "shortDescription" : "", "shape" : "[1, 9, 360, 640]", "name" : "input_frames", "type" : "MultiArray" } ], "generatedClassName" : "BallTracker", "method" : "predict" } ] Then in theory it should work in Xcode? Unfortunately this is not the case for my model though. Does my conversion code look off in any places? Do you have any other recommendations for getting the model working on Xcode? Sorry for the long thread, just haven't seen anyone else online have a similar issue and want to get to the bottom of this. Thanks!

Additionally, when I run the model with test data in VSCode, I am getting an expected output so I think the conversion itself is working, the model is just not getting properly read by Xcode for some reason: model = ct.models.MLModel("BallTracker.mlpackage") input_data = np.random.rand(1, 9, 360, 640).astype(np.float32) inputs = {"input_frames": input_data} # Predict with the model results = model.predict(inputs) print(results) # this yields valid results

I found out what it was. I can't drag the model directly from VSCode to Xcode so I first drag the file onto my desktop. This makes it show up as a txt for whatever reason so I had to import the file by adding the file from Xcode, and referencing the file in VSCode directly.

Ok thanks! This is really helpful! I had a couple of follow up questions: So the general approach to minimize run time would be to get the CVPixelBuffers from the camera and convert them into vImages so I can resize them quickly, then change these vImages into Float arrays which can be reshaped and converted to MLTensors in which we can perform the needed operations? When I am coding in Xcode, I am getting cannot find "MLTensor" in scope, though I have imported CoreML. How can I use MLTensor? Right now my model takes in MultiArrays as input, and produces a MultiArray as output. How do I set the input and outputs to be MLTensors? I have my current converter code pasted below def convert_to_coreml(model_path): logging.basicConfig(level=logging.DEBUG) model = BallTrackerNet() model.load_state_dict(torch.load(model_path, map_location='cpu')) model.eval() example_input = torch.rand(1, 9, 360, 640) # Trace the model to verify shapes traced_model = torch.jit.trace(model, example_input) model_coreml = ct.convert( traced_model, inputs=[ ct.TensorType( name="input_frames", shape=(1, 9, 360, 640), dtype=np.float32 ) ], convert_to="mlprogram", ) model_coreml.save("BallTracker2.mlpackage") return model_coreml # Run conversion try: model = convert_to_coreml("balltrackerbest.pt") print("Conversion successful!") except Exception as e: print(f"Conversion error: {str(e)}") Thanks!

Ok, I have set the minimum deployment target to iOS 18.3 but it is still unable to find MLTensor in scope. Do I need to create my own struct of MLTensor somehow? Here is my current code: import CoreML class BallTracker { let model: BallTrackerPreprocess init() { self.model = try! BallTrackerPreprocess(configuration: MLModelConfiguration()) } func processFrame(from frames: [CVPixelBuffer]) -> MLTensor? { // f1 is current, f2 is prev, f3 is previous prev guard frames.count == 3 else { return nil } let width = CVPixelBufferGetWidth(frames[0]) let height = CVPixelBufferGetHeight(frames[0]) let shape: [Int] = [3, height, width] let tensor = MLTensor(concatenating: frames.map { MLTensor(pixelBuffer: $0) }, dimension: 0) do { let prediction = try model.prediction(input_frames: tensor) return prediction.var_366 } catch { print("error: \(error)") return nil } } }