Problem: Misaligned Vision Bounding Boxes in SwiftUI
When using AVCaptureVideoPreviewLayer inside a UIViewRepresentable, my Vision bounding boxes were either stretched, offset to the side, or shrunken in the center. The standard GeometryReader approach failed because it reported the full screen dimensions, while the camera feed was being letterboxed or aspect-filled.
The Solution: Native Layout Bridging
I moved away from artificial scaling factors (like 1.33 or manual multipliers) and implemented a "Pure Natural" layout bridge.
Capturing the "Real" Video Frame I added a closure (onLayout) to my UIView subclass. This allowed the UIKit layer to report its actual non-zero dimensions (e.g., 832×420) back to SwiftUI only after the layout math was finalized by the system. This eliminated the (0,0) size errors during initialization.
Standardizing the Vision Request I set the VNImageCropAndScaleOption to .scaleFill. This ensures that the Vision coordinate system (0.0 to 1.0) maps exactly to the edges of the reported PreviewView bounds, making the math predictable regardless of the device aspect ratio.
The "Natural" Coordinate Flip By using the actual reported videoFeedSize, the conversion from Vision's bottom-left origin to SwiftUI's top-left origin became a simple subtraction without needing complex [ratios:]
This is the closest I've reached so far, it's not bad considering it's a test run off a photo on the computer screen, and not a real scoring system, like a live photo.
Topic:
Machine Learning & AI
SubTopic:
Core ML