Post

Replies

Boosts

Views

Activity

Display bounding box in SceneKit
Hello, I am using YOLOv3 with Vision to classify objects during my AR session. I want to render the bounding boxes of the detected objects in my screen view. Unfortunately, the bounding boxes are are placed too far down and have a wrong aspect ratio. Does someone know what the issue might be? This is how I am currently transforming the bounding boxes. Assumptions: The app is in portrait mode Vision request is performed with centerCrop and orientation .right. Fix the coordinate origin of vision: let newY = 1 - boundingBox.origin.y     let newBox = CGRect(x: boundingBox.origin.x, y: newY, width: boundingBox.width, height: boundingBox.height) Undo center cropping of Vision: let imageResolution: CGSize = currentFrame.camera.imageResolution // Switching height and width because the original image is rotated let imageWidth = imageResolution.height let imageHeight = imageResolution.width // Square inside of normalized coordinates. let roi = CGRect(x: 0, y: 1 - (imageWidth/imageHeight + ((imageHeight-imageWidth) / (imageHeight*2))), width: 1, height: imageWidth / imageHeight) let newBox = VNImageRectForNormalizedRectUsingRegionOfInterest(boundingBox, Int(imageWidth), Int(imageHeight), roi) Bring coordinates back to normalized form: let imageWidth = imageResolution.height let imageHeight = imageResolution.width let transformNormalize = CGAffineTransform(scaleX: 1.0 / imageWidth, y: 1.0 / imageHeight) let newBox = boundingBox.applying(transformNormalize) Transform to scene view: (I assume the error is here. I found out while debugging that the aspect ratio of the bounding box changes here.) let viewPort = sceneView.frame.size let transformFormat = currentFrame.displayTransform(for: .landscapeRight, viewportSize: viewPort) let newBox = boundingBox.applying(transformFormat) Scale up to viewport size: let viewPort = sceneView.frame.size let transformScale = CGAffineTransform(scaleX: viewPort.width, y: viewPort.height) let newBox = boundingBox.applying(transformScale) Thanks in advance for any help!
1
0
2.0k
Sep ’22
How to turn AR SpriteKit App into AR SceneKit App
Hello! I am new to ARKit and currently working on this sample SpriteKit AR application from Apple. To use ARKits debug option of visualizing the feature points I am trying to change the app from an AR SpriteKit base to an AR SceneKit base. (I thought it should be possible after watching this Apple session.) Therefore I made some changes. But unfortunately, the application doesn't show the camera input anymore and throws an exception after some time. The object detector still correctly identifies objects but the main screen stays gray. Help would be much appreciated! Best regards! Main changes: @IBOutlet weak var sceneView: ARSCNView! // @IBOutlet weak var sceneView: ARSKView! class ViewController: ARSCNViewDelegate {} // class ViewController: ARSKViewDelegate {} Put the SpriteKit overlayScene into a SCNPlane. Had to change how child nodes get hidden when relocalization happens.
1
1
849
Jun ’22
Display bounding box in SceneKit
Hello, I am using YOLOv3 with Vision to classify objects during my AR session. I want to render the bounding boxes of the detected objects in my screen view. Unfortunately, the bounding boxes are are placed too far down and have a wrong aspect ratio. Does someone know what the issue might be? This is how I am currently transforming the bounding boxes. Assumptions: The app is in portrait mode Vision request is performed with centerCrop and orientation .right. Fix the coordinate origin of vision: let newY = 1 - boundingBox.origin.y     let newBox = CGRect(x: boundingBox.origin.x, y: newY, width: boundingBox.width, height: boundingBox.height) Undo center cropping of Vision: let imageResolution: CGSize = currentFrame.camera.imageResolution // Switching height and width because the original image is rotated let imageWidth = imageResolution.height let imageHeight = imageResolution.width // Square inside of normalized coordinates. let roi = CGRect(x: 0, y: 1 - (imageWidth/imageHeight + ((imageHeight-imageWidth) / (imageHeight*2))), width: 1, height: imageWidth / imageHeight) let newBox = VNImageRectForNormalizedRectUsingRegionOfInterest(boundingBox, Int(imageWidth), Int(imageHeight), roi) Bring coordinates back to normalized form: let imageWidth = imageResolution.height let imageHeight = imageResolution.width let transformNormalize = CGAffineTransform(scaleX: 1.0 / imageWidth, y: 1.0 / imageHeight) let newBox = boundingBox.applying(transformNormalize) Transform to scene view: (I assume the error is here. I found out while debugging that the aspect ratio of the bounding box changes here.) let viewPort = sceneView.frame.size let transformFormat = currentFrame.displayTransform(for: .landscapeRight, viewportSize: viewPort) let newBox = boundingBox.applying(transformFormat) Scale up to viewport size: let viewPort = sceneView.frame.size let transformScale = CGAffineTransform(scaleX: viewPort.width, y: viewPort.height) let newBox = boundingBox.applying(transformScale) Thanks in advance for any help!
Replies
1
Boosts
0
Views
2.0k
Activity
Sep ’22
How to turn AR SpriteKit App into AR SceneKit App
Hello! I am new to ARKit and currently working on this sample SpriteKit AR application from Apple. To use ARKits debug option of visualizing the feature points I am trying to change the app from an AR SpriteKit base to an AR SceneKit base. (I thought it should be possible after watching this Apple session.) Therefore I made some changes. But unfortunately, the application doesn't show the camera input anymore and throws an exception after some time. The object detector still correctly identifies objects but the main screen stays gray. Help would be much appreciated! Best regards! Main changes: @IBOutlet weak var sceneView: ARSCNView! // @IBOutlet weak var sceneView: ARSKView! class ViewController: ARSCNViewDelegate {} // class ViewController: ARSKViewDelegate {} Put the SpriteKit overlayScene into a SCNPlane. Had to change how child nodes get hidden when relocalization happens.
Replies
1
Boosts
1
Views
849
Activity
Jun ’22