I have CVPixelBuffers in YCbCr (422 or 420 biplanar 10 bit video range) coming from camera. I see vImage framework is sophisticated enough to handle a variety of image formats (including pixel buffers in various YCbCr formats). I was looking to compute histograms for both Y (luma) and RGB. For the 8 bit YCbCr samples, I could use this code to compute histogram of Y component. CVPixelBufferLockBaseAddress(pixelBuffer, .readOnly) let bytesPerRow = CVPixelBufferGetBytesPerRowOfPlane(pixelBuffer, 0) let baseAddress = CVPixelBufferGetBaseAddressOfPlane(pixelBuffer, 0) let height = CVPixelBufferGetHeightOfPlane(pixelBuffer, 0) let width = CVPixelBufferGetWidthOfPlane(pixelBuffer, 0) CVPixelBufferUnlockBaseAddress(pixelBuffer, .readOnly) var buffer = vImage_Buffer(data: baseAddress, height: vImagePixelCount( height), width: vImagePixelCount(width), rowBytes: bytesPerRow) alphaBin.withUnsafeMutableBufferPointer { alphaPtr in let error = vImageHistogramCalculation_Planar8(&buffer, alphaPtr.baseAddress!, UInt32(kvImageNoFlags)) guard error == kvImageNoError else { fatalError("Error calculating histogram luma: \(error)") } } How does one implement the same for 10 bit HDR pixel buffers, preferably using new iOS 16 vImage APIs that provide lot more flexibility (for instance, getting RGB histogram as well from YCbCr sample without explicitly performing pixel format conversion)?

I have the following code adapted from AVCamManual sample code to set white balance. I still see crash reports in analytics where exception is raised in setting WB: *** -[AVCaptureDevice temperatureAndTintValuesForDeviceWhiteBalanceGains:] whiteBalanceGains contain an out-of-range value - red, green, and blue gain Here is my code, it is not clear how things are turning out of range. public func normalizedGains(gains:AVCaptureDevice.WhiteBalanceGains) -> AVCaptureDevice.WhiteBalanceGains { var g = gains if let device = videoDevice { g.redGain = max(1.0, g.redGain) g.blueGain = max(1.0, g.blueGain) g.greenGain = max(1.0, g.greenGain) g.redGain = min(device.maxWhiteBalanceGain, g.redGain) g.blueGain = min(device.maxWhiteBalanceGain, g.blueGain) g.greenGain = min(device.maxWhiteBalanceGain, g.greenGain) } return g } And my code to set WB: public func setTemperatureAndTint( colorTemperature:Float?, tint:Float?) { if let device = videoDevice { var tint = tint var colorTemperature = colorTemperature if colorTemperature == nil { colorTemperature = device.temperatureAndTintValues(for: device.deviceWhiteBalanceGains).temperature } if tint == nil { tint = device.temperatureAndTintValues(for: device.deviceWhiteBalanceGains).tint } let temperatureTint = AVCaptureDevice.WhiteBalanceTemperatureAndTintValues(temperature: colorTemperature!, tint: tint!) NSLog("Setting tint \(temperatureTint.tint)") do { try device.lockForConfiguration() device.setWhiteBalanceModeLocked(with: normalizedGains(gains: device.deviceWhiteBalanceGains(for: temperatureTint)) , completionHandler: nil) device.unlockForConfiguration() wbLockedtoGray = false } catch { NSLog("Unable to change White balance gain \(error)") } } } Is there anything I am doing wrong?

I am trying to use a CIColorKernel or CIBlendKernel with sampler arguments but the program crashes. Here is my shader code which compiles successfully. extern "C" float4 wipeLinear(coreimage::sampler t1, coreimage::sampler t2, float time) { float2 coord1 = t1.coord(); float2 coord2 = t2.coord(); float4 innerRect = t2.extent(); float minX = innerRect.x + time*innerRect.z; float minY = innerRect.y + time*innerRect.w; float cropWidth = (1 - time) * innerRect.w; float cropHeight = (1 - time) * innerRect.z; float4 s1 = t1.sample(coord1); float4 s2 = t2.sample(coord2); if ( coord1.x > minX && coord1.x < minX + cropWidth && coord1.y > minY && coord1.y <= minY + cropHeight) { return s1; } else { return s2; } } And it crashes on initialization. class CIWipeRenderer: CIFilter { var backgroundImage:CIImage? var foregroundImage:CIImage? var inputTime: Float = 0.0 static var kernel:CIColorKernel = { () -> CIColorKernel in let url = Bundle.main.url(forResource: "AppCIKernels", withExtension: "ci.metallib")! let data = try! Data(contentsOf: url) return try! CIColorKernel(functionName: "wipeLinear", fromMetalLibraryData: data) //Crashes here!!!! }() override var outputImage: CIImage? { guard let backgroundImage = backgroundImage else { return nil } guard let foregroundImage = foregroundImage else { return nil } return CIWipeRenderer.kernel.apply(extent: backgroundImage.extent, arguments: [backgroundImage, foregroundImage, inputTime]) } } It crashes in the try line with the following error: Fatal error: 'try!' expression unexpectedly raised an error: Foundation._GenericObjCError.nilError If I replace the kernel code with the following, it works like a charm: extern "C" float4 wipeLinear(coreimage::sample_t s1, coreimage::sample_t s2, float time) { return mix(s1, s2, time); }

It seems AVAssetWriter is rejecting CVPixelBuffers with error -12743 when appending NSData for kCVImageBufferAmbientViewingEnvironmentKey for HDR videos. Here is my code: var ambientViewingEnvironment:CMFormatDescription.Extensions.Value? var ambientViewingEnvironmentData:NSData? ambientViewingEnvironment = sampleBuffer.formatDescription?.extensions[.ambientViewingEnvironment] let plist = ambientViewingEnvironment?.propertyListRepresentation ambientViewingEnvironmentData = plist as? NSData And then attaching this data, CVBufferSetAttachment(dstPixelBuffer, kCVImageBufferAmbientViewingEnvironmentKey, ambientViewingEnvironmentData! as CFData, .shouldPropagate) No matter what I do, including copying the attachment from sourcePixelBuffer to destinationPixelBuffer as it is, the error remains! var attachmentMode:CVAttachmentMode = .shouldPropagate let attachment = CVBufferCopyAttachment(sourcePixelBuffer!, kCVImageBufferAmbientViewingEnvironmentKey, &attachmentMode) NSLog("Attachment \(attachment!), mode \(attachmentMode)") CVBufferSetAttachment(dstPixelBuffer, kCVImageBufferAmbientViewingEnvironmentKey, attachment!, attachmentMode) I need to know if there is anything wrong in the way metadata is copied.

I am processing CVPixelBuffers received from camera using both Metal and CoreImage, and comparing the performance. The only processing that is done is taking a source pixel buffer and applying crop & affine transforms, and saving the result to another pixel buffer. What I do notice is CPU usage is as high a 50% when using CoreImage and only 20% when using Metal. The profiler shows most of the time spent is in CIContext render: let cropRect = AVMakeRect(aspectRatio: CGSize(width: dstWidth, height: dstHeight), insideRect: srcImage.extent) var dstImage = srcImage.cropped(to: cropRect) let translationTransform = CGAffineTransform(translationX: -cropRect.minX, y: -cropRect.minY) var transform = CGAffineTransform.identity transform = transform.concatenating(CGAffineTransform(translationX: -(dstImage.extent.origin.x + dstImage.extent.width/2), y: -(dstImage.extent.origin.y + dstImage.extent.height/2))) transform = transform.concatenating(translationTransform) transform = transform.concatenating(CGAffineTransform(translationX: (dstImage.extent.origin.x + dstImage.extent.width/2), y: (dstImage.extent.origin.y + dstImage.extent.height/2))) dstImage = dstImage.transformed(by: translationTransform) let scale = max(dstWidth/(dstImage.extent.width), CGFloat(dstHeight/dstImage.extent.height)) let scalingTransform = CGAffineTransform(scaleX: scale, y: scale) transform = CGAffineTransform.identity transform = transform.concatenating(scalingTransform) dstImage = dstImage.transformed(by: transform) if flipVertical { dstImage = dstImage.transformed(by: CGAffineTransform(scaleX: 1, y: -1)) dstImage = dstImage.transformed(by: CGAffineTransform(translationX: 0, y: dstImage.extent.size.height)) } if flipHorizontal { dstImage = dstImage.transformed(by: CGAffineTransform(scaleX: -1, y: 1)) dstImage = dstImage.transformed(by: CGAffineTransform(translationX: dstImage.extent.size.width, y: 0)) } var dstBounds = CGRect.zero dstBounds.size = dstImage.extent.size _ciContext.render(dstImage, to: dstPixelBuffer!, bounds: dstImage.extent, colorSpace: srcImage.colorSpace ) Here is how CIContext was created: _ciContext = CIContext(mtlDevice: MTLCreateSystemDefaultDevice()!, options: [CIContextOption.cacheIntermediates: false]) I want to know if I am doing anything wrong and what could be done to lower CPU usage in CoreImage?

I need to know correct color conversion matrices for converting YCbCr422 and YCbCr420 10 bit video range sample buffers (BT.2020 color space) to RGB. AVFoundation framework mentions AVVideoYCbCrMatrix_ITU_R_2020 which is a string constant. But I need to know the full matrix that can be used to perform color conversion. I have this matrix for full range BT2020, not sure if this is correct and what is the correct way to adapt it to video range. let colorMatrixBT2020_fullRange = ColorConversion(matrix: matrix_float3x3(columns: (simd_float3(1.0, 1.0, 1.0), simd_float3(0.000, -0.11156702/0.6780, 1.8814), simd_float3(1.4746, -0.38737742/0.6780, 0.000))), offset: vector_float3(0.0, -0.5, -0.5))

I want to know the correct way to enable Apple log on AVCaptureDevice. I understand one can query device formats, query whether a format supports AVCaptureColorSpace_AppleLog, and set the format and set the activeColorSpace to this value. Is that the way to enable Log?

The native camera app on iPhone 15 can record videos directly in external hard drive. Is there an API to achieve the same in Photos framework?

This issue is driving me crazy. I load an NSAttributedString in UITextView and within moments after loading the foregroundColor attribute of text is erased(i.e becomes white) without me doing anything. Here is the code and NSLog dump. How do I debug this I wonder? class ScriptEditingView: UITextView, UITextViewDelegate { var defaultFont = UIFont.preferredFont(forTextStyle: .body) var defaultTextColor = UIColor.white private func commonInit() { self.font = UIFont.preferredFont(forTextStyle: .body) self.allowsEditingTextAttributes = true self.textColor = defaultTextColor self.backgroundColor = UIColor.black self.isOpaque = true self.isEditable = true self.isSelectable = true self.dataDetectorTypes = [] self.showsHorizontalScrollIndicator = false } } And then in my ViewController that contains the UITextView, I have this code: textView = ScriptEditingView(frame: newTextViewRect, textContainer: nil) textView.delegate = self view.addSubview(textView) textView.allowsEditingTextAttributes = true let guide = view.safeAreaLayoutGuide // 5 textView.translatesAutoresizingMaskIntoConstraints = false NSLayoutConstraint.activate([ textView.leadingAnchor.constraint(equalTo: guide.leadingAnchor), textView.trailingAnchor.constraint(equalTo: guide.trailingAnchor), textView.topAnchor.constraint(equalTo: view.topAnchor), textView.bottomAnchor.constraint(equalTo: view.bottomAnchor) ]) textView.attributedText = attributedString NSLog("Attributed now") dumpAttributesOfText() DispatchQueue.main.asyncAfter(deadline: .now() + 1.0) { NSLog("Attributes after 1 sec") self.dumpAttributesOfText() } And here is code to dump attributes of text: private func dumpAttributesOfText() { textView.attributedText?.enumerateAttributes(in: NSRange(location: 0, length: textView.attributedText!.length), options: .longestEffectiveRangeNotRequired, using: { dictionary, range, stop in NSLog(" range \(range)") if let font = dictionary[.font] as? UIFont { NSLog("Font at range \(range) - \(font.fontName), \(font.pointSize)") } if let foregroundColor = dictionary[.foregroundColor] as? UIColor { NSLog("Foregroundcolor \(foregroundColor) at range \(range)") } if let underline = dictionary[.underlineStyle] as? Int { NSLog("Underline \(underline) at range \(range)") } }) } The logs show this: 2022-07-02 13:16:02.841199+0400 MyApp[12054:922491] Attributed now 2022-07-02 13:16:02.841370+0400 MyApp[12054:922491] range {0, 14} 2022-07-02 13:16:02.841486+0400 MyApp[12054:922491] Font at range {0, 14} - HelveticaNeue, 30.0 2022-07-02 13:16:02.841586+0400 MyApp[12054:922491] Foregroundcolor UIExtendedGrayColorSpace 1 1 at range {0, 14} 2022-07-02 13:16:02.841681+0400 MyApp[12054:922491] range {14, 6} 2022-07-02 13:16:02.841770+0400 MyApp[12054:922491] Font at range {14, 6} - HelveticaNeue, 30.0 2022-07-02 13:16:02.841855+0400 MyApp[12054:922491] Foregroundcolor kCGColorSpaceModelRGB 0.96863 0.80784 0.27451 1 at range {14, 6} 2022-07-02 13:16:03.934816+0400 MyApp[12054:922491] Attributes after 1 sec 2022-07-02 13:16:03.935087+0400 MyApp[12054:922491] range {0, 20} 2022-07-02 13:16:03.935183+0400 MyApp[12054:922491] Font at range {0, 20} - HelveticaNeue, 30.0 2022-07-02 13:16:03.935255+0400 MyApp[12054:922491] Foregroundcolor UIExtendedGrayColorSpace 1 1 at range {0, 20}

I noticed on iOS 17 that calling AppTransaction.shared fails with following errors. Here is the code: let result: VerificationResult<AppTransaction> = try await AppTransaction.shared NSLog("Result \(result)") It does not hit the NSLog statement and instead throws the following error logs. Error getting app transaction: Error Domain=ASDErrorDomain Code=500 "(null)" UserInfo={NSUnderlyingError=0x281815050 {Error Domain=AMSErrorDomain Code=301 "Invalid Status Code" UserInfo={NSLocalizedDescription=Invalid Status Code, AMSURL=https://mzstorekit-sb.itunes.apple.com/inApps/v1/receipts/createAppReceipt?REDACTED, AMSStatusCode=401, AMSServerPayload={ errorCode = 500317; }, NSLocalizedFailureReason=The response has an invalid status code}}} Received error that does not have a corresponding StoreKit Error: Error Domain=AMSErrorDomain Code=301 "Invalid Status Code" UserInfo= {NSLocalizedDescription=Invalid Status Code, AMSURL=https://mzstorekit- sb.itunes.apple.com/inApps/v1/receipts/createAppReceipt?REDACTED, AMSStatusCode=401, AMSServerPayload={ errorCode = 500317; }, NSLocalizedFailureReason=The response has an invalid status code} Received error that does not have a corresponding StoreKit Error: Error Domain=ASDErrorDomain Code=500 "(null)" UserInfo=.{NSUnderlyingError=0x281815050 {Error Domain=AMSErrorDomain Code=301 "Invalid Status Code" UserInfo={NSLocalizedDescription=Invalid Status Code, AMSURL=https://mzstorekit-sb.itunes.apple.com/inApps/v1/receipts/createAppReceipt?REDACTED, AMSStatusCode=401, AMSServerPayload={ errorCode = 500317; }, NSLocalizedFailureReason=The response has an invalid status code}}}

I see these errors when presenting UIActivityViewController with a video file. [ShareSheet] Failed to request default share mode for fileURL:file:///var/mobile/Containers/Data/Application/B0EB55D3-4BF1-430A-92D8-2231AFFD9499/Documents/IMG-0155.mov error:Error Domain=NSOSStatusErrorDomain Code=-10814 "(null)" UserInfo={_LSLine=1538, _LSFunction=runEvaluator} I don't understand if I am doing something wrong and what the error means. The share sheet shows anyways.

I am planning to convert a paid app to freemium. I would like existing paid users to remain unaffected in this process. In this question, I am focussing on volume purchase users (both existing and future). The info on Apple developer website advises to use original Store Kit if one needs to support Volume Purchase users: You may need to use the Original API for in-app purchase for the following features, if your app supports them: The Volume Purchase Program (VPP). For more information, see Device Management. Does that mean I can't use StoreKit2 to verify receipts of volume purchases made before the app went freemium (to get original purchase version and date), OR, the API can not be used to make in-app volume purchases and perhaps, users will not be able to make volume purchases from AppStore, OR, both?

Dear AVKit Engineers, I see a strange bug in AVKit. Even after the viewcontroller hosting AVPlayerViewController is dismissed, I see CPU spiking to over 100% which is caused by some animation code still running after AVPlayerViewController no more exists ([AVMobileChromelessControlsViewController__animateSliderToTintState:duration:completionHandler:]). How does this code continue to run even after AVPlayerViewController is no more? And what can I do to fix it?

I have tried everything but it looks to be impossible to get MTKView to display full range of colors of HDR CIImage made from CVPixelBuffer (in 10bit YUV format). Only builtin layers such as AVCaptureVideoPreviewLayer, AVPlayerLayer, AVSampleBufferDisplayLayer are able to fully display HDR images on iOS. Is MTKView incapable of displaying full BT2020_HLG color range? Why does MTKView clip colors no matter even if I set pixel Color format to bgra10_xr or bgra10_xr_srgb?  convenience init(frame: CGRect, contentScale:CGFloat) {         self.init(frame: frame)         contentScaleFactor = contentScale     }     convenience init(frame: CGRect) {         let device = MetalCamera.metalDevice         self.init(frame: frame, device: device)         colorPixelFormat = .bgra10_xr         self.preferredFramesPerSecond = 30     }     override init(frame frameRect: CGRect, device: MTLDevice?) {         guard let device = device else {             fatalError("Can't use Metal")         }         guard let cmdQueue = device.makeCommandQueue(maxCommandBufferCount: 5) else {             fatalError("Can't make Command Queue")         }         commandQueue = cmdQueue         context = CIContext(mtlDevice: device, options: [CIContextOption.cacheIntermediates: false])         super.init(frame: frameRect, device: device)         self.framebufferOnly = false         self.clearColor = MTLClearColor(red: 0, green: 0, blue: 0, alpha: 0)     } And then rendering code:  override func draw(_ rect: CGRect) {         guard let image = self.image else {             return         }         let dRect = self.bounds         let drawImage: CIImage         let targetSize = dRect.size         let imageSize = image.extent.size         let scalingFactor = min(targetSize.width/imageSize.width, targetSize.height/imageSize.height)         let scalingTransform = CGAffineTransform(scaleX: scalingFactor, y: scalingFactor)         let translation:CGPoint = CGPoint(x: (targetSize.width - imageSize.width * scalingFactor)/2 , y: (targetSize.height - imageSize.height * scalingFactor)/2)         let translationTransform = CGAffineTransform(translationX: translation.x, y: translation.y)         let scalingTranslationTransform = scalingTransform.concatenating(translationTransform)        drawImage = image.transformed(by: scalingTranslationTransform)         let commandBuffer = commandQueue.makeCommandBufferWithUnretainedReferences()         guard let texture = self.currentDrawable?.texture else {             return         }         var colorSpace:CGColorSpace                 if #available(iOS 14.0, *) {             colorSpace = CGColorSpace(name: CGColorSpace.itur_2100_HLG)!         } else {             // Fallback on earlier versions             colorSpace = drawImage.colorSpace ?? CGColorSpaceCreateDeviceRGB()         }         NSLog("Image \(colorSpace.name), \(image.colorSpace?.name)")         context.render(drawImage, to: texture, commandBuffer: commandBuffer, bounds: dRect, colorSpace: colorSpace)         commandBuffer?.present(self.currentDrawable!, afterMinimumDuration: 1.0/Double(self.preferredFramesPerSecond))         commandBuffer?.commit()     }

I understand the CVBufferSetAttachment simply appends metadata attachment to the sample buffer in a dictionary. But I see there are no errors in appending metadata that is contradictory in nature. For instance, for the sample buffers received from camera in HDR mode which are in YUV422 10 bit biplanar format, both the following succeed: CVBufferSetAttachment(testPixelBuffer!, kCVImageBufferColorPrimariesKey, kCVImageBufferColorPrimaries_ITU_R_2020, .shouldPropagate) CVBufferSetAttachment(testPixelBuffer!, kCVImageBufferTransferFunctionKey, kCVImageBufferTransferFunction_ITU_R_2100_HLG, .shouldPropagate) CVBufferSetAttachment(testPixelBuffer!, kCVImageBufferYCbCrMatrixKey, kCVImageBufferYCbCrMatrix_ITU_R_2020, .shouldPropagate) Or CVBufferSetAttachment(testPixelBuffer!, kCVImageBufferColorPrimariesKey, kCVImageBufferColorPrimaries_ITU_R_709_2, .shouldPropagate) CVBufferSetAttachment(testPixelBuffer!, kCVImageBufferTransferFunctionKey, kCVImageBufferTransferFunction_ITU_R_709_2, .shouldPropagate) CVBufferSetAttachment(testPixelBuffer!, kCVImageBufferYCbCrMatrixKey, kCVImageBufferYCbCrMatrix_ITU_R_709_2, .shouldPropagate) So one could set the color primaries and transfer function to be BT.709 format for sample buffers that are in 10 bit HDR. I see no errors when either sample buffer is appended to AVAssetWriter. I am wondering how attachments actually work and how AVFoundation resolves the contradictions?