Hi darkmaterial,
I'm not answering you question directly, cause I don't remember exactly what you are missing.
but, I want to write some info I wrote for my self next time I'm going to deal with CALayers and AppKit hope it helps, read and check it, good chance the answer is there :
Difference between uiview-layer and nsview layer
https://stackoverflow.com/questions/9551992/how-to-add-a-calayer-to-an-nsview-on-mac-os-x
https://stackoverflow.com/questions/15966219/calayer-drawrect
Very important slide show of WWDC 2012 about NSView layers and drawing
https://docs.huihoo.com/apple/wwdc/2012/session_217__layerbacked_views_appkit__core_animation.pdf
https://www.objc.io/issues/14-mac/appkit-for-uikit-developers/
By default, AppKit views are not backed by Core Animation layers; layer-backing support has been integrated into AppKit retroactively. But while you never have to worry about this with UIKit, with AppKit there are decisions to make. AppKit differentiates between layer-backed and layer-hosting views
If you want to interact with the layer in such ways, then you have to go one step further. Overriding NSView‘s wantsUpdateLayer method to return YES enables you to change the layer’s properties. If you do this though, AppKit will no longer call the view’s drawRect: method. Instead, updateLayer will be called during the view update cycle, and this is where you can modify the layer.
To make appKit layers works like UIKit layers
wantsLayer = YES
wantsUpdateLayer return YES
And now - (void)updateLayer will be called instead of drawRect:
Make sure layerContentsRedrawPolicy property to NSViewLayerContentsRedrawOnSetNeedsDisplay
This way, you have control over when the layer contents need to be redrawn. A frame change will not automatically trigger a redraw anymore; you are now responsible for triggering it by calling -setNeedsDisplay:
Again all the above is note I have for my self, incase I want to do something with CALayers ( rarely).
I hope this help you.
Good luck