From my experience this seems to be expected. The scheduler will only let your threads run on p-cores if the load is high enough. So as long as you don't have any audible drop-outs you should be fine.