Is there anyone else who has run into this issue? Or am I doing anything wrong?

I have tried to work around the problem by using the .onScrollVisibilityChange modifier to avoid render views that are not visible in the scrollview. The performance is improved a tiny bit, but it is still far from acceptable.

As @juraj noted, this problem seems to have been fixed in 15.2 :-)