I found a pretty good way to handle this: I switched to a "source" view which is the first-person view without rotation and foreground stuff. So it looks like you're still looking out of the person's eyes, but also not really because it ignores the head rotation. This makes it way easier to handle for me.
The real panel references the source view with rotation and foreground things. References are updated live, so I immediately see how the end result looks.
