@zweifel your XML looks like it's missing some stuff, but if it's working...
Have you looked at this tool to help generate your XML:
Anyway, try moving your "windowTarget" into your placement options.
Using your XML it would look like this:
<lticm:options name="course_navigation">
<lticm:property name="enabled">true</lticm:property>
<lticm:property name=" windowTarget ">_blank </lticm:property>
</lticm:options>
<lticm:options name="user_navigation">
<lticm:property name="url">https://.starfishsolutions.com/starfish-ops/support/ltiLogin.html</lticm:property>
<lticm:property name="enabled">true</lticm:property>
<lticm:property name=" windowTarget ">_blank </lticm:property>
</lticm:options>
Placing the "windowTarget" in my XML like this is working, launching my LTI tool in a new browser tab.
I hope this helps.