FTS will automatically target enemy ships if they're actively attacking you. However, if they're just cruising, and you're not at red alert, then yes, FTS will target everything in scanner range. That's what you get for a cheap targeting solution!
The other mode of FTS essentially just stops the enemy-ship targeting during red alert conditions, allowing you to target everything.
If you're looking for finer control over what you target, the Targeter OXP or MTS might be the one for you. From memory, I think both have a "powered ship" mode, or something like it.
Edit to add: I find, when I'm in a high-density part of space and I need to target a specific ship, that I switch back to the standard (core) method of targeting - that of pressing "t" to turn on missile targeting, and then moving the reticle over a ship to select it.
Last edited by phkb on Thu Feb 22, 2018 8:19 am, edited 2 times in total.