What can I do to avoid occasional unpleasantness?
As in other Mediterranean countries with similar cultures, you should observe local customs. Do things with others (a female or male companion you know, or a mixed group) when possible. If you’re traveling alone, introduce yourself to Turkish women or families, ask a question or strike up a conversation so that you are informally included with them.
In Turkey, as in many other countries, social encounters between men and women who are not relatives or close friends are conducted much more formally than they might be in Europe or–especially–Australia, Canada, or the USA. Also, this formality is maintained for a much longer time.
How can I be “more formal”?
Dress neatly and act reserved. Be pleasant, but don’t smile readily at men you don’t know, even when conducting business (registering at a hotel, taking a taxi ride, etc). Be correct and formal, even on the third and fourth encounter. If a man responds by being overly friendly, you should be overly formal. Keep control of the situation, keep it on yourterms.
What else can I do?
Set the rules for each encounter. Do things in public, or in group settings in which you know most of the people. Avoid being alone in private with a man or men you do not know well—especially in a car. (Mixed groups, including both Turkish men and Turkish women, are usually fine.)
Cultural signals passed between men and women, and the expectations, might be quite different, and not what is intended. Neither person is wrong or right, just different. You need to be on the same wavelength for your signals to be taken as you mean them.
If you’re not interested, you may give a signal to a Western man (“Get lost!”) and he’s supposed to wander away. To a Mediterranean male, “Get lost!” might be taken merely as a pro-forma protest meant to protect your honor, to show that you didn’t yield to his charms too easily. You’re expected to protest, whether you’re interested in him or not.