In my opinion, the biggest issue is that people do think poly and swinging are the same thing, when clearly they are not. Honestly, just assuming anything about someone else’s relationship in general is the problem.
Also, television, the internet, etc. hasn’t helped. People see Sister Wives and think all poly is a man with multiple wives, when polyandry is a thing and has been for tens of thousands of years. But sex sells.
There’s nothing wrong with whatever iteration of any of it people choose to do in their own lives. But like you, I prefer a proper understanding.
Thank you for sharing.