On top of my last comment about minimums, its important to note that no system is perfect. Even in real life it still has flaws. The link below is to an article about the current NBA ASG issue but you can read it in the name of the link itself about undeserving players.
https://clutchpoints.com/nba-all-star-news-many-undeserving-players-got-votes-from-other-players/
Make the minimum games adjustment alone and that should have a large positive effect that could then allow for the other suggestions i posted here.