I have a lot of theories on why people are discouraged to both 1) enter, and 2) vote. (Maybe people don't want to vote for entries because it takes votes away from their own entries, for example. I'm not saying this is how I feel. When I enter, I vote.)
I would encourage the whole voting system/judging system to be revisited.
For example, the Avatar Library contest I think works very well.
1 person sets the theme, judges the entries, and picks the winner.
Winner sets the new theme, and so on.
It's the easiest contest to enter and judge, the vibe is so friendly and encouraging, and different people actually win each time. So there's fairness. Just my opinion. It's been so much fun (when I've had time, and when I haven't had time, the contest easily goes on with those who do.) I don't think people's feelings get offended because every "Winner" has their own unique taste. It's only up to one person who decides the winner, so if you don't win, what's the big deal? You could easily win the next round.
I think simplicity is key. The more complications, the more people are discouraged. For example, each voter picking a Top 3 might be cumbersome for some.