I understand what you are saying because each year becomes less of people showing up in costumes at my house. In the beginning, they all did. Now it's very few that do. I didn't do this for this year but thinking for next year of adding to both my mailed invitation and my facebook invitation a comment that says something like: Uncostumed guests will be costumed by one of the creepy hosts. Ready to take a chance on what they will give you?" I have a lot of different witch hats so I am thinking I could have a lot of people in witch hats and a creepy cloth "cape." I think guests would actually have fun with that and maybe the year after putting some effort into it themselves when they see how easy it can be to have any sort of costume. I have tried costume contests with both silly and nice prizes (different years) and it is still always the same people who bother to come in a costume. Good luck and if you find something that works, let me know please! 
