Hmm. I've never actually hosted before but when other people host and when I play alone on my server I always get better performance in an online environment. It should not really go down unless you're internet is atrocious.

Make sure no one is running anything but the required mods and that they are all up-to-date for everyone. I'm betting this is your problem. How is your performance on the server when you're by yourself? Better or worse than in SP?

You can also mess with the Active Limiter in the Virtual AI module if need be. Your total number of units seems fine for a map that size. Probably about the same as I'd use to start (I'd probably start with 100 civ with the filtering you're using and 30 Mil with higher filtering than what you're using with putting some single unit cust modules to fill the gaps; but if you're not getting the 80+ message objective-wise you sound ok with what you're doing).