At 11pm I usually get the urge to go work out, go to the grocery, get some Taco Bell, etc. Most people make fun of me for this, but it is really some you should try. For example, when I just went to the grocery at about 11:30. I got there and got the closest parking spot. There were like two other customers in the store. There was no line when I went to check out, and then I walked through the cold to my car in a couple of seconds because I was parked so close. Awesome.
Compare that to going at noon on a Saturday or after work on a weekday. You have to park far away, it's crowded, you have to wait in line, and then walk all the way back to your car. Why would you do that?
This concept applies to the gym as well. I usually go about 10:30pm or 11:00pm. There are usually less than 10 people there so I don't have to wait to use any machines and there aren't those really intimidating, ripped beyond belief guys (you know the guys I'm talking about) either. I much prefer going late versus when it's crowded.
So you can go at "normal" times, but I will continue to go at my times and have a way better experience. Just my two cents.