I love Governors Island but never seem to make it over during the weekend, so I was super excited to learn that the National Parks Service offers public visits during the week. Did everyone already know about this? I definitely didn't but now feel the need to tell everyone.
There's a variety of tours programs offered, from a history of artillery on the island to a tour of the interior of Castle Williams, all for free, and they include ferry service from Manhattan. You won't get to wander around the island by yourself afterward but they look like a terrific way to get an in-depth view of the island's history without having to worry about the weekend crowds. Tickets for all the programs are given out an hour in advance, so just be sure to arrive early to get one.
You can find all the details over on the National Parks Service site: http://www.nps.gov/gois/planyourvisit/2013-weekday-programming-schedule.htm