I have been to the Kentucky Derby and it's quite an experience. I'm not a horse race person or gambler but you don't have to be to enjoy the Derby.
It's pretty expensive for a short vacation. Our seats were in the low end grandstand which was near the starting gate. I think a Ky Oaks/ Ky Derby two day tickets were about $400 to $500 each. We did have an issue with tickets. We purchased tickets from a ticket broker and were told the tickets were general seating for our section. But all grandstand seats are assigned seating. We had a large group and this wasn't an issue except for the actual Derby and some of us had to sit apart because our seats weren't together. This was in 2010.
We stayed in a hotel that was close enough to walk to Churchill Downs, a little less than a mile. This was about $400 to $500 per night. So a three night, two day vacation was about $2,500 for two people.
There may be some better options for lodging. I think there are some hotels that are farther out and less expensive and provide shuttles. You could stay farther out and drive in. There is parking at University of Louisville Papa John's football stadium but that's designed for 55,000 fans and the Derby has 165,000.
Of course the Signature Drinks are a Lily (a Vodka drink) for the Oaks and a Mint Julep (a sweet Bourbon drink) for the Derby. But you can also get a beer.
Have fun, you will enjoy it.