The Jungle Club in Koh Samui is a hidden gem offering stunning panoramic views of the Gulf of Siam. It is technically a resort with an incredible infinity pool that is only accessible if you are guest...but you can still enjoy the insane view from the restaurant/bar.
I highly recommend making a reservation since it does book out. Its located on top of a huge hill in the jungle and my poor motorbike couldn't make it up the climb so we had to park at the bottom and walk.
They do offer 4x4 transportation from the town if needed.