World Class is the biggest fitness center chain in Iceland. They have really fair prices for memberships (drop ins are expensive) and you can also go to the public pools that are connected to their gyms. This one in Laugardalur is their biggest and I highly recommend it if you're staying a while.