A new site for a town’s centre leisure centre has been agreed but a councillor has warned the cost will be “enormous”.