No need for Swansea to panic yet despite Premier League worries