Try a different activation function than relu (e.g. elu, tanh)