I would say yes. When I was training people on a "CELTA equivalent" course we had quite a few qualified primary school teachers and they found the switch to TEFL quite an adjustment and didn't manage to bring their teaching skills to bear until the 3rd or 4th week of the course. Some of them looked like they might fail and were worse than people with no teaching experience early on. This will particularly be so if you are thinking of switching to teaching adults. As a TEFL manager employing teachers with no previous TEFL jobs on their CV I would expect the same thing- that the people with TEFL qualifications would be able to cope much better from day one but that the people with more impressive training in teaching other subjects might eventually outstrip them- but not before causing some problems when they first started. If that is the choice, most directors would choose the TEFL course people every time.
The exception to this is if you will be teaching maths abroad, usually meaning in an international school