avfilter/avf_showspectrum: Fix allocation check

If s->stop is set, the return value would be overwritten
before being checked. This bug was introduced in the switch
to AV_TX in 014ace8f98cc4a1a88e7a6d5890cef628eb9e8b0.

Signed-off-by: Andreas Rheinhardt <andreas.rheinhardt@outlook.com>
(cherry picked from commit 0992c19c3046973047b658c0e105dede404fb0c5)
Signed-off-by: Michael Niedermayer <michael@niedermayer.cc>