"it seems to me that it's worse than having no second factor at all"

I agree with you totally. A second authentication that is much weaker than the first (prone to social engineering or even googling like the endemic first name of your favorite uncle questions) can be worse than just having one strong authentication. For some reason many companies think that building a bridge out of very many weak components makes a strong bridge. That is not true though, you have a weak bridge in the end.