I wrote a blog post on the difficulties of memory-efficient training, which seems relevant: http://mitchgordon.me/machine/learning/2020/01/13/do-we-real...

The methods discussed there take a different angle at the problem.