I really think the idea of pushing/pinging search engines to come get content should really be linked to the sitemaps protocol if anything, to me that seems the most logical solution.

Basically, your main sitemap file would consist of a sitemapindex which then links to at least 2 more separate files. The first being your recently updated list that gets flushed when a search engine hits your site, and the rest containing a full index of the content on your site.