> immediately, there were more bugs needing more fixing
Thing about releasing things publicly for the first time is: even if you've rigorously tested everything and ironed out bugs, users of the service will report bugs you never even thought of checking for. They will be running weird browser configs, do weird things with your service, and otherwise use it in ways you never dreamed of, surfacing obscure sharp edges in your code. This is why, upon public release you have to be ready to patch quickly.
As the author, what really confused me about this app... was there is only a single user facing form. You add your email address and your HN username. The bugs came from the dataset mostly.
The crawler actually DID use the API. I'm using crawler as it crawled the API. It would get the MAX ID from the API, then request every ID between the last one it found and that one.
One of the things that would cause a problem early on is that a request would bring back a "null" response for an ID that should exist, so it would pause and retry after 5 seconds, and do that 5 times. Sometimes it would still not have a response, so any reply to that response would then crash on a referential integrity check. So I had to remove the integrity so that that reply would still come in, even when I couldn't request the parent for whatever reason (caching?).
It was little things like that that would happen randomly at 3am, and kill the client for hours until I woke up and saw there was an exception in the log, and no threads had downloaded in hours.
Heaven forbid I was on vacation when it went down. No SSH key to remote in. I would have to hard reboot the droplet from the DO admin panel.
Thing about releasing things publicly for the first time is: even if you've rigorously tested everything and ironed out bugs, users of the service will report bugs you never even thought of checking for. They will be running weird browser configs, do weird things with your service, and otherwise use it in ways you never dreamed of, surfacing obscure sharp edges in your code. This is why, upon public release you have to be ready to patch quickly.