There's a fair amount of experimental work happening trying different parsing and resolution procedures such that the training data reflects an AST and or predicts nodes in an AST as an in-filling capability.

Do you know if any such experimental work is using a special tokenizer for example in Lisp a special token for left or right parenthesis?

We plan to do both. We want to give back to Engrafo where it makes sense, but we also have a unique opportunity given our other semantic features along with the paper metadata from our corpus to build upon this experience. This release is an MVP reading experience, but in the future we plan to add a number of things like: user highlighting, direct linking to authors and citations, and collaborative commenting.

+1 to what @undefined said.

If all goes well, we hope to act as a front-facing host for the Engrafo engine. That'll enable the ArXiv Vanity team to focus their efforts on improving the conversion process (which is what they'd like to focus on) while we can handle the logistics of serving this up to users (as quickly as possible).

I'm really excited about the opportunity to collaborate and support the ArXiv Vanity / Engrafo team!

(I also work on the Semantic Scholar team)

1. Totally agree. We know how we can / will do this and plan to do so given enough interest in the MVP reading experience.

2. The url structure is indeed for SEO purposes. (We get a large majority of users discovering our paper pages through organic search)

The proposed shorter format could maybe be a redirect then? Quickly being able to turn one link into the other would be quite useful.

Definitely, thanks!

Disclaimer -- I work at Moz

Another great resource on, and as, content marketing similar in depth to the aforementioned Beginner's Guide to SEO:

https://moz.com/beginners-guide-to-content-marketing