I am genuinely delighted that `dcc` (the system I play with my friends) is in the list of systems currently supported: https://rpg.actor/systems and I might participate as a result. Very cool.
Ghidra is a very impressive piece of software with a deep bench of functionality. The recent couple major releases that move to a more integrated Python experience have been very nice to use.
Kaitai is absolutely one of my favorite projects. I use it for work (parsing scientific formats, prototyping and exploring those formats, etc) as well as for fun (reverse engineering games, formats for DOSbox core dumps, etc).
I gave a guest lecture in a friend's class last week where we used Kaitai to back out the file format used in "Where in Time is Carmen Sandiego" and it was a total blast. (For me. Not sure that the class agreed? Maybe.) The Web IDE made this super easy -- https://ide.kaitai.io/ .
(On my youtube page I've got recordings of streams where I work with Kaitai to do projects like these, but somehow I am not able to work up the courage to link them here.)
RE, especially of older and more specialized software, involves dealing with odd undocumented binary formats. Which you may have to dissect carefully with a hex editor and a decompiler, so that you can get at the data inside.
Kaitai lets you prototype a parser for formats like that on the go, quick and easy.
A shot in the dark, but maybe you could give me a hint. Recently, I was interested in extracting sprites from an old game. I was able to reverse the file format of the data archive, which contained the game assets as files. However, I got stuck because the image files were obviously compressed. By chance, I found an open source reimplementation of the game and realised it was LZ77+Huffman compressed, but how would one detect the type of compression and parameters with only the file? That seems a pretty hard problem or are there good heuristics to detect that?
Some simpler cases like various RLE-type encodings can be figured out with that pattern recognizing brain - by staring at them really really hard.
For harder cases? You take the binaries that read or write your compressed files, load them in your tool (typically Ghidra nowadays), and track down the code that does it.
Then you either recognize what that code does (by staring at it really really hard), or try to re-implement it by hand while reading up on various popular compression algos in hope that doing this enlightens you.
Third option now: feed the decompiled or reimplemented code to the best LLM you have access to, and ask it. Those things are downright superhuman at pattern matching known algorithms, so use them, no such thing as "cheating" in RE.
The "hard mode" is compression implemented in hardware, with neither a software encoder or a software decoder available. In which case you better be ready for a lot of "feed data to the magic registers, see results, pray they give you a useful hint" type of blind hardware debugging. Sucks ass.
The "impossible" is when you have just the compressed binaries, with no encoder or decoder or plaintext data available to you at all. Better hope it's something common or simple enough or it's fucking hopeless. Solving that kind of thing is cryptoanalysis level of mind fuck and I am neither qualified enough nor insane enough to advise on that.
Another thing. For practical RE? ALWAYS CHECK PRIOR WORK FIRST. You finding an open source reimplementation? Good job, that's what you SHOULD be doing, no irony, that's what you should be doing ALWAYS. Always check whether someone has been there and done that! Always! Check whether someone has worked on this thing, or the older version of it, or another game in the same engine - anything at all. Can save you literal months of banging your head against the wall.
Thanks for your reply and advice! I guess what you describe as "impossible" is the case I am mostly interested in, though more for non-executable binary data. If I am not mistaken, this goes under the term "file fragment classification", but I have been wondering if practitioners might have figured out some better ways than what one can find in scholarly articles.
And yes, searching for the reimplementation beforehand would have saved me some hours :D
It's not about the data being executable. It's about having access to whatever reads or writes this data.
Whatever reads or writes this data has to be able to compress or decompress it. And with any luck, you'll be able to take the compression magic sauce from there.
I understood "binaries" in "compressed binaries" as "executables", e.g. like a packed executable, but I see that you mean indeed a binary file (and not e.g. a text file).
It has links to some of the panel reports that led to the founding of NCSA, but the OSTI website has been having intermittent 502s for me this morning.
The original "black proposal" was online on the NCSA website, but seems to have been missed in a website reorg; wayback has it here: https://web.archive.org/web/20161017190452/http://www.ncsa.i... . It's absolutely fascinating reading, over 40 years later.
I read this in high school, but not because it was assigned. At the time I was really into "rare" Queen MP3s, and there's a studio recording of the fast version of "We Will Rock You" where Brian May reads a passage from this book before the music starts. An odd way to be inspired to read a book, but I still think I got a fair bit out of it.
'A studio version was made for John Peel's BBC Radio 1 show at the Maida Vale Studios on 28 October 1977 and first broadcast soon after on 14 November. It includes a separate section that begins with an abridged session version of the original comprising the first verse, chorus and guitar outro - this part has become known as We Will Rock You (slow). Between the two parts there is a brief reading of Hermann Hesse's novel Siddhartha, used in a BBC Radio documentary. This audio was found on the BBC tapes being reused to record the session and was retained by the band.'
That's the one! And all this time I'd been holding on to something I read on a mailing list or website back then, attributing the reading to Brian May "just because he liked that book."
Thanks for sharing it ! Not everyone listen to music only for the beat. It looks more like a success story than an odd one if you ask me. Music must inspires.
I do so love the TI-99/4A. I've been going through some TI-99/4A disk archives from the Chicago Texas Instruments User Group (which streams its meetings every month) and one of my favorite finds so far was this fun demo:
I don't think that it's fair to compare the rendering to what is currently in use in the scientific community, for two main reasons:
The first is that different types of rendering have different uses; typically in scientific visualization this is broken down into essentially "viz for self, viz for peers, viz for others" and oftentimes the most well-used rendering engines are targeted squarely at the first and second categories. The visual language in those categories is qualitatively different than that used for more "outward facing" renderings.
The second reason is that I disagree with your assertion about the quality of the visualization techniques in use within science. There are some truly spectacular visualization engines for cosmology and galaxy formation -- just to pick two examples off the top of my head, the work done by Ralf Kaehler or that by Dylan Nelson. (There are many really good examples, however, and I feel guilty not mentioning more.)
As I said in another, rather terse and unelaborated comment, though, this is really, really impressive work. I think it's important that in praising it, however, we don't discount the work that's been done elsewhere. This need not be zero-sum.
I don’t mean to discount any other work. I have already disclaimed that I don’t work in academia and rely on second-hand feedback from my classmates (in Europe)—for example, the Fortran implementation of Yoshida’s method from N years ago that nobody could modify, or the pressure for publication. Building (or learning) a new rendering engine would be a losing strategy in an academic career, as it is a much more difficult path to getting published. There are far fewer postdoc positions than PhD positions, and rendering skills won’t help in this competition.
Regarding the work of Ralf Kaehler: I have seen his renderings and looked through his articles, but to the best of my knowledge, no source code is publicly available. I don’t consider it fair to count it as something actively used in the field, beyond his lab and affiliated projects.
Disclaimer: that doesn't mean that there are no others, but their availability to researchers is limited to be widely spread.
I skimmed the documentation and didn't see any reference as to whether Markwhen supports dependencies? I.e. MSProject-style make one event dependent on another task ending or starting.
One feature of DOSBox-X I've come to really appreciate when reverse engineering is that you can toggle the debug log on and off. Additionally, it can display the current VGA palette in the main window.
reply