The goal behind most "clean" software design in general is to eliminate the possibility of failure via constraints. That's the pattern I've seen over the years. Of course, the map is not the territory - you need to make sure the reachable set within the constraints is actually a subset of the real reachable set. Which may be underspecified or unknown a priori (as if you could've really specified the true reachable set, why didn't you just encode those rules?)
So I'm sympathetic to the criticism, especially since composition of formal methods & analyzing their effects is still very much a hard problem (and not just computationally - philosophically, often, for the reason I listed above).
That being said, I don't know a better solution. Begging the agent with prompts doesn't work. Are you suggesting some kind of mechanistic interpretability, maybe?
> And meanwhile (gestures broadly around everywhere) maybe humans actually aren't doing such an optimal job of running and governing everything important in the world?
The issue with this is that you want to impune, in the grand scheme of things, a small few individuals. And so you want to institute an AI system. Which are controlled by the same individuals (or at least the same class of individuals, with the reach to abuse such a system).
I'll hear you out if AI becomes truly decentralized. Until then, no, this line of rhetoric is just justification for the surveillance state that's to come (to be fair, the surveillance state would pick yet another justification, regardless).
What exactly is the Platonic Representation Hypothesis?
You just don't "learn reality" by getting good at representations. You can learn a data set. You can learn a statistical regularity in things such as human languages. You can analyze the concept spaces of LLM's and compare them numerically. I agree with that.
What the hell does "learning an objective shared reality" mean?
This reminds me of EY saying that a solomonoff inductor would learn all of physics in a few days of a 1920x1080 data stream. Either it's false (because it needs to do empirical testing itself), or it's true, but only if you presuppose the idea that it has a perfect model of all the interactions of the world and can decide between all theories a priori... so then why are we even asking if it's a "perfect learner"? It already has a model for all possible interactions already, there's nothing out of distribution. You might argue, "Well, which model is the correct one?" That's the wrong question already - empirical data is often about learning what you didn't know that you didn't know, not just learning about in-distribution unknowns.
I just get an ick because I associate people talking about this hypothesis as if "LLM's converge on shared objective reality => they are super smart and objective, unlike humans". LLM's can be smart. They can even be smarter than humans. It's also true that empiricism is king.
I developed from a very early age a sort of "always assume the worst about yourself" mentality.
I think part of it was influenced by social media (I was a tween debatelord). Part of it was self improvement (only focus on yourself! get ahead! never blame the enviornment!). Part of it was genuinely depressing things in my life.
As an example, I was obsessed with "finding my passion" at some point. Looking back, I was looking for a way to say, "This thing I'm committed to is way more important than all the other things in my life, so I don't need to go do them". As another example, frequently I would go into epistemic spirals - I was aware of psychoanalysis, so clearly there's capability for deep self delusion. But how do I know the navel gazing isn't self delusion? How do I know framing it as "navel gazing" is not an attempt to cope? And infinite recursion ensues. Another example is constantly feeling like I needed to steelman opponents, and so I would do the utmost research and understand the "best" arguments for the opponent's side before responding.
Incidentally, I think this is why I loved computer science so much - because you often proved worst case guarantees. I had a deep disdain for heuristic solutions.
But this mentality is still bad. Let's take the steelman example. How could steelmanning your opponent possibly be a bad thing? Well, are you actually steelmanning them, or are you trying to find some sort of greater upper bound to their argument, then attacking that... for what? Efficiency? Feeling secure in yourself? Why not actually listen to them? Oh, but surely if they accept premises A, B, C, then D, E, F must follow! Do they, though? Is it possible they could not go down that route, and for valid reasons?
It's still a deep contradiction I work through, since to me personally, all of these things invoke a deep "you are not being remotely rational or moral" gut feeling when I do go down those routes. But I know that I need to sit more in grey zones and just.... live in the grey.
(I still love formal computer science and dislike heuristics. But it's much more balanced now.)
Oh and I should mention, the desire to hear everybody out too. Incidentally, on the first few times I had these types of revalations, of course I would go and completely go extreme in the opposite way.
Curious, which app/ forum/ subreddit/ group were you a tween debatelord on, and in what years? (got a link, so we can see?) To what extent did your formation depend on that crowd and its cultural values?
It's always amusing to see what crimes people demand to have strict liability for, yes. "He posted a wrong location online, of course that'd disrupt the search for the wolf, right to jail, right away".
Once you have centralization, "composition" is not so hard. You get to define all your edge cases, define how you see the real world. Everybody doesn't have their own way of doing things, you have only one way of doing things.
Of course, then comes the extension of the software. People will see the world differently. And we have not algorithmically figured out how domains themselves evolve. The centralization abstraction breaks because people disagree and have different use cases.
I don't see how you get around this fundamental limitation. Are you going to impose yet another secret standard on everybody to get the interoperability you want? If you had full control over the world, yes, things are easy.
I'm not saying this as a diss. I truly do believe centralization works. AWS? Palantir? Building the largest centralized platforms in history and having everybody go through your tooling, when executed carefully, is a dummy effective strategy. In the past, monopolies were effectively this too (though I'd say buying steel is much different than "buying" arbitrary turing-complete services to help deal with a wide variety of semantic issues, and that's what precisely makes the 'monopoly' model break in the 21st century). And hey, at least AWS is a pretty good service, insofar that it makes certain things braindead easy. Is it a "good" service, intrinsically or whatever? I don't know.
No I mean like, centralization is unfortunately the thing that just works.
I work at a company that thinks extremely deeply about interoperability issues and everybody is on the opposite side: it can be said that we were made as a response to xkcd 927, to try and solve the issue.
I think the company is right in that semantic decentralization with interoperability would be a good end goal, but I think just plain darwinism explains the necessity of the opposite.
> Although the council had planned to implement Oracle "out-of-the-box," it created several customizations including a banking reconciliation system that failed to function properly. The council struggled to understand its cash position and was unable to produce auditable accounts. It has spent more than £5 million on manual workaround labor.
Not a great example of a single centralised system. The errors came from trying to write custom reconciliation code between two systems, the ERP and the bank - perfect example of the problems OP raises.
Fair point but AWS is also highly extensible, and i'm not sure about Palantir but i guess it must be too to a point? Maybe it's a classic case of good abstractions vs bad ones
It might just be social. When I use the open source http library, much of the reason I use it is because someone has put in the work of making sure it actually works across a diverse set of software and hardware platforms, catching common dumb off by ones, etc.
Sure, the LLM theoretically can write perfect code. Just like you could theoretically write perfect code. In real life though, maintenance is a huge issue
Sorry this is a bit of a tangent, but I noticed you also released UD quants of ERNIE-Image the same day it released, which as I understand requires generating a bunch of images. I've been working to do something similar with my CLI program ggufy, and was curious of you had any info you could share on the kind of compute you put into that, and if you generate full images or look at latents?
Is quantization a mostly solved pipeline at this point? I thought that architectures were varied and weird enough where you can't just click a button, say "go optimize these weights", and go. I mean new models have new code that they want to operate on, right, so you'd have to analyze the code and insert the quantization at the right places, automatically, then make sure that doesn't degrade perf?
Maybe I just don't understand how quantization works, but I thought quantization was a very nasty problem involving a lot of plumbing
So I'm sympathetic to the criticism, especially since composition of formal methods & analyzing their effects is still very much a hard problem (and not just computationally - philosophically, often, for the reason I listed above).
That being said, I don't know a better solution. Begging the agent with prompts doesn't work. Are you suggesting some kind of mechanistic interpretability, maybe?
reply