Do these scores actually mean anything? Isn’t the LLM just making up something? If you ran the exact same prompt through 10 times would you get those same scores every single time?
Yes I'd be interested in that answer too - these scores are most likely just generated in an arbitrary way, given how LLMs work. Given how they work in generating text it didn't actually keep a score and add to it each time it found a plus point in the skill as a human might in evaluating something.
At this point I'd discount most advice given by people using LLMs, because most of them don't recognise the inadequacies and failure modes of these machines (like the OP here) and just assume that because output is superficially convincing it is correct and based on something.
Do these skills meaningfully improve performance? Should we even need them when interacting with LLMs?
They aren't arbitrary, as I said earlier I got the LLM to de a detailed analysis first, then summarise. If I was doing this "properly" for something I was doing myself I'd go through the LLM summary point by point and challenge anything I didn't think was right and fix things in the skill where I thought it was correct.
You aren't going to have much success with LLMs if you don't understand that their primary goal is to produce plausible and coherent responses rather than ones that are necessarily correct (although they may be - hopefully).
And yes, Skills *do* make a significant difference to performance, in exactly the same way that well written prompts do - because that's all they really are. If you just throw something at a LLM and tell it "do something with this" it will, but it probably won't be what you want and it will probably be different each time you ask.
It would be interesting to see one of these evals and how it generated the score, to work out whether it is in fact arbitrary or based on some scale of points.
I found the summary above devoid of useful advice, what did you see as useful advice in it?
> if you don't understand that their primary goal is to produce plausible and coherent responses rather than ones that are necessarily correct (although they may be - hopefully).
If you really believe this you should perhaps re-evaluate the trust you appear to place in the conclusions of LLMs, particularly about their own workings and what makes a good skill or prompt for them.
> It would be interesting to see one of these evals and how it generated the score, to work out whether it is in fact arbitrary or based on some scale of points.
So go repeat the exercise yourself. I've already said this was a short-enough-to-post rollup of a much longer LLM assessment of the skills and that while most of the points were fair, some were questionable. If you were doing this "for real" you'd need to assess the full response point-by-point and decide which ones were valid.
> If you really believe this you should perhaps re-evaluate the trust you appear to place in the conclusions of LLMs, particularly about their own workings and what makes a good skill or prompt for them.
What on earth are you on about? The whole point of of the sentence you were replying to was that you can't blindly trust what comes out of them.
I'm saying that your agreement that they produce plausible but sometimes false text is contradicted by the trust you seem to have in their output and self-analysis, which is plausible but unlikely to be correct.
Yes of course there's a risk it may still be incorrect but querying the LLM with the limited facilities it provides for introspection is more likely to have at least some connection with facts than the alternative that some people use, which is to simply guess as to why it produced the output it did.
If you have an alternative approach, please share.
No of course you wouldn't because LLMs are nondeterministic. But the scores would likely be in the same ballpark. The scores I posted are the result of a much more detailed analysis done by the LLM, which was far too long to post. I eyeballed it, most of the points seemed fair so I asked it to summarise and convert into scores.
It's really interesting reading about how these folks view LLMs. Yeah, they're transformative, but I don't know that we're going to be eating ramen in a Neo-Tokyo street bar anytime soon. So much "A.G.I" mentioned in the article.
I find it interesting how a lot of cyberpunk does not really include AI or does not present it in transformative way. There is a lot of mind uploading, implants, corpo fun and overall technology permeating all aspects of life, but often AI itself does not actually play a big role.
Counterexamples that come to mind are Neuromancer (AI driving the plot) and Blade Runner (AI antagonists.)
A compromise thesis might be that in cyberpunk media, AI is at never powerful or motivated to fundamentally reform the worldwide crapsack economic system. They don't abolish corporations, although they might take them over.
Of course, if there was a story about an AI taking over the world into a post-scarcity society, it probably wouldn't be filed under "cyberpunk" either...
Rampant capitalism is kinda genre-defining for Cyberpunk so Cyberpunk without corporations wouldn't really be Cyberpunk. _The Matrix_ only qualifies as Cyberpunk because within the matrix the machines effectively control the capitalist power structures to exert their influence.
Abundance/scarcity isn't really about availability, it's more about access. You can have a cyberpunk story in a "post-scarcity" setting in the sense of availability (due to sci-fi tech) but you can't have it without unequal access to those resources.
Right: I'm implying that the genre definition itself places an upper-bound on how impactful AI is "allowed" to be, which creates a kind of (heh) no-so-anthropic principle, ex:
A: "Why isn't there more AI in cyberpunk media?"
B: "There's a decent amount already, as characters or tools."
A: "But why didn't those authors address its potential to be even bigger?"
B: "Some did, but that makes stories we don't categorize as cyberpunk."
Agreed, which is why The Culture (series) isn't cyberpunk but The Polity (by Neal Asher) kinda skirts the line, in many ways they are similar except resource inequality still exists on a wide/policy scale in the latter.
AIs are in plenty of cyberpunk stories, but your comment did make me think that they are often rather stereotypically “alien entity characters” and not a kind of corporate technology / weapon that is controlled by a specific organization.
Which is a shame, as it seems to me that the overwhelming risk of AI is from the latter scenario, and not as a rogue individual entity.
I think you can look at Star Trek as a fairly grounded example of where current LLMs could go: the ship's computer is not autonomous in any way but it does accept fairly vague instructions and you can apparently vibe-code the holodeck.
AI is one of the core parts of cyberpunk, through androids / humanoid robots. Blade Runner is completely built on the protagonist having to interact with rogue artificial intelligence.
It's because they're really good at the kind of busywork the average white collar job requires. Most people are out there writing documents and making presentations. Only when you use them for actual complexity does the shortfall become clear.
I'm going to write a silly comment here:
For a moment I thought you wrote "... LLMs. Yeah, they're transformative, but I don't know that they're going to be eating ramen in a Neo-Tokyo street bar anytime soon."
I liked that mental image a lot! (I try to maintain being uncertain whether Deckard was a replicant)
I guess that would work until they started auditing your prompts. I suppose you could just have a background process on your workstation just sitting there Clauding away on the actual problem, while you do your development work, and then just throw away the LLM's output.
I’ve contributed a fair amount over the past few months of primarily AI generated content that I mainly just edit for the usual AI tropes and it’s pretty much all still up.
reply