This is very reminiscent of the "everyone's a Russian bot" era of social media, where everyone would just lob that accusation at people without any real proof.
The article does not put things in context. Raising $7 Billion to continue innovating and serving a frontier model is not that much when you compare that Anthropic and Google are paying $1B per month for X data centre just to cope with inference demand.
Congrats on launch. I have experienced these issues first hand with `Open Finance` a few years ago.
I feel that you'll end up being an automation agency (you mentioned UiPath), companies who have the skills and capacity to build, will not need your service. But those who want the full service, you might fill a gap.
I might have a different take, I’m happy with this price per token so only those who’re using it for value would use for what they want.
There are so many useless cases such as people bragging about their token consumption that has no product and no value add, or those with OpenClaw doing useless automation that could be a Python script.
Agree on the point of that the shortage of tokens is causing a bit more responsible behavior with it comes to AI use, which is not necessarily a bad thing. The scarcity is a bit sad for solo-devs, but there's also hope this may encourage less slop and more thoughtful use of the tools while society adjusts.
There was an interesting discussion with creator of PI how even if LLMs are producing less errors than humans, they are producing them 10x faster and issues can compound a lot faster too. Introducing intentional breaks, even if by necessity, can help with that and not taking shortcuts that can be solved by throwing millions of tokens at any problem.
Sounds like they don’t have a moat at all. It’s like software consultancy with a data centre. And then the article mentions many customers using these models on prem (so data centre is not really a plus).
What’s stopping any country backed startup from fine-tuning small open source models?
Maybe because distilling small models from bigger ones that you control gives you better small models than fine-tuning from bigger models you don't control?
(I am not claiming it is the case, but stating this as an assumption)
their moat is where they are based from and that they are making their own models. they have been before the distillation era in the open-weights model.
their model's efficacy for the mainstream comparisons may not be up to the task, but they are pivoting to their own lane for it. but the scope beyond the local market, it is yet to be seen.
No one in Europe will buy from a random startup, the consultancy part is a MUST to do businesses with big corps, banks, finances, insurances, governs, public administration ...
The "consultancy" is their moat: if there are already in the company they will catch up most of the opportunities despite not having the SOTA model.
In Europe procurement cycles are insane, if you are somehow a "trusted vendor" you get a priority line, otherwise you need a lot of political support or ties with some C-level in a company.
Moreover, a lot of companies don't want to send their data to external providers(unless it's Microsoft, but it's a different story ...)
I think many would assume "not enterprise" or "not datacenter grade" when someone says "Standard GPUs", but maybe that specific phrase have a specific meaning I'm not familiar with.
Edit: I just tried a 4B model on a RTX Pro 6000, getting ~500 tok/s with llama.cpp not even trying to optimize or change anything, just default settings. I'm sure with vLLM it'd be a lot faster already, still before manually tuning configs. I wouldn't call that card "Standard GPU" either FWIW, but it makes the claimed performance numbers feel not as exciting, especially given the hardware they were using.
- model size: 2B is just for this preview (it was faster to implement), our article explains how we expect to support large frontier MoE at 1,000 to 5,000 tokens/s
- reaching 500 tok/s, or even up to ~1,000 tok/s, on a consumer GPU card is possible with existing inference engines like vLLM. But there is a ceiling.
The hard part comes we you try to be faster than that: these frameworks won't scale higher just by adding GPUs or using faster GPUs. There is a "glass ceiling" due to microseconds lost everywhere in the stack (grid syncs, inter-GPU comms, kernel launches, CPU sampling, etc.).
All our work at Kog is about removing these bottlenecks.
Thank you for explaining. Do you think there are still opportunities for stack optimizations to meaningfully speed up inference on single consumer-grade GPUs?
I'm sure there are, and I really hope we can work on consumer-grade GPUs at some point.
It should be possible to apply the same methodology (digging deep into the hardware details to understand all its little characteristics, and rethinking the inference stack around that).
What a lot of use on here are salivating for is the ability to run these on prosumer hardware at home. So we tend to jump to the conclusion that "standard" means "consumer-grade" because that's what we want to see.
Still, very cool work!
The blog makes it clear that "standard" GPU here is in opposition to purpose-built hardware like Cerebras. The selling point is reaching the same order of magnitude in generative speed as those approaches.
reply