While that particular API might be nice, and people and companies should probably push back against the obfuscation, in the end it doesn't really matter. When I hand off between different models I already have the first model prepare a markdown file for the second rather than just importing the entire original thread wholesale, because that's expensive anyhow, and also rather unfocused. They can't get their models to stop generating that sort of checkpoint because that's a fundamental operation necessary for all the harnesses to work anyhow.
The fundamental technology of LLMs and arguably AI in general strongly cuts against that sort of lockin. Handoff is a fundamental capability. There's no option to encrypt the docs or write it in some dialect only one model understands because humans need to understand it to, which stops that whole line dead in its tracks for at least the forseeable future. An AI can already today pick up such pieces, how much more easily will they do it tomorrow?
If they want to lock me in, they're going to need to provide a feature that I need so badly I can't switch and nobody else has. It is hard to see what that would be, other than being a generally better model.
As a heavy pi user - this doesn’t help, for codex. The thinking traces are encrypted and you cannot move that session over to eg your local ds4. It will lack those precious thinking traces that you absolutely need to continue a session productively.
They can be deterministic. We did this at Groq, if you sent a request with exactly the same input token, seed and temperature value you would get precisely the same result every time.
This is harder to do on other architectures that themselves aren't fully deterministic though.
Was the temperature 0? Cause unless I don't understand it right, any non-zero temperature implies probabilistic next token prediction.
You did mention, seed, which I haven't seen available anywhere else (in my limited experience) and it can explain a reproducible inference result, but I feel like this matches the "letter of the law" meaning of deterministic rather than the "spirit of the law".
Using a PRNG and a seed, you can have paeudo-randomness and determinism. The pseudo-randomness is the key piece, it’s random to the extent that it cannot be distinguished from true randomness in polynomial time classically.
Do you mean that you get the exact same result (as word for word) because from Groq's documentation, it says that determinism is best effort, and also it's not guaranteed across model versions : https://console.groq.com/docs/prompting?_sm_vck=tMW2S7k17245...
My understanding is that it's best effort because even if you control the seed/temperature, the model version and hardware specs you still have floating point math that will introduce rounding differences
>LLM responses are certainly not idempotent, as they are not even deterministic.
Isn't that more due to an optimization and not how the LLM itself runs?
Like a MoE LLM run on a single input should give the same output each time. But this is inefficient, as any given token is hitting 1 (or maybe 2 or 3) experts at a time, meaning all the other experts are doing absolutely nothing. So you upgrade it to take in multiple requests. But then any given expert can become a bottleneck, so when too many requests need a given expert, some of them are routed to a second or third best expert instead. Within the context of any single request, this looks like non-determinism, but it is still deterministic when considering the full batch.
For everyday users and everyday use cases, that is enough to treat it as non-deterministic (the harness might also send in unique data like current time which means one can never have the exact same request twice), but when talking about LLMs more theoretically, I think we need to consider they can still be ran deterministically even if that isn't as optimized.
Similar with temperature. 0 means deterministic, but anything higher with a seeded value is deterministic. If anything, temperature is us purposefully adding non-determinism to agents because they were too deterministic.
It is much more subtle than your specific example, which is strictly speaking a bug, though ofc it has been used during pretraining for efficiency purposes. Sglang and miles have been working towards full determinism in open source codebases, so the LLMs can help explain the subtleties encountered in actual projects if you point them to these repos.
In the simplest possible case, a distributed addition of floating point numbers is not deterministic if you don't specify the order of the addition operations. If floats are added in a first-come first-serve fashion (the simplest implementation of a reduction operation) you already lose determinism. These could be activations from multiple experts, but it could also be adding chunks of a matrix operation that uses multiple CUDA cores.
If you manage to achieve determinism in temperature zero, it is possible to extend it to determinism at higher temperatures, because at that point you only have to keep the pseudo-random-number generator state in sync across parallel instances and this problem has been solved.
I appreciate the replies on the determinism point and I’ve learned some new things here. In any case I probably should not have tagged that on, as my main point was to share that the sort of property that parent is talking about (whether true for all LLMs/providers/harnesses or not) is statelessness, not idempotency.
It's a more or less solved problem (e.g. DeepSeek has batch-invariant kernels, vLLM also has reproducible outputs etc), with an obvious caveat that you need to lock your hardware setup, model snapshot, and inference stack if you want any useful reproducibility across longer time periods. It's up to specific providers to actually implement it to any extent, of course commercial providers are incentivized to do the opposite.
The ring sizing has been such a debacle, first with needing to buy (!) their special ring sizing kit because standard sizing wouldn’t be accurate they said, and now they say that the ring sizing kits weren’t true to size and suggest this:
> When in doubt, order a larger size. You can always adjust a larger Index 01 to feel smaller with a foam adhesive or clip but you can’t make it larger!
Pebble never said that anyone "needs" to buy the ring sizing kit. They did suggest that you buy the sizing kit from them for $10, OR 3D print it at home, OR visit a jeweler.
The ring has a button on it that you are going to press often, and you will likely wear it on a different finger than you would normally wear a ring on. That is/was the advantage of the sizing kit IMO: you can try the different sizes and wear the "dummy" ring for a couple of hours or a day and get a better idea of what size and finger will work best for you.
I think calling the ring sizing a "debacle" is being a little over-dramatic. It is a shame the ring kit is off on sizing, but how big of a problem it is, is kind of an unknown right now. It's a limited production product, there will be problems at the start, just like there have been with the Pebble Time 2.
I could be misremembering but I seem to recall them saying that their sizing (the reason you had to buy the special sizing kit) was slightly smaller than normal, and now it turns out the rings are coming out slightly bigger, could it be that all along we should've just used normal US ring sizing?
to be fair, most smart rings (oura, samsung, even the cheap chinese ones on aliexpress) use ring sizing kits. it seems to be standard practice in this space.
Oura Rings (I don't know about the Index) aren't just like other kinds of jewelry. There are protrusions inside the Ring, just like on other wearables like the Apple Watch, for optical sensors that change how it fits. And these smart rings cannot be stretched or modified in any way.
agree, not being able to guarantee or even know, it seems, how your sizing works is a deal breaker and i might have to cancel my order until i see how the first reviews come out :/
Yeah, this annoyed me. I live right next door to a jeweler who could have given me a standard sizing easily, but I bought the kit at their suggestion/mild insistence.
What's worse, I threw it out when I was done, given that it was a PLA sheet with no other apparent use, with it being nonstandard and all. So now I'm not even sure if I need to size up, because I'm not going to buy a second sizing kit to refresh myself on the fit.
My child has an SE, and the only downside we’ve seen is that without a connected iPhone or WiFi network, the battery life is unimaginably bad. That little LTE radio works hard in the school.
I'll never understand why people think this looks human:
What Made This Time Different
This time, I didn't just install FreeBSD.
I created a system for learning and success.
Clear goal: FreeBSD as my daily driver
Daily habit: 10 minutes minimum
Accountability: post the journey on Linkedin
Gee, why not let the agent try FreeBSD for you and do the posting directly
It's a funny video but his stance is quite principled, as he refuses to use any non-free software even if it's horribly inconvenient. Which is the exact opposite of what we have here.
Because those type of people are actually some type of narcissist powerpoint bots. Only interested in their "journey" and putting that next to a picture (of herself probably) climbing a mountain.
Here is what should worry everybody who care about FreeBSD and the "foundation", it seems it is being run into the ground [0]
> The board has said the deficit spending is intentional. They are drawing on reserves to invest more in the project, which makes sense in principle.
> But at roughly the 2024 burn rate, the reserve fund might last about 4 to 5 years.
> One thing that caught my attention is that the EU Cyber Resilience Act starts in September 2026, and the Foundation already has six workstreams running to prepare for it. That kind of work costs money, and right now a lot of it seems to be funded from the same reserves that are shrinking.
This is also why you are hearing thing like FreeBSD "is investing in laptop support" or a KDE installer.
But this is not what is gonna "save" FreeBSD. People get interested and choose FreeBSD because it a great server OS, because of the ZFS support, jails, PF, stability and the coherent system overall.
They would do more for adoption by just making sure most VPS hosting services have it as an option along Linux than wasting infinite money on supporting "laptops".
See also [1], people are getting tired of that level of hypocrisy and waste of money.
I'm trying to remember if anyone complained like this about the Apple Card offers in Apple Wallet. For some reason advertising their credit card is completely fine, but advertising their movie is where people get out the pitchforks? Not defending either, I think both are egregious. I just think it's interesting.
Completely agree! Felt like such a refresh right up until the suggestions to follow on the Musk platform – it’s basically the same as suggesting to follow on Truth Social nowadays, only that Musk has more money than Trump could ever dream off
A more appropriate term is “stateless”. LLM responses are certainly not idempotent, as they are not even deterministic.