ARC's harness is just straight up broken. No serious harness removes reasoning context between each step. Not only does this significantly lower performance over all reasoning LLMs, but it also increase cost as you destroy the cache on every turn. Tossing the oldest entry when context fills up instead of using compaction is equally bad with the same issues.
And that's not a joke, I made the mistake of installing this on my Windows machine just to test it out quickly last week.
It created 2 new users and then assigned new NTFS permissions for every single file under my user directory to them. This of course wrecked havoc, ssh refused to work, several applications refusing to start and a ton of permission errors. It did this without even a warning in the background and it also does not undo any of it when you uninstall it.
It took around 3 hours to fix it by updating ~10 million NTFS permissions for every single file under my user directory.
I would love to see your prompt(s) and how you managed to make it do this.
I’ve been using Codex in full access mode constantly for the past couple months on a Pro plan and I haven’t had a single incident like this. I’ve used it across macOS, Linux, and Windows as well, so my usage isn’t limited to a specific setup either.
I never sent any prompts to it, it did all of this during the installation. Here is one related GitHub issue I found for it https://github.com/openai/codex/issues/12343 Looks like they don't intend to change this.
Your earlier post is misleading, since "it" sounds like the model, not the installer. (of course if the latter was created by the former, its basically the same complaint, but I suspect what you are talking about was a deliberate design decision by a human)
The problem doesn't exist when granting codex full access. Codex makes you choose upon first launch to "Set up default sandbox" or to "Use non-admin sandbox". The default option creates two new users and messes with NTFS permissions.
> I’ve been using Codex in full access mode constantly for the past couple months on a Pro plan and I haven’t had a single incident like this. I’ve used it across macOS,
That is not how it works: the "but it works for me" is a logical fallacy.
Haven't you followed a bit all the posts about models, for example, escaping their environments?
The "but it works for me" is a random data point: an anecdote.
That there are people / companies where models ran wild and destroyed files / messed up projects is a fact.
When there are documented cases of stuff gone wrong, people must find ways to protect their data.
We literally see posts frontpage, daily, about AI sandboxing and we regularly see posts about AI escapes or AI SNAFUs.
And you come and post "show me your prompt, for it works on my machine"...
If you’re doing that anyway you might as well install a hypervisor and layer your OS on top. That way you can snapshot before your LLM with root fucks everything up again.
I've been using GPT through opencode for quite some time on linux. I have had a great time with getting it to help me through computer use and it has been a game changer for me.
For instance recently, media downloads on my NAS became quite a bit slower. I asked it to investigate and it quickly got back to me saying the issue was the ethernet link had been downgraded to 100Mbps from 1Gbps. It even went through journalctl to tell me when this had happened. To fix it I just had to plug an ethernet cable out and back in. Now this is the kind of thing, earlier I would have to spend an evening on. But given an open system like Linux, a well made safe model just makes it so much more accessible than having to RTFM for one off things I am sure to forget in a month.
I dont really let it run loose through my systems, and keep an eye on the thinking traces it puts out and the permissions it asks for. But now I dont have to slog through manuals and deal with gruff people on the internet. I can choose to do so after the fact, depending on how much free time and curiosity I have.
We're not that far off. But it will not just be for AI, but all software. Mobile OSes has already normalized the expectation that you don't control what can run on your own devices.
Chrome and Firefox uses two different scaling algorithms that is probably contributing a lot more to this difference. Chrome is more blurry in general while Firefox is sharper but has slightly more ringing artifacts. Personally I prefer the Firefox version.
That's why you should prefer LPIPS to PSNR. LPIPS is a learned metric that uses subjective human input.
That said, it's limited due to the training set, iirc. So we still use PSNR regularly — it's an easily-understood metric with widely-understood limitations, which is in practice not too bad. Devil you know, etc.
Ironically when we did compression quality evaluations, me and my colleague found that PSNR way better matches our perception of quality, than LPIPS or FLIP. Our assumption is that LPIPS is biased towards quality artifacts that is not representative of compression quality artifacts.
It depends on the target audience. Many people prefer the oversharpened content. I always find it incredible how so many people never change the oversharpened and oversaturated default settings on TVs.
I do prefer the firefox look too! I'd be curious to hear your take on what could be the root cause of this. I'm far from a image rendering expert. But I remember when inspecting the edges and curves of the images in a DCT visualizer they were all in the AC coefficients which is what led me to the conclusion of the post.
I'm far from an image rendering expert myself, but I don't think there is a single root cause here. I don't doubt the partial decoding would affect the final result just as the scaling algorithm itself, so in the end it's going to be a combination of it all.
The reason I believe the scaling algorithm is contributing more in your particular example, is because it looks almost exactly like a classic B/W comparison between a sharp ringing heavy algorithm and a more blurry one.
It would be interesting to see how your example would look if you used colors instead, where the ringing artifacts become a lot more obvious. Like this one: https://twinlens.app/compare?share=8fb890a5820e
Thanks for these! Indeed I understand your point now. I'll add a little note on the post later today.
Another interesting bit is that It wasn't so easy to get an image showing the issue a clear as we had it. My hunch is that moiré patterns might have a role in this, as most images where the the issue showed, had some kind of grid/repetition. I suppose some images might be more affected by the IDCT and other more affected by the scaling algorithm, making it more or less visible. It's a mix like you said!
I wonder if this is a gamma correction related. JPEG encodes directly in a non linear color encoding (full-range YCbCr), so the partial decoding may be effectively scaling without gamma correction.
Why is it replacing true/false with T/F? true/false is already 1 token in all tokenizer I've seen. Even worse is replacing null with ∅. ∅ is a special unicode symbol that takes up 2 tokens compared to the 1 token for null...
Brand new GitHub account, brand new HN account. I stay far away from projects like this these days, they can easily be malicious. GitHub needs some kind of indicator for projects authored by tenured developers with a real identity.
Maybe they want to be tokenizer agnostic? Then they would need to go by character count, right? Despite these inconsistencies, has anyone actually verified their promise? 350 vs. 10.000 tokens would still be very valuable even if they mess up some edge cases
Agreed. This is what happens when you confuse token counts with byte counts.
A trivial test through tiktoken [1] (though technically you really have to match the tokenizer to the specific LLM) would have shown them that ∅ was a poor choice.
Even from the perspective of learned training data, you can probably just intuit that from a frequency standpoint alone the empty-set symbol ∅ can’t possibly have appeared that often outside of things like set theory and logic.
On Windows over allocating virtual memory (commit charge) is still bad. Unlike on Linux, once the total commit charge hits your physical RAM limit + SWAP limit, Windows will start crashing your applications and refuse to allocate more virtual memory, even if that memory has not been touched yet.
This is also why it is very important to have plenty of SWAP space on Windows, even if you have 64 GiB+ of memory. Because applications love to over allocate commit charge.
I've been using this DeepSeek model the whole day today after building with 5.6 Luna extensively over the last week and I would disagree, at least for Rust + OpenGL.
DeepSeek just spend almost 2 hours trying to figure out why terrain textures were not working. It tried everything over and over again, it even had reference code for meshes on how to setup the rendering with materials, and it could just not do it.
I finally gave up and gave it to GPT-5.6 Luna instead, and figure out in a single prompt after 20 seconds, that the terrain mesh was being initialized with None in the material slot.
Other tasks it has managed to figure out at least, but it is significantly slower than GPT-5.6 Luna and it requires a lot more iterations.
That's roughly my experience. Luna is extremely efficient and at higher levels of reasoning and longer running tasks more capable.
Reading the DS reasoning is wild, it's constantly going in circles. The most minor lack of clarity in your prompt and it will spend ages going back and forth on what you meant. It reasons 5x longer than the preview which makes it really slow now as well. We did a lot of work to nudge it to be decisive and improve our evaluation setup to there's more clarity, and it helped but only marginally.
Ours is a full-stack app one shot test so it includes backend, frontend, design, and QA/testing. It's graded by Opus xhigh and Sol xhigh and the grades are averaged.
DS4 preview would finish in 20 minutes flat on high reasoning and grades 6/10. Luna high gets 9/10 in about 30 minutes. DS4-final is crazy - at high thinking it's taking over an hour and getting ~8 but only had one successful run as I got tired of waiting so long after many early abort/retries trying to debug why thinking was so long. The lowest thinking still takes over 45 minutes, and with thinking off it actually is finally closer to preview in time but actually get's a much more varying result anywhere from incomplete to 6 it seems.
Costs per run DS4 is best but not actually by a lot as it's spending 10x the tokens with all the reasoning and mistakes. It's a very brute force model and I really preferred preview in many ways for how predictably fast it was.
Side note, Spark 1.2 is a nice model for this test, best in frontend design and fastest to get results together, though not nearly as efficient as Luna. Grok scores similarly to Spark but at like $50/run vs the contributor Spark costing $1.50.
Edit 2: btw it tests a team of agents working together in a special harness and stack. So 20 minutes is for 8 agents essentially. That said everything was built around DS as it was the cheapest to iterate against so even with that advantage the new one struggles.
One of our sites that don't use any kind of tracking or cookies other then a site preference cookie, got so many complaints that we had to put up a cookie banner that simply says "we do not use any tracking cookies"...
Users complained that we were non-compliant with EU regulations due to the missing cookie banner.
Because of the legal wording of these support tickets, the support team would often raise them to dev or legal as well, and since this is a non-revenue driving part of our business (the reason we don't track in the first place) we decided to just put up a cookie banner explaining why we don't need a cookie banner.
reply