> And for anyone at Twitter who was depending on the network of tweets being a Directed Acyclic Graph, I'm so terribly sorry.
I love the idea that there's someone out there with code that resolves retweet chains recursively, who's about to be in for a great head scratcher of a bug.
It’d be a simple check that anything referenced has to have a lower ID (and hence time stamp).
I find this bug more interesting:
> Also, it seems like Twitter doesn't actually care about the username and just resolves URLs based on the tweet ID. I'm sure lots of people already knew that but it's new to me.
They’re not validating the parent directory matches the actual tweet. I wonder if that’s an actual bug or intentional to allow for handle renaming to not break existing links.
That seems incredibly brittle due to making assumptions about the ID format that twitter has no obligations to keep. It just so happens that if your logic had been if(newId > olderId) you would have survived their new format (due to the fact that the timestamp leads the integer) but that'd be a win based on pure luck. The example the engineer came up with was seven years old so it had no way of foreseeing the ID format change.
It's a good point but I would suspect that the timestamp being prefix is no accident. If your IDs were sortable in the past it's probably a good a idea to keep them sortable. And since tweets more often than not refer to tweets in the same time range or are paginated together with tweets in the same time range (roughly) having time as a prefix has other advantages
This bug actually has some security implications because you can make it seem as though an account has tweeted something when really it was another one. A casual observer might not notice the discrepancy between the name before and after the click.
>what's really amazing is that twitter programmers thought about this edge case and made sure the tweet would not display itself
Twitter Engineer
>We didn't think of this edge case. Someone did this about 7 years ago and the recursive hydration would make a tweet service crash by simply loading the tweet in a browser. It took a principal engineer an entire day of wading through heap dumps to figure out what was happening.
TIL: debugging via memory dumps is a Principal Engineer level skill.
Anyone here actually do this? I read about it in Release It and it sounds by far like the closest thing there is to a super power when it comes to solving production incidents. I've never actually seen anyone do it though.
Recently saw a video on this technique from Dotnet Conf. Piqued my curiosity again, and now this. I've really gotta learn this.
The fact the principal engineer was doing this does not mean doing this is a principal engineer skill. There's lots of software engineers who can deal with coredumps which is pretty much the same idea.
I have done it once successfully in 10 years (.NET dev). Would recommend having any other kind of logging or instrumentation in place so you don't have to do it. It's still worth learning WinDbg and sosclr.
In my company, we used to have a plugin for our bug tracker to automatically analyze .NET core dumps with WinDbg (if they were attached to a bug) and extract some useful information. We used to do this relatively often, for a shipped product, not a live service, especially if we found memory leaks.
Would you say something like that is worth to set up?
I noticed EC2 now has an API to get memory dumps. Theoretically you could automate collecting memory dumps when an unhealthy instance is pulled out of a load balancer. Then some automated analysis could happen, and allow further manual analysis.
Not sure how much it cost, but it was definitely helpful - even the fact that it was obvious which team needed to take a look first based on the objects that had leaked often made it worth it.
I remember spending quality time with coredumps and gdb back in 2012/2013, when a prototype supercar dashboard we were building crashed on certain CSS animations.[ß]
The call chain went through GTKWebkit, Wayland and all the way to Pango and Cairo. Getting that part untangled took a long afternoon. Figuring out the root cause was another two full days.
The topmost parts of the stack above could be dealt with breakpoints, but even with pango/cairo libs from a debug build it was painful. The failing function could only be single-stepped, trying to place breakpoints inside it would not work. In the end it was an unhandled divide-by-zero deep inside the rendering library.
WTF? If you already have the infrastructure to coredump, they are without a doubt the most convenient way to debug. A stacktrace does not even begin to compare. It is like limiting yourself to printf-debugging in the presence of gdb.
Actually, it exactly is! Now I'm not sure if you were /s or not.
For the code that implements basic state and invariant checks (ie ships with asserts compiled in), crashes are usually exceedingly rare and limited to one of these checks failing. Debugging them requires a stack trace and, optionally, some context related to the check itself. If the program dumps this info on crash, the fix can typically be made in less time it takes to retrieve/receive the coredump and start looking at it. If it can't be fixed this way, then it's to the coredump we go.
On the other hand if the code is prone to segfaulting on a whim, requiring dissecting its state to trace the cause down, then, yeah, it's a coredump case too. But a code like that shouldn't be running in production to begin with.
Sure, if by miraculous chance you happen to have printf'd exactly the state you required to figure out the assert/crash, "you don't need gdb". You could also find -- by divine inspiration -- what went wrong just by looking at the line number where the assert failed. But it's still WTF-y to argue that therefore, an actual {,post-mortem} debugger is "a last resort tool".
I did this in my second year as a professional coder and it took me a while (a week? a week and a half?) to understand what to do and what I was seeing. I would prefer never to have to do it again.
I actually just came across this account recently and was tempted to make some sort of reply bot on whether they were prime or not. I was also trying to find some of the "funnier" numbers to see if they had a disproportionate number of likes (certainly not to like them myself...) but gave up after realising they had almost 50k tweets.
So this brings up a question I've long had, but didn't want the distraction of researching: Is there an easy way to get to some point in a user's timeline? e.g. first tweet, or November 1, 2020?
Recursion isn't the problem. Not keeping track of seen tweets is the problem. Recursion can be used to detect cycles and traverse a cyclic graph in a way that doesn't blow up.
It’s a lot easier to just have a depth limit on such non-cyclic graphs than keep an in-memory list of previously seen nodes. its interesting for sure! but a much rarer edge-case imo
When doing recursion (Postgres recursive CTE) I keep a path on the latest edges and check to see new edges aren't already visited, so same nodes can appear in multiple branches but not on the same branch. Works flawlessly.
Irritation would probably better in terms of not blowing up your memory requirements. Just keep hashes of all visited nodes and a stack or queue of to-be-visited nodes and loop until you have no more to-be-visited nodes.
I love the idea that there's someone out there with code that resolves retweet chains recursively, who's about to be in for a great head scratcher of a bug.