If there is such a level of privilege in Twitter's stack, that says a great deal about their technology. Insiders must not be able to act as users except in prescribed ways requiring two-person control, logged and 100% audited. Glass-breaking privilege escalation should set off every pager in the company.
Sorry, but would you mind expanding slightly on how you would implement such a system?
In my understanding once you remove all the layers of abstraction as some point it's a bunch of databases and data stores. Someone has to manage them. Why wouldn't a breach of those users be able to do whatever they want?
And a higher level, someone is writing the code to implement such a stringent access system. Why wouldn't a breach of those users (or a rogue employee) be able to accomplish bad things?
Glad you asked. "There is a database and some guy is the DBA" is a very outdated architecture that can get you passing grades as an undergraduate and that's about all its good for. You should not take as a given that the right to modify datastores falls ultimately upon some individual. It is possible to permanently discard this ability, and organizations should strive for that.
I'm guessing you work/have recently worked at a big tech company (FANG or one of the ~5 other companies of comparable size) and are seriously overestimating how common their best practices are. Unless by "passing grades as an undergraduate" you mean "bonuses and promotions at a majority of the companies that handle your data every day"
G did not really get serious about infrastructure security until after the China hack (and more-so after NSA/Snowden) and didn't really get serious about insider risk until after "gcreep". Still, I don't understand the reluctance of the industry at large to learn the lessons of other people's failures. Why does each company need to separately discover that insider risk cannot be prevented by recruiting, it has to be prevented in code and hardware?
Building a large-scale information system is like building a nuclear power station. There are a million ways to screw it up and only a few recognized right ways. If you ignore the best practices, it will eventually destroy your company and harm your users. Twitter have nuked themselves here. How can they come back from this? It sure looks like an insider risk mitigation system would have been money well spent.
I had a fairly high level of Gmail and Gaia administrator access for a while when I worked there, including the post Snowden era. Resetting the password on an account would indeed trigger an audit event, and I'd be asked what was going on. I could provide any plausible sounding reason and that was sufficient, it wasn't really investigated. And that was the right level of oversight because as far as I know nobody with that kind of access ever abused it by making up a plausible sounding reason.
Stopping bad insiders is really hard. Attempting to do it makes most organisations totally dysfunctional. There is one very famous kind of company that combats bad insiders regularly and with huge quantities of systems - a bank. Investment banks in particular. Whenever you read about 'rogue traders' they inevitably had to do a lot of stuff to disable all the various security systems trying to catch rogue traders.
Institutionally distrusting your own employees can lead to seriously messed up IT systems. It's one of the reasons that bank employees are notoriously unable to access so many ordinary external websites, or services like Slack. It's how you can get "administrators" that can't read the logs of the service they supposedly administer. Encrypted messaging services in particular are poison to an org that's trying to stop employees exfiltrating valuable data. Google can just about do a good job of it because it has an essentially unlimited budget, which it spends on rolling its own tools for absolutely everything and integrating it all into one uber-architecture. From an economics perspective this makes no sense - comparative advantage etc - and thus basically no other company can do it that way. They have to buy or deploy open source tools that use a wide array of threat models and security systems but 95% of them will assume a trusted admin. Then try and hack things on top to restrict what rogue admins can do. It's deeply unpleasant.
Having been in several situations - As Gaia admin, working for big budget low competence IT for a "major" company, and as a shoestring SRE on a household name that's still held together by duct tape in some corners - it weird what is obvious, what is possible, and what level of escalation would be required for what kind of attack. It would have be possible and even trivial for me to impersonate a user at any of the three. At Google, I would have left indelible tracks that would have gotten me fired, see Gcreep (whom, oddly enough, I replaced - I was the next SRE hire at Google Kirkland after he'd been sacked). At the largeco, the tracks would have been indecipherable; nobody would have been able to notice. The logging wasn't there. The ability to analyze what logs they had wasn't there. As a shoestring engineer, I'm pretty sure I would have clear knowledge of who did what if something were discovered, but I would have a significant problem finding it unless something were obviously wrong. I know I can't stop a rogue admin; my team is small enough and needs to react fast enough that we can't spare time for access controls or break-glass, even if they were handed to us on a silver platter.
I'm quite concerned about what that means and what this means, and I'm watching this intently. Probably for nothing; I know this is in the realm of risk we're unprepared for, and can't prepare for. Darned if I don't worry anyway.
Thats not what I meant, sorry. How do you implement such a system? So theres a team to manage the datastores, but that changes nothing that on some level someone somewhere has root passwords and/or filesystem access and/or ability to modify the fleet.
We all know access controls and multiple operators are good, yeah. But at the heart of it is still a bunch of linux machines that have to be managed and deployed to. Which as far as I know has no mechanism for check with operator x before running command from operator 0.
I know nothing about twitter's architecture but it could be:
- at-rest encryption of the datastores with the content encryption key protected by a HSM. A KMS (key management system) would be the interface to retrieve the key, with access control enabled. An even better solution would be to have the HSM cipher/decipher the data directly, thus the encryption key would never leave the HSM (or the encryption key is also ciphered by the HSM). But performance-wise it is not realistic.
- in-transit encryption from the client to the datastore. No end-to-end encryption more likely thus allowing admins who have access to encryption termination hosts (reverse proxy, twitter backend app, datastore,etc) to read (and maybe alter) the data by doing memory dumps
- access control for datastore operations: allowing only the twitter backend and some privileged users to read/write in the datastores, etc.
Doing end-to-end encryption from the client to the datastore with a key per client is possible but it would make the solution very complex to operate and not performant.
The tl;dr is that they use hardware security modules (HSMs) with quorum-based access controls. Any administrative actions such as deploying software or changing the list of authorized operators requires a quorum of operators to sign a command for that action using their respective private keys.
While this system was designed specifically around protecting customers' private keys, you could imagine a similar system around large databases.
Not that either. It feel like the conversation around these things is stuck in the far past. Large-scale organizations can and have driven the number of people with root passwords to zero. "Filesystem access" shouldn't be as easy as you're implying and it also shouldn't be of any use, since everything in the files ought to be separately encrypted with keys that can only be unwrapped by authorized systems.
Even the last thing you said about Linux systems starting processes ... even a minor application of imagination can lead you to think of an init daemon that can enforce the pedigree of every process on the machine.
I don't think this is going anywhere. You just keep dodging the question while acting elitist about a topic it is becoming clear you don't actually know much about..
The software has to get there somehow. The images have to get created somehow. The databases need to stay running somehow. At the end of the day they are machines that need to be managed. Just because you don't have people SSH'ing in and SFTP'ing files around changes nothing about that. And I'm not talking about doing that anyway, or any of the other things you keep telling me I don't understand are bad practice (you're wrong).
Hand waving and mumbling 'old tech, newb' doesn't help in the slightest. I've been writing software with a small side of infrastructure management for 10+ years. Not all of us work at FAANG and magically know how things work on that scale.
> Not that either. It feel like the conversation around these things is stuck in the far past. Large-scale organizations can and have driven the number of people with root passwords to zero. "Filesystem access" shouldn't be as easy as you're implying and it also shouldn't be of any use, since everything in the files ought to be separately encrypted with keys that can only be unwrapped by authorized systems.
OK, what about the people who have physical access?
> even a minor application of imagination can lead you to think of an init daemon that can enforce the pedigree of every process on the machine.
> OK, what about the people who have physical access?
What about them? Nothing about physical presence should lead to userdata access, nor the ability to act as users, if the application-layer security is squared away. In any case, physical security is by far the easiest of these topics to handle. Keeping people out of buildings is a human undertaking with 1000s of years of solid doctrine.
> Who watches the init daemon?
Another important question! If you don't know what's running on your box, you really don't have a security story at all.
One can trade data navigability and a performance hit for opacity of content.
Encrypted rows of data are meaningless to an "admin" that can query to its heart's content but will never be able to decrypt the result set. On the other hand, the layers on top (such as the web-tier that emits the plaintext) may have the keys to decrypt, but lack the privs to run around in the database; from that level, they must pass along the user's credentials to obtain user specific content.
Since people don't search by content on Twitter (afaik) and only 'meta-data' indexes are used (such as hash-tags, follower, following, date) this is entirely doable for something like Twitter.
There is also 'Homomorphic Encryption', but I'm not sure the tech there has reached acceptable performance levels.
> requiring two-person control, logged and 100% audited
That would be good from a security perspective, but it would cost additional training, require more support staff, increase response time between request and resolve, make the system more complex and possible fragile, and take development resources away from profit centers.
Most companies has likely, at best, the same security at their internal support center as their accounting department, and given how common CEO fraud is, it mean social engineering will likely continue to be a major attack vector for a long time.
this assumes two things: that there is a security model that would prevent this attack that they should have implemented, and that alarms _weren't_ set off. Both of those are weak assumptions.
I don't think parent was assuming those measures are implemented. They were saying that they should be implemented and if they are not, it betrays seriously poor security posture at Twitter.
