15·
12 hours agoOur CFO’s social security number, contact info, and just about everything you’d need to impersonate them inside a random shell script that was being passed around like drugs at a party for anyone to use. Oh and it had an API key to our payments processor hard coded into it.
That was the tip of the iceberg of how bad the systems were at the company. All of these are from the same company:
- A fintech based company with no billing team
- An event system that didn’t event
- A permissions system that didn’t administer permissions
- A local cache for authentication sessions. Which means that requests would intermittently fail auth because the session was only on one replica. If you hit any of the other ones, you’d get an unauthenticated error
- A metrics collection system that silently lost 90% of it’s data
- Constant outages due to poorly designed and implemented systems (and lack of metrics… hmmm)
- Everything when I joined was a single gigantic monolith that was so poorly implemented they had to run at least 3 different versions of it in different modes to serve different use cases (why the fuck did you make it a monolith then?!)
- The subscriptions system was something like 20 or 30 database tables. And they were polymorphic. No one could touch the system without it breaking or that person declaring failure, which leads me to …
- A database schema with over 350 tables, many of which were join tables that should have been on the original table (fuck you scala/java for the limitations to the number of fields you can have in a case class). Yes you read that right. Table A joined to table B just to fill in some extra data that was 1:1 with table A. Repeat that a few dozen times
- History tables. Not separate from the original table, but a table that contained the entire history of a given piece of data. The worst example was with those extraneous join tables I just mentioned. If you went and changed a toggle from true to false to true to false, you’d have 4 records in the same table. One for each of those small changes. You’d have to constantly try to figure out what the ‘latest’ version of the data was. Now try joining 5 tables together, all of them in this pattern.
- Scala… I could go on a tirade about how bad scala is but needless to say, how many different error handling mechanisms are there? Scala decided to mix all of them together in a blender and use them all together. Scala is just two white paper languages in a trenchcoat. Never use it in a production system
- A dashboard for “specialists” that was so easy to overwhelm that you could do it by breathing on it due to the LACK of events that it needed
- Passwords stored in plain text (admittedly this was in the systems of the company we acquired while I was there). Doesn’t matter if they were actually <insert algorithm here>, they were visible in a dashboard accessible by employees. Might as well have been plain text
- A payments system that leaked it’s state into a huge part of the rest of the system. The system ended up being bifurcated across two systems, I was brought in to try to clean up some of the mess after only a couple of months. I desperately tried to get some help because I couldn’t do it solo. They ended up giving me the worst engineer I’ve ever worked with in my 15 year career, and I’ve seen some bad engineers. Looking back, I’m reasonably confident he was shoving our codebase into an AI system (before it was approved/secured, so who knows who had access) and not capable of making changes himself. I could make several posts about this system on its own
- I could go on but I’ll cut it off there
A data ingestion service that was processing ~15 billion logs each day that was duplicating each of those logs 2-4 times in memory as a part of the filtering logic. No particular reason nor need to do it. When I profiled the system it was BY FAR the largest hog of CPU and memory.
The engineer who wrote it once argued with me about writing comparisons a == b vs b == a because one was technically more efficient … in a language we weren’t using.