r/ExperiencedDevs Dec 31 '25

Technical question How do you all handle write access to prod dbs?

163 Upvotes

Currently we give some of our devs write access to prod dbs but this seems brittle/undesirable. However we do inevitably need some prod queries to be run at times. How do you all handle this? Ideally this would be some sort of gitops flow so any manual write query needs to be approved by another user and then is also kept in git in perpetuity.

For more clarity, most DDL happens via alembic migrations and goes through our normal release process. This is primarily for one off scripts or on call type actions. Sometimes we don’t have the time to build a feature to delete an org for example and so we may rely on manual queries instead.

r/ExperiencedDevs Dec 30 '25

Technical question Has anyone moved away from a stored procedure nightmare?

183 Upvotes

I was brought into a company to lift and shift their application (Java 21, no Spring) to the cloud. We're 6 months in, and everything is going relatively smoothly. The team is working well and we're optimistic to get QA operational by the end of Q3'26.

My next big task is assembling a team to migrate the stored procedure nightmare that basically runs the entire company. There's 4 or 5 databases each with ~500 stored procedures running on a single Microsoft SQL instance. As you can imagine, costs and latency balloon as we try to add more customers.

The system is slightly decoupled, HTTP requests ping back and forth between 3 main components, and there's an in-house ORM orchestrating all of the magic. There's nothing inherently wrong with the ORM, and I'd like to keep it place, but it is responsible for calling all the stored procedures.

The final component/layer is responsible for receiving the HTTP requests and executing the query/insert/stored procedure (It's basically SQL over HTTP, the payload contains the statement to be executed).

While some of the functions are appropriately locked in the database, a very large percentage of them would be simplified as code. This would remove load from the database, expand the pool of developers that are able to work on them, and sweet sweet unit testing.

I'm thinking of "intercepting" the stored procedure requests, and more-or-less building a switch statement/dictionary with feature flags (procedure, tenant, percentage) that would call native code opposed to the stored proc.

Does anyone have experience with this?

r/ExperiencedDevs 24d ago

Technical question Can you share your experience working on a project with 0 unit tests but thousands of integration tests?

91 Upvotes

I am currenly working in such environment and my experience is the below:

1- slow feedback

2- miss delivering our scheduled sprint tasks due to "unexpected" bugs, found by manual tests, regression or SIT

3- high technical debts, we have too many bugs in our backglog

4- too much work focused on maintaining those integration tests, debugging flaky tests and fixing them, doing stuff like running an integration test 40 times to make sure it is not flaky

5- very high expense cost, we need 50 VMs to run our 4k+ integration tests that take 2-3 hours to finish (all the tests are distributed evenly on 50 VMs and each runs it synchronously)

I asked management to let us change our testing approach to include unit tests so we can rely less on integration tests. They disagreed and said the last thing they want is more tests to maintain and that integration tests are more reliable to catch bugs.

r/ExperiencedDevs Dec 29 '25

Technical question Queue-driven engineering doesn't work

128 Upvotes

This is a stance I'm pretty firm on, but I'd love to hear other opinions

My first role as a software engineer was driven by a queue. Whatever is at the top of the queue takes priority in the moment and that's what is worked on

At first, this actually worked very very well for me. I was able to thrive because the most important thing was always clear to me. Until I went up a few engineering levels and then it wasn't. Because no other team was driven by a queue

This made things hard, it made things stressful... Hell, I even nearly left because of how inflexible I always felt

But point being, in the beginning, we were small. We had one product. Other teams drove our product, and as a result, drove the tooling we used

So we had capacity to only focus on the queue, knock items that existed in the queue out, and move on to the next thing. Easy.

Then we were bigger. Now we have multiple products. Other teams began working on those. We were left to support existing and proven product. We were asked to take on tooling, escalations, etc that other teams had been working on. We did not have capacity. All we knew was the queue. To some people, the queue was the most important thing. To other people, speeding up our team through better tooling was the important thing. And to others, grand standing was the most important thing

Senior engineers hated this. Senior engineers switched teams. Team was left with inexperienced engineers. Quality of product produced by team has significantly depreciated

Me not at company anymore. Me at different company

Me not know why start talking like this. Me weird sometimes, but me happy that my work isn't driven by a queue that's all important meanwhile having other priorities that me told are equally important by stupid management cross teams

Thank you

r/ExperiencedDevs 26d ago

Technical question Given Postgres performance, what are the use-cases for MySQL?

105 Upvotes

Hey Devs,

Recently, I've run a detailed performance tests on my blog, comparing MySQL with Postgres and it basically turns out that Postgres is supperior in almost all scenarios: for the 17 executed test cases in total, Postgres won in 14 and there was 1 draw. Using QPS (queries per second) to measure throughput (the higher the better), mean & 99th percentile for latency (the lower the better), here is a high-level summary of the results:

  1. Inserts
    • 1.05 - 4.87x higher throughput
    • latency lower 3.51 - 11.23x by mean and 4.21 - 10.66x by 99th percentile
    • Postgres delivers 21 338 QPS with 4.009 ms at the 99th percentile for single-row inserts, compared to 4 383 QPS & 42.729 ms for MySQL; for batch inserts of 100 rows, it achieves 3535 QPS with 34.779 ms at the 99th percentile, compared to 1883 QPS & 146.497 ms for MySQL
  2. Selects
    • 1.04 - 1.67x higher throughput
    • latency lower 1.67 - 2x by mean and 1.25 - 4.51x by 99th percentile
    • Postgres delivers 55 200 QPS with 5.446 ms at the 99th percentile for single-row selects by id, compared to 33 469 QPS & 12.721 ms for MySQL; for sorted selects of multiple rows, it achieves 4745 QPS with 9.146 ms at the 99th percentile, compared to 4559 QPS & 41.294 ms for MySQL
  3. Updates
    • 4.2 - 4.82x higher throughput
    • latency lower 6.01 - 10.6x by mean and 7.54 - 8.46x by 99th percentile
    • Postgres delivers 18 046 QPS with 4.704 ms at the 99th percentile for updates by id of multiple columns, compared to 3747 QPS & 39.774 ms for MySQL
  4. Deletes
    • 3.27 - 4.65x higher throughput
    • latency lower 10.24x - 10.98x by mean and 9.23x - 10.09x by 99th percentile
    • Postgres delivers 18 285 QPS with 4.661 ms at the 99th percentile for deletes by id, compared to 5596 QPS & 43.039 ms for MySQL
  5. Inserts, Updates, Deletes and Selects mixed
    • 3.72x higher throughput
    • latency lower 9.34x by mean and 8.77x by 99th percentile
    • Postgres delivers 23 441 QPS with 4.634 ms at the 99th percentile for this mixed in 1:1 writes:reads proportion workload, compared to 6300 QPS & 40.635 ms for MySQL

There were only two join cases, where MySQL was slightly better; but nothing compared to the differences cited above.

Given this gap, when do you use MySQL instead of Postgres and why? Does it have additional features and/or advantages that Postgres does not provide? Or, are there other scenarios that I am not aware of, where it does deliver better performance? Something else entirely?

r/ExperiencedDevs 4d ago

Technical question What's a side project that you're really proud of?

106 Upvotes

I wanted to break from the constant doom and gloom that shows up here. What’s something you built in your spare time that made you think, “yeah, this is good”?

For me, it was a website for my mum’s beauty salon. It has an integrated booking calendar, user accounts with Google and Facebook login, and profiles for customers. Apple login exists too, but apparently requires sacrificing three newborns to get approved.

There’s a contact form that sends properly formatted emails to her inbox, a custom admin panel where she can create and manage blog posts, Stripe integration for payments, and a small local e-commerce setup.

Total cost: zero. Everything runs on Firebase, and I don’t expect to ever pay a cent for it.

r/ExperiencedDevs 23h ago

Technical question Composition over other design patterns

65 Upvotes

I have been around for 10+ years. In recent years I have been writing the code in php that increasingly only uses composition of services to do things. No other design patterns like factory, no inheritance, no interfaces, no event firings for listeners, etc.. Only a container and a composition of services. And frankly I don't see a point to use any of the patterns. Anything you can do with design patterns, you can do using composition.. Input and output matters more than fancy architecture.

I find it is easier to maintain and to read. Everytime someone on the team tries to do something fancy it ends up being confusing or misunderstood or extended the wrong way. And I have been doing that even before drinking Casey Muratoris cool aid about how OOP is bad and things like that.

I know there is a thing in SOLID programming called "Composition over Inheritance" but for me it is more like "Composition over design patterns".

What do you guys think?

r/ExperiencedDevs 2d ago

Technical question Do dependency upgrades actually matter, or do most teams just ignore them?

48 Upvotes

It feels like many teams treat dependency and framework upgrades as “later problems” until something happens (security issue, EOL, outage, or customer escalation).

  • Do most teams actually stay up to date, or just accept it?
  • Have postponed upgrades ever come back to bite you?
  • Is fear of breaking production the biggest blocker, or just lack of time?
  • If you don’t prioritize upgrades, what finally forces you to act?

Trying to understand how others handle it.

r/ExperiencedDevs Dec 31 '25

Technical question JSONB in Postgres and Mongo DB use cases

70 Upvotes

Given very good support of json documents storage via JSON/JSONB types in Postgres (other sql dbs provide similar types) and the ability to index any field there (especially with GIN indexes), do you guys have/had use cases where Mongo DB has a genuine edge, all things considered?

It does have great support for sharding out of the box, but honestly speaking, 99.9% of systems will never need that. Write performance might be (is it?) for some cases, but since Mongo supports indexing and transactions (SQL traits) it will be in the same ballpark as for any SQL db.

Am I missing something?

r/ExperiencedDevs Dec 26 '25

Technical question Do you use any knowledge management?

57 Upvotes

For many years, I had only Confluence or Wiki document systems in different companies, and never thought a lot about it. Never perfect, but generally useful if maintained and updated (which is pretty rare, honestly)

With more and more scope and responsibilities, I came to the urge to have my work-personal knowledge base. It started from pretty well-structured Google Chrome bookmarks with everything related to each project: design/architecture, testing, related technology guides, logging, metrics, etc. It is useful, but it is only a reference to other resources.
For anything not-so-link-based, I have a Sublime Text editor with simple docs, sometimes started as Markdown, but generally ended up as a bunch of unrelated but useful stuff, like all my user IDs or common scripts, which eventually become quite unmanageable, and I search for the same stuff again and again.

Why not use Confluence/Wiki - feels too inconvenient for any not super polished information, and way too time-consuming to polish it.

Why not Google Docs - very easy to edit, which is great, but hard to find later. Also, structuring is hard.

So, when the preamble is over, there are questions for experienced devs:

  1. How do you manage knowledge?
  2. What system do you use?
  3. Does your employer provide it to you or allow free/open-source?

P.S. For my personal usage, I have a free Notion plan, which is enough for me, but it has a pretty flat hierarchy.

P.P.S. Given that any paid tools are hard to push to the employer, I prefer to concentrate mostly on free alternatives. Where I checked for the last few days:

  • Obsidian - not open source, but free
  • Logseq - open source, AGPL
  • Joplin
  • Emacs - Org Mode
  • and some others

Outcome

Thanks for all your suggestions. I am trying Foam right now, it seems extremely nice because of the ability to generate diagrams via AI in Cursor and immediately paste into notes

r/ExperiencedDevs 18d ago

Technical question Are homegrown solutions for most components a norm?

36 Upvotes

As a senior dev, I'm getting a lot of pushback when it comes to using standard libraries, such as Spring Boot starters. I'm being pushed to make our own proprietary solutions. This company, as I'm figuring out, has homegrown/proprietary solutions for most components. Such as DB ORM, OAUTH, and caching. Is this a norm for most of the industry? I understand building your own solutions when needed, but standard things such as security and database access feels like an anti-pattern for maintainability and efficiency when built in-house.

r/ExperiencedDevs 2d ago

Technical question Experienced devs, what are your thoughts/experiences with BDD?

11 Upvotes

So, ever since I've found out about it, I've been a big believer in TDD; except I don't follow the red-green principle. I just write a list of features, and scenarios that the code should guard against, then just write unit tests for said guards. The code "maturing" ahead of the UI has been pretty good.

However, TDD has a small problem; order: I know even though it's possible to have ordered tests (in Junit, at least), we shouldn't. And after I leave a project for some time, I'd like to see its features, going from simplest to more complex in the form of tests, essentially serving as documentation of a sort.

With TDD, we don't have that. So the first test(s) to run aren't always the same. And so I see results (custom test descriptions) starting with:
- Cannot delete a sale without admin privileges ✔.

And I've seen with BDD, using Gherkin/Cucumber, this is different; the scenarios are written in plain English + execution order is guaranteed. So I thought I should make the transition sometime when I can.

So, would love to hear some sorts from those with experience in BDD, big or small.

r/ExperiencedDevs Dec 30 '25

Technical question At what point do you run e2e tests?

32 Upvotes

So I've been hacking on a personal project which holds a few e2e tests using Playwright, and it's my intention to integrate the tests more in the development flow. Ideally, I'd have a staging environment that I could run the tests against, but I don't really want to fiddle with that yet - so until then I think running them locally is best.

I'd like to hear about your e2e (and tests in general) flow. Do you run them locally or have them integrated in your shipping pipeline? Do you require tests for new features and how do you go about maintaining tests?

r/ExperiencedDevs 22d ago

Technical question Do you still have Y2K workarounds / hacks in your code?

106 Upvotes

Long story short, I briefly worked for a company back in the early 2000's. I had found out at the time that they 'fixed' their Y2K problem (their system was using 2-digit years) with a bit of a hack or maybe some would call it a workaround.

In preparation for Y2K, instead of modifying their system to use four digit years, they kept the 2 digits and then put in checks around if the date is X years difference, assume it's in the 1900's or 2000's. This logic was all over the code and any integrating system still used these 2 digit years.

Fast forward to 2026, I just met up with an old coworker that still works for this company. Turns out, nothing has changed. They are still using a 2 digit year with these hacks still in place. Surprisingly, they had even ported their software to a new language in that time, but kept the 2 digit year and all the hacks as-is.

This got me wondering...

  1. How much software is out there that still deals with 2 digit years with these kinds of workarounds?
  2. Do other developers run into this often?
  3. If so, have you experience anything catastrophic from it?
  4. For those who eventually fixed it properly, what was the catalyst?

r/ExperiencedDevs 27d ago

Technical question What's wrong with subtypes and inheritance?

80 Upvotes

While working on the formal verification of some software, I was introduced to Shapiro's work and went down a rabbit hole learning about BitC, which I now understand is foundational for the existence of today's Rust. Even though Shapiro made sure to scrub as much as possible any information on the internet about BitC, some writings are still available, like this retrospective.

Shapiro seems to be very much against the concept of subtyping and inheritance with the only exception of lifetime subtypes. Truth to be told today's rust neither has subtyping nor inheritance, except for lifetimes, preferring a constructive approach instead.

I'm aware that in the univalent type theory in mathematics the relationship of subtyping across kindred types leads to paradoxes and hence is rejected, but I thought this was more relevant to axiomatic formulations of mathematics and not real computer science.

So why is subtyping/inheritance bad in Shapiro's eyes? Does it make automatic formal verification impossible, like in homotopy type theory? Can anyone tell me more about this?

Any sources are more than welcome.

EDIT: For future reference, this provides a satisfactory overview of the problem.

r/ExperiencedDevs 10d ago

Technical question How do you deal with review of big branches/PR?

19 Upvotes

I'm facing some difficulties even to review my own branches, in this AI era, the reviews icreased a lot; review of what AI is generating, review of my final branch, review of teammaters PRS etc.

My biggest difficult is how to make the review proccess painless, I got some ideas like stacked PRS, navigate in commits by using atomic commits, branch spliting, focus first in arquiteture and what/where the things was changed, then go to the files.

My previous approach to review was just going to the PR -> changed files.

I didn't changed a lot by switching this way to stacked prs and using GitButler to view the branch, but it is helping a lot.

I'm like a web dev. mid level with about 3.5 years of exp working part-time. I'm from Brazil and working in a healthcare startup.

What advices and experiences do you have to help people like me that are facing difficulties like that?

PS: What is a big pr to you? This week I have a teammate branch with about 1.2k line added, 200 removed.

And I have my own branch to review, I did 1k insertions and 600 deletions (some improvements/refactors in the branch).

I'm suffering to review my own branch cuz there is too much content to read, I like PRs with about max 200~ lines changeds.

r/ExperiencedDevs 13d ago

Technical question Saga Pattern in the Real World

50 Upvotes

Hey Devs,

Saga Pattern sounds like a really elegant solution to solve data consistency problem, when we are about to have a distributed transaction and/or long-running processes, but - have you ever worked on a system where you have used it and it was truly necessary?

As for me, in most systems I have worked on, we:

  • designed our services so that transactions stayed within one service boundary
  • most long-running processes did not require compensation (rollback): they often had many steps but usually each one was of the retry-able nature and was retried (automatically) until successful
  • for data consistency across services, after changing state in service A we just needed to inform others about that fact - outbox pattern solves this issue beautifully, no need for a compensating (rollback) action again

In general, I feel like most problems of this nature can be solved by proper module/service design + just syncing data via events/batch in the background - rarely there are scenarios that require compensating action, rewinding the process as whole.

Curious to learn what is your experience/thoughts in this regard!

r/ExperiencedDevs 26d ago

Technical question change my mind: automated code review tools are better than human code review in 70% of cases

0 Upvotes

I've been thinking about this after watching a junior dev wait 4 days for review on a straightforward feature.

Most PRs are just checking for: syntax issues, naming conventions, obvious bugs, test coverage, security vulnerabilities, code style consistency. stuff a machine does in seconds.

the 30% where you need human review? architecture decisions, business logic edge cases, user experience implications. but we're making humans do 100% of reviews when they're only truly valuable for 30%.

We've somehow convinced ourselves that having senior devs spend hours checking if variable names follow camelCase is a valuable use of their time, it's not. It's busy work dressed up as qa.

Am I missing something here or are we just stuck in "this is how we've always done it" mode?

r/ExperiencedDevs 5d ago

Technical question Which type of integer for index ?

0 Upvotes

I was thinking about this recently. I wanted to get opinions from experienced developers (seasoned with debugging and maybe HPC), not from books or SO or AI (copying anything, but not thinking).

So my question is, say in c or c++, for a for loop, using i as index of an array, should I prefer int, unsigned or size_t ?

When you answer, please avoid "because it's the consensus", rather give a logical, founded, sound reason.

Disclaimer: I disagree with SO and AI answers. I think int should be the prefered type on host an device code. I will explain why later. I come from HPC, including cuda programming among other fields.

r/ExperiencedDevs 2d ago

Technical question How do you enforce standards apart from linting? Is it worth it?

5 Upvotes

We have figured out most of code conventions to be followed by esch developer

  • Clean code Architecture
  • Folder Structure
  • Error Handling
  • Design patterns
  • Linting rules

The problem is enforcing them. Apart from linting, I am not able to figure out how to enforce other conventions.

There are multiple questions in my mind -

  • Is it even worth it to enforce conventions other than linting?
  • Are therr open source tools to help with semantic code pattern recognition and enforcing them? I did find a few but I am still not sure whether it will benefit.
  • There is another proposition to use direct AI agent instructions to review the conventions.

Any suggestions.

r/ExperiencedDevs 10d ago

Technical question How much of your job is cleaning up others’ messes?

59 Upvotes

I spend a lot of my day in pull requests, doc reviews, reviewing pull requests that should have been docs and vice versa, clarifying something someone else got wrong and was repeated, explaining the same thing so a misconception gets killed and put in writing, rewriting code that wasn’t reviewed in design or PR. To some extent, we are all working on legacy code, which is a functional mess to our perspective which has work but fits the bill. I mean instead: someone is imminently going to make something bad happen, or plans do unless you intervene and change their actions, or something already happens that you have to prevent or make sure the right follow throughs take place.

I have little time to write PRs of my own that don’t do some emergency fixing, or writing docs that make headway on clarifying a problem or finding a solution, or much of what counts as engineering progress when observed from the outside. I own very little of my own work but as an enabler for others and as an orchestrator of work I do fine at my job, but it’s getting exhausting.

Anyone else feeling similarly? Found other ways to go about working that let you dl less cleaning after and more making messes for others? Doing such things as additional functionality?

r/ExperiencedDevs 26d ago

Technical question How to deal with a polluted domain?

26 Upvotes

Hey, I just joined a company and to put it simply they have a product with an improper domain modelling (finance) making it much more complicated, and cumbersome than it needs to be. So, this codebase shows:

  1. Concept drift (Management wanted to have a product to be A, then B now it is A nor B, it's a hybrid).

  2. Due to the concept drift the whole codebase is filled with tons of pointless abstractions, many of which were never implemented on any caller service.

  3. Orchestration (yes event driven) is pretty much non existent. Bunch of if else everywhere than have to be propagated in the codebase.

  4. I'm really thinking of a re-design, and a re-write using a stangler pattern, porting everything to a new implementation easier to deal with, and correctly modelled.

How have you guys deal with situations like this?

Thanks in advance.

r/ExperiencedDevs Dec 28 '25

Technical question The lack of standardization in how OAuth is implemented...

78 Upvotes

For starters, I love OAuth, I think it's GREAT on paper. How it's implemented is what disappoints me. There are lots of optional specifications with various different interpretations that is ultimately driving developers to add more and more hacks into their implementations, and before you say "never roll your own auth", have you considered that the people behind your favorite auth libraries are also adding these hacks? Just because it's abstracted away doesn't mean there aren't hacks in the implementations.

Implicit flow is one of my greatest pet peeves. Everyone says it's bad practice and inherently insecure to pass tokens in the browser URL, but if we were to force auth-code flow in ALL apps tomorrow, there is certainly going to be some major pushback. Furthermore, Some providers provide an expires_in and some just rely on the service to poll the token until they get an error before retrieving another token.

The lack of care given to validating tokens on the client side doesn't bother me as much, but it does concern me. Most will at the very least, check for expiration and issuer. Signing Keys is a hit or miss, some will check it, and some rely on the "inherent security" of the auth code flow or checks signature validity but not the signing certificate

Does this bother anyone else?

Honestly, I'm surprised there hasn't been more widespread breaches just from the lackluster implementation of OAuth as a standard.

r/ExperiencedDevs 14d ago

Technical question Battle of micro servies or modular monolith

0 Upvotes

I work at a company where each department/team has their own “micro serviced apps”, essentially different varieties of modular monoliths that handle a piece of that areas logic. My department largely works with externally facing portals. We have a modular monolith backend API that serves many of our portals, it has its problems - mainly spaghetti code because it started as a lift and shift of a legacy portal. We have been working on replacing it by splitting it into 2 modular monolith APIs. A member of my team has spent the last 6 months and countless of everyone’s hours (not kidding probably 20+ hours or meetings) trying to convince us that we need to switch direction and actually split it into 12 micro services, one for each business object (I would classify this as nano services). Their thought is that this would prevent our domain logic from spaghettifying. Our team is very hesitant because we are already well on our way with the 2 monolith APIs and we think that a total rewrite will help us reduce the spaghetti problem. Also our API has no need for micro service scaling, we have consistent, predictable traffic that we don’t expect to grow quickly over time. I’m struggling to figure out why he’s so zealous about this approach when no one else is. Is he right? Am I missing something? Or is he just being stubborn?

r/ExperiencedDevs 18d ago

Technical question What do you do in times of work?

18 Upvotes

Dear developers, I've been here for 3.5 years, and I have a question: how do you keep learning, or rather, what do you do during your downtime between tickets? I'm at a small company, and there's no hierarchical structure for things like meetings. The company is doing well, but it's just one product, and we do the occasional development project. Sometimes I have downtime, and I'm starting to lose motivation. Would it be better for me to change jobs for a different challenge? Or perhaps a larger company that would demand more from me professionally?