I was literally told to set up this new service as quickly as possible and it didn’t need to be correct or best practice because this was just a proof of concept.
Here we are 6 months later and I’m still cleaning up my own mess.
It will be in production for 10 years
In 25 years someone’s going to discover a dust covered raspberry pi hosting the poc service in the back of a network cabinet and unplug it, bringing down the rest of production
Young people will be hired and spend months to learn “how it was done in the old days” (it was just old stuff piled upon other old stuff).
The motto I got from working at an ISP was “There is nothing more permanent then a temporary fix.”
Similarly, I’ve heard, “nothing more permanent than a 3/4 solution”, i.e. it works well enough that fixing other things always ends up taking priority.
You should read the Red hat article about decommissioning a data center. Over the period of 20 years there is a crazy amount of works well enough and no one documented it.
This is why i always push back that if you want it, I’m going to implement it properly, or at least in a way I can change easily later. I’m not going to make my own life difficult on your whim.
Web developers are the worst. Especially the ones that call themselves full stack. </sysadmin rant>
Don’t let the perfect be the enemy of the good. That said, it’s vanishingly rare that I’ve regretted designing something for long term use rather than just hacking it together.
Taking the extra time to document stuff, to add guard rails/error handling, to make a piece of script easier to re-use, to actually plan something rather than building it as you go… almost always a good use of time.
Amen, wizardbeard.
Same with “lift-and-shift now and fix after migration” . Everyone knows nothing will be fixed and the same problems will exist, just in the new environment.
Always implement like you’re going to turn it on tomorrow
Several years ago I had a client who needed to be able to receive files from a client of theirs. This was before Dropbox and the like, so I just went in and set up a quick and dirty FTP server. Worked fine, client got their files, all was good with the world.
A couple weeks later: “hey, we have another client that needs to send us files, can you set them up on the FTP?” Sure, no problem.
Repeat for about the next six months. The thing turned into this huge glob of a mess that miraculously enough still worked.
Finally, I call up my contact there and tell him if he wants this to be a permanent solution we should go in and set it up right, and to do that I’m going to need a bunch of information from him, who needs access to what, who should not be able to access what, etc. He says fine, why don’t you come on over and we’ll hash it all out.
I go on-site, we end up having a 2-hour meeting (billed, of course) where we go over all this stuff, plenty of notes are taken, decisions are made. We wrap up, I tell him all I need now is the list of users and their access needs and I can clean the whole thing up.
I never got the list. The thing just continued to grow and grow into some kind of unearthly abomination. Fortunately, I left that job before the thing imploded completely. Someone else got to untangle that mess.
But somewhere inside of you, it hurts, doesn’t it? One more pile of infrastructure and code that ain’t right, and it could have been different.
And of course, the next guy that comes in thinks the mess is your fault as opposed to an expression of organisational dysfunction.
This is why I put an end date on anything temporary. If you want it longer than that, it’ll have to be fully built out.
For me it is always the emergency script for fix a problem with production that ends up being forgotten about.
And this is why bug bounty hunters get paid.
Even as a PoC, you’ll want a strong foundation to build on. Even as a temporary solution, you want to know the fix won’t fail the next time its under pressure. Even under time constraints, you want a solution that you are confident works when it goes live.
