How do I DevOpsify all that as-built stuff?
I’ve got a few practical solutions or low hanging fruit for you all how to address this in a continuous manner.
And ensure that people actually start to embrace and grow from those as-builts —
You know those things people manually built and left sitting as-is in the production faced with occasional performance art on their specialized voodoo there and then?
I personally would like to call it Form of Truth that gives the actual form for all that Source of Truth stuff that everybody keeps talking about.
*Poof* you are now a DevOps witch!
Solutions for this must not be forgotten running somewhere behind a deep dark silent corner alone screaming for the absolute perfect world first.
Nor anyone should be afraid of trying out different solutions either as iterative continuous development with some potential failure with the associated learning are all part of the necessary journey to cultivate the right mindsets.
It all depends…
If you got a simple web app and/or your client is a cool kid startup who already baked all this stuff in and/or where everybody already codes and/or in your little lab perhaps straight from school exercise and/or you bought into the vendor cognitive dissonance kool aid — *how seemingly absolute simple it is* without thinking/taking care of other dependencies
— that’s cool carry on it will all work perfect I promise I’ve been there too ! :)
But after the hangover has settled —
On each domain I have to figure how fast and how far I can push things with the given the constraints of time and cost.
It is much about motivating the forces greater as it is introducing new goodies among other things motivating from the grassroots to actually adopt these things and run with it eventually from their own initiative as self-replicating solution.
Can’t we just blow it all up and re-build? a.k.a Human Capital Problem
Perhaps the most important thing to figure out is there signs of life still?
Too often I’ve come across large organizations when people just being laid off left and right after the organizational/skills development has been neglected over a long period of time and where people have been treated as disposable assets instead of appropriate talent management applied.
Can’t we just apply the delta over top as-is?
That is the end-goal yes but to get there we have to get often in controlled manner as often people are in disagreement or don’t even know what is the Form of Truth as these as-builts typically are handed over like family heirlooms.
When you finally have played long enough time as the referee what the Form of Truth is or should be you might have a big task to evict and address all the “special” cases and such.
This referee controlled alignment to help establish the agreed Form of Truth is important process to make sure we have documented and addressed all the rough/fragile edges before we attempt to put the puzzle together.
Continuous Visibility and Accountability helps
There have been plenty of examples where like-for-like replacement has been blindly done without thoughtful way to manage the technical debt efficiently out in a controlled continuous manner.
My approach aims to make it all visible and people accountable for that technical debt and more importantly give motivation to address it thru automated gap analysis after the Form of Truth has started finally to evolve.
Essentially it comes down to
The cost of change must be less than the cost of not changing.
As-builts are a symptom
Typically associated with organizations who never figured out the configuration management to begin with and the organizations/culture did nothing to cultivate the need for it like those brutalist period buildings show us.
There has been always configuration management opportunities but it was just far more easier to build it and leave it — instead of using templates, package management etc. and culture wasn’t much pushy about this.
How big is the problem depends on how long this has been going on for.
a revolution straight from the beginning?
As I often tend to suspect if we do not raise the bar to the level that everyone codes you need to approach it as such and take what they already know and develop from there instead of introducing a complete revolution with the associated anarchy and fireworks.
Of course if the organizations *poof* replaces all their workforce just like that and suddenly aligns everything associated as well.. like all that technical debt from years ago.. what dream is that until the fireworks begin?
In worst cases I’ve seen whole departments cut and then the work is outsourced to offshore vendor who in turn massively recruits numbers straight from the school without understanding the technical debt/toil derived complexities and imperfections largely caused by the business itself without the appropriate safeguards in place to avoid it.
Tools in their native format
It is often better not to introduce say instead yet another complicated DSL or radical new concepts from whole another domain that require electroshock therapy Zaps in the network of human neural networks that are conditioned to do the same thing the same way they have always done — in their native format which can be translated to configuration management to form the “Form of Truth”.
When you have the “Form of Truth” figured out it is much easier to introduce the “Source of Truth” thinking that is all the rage now but yet forgets the imperfect world at large.
This is because typically Source of Truth is much further alien concept than the Form of Truth — as many of these organizations have had loose attempts to say templatize their stuff but often failing because it was never connected to anything to hold anyone accountable for it.
Like in one organization there was a template which they used mail merge from excel spreadsheets to come up with a configuration — except it was seen as the ideal form not something to honor or keep as authoritative.
What I did was extract the source of truth from that complex as-built topology with my tools and started to model the template in appropriate documentation system forming the Form of Truth both on a documentation system they already knew and in the native configuration format they understood and tried to template without success earlier.
From the combined Source of Truth and Form of Truth that was expected to match with the as-built to be, I used my diff tools to then generate automatic gap reports across the whole topology of as-builts:
- Where these gaps exactly are and
- How much of it
From which both the management and the people themselves at grassroots were then able to hold accountability for both future changes as well as to clean up the toil that now shows up as quantified numbers — to use as a benchmark for KPIs and to motivate future continuous improvement.
From there it naturally depends on how well the culture is adapted to accountability but numbers I would certainly argue makes this job easier?
Target the State in Flux
It’s best to first figure out the typical changes — or — what the current voodoo performance art does as it is more important to figure out the state in flux so you can motivate the people to steer off from the performance art towards DevOpsifying their as-built art projects.
As well as help cleaning up the as-built to be in a state where the configuration management can take over — which can be done with the right automated accountability or numbers management.
Any opportunity where the as-built itself is in flux
One nice example was with where client was replacing their as-built physical infrastructure and what process the people in there liked to call as like-for-like replacement in order to avoid vertical organization accountability as labeling things as “like for like” tended not to get as scrutiny as if it was labeled as a replacement.
Most Low hanging fruit was not in the change itself but more in the preparation, change monitoring and stuff like that to reduce the changes from taking months to overnight.
Too often organizations are focused on the change itself but not the associated difficulties.
In reality it is never anything like-for-like when it comes as-built
In addition the particular type of change itself I was targeting in my example was the typical messy performance art of:
- Manually collecting “Source of Truth” data
- Put the Source of Truth into excel spreadsheets
- Manual change documentation
- Requiring tens of people on their keyboards logging into things
- Checking manually things across all over the place
- Vague change documentation
The above all presented the perfect opportunity to DevOpsify the topology as it was a big change replacing the underlying physical assets.
Native tools examples
As an example here are few things I’ve thrown in when facing adversity on organizations that have needed the first push aligned to the example above.
- Automate the collection of Source of Truth
- Sync inventory with the Source of Truth
- Create the Form of Truth
- Hold accountability from the Form of truth
- Automate change documentation using pattern-variable-templates
- Automate device interaction for the changes and more importantly
- Automate the change monitoring which is typically the low hanging fruit
The above can represent the initial push to get where we want to be and it needs to be treated as evolving continuous development that both raises the barrier as well as cultivates the evolving mindset from the stagnation of accepting how things are.
The way to change the mindsets
Come from positive reinforcement —
Like when people are seeing the positive change from simple tools and simple solutions it is much easier to change the minds who are now more adapted to think change is possible.
And when these are appropriately celebrated to override the LNR-JS.
