Data Engineers Should Be Held To The Same Standards As Bakers

I shipped a product on time, under budget, that is going to be used by lots of people. No corners cut, no requirements dropped, no fudging the definition of done. It's a data pipeline that ingests images which will be used as input to a computer vision model, and it's going to power sustainability projections going 25 years into the future. Everything is approved, everyone is happy.

When you read that, was there any part of you that was impressed? Did you send a small "Congrats" my way in your head? Or maybe you felt kind of jealous that something that was planned actually came to fruition?

Maybe we should be asking ourselves why this isn't the norm everywhere.

It's all a bit silly

At the core of it, really, it's a little crazy to think or feel any of these things when you consider the context. Someone transferred me money on a regular basis while I used the skills I possess to make something they needed. I simply did what I was paid to do. That's it. There's nothing spectacular about it. That is, until we take a serious look at the state of the software and data industry. Imagine I was a baker telling the same story...

I baked a loaf of bread. It's a sourdough made from a starter that I tended to for a week until it was delicately bubbling. The manager and owner of the bakery inspected the loaf and approved its light, airy quality — they were happy with the texture, too. Something I made was deemed good enough to sell to someone who relied on the bread to be their breakfast. The thing I baked ticked the boxes of a tasty loaf of bread, as planned, and the recipe I used will be adopted by the bakery for the next 25 years. Would delivering baked goods on time be perceived as even a little bit remarkable?

Arguments can be made that building a data pipeline is more complex than baking a loaf of bread. But those arguments hold as much water as the sieve I would use to filter the flour for my sourdough starter. Like software engineering, there is an art and science to baking bread (mind you, that statement should be reversed since bread-making has been around for millenia, but I digress...). And from a guy with Coeliac Disease, I promise you there are a lot of horrible outcomes when baking. The benefit that writing code has over baking is that you can copy-and-paste someone else's solution and run it immediately to figure out if it's any good. Then you can modify, reduce, and enhance your starter code into an elegant and effective product. To do that with sourdough, you need to use a bunch of ingredients and wait a week to see if your thing rises into a fluffy creation of deliciousness or collapses into a brick-like mound of salt and starch. The iteration cycle of the latter is way more expensive, both in terms of time and materials. Now what if I was a hypothetical surgeon instead of a hypothetical baker telling a similar story? I'll spare you the gruesome details of what an unsuccessful abdominal surgery might look like.

So why are we so amazed when a tech project is actually delivered on time? One answer is market comparison: most tech projects are doomed before they start. But it's embarrassingly wrong to define the probability of success of a data pipeline or web app on previous outcomes in this way. Really, why are we so amazed when a product is shipped? I pose this as a rhetorical question rather than one of warm-feelings-and-daisies while we sit through Retros on What Went Well, What Could Have Gone Better, and What Was Out Of Our Control. Regardless of the number of "ceremonies", if a team doesn't ship a single thing, none of them should have been hired in the first place, yet running into teams in this situation is the norm. Instead of looking for catharsis in a post-mortem after a project fails, I subscribe to the idea that we need to raise our expectations of the quality of the things we generate. We spend an enormous amount time in meetings, typing on a computer, and pushing code to production. It's unacceptable to chalk up non-delivery to "we didn't get the scope right".

A pandemic of mediocrity

This isn't a problem that's unique to the tech industry. I used to coordinate programs for persons with special needs in the recreation branch of a local government. I was put on a committee to review safety standards across our operations while being told that it would "greatly enhance your career opportunities". I was nominated to lead a sub-committee of this committee (as you do) that was tasked with developing emergency response procedures and protocols to respond to someone having a seizure. (I just want to pause and note how fucked this was. While I was an accredited allied health professional at the time, it's fucking ridiculous that a bunch of recreationists were put in charge of dictating how anyone should respond to a medical emergency. But, again, I digress...). The two other members of the sub-committee were total incompetents, so the hard-working-do-whatever-you're-told-when-it-comes-to-paid-work habit I've developed over a life of being guilted into blending my self-worth with my career took over. I researched what other municipalities, private recreation and sports clubs, hospitals, and emergency medical services were doing. I read every publicly available report or protocol that I could find. I drafted and re-drafted a 10, 20, and, eventually, 36 page document outlining the evidence-based rationale for the one-page protocol that was proposed to be our new way to respond to seizure-related emergencies. (Again, why the fuck was I responsible for producing any of this? Anway...).

When I presented the document to the committee, the two dudes on the sub-committee nodded along and smiled with pride at what they would be given credit for. I finished my 15 minute slide deck (referred to as "PowerPoint slides" at the time) and was ready for questions, comments, and criticisms. The first thing spoken was from a woman who was the highest in the org chart on the committee. She was paid (notice I did not say 'earned') a very good salary and held a position that affected a lot of recreation services that were provided to a city of 800K people. She expressed how impressed she was that I delivered what was asked of me, was amazed that I completed the task by the agreed upon date even though that should be utterly unremarkable, and commended me for doing what I said I was going to do. Everyone on the committee nodded along in agreement.

This was a pivotal moment in my life. I remember looking at the faces of the people sitting in that meeting, all of whom were 5-15 years my senior. Utterly checked out, the bags under their eyes, the defeated postures, the deflated spirits. I felt like I was staring down the barrel of a gun at what my life was going to be. Later that month, I accepted an offer to do a PhD and finalised plans to move across the world, as far away as I could get from those sad, sad faces. Ten years later, I myself have bags under my eyes, but I attribute them to having two kids under the age of 5 who I love chasing after every day.

As we know, culture is something that's hard to define. Further, the corporate world has contrived and synthesised different ideas of 'culture' that amount to not a whole lot. There is, however, a partial definition or notion that I can get behind: culture is defined by what you let happen. Gary made a sexualised comment about Sarah and no one said anything? A culture of harrassment starts to build. Finn made a questionable change to the analytics dashboard that everyone knew about, but no one corrected, and now the monthly financial roll-up looks better than everyone knows it should be? A culture of dishonesty and fraud begins to form. Everyone knows that Alex doesn't check the code that ChatGPT wrote for them but they're incredibly productive so nobody bothers mentioning it? A nightmare of cutting corners and horrendous quality becomes the norm. This is a deficient philosophy on how to define culture, so it will forever be incomplete, but it gives context to how we end up accepting missed deadlines, shitty products, and wasted resources in the workplace.

The biggest craters created by this flaming asteroid of low expectations are 1) colossal amounts of squandered money and 2) destruction of the human soul. On the former, it's infuriating to constantly witness promises made by smiling assassins that an analytics platform will be automated with an AI tool. And the solution is an over-complicated custom command line tool built by our buddy Alex that's meant to be used by analysts who can't code and do not want to code. What's more, the tool doesn't even work because Alex never bothered to test it within the client's firewall! This scenario plays out over and over and over in the places I've worked. Quite simply, it needs to stop. If you ever hear the term "lift-and-shift", for example, you're being conned into a cloud migration that will cost millions per year and will look and operate exactly like the on-premise infrastructure you already have. There are red flags to look out for, but that conversation's for another day.

The latter consequence of low expectations is the more serious one. It's something that some people may never recover from. The average career of a paramedic is 6-8 years because of the physical and psychological strain the job puts on a person. The career of a physiotherapist is about the same, mainly because they become someone's personal psychologist 8 times a day while treating back pain or a sore knee. The trajectory of a data or software engineer seems to be a bit different: they burn out from the mental stress of working in dishonest and fraudulent cultures that produce unsatisfying work, but the engineer stays in the profession. Only they're a shell of a human. How many developers or programmers do you know who are a little sad all the time? I wonder what caused them to be like that.

What does this mean for the future of tech?

Without tooting my own horn, the organisation I'm currently consulting for is getting way more value than they thought they would when they contracted me. Not only have I delivered a tranche two months ahead of schedule, I've created their first working data model (also known as dimension and fact tables or a Star Schema) for this and other existing projects that, for the first time, enables their analysts to actually compare data across two isolated parts of the business. I've saved them money on contractors by automating manual tasks. The output from the core product will be used to affect legislation of the kind that will change thousands of lives in small, significant ways. I share this not as a come-and-see-how-good-I-look, but as a suggestion that maybe — just maybe — we don't need to accept a project delivered on time as an outstanding achievement that's rarer than blue steak. What kind of culture are we building when we let that happen? What if finishing things on time and to scope were the norm? Honestly, where could we be as a society? Don't get me wrong. I definitely enjoyed a couple of glasses of whiskey when I hit the big green button and my pipeline finished with no errors (even the logs were clean!). We should absolutely celebrate milestones. The celebration is just sweeter when we're on schedule and our clients are happy.

Some thoughts on what worked:

1. How did you deliver more value than expected in your contract?

I asked what the organisation needed and listened. This is a surprisingly rare skill in engineers. Even better, the organisation was honest with the answers to the questions I asked - as a consultant, I actually knew what I needed to fix.

2. How was the delivery so far ahead of schedule?

This may be shocking, but I read the documentation for the tools I used. I know — wild, right? But seriously, this helped avoid a lot of pitfalls (like how AWS provides pre-defined managed policies for CodeDeploy that needed to be pieced together with the private subnets this org uses and authenticate correctly with GitHub Actions).

3. How were you able to prevent scope creep?

Again, ask questions and listen — then ask questions again. People talk about "priorities" like some amorphous state of nirvana, but it's real easy to figure out an organisation's true priorities when you identify the things that are repeated often. With the org's priorities well understood, I was able to bat away unnecessary tasks that were floated my way. "I can definitely modify that banner in the dashboard. Before I shift my focus away from developing the ingestion pipeline, that needs to be finished by the end of Q2, how often is that dashboard used? I just want to be sure that my time is allocated apporpriately."

4. How did you find time to engineer a pipeline by yourself from scratch?

I am incredibly lucky to be part of a wonderful community of people who are smarter and more experienced than I am. I found them online as fellow readers of a blog written by my dear friend and Co-Director, Nik Suresh.

I avoided meetings like the plague, which is way easier to do as a contractor than as an employee, but still possible when you make it clear to everyone that you're working to deliver a high priority product (refer to my rant above on priorities). A lot less Scrum (and by that I mean none) and way more Extreme Programming. Let me be clear: this is a call to management to let your team work — standups that turn into hour long chinwags every day will kill productivity, motivation, and (most importantly) joy.

One thing remains unanswered: How does a guy who's allergic to gluten know so much about making sourdough?

I Googled it.