Saturday, November 14, 2026

How I weaned myself off buttons

And what a rainy afternoon in a Tesla taught me about AI-ready architecture

Up until a year ago, I was a die-hard physical-button purist when it came to operating a car.

Across the vehicles I had owned and driven over the years - an original British Rover (not a Land Rover - the old-school Rover), a Mazda 3, and for the past decade a diesel Hyundai i40 - I loved having real switches and dials under my fingers. Once you build muscle memory for a cabin, you know where every control lives and what it feels like. You can adjust almost anything in the car without taking your eyes off the road for a millisecond.

That preference hardened into dogma about ten years ago when a friend of mine bought a Ford Focus. Ford had just moved most of the centre stack onto a touchscreen, and changing the air-conditioning temperature required a multi-step choreography without any physical fallback: look away from the road, hunt down the climate icon on the screen, wait for the virtual temperature gauge to pop up, and swivel your finger along the scale. On a smartphone sitting on your couch, that is a perfectly fine interface. At 100 km/h on a freeway, it is a terrible way to stay alive.

From that day on, I was preconditioned against touchscreen-heavy cars - and Tesla sat right at the top of my no list. A cabin with a giant tablet bolted to the dash and almost zero physical buttons? - No thank you. That cannot be comfortable, and it cannot possibly work. I was so certain of it that I did not even have the curiosity to sit in one.

Act I: Shopping for an EV (Anything But a Tesla)

About a year ago, thanks to Australia's FBT-exempt novated lease incentive for electric vehicles, I found myself in the market for a new car - and the math meant it had to be an EV.

I ruled Tesla out from the start. For the record, Elon Musk's public political views had nothing to do with it; I do not take sides there, and my filter was purely engineering - if a product is good, I will buy it. I just happened to be convinced that Tesla's product UX was fundamentally broken, and I was not wild about the exterior styling either.

While researching the market, I did notice that about a month earlier Tesla had finally launched Full Self-Driving (Supervised) in Australia. Did that change my mind? - Not in the slightest. The keyword was Supervised. Sitting in the driver's seat required to pay full, undivided attention to the road while not actually driving sounded twice as boring and exhausting as just driving the car myself.

So I narrowed my shortlist down to three contenders with proper dashboards and real buttons: the Kia EV5, the Kia EV9, and the BYD Sealion 7. The reviews were solid - efficiency was not class-leading, sure, but owners generally liked them - and all that was left was to go test-drive them.

Act II: One Rainy Public Holiday, Two Different Centuries

On a grey, overcast Victorian public holiday, I finally had a free day to visit the dealerships. Actually, "nice day off" is a stretch: I had slept poorly the night before, I was exhausted, and I had a nagging headache.

I was driving toward the Kia dealership in the rain when I ran into a busy parking situation right next door - and purely by accident ended up pulling into the Tesla dealership lot just before it. Sitting there with my headache, I figured: well, since I am already parked here, I might as well walk in and cross it off the list.

I had no booking. I walked up to the counter, a staff member asked if I was looking at the cars, and within thirty seconds they offered me an on-the-spot test drive and asked if I wanted to try Full Self-Driving.

"Paperwork?" - They unlocked their phone, opened a form, and handed their phone over to me. I typed my name and driver's licence details and scribbled with my finger. That was it. They walked me outside, authorised the car over the air, spent two minutes showing me how to adjust the steering wheel and mirrors, and closed the door. They did not even get in the car with me. "Take it for 40 minutes, no rush - here is a good route for FSD."

Off I went into the wet traffic. The feel on the road was a bit different - and obviously much better and quieter than my ten-year-old Hyundai i40, and within a couple of minutes I engaged FSD. The car just kept going.

After ten minutes, something strange happened in my head. Look at the state I was in: it was overcast and raining, my head was throbbing, I was running on bad sleep, and I was sitting in a completely unfamiliar car where I had zero muscle memory for any of the controls. And the car was just... driving. It was not tired. It did not have a headache. It was never rushed, never distracted.

I sat there feeling not bored, but deeply grateful. If you have kids, you know the feeling when you are sick and exhausted and a relative or a trusted babysitter walks through the door and takes over for an hour. That was what the car was doing for me. Whenever I am not at my best - when I do not want the sport of driving and just need to get safely from point A to point B - the car has my back.

Why had I gotten it so backwards when I dismissed "Supervised FSD" on paper? - Turns out reading about FSD or watching YouTube videos of it is like reading a manual about sex: technically informative, emotionally useless. You cannot reason your way into how your nervous system reacts when a two-tonne machine smoothly negotiates a rainy Melbourne roundabout on its own while you nurse a headache. It felt like 2007 all over again, the week the original iPhone came out: suddenly there was the iPhone, and then there were "all the other phones".

When I handed the car back forty minutes later, I already knew I was buying a Tesla. Still, the Kia dealership was right next door, so I drove over to keep my original plan honest.

Walking into Kia felt like stepping through a time portal into the early 2000s. The yard was a maze of double- and triple-parked cars. When someone finally freed up, I asked to test-drive an EV9 (they only had the base trim on the lot; if I wanted the GT-Line, that would be a six-month wait).

Before we could go anywhere, we had to do the paperwork. They handed me a physical sheet of paper - one of those forms that has been photocopied off a photocopy 150 times until the text is fat, fuzzy, and tilted three degrees off-axis. I filled it out by hand with a pen and handed it back to the receptionist. What was the very first thing she did? - She laid my handwritten sheet next to her keyboard and started typing every single field into her computer by hand.

Then we waited while someone hunted down the physical key fob. Then we had to un-double-park the EV9. Finally, we pulled out onto the road with the salesperson riding shotgun beside me.

The EV9 drove fine - it was electric, so it was snappy compared to my old diesel Hyundai - but it was just another car. The cabin was a wall of physical buttons I had yet to acquaint myself with. Halfway through the drive, the salesperson asked, "Want to try the heated steering wheel?" Sure, why not. I pressed the button for Level 1. It was not even a cold day, and within three minutes my palms were burning and I swiftly did another hunt for an off button.

Everything worked, yet all of it felt like doing things the old way when there was no longer any reason to. When I finished with the EV9, my headache was still there and my curiosity was gone. I did not even bother driving over to BYD to try the Sealion 7. Neither Kia nor BYD could drive itself, and once you cross that line, the rest of the spec sheet falls off the table.

Act III: Why the Missing Buttons Suddenly Made Sense

A few weeks after picking up my Tesla, I was driving home on a genuinely cold Melbourne evening. A couple of minutes into the trip, I noticed a gentle, pleasant warmth in my hands. Until that moment, I was not even sure the car had a heated steering wheel. The car had quietly turned it on by itself - not at a blistering Level 1 or Level 2, but just warm enough to take the chill off the rim without ever drawing attention to itself.

That was when the whole "buttonless" philosophy clicked for me. I did not need a Level 1 / Level 2 steering-wheel button in the Kia; I needed a continuous closed-loop controller tied to cabin temperature so the button did not have to exist at all.

Why had I hated the idea of Tesla's touchscreen for so many years? - Because I was evaluating it under the assumption that I was the full-time operator of the machine. Tesla designed the entire vehicle around a much bigger product bet: having to drive the car yourself is a temporary legacy state until autonomy is solved.

Once you design backwards from that end-state, two things happen:

  1. When FSD is driving, taking your eyes off the road for two seconds to tap a screen is no longer a safety hazard. And to give Tesla credit where it is due, even when you do drive manually, they did not repeat the Ford Focus mistake: your speed sits right in your peripheral vision at the top-right corner of the screen, the climate temperature targets stay pinned to the bottom bar where muscle memory can find them, and the steering wheel still has two tactile 4-way scroll-wheel joysticks, customisable buttons around your thumbs, and a turn-signal stalk on the column.
  2. In a daily point-A-to-point-B drive, almost every button you used to press was just manual ceremony. I stopped carrying a physical wallet years ago - Australia has been effectively cashless for a long time - so the only two things left in my pockets were my phone and my car keys. Now it is just my phone. I walk up to the car, pull the door handle (Button #1), and sit down. It recognises my phone, loads my profile, and automatically moves the seat, mirrors, steering wheel, and even the motorised air-conditioning vents to my exact positions. I click the seatbelt buckle (Button #2), which tells the car I am ready to go, tap my destination, and let FSD take over. When I arrive, I unbuckle - which automatically shifts the car into Park - step out, and walk five metres away while it locks itself.

The door handle and the seatbelt buckle - those are the only two buttons you need to know. Once you enter the destination, everything else between my driveway and the supermarket is handled by the car itself.

Act IV: The Architecture Punchline - Why Grok Worked and Why Enterprise AI Fails

"Of course," I hear you saying, "what about voice commands? Cars have had voice buttons on the steering wheel for many years!" - Sure, and I hated almost all of them.

Until recently, I barely touched Tesla's built-in voice button outside of opening the glovebox. For starters, English is not my first language, and pre-LLM voice recognition was hopeless with my accent on local Australian names. Asking it to navigate to Coles supermarket (C-O-L-E-S) worked maybe five times out of ten; the rest of the time it heard "Cold" or went looking for "Kohl's". Worse still was my phonebook. Pretty much the only person I ever call over a regular cellular voice call today is my wife, and her contact entry in my phone is saved as Camomile :) - with an old-school colon-and-bracket smiley face at the end. Any pre-LLM integrated voice parser I tried failed to resolve "Call Camomile" against Camomile :).

Then, about half a year ago, Tesla added Grok to the cabin. At first, it was basically ChatGPT circa 2023 bolted into the dashboard: fun to chat with on a long drive, completely unable to touch the car.

With the recent updates, however, Grok crossed the line from chatbot to operator. Now you can hold the mic button (or just say "Hey Grok"), have a normal conversation to research a place, say "take me there", and it pushes the destination straight into navigation. You can ask it to bump up the air-conditioning temperature or call your wife regardless of what punctuation lives in her contact card. Better yet, on a complex piece of machinery whose software mutates every few weeks over the air, Grok acts as an interactive tutor: you can ask "Does the car have feature X? How do I use it? I am stuck, what do I do?" and it will explain the behaviour and pop open the exact Settings menu pane on the screen so you can flip the toggle yourself.

Pause for a second and ask the systems-engineering question: Why was Tesla able to turn a passive LLM into a full cabin co-pilot via a routine over-the-air software update?

Now, I am not in the car-building business, and I know every modern car has plenty of digital buses under the skin. But Tesla's decade-old bet - that the car would eventually drive itself and therefore every single subsystem, right down to the air-conditioning vent vanes and seat heaters, had to be electronically actuated and software-controlled rather than tied to mechanical sliders - handed them the exact foundational platform that agentic AI requires. When the LLM brain finally arrived in 2025-2026, all the actuators and state APIs were already sitting there waiting for it.

And that brings me back to the Kia dealership and its 150th-generation crooked photocopy.

Right now, I watch many companies trying to "adopt AI" the Ford Focus / Kia way: they take legacy business processes held together by manual handoffs, tribal knowledge, and humans re-typing paper forms into desktop terminals, and they bolt an LLM chat widget on top. Then they wonder why their expensive new AI can only summarise meetings instead of actually operating the business.

You cannot bolt an AI agent onto mechanical linkages. Before AI can reason over how your company runs - let alone safely mutate and operate your workflows - you have to do the unglamorous platform work Tesla did ten years ago. That does not mean becoming "AI-first" as a slogan; it means making your business processes clearly documented, machine-readable, and API-first down to the last air vent. Once every state transition and actuator in your organisation is exposed as a clean, software-addressable interface, plugging an LLM on top to orchestrate it is just an over-the-air update.

In the end, I did not really wean myself off buttons - Tesla just turned them into an API, and then they stopped mattering. If your enterprise processes still start with a 150th-generation crooked photocopy, no AI model is going to save you. It will just re-type the paper faster.


P.S. Why my AI-adopter engineer friends still reply with a flat "okay" when I tell them my car drives itself - and why some breakthroughs have to be felt in the driver's seat before your brain believes they exist - is a story for the next post.

Tuesday, March 26, 2024

Rolling velocity of unladen human - the story of inner trust

Working from home has been a new normal for me for quite a while - I've been doing it for the past 7 years and don't see myself longing for daily trips to the office any time soon. But it has its challenges - for me, I need to get out regularly, broadly speaking. Otherwise the subtle sense of unrest begins to accumulate, layer by layer, until nothing seems enjoyable anymore.

For me, the solution lies not in traditional avenues of escape like bars and clubs, but rather in the exhilarating rush of speed. At the age of 33 I obtained my motorcycle licence, relishing the sensation of propelling myself through the air. Yet, motorcycling is not a carefree experience, and even more so as you grow older, have a dear wife, daughter and other responsibilities. I still enjoy it immensely but it's not the kind of experience where you just let go and enjoy the wind. That is, not the thing I would do when I'm tired and have some layers of stress to undo.

During a conversation with my psychologist I confirmed that my inner love for the sensation of wind rushing past my skin is true. Determined to experience this freedom with the need for control, I considered various options. Skydving with an instructor crossed my mind, but then the eureka moment struck: a theme park! I always loved them when I was a kid but was too scared to do the larger rides.

Now, at this stage of my life it looked like a perfect opportunity to both embrace childhood dreams & fears, as well to enjoy the wind!

Living in Melbourne, Gumguya looked like the best option. With my ticket booked, I showed up ready to try my best.

My first conquest was TNT, though I chose a seat farther from the front. Even before we started the climb, adrenaline stirred, coaxing layers of pent-up stress to the surface. And I was getting ready to scream, giving myself permission to scream, as much as I need. It felt like a well-mixed cocktail of "doing the right thing", taking the right care of myself, and being so nervous & scared at the same time. As we hurled downwards, it came time to scream. And I screamed, screamed my lungs out. At one point of the ride, a young lady on my left shouted asking if I'm OK. I could only show her thumbs up.

Once disembarked, with my heart pumping, I had only one thought: "AGAIN!". And on I went, again, and then again, just enjoying myself screaming through the wind.

Then I felt my body and brain are well overstimulated and it's time to absorb. I went for a 30 minute walk around the place until my body stopped feeling all spongy and the heart rate halved back to my normal one.

The next challenge was Project Zero. That scared me a lot. At one pint I got to the front of the line but ultimately baulked. The fear grip was too tight. I could've forced myself but realised I'm here to have fun, and it wasn't fun. Just fear.

I went for another walk, thinking. The whole situation with Project Zero didn't make sense - it's absolutely safe, nothing can happen to me, yet I'm scared, utterly scared. It was irrational - when entering a freeway on my motorbike, I hit cruise control on 100km/h, take my hands of the handlebars and enjoy the ride steering the bike with my body lean - this is objectively a much more scary situation, with danger of death included. Again, irrational.

I thought about astronauts for a moment - they've been truly launched into the void, without any guard rails... Somehow the thought about astronauts lifted me up and gave me courage. I was still scared, but for some reason, courage contained that fear. Renaming Project Zero to Gagarin my head, I did a ride, in the middle seat. Surprisingly it wasn't as scary as my first time TNT, potentially because on Gagarin your body has much more touch points with the sled, whereas with TNT you sit in a kind of a harness.

Then it hit me - my fear is caused by lack of trust!. I didn't trust, so I naturally tried to be in control, of which you have zero when strapped to a moving cart, and then you panic! All I have to do is to trust these contraptions to give me a good ride and have fun. This was the matter of truly letting go! From there it all was just one great ride. I did Gagarin again, on the front seat - and it was nice! I did TNT twice again, on the front row and didn't scream - didn't want to, just leaned on the wind and enjoyed the ride.

The final test to my rediscovery of inner trust was riding Rebel. I just sank into the harness and enjoyed the physics. My jaw was so relaxed I had to pick it up after gravity forces pulled it out during the downswing. Sitting with your legs up in the air at the height of a ten story building is supposed to be scary, but since I trusted nothing could happen to me, I had no reasons for longing to be in control and then be scared & stressed because I obviously couldn't be in control. And after a full day of such an active meditation it was an easy thing to do.

I'm now curious if a theme park will do the trick the next time I feel I need to "get out".

Wednesday, February 7, 2024

Guarding GitHub secrets in your organization

GitHub Actions Secrets security model is a breeze to understand when you are dealing with public, open-source projects. Here, contributors are encouraged to fork your repository and submit changes through a pull request.

But when you're working within a GitHub organization, where collaboration happens on private repositories, the game changes a bit. Forking is usually disabled and contributors are encouraged to commit their proposed changes to a separate branch and then do a pull request to the main branch. To keep things secure, branch protection rules are set for sensitive branches like main to prevent unauthorized push.

Sounds secure, right? But here's the catch: GitHub Action secrets are indeed available when running your workflow from a branch. This is a stark contrast to the fork model for public repos, where secrets are not passed to the runner when a workflow is triggered from a forked repository [1].

People who are new to GitHub orgnizations may be caught off guard here. For instance, if you have a secret, e.g. personal access token, that allowes pushing into a protected branch, you can request this secret into your workflow that runs in a branch and still push your code into the protected branch, effectively bypassing the branch protection rules.

One may argue that if you don't trust your colleagues users to such an extent then you defeinitely have other more important issue to solve (Conway's Law) but let's face it - we all occasionally do silly things innocently - it's part of being human.

But don't worry, there are native solutions to guard against such mishaps.

Let's dive into a practical use case. Suppose I'm a repo admin and would like to periodically run a code generation workflow, say, to update my auto-generated Opentofu configuraiton based on some external data. My main branch is protected, requiring all pull requests to be manually approved - this is, again, to encourage everyone in my organization to contribute by creating branches in my repo, while leaving me with the final say on each pull request to decide if the contribution makes sense.

One approach I could take is to issue a personal access token (PAT), define it as secret in the repo, and then use it in the checkout action to interact with the git repo as yourself:


   steps:
    - name: Checkout
      uses: actions/checkout@v4.1.1
      with:
        token: ${{ secrets.my_personal_access_token }}
  
    - name: Code gen
      run: ./generate_tf.sh
  
    - name: Commit changes if any
      uses: stefanzweifel/git-auto-commit-action@v5.0.0

While this approach works and seems to do what I want, it's severly flawed:

  • Any org member with the write access to the repo can access my PAT, e.g. when running a workflow in a branch, and hence to commit to the main branch.
  • Making things worth, they will get write access to any repository I have write access to, including my private repositories outside the organization in question!
    • Yes, with Fine-grained access tokens I can now limit token access to a particular repo, but as of Feb 2024 that's limited only to repos in my own account, i.e. to have a PAT applicable to repos in your orgnization, I will need to create a legacy token that allows writes everywhere my user has access to.

Bottom line - don't use PATs in organizations unless you are really willing to give access to everything you got on GitHub.

"Of course" I hear you saying, "let's not use my user, let's use a user. E.g. let's create a user called gh-bot, allow it to bypass pull request protection in certain repos, issue a PAT for them and use it?" - Sure, it will work but is similarly problematic, though not as dangerous as with your own PATs:

  • You need to take care of guarding credentials for that user, such as using a password manager, etc. Whoever has acess to these credentials, will have write access to the relevant repos.
  • If you reuse this user over multiple repos, then whoever gets access to the PAT (again, by running a workflow in the branch on one repo) will have access to all those repos as well.
  • You can of course define one such user for every repo but it's wastefull because you'll need to pay for each of them to have them in your org, and credential management gets even worth.

So what should we do? - We need some kind of machine user, a service account in GCP terms or IAM Role in AWS terms. Thankfully there is a such thing - GitHub Apps. Yes, it may sound confusing - we need an identity, not another app to develop; but don't worry, it provides exactly that and you won't need to write any app code. Here is how it works:

  • You create a GitHub app. You don't need to use real Homepage URLs or Callback URL, neither implement webhooks - just enter values that makes sense, context wise. Following my previous example let's call it codegen.myrepo.mycomapany.com.
  • In the app's permissions section, grant the app Read & Write access to Contents.
  • Add the app to the list of entities allowed to bypass pull request protection rules for your repo's main branch.
  • Store App ID and private key in your repo env vars / secrets.
  • Use actions/create-github-app-token to create temporary token to access your github repository.
Here is how your workflow will look like:

    steps:
    - uses: actions/create-github-app-token@v1.7.0
      id: codegen-bot-token
      with:
        app-id: ${{ vars.CODEGEN_APP_ID }}
        private-key: ${{ secrets.CODEGEN_PRIVATE_KEY }}

    - name: Checkout
      uses: actions/checkout@v4.1.1
      with:
        token: ${{ steps.codegen-bot-token.outputs.token }}

    - name: Code gen
      run: ./generate_tf.sh
  
    - name: Commit changes if any
      uses: stefanzweifel/git-auto-commit-action@v5.0.0

This solves the issue of having proper machine users - you can have one app (or more!) per repo and there is neither overhead of managing them nor additional cost incurred.

There is still one issue remaining - any user in your org with the write access to the repo can still have access to the app's private key defined in the repo's secret and hence bypass the main branch protenction rules. Indeed, the blast radius is now very limited to just one branch in one repo, but still, it feels pointless to have branch protection that can be easily (and accidentally) be bypassed by any user in your org.

Thankfully, again, there is a native solution for this - GitHub Actions Environments. (Not to be confused with GitHub Actions Environment Variables. Environments allow fine-grained control to who can access secrets and env vars defined in them.

So instead of storing our app ID / private key in the repo level-secrets, let's:

  • Create a new environment called "codegen"
  • Limit access to that environemnt to the main branch
  • Move our app ID and secret from repolevel into the environment-level

The last touch, is to request access to this environment within the workflow file: Here is how your workflow will look like:


    environment: codegen
    steps:
      ...

With that change in place, re-running your workflow will only work if run on the main. If your colleague will try to run it on any other branch in the repo (with innocous intentions most of the time), they won't be allowed:

That's it! Now our repo access is properly configured!

As you can see, designing GitHub actions workflows with princial of least priviledge mind is not obvious and quite often a well working setup can be surprisingly well open to unintended access. I hope this summary of my research on the subject will save you some time.

References

Tuesday, November 28, 2023

Practical DRY vs code readability

About 13 ago, Django docs formally introduced me to the DRY principal - Don't Repeat Yourself, as in "Don't pollute your project with almost identical pieces of code, but rather refactor them into a shared, reusable code base".

It sounds easy and is indeed something people have doning since at least the Multics days. However trying to enforce this principal on myself was quite a struggle. Whenever I spotted repetetion emerging while writing a code, I felt compelled to refactor it immediately. Needless to say, attempting to build a shared functionality for something still in heavy development is tiresome. The context switch between writing what you want to create and refactoring as you go wore me out heavily.

It took me some time to come terms with the perfectionist in myself that I do not have to DRY, that it's a guide, not a law.

Nowadays, when prototyping, I shamelessly repeat myself. Once I get the core functionality working, I review it and apply the following priciples of my own:

  1. Readability trumps DRY
  2. Repeating something twice is totally fine
  3. If repeating logic appears in thee or more places - I'll refactor it out IF it doesn't sacrifice readbility
  4. If I can't refactor with sacrificing readability - I probably have to redesign my code all together

Kick-start your new project with terraformed GCP-GitHub actions key-less auth

In the past, to gain access to our GCP env inside GitHub actions, we used GitHub secrets to store GCP service account keys. It worked but for me it always felt like walking a thin line. Thankfully now GitHub support OICD tokens and we can setup GCP Workload Identity Federation to grant key-less access for our GitHub actions to our GCP environment.

There are plenty of guides out there on how to do it but it takes some effort to follow them, particularly if you want to terraform everthing - it adds the extra work of bootstrapping terraform configuration itself (using local state to create remote state storage, upload state, switch to impersonalization, etc.). Hence, after repeating this a couple of times I decided to have repository template to save time to me and hopefully you as well.

Here is it: https://github.com/zarmory/gcp-github-federation-terraform

What do you get?

After cloning and configuring this repo, with a couple of commands, you'll get the following:
  • Terraform state bucket created
  • Terraform service account created and permissions assigned
  • GitHub OIDC federation set up
  • Sample GitHub Actions workflows to validate and apply your configuration

All in all just ~100 lines of terraform code, including comments. Basically, just clone, configure and start building.

All of the code is meant to serve as a working example to encourage you hack and modify (rather than highly abstracted resuable module of sorts).

This is merely an annoucement post - if interested, please continue to the repo README for further details.

Thursday, November 16, 2023

Getting GNU Make cloud-ready

The title looks cheeky, but so is the issue.

After release of GNU Make 4.2, the team went quiet for 4 years; however since the COVID times they've been releasing a new minor version about once a year. Surprisingly, upgrade to GNU Make 4.4 caused issues with Google Cloud SDK - the gcloud command.

When working on internal projects, I like to have CLOUDSDK_CORE_PROJECT environment variable populated, but I don't wan't to preset it to a fixed value, because every person on the team have their own playground project, which I want the tool to use as the deployment target. So I came with the following Makefile:


CLOUDSDK_CORE_PROJECT ?= $(shell gcloud config get-value project)
export CLOUDSDK_CORE_PROJECT

release:
	@echo Deploying to project $(CLOUDSDK_CORE_PROJECT)

This way my toolchain will pick user's default project, which usually points to their playground. And if someone wants things done differently they can set CLOUDSDK_CORE_PROJECT explicity, e.g. through .envrc - nice and simple.

This worked very well for years until I upgraded my system and started hitting the following cryptic errors when running make:


$ make
ERROR: (gcloud.config.get-value) The project property is set to the empty string, which is invalid.
To set your project, run:

  $ gcloud config set project PROJECT_ID

or to unset it, run:

  $ gcloud config unset project
ERROR: (gcloud.config.get-value) The project property is set to the empty string, which is invalid.
To set your project, run:

  $ gcloud config set project PROJECT_ID

or to unset it, run:

  $ gcloud config unset project
Deploying to project

After quite a bit of reading and bisecting upgraded packages (which is relatively easy with NixOS) I found that Make 4.4.x is the culprit. Reading through the release notes it was surprised to find a long list of backward incompatability warnings - quite astonishing for such a mature and feature-complete tool like GNU Make. Among them, the following paragraph caught my attention:

Previously makefile variables marked as export were not exported to commands started by the $(shell ...) function. Now, all exported variables are exported to $(shell ...). If this leads to recursion during expansion, then for backward-compatibility the value from the original environment is used. To detect this change search for 'shell-export' in the .FEATURES variable.

Bingo! After that I could quickly reproduce the issue:


$ CLOUDSDK_CORE_PROJECT= gcloud config get-value project
ERROR: (gcloud.config.get-value) The project property is set to the empty string, which is invalid.
To set your project, run:

  $ gcloud config set project PROJECT_ID

or to unset it, run:

  $ gcloud config unset project

So what happens is:

  • CLOUDSDK_CORE_PROJECT is not set, so Make calls the default
  • Since this variable is export'ed, Make makes it available to the shell assigning the empty string a value, which breaks gcloud

The fix is simple, though hacky:


CLOUDSDK_CORE_PROJECT ?= $(shell unset CLOUDSDK_CORE_PROJECT; gcloud config get-value project)
export CLOUDSDK_CORE_PROJECT

release:
	@echo Deploying to project $(CLOUDSDK_CORE_PROJECT)

I.e. if the variable is set, the default will not be called. But if it's not, we clear the empty variable from the subshell environment, thus preventing things from breaking.

Eventually trivial, such small issues can easily eat out several hours of one's time, hence I'm sharing this hopefully useful nugget of knowledge on how to make your Makefile cloud-ready :)

Friday, November 11, 2022

Pointing Mozilla SOPS into the right direction

Mozilla SOPS is a neat way to manage your secrets in git. I've been using it a lot in the last years in various project and so far I'm very happy with it.

Today I stumbled up a problem where sops refused to decode my file:


Failed to get the data key required to decrypt the SOPS file.

Group 0: FAILED
  projects/foo/locations/global/keyRings/foo-keyring/cryptoKeys/foo-global-key: FAILED
    - | Error decrypting key: googleapi: Error 403: Cloud Key
      | Management Service (KMS) API has not been used in project
      | 123xxxxxx before or it is disabled. Enable it by visiting
      | https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=123xxxxxxx
      | then retry. If you enabled this API recently, wait a few
      | minutes for the action to propagate to our systems and
      | retry.
      | Details:
      | [
      |   {
      |     "@type": "type.googleapis.com/google.rpc.Help",
      |     "links": [
      |       {
      |         "description": "Google developers console API
      | activation",
      |         "url":
      | "https://console.developers.google.com/apis/api/cloudkms.googleapis.com/overview?project=123xxxxxxx"
      |       }
      |     ]
      |   },
      |   {
      |     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
      |     "domain": "googleapis.com",
      |     "metadata": {
      |       "consumer": "projects/123xxxxxxx",
      |       "service": "cloudkms.googleapis.com"
      |     },
      |     "reason": "SERVICE_DISABLED"
      |   }
      | ]
      | , accessNotConfigured

Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

That is, sops complained that Google KMS service I use to encrypt/decrypt the keys behind the scenes is diabled in my proejct. Which didn't make sense - after all, I created KMS keys in that project so the service must be enabled. I inspected the project id 123xxxxxxx the error was referring to and was surprised to find out that it belongs to a project bar and not the project foo I was working on (and the one where KMS keys where stored at).

After checking environment variables, KMS key location in the encrypted file I had no other options but to try strace on sops binary to find out was causes sops to go with project bar instead of foo. And bingo - it looked at ~/.config/gcloud/application_default_credentials.json file which has quota_project_id parameter pointing straight to bar.

One easy fix is to run gcloud auth application-default set-quota-project foo. It basically tells Google SDK to use foo as a billing project when calling KMS service (KMS API distinguishes between calling project and resource-owning project as explained here. It works but it's a fragile solution - if you are working on several projects in parallel you need to remember to switch back and forth to the correct project since these particular applicatoin-default settings can not be controlled from environment variables.

What is there was a way to simply tell sops (and others) to use project owning the resource (the KMS key in my case) as a billing project as well? Apparently there is a way:


gcloud auth application-default login --disable-quota-project
...
Credentials saved to file: [~/.config/gcloud/application_default_credentials.json]

These credentials will be used by any library that requests Application Default Credentials (ADC).
WARNING: 
Quota project is disabled. You might receive a "quota exceeded" or "API not enabled" error. Run $ gcloud auth application-default set-quota-project to add a quota project.
And voilĂ  - it works!