Trying to run `terraform validate` with TF_PLUGIN_CACHE_DIR set

If I run terraform providers lock -platform=darwin_amd64 -platform=linux_amd64 -platform=darwin_arm64 I get no changes to my lock file. I include all 3 platforms as I’m on MacBook Pro with M1 chipset, we have others on Intel, and we have Linux and WSL users, and Linux pipelines.

When I then run terraform validate, I get an error.

$ terraform validate
╷
│ Error: registry.terraform.io/hashicorp/archive: the cached package for registry.terraform.io/hashicorp/archive 2.4.2 (in .terraform/providers) does not match any of the checksums recorded in the dependency lock file
│ 
│ 
╵

This command is also run as part of the pre-commit hook and took me a little while to track down to it being Terraform, and not TFLint causing the problem.

From reading various GitHub issues on this, with the most important one being Caching not usable in 0.14.x due to lock file checksums · Issue #27769 · hashicorp/terraform · GitHub, it would seem attempting to use TF_PLUGIN_CACHE_DIR (the idea was to not have every provider installed/copied into every repositories .terraform directory when the terraform init is done) is not a good idea at the moment…

Is there anything that can be done to make this work? Considering the issue has been around a while, it would seem not.

I think that TFLint is also doing something “wrong” as an invalid value is written to the lock file (writing the lock file when NOT specifically upgrading providers feels VERY wrong to me!). Whilst Hashicorp has no support for TFLint, if there is any knowledge/experience that is sharable, it would be nice to know.

Hi @richard.quadling,

The global plugin cache – what you’re configuring with the TF_PLUGIN_CACHE_DIR environment variable – is relevant only to commands that perform provider installation. That includes terraform init (which installs into the working directory cache), but does not include terraform validate.

Commands like terraform validate expect provider installation to already have completed, and thus expect to find all of the locked providers in the working directory’s own provider plugin cache, in .terraform/providers.

This error suggests to me then that you have a hashicorp/archive v2.4.2 package under .terraform/providers that doesn’t match the checksums in the lock file. Since you’ve described using a package that doesn’t really exist – there is no hashicorp/archive release for darwin_arm64 – I assume you’re intending to use packages you built yourself, in which case you have essentially intentionally corrupted your global cache with a modified build of the packages, which has confused Terraform because it expects the cache directory to match what would’ve come from upstream.

If your goal is to maintain your own version of this provider then the most straightforward answer would be to publish your builds to a Terraform registry and install from there. However, if you need Terraform to believe that this provider really is hashicorp/terraform specifically then there’s another option:

  • Place your custom-built provider in one of the implied local mirror directories. Terraform will prefer to use the local mirror instead of the origin registry for any provider it finds in these directories, assuming you haven’t explicitly overridden Terraform’s defaults.
  • Set the version number in your mirror directory to something like 2.4.2+rquadling1 so that it’s clear to Terraform that you intend this to be a separate set of builds from the upstream 2.4.2. Then Terraform won’t try to use the official checksums to validate your local build.
  • Delete the directory you manually placed in the global plugin cache directory. The global plugin cache is supposed to be maintained automatically by Terraform rather than manually by you, and Terraform will automatically link or copy the package from the provider mirror directory into the cache directory when it installs the provider from the local mirror.
  • Probably also delete .terraform/providers since it seems like you have at least one stale package in there which will probably confuse things.

With the above setup I would expect terraform init to detect that the local mirror directory contains a hashicorp/template package and so use the local mirror directory as the authority for which packages are available for that provider. It will then find that the most recent available version of the provider is 2.4.2+rquadling1, and so install that version. (If you run terraform init with the environment variable TF_LOG=trace then it should give you some feedback about where it’s searching and what decisions it is making based on what it found.)

One remaining wrinkle is that terraform providers lock is designed under the assumption that most people want their dependency lock file to contain checksums matching the official provider releases, and so by default it’ll lock the packages from upstream instead of the ones in your filesystem mirror directory. You can override that using the -fs-mirror option:

terraform providers lock -fs-mirror=/path/to/your/mirror/root -platform=darwin_amd64 -platform=linux_amd64 -platform=darwin_arm64 hashicorp/archive

The lock file entries for hashicorp/archive should then be the ones for 2.4.2+rquadling1, instead of the latest upstream release.

Details aside, the main thing to keep in mind here is that Terraform expects a particular (provider source, version number) pair to have a fixed set of package checksums that’s set in stone once the provider is published. Therefore if you want to produce your own builds that intentionally don’t match the upstream you need to both configure Terraform to install from somewhere other than upstream and use a distinct version number (by adding a build metadata suffix) so Terraform doesn’t think you’re being attacked with a maliciously-modified package.

Hi.

I’m NOT building my own provider or compiling one. The issue is only related to the use of the cache.

If I delete the repo’s .terraform/providers and run the terraform lock with all the platforms and then immediately run terraform validate, … all WITHOUT the cache envvar, no errors.

If I reinstate the envvar, delete the contents of the directory pointed to by the envvar, delete the contents of the repo’s .terraform/providers, and repeat the lock and validate … error.

I’m on a MacBook Pro M1 running the latest version of MacOS Sonoma 14.4.1

I did notice that when using the cache, all the files are symlinked.

When NOT using the cache, the files are copied.

When I use the Apparancy app (I think it is part of MacOS), and open the installed file:

~/dt/ecs-mass-cancellation-infrastructure/.terraform/providers/registry.terraform.io/hashicorp/archive/2.4.2/darwin_arm64 $ ll
-rwxr-xr-x@ 1 arrayqueue  staff    14M  8 Apr 14:12 terraform-provider-archive_v2.4.2_x5*

(Terraform has put the file in a Darwin ARM64 path), Apparancy says the file is an Apple Silicon - 64-bit Executable.

This is not anything I’ve built.

Having said all of this, I’m on TF v1.7.5 and it looks like v1.8 has an improvement on this.

What I’d like to have is all the versions in the cache directory for local development as we have a LOT of projects and we have tools to help us work out what projects are out of date with providers and modules (not perfect, just “good enough” at the moment).

What is odd is that the terraform plan / apply has absolutely no problem! It was only when I tried to get the cache working locally did we get this issue.

It doesn’t seem to matter which repo we use. The issue is always there (that COULD be that I’ve concentrated on the same type of repo and they all have Archive … I’ll do some more checks).

But the provider on my machine is one that Terraform has grabbed, rather than one I’ve built. Of course, if Terraform grabbed source, compiled it, and stored the result somehow … well … that WOULD be amazing!!! I know it hasn’t (and now I’ll be doubly amazed when you say “Of course it does that!”).

Hi @richard.quadling,

Sorry for misunderstanding yesterday. On re-read I think I misunderstood hashicorp/archive as hashicorp/template and so was answering under that incorrect assumption. (Folks compiling their own hashicorp/template has been a common cause of problems shaped like this in the past, and so I jumped to an invalid conclusion by reading carelessly.)

Ignoring all of the commentary about local copies of providers, I think my closing remark still applies here:

The main thing to keep in mind here is that Terraform expects a particular (provider source, version number) pair to have a fixed set of package checksums that’s set in stone once the provider is published.

It seems like for some reason your local cache directory (.terraform/providers) is ending up containing a package that disagrees with the checksums you previously calculated, and so Terraform is raising an error. My idea for what might be causing it was invalid, but nonetheless that seems to be where the problem originates.

I think then the big question is: why doesn’t the package under .terraform/providers match the checksums in your lock file? You mentioned that the package in there is a symlink to the global cache, which matches my expectations. That also means that we can assume that the global cache and the local cache are in agreement, and so the disagreement must be between both of the caches and the dependency lock file.


Since I can’t see the files on your computer, I tried to retrace your steps and build a lock file containing entries for the same provider version your error message reports.

I started by creating a Terraform module that consists only of a requirement for that exact provider version, just to make sure I was matching you as closely as possible:

terraform {
  required_providers {
    archive = {
      source  = "hashicorp/archive"
      version = "2.4.2"
    }
  }
}

Then I ran the same locking command you reported in your first comment:

$ terraform providers lock -platform=darwin_amd64 -platform=linux_amd64 -platform=darwin_arm64
- Fetching hashicorp/archive 2.4.2 for darwin_amd64...
- Retrieved hashicorp/archive 2.4.2 for darwin_amd64 (signed by HashiCorp)
- Fetching hashicorp/archive 2.4.2 for linux_amd64...
- Retrieved hashicorp/archive 2.4.2 for linux_amd64 (signed by HashiCorp)
- Fetching hashicorp/archive 2.4.2 for darwin_arm64...
- Retrieved hashicorp/archive 2.4.2 for darwin_arm64 (signed by HashiCorp)
- Obtained hashicorp/archive checksums for darwin_amd64; This was a new provider and the checksums for this platform are now tracked in the lock file
- Obtained hashicorp/archive checksums for linux_amd64; This was a new provider and the checksums for this platform are now tracked in the lock file
- Obtained hashicorp/archive checksums for darwin_arm64; This was a new provider and the checksums for this platform are now tracked in the lock file

Success! Terraform has updated the lock file.

Review the changes in .terraform.lock.hcl and then commit to your
version control system to retain the new checksums.

Notice that my command reported that it was adding an entirely new entry to the lock file. Terraform produces different messages when it’s dealing with a provider that’s already tracked in the lock file, so if yours is different then maybe we can dig deeper into what might behave differently when there’s a lock file entry already present.

For now though, here’s the lock file I was left with:

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/archive" {
  version     = "2.4.2"
  constraints = "2.4.2"
  hashes = [
    "h1:1eOz9vM/55vnQjxk23RhnYga7PZq8n2rGxG+2Vx2s6w=",
    "h1:G4v6F6Lhqlo3EKGBKEK/kJRhNcQiRrhEdUiVpBHKHOA=",
    "h1:WfIjVbYA9s/uN2FwhGoiffT7CLFydy7MT1waFbt9YrY=",
    "zh:08faed7c9f42d82bc3d406d0d9d4971e2d1c2d34eae268ad211b8aca57b7f758",
    "zh:3564112ed2d097d7e0672378044a69b06642c326f6f1584d81c7cdd32ebf3a08",
    "zh:53cd9afd223c15828c1916e68cb728d2be1cbccb9545568d6c2b122d0bac5102",
    "zh:5ae4e41e3a1ce9d40b6458218a85bbde44f21723943982bca4a3b8bb7c103670",
    "zh:5b65499218b315b96e95c5d3463ea6d7c66245b59461217c99eaa1611891cd2c",
    "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
    "zh:7f45b35a8330bebd184c2545a41782ff58240ed6ba947274d9881dd5da44b02e",
    "zh:87e67891033214e55cfead1391d68e6a3bf37993b7607753237e82aa3250bb71",
    "zh:de3590d14037ad81fc5cedf7cfa44614a92452d7b39676289b704a962050bc5e",
    "zh:e7e6f2ea567f2dbb3baa81c6203be69f9cd6aeeb01204fd93e3cf181e099b610",
    "zh:fd24d03c89a7702628c2e5a3c732c0dede56fa75a08da4a1efe17b5f881c88e2",
    "zh:febf4b7b5f3ff2adff0573ef6361f09b6638105111644bdebc0e4f575373935f",
  ]
}

This contains all of the official zh: checksums provided by the provider development team when they published this release – those are checksums of the .zip packages used for distribution, rather than the contents – and it also contains the three h1: checksums that represent the contents of the darwin_amd64, linux_amd64, and darwin_arm64 packages for this provider, calculated locally by the terraform providers lock command after it downloaded them.

Does what I’ve shown above agree with yours? If there are any differences in either the terraform providers lock output or the resulting lock file then that’ll hopefully give us a better thread to tug on.

(I did all of this without a global plugin cache configured, just as a baseline. If we do find that your results differ from mine then I’ll try again with a plugin cache to see if my results will match yours better in that case.)

TL;DR; I couldn’t repeat it in a clean setup. So I must have gotten something wrong somewhere/somehow!

So. Here we go!

$ find . -type f | grep -v idea
./archive.tf
./.terraform-version

.terraform-version:

1.7.5

.achive.tf

terraform {
  required_version = ">= 1.3.0"
  required_providers {
    archive = {
      source  = "hashicorp/archive"
      version = "2.4.2"
    }
  }
}
data "archive_file" "this_file" {
  type        = "zip"
  source_file = "${path.module}/archive.tf"
  output_path = "${path.module}/archive.zip"
}

No lock file and no TF_ envvars:

$ env | grep TF_

Running terraform providers lock -platform=darwin_amd64 -platform=linux_amd64 -platform=darwin_arm64

$ terraform providers lock -platform=darwin_amd64 -platform=linux_amd64 -platform=darwin_arm64
- Fetching hashicorp/archive 2.4.2 for darwin_amd64...
- Retrieved hashicorp/archive 2.4.2 for darwin_amd64 (signed by HashiCorp)
- Fetching hashicorp/archive 2.4.2 for linux_amd64...
- Retrieved hashicorp/archive 2.4.2 for linux_amd64 (signed by HashiCorp)
- Fetching hashicorp/archive 2.4.2 for darwin_arm64...
- Retrieved hashicorp/archive 2.4.2 for darwin_arm64 (signed by HashiCorp)
- Obtained hashicorp/archive checksums for darwin_amd64; This was a new provider and the checksums for this platform are now tracked in the lock file
- Obtained hashicorp/archive checksums for linux_amd64; This was a new provider and the checksums for this platform are now tracked in the lock file
- Obtained hashicorp/archive checksums for darwin_arm64; This was a new provider and the checksums for this platform are now tracked in the lock file

e[1me[32mSuccess!e[0m e[1mTerraform has updated the lock file.e[0me[0m

Review the changes in .terraform.lock.hcl and then commit to your
version control system to retain the new checksums.

.terraform.lock.hcl

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/archive" {
  version     = "2.4.2"
  constraints = "2.4.2"
  hashes = [
    "h1:1eOz9vM/55vnQjxk23RhnYga7PZq8n2rGxG+2Vx2s6w=",
    "h1:G4v6F6Lhqlo3EKGBKEK/kJRhNcQiRrhEdUiVpBHKHOA=",
    "h1:WfIjVbYA9s/uN2FwhGoiffT7CLFydy7MT1waFbt9YrY=",
    "zh:08faed7c9f42d82bc3d406d0d9d4971e2d1c2d34eae268ad211b8aca57b7f758",
    "zh:3564112ed2d097d7e0672378044a69b06642c326f6f1584d81c7cdd32ebf3a08",
    "zh:53cd9afd223c15828c1916e68cb728d2be1cbccb9545568d6c2b122d0bac5102",
    "zh:5ae4e41e3a1ce9d40b6458218a85bbde44f21723943982bca4a3b8bb7c103670",
    "zh:5b65499218b315b96e95c5d3463ea6d7c66245b59461217c99eaa1611891cd2c",
    "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
    "zh:7f45b35a8330bebd184c2545a41782ff58240ed6ba947274d9881dd5da44b02e",
    "zh:87e67891033214e55cfead1391d68e6a3bf37993b7607753237e82aa3250bb71",
    "zh:de3590d14037ad81fc5cedf7cfa44614a92452d7b39676289b704a962050bc5e",
    "zh:e7e6f2ea567f2dbb3baa81c6203be69f9cd6aeeb01204fd93e3cf181e099b610",
    "zh:fd24d03c89a7702628c2e5a3c732c0dede56fa75a08da4a1efe17b5f881c88e2",
    "zh:febf4b7b5f3ff2adff0573ef6361f09b6638105111644bdebc0e4f575373935f",
  ]
}

Run terraform -init

$ terraform init 

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/archive from the dependency lock file
- Installing hashicorp/archive v2.4.2...
- Installed hashicorp/archive v2.4.2 (signed by HashiCorp)

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

The only provider in .terraform/providers:

./.terraform/providers/registry.terraform.io/hashicorp/archive/2.4.2/darwin_arm64:
total 29584
drwxr-xr-x  3 arrayqueue  staff    96B 10 Apr 16:34 ./
drwxr-xr-x  3 arrayqueue  staff    96B 10 Apr 16:34 ../
-rwxr-xr-x  1 arrayqueue  staff    14M 10 Apr 16:34 terraform-provider-archive_v2.4.2_x5*

Run terraform validate:

$ terraform validate
Success! The configuration is valid.

So far so good.

Now removing the lock and providers, setting the envvar, creating an empty directory for the envvar and rerunning.

$ export TF_PLUGIN_CACHE_DIR=$HOME/dev/tf-plugin-cache && mkdir $TF_PLUGIN_CACHE_DIR && ls -al $TF_PLUGIN_CACHE_DIR
total 0
drwxr-xr-x  2 arrayqueue  staff    64B 10 Apr 16:38 ./
drwxr-xr-x  9 arrayqueue  staff   288B 10 Apr 16:38 ../

$ rm -rf .terraform
.terraform/providers/registry.terraform.io/hashicorp/archive/2.4.2/darwin_arm64/terraform-provider-archive_v2.4.2_x5
.terraform/providers/registry.terraform.io/hashicorp/archive/2.4.2/darwin_arm64
.terraform/providers/registry.terraform.io/hashicorp/archive/2.4.2
.terraform/providers/registry.terraform.io/hashicorp/archive
.terraform/providers/registry.terraform.io/hashicorp
.terraform/providers/registry.terraform.io
.terraform/providers
.terraform

$ rm .terraform.lock.hcl 
remove .terraform.lock.hcl? y
.terraform.lock.hcl

$ find . -type f | grep -v idea
./archive.tf
./.terraform-version

All reset.

Locks

$ terraform providers lock -platform=darwin_amd64 -platform=linux_amd64 -platform=darwin_arm64
- Fetching hashicorp/archive 2.4.2 for darwin_amd64...
- Retrieved hashicorp/archive 2.4.2 for darwin_amd64 (signed by HashiCorp)
- Fetching hashicorp/archive 2.4.2 for linux_amd64...
- Retrieved hashicorp/archive 2.4.2 for linux_amd64 (signed by HashiCorp)
- Fetching hashicorp/archive 2.4.2 for darwin_arm64...
- Retrieved hashicorp/archive 2.4.2 for darwin_arm64 (signed by HashiCorp)
- Obtained hashicorp/archive checksums for darwin_amd64; This was a new provider and the checksums for this platform are now tracked in the lock file
- Obtained hashicorp/archive checksums for linux_amd64; This was a new provider and the checksums for this platform are now tracked in the lock file
- Obtained hashicorp/archive checksums for darwin_arm64; This was a new provider and the checksums for this platform are now tracked in the lock file

Success! Terraform has updated the lock file.

Review the changes in .terraform.lock.hcl and then commit to your
version control system to retain the new checksums.

.terraform.lock.hcl

# This file is maintained automatically by "terraform init".
# Manual edits may be lost in future updates.

provider "registry.terraform.io/hashicorp/archive" {
  version     = "2.4.2"
  constraints = "2.4.2"
  hashes = [
    "h1:1eOz9vM/55vnQjxk23RhnYga7PZq8n2rGxG+2Vx2s6w=",
    "h1:G4v6F6Lhqlo3EKGBKEK/kJRhNcQiRrhEdUiVpBHKHOA=",
    "h1:WfIjVbYA9s/uN2FwhGoiffT7CLFydy7MT1waFbt9YrY=",
    "zh:08faed7c9f42d82bc3d406d0d9d4971e2d1c2d34eae268ad211b8aca57b7f758",
    "zh:3564112ed2d097d7e0672378044a69b06642c326f6f1584d81c7cdd32ebf3a08",
    "zh:53cd9afd223c15828c1916e68cb728d2be1cbccb9545568d6c2b122d0bac5102",
    "zh:5ae4e41e3a1ce9d40b6458218a85bbde44f21723943982bca4a3b8bb7c103670",
    "zh:5b65499218b315b96e95c5d3463ea6d7c66245b59461217c99eaa1611891cd2c",
    "zh:78d5eefdd9e494defcb3c68d282b8f96630502cac21d1ea161f53cfe9bb483b3",
    "zh:7f45b35a8330bebd184c2545a41782ff58240ed6ba947274d9881dd5da44b02e",
    "zh:87e67891033214e55cfead1391d68e6a3bf37993b7607753237e82aa3250bb71",
    "zh:de3590d14037ad81fc5cedf7cfa44614a92452d7b39676289b704a962050bc5e",
    "zh:e7e6f2ea567f2dbb3baa81c6203be69f9cd6aeeb01204fd93e3cf181e099b610",
    "zh:fd24d03c89a7702628c2e5a3c732c0dede56fa75a08da4a1efe17b5f881c88e2",
    "zh:febf4b7b5f3ff2adff0573ef6361f09b6638105111644bdebc0e4f575373935f",
  ]
}

Testing terraform validate fails as no terraform init yet run.

$ terraform validate
╷
│ Error: registry.terraform.io/hashicorp/archive: there is no package for registry.terraform.io/hashicorp/archive 2.4.2 cached in .terraform/providers
│ 
│ 
╵

Running terraform init

$ terraform init

Initializing the backend...

Initializing provider plugins...
- Reusing previous version of hashicorp/archive from the dependency lock file
- Installing hashicorp/archive v2.4.2...
- Installed hashicorp/archive v2.4.2 (signed by HashiCorp)

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.

We now have a new file in the cache directory

$ ls -alR $TF_PLUGIN_CACHE_DIR
(snipped out the directory elements)
/Users/arrayqueue/dev/tf-plugin-cache/registry.terraform.io/hashicorp/archive/2.4.2/darwin_arm64:
total 29584
drwxr-xr-x  3 arrayqueue  staff    96B 10 Apr 16:43 ./
drwxr-xr-x  3 arrayqueue  staff    96B 10 Apr 16:43 ../
-rwxr-xr-x  1 arrayqueue  staff    14M 10 Apr 16:43 terraform-provider-archive_v2.4.2_x5*

Local files:

$ find . -type f | grep -v idea
./archive.tf
./.terraform-version
./.terraform.lock.hcl

No provider for a “file”, so check for “links”:

$ find . -type l | grep -v idea
./.terraform/providers/registry.terraform.io/hashicorp/archive/2.4.2/darwin_arm64

$ ls -alR .
(snipped)
./.terraform/providers/registry.terraform.io/hashicorp/archive/2.4.2:
total 0
drwxr-xr-x  3 arrayqueue  staff    96B 10 Apr 16:43 ./
drwxr-xr-x  3 arrayqueue  staff    96B 10 Apr 16:43 ../
lrwxr-xr-x  1 arrayqueue  staff    96B 10 Apr 16:43 darwin_arm64@ -> /Users/arrayqueue/dev/tf-plugin-cache/registry.terraform.io/hashicorp/archive/2.4.2/darwin_arm64

At this stage, the .terraform.lock.hcl file is unmodified. The same 3 h1s.

Running terraform validate:

$ terraform validate
Success! The configuration is valid.

Which is odd. I changed the required version to ~> 2.0. No difference.

So this suggests that there’s something else going on.

Added some extra providers:

terraform {
  required_version = ">= 1.3.0"
  required_providers {
    archive = {
      source  = "hashicorp/archive"
      version = "~> 2.0"
    }
    aws = {
      source  = "hashicorp/aws"
      version = "~> 5"
    }
    cloudflare = {
      source  = "cloudflare/cloudflare"
      version = "~> 4.0"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.0"
    }
  }
}

The cache directory has the extra files (as expected), and the validation passes.

Moving the location of the plugin directory to the usual documented location $HOME/.terraform.d/plugin-cache

$ export TF_PLUGIN_CACHE_DIR=$HOME/.terraform.d/plugin-cache && mkdir $TF_PLUGIN_CACHE_DIR && ll $TF_PLUGIN_CACHE_DIR
total 16
drwxr-xr-x  3 arrayqueue  staff    96B  9 Apr 10:03 ./
drwxr-xr-x  7 arrayqueue  staff   224B  9 Apr 11:24 ../
-rw-r--r--@ 1 arrayqueue  staff   6.0K  9 Apr 10:03 .DS_Store

Do all the things. No change there!

And so now I’M stuck!

And so, I’ll leave this for now.

I’ll slowly add more to my test setup and see if it is something daft like Make getting in the way (we have a LOT of Makefile content to help reduce the amount of typing and Make is sort of everywhere, and so a nice simple way of doing everything everywhere all the time … ish).