Creating Resources in Terraform Testing Experiment

dgem · September 14, 2022, 8:25am

Hi,
Very interested in the terraform test experiment and thought I’d try it our with a k8s iam pod role module we have.
I can do simple outputs, but wanted to use the module in combination with the kubernetes provider to deploy a job using the pod role (e.g. to describe a KMS key or access a secure bucket).
Although the tests seem to work, I don’t see any resources being created on the k8s cluster!

Here is a snippet of my test:

resource "random_string" "pod_role" {
  length  = 5
  lower   = true
  upper   = false
  special = false
  numeric = true
}

module "main" {
  source = "../.."

  # this modules doesn't have defaults for most input vars
  pod_role            = "tf-test-pod-role-${random_string.pod_role.result}"
  issuer_host_path    = "/platform-eks/${var.cluster_name}/IssuerHostPath"
  cluster_namespace   = "cipy-tests"
  serviceaccount_name = "terraform-test"
  statement_list      = [{ "effect" : "Allow", "actions" : ["kms:DescribeKey", "kms:ListKeys"], "resources" : ["*"] }]
}

resource "kubernetes_service_account_v1" "list_keys" {
  metadata {
    name        = "terraform-test"
    namespace   = "cipy-tests"
    annotations = { "eks.amazonaws.com/role-arn" = module.main.pod_role_arn }
  }
}

resource "kubernetes_job_v1" "list_keys" {
  metadata {
    name      = "tf-test-pod-role-${random_string.pod_role.result}"
    namespace = "cipy-tests"
  }
  spec {
    template {
      metadata {
      }
      spec {
        container {
          name    = "amazon"
          image   = "aws-cli"
          command = ["kms", "list-keys"]
        }
        restart_policy       = "Never"
        service_account_name = "terraform-test"
      }
    }
    backoff_limit = 2
  }
  wait_for_completion = true
  timeouts {
    create = "2m"
    update = "2m"
  }
}

resource "test_assertions" "pod_role" {
  depends_on = [
    kubernetes_job_v1.list_keys,
    kubernetes_service_account_v1.list_keys
  ]
  component = "pod_role"

  equal "pod_role" {
    description = "pod role name"
    got         = module.main.pod_role
    want        = "tf-test-pod-role-${random_string.pod_role.result}"
  }

  check "pod_role_arn" {
    description = "role arn contains correct account"
    condition   = can(regex(var.provider_aws_account, module.main.pod_role_arn))
  }
}

resource "test_assertions" "aws_eks_cluster" {
  depends_on = [
    kubernetes_job_v1.list_keys,
    kubernetes_service_account_v1.list_keys
  ]
  component = "kubernetes"

  equal "name" {
    description = "check found the right cluster"
    got         = data.aws_eks_cluster.example.name
    want        = "sandbox-v2"
  }

  equal "version" {
    description = "check running suppoted version"
    got         = data.aws_eks_cluster.example.version
    want        = var.supported_k8s_version
  }

  equal "active" {
    description = "check cluster is active"
    got         = data.aws_eks_cluster.example.status
    want        = "ACTIVE"
  }
}

resource "test_assertions" "kubernetes_service_account" {
  depends_on = [
    kubernetes_job_v1.list_keys,
    kubernetes_service_account_v1.list_keys
  ]
  component = "kubernetes"

  check "uid" {
    description = "check uid assigned"
    condition   = can(regex("^[0-9a-fA-F]{8}\\b-[0-9a-fA-F]{4}\\b-[0-9a-fA-F]{4}\\b-[0-9a-fA-F]{4}\\b-[0-9a-fA-F]{12}$", kubernetes_service_account_v1.list_keys.id))
  }

  equal "data" {
    description = "deliberate failure"
    want        = []
    got         = data.kubernetes_resource.list_keys_sa.metadata
  }
}

Note the “deliberate failure” doesn’t fail and that’s why I’m wondering if the k8s provider is really being invoked in the way I’d expect…

Is terraform test capable of creating real resources, or are they somehow “mocked”?

apparentlymart · September 14, 2022, 3:40pm

Hi @dgem,

The terraform test command is approximately automating the following steps for each of your directories and reporting a failure if any of them produce errors:

terraform init to install the dependencies
terraform apply to try creating everything
terraform destroy to try destroying everything

One way to gather more info here would be to switch into the directory for this test scenario, then clean up any leftover working directory state the test command might have created:

cd tests/example
rm -rf .terraform terraform.tfstate

Then run the commands above manually and see if you see any errors that don’t appear when you run terraform test. If so, that would suggest that there’s a bug where the test harness is not properly reacting to all errors.

dgem · September 15, 2022, 9:01am

Hi @apparentlymart, thanks for the speedy response!

We run all the infra related code in pipelines, so it’s a bit tricky to poke around in directories after the fact.

However, I did do a run of terraform test with TF_LOG=debug and I can see there was indeed an error submitting the Kubernetes Job (the admission controller rejected it), as you can see in the following trace snip. The trace starts on the sucessful creation of a Kuberenetes ServiceAccount, which would be then used by the Job :

2022-09-14T15:49:11.138Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: -----------------------------------------------------
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: 2022/09/14 15:49:11 [DEBUG] Kubernetes API Response Details:
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: ---[ RESPONSE ]--------------------------------------
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: HTTP/2.0 200 OK
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: Content-Length: 619
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: Audit-Id: beb2eb58-b862-4ca1-8e9d-3322d50edd49
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: Cache-Control: no-cache, private
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: Content-Type: application/json
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: Date: Wed, 14 Sep 2022 15:49:11 GMT
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: X-Kubernetes-Pf-Flowschema-Uid: 532e30a3-e5e8-40b3-8455-cd7370e1181c
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: X-Kubernetes-Pf-Prioritylevel-Uid: c284d965-6edd-45b6-b72c-3ca183dbd89c
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: 
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: {
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:  "kind": "ServiceAccount",
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:  "apiVersion": "v1",
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:  "metadata": {
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:   "name": "terraform-test",
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:   "namespace": "cipy-tests",
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:   "uid": "15ecfdf2-d0f7-4485-8678-48dbc39fc4d9",
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:   "resourceVersion": "358535797",
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:   "creationTimestamp": "2022-09-14T15:49:11Z",
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:   "annotations": {
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:    "eks.amazonaws.com/role-arn": "arn:aws:iam::XXXXXXXXXXX:role/tf-test-pod-role-le6sh"
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:   },
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:   "managedFields": [
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:    {
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:     "manager": "HashiCorp",
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:     "operation": "Update",
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:     "apiVersion": "v1",
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:     "time": "2022-09-14T15:49:11Z",
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:     "fieldsType": "FieldsV1",
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:     "fieldsV1": {
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:      "f:automountServiceAccountToken": {},
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:      "f:metadata": {
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:       "f:annotations": {
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:        ".": {},
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:        "f:eks.amazonaws.com/role-arn": {}
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:       }
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:      }
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:     }
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:    }
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:   ]
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:  },
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:  "automountServiceAccountToken": true
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: }
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: 
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: -----------------------------------------------------
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: 2022/09/14 15:49:11 [DEBUG] Configuration contains 0 secrets, saw 0, expected 1
2022-09-14T15:49:11.141Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: 2022/09/14 15:49:11 [TRACE] Waiting 500ms before next try
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: 2022/09/14 15:49:11 [DEBUG] Kubernetes API Response Details:
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: ---[ RESPONSE ]--------------------------------------
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: HTTP/2.0 403 Forbidden
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: Content-Length: 1211
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: Audit-Id: 2044070a-f3fc-4641-8d55-6931f76d27c3
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: Cache-Control: no-cache, private
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: Content-Type: application/json
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: Date: Wed, 14 Sep 2022 15:49:11 GMT
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: X-Kubernetes-Pf-Flowschema-Uid: 532e30a3-e5e8-40b3-8455-cd7370e1181c
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: X-Kubernetes-Pf-Prioritylevel-Uid: c284d965-6edd-45b6-b72c-3ca183dbd89c
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: 
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: {
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:  "kind": "Status",
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:  "apiVersion": "v1",
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:  "metadata": {},
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:  "status": "Failure",
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:  "message": "admission webhook \"polaris.fairwinds.com\" denied the request: \nPolaris prevented this deployment due to configuration problems:\n- Job: someCo tags are misconfigured on controller\nMandatory tags:\nsomeCo/alert-channel\nsomeCo/entity\nsomeCo/environment\nsomeCo/owner\nsomeCo/product\nsomeCo/service\nsomeCo/version\n\n- Pod: NodeSelector is misconfigured on pod\n- Container amazon: CPU limits should be set\n- Container amazon: Memory requests should be set\n- Container amazon: CPU requests should be set\n- Container amazon: Memory limits should be set\n",
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:  "reason": "\nPolaris prevented this deployment due to configuration problems:\n- Job: someCo tags are misconfigured on controller\nMandatory tags:\nsomeCo/alert-channel\nsomeCo/entity\nsomeCo/environment\nsomeCo/owner\nsomeCo/product\nsomeCo/service\nsomeCo/version\n\n- Pod: NodeSelector is misconfigured on pod\n- Container amazon: CPU limits should be set\n- Container amazon: Memory requests should be set\n- Container amazon: CPU requests should be set\n- Container amazon: Memory limits should be set\n",
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5:  "code": 403
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: }
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: 
2022-09-14T15:49:11.147Z [DEBUG] provider.terraform-provider-kubernetes_v2.13.1_x5: -----------------------------------------------------
2022-09-14T15:49:11.147Z [WARN]  unexpected data: registry.terraform.io/hashicorp/kubernetes:stderr="{"@caller":"github.com/hashicorp/terraform-plugin-sdk/v2@v2.12.0/internal/logging/helper_schema.go:21","@level":"trace","@message":"Called downstream","@module":"sdk.helper_schema","@timestamp":"2022-09-14T15:49:11.147488Z"}"
2022-09-14T15:49:11.149Z [ERROR] vertex "kubernetes_job_v1.list_keys" error: Failed to create Job! API error: admission webhook "polaris.fairwinds.com" denied the request: 
Polaris prevented this deployment due to configuration problems:
- Job: tags are misconfigured on controller
Mandatory tags:
someCo/alert-channel
someCo/entity
someCo/environment
someCo/owner
someCo/product
someCo/service
someCo/version

- Pod: NodeSelector is misconfigured on pod
- Container amazon: CPU limits should be set
- Container amazon: Memory requests should be set
- Container amazon: CPU requests should be set
- Container amazon: Memory limits should be set

dgem · September 15, 2022, 9:43am

Hi @apparentlymart,

I’ve been having a look at terraform/test.go at main · hashicorp/terraform · GitHub and noticed that at terraform/test.go at b54017ef83a563422bf06395ef30ef1e5579ad27 · hashicorp/terraform · GitHub there is a comment to ignore any apply failures (or that’s what I’m reading).

	state, diags = c.testSuiteApply(ctx, plan, suiteDirs, providerFactories)
	if diags.HasErrors() {
		// We don't return here, unlike the others above, because we want to
		// continue to the destroy below even if there are apply errors.
		synthError(
			"apply",
			"terraform apply",
			"failed to apply the created plan",
			diags,
		)
	}

	// By the time we get here, the test provider will have gathered up all
	// of the planned assertions and the final results for any assertions that
	// were not blocked by an error. This also resets the provider so that
	// the destroy operation below won't get tripped up on stale results.
	ret = testProvider.Reset()

	state, diags = c.testSuiteDestroy(ctx, state, suiteDirs, providerFactories)
	if diags.HasErrors() {
		synthError(
			"destroy",
			"terraform destroy",
			"failed to destroy objects created during test (NOTE: leftover remote objects may still exist)",
			diags,
		)
	}

	return ret, state

it looks to me like the evaluation of the test suite runs terraform/test.go at main · hashicorp/terraform · GitHub doesn’t check for the apply error afaics, but this is based on a n00bs first glance.

Is it worth raising an issue to progress this?

apparentlymart · September 15, 2022, 10:25pm

Hi @dgem,

My read of the code you linked to is that if it encounters an error then it will record it as part of the return value of this function (in the ret value) but it then continues to run c.testSuiteDestroy because otherwise anything already created before the error would be left running. The goal of terraform test is to try to destroy everything it created before it returns.

I do think there is some kind of bug here because we can see the provider detecting an error from the API in those logs as you say. But I don’t think the code you linked to specifically is obviously wrong, because it does seem to be writing an error into ret before returning.

It certainly wouldn’t hurt to create a GitHub issue to track this weird behavior. I see we already have an issue here which seems similar to what you’ve described, so I think it would be best to record your new example as a comment on this issue so that it’s all together in one place:

github.com/hashicorp/terraform

terraform test for module with k8s_secret resource not failing correctly

opened 09:46PM - 01 Feb 22 UTC

pindge

bug cli confirmed v1.1 experiment/module_testing

### Terraform Version ``` (base) ubuntu@:~/datakube/modules/terraform_test$ te…rraform version Terraform v1.1.4 on linux_amd64 ``` ### Terraform Configuration Files the `main.tf` file ```terraform variable "input" { type = string } variable "namespace" { type = string default = "processing" } variable "test_process" { type = string default = "test" } output "foo" { value = "foo value ${var.input}" } output "k8s_creds" { value = kubernetes_secret.uplifted_user_creds.metadata[0].name } resource "kubernetes_secret" "uplifted_user_creds" { metadata { name = "${var.namespace}-aws-creds-${var.test_process}" namespace = var.namespace } data = { } type = "Opaque" } ``` the `test.tf` file ``` terraform { required_providers { test = { source = "terraform.io/builtin/test" } } } module "main" { source = "../.." input = "boop" } resource "test_assertions" "foo" { component = "foo" equal "output" { description = "output \"foo\" value" got = module.main.foo want = "foo value boop" } } resource "test_assertions" "creds" { component = "cred" equal "output" { description = "output \"foo\" value" got = module.main.k8s_creds want = "foo value boop" } } ``` ### Debug Output ``` (base) ubuntu@:~/datakube/modules/terraform_test$ terraform test ╷ │ Warning: The "terraform test" command is experimental │ │ We'd like to invite adventurous module authors to write integration tests for their modules using this command, but all of the behaviors of this command are currently experimental and may change based on feedback. │ │ For more information on the testing experiment, including ongoing research goals and avenues for feedback, see: │ https://www.terraform.io/docs/language/modules/testing-experiment.html ╵ Success! All of the test assertions passed. ``` ### Expected Behavior the `test_assertions` `creds` should fail as the output of `kubernetes_secret.uplifted_user_creds.metadata[0].name` should be `processing-aws-creds-test`. ### Actual Behavior the test for it passed, indicating `terraform test` sees the output of `kubernetes_secret.uplifted_user_creds.metadata[0].name` equal to `foo value boop` ### Steps to Reproduce `terraform test` ### Additional Context Nil ### References N/A

In that issue there’s also a link to another issue:

github.com/hashicorp/terraform

`terraform test`: error while creating resources under test results in false success output

opened 01:29PM - 10 Dec 21 UTC

drewmullen

bug confirmed explained v1.1 experiment/module_testing

### Terraform Version  using RC due to [bug](https://github.com/hashicorp/terraform/issues/30071). I have also tested these issues with 1.0.8 ``` $ tf -v Terraform v1.1.0-rc1 on darwin_amd64 ``` ### Terraform Configuration Files **Scenario 1:** In [this branch](https://github.com/drewmullen/terraform-aws-vpc/tree/ipam-ipv4-tftest/tests/examples_ipam_vpc) I expect the terraform test to take about 45m (the resources take a long time to destroy) but it only takes about 3m. **Scenario 2:** When i that didnt work as expected i tried purposely getting tests to fail and i was unable to. I distilled a [very simple PR](https://github.com/aws-ia/terraform-aws-label/pull/4) to demonstrate what should be an easy failure ```terraform resource "test_assertions" "expect_failure" { component = "tags" check "fail" { description = "this test should fail" condition = true == false } equal "scheme" { description = "should fail" got = true want = false } } ``` ### Debug Output  Will post upon request ### Expected Behavior  **Scenario 1:** tests should take about 45m to complete **Scenario 2:** tests should fail ### Actual Behavior  **Scenario 1:** tests completed within 5m **Scenario 2:** tests pass ### Additional Context  Conversation started on [discuss.hashicorp.com](https://discuss.hashicorp.com/t/terraform-test-does-not-fail-when-it-should/32861) ### References

The discussion there actually suggests something that might be going on with your example: you have two different test_assertions resources which have the same value for the component argument, and so I wonder if they are colliding with each other and the successful one is overwriting the failing one. Can you try making sure all of your assertion resources have a unique component and see if that changes the outcome?

This use of resources to represent assertions was a compromise to avoid adding entirely new language features just for the experiment, and this requirement for the component keys to be unique across all of them is an artifact of that compromise. We’re expecting to replace the resources with real condition syntax in a future incarnation of the experiment, and so if this is the cause for you then I think we’ll have to accept it as a quirk of the current prototype and fix it with native syntax in a later release.

dgem · September 16, 2022, 1:02pm

Thanks, @apparentlymart,
I’ll make the changes you suggest and continue the investigation… really hoping this experiment get out and in use, nice work