Regression testing of custom provider

[lechnerc77]

We are enhancing our test setup for a custom provider in order to identify regression concerning the resource schemas.
Idea is to execute the following flow within a test case:

• Execute test step leveraging the ExternalProviders (leaving out ProtoV6ProviderFactories)
• Execute a consequent test step that uses the ProtoV6ProviderFactories which then instantiates the current version/dev version of the provider and do a plan only

As pseudo code this would look like this:

Steps: []resource.TestStep{
				{
					ExternalProviders: map[string]resource.ExternalProvider{
						"btp": {
							Source:            "org/myprovider",
							VersionConstraint: getRegressionTestBaseVersion(),
						},
					},

					Config: hclProviderFor(user) + hclResourceDirectoryWithFeatures("uut", "my-new-directory-feat", "This is a new directory with features"),
					Check: resource.ComposeAggregateTestCheckFunc(
						resource.TestMatchResourceAttr("xxx_directory.uut", "id", regexpValidUUID),
					),
				},
				{
					ProtoV6ProviderFactories: getProviders(rec.GetDefaultClient()),
					Config:                   hclProviderFor(user) + hclResourceDirectoryWithFeatures("uut", "my-new-directory-feat", "This is a new directory with features"),
					PlanOnly:                 true,
				},
			},

This gives us some errors that are hard to say how to overcome them:

Error: Failed to query available provider packages
            
Could not retrieve the list of available versions for provider org/myprovider: no available releases match the given constraints 

Error: Missing required argument
            
The argument "xyz" is required, but was not set.

Letting the tests run with either only the ExternalProvider or only the ProtoV6ProviderFactories works fine. The combination as depicted above fails.

Are we on the completely wrong path with this setup or should this work and we are having something wrong when wiring things up

[PaulW]

Hi @lechnerc77

I just wondered if you’d managed to get anywhere with this issue? I have today raised what I suspect is an identical problem:

Acceptance Testing using a local External Provider

Hi, I am currently trying to migrate a private Terraform Provider from the legacy SDK V2 over to the Plugin Framework. I’m happy that the plan generated using the new resource provider is a no-op when switching from the legacy SDK provider to the framework provider as I have tested this manually. However I am trying to write an acceptance test as per the guide here and have hit a problem: As this is a private provider I need the ExternalProviders to be picked up from a local directory rather …

If you managed to fix this and could provide any advice that would be appreciated.

Thanks.

Paul

[lechnerc77]

Hi @PaulW ,

this looks like exactely the same problem. Unfortunately we were not able to resolve this issue via the integrated test framework.

Our current solution is to setup a real sample infrastructure via the current version of our provider, then update the provider and execute a terraform plan to determine deviations that we might have missed. This is executed via GitHub Actions. This is a bit tedious, but we did not yet find any better solution

Best,
Christian

[PaulW]

Hi @lechnerc77

Thanks for getting back to me. Sorry to hear you didn’t manage to fix this. Sounds like we might need to do what you have done too.

I’ll be sure to post back here though if I do manage to solve this.

Thanks,

Paul

[bendbennett]

Hi @lechnerc77

Sorry you ran into trouble here.

I’m intrigued by the error that you’re seeing when you run both test steps, but not when you run each test step in isolation.

We have a number of acceptance tests on the random provider, which operate in a similar way by running a test step with a previous version of the provider, and then run a test step with local provider code as defined by Proto<V5/V6>ProviderFactories.

For example, TestAccResourcePassword_OverrideSpecial_FromVersion3_3_2 runs a test step with v3.3.2 of the random provider, then another test step using the local provider code.

Under these circumstances, I can only reproduce the error that you’re reporting by altering the VersionConstraint that is being used for the ExternalProvider to a version that doesn’t exist. Specifically:

func providerVersion332() map[string]resource.ExternalProvider {
	return map[string]resource.ExternalProvider{
		"random": {
			VersionConstraint: "3.3.3",
			Source:            "hashicorp/random",
		},
	}
}

In this instance, 3.3.3 specifies a version that doesn’t exist and I see the following error:

        Error: Failed to query available provider packages
        
        Could not retrieve the list of available versions for provider
        hashicorp/random: no available releases match the given constraints 3.3.3

It’s interesting that in the case you describe, the error that you see does not output the version constraint specified by VersionConstraint in your configuration.

Are you able to confirm that when the VersionConstraint is hardcoded in your configuration to a value that specifies an available version that you see the same error?

[lechnerc77]

Hi @bendbennett ,

when analyzing the error we also had that value hard coded pointing to an existing version the Terraform registry (I think we tried every variant on how to set parameters when trying to pin the issue down ). The result remained unchanged.

We also debugged things to quite some extent and the VersionConstraint contained the right value. As mentioned this worked fine when executing the two steps in isolation.

[bendbennett]

Hi @lechnerc77, thanks for confirming.

Couple of further questions:

• Assume you’re using terraform-plugin-testing for the acceptance tests? Can you confirm which version you are using. If you are using v1.6.0, can you confirm that you see the same error with v1.5.1?
• Are you able to share the raw configuration that you are populating the Config field with (redacting any sensitive data)?

Thanks

[lechnerc77]

Hi @bendbennett ,

we executed the tests on both versions The upgrade to 1.6.0 was in between. We failed on both versions.

The data in this case is the following:

The hclProviderFor returns the following data:

return fmt.Sprintf(`
provider "btp" {
    cli_server_url = "%s"
    globalaccount  = "%s"
    username       = "%s"
    password       = "%s"
    idp            = "%s"
}
    `, cliServerURL, testGlobalAccount, user.Username, user.Password, user.Idp)
}

The hclResourceDirectoryWithFeatures returns:

return fmt.Sprintf(`resource "btp_directory" "%s" {
        name        = "%s"
        description = "%s"
		features    = ["DEFAULT","ENTITLEMENTS","AUTHORIZATIONS"]
    }`, resourceName, displayName, description)

Best,
Christian

[bendbennett]

Hi @lechnerc77, thank you for sharing that. I can’t see any obvious issues with the configuration.

Glancing back at your original post, I noticed that the second test step is specifying PlanOnly: true. The release notes for v1.6.0 of terraform-plugin-testing mention the following:

helper/resource: Implicit terraform refresh commands during each TestStep have been removed to fix plan check and performance issues, which can cause new test failures when testing schema changes (e.g. state upgrades) that have a final TestStep with PlanOnly: true. Remove PlanOnly: true from the final TestStep to fix affected tests which will ensure that updated schema changes are applied to the state before attempting to automatically destroy resources. (#223)

Could you perhaps check to see if removing PlanOnly: true as described in the release notes has any effect in this case?

[lechnerc77]

@bendbennett I just tried the same setup without the PlanOnly parameter. The error remains the same:

Error: Failed to query available provider packages
            
Could not retrieve the list of available versions for provider org/myprovider: no
available releases match the given constraints 

Error: Missing required argument
            
The argument "xyz" is required, but was not set.

[PaulW]

Hi,

I don’t think my problem was exactly the same as i am using a locally built provider but thought it worth explaining how I have solved this.

I am on a Mac M2 so the architecture is darwin_arm64.

I’ve built my provider and placed it in

/$HOME/.terraform.d/plugins/registry.terraform.io/org/myprovider/<version>/darwin_arm64
Then my .terraformrc file (in $HOME) looks like this:

plugin_cache_dir = "$HOME/.terraform.d/plugin-cache"
provider_installation {
  filesystem_mirror {
    path = "$HOME/.terraform.d/plugins"
  }
  direct {
    exclude = ["org/*/*"]
  }
}

NOTE that in the .terraformrc file, for the path $HOME needs to be expanded to the real value rather than using the environment variable

Then my migration test defines this for the external provider:

resource.Test(test, resource.TestCase{
		Steps: []resource.TestStep{
			{
				ExternalProviders: map[string]resource.ExternalProvider{
					"myprovider": {
						VersionConstraint: "<version>",
						Source:            "org/myprovider",
					},
				},

This now works, picks up the external provider and runs the test correctly.

Hope this helps!

Paul

[bendbennett]

Hi @lechnerc77,

Are you able to provide any further clarification on what The argument "xyz" is required, but was not set refers to? Does the schema differ between the version used by the provider specified in ExternalProviders and that used by the provider specified by ProtoV6ProviderFactories? If so, are you able to share the code for the UpgradeState method? It would be helpful if you could also show the schema for both versions, if they differ.

It would also be useful to see the log output from the failing test run. If you run the test with TF_LOG=TRACE and perhaps save the output to a gist and link that here that would be helpful.

[lechnerc77]

Hi @bendbennett ,

the parameter mentioned is called globalaccount and part of the provider configuration
There is no deviation between the schemas, we did not implement the method UpgradeState. The test setup should be a measure for us to detect accidental schema changes that ight have been missed in our test setup.

You find the trace here: Error Terraform Test schema changes

[bendbennett]

Hi @lechnerc77,

Sorry for the delay in getting back to you. Thank you for supplying the logs. Looking through this I don’t see anything obvious that could lead to the issue you’re encountering. Couple of further questions:

• Do you have any environment vars set during the running of the tests, for instance TF_ACC_PROVIDER_NAMESPACE?
• Are your colleagues able to reproduce the same issue that you see running the same tests?
• Can you link to the source code for the tests?
• If you clone the random provider, and run TestAccResourcePassword_OverrideSpecial_FromVersion3_3_2, does this test execute successfully in your environment?

[lechnerc77]

Hi @bendbennett ,

we are currently a bit busy with getting the provider GA, so answers take a bit:

• There are no TF env variables set
• Yes team members see the same error when doing the tests
• I can link the test, but it does currently not contain the part mentioned in the error as we removed it after the unsuccessful tests: terraform-provider-btp/internal/provider/resource_directory_test.go at main · SAP/terraform-provider-btp (github.com)
• I tested the setup with the random provider inside of the same devcontainer that we use for our provider and it worked.