Internal Server Error when applying on openstack

Hi,

I was trying to modify a somewhat old deployment on OpenStack. When I run terraform apply it’s failing to refresh the state of some items. I doesn’t give me a useful error but crashes with an “Internal Server Error”.

I’m using:

Terraform v0.11.11
+ provider.null v2.1.2
+ provider.openstack v1.26.0
+ provider.template v2.1.2

The original deployment was created with the same Terraform version but provider versions:

+ provider.null v2.1.0
+ provider.openstack v1.16.0
+ provider.template v2.1.0

I also tried applying with the original provider versions.

Running Terraform with TF_LOG=DEBUG and -parallelism=1 I get some more verbose output, like the following

openstack_compute_instance_v2.k8s_data_node_ext_net[2]: Refreshing state... (ID: a5204092-2d50-451b-9292-12238f26b748)
2020-03-05T16:17:51.135Z [DEBUG] plugin.terraform-provider-openstack_v1.26.0_x4: 2020/03/05 16:17:51 [DEBUG] OpenStack Region is: jic
2020-03-05T16:17:51.135Z [DEBUG] plugin.terraform-provider-openstack_v1.26.0_x4: 2020/03/05 16:17:51 [DEBUG] OpenStack Endpoint for compute: http://josc.jic.it:8774/v2.1/
2020-03-05T16:17:55.111Z [DEBUG] plugin.terraform-provider-openstack_v1.26.0_x4: 2020/03/05 16:17:55 [DEBUG] Retrieved Server a5204092-2d50-451b-9292-12238f26b748: &{ID:a5204092-2d50-451b-9292-12238f26b748 TenantID:2f3ba940e96a4bb49b680dd4708f0c80 UserID:9463d48df8d54abca70e9b1954a6381c Name:tdm-stage-k8s-data-node-ext-net-3 Updated:2020-02-12 11:30:51 +0000 UTC Created:2019-07-10 13:28:38 +0000 UTC HostID:c6880f6db5e8e7b898231a6d044d111c67597f02b5b9b51095828f1d Status:ACTIVE Progress:0 AccessIPv4: AccessIPv6: Image:map[id:52a33989-1ed3-4240-846f-25e1532f3a8d links:[map[href:http://josc.jic.it:8774/images/52a33989-1ed3-4240-846f-25e1532f3a8d rel:bookmark]]] Flavor:map[id:ad25e075-11f0-412e-8514-d6b4c83b362f links:[map[href:http://josc.jic.it:8774/flavors/ad25e075-11f0-412e-8514-d6b4c83b362f rel:bookmark]]] Addresses:map[external_net:[map[OS-EXT-IPS-MAC:mac_addr:fa:16:3e:04:ed:61 OS-EXT-IPS:type:fixed addr:172.30.10.74 version:4]]] Metadata:map[depends_on:e9618922-1eef-4de8-9881-89777f573a37 kubespray_groups:kube-node,k8s-cluster,no-floating,k8s-data-node, ssh_user:ubuntu] Links:[map[href:http://josc.jic.it:8774/v2.1/servers/a5204092-2d50-451b-9292-12238f26b748 rel:self] map[href:http://josc.jic.it:8774/servers/a5204092-2d50-451b-9292-12238f26b748 rel:bookmark]] KeyName:kubernetes-tdm-stage AdminPass: SecurityGroups:[map[name:default] map[name:tdm-stage-k8s-worker] map[name:tdm-stage-k8s]] AttachedVolumes:[] Fault:{Code:0 Created:0001-01-01 00:00:00 +0000 UTC Details: Message:}}
2020-03-05T16:17:55.111Z [DEBUG] plugin.terraform-provider-openstack_v1.26.0_x4: 2020/03/05 16:17:55 [DEBUG] OpenStack Region is: jic
2020-03-05T16:17:55.111Z [DEBUG] plugin.terraform-provider-openstack_v1.26.0_x4: 2020/03/05 16:17:55 [DEBUG] OpenStack Endpoint for compute: http://josc.jic.it:8774/v2.1/
2020-03-05T16:18:01.053Z [DEBUG] plugin.terraform-provider-openstack_v1.26.0_x4: 2020/03/05 16:18:01 [DEBUG] Addresses: map[string]interface {}{"external_net":[]interface {}{map[string]interface {}{"OS-EXT-IPS-MAC:mac_addr":"fa:16:3e:04:ed:61", "OS-EXT-IPS:type":"fixed", "addr":"172.30.10.74", "version":4}}}
2020-03-05T16:18:01.053Z [DEBUG] plugin.terraform-provider-openstack_v1.26.0_x4: 2020/03/05 16:18:01 [DEBUG] allInstanceAddresses: []openstack.InstanceAddresses{openstack.InstanceAddresses{NetworkName:"external_net", InstanceNICs:[]openstack.InstanceNIC{openstack.InstanceNIC{FixedIPv4:"172.30.10.74", FixedIPv6:"", MAC:"fa:16:3e:04:ed:61"}}}}
2020-03-05T16:18:01.053Z [DEBUG] plugin.terraform-provider-openstack_v1.26.0_x4: 2020/03/05 16:18:01 [DEBUG] getAllInstanceNetworks: []openstack.InstanceNetwork{openstack.InstanceNetwork{UUID:"2f0db58d-fd9f-4cd8-83fb-59c225a06dc0", Name:"external_net", Port:"", FixedIP:"172.30.10.74", AccessNetwork:false}}
2020-03-05T16:18:01.054Z [DEBUG] plugin.terraform-provider-openstack_v1.26.0_x4: 2020/03/05 16:18:01 [DEBUG] flattenInstanceNetworks: []map[string]interface {}{map[string]interface {}{"access_network":false, "fixed_ip_v4":"172.30.10.74", "fixed_ip_v6":"", "mac":"fa:16:3e:04:ed:61", "name":"external_net", "port":"", "uuid":"2f0db58d-fd9f-4cd8-83fb-59c225a06dc0"}}
2020-03-05T16:18:01.054Z [DEBUG] plugin.terraform-provider-openstack_v1.26.0_x4: 2020/03/05 16:18:01 [DEBUG] OpenStack Instance Network Access Addresses: 172.30.10.74,
2020/03/05 16:18:12 [ERROR] root.compute: eval: *terraform.EvalRefresh, err: openstack_compute_instance_v2.k8s_data_node_ext_net.2: Internal Server Error
2020/03/05 16:18:12 [ERROR] root.compute: eval: *terraform.EvalSequence, err: openstack_compute_instance_v2.k8s_data_node_ext_net.2: Internal Server Error
2020/03/05 16:18:12 [TRACE] [walkRefresh] Exiting eval tree: module.compute.openstack_compute_instance_v2.k8s_data_node_ext_net[2]
2020/03/05 16:18:12 [TRACE] root.compute: eval: *terraform.EvalSequence

Does anyone have an idea of what may be going on here? I can provide more log output should it be helpful.

Cheers,

Luca

Have you used openstack commands to pull the details of that instance to compare against the state file? That exercise will also tell you if the native openstack client is able to pull all of the metadata. Most of the TF OStack problems I have had were related to an openstack problem and had nothing to do with TF. Good luck

Indeed, it turned out to be an openstack problem.

The Terraform problem in this case is that it crashes with an internal error instead of a useful error message :-/

Thanks for the tip!

Luca

1 Like