Terraform AWS gitlab pipeline plan not recognizing changes

I have few Terraform pipelines running on gitlab and this 1 pipeline that I am trying to setup for AWS does not want to work. Even though I am setting up the access key and secret as an environment variable it still does not run successfully terraform init unless I pass the access key and secret as backend-config like this: terraform init -backend-config="access_key=$AWS_ACCESS_KEY_PIPELINE_TEST" -backend-config="secret_key=$AWS_ACCESS_KEY_PIPELINE_SECRET" -backend-config="region=$AWS_DEFAULT_REGION"

Even more confusing is when I run just terraform plan it does not detect change but when I pass the var-file it works just fine. I am not sure what is the issue here. If anyone has faced this issue then please suggest what could I be doing wrong. Below is my gitlab CI file

image:
  name: hashicorp/terraform:latest
  entrypoint:
    - '/usr/bin/env'
    - 'PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin'

variables:
  hcm_dev_PLAN: hcm_dev_plan.tfplan
  hcm_shared_PLAN: hcm_shared_plan.tfplan
  hcm_prod_PLAN: hcm_prod_plan.tfplan

stages:
  - test
  - validate
  - plan
  - apply


.validate: &ValidateAnchor
  script:
    - |
      echo "I am inside the validate anchor step"
      if [[ ${CI_JOB_NAME} == "validate" ]]; then
        export access_key=${AWS_ACCESS_KEY_PIPELINE_TEST}
        export secret_key=${AWS_ACCESS_KEY_PIPELINE_SECRET}
        export region=${AWS_DEFAULT_REGION}
        terraform init -backend-config="access_key=$AWS_ACCESS_KEY_PIPELINE_TEST" -backend-config="secret_key=$AWS_ACCESS_KEY_PIPELINE_SECRET" -backend-config="region=$AWS_DEFAULT_REGION"
        echo "DEV successfully initialized!"
        terraform fmt
        terraform validate
      fi

.plan: &PlanAnchor
  script:
    - |
      if [[ ${CI_JOB_NAME} == "plan" ]]; then
        export access_key=${AWS_ACCESS_KEY_PIPELINE_TEST}
        export secret_key=${AWS_ACCESS_KEY_PIPELINE_SECRET}
        export region=${AWS_DEFAULT_REGION}
        terraform init -backend-config="access_key=$AWS_ACCESS_KEY_PIPELINE_TEST" -backend-config="secret_key=$AWS_ACCESS_KEY_PIPELINE_SECRET" -backend-config="region=$AWS_DEFAULT_REGION"
        echo "DEV successfully initialized!"
        terraform plan -out=$hcm_dev_PLAN
      fi


validate:
  <<: *ValidateAnchor
  stage: validate
  only:
    changes:
      - *

plan:
  <<: *PlanAnchor
  stage: plan
  only:
    changes:
      - *
  artifacts:
    name: plan
    paths:
      - "$CI_PROJECT_DIR"
    exclude:

Below is the output of the plan

DEV successfully initialized!
No changes. Your infrastructure matches the configuration.
Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

What could be the reason of not persisting the credentials and why does the plan needs the var file to detect the changes?
Please suggest

Encountering issues with Terraform AWS GitLab pipeline plan not recognizing changes can be a bit perplexing, but fear not – let’s dive into some potential solutions.

  1. AWS Integration Configuration: Ensure that your AWS integration configuration within GitLab CI/CD is accurate. Double-check the credentials, IAM roles, and permissions to make sure Terraform can effectively communicate with your AWS environment.
  2. Terraform Init: Run terraform init within your GitLab pipeline to ensure that all the necessary providers and modules are correctly initialized. This step is crucial for setting up the working directory with the required configuration.
  3. Check for Changes: Review your GitLab CI/CD configuration to confirm that it’s configured to detect changes appropriately. This includes checking the paths specified in your GitLab CI/CD file and making sure they align with the actual changes made in your Terraform codebase.
  4. Pipeline Execution Logs: Examine the logs of your GitLab pipeline execution for any error messages or warnings related to the Terraform plan. This can provide valuable insights into what might be causing the changes not to be recognized.
  5. Version Compatibility: Verify that you are using compatible versions of Terraform, AWS provider, and other dependencies. Sometimes, issues can arise due to incompatibilities between different versions.

Remember, each project’s setup can be unique, so these are general suggestions. If the problem persists, consider sharing more details about your GitLab CI/CD configuration and Terraform code for the community to provide more targeted assistance.

Have you considered any of these steps yet, or do you have additional details to share about your setup?

Thanks @Roshan for responding to my query. Please find my response below

  1. AWS Integration Configuration: The same credentials are configured a gitlab group level which means those are available to be defined within the pipeline. Also I have tried to do a word to word match of those variables vs what I have in my CI file and I did not find any difference. Using the same credentials I can run CLI commands which I guess means the credentials are valid.

  2. Terraform init: As you can see my CI where if I just run terraform init without the backend config entires then I get access denied error like below: Successfully configured the backend "s3"! Terraform will automatically use this backend unless the backend configuration changes. Error refreshing state: AccessDenied: Access Denied status code: 403, request id: NR4QSTZ09WCFBVXG, host id: PBXYZbQfp+zlkA0gGLR8LJ55nY5ve+/Tnm5AkWt2Ox2VPK7ZPlA40wRbmwATdUs1WCy5uoFVH6Q=

  3. Check for Changes: All my changes are in the root location of the project so I am just using $CI_PROJECT_DIR variable to access the location of my working directory.

  4. Pipeline Execution Logs: I have even tried echo statements to ensure that the pipeline flow is working just fine which seems to be just fine.

  5. Version Compatibility: Below is my provider file

provider "aws" {
  #required_version = ">= 0.13.1"
  region = "us-east-1"
}

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = ">= 4.9"
    }
  }
  backend "s3" {
    bucket = "terraform-dev-statefile-bucket"
    key    = "statefile-name.tfstate"
    region = "us-east-1"
  }
}

when I run terraform --version along with init(with backend-config) and validate here is the output

Terraform has been successfully initialized!
You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.
If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
DEV successfully initialized!
variables.tf
╷
│ Warning: Argument is deprecated
│ 
│   with module.s3.aws_s3_bucket.create-s3-bucket,
│   on .terraform/modules/s3/storage/s3/main.tf line 1, in resource "aws_s3_bucket" "create-s3-bucket":
│    1: resource "aws_s3_bucket" "create-s3-bucket" {
│ 
│ Use the aws_s3_bucket_server_side_encryption_configuration resource instead
╵
Success! The configuration is valid, but there were some
validation warnings as shown above.
Terraform v1.7.0
on linux_amd64
+ provider registry.terraform.io/hashicorp/aws v5.32.1
Cleaning up project directory and file based variables
00:01
Job succeeded

Please let me know if you need any more information

I changed the following

        export AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_PIPELINE_TEST}
        export AWS_SECRET_ACCESS_KEY=${AWS_ACCESS_KEY_PIPELINE_SECRET}
        export AWS_DEFAULT_REGION=${AWS_DEFAULT_REGION}

That worked for terraform init without the backend-config but not for terraform plan as it is still not detecting changes if I dont pass the var-file

Meticulously configuring the AWS access key and secret as environment variables in their GitLab CI/CD settings, they found that the terraform init command consistently failed to authenticate with AWS, unless they passed the credentials explicitly as backend configuration parameters. Even after successfully initializing Terraform, they faced another perplexing situation: when running terraform plan , the command failed to detect any changes in their infrastructure code unless they explicitly provided a var-file.