Boundary kex_exchange_identification: Connection closed by remote host error

Hello,

I’m having some issues when trying to connect my ssh target. I have checked github issues and some topics that people facing the same issue. They say that this is related to your localhost ssh configuration or the server ssh might be disabled. None is my case.

I was able to connect to it a few weeks back but I re-deployed the target server in an automated fashion and added it to the boundary as a target but this time, I’m unable to connect to it.

boundary credential-libraries create vault \
      -credential-store-id csvlt_TQJJIiFkhG \
      -vault-path "internal/data/bastion" \
      -name "vault-cred-library" \
      -credential-type ssh_private_key

Credential Library information:
  Created Time:          Fri, 04 Nov 2022 15:51:20 EDT
  Credential Store ID:   csvlt_TQJJIiFkhG
  ID:                    clvlt_dsSLPk1xDT
  Name:                  vault-cred-library
  Type:                  vault
  Updated Time:          Fri, 04 Nov 2022 15:51:20 EDT
  Version:               1

  Scope:
    ID:                  p_eOIb2ZWorP
    Name:                xxx Project
    Parent Scope ID:     o_RkkqCJgtQu
    Type:                project

  Authorized Actions:
    no-op
    read
    update
    delete

  Attributes:
    HTTP Method:         GET
    Path:                internal/data/bastion

  Credential Type:
    ssh_private_key

Configured the host and host set. When I tried to connect to the ssh target.

$ boundary connect ssh -target-id ttcp_fHVqBPUpj5
Credentials:
  Credential Source ID:   clvlt_dsSLPk1xDT
  Credential Source Name: vault-cred-library
  Credential Store ID:    csvlt_TQJJIiFkhG
  Credential Store Type:  vault
  Credential Type:        ssh_private_key
  Secret:
        private_key:   -----BEGIN RSA PRIVATE KEY-----
        <REDACTED>
        -----END RSA PRIVATE KEY-----
        username:      ubuntu

kex_exchange_identification: Connection closed by remote host
Connection closed by 127.0.0.1 port 49745

Worker node logs

2022-11-04T19:59:53.082730575Z boundary-worker-576474fffc-dk98z {"id":"NzPRpFZEyz","source":"https://hashicorp.com/boundary/boundary-worker-576474fffc-dk98z/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).handleProxy","data":{"msg":"session successfully activated","session_id":"s_BguVCHWiHJ"}},"datacontentype":"application/cloudevents","time":"2022-11-04T19:59:53.082415291Z"}
2022-11-04T19:59:53.094185626Z boundary-worker-576474fffc-dk98z {"id":"ftSUX0zNjh","source":"https://hashicorp.com/boundary/boundary-worker-576474fffc-dk98z/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).handleProxy","data":{"connection_id":"sc_VecwnkOY9k","msg":"connection successfully authorized","session_id":"s_BguVCHWiHJ"}},"datacontentype":"application/cloudevents","time":"2022-11-04T19:59:53.093805172Z"}
2022-11-04T20:00:07.609454Z    Stream canceled unexpected EOF for hashicorp/boundary-worker-576474fffc-89jpc (boundary-worker)
2022-11-04T20:00:18.604417Z    Stream canceled unexpected EOF for hashicorp/boundary-worker-576474fffc-n5469 (boundary-worker)
2022-11-04T20:00:42.725100036Z boundary-worker-576474fffc-dk98z {"id":"Fk39fspwJI","source":"https://hashicorp.com/boundary/boundary-worker-576474fffc-dk98z/worker","specversion":"1.0","type":"error","data":{"error":"error dialing endpoint: dial tcp 44.198.187.113:22: connect: connection timed out","error_fields":{},"id":"e_SNnREZrgny","version":"v0.1","op":"worker.(Worker).handleProxy","info":{"endpoint":"tcp://44.198.187.113:22","msg":"error handling proxy","session_id":"s_9mUqiFg6mp"}},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:42.724239627Z"}
2022-11-04T20:00:42.725162967Z boundary-worker-576474fffc-dk98z {"id":"eY4oa7QFxR","source":"https://hashicorp.com/boundary/boundary-worker-576474fffc-dk98z/worker","specversion":"1.0","type":"error","data":{"error":"failed to close WebSocket: failed to read frame header: EOF","error_fields":{},"id":"e_HTSPUPEMZC","version":"v0.1","op":"worker.(Worker).handleProxy","info":{"msg":"error closing client connection"}},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:42.724746622Z"}
2022-11-04T20:00:42.734570165Z boundary-worker-576474fffc-dk98z {"id":"sd5BPlSnlL","source":"https://hashicorp.com/boundary/boundary-worker-576474fffc-dk98z/worker","specversion":"1.0","type":"error","data":{"error":"could not find session ID \"s_9mUqiFg6mp\" in local state after closing connections","error_fields":{},"id":"e_nrSvMLSN6A","version":"v0.1","op":"session.closeConnections","info":{"msg":"error marking connection closed in state"}},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:42.734143521Z"}
2022-11-04T20:00:53.352048180Z boundary-worker-576474fffc-dk98z {"id":"VUbAnEsDHI","source":"https://hashicorp.com/boundary/boundary-worker-576474fffc-dk98z/worker","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"worker.(Worker).cleanupConnections","data":{"connection_id":"sc_VecwnkOY9k","msg":"terminated connection due to cancellation or expiration","session_id":"s_BguVCHWiHJ"}},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:53.351501415Z"}

Controller logs

boundary-controller-7cd9cb6c84-p8q5d {"id":"Xm0tn8dc0T","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-p8q5d/controller","specversion":"1.0","type":"error","data":{"error":"WARNING: credential will expire before job scheduler can run","error_fields":{},"id":"e_af4aln8JjM","version":"v0.1","op":"vault.(Repository).Issue","request_info":{"id":"gtraceid_YJBXhD7HEZtnTlMn3VHj","method":"POST","path":"/v1/targets/ttcp_fHVqBPUpj5:authorize-session","public_id":"at_bjZdQtTAms","client_ip":"10.5.237.173"},"info":{"runJobsInterval":60000000000}},"datacontentype":"application/cloudevents","time":"2022-11-04T19:59:52.866956549Z"}
boundary-controller-7cd9cb6c84-p8q5d {"id":"zANJ9Y6RHW","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-p8q5d/controller","specversion":"1.0","type":"observation","data":{"latency-ms":124.162454,"request_info":{"id":"gtraceid_YJBXhD7HEZtnTlMn3VHj","method":"POST","path":"/v1/targets/ttcp_fHVqBPUpj5:authorize-session","public_id":"at_bjZdQtTAms","client_ip":"10.5.237.173"},"start":"2022-11-04T19:59:52.756930253Z","status":200,"stop":"2022-11-04T19:59:52.881092707Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T19:59:52.88127846Z"}
boundary-controller-7cd9cb6c84-rnrws {"id":"3CLFvKApUf","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-rnrws/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.78192,"request_info":{"id":"gtraceid_hT6BLHAwZfmzgQJpBjmy","method":"GET","path":"/health","client_ip":"10.5.104.226"},"start":"2022-11-04T19:59:57.607645062Z","status":200,"stop":"2022-11-04T19:59:57.608426952Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T19:59:57.608448032Z"}
boundary-controller-7cd9cb6c84-rnrws {"id":"NGHDWoMSmQ","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-rnrws/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.682659,"request_info":{"id":"gtraceid_G6i5dCeRgRXsHy9AXQzc","method":"GET","path":"/health","client_ip":"10.5.104.226"},"start":"2022-11-04T19:59:57.607953586Z","status":200,"stop":"2022-11-04T19:59:57.608636235Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T19:59:57.608657785Z"}
boundary-controller-7cd9cb6c84-cbhq6 {"id":"0buFrtUYOx","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-cbhq6/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.533577,"request_info":{"id":"gtraceid_Jb4Ya4MssfEoQowM7KQC","method":"GET","path":"/health","client_ip":"10.5.174.119"},"start":"2022-11-04T19:59:57.760616677Z","status":200,"stop":"2022-11-04T19:59:57.761150254Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T19:59:57.761166174Z"}
boundary-controller-7cd9cb6c84-cbhq6 {"id":"BlNAHpmeui","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-cbhq6/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.454356,"request_info":{"id":"gtraceid_1zH5St81FDBrrCqbc7Wb","method":"GET","path":"/health","client_ip":"10.5.174.119"},"start":"2022-11-04T19:59:57.76082586Z","status":200,"stop":"2022-11-04T19:59:57.761280206Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T19:59:57.761300026Z"}
boundary-controller-7cd9cb6c84-p8q5d {"id":"RLxYJ5AvzH","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-p8q5d/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.642881,"request_info":{"id":"gtraceid_n7mmz4bUV8Lr2gcJIcyK","method":"GET","path":"/health","client_ip":"10.5.201.244"},"start":"2022-11-04T19:59:57.784529273Z","status":200,"stop":"2022-11-04T19:59:57.785172154Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T19:59:57.785199535Z"}
boundary-controller-7cd9cb6c84-p8q5d {"id":"j51N5fm38r","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-p8q5d/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.510028,"request_info":{"id":"gtraceid_rcx63F63NM8vq7MZJxui","method":"GET","path":"/health","client_ip":"10.5.201.244"},"start":"2022-11-04T19:59:57.785859416Z","status":200,"stop":"2022-11-04T19:59:57.786369454Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T19:59:57.786422025Z"}
boundary-controller-7cd9cb6c84-rnrws {"id":"Rb4dscbT4w","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-rnrws/controller","specversion":"1.0","type":"system","data":{"version":"v0.1","op":"session.WorkerStatusReport","data":{"controller_id":"w_pI5lT4RsOP","count":1,"msg":"marked unclaimed connections as closed"}},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:05.134133324Z"}
boundary-controller-7cd9cb6c84-rnrws {"id":"tIULgeRSxZ","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-rnrws/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.510007,"request_info":{"id":"gtraceid_xbZVSgGVPfaRElpedLjQ","method":"GET","path":"/health","client_ip":"10.5.104.226"},"start":"2022-11-04T20:00:07.606409509Z","status":200,"stop":"2022-11-04T20:00:07.606919526Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:07.606938446Z"}
boundary-controller-7cd9cb6c84-rnrws {"id":"pHCzUZ8S8u","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-rnrws/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.495276,"request_info":{"id":"gtraceid_tcMYs4ku5wnLiS1iVgeY","method":"GET","path":"/health","client_ip":"10.5.104.226"},"start":"2022-11-04T20:00:07.607673986Z","status":200,"stop":"2022-11-04T20:00:07.608169042Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:07.608192143Z"}
boundary-controller-7cd9cb6c84-cbhq6 {"id":"ldL10CQ4S1","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-cbhq6/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.474486,"request_info":{"id":"gtraceid_WR43dnGo6XmeUNIezYJh","method":"GET","path":"/health","client_ip":"10.5.174.119"},"start":"2022-11-04T20:00:07.761874265Z","status":200,"stop":"2022-11-04T20:00:07.762348731Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:07.762373051Z"}
boundary-controller-7cd9cb6c84-cbhq6 {"id":"aBOJAr2FY5","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-cbhq6/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.329385,"request_info":{"id":"gtraceid_KncgNLQRhxOq9Da82GgF","method":"GET","path":"/health","client_ip":"10.5.174.119"},"start":"2022-11-04T20:00:07.762767796Z","status":200,"stop":"2022-11-04T20:00:07.763097171Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:07.763118331Z"}
boundary-controller-7cd9cb6c84-p8q5d {"id":"N6tiHcemM3","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-p8q5d/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.469818,"request_info":{"id":"gtraceid_hSrpjN7YM85VGN5bfw1I","method":"GET","path":"/health","client_ip":"10.5.201.244"},"start":"2022-11-04T20:00:07.781472179Z","status":200,"stop":"2022-11-04T20:00:07.781941997Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:07.781963127Z"}
boundary-controller-7cd9cb6c84-p8q5d {"id":"D1rBTetSYf","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-p8q5d/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.436478,"request_info":{"id":"gtraceid_BKYvYJVWSf618A4b9oYT","method":"GET","path":"/health","client_ip":"10.5.201.244"},"start":"2022-11-04T20:00:07.783157147Z","status":200,"stop":"2022-11-04T20:00:07.783593605Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:07.783616415Z"}
boundary-controller-7cd9cb6c84-rnrws {"id":"KmievSl0hj","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-rnrws/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.508247,"request_info":{"id":"gtraceid_JipkC8BnF8j5Csb54m5o","method":"GET","path":"/health","client_ip":"10.5.104.226"},"start":"2022-11-04T20:00:17.606775748Z","status":200,"stop":"2022-11-04T20:00:17.607283955Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:17.607301975Z"}
boundary-controller-7cd9cb6c84-rnrws {"id":"OTAuObmVYr","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-rnrws/controller","specversion":"1.0","type":"observation","data":{"latency-ms":1.106855,"request_info":{"id":"gtraceid_BP4hel0gQaSY5YLhP1gA","method":"GET","path":"/health","client_ip":"10.5.104.226"},"start":"2022-11-04T20:00:17.606775728Z","status":200,"stop":"2022-11-04T20:00:17.607882563Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:17.607920183Z"}
boundary-controller-7cd9cb6c84-cbhq6 {"id":"ifoABfjwRo","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-cbhq6/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.528247,"request_info":{"id":"gtraceid_ZfAFud827L3ZxHWCFrTk","method":"GET","path":"/health","client_ip":"10.5.174.119"},"start":"2022-11-04T20:00:17.760992567Z","status":200,"stop":"2022-11-04T20:00:17.761520814Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:17.761554084Z"}
boundary-controller-7cd9cb6c84-cbhq6 {"id":"tVhSyeSDkl","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-cbhq6/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.530057,"request_info":{"id":"gtraceid_keKl78S58nRSRNzC77ip","method":"GET","path":"/health","client_ip":"10.5.174.119"},"start":"2022-11-04T20:00:17.760995237Z","status":200,"stop":"2022-11-04T20:00:17.761525284Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:17.761541824Z"}
boundary-controller-7cd9cb6c84-p8q5d {"id":"r1S0vr8APw","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-p8q5d/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.385906,"request_info":{"id":"gtraceid_lkEDAU9EJMSUUMBsGAcV","method":"GET","path":"/health","client_ip":"10.5.201.244"},"start":"2022-11-04T20:00:17.779449904Z","status":200,"stop":"2022-11-04T20:00:17.7798358Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:17.779850641Z"}
boundary-controller-7cd9cb6c84-p8q5d {"id":"QhLpVZ38hd","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-p8q5d/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.410957,"request_info":{"id":"gtraceid_cCWCTLS5pDlQqAUhSsBg","method":"GET","path":"/health","client_ip":"10.5.201.244"},"start":"2022-11-04T20:00:17.78099164Z","status":200,"stop":"2022-11-04T20:00:17.781402607Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:17.781456768Z"}
boundary-controller-7cd9cb6c84-rnrws {"id":"m9F9Av6dl8","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-rnrws/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.555118,"request_info":{"id":"gtraceid_8fvyM2b34R7ePDfHfY4w","method":"GET","path":"/health","client_ip":"10.5.104.226"},"start":"2022-11-04T20:00:27.606993819Z","status":200,"stop":"2022-11-04T20:00:27.607548917Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:27.607563497Z"}
boundary-controller-7cd9cb6c84-rnrws {"id":"ssHd8O4QWG","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-rnrws/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.505007,"request_info":{"id":"gtraceid_poRhVbHKHgAiFqTDdMSi","method":"GET","path":"/health","client_ip":"10.5.104.226"},"start":"2022-11-04T20:00:27.607216152Z","status":200,"stop":"2022-11-04T20:00:27.607721139Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:27.607739959Z"}
boundary-controller-7cd9cb6c84-cbhq6 {"id":"4zYlt5RTpv","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-cbhq6/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.521566,"request_info":{"id":"gtraceid_HqPrr6pxIepB5jjtoTnC","method":"GET","path":"/health","client_ip":"10.5.174.119"},"start":"2022-11-04T20:00:27.760536918Z","status":200,"stop":"2022-11-04T20:00:27.761058484Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:27.761076475Z"}
boundary-controller-7cd9cb6c84-cbhq6 {"id":"MEFegDWVO8","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-cbhq6/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.517886,"request_info":{"id":"gtraceid_itpo80Tq890X1PBiU0v1","method":"GET","path":"/health","client_ip":"10.5.174.119"},"start":"2022-11-04T20:00:27.760536908Z","status":200,"stop":"2022-11-04T20:00:27.761054794Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:27.761089335Z"}
boundary-controller-7cd9cb6c84-p8q5d {"id":"3Hj6mj3BUu","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-p8q5d/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.413177,"request_info":{"id":"gtraceid_xLm8tv6jGAcmjYokPczs","method":"GET","path":"/health","client_ip":"10.5.201.244"},"start":"2022-11-04T20:00:27.782831118Z","status":200,"stop":"2022-11-04T20:00:27.783244315Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:27.783259766Z"}
boundary-controller-7cd9cb6c84-p8q5d {"id":"tdzO4v2J3g","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-p8q5d/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.316036,"request_info":{"id":"gtraceid_YtJ8Y77CdhXODgBFzuey","method":"GET","path":"/health","client_ip":"10.5.201.244"},"start":"2022-11-04T20:00:27.78352973Z","status":200,"stop":"2022-11-04T20:00:27.783845776Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:27.783864276Z"}
boundary-controller-7cd9cb6c84-rnrws {"id":"2PVmlttKBp","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-rnrws/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.465786,"request_info":{"id":"gtraceid_EHDZk5VkNDZz4OIOKOHA","method":"GET","path":"/health","client_ip":"10.5.104.226"},"start":"2022-11-04T20:00:37.607251054Z","status":200,"stop":"2022-11-04T20:00:37.60771681Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:37.607738271Z"}
boundary-controller-7cd9cb6c84-rnrws {"id":"wPmKexk1kN","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-rnrws/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.350745,"request_info":{"id":"gtraceid_CYhriDyEAFYm98XPtov6","method":"GET","path":"/health","client_ip":"10.5.104.226"},"start":"2022-11-04T20:00:37.608387689Z","status":200,"stop":"2022-11-04T20:00:37.608738424Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:37.608760944Z"}
boundary-controller-7cd9cb6c84-cbhq6 {"id":"32C321bRBi","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-cbhq6/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.635169,"request_info":{"id":"gtraceid_U2dhlbfRNerQAMybBVsN","method":"GET","path":"/health","client_ip":"10.5.174.119"},"start":"2022-11-04T20:00:37.760419832Z","status":200,"stop":"2022-11-04T20:00:37.761055001Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:37.761096911Z"}
boundary-controller-7cd9cb6c84-cbhq6 {"id":"aZrUTVF9Xv","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-cbhq6/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.615078,"request_info":{"id":"gtraceid_VQInDoo6jltJVd1tApl8","method":"GET","path":"/health","client_ip":"10.5.174.119"},"start":"2022-11-04T20:00:37.760465623Z","status":200,"stop":"2022-11-04T20:00:37.761080701Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:37.761100761Z"}
boundary-controller-7cd9cb6c84-p8q5d {"id":"TpRRW2RazT","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-p8q5d/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.956186,"request_info":{"id":"gtraceid_0qnVxTWV3cf04N6fyqFT","method":"GET","path":"/health","client_ip":"10.5.201.244"},"start":"2022-11-04T20:00:37.77905866Z","status":200,"stop":"2022-11-04T20:00:37.780014846Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:37.780059337Z"}
boundary-controller-7cd9cb6c84-p8q5d {"id":"ccUcjF6sBM","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-p8q5d/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.409277,"request_info":{"id":"gtraceid_6S1vRqY27qTFj1EUxodb","method":"GET","path":"/health","client_ip":"10.5.201.244"},"start":"2022-11-04T20:00:37.781092524Z","status":200,"stop":"2022-11-04T20:00:37.781501841Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:37.781747655Z"}
boundary-controller-7cd9cb6c84-rnrws {"id":"lseX3Vtif8","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-rnrws/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.447696,"request_info":{"id":"gtraceid_jmwqMDUMdsKznyxsB1co","method":"GET","path":"/health","client_ip":"10.5.104.226"},"start":"2022-11-04T20:00:47.606293926Z","status":200,"stop":"2022-11-04T20:00:47.606741582Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:47.606763532Z"}
boundary-controller-7cd9cb6c84-rnrws {"id":"UG4OJZOOmj","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-rnrws/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.491847,"request_info":{"id":"gtraceid_QbEe6HkTCX8Zlo6MxIMl","method":"GET","path":"/health","client_ip":"10.5.104.226"},"start":"2022-11-04T20:00:47.615229084Z","status":200,"stop":"2022-11-04T20:00:47.615720901Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:47.615743311Z"}
boundary-controller-7cd9cb6c84-cbhq6 {"id":"0nbPRuRToi","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-cbhq6/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.584078,"request_info":{"id":"gtraceid_k3SDuSLz29mEyYnVHv11","method":"GET","path":"/health","client_ip":"10.5.174.119"},"start":"2022-11-04T20:00:47.760535287Z","status":200,"stop":"2022-11-04T20:00:47.761119365Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:47.761135665Z"}
boundary-controller-7cd9cb6c84-cbhq6 {"id":"LnF1tMTplk","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-cbhq6/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.554708,"request_info":{"id":"gtraceid_haAHHxXKCWobvRCJ5MgB","method":"GET","path":"/health","client_ip":"10.5.174.119"},"start":"2022-11-04T20:00:47.7607671Z","status":200,"stop":"2022-11-04T20:00:47.761321828Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:47.761341118Z"}
boundary-controller-7cd9cb6c84-p8q5d {"id":"Ye5FVW9jB4","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-p8q5d/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.465918,"request_info":{"id":"gtraceid_nQ4uNDY5DC5wXRhDOiNC","method":"GET","path":"/health","client_ip":"10.5.201.244"},"start":"2022-11-04T20:00:47.778692496Z","status":200,"stop":"2022-11-04T20:00:47.779158474Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:47.779179704Z"}
boundary-controller-7cd9cb6c84-p8q5d {"id":"0vbMtl0hBX","source":"https://hashicorp.com/boundary/boundary-controller-7cd9cb6c84-p8q5d/controller","specversion":"1.0","type":"observation","data":{"latency-ms":0.844865,"request_info":{"id":"gtraceid_gwCxXP8O3KLGOQiPm4rB","method":"GET","path":"/health","client_ip":"10.5.201.244"},"start":"2022-11-04T20:00:47.780371674Z","status":200,"stop":"2022-11-04T20:00:47.781216559Z","version":"v0.1"},"datacontentype":"application/cloudevents","time":"2022-11-04T20:00:47.781239999Z"}

Beside that, I can connect to the server without boundary. What do you think would cause such an issue? As I was fine to connect few weeks back, I can’t figure why I’m unable to do so now.

Hey @ercindemir0

Thanks for the interest in Boundary! A kex_exchange_identification Is a SSH protocol level error, unfortunately for tcp targets you wont get much information for the Boundary logs for this.

What version of Boundary are you running? The reason I ask is because when you attach a ssh_private_key credential to a target the boundary connect ssh command attempts to consume that credential in order to auth the connection, if the command does this it should not be printing the credential information in the CLI.

Another option you can try and test is run boundary connect without the ssh. This will open a localhost proxy for you that you can then pass your private key directly using -i. This is effectively what Boundary is attempting to do

Hey @louisruch,

Thank you for the reply. I was using 0.10.1 and just upgraded to 0.11.0 to see if the issue gets resolved. The output for both versions is the same. The exact same configuration was working before, I don’t know what caused such an error. My primary motivation is to use boundary desktop app rather than the cli. Do you happen to know how I can troubleshoot kex_exchange_identification error?

ps: I’ve also tried tailing ssh logs with sudo tail -f /var/log/auth.log | grep ssh but seems like boundary doesn’t hit the server. Because I see authentication logs while I’m connecting to the server without boundary but with boundary nothing goes to the log. And also enabling remote login in the sharing settings of mac, didn’t work for me (I saw some people recommending this for such errors.)

Any leads to help me get this working will highly be appreciated!

I am not clear on your exact setup, but have you confirmed you can connect to the target from the box your worker binary is running from?

I’ve found the solution. In my case, the security group of the instance didn’t allow the machine. So the issue resolved when I opened the port up. Thanks for the help.