consul connect envoy 启动acls和tls后，grpc连接异常

consul dev用起来都很正常，但开了加密和校验，难度翻了好几倍。

首先看一下consul connect enovy的日志：

2022-10-12 10:38:10.418][45382][info][config] [source/server/configuration_impl.cc:97] loading 1 cluster(s) [2022-10-12 10:38:10.522][45382][info][config] [source/server/configuration_impl.cc:101] loading 0 listener(s) [2022-10-12 10:38:10.522][45382][info][config] [source/server/configuration_impl.cc:113] loading stats configuration [2022-10-12 10:38:10.523][45382][info][runtime] [source/common/runtime/runtime_impl.cc:463] RTDS has finished initialization [2022-10-12 10:38:10.523][45382][info][upstream] [source/common/upstream/cluster_manager_impl.cc:221] cm init: initializing cds [2022-10-12 10:38:10.526][45382][warning][main] [source/server/server.cc:784] there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections [2022-10-12 10:38:10.527][45382][info][main] [source/server/server.cc:905] starting main dispatch loop [2022-10-12 10:38:25.523][45382][warning][config] [source/common/config/grpc_subscription_impl.cc:118] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.cluster.v3.Cluster [2022-10-12 10:38:25.523][45382][info][upstream] [source/common/upstream/cluster_manager_impl.cc:225] cm init: all clusters initialized [2022-10-12 10:38:25.523][45382][info][main] [source/server/server.cc:886] all clusters initialized. initializing init manager [2022-10-12 10:38:40.524][45382][warning][config] [source/common/config/grpc_subscription_impl.cc:118] gRPC config: initial fetch timed out for type.googleapis.com/envoy.config.listener.v3.Listener [2022-10-12 10:38:40.524][45382][info][config] [source/server/listener_manager_impl.cc:841] all dependencies initialized. starting workers [2022-10-12 10:39:06.293][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 55s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:39:18.840][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 68s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:39:48.373][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 97s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:39:57.092][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 106s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:40:10.912][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 120s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:40:23.269][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 132s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:40:50.591][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 160s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:41:04.465][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 173s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:41:20.063][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 189s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:41:35.213][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 204s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:41:40.294][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 209s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:41:54.596][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 224s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:42:11.217][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 240s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:42:24.340][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 253s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:42:30.040][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 259s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:42:33.164][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 262s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination [2022-10-12 10:42:58.807][45382][warning][config] [./source/common/config/grpc_stream.h:196] DeltaAggregatedResources gRPC config stream closed since 288s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination

DeltaAggregatedResources gRPC config stream closed since 55s ago: 14, upstream connect error or disconnect/reset before headers. reset reason: connection termination

consul的日志：

2022-10-12T10:38:10.915+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50406": tls: first record does not look like a TLS handshake 2022-10-12T10:38:12.529+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50408": tls: first record does not look like a TLS handshake 2022-10-12T10:38:12.549+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50410": tls: first record does not look like a TLS handshake 2022-10-12T10:38:14.088+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50414": tls: first record does not look like a TLS handshake 2022-10-12T10:38:16.652+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50416": tls: first record does not look like a TLS handshake 2022-10-12T10:38:17.414+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50418": tls: first record does not look like a TLS handshake 2022-10-12T10:38:22.881+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50424": tls: first record does not look like a TLS handshake 2022-10-12T10:38:31.523+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50430": tls: first record does not look like a TLS handshake 2022-10-12T10:38:36.600+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50436": tls: first record does not look like a TLS handshake 2022-10-12T10:38:36.811+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50438": tls: first record does not look like a TLS handshake 2022-10-12T10:38:40.302+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50440": tls: first record does not look like a TLS handshake 2022-10-12T10:38:45.743+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50446": tls: first record does not look like a TLS handshake 2022-10-12T10:39:06.293+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50468": tls: first record does not look like a TLS handshake 2022-10-12T10:39:10.438+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50470": tls: first record does not look like a TLS handshake 2022-10-12T10:39:18.839+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50480": tls: first record does not look like a TLS handshake 2022-10-12T10:39:27.425+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50488": tls: first record does not look like a TLS handshake 2022-10-12T10:39:34.695+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50498": tls: first record does not look like a TLS handshake 2022-10-12T10:39:48.373+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50504": tls: first record does not look like a TLS handshake 2022-10-12T10:39:52.666+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50512": tls: first record does not look like a TLS handshake 2022-10-12T10:39:57.091+0800 [WARN] agent: [core]grpc: Server.Serve failed to complete security handshake from "127.0.0.1:50516": tls: first record does not look like a TLS handshake

首先，consul的配置很重要，一开始我也是这样想的，后面把研究反向对到envoy身上，为啥envoy集群起不来。

consul：

Consul = 1.13.2

Envoy=1.23.0
ACLs = Enabled
TLS = Enabled

client的配置：记得grpc端口要，默认好像是不开的。

"ca_file": "/etc/ssl/certs/foobar-consul-ca.pem","cert_file": "/etc/consul/client1.dc1.consul.pem","key_file": "/etc/consul/client1.dc1.consul.key","connect": {"enabled": true},"ports": {"grpc": 8502,"https": 8501}, consul connect envoy --sidecar-for foobar -admin-bind localhost:19000

还有添加系统环境

CONSUL_HTTP_SSL=true CONSUL_HTTP_ADDR=127.0.0.1:8501 CONSUL_CACERT=/etc/ssl/certs/consul-ca.pem CONSUL_CLIENT_CERT=/etc/consul/client1.dc1.consul.pem CONSUL_CLIENT_KEY=/etc/consul/client1.dc1.consul.key

这个点，基本都能在网上找到。但其实还差了些。

CONSUL_GRPC_ADDR=https://127.0.0.1:8502

少了这个，因为envoy是要用到grpc的，而且consul的grpc专门就是给envoy 的xdc 用的。

就这样就可以跑起来的话，就没下面的什么事情了。

但我情况有点怪，系统环境感觉不生效，直接输出的都正常，就是grpc有问题。我也想过在connect命令行里面加更多的配置，但把token和ca的参数都带上，还是不行。

consul connect envoy \-grpc-addr=https://localhost:8502 \-ca-file=/etc/consul.d/ssl.ca.d/ssl.chain.pem \-client-cert=/etc/consul.d/ssl.crt.pem \-client-key=/etc/consul.d/ssl.key.pem \-http-addr=https://localhost:8501 \-tls-server-name=localhost \-token=... \-admin-bind 127.0.0.1:19005 \-envoy-version=1.14.2 \-sidecar-for some-service

看到别人说的一个例子，试了一下，居然正常了。其实本来想着consul来直接带着一堆参数来辅助调用envoy，但还是要把tls的参数在命令行带过去。

consul的文档变得有点快，但看起来越来越清晰，之前看1.13.1的文档，配置啥的东一块西一块，陷阱都有说，但不是在一起！！！

参考：

Envoy Proxy breaks when enabling Consul TLS · Issue #7926 · hashicorp/consul · GitHub

环境参数    Commands | Consul | HashiCorp Developer

总结

以上是生活随笔为你收集整理的consul connect envoy 启动acls和tls后，grpc连接异常的全部内容，希望文章能够帮你解决所遇到的问题。

如果觉得生活随笔网站内容还不错，欢迎将生活随笔推荐给好友。