Membri Hazelcast in 2 diversi host Docker
Sep 29 2020
Lo scenario è costituito da 2 VM, ciascuna con docker e memebers Hazelcast in esecuzione come contenitori.
Leggendo questa guida https://hazelcast.com/blog/configuring-hazelcast-in-non-orchestrated-docker-environments/Posso ottenere l' indirizzo IP pubblico dello scenario 3 , la mappatura delle porte e il metodo di rilevamento TCP lavorando con un membro per nodo. Ma se aggiungo un membro a uno dei nodi, prende il posto dell'altro membro nel cluster o registra i problemi di connessione. Quindi non sono in grado di far funzionare il cluster con più di un membro per nodo.
La configurazione in entrambi i nodi è:
hazelcast:
network:
join:
multicast:
enabled: false
tcp-ip:
enabled: true
member-list:
- 10.132.0.2:5701
- 10.128.0.3:5701
- 10.128.0.3:5702
Il nodo Container in 10.132.0.2viene eseguito con:
docker run -v `pwd`:/mnt --rm --name member1 -e "JAVA_OPTS=-Dhazelcast.local.
publicAddress=10.132.0.2 -Dhazelcast.config=/mnt/hazelcast.yml" -p 5701:5701 hazelcast/hazelcast:4.0.1
########################################
# JAVA_OPTS=-Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/opt/hazelcast/logging.properties -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED -Dhazelcast.local.publicAddress=10.132.0.2 -Dhazelcast.config=/mnt/hazelcast.yml
# CLASSPATH=/opt/hazelcast/*:/opt/hazelcast/lib/*
# starting now....
########################################
+ exec java -server -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/opt/hazelcast/logging.properties -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED -Dhazelcast.local.publicAddress=10.132.0.2 -Dhazelcast.config=/mnt/hazelcast.yml com.hazelcast.core.server.HazelcastMemberStarter
Sep 29, 2020 6:35:23 AM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/mnt/hazelcast.yml' from System property 'hazelcast.config'
Sep 29, 2020 6:35:23 AM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /mnt/hazelcast.yml
Sep 29, 2020 6:35:24 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [4.0.1] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [10.128.0.3, 10.132.0.2]
Sep 29, 2020 6:35:24 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [4.0.1] Prefer IPv4 stack is true, prefer IPv6 addresses is false
Sep 29, 2020 6:35:24 AM com.hazelcast.instance.AddressPicker
WARNING: [LOCAL] [dev] [4.0.1] Could not find a matching address to start with! Picking one of non-loopback addresses.
Sep 29, 2020 6:35:24 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [4.0.1] Picked [172.17.0.2]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true
Sep 29, 2020 6:35:24 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [4.0.1] Using public address: [10.132.0.2]:5701
Sep 29, 2020 6:35:24 AM com.hazelcast.system
INFO: [10.132.0.2]:5701 [dev] [4.0.1] Hazelcast 4.0.1 (20200409 - e086b9c) starting at [10.132.0.2]:5701
Sep 29, 2020 6:35:24 AM com.hazelcast.system
INFO: [10.132.0.2]:5701 [dev] [4.0.1] Copyright (c) 2008-2020, Hazelcast, Inc. All Rights Reserved.
Sep 29, 2020 6:35:24 AM com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator
INFO: [10.132.0.2]:5701 [dev] [4.0.1] Backpressure is disabled
Sep 29, 2020 6:35:25 AM com.hazelcast.instance.impl.Node
INFO: [10.132.0.2]:5701 [dev] [4.0.1] Creating TcpIpJoiner
Sep 29, 2020 6:35:25 AM com.hazelcast.cp.CPSubsystem
WARNING: [10.132.0.2]:5701 [dev] [4.0.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
Sep 29, 2020 6:35:26 AM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl
INFO: [10.132.0.2]:5701 [dev] [4.0.1] Starting 2 partition threads and 3 generic threads (1 dedicated for priority tasks)
Sep 29, 2020 6:35:26 AM com.hazelcast.internal.diagnostics.Diagnostics
INFO: [10.132.0.2]:5701 [dev] [4.0.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
Sep 29, 2020 6:35:26 AM com.hazelcast.core.LifecycleService
INFO: [10.132.0.2]:5701 [dev] [4.0.1] [10.132.0.2]:5701 is STARTING
Sep 29, 2020 6:35:26 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.132.0.2]:5701 [dev] [4.0.1] Connecting to /10.128.0.3:5701, timeout: 10000, bind-any: true
Sep 29, 2020 6:35:26 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.132.0.2]:5701 [dev] [4.0.1] Connecting to /10.128.0.3:5702, timeout: 10000, bind-any: true
Sep 29, 2020 6:35:26 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.132.0.2]:5701 [dev] [4.0.1] Connecting to /10.132.0.2:5703, timeout: 10000, bind-any: true
Sep 29, 2020 6:35:26 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.132.0.2]:5701 [dev] [4.0.1] Connecting to /10.132.0.2:5702, timeout: 10000, bind-any: true
Sep 29, 2020 6:35:36 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.132.0.2]:5701 [dev] [4.0.1] Could not connect to: /10.128.0.3:5702. Reason: SocketTimeoutException[null]
Sep 29, 2020 6:35:36 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.132.0.2]:5701 [dev] [4.0.1] Could not connect to: /10.128.0.3:5701. Reason: SocketTimeoutException[null]
Sep 29, 2020 6:35:36 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.132.0.2]:5701 [dev] [4.0.1] Could not connect to: /10.132.0.2:5702. Reason: SocketTimeoutException[null]
Sep 29, 2020 6:35:36 AM com.hazelcast.internal.cluster.impl.TcpIpJoiner
INFO: [10.132.0.2]:5701 [dev] [4.0.1] [10.132.0.2]:5702 is added to the blacklist.
Sep 29, 2020 6:35:36 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.132.0.2]:5701 [dev] [4.0.1] Could not connect to: /10.132.0.2:5703. Reason: SocketTimeoutException[null]
Sep 29, 2020 6:35:36 AM com.hazelcast.internal.cluster.impl.TcpIpJoiner
INFO: [10.132.0.2]:5701 [dev] [4.0.1] [10.128.0.3]:5702 is added to the blacklist.
Sep 29, 2020 6:35:36 AM com.hazelcast.internal.cluster.impl.TcpIpJoiner
INFO: [10.132.0.2]:5701 [dev] [4.0.1] [10.128.0.3]:5701 is added to the blacklist.
Sep 29, 2020 6:35:36 AM com.hazelcast.internal.cluster.impl.TcpIpJoiner
INFO: [10.132.0.2]:5701 [dev] [4.0.1] [10.132.0.2]:5703 is added to the blacklist.
Sep 29, 2020 6:35:37 AM com.hazelcast.internal.cluster.ClusterService
INFO: [10.132.0.2]:5701 [dev] [4.0.1]
Members {size:1, ver:1} [
Member [10.132.0.2]:5701 - 69284e57-ce61-405c-87d3-1e9ea46b2bed this
]
Sep 29, 2020 6:35:37 AM com.hazelcast.core.LifecycleService
INFO: [10.132.0.2]:5701 [dev] [4.0.1] [10.132.0.2]:5701 is STARTED
I contenitori in node 10.128.0.3vengono eseguiti con:
docker run -v `pwd`:/mnt --rm --name member2 -e "JAVA_OPTS=-Dhazelcast.local.p
ublicAddress=10.128.0.3:5701 -Dhazelcast.config=/mnt/hazelcast.yml" -p 5701:5701 hazelcast/hazelcast:4.0.1
########################################
# JAVA_OPTS=-Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/opt/hazelcast/logging.properties -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.management.internal=ALL-UNNAMED -Dhazelcast.local.publicAddress=10.128.0.3:5701 -Dhazelcast.config=/mnt/hazelcast.yml
# CLASSPATH=/opt/hazelcast/*:/opt/hazelcast/lib/*
# starting now....
########################################
Sep 29, 2020 6:36:54 AM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/mnt/hazelcast.yml' from System property 'hazelcast.config'
Sep 29, 2020 6:36:54 AM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /mnt/hazelcast.yml
Sep 29, 2020 6:36:55 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [4.0.1] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [10.128.0.3, 10.132.0.2]
Sep 29, 2020 6:36:55 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [4.0.1] Prefer IPv4 stack is true, prefer IPv6 addresses is false
Sep 29, 2020 6:36:55 AM com.hazelcast.instance.AddressPicker
WARNING: [LOCAL] [dev] [4.0.1] Could not find a matching address to start with! Picking one of non-loopback addresses.
Sep 29, 2020 6:36:55 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [4.0.1] Picked [172.17.0.2]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind any local is true
Sep 29, 2020 6:36:55 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [4.0.1] Using public address: [10.128.0.3]:5701
Sep 29, 2020 6:36:55 AM com.hazelcast.system
INFO: [10.128.0.3]:5701 [dev] [4.0.1] Hazelcast 4.0.1 (20200409 - e086b9c) starting at [10.128.0.3]:5701
Sep 29, 2020 6:36:55 AM com.hazelcast.system
INFO: [10.128.0.3]:5701 [dev] [4.0.1] Copyright (c) 2008-2020, Hazelcast, Inc. All Rights Reserved.
Sep 29, 2020 6:36:56 AM com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator
INFO: [10.128.0.3]:5701 [dev] [4.0.1] Backpressure is disabled
Sep 29, 2020 6:36:56 AM com.hazelcast.instance.impl.Node
INFO: [10.128.0.3]:5701 [dev] [4.0.1] Creating TcpIpJoiner
Sep 29, 2020 6:36:56 AM com.hazelcast.cp.CPSubsystem
WARNING: [10.128.0.3]:5701 [dev] [4.0.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mode! Please note that UNSAFE mode will not provide strong consistency guarantees.
Sep 29, 2020 6:36:58 AM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl
INFO: [10.128.0.3]:5701 [dev] [4.0.1] Starting 2 partition threads and 3 generic threads (1 dedicated for priority tasks)
Sep 29, 2020 6:36:58 AM com.hazelcast.internal.diagnostics.Diagnostics
INFO: [10.128.0.3]:5701 [dev] [4.0.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to the JVM arguments.
Sep 29, 2020 6:36:58 AM com.hazelcast.core.LifecycleService
INFO: [10.128.0.3]:5701 [dev] [4.0.1] [10.128.0.3]:5701 is STARTING
Sep 29, 2020 6:36:58 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5701 [dev] [4.0.1] Connecting to /10.128.0.3:5702, timeout: 10000, bind-any: true
Sep 29, 2020 6:36:58 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5701 [dev] [4.0.1] Connecting to /10.132.0.2:5703, timeout: 10000, bind-any: true
Sep 29, 2020 6:36:58 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5701 [dev] [4.0.1] Connecting to /10.132.0.2:5702, timeout: 10000, bind-any: true
Sep 29, 2020 6:36:58 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5701 [dev] [4.0.1] Connecting to /10.132.0.2:5701, timeout: 10000, bind-any: true
Sep 29, 2020 6:36:58 AM com.hazelcast.internal.nio.tcp.TcpIpConnection
INFO: [10.128.0.3]:5701 [dev] [4.0.1] Initialized new cluster connection between /172.17.0.2:56429 and /10.132.0.2:5701
Sep 29, 2020 6:37:05 AM com.hazelcast.internal.cluster.ClusterService
INFO: [10.128.0.3]:5701 [dev] [4.0.1]
Members {size:2, ver:2} [
Member [10.132.0.2]:5701 - 69284e57-ce61-405c-87d3-1e9ea46b2bed
Member [10.128.0.3]:5701 - aa22f242-cc82-44ff-9dc1-06678d14420e this
]
Sep 29, 2020 6:37:06 AM com.hazelcast.core.LifecycleService
INFO: [10.128.0.3]:5701 [dev] [4.0.1] [10.128.0.3]:5701 is STARTED
Sep 29, 2020 6:37:08 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5701 [dev] [4.0.1] Could not connect to: /10.132.0.2:5702. Reason: SocketTimeoutException[null]
Sep 29, 2020 6:37:08 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5701 [dev] [4.0.1] Could not connect to: /10.128.0.3:5702. Reason: SocketTimeoutException[null]
Sep 29, 2020 6:37:08 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5701 [dev] [4.0.1] Could not connect to: /10.132.0.2:5703. Reason: SocketTimeoutException[null]
Finora è tutto ok, ma quando inizio il membro 3:
docker run -v `pwd`:/mnt --rm --name member3 -e "JAVA_OPTS=-Dhazelcast.local.p
ublicAddress=10.128.0.3:5702 -Dhazelcast.config=/mnt/hazelcast.yml" -p 5702:5701 hazelcast/hazelcast:4.0.1
########################################
# JAVA_OPTS=-Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/opt/hazelcast/logging.properties -XX:M
axRAMPercentage=80.0 -XX:+UseParallelGC --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-UNNAMED
--add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.base/sun.ni
o.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.manageme
nt.internal=ALL-UNNAMED -Dhazelcast.local.publicAddress=10.128.0.3:5702 -Dhazelcast.config=/mnt/hazelcast.yml
# CLASSPATH=/opt/hazelcast/*:/opt/hazelcast/lib/*
# starting now....
########################################
+ exec java -server -Djava.net.preferIPv4Stack=true -Djava.util.logging.config.file=/opt/hazelcast/logging.properti
es -XX:MaxRAMPercentage=80.0 -XX:+UseParallelGC --add-modules java.se --add-exports java.base/jdk.internal.ref=ALL-
UNNAMED --add-opens java.base/java.lang=ALL-UNNAMED --add-opens java.base/java.nio=ALL-UNNAMED --add-opens java.bas
e/sun.nio.ch=ALL-UNNAMED --add-opens java.management/sun.management=ALL-UNNAMED --add-opens jdk.management/com.sun.
management.internal=ALL-UNNAMED -Dhazelcast.local.publicAddress=10.128.0.3:5702 -Dhazelcast.config=/mnt/hazelcast.y
ml com.hazelcast.core.server.HazelcastMemberStarter
Sep 29, 2020 6:38:26 AM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Loading configuration '/mnt/hazelcast.yml' from System property 'hazelcast.config'
Sep 29, 2020 6:38:26 AM com.hazelcast.internal.config.AbstractConfigLocator
INFO: Using configuration file at /mnt/hazelcast.yml
Sep 29, 2020 6:38:26 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [4.0.1] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [10.12
8.0.3, 10.132.0.2]
Sep 29, 2020 6:38:26 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [4.0.1] Prefer IPv4 stack is true, prefer IPv6 addresses is false
Sep 29, 2020 6:38:26 AM com.hazelcast.instance.AddressPicker
WARNING: [LOCAL] [dev] [4.0.1] Could not find a matching address to start with! Picking one of non-loopback address
es.
Sep 29, 2020 6:38:26 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [4.0.1] Picked [172.17.0.3]:5701, using socket ServerSocket[addr=/0.0.0.0,localport=5701], bind
any local is true
Sep 29, 2020 6:38:26 AM com.hazelcast.instance.AddressPicker
INFO: [LOCAL] [dev] [4.0.1] Using public address: [10.128.0.3]:5702
Sep 29, 2020 6:38:26 AM com.hazelcast.system
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Hazelcast 4.0.1 (20200409 - e086b9c) starting at [10.128.0.3]:5702
Sep 29, 2020 6:38:26 AM com.hazelcast.system
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Copyright (c) 2008-2020, Hazelcast, Inc. All Rights Reserved.
Sep 29, 2020 6:38:27 AM com.hazelcast.spi.impl.operationservice.impl.BackpressureRegulator
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Backpressure is disabled
Sep 29, 2020 6:38:27 AM com.hazelcast.instance.impl.Node
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Creating TcpIpJoiner
Sep 29, 2020 6:38:27 AM com.hazelcast.cp.CPSubsystem
WARNING: [10.128.0.3]:5702 [dev] [4.0.1] CP Subsystem is not enabled. CP data structures will operate in UNSAFE mod
e! Please note that UNSAFE mode will not provide strong consistency guarantees.
Sep 29, 2020 6:38:27 AM com.hazelcast.spi.impl.operationexecutor.impl.OperationExecutorImpl
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Starting 2 partition threads and 3 generic threads (1 dedicated for priority
tasks)
Sep 29, 2020 6:38:27 AM com.hazelcast.internal.diagnostics.Diagnostics
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Diagnostics disabled. To enable add -Dhazelcast.diagnostics.enabled=true to t
he JVM arguments.
Sep 29, 2020 6:38:27 AM com.hazelcast.core.LifecycleService
INFO: [10.128.0.3]:5702 [dev] [4.0.1] [10.128.0.3]:5702 is STARTING
Sep 29, 2020 6:38:28 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Connecting to /10.132.0.2:5703, timeout: 10000, bind-any: true
Sep 29, 2020 6:38:28 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Connecting to /10.132.0.2:5702, timeout: 10000, bind-any: true
Sep 29, 2020 6:38:28 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Connecting to /10.128.0.3:5701, timeout: 10000, bind-any: true
Sep 29, 2020 6:38:28 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Connecting to /10.132.0.2:5701, timeout: 10000, bind-any: true
Sep 29, 2020 6:38:28 AM com.hazelcast.internal.nio.tcp.TcpIpConnection
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Initialized new cluster connection between /172.17.0.3:52951 and /10.132.0.2:
5701
Sep 29, 2020 6:38:35 AM com.hazelcast.internal.cluster.ClusterService
INFO: [10.128.0.3]:5702 [dev] [4.0.1]
Members {size:3, ver:3} [
Member [10.132.0.2]:5701 - 69284e57-ce61-405c-87d3-1e9ea46b2bed
Member [10.128.0.3]:5701 - aa22f242-cc82-44ff-9dc1-06678d14420e
Member [10.128.0.3]:5702 - 0dd31ea2-db2e-4e43-941a-98592e222817 this
]
Sep 29, 2020 6:38:38 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Could not connect to: /10.132.0.2:5703. Reason: SocketTimeoutException[null]
Sep 29, 2020 6:38:38 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Could not connect to: /10.128.0.3:5701. Reason: SocketTimeoutException[null]
Sep 29, 2020 6:38:38 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Could not connect to: /10.132.0.2:5702. Reason: SocketTimeoutException[null]
Sep 29, 2020 6:38:38 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Connecting to /10.128.0.3:5701, timeout: 10000, bind-any: true
Sep 29, 2020 6:38:48 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Could not connect to: /10.128.0.3:5701. Reason: SocketTimeoutException[null]
Sep 29, 2020 6:38:48 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Connecting to /10.128.0.3:5701, timeout: 10000, bind-any: true
Sep 29, 2020 6:38:58 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Could not connect to: /10.128.0.3:5701. Reason: SocketTimeoutException[null]
Sep 29, 2020 6:39:08 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Connecting to /10.128.0.3:5701, timeout: 10000, bind-any: true
Sep 29, 2020 6:39:18 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Could not connect to: /10.128.0.3:5701. Reason: SocketTimeoutException[null]
Sep 29, 2020 6:39:18 AM com.hazelcast.internal.nio.tcp.TcpIpConnectionErrorHandler
WARNING: [10.128.0.3]:5702 [dev] [4.0.1] Removing connection to endpoint [10.128.0.3]:5701 Cause => java.net.Socket
TimeoutException {null}, Error-Count: 5
Sep 29, 2020 6:39:18 AM com.hazelcast.internal.cluster.impl.MembershipManager
WARNING: [10.128.0.3]:5702 [dev] [4.0.1] Member [10.128.0.3]:5701 - aa22f242-cc82-44ff-9dc1-06678d14420e is suspect
ed to be dead for reason: No connection
Sep 29, 2020 6:39:18 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Connecting to /10.128.0.3:5701, timeout: 10000, bind-any: true
Sep 29, 2020 6:39:27 AM com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager
WARNING: [10.128.0.3]:5702 [dev] [4.0.1] This node does not have a connection to Member [10.128.0.3]:5701 - aa22f24
2-cc82-44ff-9dc1-06678d14420e
Sep 29, 2020 6:39:28 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Could not connect to: /10.128.0.3:5701. Reason: SocketTimeoutException[null]
Sep 29, 2020 6:39:28 AM com.hazelcast.internal.nio.tcp.TcpIpConnectionErrorHandler
WARNING: [10.128.0.3]:5702 [dev] [4.0.1] Removing connection to endpoint [10.128.0.3]:5701 Cause => java.net.Socket
TimeoutException {null}, Error-Count: 6
Sep 29, 2020 6:39:32 AM com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager
WARNING: [10.128.0.3]:5702 [dev] [4.0.1] This node does not have a connection to Member [10.128.0.3]:5701 - aa22f24
2-cc82-44ff-9dc1-06678d14420e
Sep 29, 2020 6:39:37 AM com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager
WARNING: [10.128.0.3]:5702 [dev] [4.0.1] This node does not have a connection to Member [10.128.0.3]:5701 - aa22f24
2-cc82-44ff-9dc1-06678d14420e
Sep 29, 2020 6:39:38 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Connecting to /10.128.0.3:5701, timeout: 10000, bind-any: true
Sep 29, 2020 6:39:42 AM com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager
WARNING: [10.128.0.3]:5702 [dev] [4.0.1] This node does not have a connection to Member [10.128.0.3]:5701 - aa22f24
2-cc82-44ff-9dc1-06678d14420e
Sep 29, 2020 6:39:47 AM com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager
WARNING: [10.128.0.3]:5702 [dev] [4.0.1] This node does not have a connection to Member [10.128.0.3]:5701 - aa22f24
2-cc82-44ff-9dc1-06678d14420e
Sep 29, 2020 6:39:48 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Could not connect to: /10.128.0.3:5701. Reason: SocketTimeoutException[null]
Sep 29, 2020 6:39:48 AM com.hazelcast.internal.nio.tcp.TcpIpConnectionErrorHandler
WARNING: [10.128.0.3]:5702 [dev] [4.0.1] Removing connection to endpoint [10.128.0.3]:5701 Cause => java.net.Socket
TimeoutException {null}, Error-Count: 7
Sep 29, 2020 6:39:48 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Connecting to /10.128.0.3:5701, timeout: 10000, bind-any: true
Sep 29, 2020 6:39:52 AM com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager
WARNING: [10.128.0.3]:5702 [dev] [4.0.1] This node does not have a connection to Member [10.128.0.3]:5701 - aa22f24
2-cc82-44ff-9dc1-06678d14420e
Sep 29, 2020 6:39:57 AM com.hazelcast.internal.cluster.impl.ClusterHeartbeatManager
WARNING: [10.128.0.3]:5702 [dev] [4.0.1] This node does not have a connection to Member [10.128.0.3]:5701 - aa22f24
2-cc82-44ff-9dc1-06678d14420e
Sep 29, 2020 6:39:58 AM com.hazelcast.internal.nio.tcp.TcpIpConnector
INFO: [10.128.0.3]:5702 [dev] [4.0.1] Could not connect to: /10.128.0.3:5701. Reason: SocketTimeoutException[null]
Sep 29, 2020 6:39:58 AM com.hazelcast.internal.nio.tcp.TcpIpConnectionErrorHandler
WARNING: [10.128.0.3]:5702 [dev] [4.0.1] Removing connection to endpoint [10.128.0.3]:5701 Cause => java.net.Socket
TimeoutException {null}, Error-Count: 8
Sembra che ci sia un problema di comunicazione tra i membri sullo stesso nodo In un altro test il membro3 ha sostituito nel cluster il membro2 e ha contrassegnato i tentativi di connessione dal nodo2 come sospetti
Le VM sono fresche su GCP e sono sulla stessa rete, ho usato questa immagine:
Google, Container-Optimized OS, 85-13310.1041.9 stable, Kernel: ChromiumOS-5.4.49 Kubernetes: 1.18.9 Docker: 19.03.9 Family: cos-stable, supports Shielded VM features, supports Confidential VM features on N2D
Risposte
1 Teg Sep 30 2020 at 19:29
Il problema è causato da member3 che pensa di essere sulla porta 5701 invece che sulla 5702. La soluzione è specificare nella configurazione la porta dove il membro ascolterà sul docker host
La configurazione per il membro3 è
hazelcast:
network:
port:
port: 5702
join:
multicast:
enabled: false
tcp-ip:
enabled: true
member-list:
- 10.132.0.2:5701
- 10.128.0.3:5701
- 10.128.0.3:5702
In questo modo il cluster funziona e ogni membro può comunicare con gli altri.
kwart Sep 30 2020 at 14:07
Il tuo scenario dovrebbe funzionare e immagino che sarà una questione ambientale. Potresti provare le seguenti 2 cose?
Imposta esplicitamente la porta anche su membro1
Utilizzare :5701anche nella configurazione del membro1: sia member-list( hazelcast.yml) che hazelcast.local.publicAddressproprietà (riga di comando).
Questo passaggio probabilmente non cambia nulla per il tuo problema, ma dovrebbe almeno evitare avvisi non correlati nei log.
Prova se membro2 è raggiungibile da membro3
Dopo aver avviato i nodi, eseguire una shell interattiva member3e provare a inviare il protocollo challenge ( HZC) all'indirizzo member2del socket di. Se vedi la risposta del protocollo ( HZC) nel terminale, la comunicazione tra i contenitori funziona correttamente. Se non vedi la risposta, controlla la configurazione del Docker e del firewall per vedere cosa può causare il problema.
docker exec -it member3 /bin/bash
# Following command runs in container.
# The first HZC line is the one you type (followed by Enter).
# The second is the reply from member2.
bash-5.0# nc 10.128.0.3 5701
HZC
HZC
Se vedi la risposta del protocollo corretta nel terminale, avremo bisogno di indagare maggiormente sul comportamento Hazelcast. Non sono stato in grado di riprodurre il problema nel mio ambiente.
ScottM Oct 01 2020 at 06:34
Se vuoi andare un po 'oltre, Hazelcast supporta le chiamate del ciclo di vita nei contenitori e dai gestori dell'orchestrazione (cioè K8s e OpenShift), per questo usi il modulo di scoperta della rete Hazelcast per quella particolare piattaforma. Questo ti libera dagli indirizzi e dalle porte hard-coding e consente di eseguire tali assegnazioni in modo dinamico in fase di esecuzione.