Receive link not reopened after being remote closed #219
Comments
|
It happened again, we have extra logging now: Not sure if this is related: Looking at the "No message received etc" logging after the error the code seems to think the connection is (still) ok. However the subscription is getting filled with messages that are no longer retrieved. We have a multi-instance solution (all with their own subscription on the same service bus) and I see this message at the same time at all of them: The other instances all continued processing messages after that error, just one did not. It's not always the same instance that fails.
|
|
Same happens to us |
|
I have the exact same problem as noted in #224. I think even more of a coincidence is that @markvandenbergh and I happen to be coworkers (we didn't know we had the same issues). For completeness sake: @yvgopal We have the exact same issue. Connection closed due to inactivity, but then on reconnection we are getting the same stacktrace. This results in a situation where it stops processing messages and we need to restart our application in order to get it to process messages again. Waiting for it to be automatically reconnected doesn't solve the problem. Is there a workaround for this, or a way for us to handle these exceptions and manually reconnect? Our MessageHandler looks like this: |
|
Just want to chime in and say how bothersome this issue is. We're currently reconnecting every 10 minutes, by creating a new client and closing the old one, as a workaround, but it's pretty messy and should not be necessary when using a service we're actually paying for. |
|
Our .NET applications (both function apps and other applications) have none of these problems, so it must be the java library that's causing issues. |
|
We will take a look and get back to you |
|
@markvandenbergh @hope4555 @las3r @tobiasgwaaler
|
|
I'll try to answer to the best of my ability @yvgopal, perhaps @markvandenbergh can answer this as well :).
|
|
Reproducing this is difficult, but I'm wondering if this issue is related to another issue: if the application is unable to reach the service-bus server for some time (say 5 minutes), it will not be able to reconnect once the network is working again. You can reproduce this by running the code below, and simply disconnecting your network entirely for a while, then reconnect. The subscription will not reconnect after this. import com.microsoft.azure.servicebus.*;
import com.microsoft.azure.servicebus.primitives.ConnectionStringBuilder;
import com.microsoft.azure.servicebus.primitives.ServiceBusException;
import java.util.concurrent.CompletableFuture;
public class TopicAndSubscription {
public static void main(String[] args) throws ServiceBusException, InterruptedException {
// insert your test topic config
ConnectionStringBuilder topicConStringBuilder = new ConnectionStringBuilder(
"",
"",
"",
"");
// insert your subcription config
ConnectionStringBuilder subConStringBuilder = new ConnectionStringBuilder(
"",
"",
"",
"");
TopicClient topicClient = new TopicClient(topicConStringBuilder);
SubscriptionClient subscriptionClient = new SubscriptionClient(subConStringBuilder, ReceiveMode.RECEIVEANDDELETE);
subscriptionClient.registerMessageHandler(new IMessageHandler() {
@Override
public CompletableFuture<Void> onMessageAsync(IMessage message) {
String body = new String(message.getBody());
System.out.println("Got message: " + body);
return CompletableFuture.completedFuture(null);
}
@Override
public void notifyException(Throwable throwable, ExceptionPhase exceptionPhase) {
System.err.println("Error: " + throwable.getMessage());
}
});
while (true) {
Thread.sleep(2_000);
String body = Long.toString(System.currentTimeMillis());
System.out.println("Sending message: " + body);
topicClient.send(new Message(body));
}
}
} |
|
Hi, @yvgopal, per our phone discussion earlier today, the MS team feels this issue was corrected as of We see similar messages in our logs in apps that are running the new versions, such as:
I don't think this is related to long disconnects from the network as all of our consumers are running on Kubernetes in Azure VMs. Can we please reopen this issue? |
|
Similar thing happens to me using service bus topic stream binder 1.1.0RC2 (see referenced issue above) |
|
@jemag This was an older issue and was fixed in SDK version 1.2.6. I suggest you start using the latest version of java SDK 1.2.11. |
|
@yvgopal the version I mentionned is for the spring-cloud-azure-servicebus-topic-stream-binder and there doesn't seem to be any newer (than 1.1.0RC2) version available on public repositories. Do you mean manually replacing the azure-servicebus dependency within the spring-cloud-azure-servicebus-topic-stream-binder? |
|
@jemag I don't know how the spring binder package is released. You should ask them to switch to the latest version of service bus SDK. Until then you can manually add maven dependency to the latest version of java SDK. |
|
kk, thank you. Will do. |
|
I still see this issue when using |
|
@theangrydev Azure service bus service closes idle links after some timeout. Azure SB java SDK reopens the links. Generic JMS clients may not. You should handle this case and recreate the sender or receiver. |
|
@yvgopal I see, thank you for the information. This is not great behaviour, I have not seen this before in similar services. I will have a look at solving the problem for generic JMS clients and possibly submit an example to the samples project. |
|
we have the same issue with jms + qpid approach |
|
@gennadiy-dubina as @yvgopal pointed out, the Azure SB Java SDK reopens the links but generic JMS clients do not necessarily handle this well if at all. So your options are:
|
|
@theangrydev thank you for response, Azure provides binder for it. so i'm going to verify this option |
|
@gennadiy-dubina any progress on your research? |
|
@ezienecker it works for us |
|
for JMS client |
We have the following stacktrace which suggests the receive link will be reopened, however this does not seem to happen as the messages are no longer retrieved from the subscription until a restart.
The logging level of this class was set at
errorwhen this happened (two or three times now), we are going to change the logging level to get more detail on this when it happens again. I will update this ticket once we get more detail.Actual Behavior
Expected Behavior
Versions
The text was updated successfully, but these errors were encountered: