[infinispan-dev] Default TCP configuration is broken.

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] Default TCP configuration is broken.

Pedro Ruivo-2
Hi team,

The 'default-jgroups-tcp.xml" has MFC protocol without the FRAG2/3
protocol. This is broken when we send a multicast message larger than
'max-credits'. It will block forever in MFC [1]. No timeouts since we
don't have the CompletableFuture at this point.

Possible solutions are:

#1 put back FRAG2/3

advantage: we have multicast flow control.
disadvantage: all messages are fragmented (unicast and multicast), that
probably requires more resources (more messages in NAKACK and UNICAST
tables?)

#2 remove MFC

advantage: probably low resources usages. TCP will handle any fragmentation.
disadvantage: we don't have multicast flow control.

#3 alternative?

Cheers,
Pedro

[1] actually, I need a thread dump to confirm it.
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Default TCP configuration is broken.

Alan Field


----- Original Message -----

> From: "Pedro Ruivo" <[hidden email]>
> To: "ispn-dev" <[hidden email]>
> Sent: Wednesday, February 22, 2017 9:59:21 AM
> Subject: [infinispan-dev] Default TCP configuration is broken.
>
> Hi team,
>
> The 'default-jgroups-tcp.xml" has MFC protocol without the FRAG2/3
> protocol. This is broken when we send a multicast message larger than
> 'max-credits'. It will block forever in MFC [1]. No timeouts since we
> don't have the CompletableFuture at this point.
>
> Possible solutions are:
>
> #1 put back FRAG2/3
>
> advantage: we have multicast flow control.
> disadvantage: all messages are fragmented (unicast and multicast), that
> probably requires more resources (more messages in NAKACK and UNICAST
> tables?)

This is my preferred solution, since removing MFC did not work for me!

Thanks,
Alan

>
> #2 remove MFC
>
> advantage: probably low resources usages. TCP will handle any fragmentation.
> disadvantage: we don't have multicast flow control.
>
> #3 alternative?
>
> Cheers,
> Pedro
>
> [1] actually, I need a thread dump to confirm it.
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Default TCP configuration is broken.

Pedro Ruivo-2
One step ahead: https://issues.jboss.org/browse/ISPN-7512

On 22-02-2017 19:41, Alan Field wrote:

>
>
> ----- Original Message -----
>> From: "Pedro Ruivo" <[hidden email]>
>> To: "ispn-dev" <[hidden email]>
>> Sent: Wednesday, February 22, 2017 9:59:21 AM
>> Subject: [infinispan-dev] Default TCP configuration is broken.
>>
>> Hi team,
>>
>> The 'default-jgroups-tcp.xml" has MFC protocol without the FRAG2/3
>> protocol. This is broken when we send a multicast message larger than
>> 'max-credits'. It will block forever in MFC [1]. No timeouts since we
>> don't have the CompletableFuture at this point.
>>
>> Possible solutions are:
>>
>> #1 put back FRAG2/3
>>
>> advantage: we have multicast flow control.
>> disadvantage: all messages are fragmented (unicast and multicast), that
>> probably requires more resources (more messages in NAKACK and UNICAST
>> tables?)
>
> This is my preferred solution, since removing MFC did not work for me!
>
> Thanks,
> Alan
>
>>
>> #2 remove MFC
>>
>> advantage: probably low resources usages. TCP will handle any fragmentation.
>> disadvantage: we don't have multicast flow control.
>>
>> #3 alternative?
>>
>> Cheers,
>> Pedro
>>
>> [1] actually, I need a thread dump to confirm it.
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Default TCP configuration is broken.

Dan Berindei
In reply to this post by Alan Field
+1 to add FRAG3 back in, with frag_size="200k" to make fragmentation
less likely, yet stay under max_credits * min_threshold.

Dan

On Wed, Feb 22, 2017 at 9:41 PM, Alan Field <[hidden email]> wrote:

>
>
> ----- Original Message -----
>> From: "Pedro Ruivo" <[hidden email]>
>> To: "ispn-dev" <[hidden email]>
>> Sent: Wednesday, February 22, 2017 9:59:21 AM
>> Subject: [infinispan-dev] Default TCP configuration is broken.
>>
>> Hi team,
>>
>> The 'default-jgroups-tcp.xml" has MFC protocol without the FRAG2/3
>> protocol. This is broken when we send a multicast message larger than
>> 'max-credits'. It will block forever in MFC [1]. No timeouts since we
>> don't have the CompletableFuture at this point.
>>
>> Possible solutions are:
>>
>> #1 put back FRAG2/3
>>
>> advantage: we have multicast flow control.
>> disadvantage: all messages are fragmented (unicast and multicast), that
>> probably requires more resources (more messages in NAKACK and UNICAST
>> tables?)
>
> This is my preferred solution, since removing MFC did not work for me!
>
> Thanks,
> Alan
>
>>
>> #2 remove MFC
>>
>> advantage: probably low resources usages. TCP will handle any fragmentation.
>> disadvantage: we don't have multicast flow control.
>>
>> #3 alternative?
>>
>> Cheers,
>> Pedro
>>
>> [1] actually, I need a thread dump to confirm it.
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Default TCP configuration is broken.

Bela Ban
Note that FRAG3 is relatively recent...

On 22/02/17 21:01, Dan Berindei wrote:

> +1 to add FRAG3 back in, with frag_size="200k" to make fragmentation
> less likely, yet stay under max_credits * min_threshold.
>
> Dan
>
> On Wed, Feb 22, 2017 at 9:41 PM, Alan Field <[hidden email]> wrote:
>>
>>
>> ----- Original Message -----
>>> From: "Pedro Ruivo" <[hidden email]>
>>> To: "ispn-dev" <[hidden email]>
>>> Sent: Wednesday, February 22, 2017 9:59:21 AM
>>> Subject: [infinispan-dev] Default TCP configuration is broken.
>>>
>>> Hi team,
>>>
>>> The 'default-jgroups-tcp.xml" has MFC protocol without the FRAG2/3
>>> protocol. This is broken when we send a multicast message larger than
>>> 'max-credits'. It will block forever in MFC [1]. No timeouts since we
>>> don't have the CompletableFuture at this point.
>>>
>>> Possible solutions are:
>>>
>>> #1 put back FRAG2/3
>>>
>>> advantage: we have multicast flow control.
>>> disadvantage: all messages are fragmented (unicast and multicast), that
>>> probably requires more resources (more messages in NAKACK and UNICAST
>>> tables?)
>>
>> This is my preferred solution, since removing MFC did not work for me!
>>
>> Thanks,
>> Alan
>>
>>>
>>> #2 remove MFC
>>>
>>> advantage: probably low resources usages. TCP will handle any fragmentation.
>>> disadvantage: we don't have multicast flow control.
>>>
>>> #3 alternative?
>>>
>>> Cheers,
>>> Pedro
>>>
>>> [1] actually, I need a thread dump to confirm it.
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

--
Bela Ban, JGroups lead (http://www.jgroups.org)

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Default TCP configuration is broken.

Radim Vansa
In reply to this post by Pedro Ruivo-2
On 02/22/2017 03:59 PM, Pedro Ruivo wrote:

> Hi team,
>
> The 'default-jgroups-tcp.xml" has MFC protocol without the FRAG2/3
> protocol. This is broken when we send a multicast message larger than
> 'max-credits'. It will block forever in MFC [1]. No timeouts since we
> don't have the CompletableFuture at this point.
>
> Possible solutions are:
>
> #1 put back FRAG2/3
>
> advantage: we have multicast flow control.
> disadvantage: all messages are fragmented (unicast and multicast), that
> probably requires more resources (more messages in NAKACK and UNICAST
> tables?)
>
> #2 remove MFC
>
> advantage: probably low resources usages. TCP will handle any fragmentation.
> disadvantage: we don't have multicast flow control.
>
> #3 alternative?

Let MFC go one message beyond the limit? I don't know if that's
desirable, but it's about having less messages on the wire = in some
buffers. I've seen FRAG2 buffering a many parts of a big
StateResponseCommand. So it might have similar effect if we allow one
big message to be sent, rather than splitting it into parts and
buffering that anyway on the receiver side.

Just my 2c

Radim

>
> Cheers,
> Pedro
>
> [1] actually, I need a thread dump to confirm it.
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev