[infinispan-dev] Reactive Streams + RxJava

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[infinispan-dev] Reactive Streams + RxJava

William Burns-3
I was thinking more about [1] and I find that I was going to implement basically reactive streams. What we have now in github is similar but it uses a very crude method of blocking the thread to prevent back pressure. This can then cause severe issues as many users have found out when they don't close iterator.

Unfortunately reactive streams is just a spec. I am proposing to add RxJava [2] as a dependency [2] in the core module to provide access to reactive streams and the various conversion methods. This library adds a bunch of support for built in back pressure, transformations and much more which would reduce the amount of code I would need to write substantially.

In regards to timing, I am thinking this is too close for 9.1, so maybe 9.2 or higher.


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Reactive Streams + RxJava

Emmanuel Bernard
You’re thinking about a pure implementation play, correct? RxJava or the Reactive Stream constructs would not be exposed to the user as API. Am I correct?
Also for posterity, we had backchannel chats about it and you said you felt vert.x was not necessarily addressing your needs. Could you elaborate a bit here?

Emmanuel

On 15 Jun 2017, at 23:20, William Burns <[hidden email]> wrote:

I was thinking more about [1] and I find that I was going to implement basically reactive streams. What we have now in github is similar but it uses a very crude method of blocking the thread to prevent back pressure. This can then cause severe issues as many users have found out when they don't close iterator.

Unfortunately reactive streams is just a spec. I am proposing to add RxJava [2] as a dependency [2] in the core module to provide access to reactive streams and the various conversion methods. This library adds a bunch of support for built in back pressure, transformations and much more which would reduce the amount of code I would need to write substantially.

In regards to timing, I am thinking this is too close for 9.1, so maybe 9.2 or higher.

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Reactive Streams + RxJava

William Burns-3


On Mon, Jun 19, 2017 at 8:34 AM Emmanuel Bernard <[hidden email]> wrote:
You’re thinking about a pure implementation play, correct? RxJava or the Reactive Stream constructs would not be exposed to the user as API. Am I correct?

Yes, that is correct. This is only for internal usage. My thought is maybe when we finally move to Java 9, we may expose the Flow API at some point, but nothing any time soon.
 
Also for posterity, we had backchannel chats about it and you said you felt vert.x was not necessarily addressing your needs. Could you elaborate a bit here?

The main difference is that RxJava [1] has a full suite of utility methods around reactive streams [2]. Vert.x has some APIs around reactive streams [3], but it is rather limited.

RxJava is about publishing/consuming your own streams without tying to any threading model (everything can be done on main thread for example). And it provides methods of mapping streams to others and even returning blocking APIs back to the user. Some methods in particular I was looking at were merge, delay, blockingIterable, rebatch and some others.

Vert.x is really beneficial with reactive programming built upon existing tools (HTTP, JDBC and others). It also has the event bus and others but this is built on the vert.x event loop, which we just don't need in Infinispan. We already have our own threading model and communication over JGroups.

My usage is to handle multiple streams of data coming from different locations and merging them together and doing some additional transformations. I could do this with either of the API, however RxJava already did a lot of heavy lifting I would have to do otherwise.

Also another side note is that it might be helpful for vert.x to change over to something more like the upcoming Flow API from Java 9 (maybe [4] which is pretty much identical currently). RxJava uses this as a base class for it's Publish/Subscribe.

 

Emmanuel

On 15 Jun 2017, at 23:20, William Burns <[hidden email]> wrote:

I was thinking more about [1] and I find that I was going to implement basically reactive streams. What we have now in github is similar but it uses a very crude method of blocking the thread to prevent back pressure. This can then cause severe issues as many users have found out when they don't close iterator.

Unfortunately reactive streams is just a spec. I am proposing to add RxJava [2] as a dependency [2] in the core module to provide access to reactive streams and the various conversion methods. This library adds a bunch of support for built in back pressure, transformations and much more which would reduce the amount of code I would need to write substantially.

In regards to timing, I am thinking this is too close for 9.1, so maybe 9.2 or higher.

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Reactive Streams + RxJava

Galder Zamarreño
In reply to this post by William Burns-3

--
Galder Zamarreño
Infinispan, Red Hat

> On 15 Jun 2017, at 23:20, William Burns <[hidden email]> wrote:
>
> I was thinking more about [1] and I find that I was going to implement basically reactive streams. What we have now in github is similar but it uses a very crude method of blocking the thread to prevent back pressure. This can then cause severe issues as many users have found out when they don't close iterator.
>
> Unfortunately reactive streams is just a spec.

Not just as spec. Java 9 will come with a set of interfaces that allows you to implement reactive streams:
http://download.java.net/java/jdk9/docs/api/java/util/concurrent/Flow.html

> I am proposing to add RxJava [2] as a dependency [2] in the core module to provide access to reactive streams and the various conversion methods. This library adds a bunch of support for built in back pressure, transformations and much more which would reduce the amount of code I would need to write substantially.

I don't think I like the idea of tying our core module to RxJava. Back when I was doing the functional map API we did consider using something like RxJava for the async multi-value streams, but decided against it to avoid tying ourselves to that.

A better alternative, if the APIs fit your use case, would be to depend on Java 9 APIs (oswego might have Java 8 version of those interfaces) and use maybe RxJava as the implementation of that API, if there's a suitable implementation for it.

If there's no choice but to use RxJava for the API part, I'd rather that happens in separate module to core.

Cheers,

>
> In regards to timing, I am thinking this is too close for 9.1, so maybe 9.2 or higher.
>
> What do you guys think?
>
> [1] https://issues.jboss.org/browse/ISPN-7865
> [2] https://github.com/ReactiveX/RxJava
> [3] https://mvnrepository.com/artifact/io.reactivex.rxjava2/rxjava/2.1.0
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Reactive Streams + RxJava

Galder Zamarreño
In reply to this post by William Burns-3

--
Galder Zamarreño
Infinispan, Red Hat

> On 19 Jun 2017, at 15:25, William Burns <[hidden email]> wrote:
>
>
>
> On Mon, Jun 19, 2017 at 8:34 AM Emmanuel Bernard <[hidden email]> wrote:
> You’re thinking about a pure implementation play, correct? RxJava or the Reactive Stream constructs would not be exposed to the user as API. Am I correct?
>
> Yes, that is correct. This is only for internal usage. My thought is maybe when we finally move to Java 9, we may expose the Flow API at some point, but nothing any time soon.

^ Ok, that wasn't clear in the original email.

If it's only for internal usage and no RxJava APIs are exposed, then I'm fine with it.

>  
> Also for posterity, we had backchannel chats about it and you said you felt vert.x was not necessarily addressing your needs. Could you elaborate a bit here?
>
> The main difference is that RxJava [1] has a full suite of utility methods around reactive streams [2]. Vert.x has some APIs around reactive streams [3], but it is rather limited.
>
> RxJava is about publishing/consuming your own streams without tying to any threading model (everything can be done on main thread for example). And it provides methods of mapping streams to others and even returning blocking APIs back to the user. Some methods in particular I was looking at were merge, delay, blockingIterable, rebatch and some others.
>
> Vert.x is really beneficial with reactive programming built upon existing tools (HTTP, JDBC and others). It also has the event bus and others but this is built on the vert.x event loop, which we just don't need in Infinispan. We already have our own threading model and communication over JGroups.
>
> My usage is to handle multiple streams of data coming from different locations and merging them together and doing some additional transformations. I could do this with either of the API, however RxJava already did a lot of heavy lifting I would have to do otherwise.
>
> Also another side note is that it might be helpful for vert.x to change over to something more like the upcoming Flow API from Java 9 (maybe [4] which is pretty much identical currently). RxJava uses this as a base class for it's Publish/Subscribe.
>
> [1] https://github.com/ReactiveX/RxJava/tree/2.x/src/main/java/io/reactivex
> [2] http://reactivex.io/RxJava/2.x/javadoc/io/reactivex/Flowable.html#blockingIterable()
> [3] https://github.com/eclipse/vert.x/tree/master/src/main/java/io/vertx/core/streams
> [4] https://mvnrepository.com/artifact/org.reactivestreams/reactive-streams/1.0.0
>  
>
> Emmanuel
>
>> On 15 Jun 2017, at 23:20, William Burns <[hidden email]> wrote:
>>
>> I was thinking more about [1] and I find that I was going to implement basically reactive streams. What we have now in github is similar but it uses a very crude method of blocking the thread to prevent back pressure. This can then cause severe issues as many users have found out when they don't close iterator.
>>
>> Unfortunately reactive streams is just a spec. I am proposing to add RxJava [2] as a dependency [2] in the core module to provide access to reactive streams and the various conversion methods. This library adds a bunch of support for built in back pressure, transformations and much more which would reduce the amount of code I would need to write substantially.
>>
>> In regards to timing, I am thinking this is too close for 9.1, so maybe 9.2 or higher.
>>
>> What do you guys think?
>>
>> [1] https://issues.jboss.org/browse/ISPN-7865
>> [2] https://github.com/ReactiveX/RxJava
>> [3] https://mvnrepository.com/artifact/io.reactivex.rxjava2/rxjava/2.1.0
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] Reactive Streams + RxJava

William Burns-3


On Tue, Jun 20, 2017 at 3:59 AM Galder Zamarreño <[hidden email]> wrote:

--
Galder Zamarreño
Infinispan, Red Hat

> On 19 Jun 2017, at 15:25, William Burns <[hidden email]> wrote:
>
>
>
> On Mon, Jun 19, 2017 at 8:34 AM Emmanuel Bernard <[hidden email]> wrote:
> You’re thinking about a pure implementation play, correct? RxJava or the Reactive Stream constructs would not be exposed to the user as API. Am I correct?
>
> Yes, that is correct. This is only for internal usage. My thought is maybe when we finally move to Java 9, we may expose the Flow API at some point, but nothing any time soon.

^ Ok, that wasn't clear in the original email.

If it's only for internal usage and no RxJava APIs are exposed, then I'm fine with it.

Yeah I am definitely not trying to expose any new API. And if we expose anything related to reactive streams I would recommend the reactive streams specification [1], which includes interfaces that the Java 9 Flow API is based off of.

But what I am proposing is the addition of RxJava for is to help with the internal handling of distributed iterator and possibly cache loader to reduce the number of threads.

Another side point is we could even use reactive streams for the remote iterator in the client (in fact it is a perfect fit there, much less complex). But I am not proposing that at this point as I don't really want to add a dependency in the client.

 

>
> Also for posterity, we had backchannel chats about it and you said you felt vert.x was not necessarily addressing your needs. Could you elaborate a bit here?
>
> The main difference is that RxJava [1] has a full suite of utility methods around reactive streams [2]. Vert.x has some APIs around reactive streams [3], but it is rather limited.
>
> RxJava is about publishing/consuming your own streams without tying to any threading model (everything can be done on main thread for example). And it provides methods of mapping streams to others and even returning blocking APIs back to the user. Some methods in particular I was looking at were merge, delay, blockingIterable, rebatch and some others.
>
> Vert.x is really beneficial with reactive programming built upon existing tools (HTTP, JDBC and others). It also has the event bus and others but this is built on the vert.x event loop, which we just don't need in Infinispan. We already have our own threading model and communication over JGroups.
>
> My usage is to handle multiple streams of data coming from different locations and merging them together and doing some additional transformations. I could do this with either of the API, however RxJava already did a lot of heavy lifting I would have to do otherwise.
>
> Also another side note is that it might be helpful for vert.x to change over to something more like the upcoming Flow API from Java 9 (maybe [4] which is pretty much identical currently). RxJava uses this as a base class for it's Publish/Subscribe.
>
> [1] https://github.com/ReactiveX/RxJava/tree/2.x/src/main/java/io/reactivex
> [2] http://reactivex.io/RxJava/2.x/javadoc/io/reactivex/Flowable.html#blockingIterable()
> [3] https://github.com/eclipse/vert.x/tree/master/src/main/java/io/vertx/core/streams
> [4] https://mvnrepository.com/artifact/org.reactivestreams/reactive-streams/1.0.0
>
>
> Emmanuel
>
>> On 15 Jun 2017, at 23:20, William Burns <[hidden email]> wrote:
>>
>> I was thinking more about [1] and I find that I was going to implement basically reactive streams. What we have now in github is similar but it uses a very crude method of blocking the thread to prevent back pressure. This can then cause severe issues as many users have found out when they don't close iterator.
>>
>> Unfortunately reactive streams is just a spec. I am proposing to add RxJava [2] as a dependency [2] in the core module to provide access to reactive streams and the various conversion methods. This library adds a bunch of support for built in back pressure, transformations and much more which would reduce the amount of code I would need to write substantially.
>>
>> In regards to timing, I am thinking this is too close for 9.1, so maybe 9.2 or higher.
>>
>> What do you guys think?
>>
>> [1] https://issues.jboss.org/browse/ISPN-7865
>> [2] https://github.com/ReactiveX/RxJava
>> [3] https://mvnrepository.com/artifact/io.reactivex.rxjava2/rxjava/2.1.0
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Loading...