[infinispan-dev] HotRod client TCK

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] HotRod client TCK

Martin Gencur
Hello all,
we have been working on https://issues.jboss.org/browse/ISPN-7120.

Anna has finished the first step from the JIRA - collecting information
about tests in the Java HotRod client test suite (including server
integration tests) and it is now prepared for wider review.

She created a spreadsheet [1]. The spread sheet includes for each Java
test its name, the suggested target package in the TCK, whether to
include it in the TCK or not, and some other notes. The suggested
package also poses grouping for the tests (e.g. tck.query, tck.near,
tck.xsite, ...)

Let me add that right now the goal is not to create a true TCK [2]. The
goal is to make sure that all implementations of the HotRod protocol
have sufficient test coverage and possibly the same server side of the
client-server test (including the server version and configuration).

What are the next step?

* Please review the list (at least a quick look) and see if some of the
tests which are NOT suggested for the TCK should be added or vice versa.
* I suppose the next step would then be to check other implementations
(C#, C++, NodeJS, ..) and identify tests which are missing there (there
will surely be some).
* Gradually implement the missing tests in the other implementations
   Note: Here we should ensure that the server is configured in the same
way for all implementations. One way to achieve this (thanks Anna for
suggestion!) is to have a shell/batch scripts for CLI which would be
executed before the tests. This can probably be done for all impls. and
both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes
useless because it uses Creaper (Java) and we need a language-neutral
solution for configuring the server.

Some other notes:
* there are some duplicated tests in hotrod-client and server
integration test suites, in this case it probably makes sense to only
include in the TCK the server integration test
* tests from the hotrod-client module which are supposed to be part of
the TCK should be copied to the server integration test suite one day
(possibly later)

Please let us know what you think.

Thanks,
Martin


[1]
https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0
[2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit
[3] https://github.com/infinispan/infinispan/pull/5012
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] HotRod client TCK

Radim Vansa
Since these tests use real server(s), many of them test not only the
client behaviour (generating correct commands according to the
protocol), but server, too. While this is practical (we need to test
server somehow, too), there's nothing all the tests across languages
will have physically in common and all comparison is prone to human error.

If we want to test various implementations of the client, maybe it would
make sense to give the clients a fake server that will have just a
scenario of expected commands to receive and pre-defined responses. We
could use audit log to generate such scenario based on the actual Java
tests.

But then we'd have to test the actual behaviour on server, and we'd need
a way to issue the commands.

Just my 2c

Radim

On 04/11/2017 02:33 PM, Martin Gencur wrote:

> Hello all,
> we have been working on https://issues.jboss.org/browse/ISPN-7120.
>
> Anna has finished the first step from the JIRA - collecting information
> about tests in the Java HotRod client test suite (including server
> integration tests) and it is now prepared for wider review.
>
> She created a spreadsheet [1]. The spread sheet includes for each Java
> test its name, the suggested target package in the TCK, whether to
> include it in the TCK or not, and some other notes. The suggested
> package also poses grouping for the tests (e.g. tck.query, tck.near,
> tck.xsite, ...)
>
> Let me add that right now the goal is not to create a true TCK [2]. The
> goal is to make sure that all implementations of the HotRod protocol
> have sufficient test coverage and possibly the same server side of the
> client-server test (including the server version and configuration).
>
> What are the next step?
>
> * Please review the list (at least a quick look) and see if some of the
> tests which are NOT suggested for the TCK should be added or vice versa.
> * I suppose the next step would then be to check other implementations
> (C#, C++, NodeJS, ..) and identify tests which are missing there (there
> will surely be some).
> * Gradually implement the missing tests in the other implementations
>     Note: Here we should ensure that the server is configured in the same
> way for all implementations. One way to achieve this (thanks Anna for
> suggestion!) is to have a shell/batch scripts for CLI which would be
> executed before the tests. This can probably be done for all impls. and
> both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes
> useless because it uses Creaper (Java) and we need a language-neutral
> solution for configuring the server.
>
> Some other notes:
> * there are some duplicated tests in hotrod-client and server
> integration test suites, in this case it probably makes sense to only
> include in the TCK the server integration test
> * tests from the hotrod-client module which are supposed to be part of
> the TCK should be copied to the server integration test suite one day
> (possibly later)
>
> Please let us know what you think.
>
> Thanks,
> Martin
>
>
> [1]
> https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0
> [2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit
> [3] https://github.com/infinispan/infinispan/pull/5012
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] HotRod client TCK

Sebastian Laskawiec
I think that's a very good point Radim. But on the other hand I think we may treat it as an implementation detail. The TCK must run without Infinispan server. 

I also believe it would be beneficial to define as least 2-3 levels of support, e.g. basic (only basic operations, no consistent hash support), advanced (all features). This way we could validate some more exotic clients such as Go (https://github.com/ugol/infinispan-go).

And in my opinion, a final version of this document go to infinispan design repository: https://github.com/infinispan/infinispan-designs

Apart from that, that's huge effort and really great job!

On Tue, Apr 11, 2017 at 5:05 PM Radim Vansa <[hidden email]> wrote:
Since these tests use real server(s), many of them test not only the
client behaviour (generating correct commands according to the
protocol), but server, too. While this is practical (we need to test
server somehow, too), there's nothing all the tests across languages
will have physically in common and all comparison is prone to human error.

If we want to test various implementations of the client, maybe it would
make sense to give the clients a fake server that will have just a
scenario of expected commands to receive and pre-defined responses. We
could use audit log to generate such scenario based on the actual Java
tests.

But then we'd have to test the actual behaviour on server, and we'd need
a way to issue the commands.

Just my 2c

Radim

On 04/11/2017 02:33 PM, Martin Gencur wrote:
> Hello all,
> we have been working on https://issues.jboss.org/browse/ISPN-7120.
>
> Anna has finished the first step from the JIRA - collecting information
> about tests in the Java HotRod client test suite (including server
> integration tests) and it is now prepared for wider review.
>
> She created a spreadsheet [1]. The spread sheet includes for each Java
> test its name, the suggested target package in the TCK, whether to
> include it in the TCK or not, and some other notes. The suggested
> package also poses grouping for the tests (e.g. tck.query, tck.near,
> tck.xsite, ...)
>
> Let me add that right now the goal is not to create a true TCK [2]. The
> goal is to make sure that all implementations of the HotRod protocol
> have sufficient test coverage and possibly the same server side of the
> client-server test (including the server version and configuration).
>
> What are the next step?
>
> * Please review the list (at least a quick look) and see if some of the
> tests which are NOT suggested for the TCK should be added or vice versa.
> * I suppose the next step would then be to check other implementations
> (C#, C++, NodeJS, ..) and identify tests which are missing there (there
> will surely be some).
> * Gradually implement the missing tests in the other implementations
>     Note: Here we should ensure that the server is configured in the same
> way for all implementations. One way to achieve this (thanks Anna for
> suggestion!) is to have a shell/batch scripts for CLI which would be
> executed before the tests. This can probably be done for all impls. and
> both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes
> useless because it uses Creaper (Java) and we need a language-neutral
> solution for configuring the server.
>
> Some other notes:
> * there are some duplicated tests in hotrod-client and server
> integration test suites, in this case it probably makes sense to only
> include in the TCK the server integration test
> * tests from the hotrod-client module which are supposed to be part of
> the TCK should be copied to the server integration test suite one day
> (possibly later)
>
> Please let us know what you think.
>
> Thanks,
> Martin
>
>
> [1]
> https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0
> [2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit
> [3] https://github.com/infinispan/infinispan/pull/5012
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--

SEBASTIAN ŁASKAWIEC

INFINISPAN DEVELOPER


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] HotRod client TCK

Radim Vansa
On 04/14/2017 03:50 PM, Sebastian Laskawiec wrote:
> I think that's a very good point Radim. But on the other hand I think
> we may treat it as an implementation detail. The TCK must run without
> Infinispan server.

So the tests in TCK won't start/configure the server, that will be
managed elsewhere?

R.

>
> I also believe it would be beneficial to define as least 2-3 levels of
> support, e.g. basic (only basic operations, no consistent hash
> support), advanced (all features). This way we could validate some
> more exotic clients such as Go (https://github.com/ugol/infinispan-go).
>
> And in my opinion, a final version of this document go to infinispan
> design repository: https://github.com/infinispan/infinispan-designs
>
> Apart from that, that's huge effort and really great job!
>
> On Tue, Apr 11, 2017 at 5:05 PM Radim Vansa <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Since these tests use real server(s), many of them test not only the
>     client behaviour (generating correct commands according to the
>     protocol), but server, too. While this is practical (we need to test
>     server somehow, too), there's nothing all the tests across languages
>     will have physically in common and all comparison is prone to
>     human error.
>
>     If we want to test various implementations of the client, maybe it
>     would
>     make sense to give the clients a fake server that will have just a
>     scenario of expected commands to receive and pre-defined responses. We
>     could use audit log to generate such scenario based on the actual Java
>     tests.
>
>     But then we'd have to test the actual behaviour on server, and
>     we'd need
>     a way to issue the commands.
>
>     Just my 2c
>
>     Radim
>
>     On 04/11/2017 02:33 PM, Martin Gencur wrote:
>     > Hello all,
>     > we have been working on https://issues.jboss.org/browse/ISPN-7120.
>     >
>     > Anna has finished the first step from the JIRA - collecting
>     information
>     > about tests in the Java HotRod client test suite (including server
>     > integration tests) and it is now prepared for wider review.
>     >
>     > She created a spreadsheet [1]. The spread sheet includes for
>     each Java
>     > test its name, the suggested target package in the TCK, whether to
>     > include it in the TCK or not, and some other notes. The suggested
>     > package also poses grouping for the tests (e.g. tck.query, tck.near,
>     > tck.xsite, ...)
>     >
>     > Let me add that right now the goal is not to create a true TCK
>     [2]. The
>     > goal is to make sure that all implementations of the HotRod protocol
>     > have sufficient test coverage and possibly the same server side
>     of the
>     > client-server test (including the server version and configuration).
>     >
>     > What are the next step?
>     >
>     > * Please review the list (at least a quick look) and see if some
>     of the
>     > tests which are NOT suggested for the TCK should be added or
>     vice versa.
>     > * I suppose the next step would then be to check other
>     implementations
>     > (C#, C++, NodeJS, ..) and identify tests which are missing there
>     (there
>     > will surely be some).
>     > * Gradually implement the missing tests in the other implementations
>     >     Note: Here we should ensure that the server is configured in
>     the same
>     > way for all implementations. One way to achieve this (thanks
>     Anna for
>     > suggestion!) is to have a shell/batch scripts for CLI which would be
>     > executed before the tests. This can probably be done for all
>     impls. and
>     > both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes
>     > useless because it uses Creaper (Java) and we need a
>     language-neutral
>     > solution for configuring the server.
>     >
>     > Some other notes:
>     > * there are some duplicated tests in hotrod-client and server
>     > integration test suites, in this case it probably makes sense to
>     only
>     > include in the TCK the server integration test
>     > * tests from the hotrod-client module which are supposed to be
>     part of
>     > the TCK should be copied to the server integration test suite
>     one day
>     > (possibly later)
>     >
>     > Please let us know what you think.
>     >
>     > Thanks,
>     > Martin
>     >
>     >
>     > [1]
>     >
>     https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0
>     > [2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit
>     > [3] https://github.com/infinispan/infinispan/pull/5012
>     > _______________________________________________
>     > infinispan-dev mailing list
>     > [hidden email]
>     <mailto:[hidden email]>
>     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>     --
>     Radim Vansa <[hidden email] <mailto:[hidden email]>>
>     JBoss Performance Team
>
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
>
> SEBASTIANŁASKAWIEC
>
> INFINISPAN DEVELOPER
>
> Red HatEMEA <https://www.redhat.com/>
>
> <https://red.ht/sig>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] HotRod client TCK

Galder Zamarreño
In reply to this post by Radim Vansa
I think there's some value in Radim's suggestion. The email was not fully clear to me initially but after reading a few times I understood what he was referring to. @Radim, correct me if I'm wrong...

Right now clients verify that they behave as expected, e.g. JS client uses its asserts, Java client uses other asserts. What Radim is trying to say is that there needs to be a way to verify they work adequately independent of their implementations.

So, the only way to do that is to verify it at the server level. Not sure what exactly he means by the fake server, but more than a fake server, I'd be more inclined to modify the server to that it can somehow act as TCK verifier. This is to avoid having to reimplement transport logic, protocol decoder...etc in a new fake server.

Cheers,
--
Galder Zamarreño
Infinispan, Red Hat

> On 11 Apr 2017, at 15:57, Radim Vansa <[hidden email]> wrote:
>
> Since these tests use real server(s), many of them test not only the
> client behaviour (generating correct commands according to the
> protocol), but server, too. While this is practical (we need to test
> server somehow, too), there's nothing all the tests across languages
> will have physically in common and all comparison is prone to human error.
>
> If we want to test various implementations of the client, maybe it would
> make sense to give the clients a fake server that will have just a
> scenario of expected commands to receive and pre-defined responses. We
> could use audit log to generate such scenario based on the actual Java
> tests.
>
> But then we'd have to test the actual behaviour on server, and we'd need
> a way to issue the commands.
>
> Just my 2c
>
> Radim
>
> On 04/11/2017 02:33 PM, Martin Gencur wrote:
>> Hello all,
>> we have been working on https://issues.jboss.org/browse/ISPN-7120.
>>
>> Anna has finished the first step from the JIRA - collecting information
>> about tests in the Java HotRod client test suite (including server
>> integration tests) and it is now prepared for wider review.
>>
>> She created a spreadsheet [1]. The spread sheet includes for each Java
>> test its name, the suggested target package in the TCK, whether to
>> include it in the TCK or not, and some other notes. The suggested
>> package also poses grouping for the tests (e.g. tck.query, tck.near,
>> tck.xsite, ...)
>>
>> Let me add that right now the goal is not to create a true TCK [2]. The
>> goal is to make sure that all implementations of the HotRod protocol
>> have sufficient test coverage and possibly the same server side of the
>> client-server test (including the server version and configuration).
>>
>> What are the next step?
>>
>> * Please review the list (at least a quick look) and see if some of the
>> tests which are NOT suggested for the TCK should be added or vice versa.
>> * I suppose the next step would then be to check other implementations
>> (C#, C++, NodeJS, ..) and identify tests which are missing there (there
>> will surely be some).
>> * Gradually implement the missing tests in the other implementations
>>    Note: Here we should ensure that the server is configured in the same
>> way for all implementations. One way to achieve this (thanks Anna for
>> suggestion!) is to have a shell/batch scripts for CLI which would be
>> executed before the tests. This can probably be done for all impls. and
>> both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes
>> useless because it uses Creaper (Java) and we need a language-neutral
>> solution for configuring the server.
>>
>> Some other notes:
>> * there are some duplicated tests in hotrod-client and server
>> integration test suites, in this case it probably makes sense to only
>> include in the TCK the server integration test
>> * tests from the hotrod-client module which are supposed to be part of
>> the TCK should be copied to the server integration test suite one day
>> (possibly later)
>>
>> Please let us know what you think.
>>
>> Thanks,
>> Martin
>>
>>
>> [1]
>> https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0
>> [2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit
>> [3] https://github.com/infinispan/infinispan/pull/5012
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <[hidden email]>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] HotRod client TCK

Galder Zamarreño
In reply to this post by Martin Gencur
Btw, thanks Anna for working on this!

I've had a look at the list and I have some questions:

* HotRodAsyncReplicationTest: I don't think it should be a client TCK test. There's nothing the client does differently compared to executing against a sync repl cache. If anything, it's a server TCK test since it verifies that a put sent by a HR client gets replicated. The same applies to all the test of local vs REPl vs DIST tests.

* LockingTest: same story, this is a client+server integration test, I don't think it's a client TCK test. If anything, it's a server TCK test. It verifies that if a client sends a put, the entry is locked.

* MixedExpiry*Test: it's dependant on the server configuration, not really a client TCK test IMO. I think the only client TCK tests that deal with expiry should only verify that the entry is expirable if the client decides to make it expirable.

* ClientListenerRemoveOnStopTest: Not sure this is a client TCK test. Yeah, it verifies that the client removes its listeners on stop, but it's not a Hot Rod protocol TCK test. Going back to what Radim said, how are you going to verify each client does this? What we can verify for all clients easily is they send the commands to remove the client servers to the server. Maybe for these and below client specific logic related tests, as Martin suggesteds, we go with the approach of just verifying that tests exist.

* Protobuf marshaller tests: client specific and testing client-side marshalling logic. Same reasons above.

* Near caching tests: client specific and testing client-side near caching logic. Same issues above.

* Topology change tests: I consider these TCK tests cos you could think that if the server sends a new topology, the client's next command should have the ID of this topology in its header.

* Failover/Retry tests: client specific and testing client-side retry logic. Same issues above, how do you verify it works accross the board for all clients?

* Socket timeout tests: again these are client specific...

I think in general it'd be a good idea to try to verify somehow most of the TCK via some server-side logic, as Radim hinted, and where that's not possible, revert to just verifying the client has tests to cover certain scenarios.

Cheers,
--
Galder Zamarreño
Infinispan, Red Hat

> On 11 Apr 2017, at 14:33, Martin Gencur <[hidden email]> wrote:
>
> Hello all,
> we have been working on https://issues.jboss.org/browse/ISPN-7120.
>
> Anna has finished the first step from the JIRA - collecting information
> about tests in the Java HotRod client test suite (including server
> integration tests) and it is now prepared for wider review.
>
> She created a spreadsheet [1]. The spread sheet includes for each Java
> test its name, the suggested target package in the TCK, whether to
> include it in the TCK or not, and some other notes. The suggested
> package also poses grouping for the tests (e.g. tck.query, tck.near,
> tck.xsite, ...)
>
> Let me add that right now the goal is not to create a true TCK [2]. The
> goal is to make sure that all implementations of the HotRod protocol
> have sufficient test coverage and possibly the same server side of the
> client-server test (including the server version and configuration).
>
> What are the next step?
>
> * Please review the list (at least a quick look) and see if some of the
> tests which are NOT suggested for the TCK should be added or vice versa.
> * I suppose the next step would then be to check other implementations
> (C#, C++, NodeJS, ..) and identify tests which are missing there (there
> will surely be some).
> * Gradually implement the missing tests in the other implementations
>   Note: Here we should ensure that the server is configured in the same
> way for all implementations. One way to achieve this (thanks Anna for
> suggestion!) is to have a shell/batch scripts for CLI which would be
> executed before the tests. This can probably be done for all impls. and
> both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes
> useless because it uses Creaper (Java) and we need a language-neutral
> solution for configuring the server.
>
> Some other notes:
> * there are some duplicated tests in hotrod-client and server
> integration test suites, in this case it probably makes sense to only
> include in the TCK the server integration test
> * tests from the hotrod-client module which are supposed to be part of
> the TCK should be copied to the server integration test suite one day
> (possibly later)
>
> Please let us know what you think.
>
> Thanks,
> Martin
>
>
> [1]
> https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0
> [2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit
> [3] https://github.com/infinispan/infinispan/pull/5012
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] HotRod client TCK

Dan Berindei
On Mon, May 8, 2017 at 2:32 PM, Galder Zamarreño <[hidden email]> wrote:

> Btw, thanks Anna for working on this!
>
> I've had a look at the list and I have some questions:
>
> * HotRodAsyncReplicationTest: I don't think it should be a client TCK test. There's nothing the client does differently compared to executing against a sync repl cache. If anything, it's a server TCK test since it verifies that a put sent by a HR client gets replicated. The same applies to all the test of local vs REPl vs DIST tests.
>
> * LockingTest: same story, this is a client+server integration test, I don't think it's a client TCK test. If anything, it's a server TCK test. It verifies that if a client sends a put, the entry is locked.
>
> * MixedExpiry*Test: it's dependant on the server configuration, not really a client TCK test IMO. I think the only client TCK tests that deal with expiry should only verify that the entry is expirable if the client decides to make it expirable.
>

I think they should be included, because this is part of the HotRod
wire specification:

* +0x0002+    = use cache-level configured default lifespan
* +0x0004+    = use cache-level configured default max idle

> * ClientListenerRemoveOnStopTest: Not sure this is a client TCK test. Yeah, it verifies that the client removes its listeners on stop, but it's not a Hot Rod protocol TCK test. Going back to what Radim said, how are you going to verify each client does this? What we can verify for all clients easily is they send the commands to remove the client servers to the server. Maybe for these and below client specific logic related tests, as Martin suggesteds, we go with the approach of just verifying that tests exist.
>
> * Protobuf marshaller tests: client specific and testing client-side marshalling logic. Same reasons above.
>
> * Near caching tests: client specific and testing client-side near caching logic. Same issues above.
>
> * Topology change tests: I consider these TCK tests cos you could think that if the server sends a new topology, the client's next command should have the ID of this topology in its header.
>
> * Failover/Retry tests: client specific and testing client-side retry logic. Same issues above, how do you verify it works accross the board for all clients?
>
> * Socket timeout tests: again these are client specific...
>
> I think in general it'd be a good idea to try to verify somehow most of the TCK via some server-side logic, as Radim hinted, and where that's not possible, revert to just verifying the client has tests to cover certain scenarios.

+1

Dan

>
> Cheers,
> --
> Galder Zamarreño
> Infinispan, Red Hat
>
>> On 11 Apr 2017, at 14:33, Martin Gencur <[hidden email]> wrote:
>>
>> Hello all,
>> we have been working on https://issues.jboss.org/browse/ISPN-7120.
>>
>> Anna has finished the first step from the JIRA - collecting information
>> about tests in the Java HotRod client test suite (including server
>> integration tests) and it is now prepared for wider review.
>>
>> She created a spreadsheet [1]. The spread sheet includes for each Java
>> test its name, the suggested target package in the TCK, whether to
>> include it in the TCK or not, and some other notes. The suggested
>> package also poses grouping for the tests (e.g. tck.query, tck.near,
>> tck.xsite, ...)
>>
>> Let me add that right now the goal is not to create a true TCK [2]. The
>> goal is to make sure that all implementations of the HotRod protocol
>> have sufficient test coverage and possibly the same server side of the
>> client-server test (including the server version and configuration).
>>
>> What are the next step?
>>
>> * Please review the list (at least a quick look) and see if some of the
>> tests which are NOT suggested for the TCK should be added or vice versa.
>> * I suppose the next step would then be to check other implementations
>> (C#, C++, NodeJS, ..) and identify tests which are missing there (there
>> will surely be some).
>> * Gradually implement the missing tests in the other implementations
>>   Note: Here we should ensure that the server is configured in the same
>> way for all implementations. One way to achieve this (thanks Anna for
>> suggestion!) is to have a shell/batch scripts for CLI which would be
>> executed before the tests. This can probably be done for all impls. and
>> both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes
>> useless because it uses Creaper (Java) and we need a language-neutral
>> solution for configuring the server.
>>
>> Some other notes:
>> * there are some duplicated tests in hotrod-client and server
>> integration test suites, in this case it probably makes sense to only
>> include in the TCK the server integration test
>> * tests from the hotrod-client module which are supposed to be part of
>> the TCK should be copied to the server integration test suite one day
>> (possibly later)
>>
>> Please let us know what you think.
>>
>> Thanks,
>> Martin
>>
>>
>> [1]
>> https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0
>> [2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit
>> [3] https://github.com/infinispan/infinispan/pull/5012
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] HotRod client TCK

Radim Vansa
In reply to this post by Galder Zamarreño
On 05/08/2017 07:10 AM, Galder Zamarreño wrote:
> I think there's some value in Radim's suggestion. The email was not fully clear to me initially but after reading a few times I understood what he was referring to. @Radim, correct me if I'm wrong...
>
> Right now clients verify that they behave as expected, e.g. JS client uses its asserts, Java client uses other asserts. What Radim is trying to say is that there needs to be a way to verify they work adequately independent of their implementations.
>
> So, the only way to do that is to verify it at the server level. Not sure what exactly he means by the fake server, but more than a fake server, I'd be more inclined to modify the server to that it can somehow act as TCK verifier. This is to avoid having to reimplement transport logic, protocol decoder...etc in a new fake server.

I think you got the idea. I am not trying to push any particular
implementation of the "fake server" - you could just tweak existing one,
but the purest and most deterministic approach would be having a script
that could look like:

expect connection A to serverX/any server
expect receive from A <base64 encoded bytes/hex encoded bytes>
send to A <... bytes>
expect connection A closed

Implementing a server that interprets such script isn't that complex;
you don't have to deal with protocol decoder (what's transport logic on
server?), because you just expect and send bytes.

Radim

>
> Cheers,
> --
> Galder Zamarreño
> Infinispan, Red Hat
>
>> On 11 Apr 2017, at 15:57, Radim Vansa <[hidden email]> wrote:
>>
>> Since these tests use real server(s), many of them test not only the
>> client behaviour (generating correct commands according to the
>> protocol), but server, too. While this is practical (we need to test
>> server somehow, too), there's nothing all the tests across languages
>> will have physically in common and all comparison is prone to human error.
>>
>> If we want to test various implementations of the client, maybe it would
>> make sense to give the clients a fake server that will have just a
>> scenario of expected commands to receive and pre-defined responses. We
>> could use audit log to generate such scenario based on the actual Java
>> tests.
>>
>> But then we'd have to test the actual behaviour on server, and we'd need
>> a way to issue the commands.
>>
>> Just my 2c
>>
>> Radim
>>
>> On 04/11/2017 02:33 PM, Martin Gencur wrote:
>>> Hello all,
>>> we have been working on https://issues.jboss.org/browse/ISPN-7120.
>>>
>>> Anna has finished the first step from the JIRA - collecting information
>>> about tests in the Java HotRod client test suite (including server
>>> integration tests) and it is now prepared for wider review.
>>>
>>> She created a spreadsheet [1]. The spread sheet includes for each Java
>>> test its name, the suggested target package in the TCK, whether to
>>> include it in the TCK or not, and some other notes. The suggested
>>> package also poses grouping for the tests (e.g. tck.query, tck.near,
>>> tck.xsite, ...)
>>>
>>> Let me add that right now the goal is not to create a true TCK [2]. The
>>> goal is to make sure that all implementations of the HotRod protocol
>>> have sufficient test coverage and possibly the same server side of the
>>> client-server test (including the server version and configuration).
>>>
>>> What are the next step?
>>>
>>> * Please review the list (at least a quick look) and see if some of the
>>> tests which are NOT suggested for the TCK should be added or vice versa.
>>> * I suppose the next step would then be to check other implementations
>>> (C#, C++, NodeJS, ..) and identify tests which are missing there (there
>>> will surely be some).
>>> * Gradually implement the missing tests in the other implementations
>>>     Note: Here we should ensure that the server is configured in the same
>>> way for all implementations. One way to achieve this (thanks Anna for
>>> suggestion!) is to have a shell/batch scripts for CLI which would be
>>> executed before the tests. This can probably be done for all impls. and
>>> both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes
>>> useless because it uses Creaper (Java) and we need a language-neutral
>>> solution for configuring the server.
>>>
>>> Some other notes:
>>> * there are some duplicated tests in hotrod-client and server
>>> integration test suites, in this case it probably makes sense to only
>>> include in the TCK the server integration test
>>> * tests from the hotrod-client module which are supposed to be part of
>>> the TCK should be copied to the server integration test suite one day
>>> (possibly later)
>>>
>>> Please let us know what you think.
>>>
>>> Thanks,
>>> Martin
>>>
>>>
>>> [1]
>>> https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0
>>> [2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit
>>> [3] https://github.com/infinispan/infinispan/pull/5012
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> --
>> Radim Vansa <[hidden email]>
>> JBoss Performance Team
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] HotRod client TCK

Gustavo Fernandes-2
In reply to this post by Galder Zamarreño


On Mon, May 8, 2017 at 12:10 PM, Galder Zamarreño <[hidden email]> wrote:
I think there's some value in Radim's suggestion. The email was not fully clear to me initially but after reading a few times I understood what he was referring to. @Radim, correct me if I'm wrong...

Right now clients verify that they behave as expected, e.g. JS client uses its asserts, Java client uses other asserts. What Radim is trying to say is that there needs to be a way to verify they work adequately independent of their implementations.

So, the only way to do that is to verify it at the server level.
Not sure what exactly he means by the fake server, but more than a fake server, I'd be more inclined to modify the server to that it can somehow act as TCK verifier.


We had a thread about Hot Rod testing last year, and another possible strategy is to use real unmodified servers have the TCK written once in a neutral language, compile/distribute it to each of the clients, where the tests would run as part of the build. More details on [1]

[1] http://infinispan-developer-list.980875.n3.nabble.com/infinispan-dev-Hot-Rod-testing-tt4031152.html#a4031213


Gustavo
 
This is to avoid having to reimplement transport logic, protocol decoder...etc in a new fake server.

Cheers,
--
Galder Zamarreño
Infinispan, Red Hat

> On 11 Apr 2017, at 15:57, Radim Vansa <[hidden email]> wrote:
>
> Since these tests use real server(s), many of them test not only the
> client behaviour (generating correct commands according to the
> protocol), but server, too. While this is practical (we need to test
> server somehow, too), there's nothing all the tests across languages
> will have physically in common and all comparison is prone to human error.
>
> If we want to test various implementations of the client, maybe it would
> make sense to give the clients a fake server that will have just a
> scenario of expected commands to receive and pre-defined responses. We
> could use audit log to generate such scenario based on the actual Java
> tests.
>
> But then we'd have to test the actual behaviour on server, and we'd need
> a way to issue the commands.
>
> Just my 2c
>
> Radim
>
> On 04/11/2017 02:33 PM, Martin Gencur wrote:
>> Hello all,
>> we have been working on https://issues.jboss.org/browse/ISPN-7120.
>>
>> Anna has finished the first step from the JIRA - collecting information
>> about tests in the Java HotRod client test suite (including server
>> integration tests) and it is now prepared for wider review.
>>
>> She created a spreadsheet [1]. The spread sheet includes for each Java
>> test its name, the suggested target package in the TCK, whether to
>> include it in the TCK or not, and some other notes. The suggested
>> package also poses grouping for the tests (e.g. tck.query, tck.near,
>> tck.xsite, ...)
>>
>> Let me add that right now the goal is not to create a true TCK [2]. The
>> goal is to make sure that all implementations of the HotRod protocol
>> have sufficient test coverage and possibly the same server side of the
>> client-server test (including the server version and configuration).
>>
>> What are the next step?
>>
>> * Please review the list (at least a quick look) and see if some of the
>> tests which are NOT suggested for the TCK should be added or vice versa.
>> * I suppose the next step would then be to check other implementations
>> (C#, C++, NodeJS, ..) and identify tests which are missing there (there
>> will surely be some).
>> * Gradually implement the missing tests in the other implementations
>>    Note: Here we should ensure that the server is configured in the same
>> way for all implementations. One way to achieve this (thanks Anna for
>> suggestion!) is to have a shell/batch scripts for CLI which would be
>> executed before the tests. This can probably be done for all impls. and
>> both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes
>> useless because it uses Creaper (Java) and we need a language-neutral
>> solution for configuring the server.
>>
>> Some other notes:
>> * there are some duplicated tests in hotrod-client and server
>> integration test suites, in this case it probably makes sense to only
>> include in the TCK the server integration test
>> * tests from the hotrod-client module which are supposed to be part of
>> the TCK should be copied to the server integration test suite one day
>> (possibly later)
>>
>> Please let us know what you think.
>>
>> Thanks,
>> Martin
>>
>>
>> [1]
>> https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0
>> [2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit
>> [3] https://github.com/infinispan/infinispan/pull/5012
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <[hidden email]>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] HotRod client TCK

Martin Gencur
In reply to this post by Galder Zamarreño
Hi,
thanks for looking at the list of tests and thanks for suggestions.
We'll incorporate them in the final list of tests.

What Radim suggest has some advantages and some drawbacks, but I see
this as an addition to the client TCK.
This approach can verify that the client sends some predefined commands
with predefined values but does that really verify that the user will
get the expected results? I'm not so sure. There can be some client-side
logic that does other modifications. Here I see room for a lot of missed
bugs.

I'd say we need real client-side tests which verify the client behavior
from the user perspective. Let me also add that I see the Java client
test suite as an etalon (reference standard). The server side behavior
has been tested through the Java test suite and other clients don't need
to test that again, IMO. The goal is to test the client-side. Having a
pre-defined configuration for server that would be used in all client
implementation tests should provide some common ground for the tests.

As to the real TCK suggested by Gustavo, I remember the discussion and
we discussed that at the clustering meeting last year.
Since the Java HotRod client test suite has about 1500 tests (maybe more
now?) we would need to rewrite all the tests in the new language. And
I'm not sure running those tests with various implementations would be
without problems. I'd love to see this working but I'm afraid that we
don't have time and resources to do this any time soon.

Martin

On 8.5.2017 13:32, Galder Zamarreño wrote:
> I think in general it'd be a good idea to try to verify somehow most of the TCK via some server-side logic, as Radim hinted, and where that's not possible, revert to just verifying the client has tests to cover certain scenarios.

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] HotRod client TCK

Anna Manukyan
In reply to this post by Galder Zamarreño
Hey Galder,
thanks a lot for review. I have updated the document based on your suggestions.

Best regards,
Anna.

----- Original Message -----
From: "Galder Zamarreño" <[hidden email]>
To: "infinispan -Dev List" <[hidden email]>
Sent: Monday, May 8, 2017 1:32:13 PM
Subject: Re: [infinispan-dev] HotRod client TCK

Btw, thanks Anna for working on this!

I've had a look at the list and I have some questions:

* HotRodAsyncReplicationTest: I don't think it should be a client TCK test. There's nothing the client does differently compared to executing against a sync repl cache. If anything, it's a server TCK test since it verifies that a put sent by a HR client gets replicated. The same applies to all the test of local vs REPl vs DIST tests.

* LockingTest: same story, this is a client+server integration test, I don't think it's a client TCK test. If anything, it's a server TCK test. It verifies that if a client sends a put, the entry is locked.

* MixedExpiry*Test: it's dependant on the server configuration, not really a client TCK test IMO. I think the only client TCK tests that deal with expiry should only verify that the entry is expirable if the client decides to make it expirable.

* ClientListenerRemoveOnStopTest: Not sure this is a client TCK test. Yeah, it verifies that the client removes its listeners on stop, but it's not a Hot Rod protocol TCK test. Going back to what Radim said, how are you going to verify each client does this? What we can verify for all clients easily is they send the commands to remove the client servers to the server. Maybe for these and below client specific logic related tests, as Martin suggesteds, we go with the approach of just verifying that tests exist.

* Protobuf marshaller tests: client specific and testing client-side marshalling logic. Same reasons above.

* Near caching tests: client specific and testing client-side near caching logic. Same issues above.

* Topology change tests: I consider these TCK tests cos you could think that if the server sends a new topology, the client's next command should have the ID of this topology in its header.

* Failover/Retry tests: client specific and testing client-side retry logic. Same issues above, how do you verify it works accross the board for all clients?

* Socket timeout tests: again these are client specific...

I think in general it'd be a good idea to try to verify somehow most of the TCK via some server-side logic, as Radim hinted, and where that's not possible, revert to just verifying the client has tests to cover certain scenarios.

Cheers,
--
Galder Zamarreño
Infinispan, Red Hat

> On 11 Apr 2017, at 14:33, Martin Gencur <[hidden email]> wrote:
>
> Hello all,
> we have been working on https://issues.jboss.org/browse/ISPN-7120.
>
> Anna has finished the first step from the JIRA - collecting information
> about tests in the Java HotRod client test suite (including server
> integration tests) and it is now prepared for wider review.
>
> She created a spreadsheet [1]. The spread sheet includes for each Java
> test its name, the suggested target package in the TCK, whether to
> include it in the TCK or not, and some other notes. The suggested
> package also poses grouping for the tests (e.g. tck.query, tck.near,
> tck.xsite, ...)
>
> Let me add that right now the goal is not to create a true TCK [2]. The
> goal is to make sure that all implementations of the HotRod protocol
> have sufficient test coverage and possibly the same server side of the
> client-server test (including the server version and configuration).
>
> What are the next step?
>
> * Please review the list (at least a quick look) and see if some of the
> tests which are NOT suggested for the TCK should be added or vice versa.
> * I suppose the next step would then be to check other implementations
> (C#, C++, NodeJS, ..) and identify tests which are missing there (there
> will surely be some).
> * Gradually implement the missing tests in the other implementations
>   Note: Here we should ensure that the server is configured in the same
> way for all implementations. One way to achieve this (thanks Anna for
> suggestion!) is to have a shell/batch scripts for CLI which would be
> executed before the tests. This can probably be done for all impls. and
> both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes
> useless because it uses Creaper (Java) and we need a language-neutral
> solution for configuring the server.
>
> Some other notes:
> * there are some duplicated tests in hotrod-client and server
> integration test suites, in this case it probably makes sense to only
> include in the TCK the server integration test
> * tests from the hotrod-client module which are supposed to be part of
> the TCK should be copied to the server integration test suite one day
> (possibly later)
>
> Please let us know what you think.
>
> Thanks,
> Martin
>
>
> [1]
> https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0
> [2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit
> [3] https://github.com/infinispan/infinispan/pull/5012
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev