Quantcast

[infinispan-dev] HotRod client TCK

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[infinispan-dev] HotRod client TCK

Martin Gencur
Hello all,
we have been working on https://issues.jboss.org/browse/ISPN-7120.

Anna has finished the first step from the JIRA - collecting information
about tests in the Java HotRod client test suite (including server
integration tests) and it is now prepared for wider review.

She created a spreadsheet [1]. The spread sheet includes for each Java
test its name, the suggested target package in the TCK, whether to
include it in the TCK or not, and some other notes. The suggested
package also poses grouping for the tests (e.g. tck.query, tck.near,
tck.xsite, ...)

Let me add that right now the goal is not to create a true TCK [2]. The
goal is to make sure that all implementations of the HotRod protocol
have sufficient test coverage and possibly the same server side of the
client-server test (including the server version and configuration).

What are the next step?

* Please review the list (at least a quick look) and see if some of the
tests which are NOT suggested for the TCK should be added or vice versa.
* I suppose the next step would then be to check other implementations
(C#, C++, NodeJS, ..) and identify tests which are missing there (there
will surely be some).
* Gradually implement the missing tests in the other implementations
   Note: Here we should ensure that the server is configured in the same
way for all implementations. One way to achieve this (thanks Anna for
suggestion!) is to have a shell/batch scripts for CLI which would be
executed before the tests. This can probably be done for all impls. and
both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes
useless because it uses Creaper (Java) and we need a language-neutral
solution for configuring the server.

Some other notes:
* there are some duplicated tests in hotrod-client and server
integration test suites, in this case it probably makes sense to only
include in the TCK the server integration test
* tests from the hotrod-client module which are supposed to be part of
the TCK should be copied to the server integration test suite one day
(possibly later)

Please let us know what you think.

Thanks,
Martin


[1]
https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0
[2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit
[3] https://github.com/infinispan/infinispan/pull/5012
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] HotRod client TCK

Radim Vansa
Since these tests use real server(s), many of them test not only the
client behaviour (generating correct commands according to the
protocol), but server, too. While this is practical (we need to test
server somehow, too), there's nothing all the tests across languages
will have physically in common and all comparison is prone to human error.

If we want to test various implementations of the client, maybe it would
make sense to give the clients a fake server that will have just a
scenario of expected commands to receive and pre-defined responses. We
could use audit log to generate such scenario based on the actual Java
tests.

But then we'd have to test the actual behaviour on server, and we'd need
a way to issue the commands.

Just my 2c

Radim

On 04/11/2017 02:33 PM, Martin Gencur wrote:

> Hello all,
> we have been working on https://issues.jboss.org/browse/ISPN-7120.
>
> Anna has finished the first step from the JIRA - collecting information
> about tests in the Java HotRod client test suite (including server
> integration tests) and it is now prepared for wider review.
>
> She created a spreadsheet [1]. The spread sheet includes for each Java
> test its name, the suggested target package in the TCK, whether to
> include it in the TCK or not, and some other notes. The suggested
> package also poses grouping for the tests (e.g. tck.query, tck.near,
> tck.xsite, ...)
>
> Let me add that right now the goal is not to create a true TCK [2]. The
> goal is to make sure that all implementations of the HotRod protocol
> have sufficient test coverage and possibly the same server side of the
> client-server test (including the server version and configuration).
>
> What are the next step?
>
> * Please review the list (at least a quick look) and see if some of the
> tests which are NOT suggested for the TCK should be added or vice versa.
> * I suppose the next step would then be to check other implementations
> (C#, C++, NodeJS, ..) and identify tests which are missing there (there
> will surely be some).
> * Gradually implement the missing tests in the other implementations
>     Note: Here we should ensure that the server is configured in the same
> way for all implementations. One way to achieve this (thanks Anna for
> suggestion!) is to have a shell/batch scripts for CLI which would be
> executed before the tests. This can probably be done for all impls. and
> both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes
> useless because it uses Creaper (Java) and we need a language-neutral
> solution for configuring the server.
>
> Some other notes:
> * there are some duplicated tests in hotrod-client and server
> integration test suites, in this case it probably makes sense to only
> include in the TCK the server integration test
> * tests from the hotrod-client module which are supposed to be part of
> the TCK should be copied to the server integration test suite one day
> (possibly later)
>
> Please let us know what you think.
>
> Thanks,
> Martin
>
>
> [1]
> https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0
> [2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit
> [3] https://github.com/infinispan/infinispan/pull/5012
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] HotRod client TCK

Sebastian Laskawiec
I think that's a very good point Radim. But on the other hand I think we may treat it as an implementation detail. The TCK must run without Infinispan server. 

I also believe it would be beneficial to define as least 2-3 levels of support, e.g. basic (only basic operations, no consistent hash support), advanced (all features). This way we could validate some more exotic clients such as Go (https://github.com/ugol/infinispan-go).

And in my opinion, a final version of this document go to infinispan design repository: https://github.com/infinispan/infinispan-designs

Apart from that, that's huge effort and really great job!

On Tue, Apr 11, 2017 at 5:05 PM Radim Vansa <[hidden email]> wrote:
Since these tests use real server(s), many of them test not only the
client behaviour (generating correct commands according to the
protocol), but server, too. While this is practical (we need to test
server somehow, too), there's nothing all the tests across languages
will have physically in common and all comparison is prone to human error.

If we want to test various implementations of the client, maybe it would
make sense to give the clients a fake server that will have just a
scenario of expected commands to receive and pre-defined responses. We
could use audit log to generate such scenario based on the actual Java
tests.

But then we'd have to test the actual behaviour on server, and we'd need
a way to issue the commands.

Just my 2c

Radim

On 04/11/2017 02:33 PM, Martin Gencur wrote:
> Hello all,
> we have been working on https://issues.jboss.org/browse/ISPN-7120.
>
> Anna has finished the first step from the JIRA - collecting information
> about tests in the Java HotRod client test suite (including server
> integration tests) and it is now prepared for wider review.
>
> She created a spreadsheet [1]. The spread sheet includes for each Java
> test its name, the suggested target package in the TCK, whether to
> include it in the TCK or not, and some other notes. The suggested
> package also poses grouping for the tests (e.g. tck.query, tck.near,
> tck.xsite, ...)
>
> Let me add that right now the goal is not to create a true TCK [2]. The
> goal is to make sure that all implementations of the HotRod protocol
> have sufficient test coverage and possibly the same server side of the
> client-server test (including the server version and configuration).
>
> What are the next step?
>
> * Please review the list (at least a quick look) and see if some of the
> tests which are NOT suggested for the TCK should be added or vice versa.
> * I suppose the next step would then be to check other implementations
> (C#, C++, NodeJS, ..) and identify tests which are missing there (there
> will surely be some).
> * Gradually implement the missing tests in the other implementations
>     Note: Here we should ensure that the server is configured in the same
> way for all implementations. One way to achieve this (thanks Anna for
> suggestion!) is to have a shell/batch scripts for CLI which would be
> executed before the tests. This can probably be done for all impls. and
> both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes
> useless because it uses Creaper (Java) and we need a language-neutral
> solution for configuring the server.
>
> Some other notes:
> * there are some duplicated tests in hotrod-client and server
> integration test suites, in this case it probably makes sense to only
> include in the TCK the server integration test
> * tests from the hotrod-client module which are supposed to be part of
> the TCK should be copied to the server integration test suite one day
> (possibly later)
>
> Please let us know what you think.
>
> Thanks,
> Martin
>
>
> [1]
> https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0
> [2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit
> [3] https://github.com/infinispan/infinispan/pull/5012
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
--

SEBASTIAN ŁASKAWIEC

INFINISPAN DEVELOPER


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [infinispan-dev] HotRod client TCK

Radim Vansa
On 04/14/2017 03:50 PM, Sebastian Laskawiec wrote:
> I think that's a very good point Radim. But on the other hand I think
> we may treat it as an implementation detail. The TCK must run without
> Infinispan server.

So the tests in TCK won't start/configure the server, that will be
managed elsewhere?

R.

>
> I also believe it would be beneficial to define as least 2-3 levels of
> support, e.g. basic (only basic operations, no consistent hash
> support), advanced (all features). This way we could validate some
> more exotic clients such as Go (https://github.com/ugol/infinispan-go).
>
> And in my opinion, a final version of this document go to infinispan
> design repository: https://github.com/infinispan/infinispan-designs
>
> Apart from that, that's huge effort and really great job!
>
> On Tue, Apr 11, 2017 at 5:05 PM Radim Vansa <[hidden email]
> <mailto:[hidden email]>> wrote:
>
>     Since these tests use real server(s), many of them test not only the
>     client behaviour (generating correct commands according to the
>     protocol), but server, too. While this is practical (we need to test
>     server somehow, too), there's nothing all the tests across languages
>     will have physically in common and all comparison is prone to
>     human error.
>
>     If we want to test various implementations of the client, maybe it
>     would
>     make sense to give the clients a fake server that will have just a
>     scenario of expected commands to receive and pre-defined responses. We
>     could use audit log to generate such scenario based on the actual Java
>     tests.
>
>     But then we'd have to test the actual behaviour on server, and
>     we'd need
>     a way to issue the commands.
>
>     Just my 2c
>
>     Radim
>
>     On 04/11/2017 02:33 PM, Martin Gencur wrote:
>     > Hello all,
>     > we have been working on https://issues.jboss.org/browse/ISPN-7120.
>     >
>     > Anna has finished the first step from the JIRA - collecting
>     information
>     > about tests in the Java HotRod client test suite (including server
>     > integration tests) and it is now prepared for wider review.
>     >
>     > She created a spreadsheet [1]. The spread sheet includes for
>     each Java
>     > test its name, the suggested target package in the TCK, whether to
>     > include it in the TCK or not, and some other notes. The suggested
>     > package also poses grouping for the tests (e.g. tck.query, tck.near,
>     > tck.xsite, ...)
>     >
>     > Let me add that right now the goal is not to create a true TCK
>     [2]. The
>     > goal is to make sure that all implementations of the HotRod protocol
>     > have sufficient test coverage and possibly the same server side
>     of the
>     > client-server test (including the server version and configuration).
>     >
>     > What are the next step?
>     >
>     > * Please review the list (at least a quick look) and see if some
>     of the
>     > tests which are NOT suggested for the TCK should be added or
>     vice versa.
>     > * I suppose the next step would then be to check other
>     implementations
>     > (C#, C++, NodeJS, ..) and identify tests which are missing there
>     (there
>     > will surely be some).
>     > * Gradually implement the missing tests in the other implementations
>     >     Note: Here we should ensure that the server is configured in
>     the same
>     > way for all implementations. One way to achieve this (thanks
>     Anna for
>     > suggestion!) is to have a shell/batch scripts for CLI which would be
>     > executed before the tests. This can probably be done for all
>     impls. and
>     > both UNIX/WINDOWS. I also realize that my PR for ISPN [3] becomes
>     > useless because it uses Creaper (Java) and we need a
>     language-neutral
>     > solution for configuring the server.
>     >
>     > Some other notes:
>     > * there are some duplicated tests in hotrod-client and server
>     > integration test suites, in this case it probably makes sense to
>     only
>     > include in the TCK the server integration test
>     > * tests from the hotrod-client module which are supposed to be
>     part of
>     > the TCK should be copied to the server integration test suite
>     one day
>     > (possibly later)
>     >
>     > Please let us know what you think.
>     >
>     > Thanks,
>     > Martin
>     >
>     >
>     > [1]
>     >
>     https://docs.google.com/spreadsheets/d/1bZBBi5m4oLL4lBTZhdRbIC_EA0giQNDZWzFNPWrF5G4/edit#gid=0
>     > [2] https://en.wikipedia.org/wiki/Technology_Compatibility_Kit
>     > [3] https://github.com/infinispan/infinispan/pull/5012
>     > _______________________________________________
>     > infinispan-dev mailing list
>     > [hidden email]
>     <mailto:[hidden email]>
>     > https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
>     --
>     Radim Vansa <[hidden email] <mailto:[hidden email]>>
>     JBoss Performance Team
>
>     _______________________________________________
>     infinispan-dev mailing list
>     [hidden email] <mailto:[hidden email]>
>     https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
>
> SEBASTIANŁASKAWIEC
>
> INFINISPAN DEVELOPER
>
> Red HatEMEA <https://www.redhat.com/>
>
> <https://red.ht/sig>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Loading...