[infinispan-dev] Adding tests for new cache mode

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] Adding tests for new cache mode

Radim Vansa
Hi,

I've decided to start working on Scattered Cache [1][2] POC. I'd like to
use most of the tests for distributed mode, but just extending
DistXxxTest with ScatteredXxxTest and overriding getCacheMode() seems
quite inelegant, though this is a common practice for repl/dist tests. I
had similar problem with Simple Cache, but I didn't need as many tests
for that.

@Parameters are not used as much in our testsuite - is there any reason
for that? And is there any better way, if I want just test everything
and exclude those tests where it does not make sense to run the test as
well?

Suggestions are welcome.

Radim

[1] https://issues.jboss.org/browse/ISPN-6645
[2] https://github.com/infinispan/infinispan/wiki/Scattered-Cache-design-doc

--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Adding tests for new cache mode

Tristan Tarrant-2
On 17/05/2016 14:10, Radim Vansa wrote:

> Hi,
>
> I've decided to start working on Scattered Cache [1][2] POC. I'd like to
> use most of the tests for distributed mode, but just extending
> DistXxxTest with ScatteredXxxTest and overriding getCacheMode() seems
> quite inelegant, though this is a common practice for repl/dist tests. I
> had similar problem with Simple Cache, but I didn't need as many tests
> for that.
>
> @Parameters are not used as much in our testsuite - is there any reason
Not using @Parameters is a mistake, IMHO, so if you're willing to
convert the relevant ones, that would be lovely.

Tristan
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Adding tests for new cache mode

Dan Berindei
On Tue, May 17, 2016 at 3:26 PM, Tristan Tarrant <[hidden email]> wrote:

> On 17/05/2016 14:10, Radim Vansa wrote:
>> Hi,
>>
>> I've decided to start working on Scattered Cache [1][2] POC. I'd like to
>> use most of the tests for distributed mode, but just extending
>> DistXxxTest with ScatteredXxxTest and overriding getCacheMode() seems
>> quite inelegant, though this is a common practice for repl/dist tests. I
>> had similar problem with Simple Cache, but I didn't need as many tests
>> for that.
>>
>> @Parameters are not used as much in our testsuite - is there any reason
> Not using @Parameters is a mistake, IMHO, so if you're willing to
> convert the relevant ones, that would be lovely.
>

+1 to switch to @Parameters as many tests as you want!

Dan
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Adding tests for new cache mode

Radim Vansa
Uuuh... if it was just so easy. Now I've played with @Factory,
@DataProvider and others (since we want to parameterize whole class, not
just invocation of method), and it's broken - I can't simply rename the
test according to parameters, it behaves differently in IntelliJ and on
command line and also with different versions of TestNG. But that's just
aesthetic concern.

Another problem is the resource tracking AbstractInfinispanTest uses:

@BeforeTest/AfterTest is ran only once per class, not per instance
@BeforeClass/AfterClass is ran once per instance, but in a loop on the
instances within the same thread, before the methods are executed

The tests are then run in the order instance1.test1(),
instance2.test1(), instance1.test2() - so this needs to be reordered
using IMethodInterceptor. Luckily, the BeforeClass invocation is lazy
and AfterClass is eager, so these methods are invoked when these should be.

So, at this point [1] I managed to get the tests running fine, and if I
omit the testName in @Test, it is reported correctly in
target/surefire-reports/. However, the output that's printed when I run
maven test does not involve the parameter, and in IntelliJ the test
subsequent result gets just marked with (x) where x is some number. When
testName is set, it's repeated for all results in reports, in IntelliJ
as well (with (x) for each method invocation), and commandline output is
not changed.

If anyone knows how to fix that, go for it - I don't know what IntelliJ
or the surefire reporter picks. Just a hint - ITest interface won't help
you, that just spoils everything.

Radim

[1] https://github.com/rvansa/infinispan/tree/t_test_factory

On 05/18/2016 05:02 PM, Dan Berindei wrote:

> On Tue, May 17, 2016 at 3:26 PM, Tristan Tarrant <[hidden email]> wrote:
>> On 17/05/2016 14:10, Radim Vansa wrote:
>>> Hi,
>>>
>>> I've decided to start working on Scattered Cache [1][2] POC. I'd like to
>>> use most of the tests for distributed mode, but just extending
>>> DistXxxTest with ScatteredXxxTest and overriding getCacheMode() seems
>>> quite inelegant, though this is a common practice for repl/dist tests. I
>>> had similar problem with Simple Cache, but I didn't need as many tests
>>> for that.
>>>
>>> @Parameters are not used as much in our testsuite - is there any reason
>> Not using @Parameters is a mistake, IMHO, so if you're willing to
>> convert the relevant ones, that would be lovely.
>>
> +1 to switch to @Parameters as many tests as you want!
>
> Dan
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Adding tests for new cache mode

Sanne Grinovero-3
I'm sorry, I can't help you but you're pretty much describing while
the Lucene and Query modules use JUnit exclusively.

Rather than spreading the @Parameter virus, I'd rather see a tendency
to move things to JUnit. It doesn't have some of the TestNG features
but it's easy to extend - I just think that for a project like this
one is better of by making custom and reliable extensions than to use
a general purpose cryptic framework.

2cents..

Sanne

On 19 May 2016 at 16:05, Radim Vansa <[hidden email]> wrote:

> Uuuh... if it was just so easy. Now I've played with @Factory,
> @DataProvider and others (since we want to parameterize whole class, not
> just invocation of method), and it's broken - I can't simply rename the
> test according to parameters, it behaves differently in IntelliJ and on
> command line and also with different versions of TestNG. But that's just
> aesthetic concern.
>
> Another problem is the resource tracking AbstractInfinispanTest uses:
>
> @BeforeTest/AfterTest is ran only once per class, not per instance
> @BeforeClass/AfterClass is ran once per instance, but in a loop on the
> instances within the same thread, before the methods are executed
>
> The tests are then run in the order instance1.test1(),
> instance2.test1(), instance1.test2() - so this needs to be reordered
> using IMethodInterceptor. Luckily, the BeforeClass invocation is lazy
> and AfterClass is eager, so these methods are invoked when these should be.
>
> So, at this point [1] I managed to get the tests running fine, and if I
> omit the testName in @Test, it is reported correctly in
> target/surefire-reports/. However, the output that's printed when I run
> maven test does not involve the parameter, and in IntelliJ the test
> subsequent result gets just marked with (x) where x is some number. When
> testName is set, it's repeated for all results in reports, in IntelliJ
> as well (with (x) for each method invocation), and commandline output is
> not changed.
>
> If anyone knows how to fix that, go for it - I don't know what IntelliJ
> or the surefire reporter picks. Just a hint - ITest interface won't help
> you, that just spoils everything.
>
> Radim
>
> [1] https://github.com/rvansa/infinispan/tree/t_test_factory
>
> On 05/18/2016 05:02 PM, Dan Berindei wrote:
>> On Tue, May 17, 2016 at 3:26 PM, Tristan Tarrant <[hidden email]> wrote:
>>> On 17/05/2016 14:10, Radim Vansa wrote:
>>>> Hi,
>>>>
>>>> I've decided to start working on Scattered Cache [1][2] POC. I'd like to
>>>> use most of the tests for distributed mode, but just extending
>>>> DistXxxTest with ScatteredXxxTest and overriding getCacheMode() seems
>>>> quite inelegant, though this is a common practice for repl/dist tests. I
>>>> had similar problem with Simple Cache, but I didn't need as many tests
>>>> for that.
>>>>
>>>> @Parameters are not used as much in our testsuite - is there any reason
>>> Not using @Parameters is a mistake, IMHO, so if you're willing to
>>> convert the relevant ones, that would be lovely.
>>>
>> +1 to switch to @Parameters as many tests as you want!
>>
>> Dan
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
>
> --
> Radim Vansa <[hidden email]>
> JBoss Performance Team
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] Adding tests for new cache mode

Radim Vansa
Thanks for the insight, Sanne.

Has this thread died, or are there any further comments or ideas? (does
anyone understand those integrations with Maven or Intellij?)

Radim

On 05/19/2016 05:47 PM, Sanne Grinovero wrote:

> I'm sorry, I can't help you but you're pretty much describing while
> the Lucene and Query modules use JUnit exclusively.
>
> Rather than spreading the @Parameter virus, I'd rather see a tendency
> to move things to JUnit. It doesn't have some of the TestNG features
> but it's easy to extend - I just think that for a project like this
> one is better of by making custom and reliable extensions than to use
> a general purpose cryptic framework.
>
> 2cents..
>
> Sanne
>
> On 19 May 2016 at 16:05, Radim Vansa <[hidden email]> wrote:
>> Uuuh... if it was just so easy. Now I've played with @Factory,
>> @DataProvider and others (since we want to parameterize whole class, not
>> just invocation of method), and it's broken - I can't simply rename the
>> test according to parameters, it behaves differently in IntelliJ and on
>> command line and also with different versions of TestNG. But that's just
>> aesthetic concern.
>>
>> Another problem is the resource tracking AbstractInfinispanTest uses:
>>
>> @BeforeTest/AfterTest is ran only once per class, not per instance
>> @BeforeClass/AfterClass is ran once per instance, but in a loop on the
>> instances within the same thread, before the methods are executed
>>
>> The tests are then run in the order instance1.test1(),
>> instance2.test1(), instance1.test2() - so this needs to be reordered
>> using IMethodInterceptor. Luckily, the BeforeClass invocation is lazy
>> and AfterClass is eager, so these methods are invoked when these should be.
>>
>> So, at this point [1] I managed to get the tests running fine, and if I
>> omit the testName in @Test, it is reported correctly in
>> target/surefire-reports/. However, the output that's printed when I run
>> maven test does not involve the parameter, and in IntelliJ the test
>> subsequent result gets just marked with (x) where x is some number. When
>> testName is set, it's repeated for all results in reports, in IntelliJ
>> as well (with (x) for each method invocation), and commandline output is
>> not changed.
>>
>> If anyone knows how to fix that, go for it - I don't know what IntelliJ
>> or the surefire reporter picks. Just a hint - ITest interface won't help
>> you, that just spoils everything.
>>
>> Radim
>>
>> [1] https://github.com/rvansa/infinispan/tree/t_test_factory
>>
>> On 05/18/2016 05:02 PM, Dan Berindei wrote:
>>> On Tue, May 17, 2016 at 3:26 PM, Tristan Tarrant <[hidden email]> wrote:
>>>> On 17/05/2016 14:10, Radim Vansa wrote:
>>>>> Hi,
>>>>>
>>>>> I've decided to start working on Scattered Cache [1][2] POC. I'd like to
>>>>> use most of the tests for distributed mode, but just extending
>>>>> DistXxxTest with ScatteredXxxTest and overriding getCacheMode() seems
>>>>> quite inelegant, though this is a common practice for repl/dist tests. I
>>>>> had similar problem with Simple Cache, but I didn't need as many tests
>>>>> for that.
>>>>>
>>>>> @Parameters are not used as much in our testsuite - is there any reason
>>>> Not using @Parameters is a mistake, IMHO, so if you're willing to
>>>> convert the relevant ones, that would be lovely.
>>>>
>>> +1 to switch to @Parameters as many tests as you want!
>>>
>>> Dan
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> --
>> Radim Vansa <[hidden email]>
>> JBoss Performance Team
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev


--
Radim Vansa <[hidden email]>
JBoss Performance Team

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev