[infinispan-dev] build broken

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[infinispan-dev] build broken

Sanne Grinovero-3
Hi all,
I know nobody cares about Query.. but some of the commits of
today/yesterday broke it.

I think both the pull-requestor and the reviewer should run a full
build before allowing any change to be pushed upstream; with a script
like this one:
https://gist.github.com/789588

it won't even prevent you to work on a different branch while tests
test are run, so while it takes some CPU, you can still look at other
stuff.

Sanne
_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] build broken

Manik Surtani
+1, good point.  My bad, I think I was the one who accepted this patch ….

On 9 Sep 2011, at 14:32, Sanne Grinovero wrote:

> Hi all,
> I know nobody cares about Query.. but some of the commits of
> today/yesterday broke it.
>
> I think both the pull-requestor and the reviewer should run a full
> build before allowing any change to be pushed upstream; with a script
> like this one:
> https://gist.github.com/789588
>
> it won't even prevent you to work on a different branch while tests
> test are run, so while it takes some CPU, you can still look at other
> stuff.
>
> Sanne
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
[hidden email]
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] build broken

Galder Zamarreno
Not sure what machine you have but tbh, if I run the tests on my MBP, I can hardly do other stuff, particularly development. I'd need a separate machine for that.

It's true that I should at least run the build to make sure the rest compiles but this stuff should be caught by CI too? Granted that it would not have picked it cos of https://issues.jboss.org/browse/ISPN-1381

Guys, we have CI for something. If something's wrong, it needs to be flagged, not wait for 1 month (that includes me of course).

On Sep 12, 2011, at 4:09 PM, Manik Surtani wrote:

> +1, good point.  My bad, I think I was the one who accepted this patch ….
>
> On 9 Sep 2011, at 14:32, Sanne Grinovero wrote:
>
>> Hi all,
>> I know nobody cares about Query.. but some of the commits of
>> today/yesterday broke it.
>>
>> I think both the pull-requestor and the reviewer should run a full
>> build before allowing any change to be pushed upstream; with a script
>> like this one:
>> https://gist.github.com/789588
>>
>> it won't even prevent you to work on a different branch while tests
>> test are run, so while it takes some CPU, you can still look at other
>> stuff.
>>
>> Sanne
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
> Manik Surtani
> [hidden email]
> twitter.com/maniksurtani
>
> Lead, Infinispan
> http://www.infinispan.org
>
>
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] build broken

Sanne Grinovero-3
On 13 September 2011 11:21, Galder Zamarreño <[hidden email]> wrote:
> Not sure what machine you have but tbh, if I run the tests on my MBP, I can hardly do other stuff, particularly development. I'd need a separate machine for that.

I have a dual core, likely it's not the hardware making the difference
:P It slows down the system a bit, fans spin up, but other
applications stay responsive enough.

> It's true that I should at least run the build to make sure the rest compiles but this stuff should be caught by CI too? Granted that it would not have picked it cos of https://issues.jboss.org/browse/ISPN-1381
>
> Guys, we have CI for something. If something's wrong, it needs to be flagged, not wait for 1 month (that includes me of course).

Right but the CI is worthless.
Since the tests fail so often that nobody reads the alarms it sends.
I'm still of the idea that master should always pass all tests - we've
been very close with just a couple of tests failing but I can't
remember having it ever seen all green/blue - not complaining and it's
my responsibility as well, just that we can't rely on CI for this
until we take the time to seriously review the testsuite.

Sanne

>
> On Sep 12, 2011, at 4:09 PM, Manik Surtani wrote:
>
>> +1, good point.  My bad, I think I was the one who accepted this patch ….
>>
>> On 9 Sep 2011, at 14:32, Sanne Grinovero wrote:
>>
>>> Hi all,
>>> I know nobody cares about Query.. but some of the commits of
>>> today/yesterday broke it.
>>>
>>> I think both the pull-requestor and the reviewer should run a full
>>> build before allowing any change to be pushed upstream; with a script
>>> like this one:
>>> https://gist.github.com/789588
>>>
>>> it won't even prevent you to work on a different branch while tests
>>> test are run, so while it takes some CPU, you can still look at other
>>> stuff.
>>>
>>> Sanne
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> --
>> Manik Surtani
>> [hidden email]
>> twitter.com/maniksurtani
>>
>> Lead, Infinispan
>> http://www.infinispan.org
>>
>>
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>
> --
> Galder Zamarreño
> Sr. Software Engineer
> Infinispan, JBoss Cache
>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] build broken

Galder Zamarreno

On Sep 13, 2011, at 12:14 PM, Sanne Grinovero wrote:

> On 13 September 2011 11:21, Galder Zamarreño <[hidden email]> wrote:
>> Not sure what machine you have but tbh, if I run the tests on my MBP, I can hardly do other stuff, particularly development. I'd need a separate machine for that.
>
> I have a dual core, likely it's not the hardware making the difference
> :P It slows down the system a bit, fans spin up, but other
> applications stay responsive enough.
>
>> It's true that I should at least run the build to make sure the rest compiles but this stuff should be caught by CI too? Granted that it would not have picked it cos of https://issues.jboss.org/browse/ISPN-1381
>>
>> Guys, we have CI for something. If something's wrong, it needs to be flagged, not wait for 1 month (that includes me of course).
>
> Right but the CI is worthless.
> Since the tests fail so often that nobody reads the alarms it sends.
> I'm still of the idea that master should always pass all tests - we've
> been very close with just a couple of tests failing but I can't
> remember having it ever seen all green/blue - not complaining and it's
> my responsibility as well, just that we can't rely on CI for this
> until we take the time to seriously review the testsuite.

I disagree :) - The CI is definitely worth it. Sure, some tests might need revisiting but a big majority of the tests are working fine:
https://infinispan.ci.cloudbees.com/job/Infinispan-master-JDK6-tcp/218/

We're down to about 13 failures in a testsuite of probably 2000 tests or so. That is very good already.

I don't think the testsuite is in such state that requires someone to spend a few days trying to get it work perfectly. It's desirable but we've got bigger issues to deal with IMO.

I'm more of a thinker that if you're working on something and you happen to encounter and test that fails randomly, fix it. You'll already have the knowledge of that code as a result of your work and will prob make it easier to fix the test or the underlying bug. IOW, I'm trying to be pragmatic here.

>
> Sanne
>
>>
>> On Sep 12, 2011, at 4:09 PM, Manik Surtani wrote:
>>
>>> +1, good point.  My bad, I think I was the one who accepted this patch ….
>>>
>>> On 9 Sep 2011, at 14:32, Sanne Grinovero wrote:
>>>
>>>> Hi all,
>>>> I know nobody cares about Query.. but some of the commits of
>>>> today/yesterday broke it.
>>>>
>>>> I think both the pull-requestor and the reviewer should run a full
>>>> build before allowing any change to be pushed upstream; with a script
>>>> like this one:
>>>> https://gist.github.com/789588
>>>>
>>>> it won't even prevent you to work on a different branch while tests
>>>> test are run, so while it takes some CPU, you can still look at other
>>>> stuff.
>>>>
>>>> Sanne
>>>> _______________________________________________
>>>> infinispan-dev mailing list
>>>> [hidden email]
>>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>>
>>> --
>>> Manik Surtani
>>> [hidden email]
>>> twitter.com/maniksurtani
>>>
>>> Lead, Infinispan
>>> http://www.infinispan.org
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> infinispan-dev mailing list
>>> [hidden email]
>>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>> --
>> Galder Zamarreño
>> Sr. Software Engineer
>> Infinispan, JBoss Cache
>>
>>
>> _______________________________________________
>> infinispan-dev mailing list
>> [hidden email]
>> https://lists.jboss.org/mailman/listinfo/infinispan-dev
>>
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Galder Zamarreño
Sr. Software Engineer
Infinispan, JBoss Cache


_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] build broken

Sanne Grinovero-3
On 13 September 2011 12:57, Galder Zamarreño <[hidden email]> wrote:

>
> On Sep 13, 2011, at 12:14 PM, Sanne Grinovero wrote:
>
>> On 13 September 2011 11:21, Galder Zamarreño <[hidden email]> wrote:
>>> Not sure what machine you have but tbh, if I run the tests on my MBP, I can hardly do other stuff, particularly development. I'd need a separate machine for that.
>>
>> I have a dual core, likely it's not the hardware making the difference
>> :P It slows down the system a bit, fans spin up, but other
>> applications stay responsive enough.
>>
>>> It's true that I should at least run the build to make sure the rest compiles but this stuff should be caught by CI too? Granted that it would not have picked it cos of https://issues.jboss.org/browse/ISPN-1381
>>>
>>> Guys, we have CI for something. If something's wrong, it needs to be flagged, not wait for 1 month (that includes me of course).
>>
>> Right but the CI is worthless.
>> Since the tests fail so often that nobody reads the alarms it sends.
>> I'm still of the idea that master should always pass all tests - we've
>> been very close with just a couple of tests failing but I can't
>> remember having it ever seen all green/blue - not complaining and it's
>> my responsibility as well, just that we can't rely on CI for this
>> until we take the time to seriously review the testsuite.
>
> I disagree :) - The CI is definitely worth it. Sure, some tests might need revisiting but a big majority of the tests are working fine:
> https://infinispan.ci.cloudbees.com/job/Infinispan-master-JDK6-tcp/218/
>
> We're down to about 13 failures in a testsuite of probably 2000 tests or so. That is very good already.
>
> I don't think the testsuite is in such state that requires someone to spend a few days trying to get it work perfectly. It's desirable but we've got bigger issues to deal with IMO.
>
> I'm more of a thinker that if you're working on something and you happen to encounter and test that fails randomly, fix it. You'll already have the knowledge of that code as a result of your work and will prob make it easier to fix the test or the underlying bug. IOW, I'm trying to be pragmatic here.

right and I agree with you that this is the best we can do right now.
But when the build broke, nobody noticed until the day after, and even
then it happened because by chance I needed a fresh build of Query.
I'm just guessing that since Jenkins is always complaining we don't
give enough attentions to his emails, making the CI less effective
than what it could be: on other projects a CI failure notification
makes me run to the computer to make sure it's immediately fixed, not
so with Infinispan as it's no news.

Sanne

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] build broken

Manik Surtani
I agree that the instability in the test suite removes the urgency to get things fixed, and that is a bad thing.

Which is why I think we should group the handful of occasionally unstable tests and put them in a separate test group so that they are not run all the time.  I'll look into this.


On 13 Sep 2011, at 12:12, Sanne Grinovero wrote:

> On 13 September 2011 12:57, Galder Zamarreño <[hidden email]> wrote:
>>
>> On Sep 13, 2011, at 12:14 PM, Sanne Grinovero wrote:
>>
>>> On 13 September 2011 11:21, Galder Zamarreño <[hidden email]> wrote:
>>>> Not sure what machine you have but tbh, if I run the tests on my MBP, I can hardly do other stuff, particularly development. I'd need a separate machine for that.
>>>
>>> I have a dual core, likely it's not the hardware making the difference
>>> :P It slows down the system a bit, fans spin up, but other
>>> applications stay responsive enough.
>>>
>>>> It's true that I should at least run the build to make sure the rest compiles but this stuff should be caught by CI too? Granted that it would not have picked it cos of https://issues.jboss.org/browse/ISPN-1381
>>>>
>>>> Guys, we have CI for something. If something's wrong, it needs to be flagged, not wait for 1 month (that includes me of course).
>>>
>>> Right but the CI is worthless.
>>> Since the tests fail so often that nobody reads the alarms it sends.
>>> I'm still of the idea that master should always pass all tests - we've
>>> been very close with just a couple of tests failing but I can't
>>> remember having it ever seen all green/blue - not complaining and it's
>>> my responsibility as well, just that we can't rely on CI for this
>>> until we take the time to seriously review the testsuite.
>>
>> I disagree :) - The CI is definitely worth it. Sure, some tests might need revisiting but a big majority of the tests are working fine:
>> https://infinispan.ci.cloudbees.com/job/Infinispan-master-JDK6-tcp/218/
>>
>> We're down to about 13 failures in a testsuite of probably 2000 tests or so. That is very good already.
>>
>> I don't think the testsuite is in such state that requires someone to spend a few days trying to get it work perfectly. It's desirable but we've got bigger issues to deal with IMO.
>>
>> I'm more of a thinker that if you're working on something and you happen to encounter and test that fails randomly, fix it. You'll already have the knowledge of that code as a result of your work and will prob make it easier to fix the test or the underlying bug. IOW, I'm trying to be pragmatic here.
>
> right and I agree with you that this is the best we can do right now.
> But when the build broke, nobody noticed until the day after, and even
> then it happened because by chance I needed a fresh build of Query.
> I'm just guessing that since Jenkins is always complaining we don't
> give enough attentions to his emails, making the CI less effective
> than what it could be: on other projects a CI failure notification
> makes me run to the computer to make sure it's immediately fixed, not
> so with Infinispan as it's no news.
>
> Sanne
>
> _______________________________________________
> infinispan-dev mailing list
> [hidden email]
> https://lists.jboss.org/mailman/listinfo/infinispan-dev

--
Manik Surtani
[hidden email]
twitter.com/maniksurtani

Lead, Infinispan
http://www.infinispan.org




_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] build broken

Sanne Grinovero-3
On 13 September 2011 13:15, Manik Surtani <[hidden email]> wrote:
> I agree that the instability in the test suite removes the urgency to get things fixed, and that is a bad thing.
>
> Which is why I think we should group the handful of occasionally unstable tests and put them in a separate test group so that they are not run all the time.  I'll look into this.

+100
I think it's acceptable to disable those tests which occasionally fail
and create a JIRA for each one to be re-enabled when we have time to
look into it. It also tracks what works with confidence vs. what might
be broken.

I don't always know if a failing test is "one of the usual ones", or a
new one which should ring an alarm bell, so this would be useful for
anyone but especially for me as I don't work on it everyday and have a
terrible memory.

Cheers,
Sanne

_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev
Reply | Threaded
Open this post in threaded view
|

Re: [infinispan-dev] build broken

Emmanuel Bernard
In reply to this post by Sanne Grinovero-3

On 9 sept. 2011, at 15:32, Sanne Grinovero wrote:

> I think both the pull-requestor and the reviewer should run a full
> build before allowing any change to be pushed upstream; with a script
> like this one:
> https://gist.github.com/789588
>
> it won't even prevent you to work on a different branch while tests
> test are run, so while it takes some CPU, you can still look at other
> stuff.

BTW, I've perfected / generalized this script to let it run *anything* on a Git clone of the current branch

https://gist.github.com/1218818

    remote.sh mvn clean install
    remote.sh gradle clean build
    remote.sh myscript.sh engage



_______________________________________________
infinispan-dev mailing list
[hidden email]
https://lists.jboss.org/mailman/listinfo/infinispan-dev