VG is triggering 2 jobs for one commit

[DVir669]

Hi,

Sometimes I have a scenario where I have one commit that triggers 2 builds in Jenkins.
Looking at the timestamp of the branch and which branch we're building I see the same:
Jenkins shows:
Build #925
vg_df842a9_Genadi_Rabinovich_new_1456394583204
Build #926
vg_df842a9_Genadi_Rabinovich_new_1456394583204

Both are running.

VG UI shows:
vg_7e1363d_oobuild001 release/master-release release/master-release dvir.mizrahi@hpe.com RUNNING Retry 25-02-2016 12:20:53 25-02-2016 12:21:07 N/A master-build/927/

vg_df842a9_Genadi_Rabinovich master vg_168e994_Dorinel_Utiu genadi@hpe.com RUNNING Retry 25-02-2016 12:03:03 25-02-2016 12:03:40 N/A master-build/925/

VG logs show:
25/02/16 10:03:49:049 INFO [com.verigreen.collector.common.log4j.VerigreenLogger] *** com.verigreen.collector.buildverification.JenkinsUpdater::register (tid: 23) *** Observer registered: CommitItem [
_branchDescriptor=BranchDescriptor [_protectedBranch=refs/heads/master, _newBranch=vg_df842a9_Genadi_Rabinovich, _commiter=genadi@hpe.com],
_mergedBranchName=vg_df842a9_Genadi_Rabinovich_new_1456394583204, _status=Running, _creationTime=2016-02-25 10:03:03.204, _runTime=2016-02-25 10:03:40.403, _endTime=null, _buildUrl=0, _isDone=false, _parent=f9608dfa-ad38-4f4b-99a8-bd0d7dd114aa, _child=null, _timeoutCounter=0, _retriableCounter=0, _triggeredAttempt=true, _buildNumberToStop=0]
*** END ***
25/02/16 10:03:59:059 INFO [com.verigreen.collector.common.log4j.VerigreenLogger] *** com.verigreen.collector.decision.decisionmaker.ProtectedBranchesDecisionMaker::decide (tid: 21) *** There are 2 not done item(s)
*** END ***
25/02/16 10:03:59:059 ERROR [com.verigreen.collector.common.log4j.VerigreenLogger] *** com.verigreen.collector.buildverification.JenkinsVerifier::triggerJob (tid: 23) *** Failed to trigger build for job [Master-Build] with branch [vg_df842a9_Genadi_Rabinovich_new_1456394583204] *** END ***
java.net.SocketTimeoutException: Read timed out
at java.net.SocketInputStream.socketRead0(Native Method) ~[?:1.7.0_65]
at java.net.SocketInputStream.read(SocketInputStream.java:152) ~[?:1.7.0_65]
at java.net.SocketInputStream.read(SocketInputStream.java:122) ~[?:1.7.0_65]
at org.apache.http.impl.io.AbstractSessionInputBuffer.fillBuffer(AbstractSessionInputBuffer.java:166) ~[httpcore-4.2.2.jar:4.2.2]
at org.apache.http.impl.io.SocketInputBuffer.fillBuffer(SocketInputBuffer.java:90) ~[httpcore-4.2.2.jar:4.2.2]
at org.apache.http.impl.io.AbstractSessionInputBuffer.readLine(AbstractSessionInputBuffer.java:281) ~[httpcore-4.2.2.jar:4.2.2]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:92) ~[httpclient-4.2.2.jar:4.2.2]
at org.apache.http.impl.conn.DefaultHttpResponseParser.parseHead(DefaultHttpResponseParser.java:62) ~[httpclient-4.2.2.jar:4.2.2]
at org.apache.http.impl.io.AbstractMessageParser.parse(AbstractMessageParser.java:254) ~[httpcore-4.2.2.jar:4.2.2]
at org.apache.http.impl.AbstractHttpClientConnection.receiveResponseHeader(AbstractHttpClientConnection.java:289) ~[httpcore-4.2.2.jar:4.2.2]
at org.apache.http.impl.conn.DefaultClientConnection.receiveResponseHeader(DefaultClientConnection.java:252) ~[httpclient-4.2.2.jar:4.2.2]
at org.apache.http.impl.conn.ManagedClientConnectionImpl.receiveResponseHeader(ManagedClientConnectionImpl.java:191) ~[httpclient-4.2.2.jar:4.2.2]
at org.apache.http.protocol.HttpRequestExecutor.doReceiveResponse(HttpRequestExecutor.java:300) ~[httpcore-4.2.2.jar:4.2.2]
at org.apache.http.protocol.HttpRequestExecutor.execute(HttpRequestExecutor.java:127) ~[httpcore-4.2.2.jar:4.2.2]
at org.apache.http.impl.client.DefaultRequestDirector.tryExecute(DefaultRequestDirector.java:712) ~[httpclient-4.2.2.jar:4.2.2]
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:517) ~[httpclient-4.2.2.jar:4.2.2]
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:906) ~[httpclient-4.2.2.jar:4.2.2]
at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:805) ~[httpclient-4.2.2.jar:4.2.2]
at com.offbytwo.jenkins.client.JenkinsHttpClient.post(JenkinsHttpClient.java:185) ~[jenkins-client-0.2.1.302.1.jar:0.2.1.302.1]
at com.offbytwo.jenkins.model.Job.build(Job.java:58) ~[jenkins-client-0.2.1.302.1.jar:0.2.1.302.1]
at com.verigreen.collector.buildverification.JenkinsVerifier.triggerJob(JenkinsVerifier.java:118) [verigreen-collector-impl-2.5.6.jar:?]
at com.verigreen.collector.jobs.CallJenkinsJob.calllingJenkinsForCreate(CallJenkinsJob.java:116) [verigreen-collector-impl-2.5.6.jar:?]
at com.verigreen.collector.jobs.CallJenkinsJob.execute(CallJenkinsJob.java:53) [verigreen-collector-impl-2.5.6.jar:?]
at org.quartz.core.JobRunShell.run(JobRunShell.java:202) [quartz-2.2.1.jar:?]
at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:573) [quartz-2.2.1.jar:?]

Thanks,
Dvir

[soninob]

@DVir669 - It is a known issue.
This could happen mostly because of a pending build.
Pending builds are not presented in Jenkins API (when requesting data on running builds). Therefor, VG is not receiving data, and base on your policy (in the config.properties), retrigger another build.
May I suggest that you will give your Jenkins a breathing room? Meaning - if you set the queue manager to 4, then set the number of executors in Jenkins to 6 or 5. It will reduce the problem.
The fact that this is a known issue is because we are not revising the running job for duplication or creating a garbage collector - feel free to create a PR for that :-)

[soninob]

Hey @DVir669, can we close this?

[DVir669]

I have written a perl script that checks the previous run and make sure it is a unique one before triggering the build process - I can commit it here (if I can find it :)) otherwise you can close this issue.

[gromano-dev]

Dvir, so you put this script in a pre-commit hook ?

[DVir669]

No, running this first thing in Jenkins, it scans the build and commits (if I remember correctly) and check if it should run a build or not. I'm not in the same job so I'm not sure I still have it (I can check if you want) but it is quite simple to implement

[soninob]

Whould love to see a PR for this man