Testing Team Meeting – Dec 07 2012

December 6, 2012

Location: #jquery-meeting on Freenode

Attending: Scott, Corey, Jörn

Time: Noon ET

QUnit

TestSwarm

  • Lots of trouble with BrowserStack lately. Apparently their broken VMs only get fixed when we report the issue. Corey got IE7 and Safari 6 VMs fixed
  • A few jobs, especially in IE8 fail to submit results with “”Error connecting to server”
  • TestSwarm rewrite well on the way. Is now using ua-parser, much less maintenance in TestSwarm itself and testswarm-browserstack.

testswarm-browserstack

  • Rewrite finished. Much less maintenance required here as well. More advanced and machine-readable log output (Splunk!) and smarter worker spawn management, and more (see commit log for more details).

Mergeatron

Experimenting for jquery core

  • To be moved to a separate testswarm account
  • Refactor grunt-testswarm to not need changes for mergeatron/PR.

Jenkins builds are put in directory /build/\/PR-\<#>. This is a problem as those are non-unique (more than 1 commit per PR). Should use the commit hash like we do for other builds. Though builds are synchronous, a run can be reset later in time. If the directory has changed, the re-run will be incorrect.

  • Also, job name: “pull #123” -> “pull #123 – commit #af04”.

Testing Team Meeting – Nov 23 2012

Making good progress towards TestSwarm 1.0 and BrowserStack/Jenkins integration. The switch to ua-parser is complete, allowing us to start testing against mobile browsers along with getting rid of a lot of maintenance overhead.

Testing Team Meeting – Oct 25 2012

October 25, 2012

Location: #jquery-meeting on Freenode

Attending: Jörn, Dave, Scott

Time: Noon ET

QUnit

  • A few issues to review, mostly feature ideas

TestSwarm

  • Create new browserSets for UI (no IE6) and Core 2.0 (no IE6, 7 and 8)
  • Add IE10 and Safari 6

Testing Team Meeting – Sep 28 2012

September 27, 2012

Location: #jquery-meeting on Freenode

Attending: Jörn, Richard, Timo

Time: Noon ET

QUnit

PRs and issues to review

  • #317 is looking for an assertion method to compare objects with different constructors
  • #320 has an example of loading tests asynchronously. Recommendation is to use autostart=false, then start(), to make that work. Other ideas?
  • Will land pull #323 unless anyone has concerns – just quotes “throws” to make old interpreters not throw up
  • #326 asks for in “inconclusive” assertion, aka “needs a test”. Opinions?
  • #314 – throw up or log a failed assertion when calling start() while already running?
  • Related to #320 and #314 is #309

TestSwarm

  • Blog post about TestSwarm
  • 1.0 Roadmap
  • web site in November? Also needs new logo: https://github.com/jquery/testswarm/issues/128
  • Timo is working on testswarm-browserstack rewrite, will land update soon, makes it a lot more reliable
  • Up next: Improve keep-alive to get rid of timed-out clients faster; should make proper use of the ping-system implemented in TestSwarm recently
  • Richard still working on ua-parser ticket: https://github.com/jquery/testswarm/issues/187

botio/mergatron

Testing Team Meeting – Sep 13 2012

September 13, 2012

Location: #jquery-meeting on Freenode

Attending: Dave, Jörn, Richard, Corey, Scott

Time: Noon ET

QUnit

1.10 works, core had to fix some issues where selectors weren’t scope to #qunit-fixture

PRs and issues to review

TestSwarm

mergatron

  • Alternative to botio for testing Pull Requests

Corey to stage that next week: https://github.com/SnapInteractive/mergeatron

Testing Team Meeting – Aug 30 2012

August 30, 2012

Location: #jquery-meeting on Freenode

Attending: Scott, Dave, John, Jörn

Time: Noon ET

QUnit

1.10.0 is live! Core to update, UI updated, Mobile to update

Will launch some nice updates for qunitjs.com and api.qunitjs.com today or tomorrow

PRs and issues to review

TestSwarm

  • Timo is back. We should have updates soon.
  • Core still has random ajax failures, Mobile is dealing with IE issues

Testing Team Meeting – Aug 16 2012

August 16, 2012

Location: #jquery-meeting on Freenode

Attending: Scott, Dave, John, Rick, Jörn

Time: Noon ET

QUnit

Landed module filter select. Now it is trivial to filter by module, so much more convenient.

A few PRs and issues to review

TestSwarm

  • Jenkins/TestSwarm/BrowserStack running pretty stable, lots of green
  • Maintaining browser mappings is annoying. Will get a little easier once testswarm-browserstack is moved to jquery orga. Maybe a bit more easier when moving TestSwarm to ua-parser.

Testing Team Meeting – Aug 03 2012

August 2, 2012

Location: #jquery-meeting on Freenode

Attending: Corey, Scott, Dave, John, Jörn

Time: Noon ET

QUnit

Need to review pull requests and other issues

  • #287 will land module select after some cleanup
  • #289 look into enhancing validTest to always accept global failure
  • #292 move .html() reset to Core (and maybe UI)
  • #291 looks invalid

TestSwarm

BrowserStack running better now, all UI jobs green without us changing anything

Corey working on post-commit hooks, will use those for testswarm and testswarm-browserstack to deploy updates in the future

John Bender working on getting Mobile fixed in IE9, a few (swarm) heisenbugs make that not as much fun

Need to move testswarm-browserstack to jquery account, use Krinkle’s fork as starting point. Ask Clark though to move his repo to keep forks and issues.

  • Merge Corey’s experiments, see /usr/local/tools/testswarm-browserstack/gnarf-test.js onjq03.jquery.com
  • Also look into upgrading to v2 BrowserStack API, node-browserstack already supports that

Move to new service box

Testing Team Meeting – Jul 19 2012

July 19, 2012

Location: #jquery-meeting on Freenode

Attending: Jörn, Timo, Scott

Time: Noon ET

QUnit

TestSwarm

  • Experiment with test-running in new windows instead of iframes has been completed. Details at https://github.com/jquery/testswarm/issues/195. Long story short: popups are worse than iframes, we’ll have to stick with iframes for now and instead let those bugs depend on the implementation of the new “in-host” system where we don’t use a frame-relationship with a constant long-lived “run page”.

Jenkins integration

  • botio integration stalled until EOL servers are dealt with
  • See also “Perf regression testing”

Perf regression testing

There’s two parts to perf testing. One is the performance of a regular test suite run, to catch overall issues and slowness:

  • We’re going to implement a json report aside from the html-snapshot report of each test run
  • Aside from the assertion results, we’re going to add the test duration of each test
  • in Jenkins we will use that json report to form an “jUnit Ant XML” report card, so that Jenkins will generate historic graphs for each job with the performance over time of each project as a whole, as well as each individual module, test and assertion.

The other is dedicated jsperf-like testing with repetition of certain specific tasks that should perform well. This likely requires the introduction of a framework or plugin in addition to core QUnit.

Move to new service box

gnarf has given krinkle access to jq03. Right now just has a fresh testswarm clone, nothing is set up yet. Krinkle intends to set up and document and/or puppetize it over the coming 1-2 weeks. If it needs to happen earlier, he is willing to document is somewhere and have someone else implement it (most if not all is documented very well at https://github.com/jquery/testswarm/wiki/Automated-Distributed-Continuous-Integration-for-JavaScript )

Testing Team Meeting – Jun 21 2012

The first testing and CI meeting happened today, and will continue to happen bi-weekly, Thursday at Noon EST. Here’s the agenda and meeting notes.

June 21, 2012

Location: #jquery-meeting on Freenode

Attending: Corey, John Bender, Jörn

Time: Noon ET

QUnit

TestSwarm

Jenkins integration

Deploying -git builds for Core, UI, QUnit and Mobile, though no consistent headers yet, and a very different setup for Mobile

Get sizzle on Jenkins and TestSwarm

  • Added grunt.js, will create job to run `lint` test once that is fixed

Perf regression testing

  • John Bender to explore this

Move to new service box

  • Sometime after the conference
  • Will remove `scp` step from git-build
  • Ask Clark to help with that, he can invest a few hours