Wikipedia:WikiProject User warnings/Testing

From Wikipedia, the free encyclopedia
Shortcut

This page is the tracking page for efforts on the English Wikipedia to improve the quality of user talk warnings. Links to this project on other Wikipedias can be found on the cross-wiki hub on Meta.

Scope[edit]

The purpose of this page is to measure the efficacy of different user talk templates through randomized testing. This effort does not entail creating new categories of user warnings or to organize them more efficiently, but rather to improve on the quality of our current communication methods.

We're aiming to fine tune the template messages we send to editors, in order to encourage more good faith contributors and discourage outright vandals, spammers, and other bad faith contributors.

Participants[edit]

Sign up here if you'd like to stay notified about testing updates.

  1. User:Steven (WMF)
  2. User:Maryana (WMF)
  3. User:Staeiou
  4. User:EpochFail
  5. User:Vitor Mazuco
  6. User:Philippe (WMF)
  7. User:Kudpung
  8. User:Kubigula
  9. User:DGG
  10. User:Fluffernutter
  11. User:Wikipelli
  12. Ebe123
  13. Σ
  14. User:The Blade of the Northern Lights
  15. User:Writ Keeper
  16. Eraserhead1 <talk> 21:53, 25 October 2011 (UTC)[reply]
  17. User:Jtmorgan
  18. User:Chzz
  19. Hurricanefan25
  20. Ljhenshall
  21. XLinkBot (forced by its operators)
  22. Beetstra (as one of the operators of XLinkBot)
  23. Andy Mabbett (Pigsonthewing)
  24. User:Racconish
  25. User:Recon Etc
  26. User:Jayen466
  27. User:gwickwire (talk)
  28. User:Thogo
  29. User:Logan
  30. User:Snowolf
  31. User:Johannes Rohr (WMDE)
  32. User:mabdul (AFC related)
  33. Katarighe
  34. Martijn Hoekstra (talk)
  35. User:Gaijin42
  36. User:DMacks
  37. User:Rcsprinter
  38. User:JohnCD
  39. User:Ciphers
  40. Nathan2055talk - contribs 19:49, 27 June 2012 (UTC)[reply]
  41. Rich Farmbrough, 02:49, 15 September 2012 (UTC).[reply]
  42. User:Lexein
  43. User:Rdicerb_(WMF)

Tasks[edit]

Any effort to make templates more simple, friendly, and accessible is welcome. Specifically, you can:

  • Draft new templates or assess the templates that are currently in the draft phase and suggest improvements in their content or structure. We aim to lessen the bitey-ness of warnings, but templates should still comply with the design guidelines and the usage and layout best practices.
  • Suggest which templates to test next and what changes should be made.
  • Help us analyze our data. We're using mixed methods, with both quantitative statistics and qualitative coding.
  • Recruit more participants.

Tests[edit]

Templates have been tested in the following places:

Those pages each have lists and dates of all the templates tested including results if available. The following are tracking tables with start and end dates. For tests in projects other than English Wikipedia, please see our hub on Meta.

Test name Brief description Date started Date ended Duration Tracking templates Data (if available)
Huggle 1 Level 1 vandalism warning. 3 test templates (image/no image; teaching, personal) and 1 control July 19, 2011 August 5, 2011 ~2 weeks {{z49}} {{z50}} {{z51}} {{z52}} {{z53}} {{z54}} {{z55}} {{z56}}
Huggle 2 Level 1 vandalism warning. 2 test templates (personal, personal + no directives) and 1 control September 25, 2011 October 10, 2011 ~2 weeks {{z57}} {{z58}} {{z59}}
Huggle 3 All level 1 warnings (except vandalism), 9 total. 1 test template (personal + no directives) and 1 control October 18, 2011 November 19, 2011 1 month {{z60}} {{z61}} {{z62}} {{z63}} {{z64}} {{z65}} {{z66}} {{z67}} {{z68}} {{z69}} {{z70}} {{z71}} {{z72}} {{z73}} {{z74}} {{z75}} {{z76}} {{z77}}
Twinkle 1 AfD and PROD notices, 2 total. 1 test template (personal + no directives) and 1 control November 9, 2011 December 9, 2011 1 month {{z78}} {{z79}} {{z81}} {{z82}}
Huggle short 1 Level 1 vandalism warning. 2 test templates (personal + no directives, short) and 1 control November 8, 2011 December 9, 2011 1 month {{z84}} {{z85}} {{z86}}
XLinkBot Level 1-4 spam warnings, 4 total. 1 test template (personal + no directives) and 1 control; Welcomeg and welcomeanon, 1 test template (personal + no directives) and 1 control November 17, 2011 December 17, 2011 1 month spam 1-4: {{z87}} {{z88}} {{z89}} {{z90}} {{z91}} {{z92}} {{z93}} {{z94}}; welcomes: {{z95}} {{z96}} {{z97}} {{z98}}
Huggle short 2 All issue-specific level 1 warnings, 9 total. 1 test template (short) and 1 control November 22, 2011 December 22, 2011 1 month {{z99}} {{z100}} {{z101}} {{z102}} {{z103}} {{z104}} {{z105}} {{z106}} {{z107}} {{z108}} {{z109}} {{z110}} {{z111}} {{z112}} {{z113}} {{z114}} {{z115}} {{z116}}
SDPatrolBot Newbie CSD tag removal warning, 1 test template (personal + no directives) and 1 control November 3, 2011 January 3, 2012 2 months — Wikitable
Shared IP archiving archiving of shared and dynamic IP talk pages regularly to remove irrelevant warnings December 19, 2011 February 19, 2012 2 months — List of all archived talk pages, SharedIPArchiveBot's log
ImageTaggingBot Image notifications about sourcing/licensing, 4 total. 2 test templates (directives, no directives) and 1 control. December 21, 2011 February 21, 2012 30 days (minimum) {{z131}} {{z132}} {{z133}} {{z134}} {{z135}} {{z136}} {{z137}} {{z138}} {{z139}} {{z140}} {{z141}} {{z142}}
CorenSearchBot Copyvio notifications, 6 total. 1 test template (personal + no directives) and 1 control; welcomelaws, 1 test (short) and 1 control. January 12, 2012 January 31, 2012 1 month {{z117}} {{z118}} {{z119}} {{z120}} {{z121}} {{z122}} {{z123}} {{z124}} {{z125}} {{z126}} {{z127}} {{z128}}
28bot Test edit warning, registered and unregistered, 2 total. 1 test template (personal + no directives) and 1 control January 19, 2012 March 27, 2012 {{z143}} {{z144}} {{z145}} {{z146}}

Testing method[edit]

The following are the requirements to conduct comparative A/B testing of any user talk template. Doing randomized experiments allows us to get hard data about what kinds of content are most successful at helping us achieve these goals. What you'll need is...

  1. A "randomizer" that delivers all the templates in your test. This is the template that should be included in the configuration of whichever bot or tool you are testing with, and it randomly delivers one of the templates via a parser function.
  2. A control, usually the existing default template. Note that you should replicate the default in a new template rather than use the current template page, in order to avoid including old instances of the default in your experiment.
  3. A new version or versions of the template you want to test. Try to use a canonical name that matches the type, purpose, and level of the warning you're interested in.
  4. A Z number tracking template for all templates being tested. If you do not include a separate Z number in each template, you will lose track of your test cases once they are substituted.

In some cases, such as for bots where all contribs are by one account, this method can be greatly simplified.

Analysis[edit]

We have so far used a mixed method of both quantitative measurement and qualitative assessment. If you'd like to help sort and analyze tests, please sign up above.

Testing results from all projects are available here.

Retrieved from "https://en.wikipedia.org/w/index.php?title=Wikipedia:WikiProject_User_warnings/Testing&oldid=635583553"