Wikipedia:Bots/Requests for approval/KiranBOT 8

Add links
From Wikipedia, the free encyclopedia
The following discussion is an archived debate. Please do not modify it. To request review of this BRFA, please start a new section at Wikipedia:Bots/Noticeboard. The result of the discussion was  Approved.

KiranBOT 8[edit]

Operator: Usernamekiran (talk · contribs · SUL · edit count · logs · page moves · block log · rights log · ANI search)

Time filed: 18:02, Thursday, October 19, 2023 (UTC)

Automatic, Supervised, or Manual: automatic

Programming language(s): pywikibot

Source code available: github

Function overview: archive entries/blurbs from Template:In the news

Links to relevant discussions (where appropriate): requested at BOTREQ, and further discussion at Wikipedia talk:In the news.

Edit period(s): once per day

Estimated number of pages affected: archive bot, will create one page per month

Exclusion compliant (Yes/No): No

Already has a bot flag (Yes/No): Yes

Function details: The bot goes through the edit revisions/diffs of Template:In the news, and archives the entries that have been added.

  1. If a new line begins with *[[ or * [[ then the bot considers it as a recent death. If it begins with <!--, it considers the entry as news/event.
  2. the bot excludes lines beginning with | (if there is a white-space after the pipe), and [[Image:
  3. based on the date of first addition (from diff), the bot adds the entry to header of corresponding date, including diff, editor, and time. If a news entry is updated (eg death toll), then the bot adds the new updated entry under the last matching entry (more details in 8.2).
  4. the entries are added to the archive page in following format: <entry> <special:diff/diff|updated> by <username>, <timestamp>
  5. when an entry is removed, the bot looks for that entry in the "current archive page", and in "previous archive page". in case it is a recent death/RD, the removal update is appended to the same line. otherwise, the removal entry is added below the last matching entry.
    1. in case the entry is not found in both the pages, the entry is then added to the current archive page.
  6. so far the only issue is with "currentevents" parameter within the ITN template. They are being treated as normal news. The folks at ITN are okay with this, but I am working on it. It may take a while to come up with a solution as I am currently very busy in real life.
  7. as there were no particular standards/MOS back in the day, the archive pages of the early days will look a bit odd. But I am planning to fix them manually (ie using AWB, or some other script/program).
  8. I tested the program extensively, and it has only two issues so far.
    1. the first is regarding "currentevents" parameter
    2. the second issue arises because of the way diff is presented in html. to circumvent multiple other issues, the program captures all the additions/removals/changes from one edit, and saves all the changes to archive page in one single edit. because of the way these changes are saved, sometimes the update message doesnt go below the last matching entry of original addition/update, and ends up with a message "updated by" in diff's date.
  9. the folks at the WT:ITN are okay with that. I will also be working to improve the program, and fix these issues.

In the first run, the bot will archive all the entries starting from March 2004. That would be around 40,000 edits. After that I will setup a daily cronjob to go through latest 100 edits. The current average is around 50 edits per day for the ITN template since creation, and 7 edits per day since last 8 years, but we should keep it 100 revisions for "just in case". I have already implemented a check in program based on revision ID/diff, so there would be no repeated archrivals.

Kindly feel free to ask any questions/doubts you have. Regards, —usernamekiran (talk) 18:02, 19 October 2023 (UTC)[reply]

PS: I will post links to sandbox/trial edits shortly. —usernamekiran (talk) 18:09, 19 October 2023 (UTC)[reply]

Discussion[edit]

  • As the initiator of the original request, this is pretty much what I was hoping to see so that we at ITN can track how the ITN template box changes. Excellent work! --Masem (t) 00:44, 20 October 2023 (UTC)[reply]

Approved for trial (50 edits or 30 days, whichever happens first). Please provide a link to the relevant contributions and/or diffs when the trial is complete. Primefac (talk) 09:20, 24 October 2023 (UTC)[reply]

Trial complete. The bot created the archive page Wikipedia:In the news/Featured/February 2004 (for consistency with ITN candidates archive at Wikipedia:In the news/Candidates/April 2005). Everything worked as expected. Requesting an extended trial with 2,000 edits. —usernamekiran (talk) 13:49, 24 October 2023 (UTC)[reply]

I just added a new function to update the Wikipedia:In the news/Featured/Archives (consistent with /Candidates/Archives) as the bot creates a new monthly archive page. —usernamekiran (talk) 19:55, 24 October 2023 (UTC)[reply]

BAG assistance needed The trial went as expected, and new functionality of updating index worked as expected in sandbox. Is it possible to get this task approved? —usernamekiran (talk) 13:09, 31 October 2023 (UTC)[reply]

doing some experiments, no hurry. —usernamekiran (talk) 05:01, 2 November 2023 (UTC)[reply]

{{BAG assistance needed}} based on the discussion at WT:ITN#Archive of ITN postings, no changes were made to the original program except for changing the target location for archive pages. requesting extended trial for 500 edits. —usernamekiran (talk) 02:40, 17 November 2023 (UTC)[reply]

Hi usernamekiran, I have a couple requests. (1) Can we make the header, i.e. the green box at the top, into a template so the wording/formatting can be updated in one place if ever necessary? (2) Can you batch the edits together when you update a page? All of the edits here are done in quick succession using information that should be available to you at the start of the run, so it should be doable in one edit. This should greatly reduce the estimated 40,000 edits. — The Earwig (talk) 07:07, 17 November 2023 (UTC)[reply]
@The Earwig: Hi. (1) I had thought about the header to be transcluded something like {{Wikipedia:In the news/Featured/Archives/header}}. The header for all the candidates archive page is same (Wikipedia:In the news/Candidates/November 2023), but in our case, there is one extra line: "The relevant discussions for additions of entries to the Wikipedia talk:In the news, kindly see Wikipedia:In the news/Candidates/November 2023, or the previous month's page thereof." I am not sure how can we come up with something so that the header could be updated from single place/edit, as even if we use substitution somehow, it wouldnt be editable from single location in the future. Kindly let me know if you have any suggestions/ideas regarding that. (2) Regarding the edits in rapid succession: I was having a lot of difficulties because of the way mediawiki presents diffs in the html format. eg, even if a single word (eg death toll) is updated from 10 to 15, then the diff is presented as the original line completely removed, and an entire new line with the updated word as a totally new line. To circumvent this issue − and to avoid repeated entries, the bot relies on diff IDs already present in the archive page. Also, in current days, the ITN template gets edited/updated around 5 to 7 times in a day. So, in the first run, bot will archive all the entries (around 40,000), and then it will run everyday to archive the new 5 to 7 entries. In short: appending all the changes in single edit for a single run is not feasible (also difficult, as lot of entries do not go in same months, eg a blurb being added on 29 Nov, and being removed on 2 Dec). This can be resolved by adding a delay of 3 (or 5) seconds between each save operation. —usernamekiran (talk) 17:03, 17 November 2023 (UTC)[reply]
@Usernamekiran: For (1), I don't see why we can't have the bot transclude a template like you said... we can do that variable text for the month/year with either a parameter or parser functions. I can make an example to show you if you want. For (2), could you share the bot's code? It would make it easier for me to understand the issue. — The Earwig (talk) 08:47, 20 November 2023 (UTC)[reply]
@The Earwig: Hi. I added the link to github in this same edit. The code is very rudimentary/crude. Also, it was developed over a long period with a few breaks of few days, so it might look a little odd. I tried to add comments wherever there might be doubt. —usernamekiran (talk) 10:25, 20 November 2023 (UTC)[reply]
@The Earwig: Hi. If there are any possible improvements in the code, please let me know. —usernamekiran (talk) 06:21, 24 November 2023 (UTC)[reply]
@The Earwig I created Wikipedia:In the news/Posted/Archives/header, it is working as expected at Wikipedia:In the news/Featured/February 2004. @Primefac We are good to go for another trial. The Earwig's both requests have been addressed — transcluding the header, and a delay of 3 seconds between each save operation. —usernamekiran (talk) 14:25, 24 November 2023 (UTC)[reply]
Thanks for the header template, looks good – I am busy with holidays/family obligations over the next couple days so I won't have a chance to review the code until later, but I still believe we should be able to do this task in far fewer edits. — The Earwig (talk) 03:25, 25 November 2023 (UTC)[reply]
Hi. I'm not sure what is the issue with number of edits. I mean, these edits will take place currently non-existent/unwatched pages. The task for creating complete past archive should take around 34 hours (40,000 diffs — 1 edit per 3 seconds). After that, on an average there are only 5 to 7 entries to archive per day. The bot will run only once per day, running for ~25 seconds. I think reworking the code for that wouldn't be feasible. —usernamekiran (talk) 03:55, 25 November 2023 (UTC)[reply]
update: I made some changes to the code, so that it would perform all the operations on text files. I thought of first creating the text files, then saving these files as wikipedia pages. But without making any other changes than saving data to text file(s) vs page(s), the accuracy decreased a lot. —usernamekiran (talk) 10:16, 16 December 2023 (UTC)[reply]
Hey usernamekiran, thanks a lot for doing the extra work to address my concern, and sorry I've been so unresponsive here. How much did the accuracy decrease? I am surprised by this; I would expect the behavior to be mostly the same. Is there an issue that the text file is not getting initialized to the existing content of the wiki page (if any)?
I'm approving you for a second trial. You don't have to start this right away if you'd like to discuss the accuracy issue more first, but you can if you think it's ready, so we can evaluate the new behavior. Approved for trial (50 edits). Please provide a link to the relevant contributions and/or diffs when the trial is complete. — The Earwig (talk) 04:26, 21 December 2023 (UTC)[reply]
on it. —usernamekiran (talk) 20:42, 22 December 2023 (UTC)[reply]

Trial complete. @The Earwig: Hello. I ran the original program, and User:KiranBOT/sandbox/Posted/February 2004 was created. With another method, I saved all the changes to txt file, and then copy-pasted all the text to User:KiranBOT/sandbox5. Except for saving individual entries to page vs saving individual edits to txt file (and some edit counting mechanism), there were no changes in the program at all. But the file program captured some images (undesired, original program skipped them), and the file program couldn't properly check the diff ID's already present in the file — that is resulting in multiple same entries being added (some of these repeated entries do not have timestamps). Both the programs are in this repository on github. —usernamekiran (talk) 14:43, 26 December 2023 (UTC)[reply]

@Usernamekiran Thanks. On User:KiranBOT/sandbox/Posted/February 2004, several entries are labeled RD, I assume for recent deaths, but they are not deaths? — The Earwig (talk) 16:33, 26 December 2023 (UTC)[reply]
@The Earwig: yes, you are correct. I've explained about that in the "function details" section in points 1, and 7. As there was no standard in the early days, it would be very difficult to to differentiate between news blurb, and and recent deaths. I was thinking about creating a simple regex for AWB to find lines that begin with RD, and count the characters between first occurrence of [[, and last occurrence of the closing bracket of the same line. If there are more than a certain number of characters, then the regex would remove the "RD". Implementing this regex in current program would bulk up/clutter the code, and it would also be redundant once the MOS/standardisation comes in. Implementing it later would cause only one edit per page. —usernamekiran (talk) 19:07, 26 December 2023 (UTC)[reply]

 Approved. Please run the version that prepares the edits in a file for the initial historical run. I don't entirely understand why your code produces duplicate entries, but perhaps you could write a script to clean up the duplicate lines and any other undesired things you think are easy to fix (like images) before saving. Maybe that approach would be easier than making changes to the main code? In any case, I'm satisfied this task is low-risk enough and any issues can be corrected after the fact. — The Earwig (talk) 16:48, 28 December 2023 (UTC)[reply]

The above discussion is preserved as an archive of the debate. Please do not modify it. To request review of this BRFA, please start a new section at Wikipedia:Bots/Noticeboard.
Retrieved from "https://en.wikipedia.org/w/index.php?title=Wikipedia:Bots/Requests_for_approval/KiranBOT_8&oldid=1192294207"