Wikipedia talk:reFill/technical

Please use this talk page for technical discussions regarding reFill. Curb Safe Charmer (talk) 22:46, 7 February 2022 (UTC)[reply]

Upgrades[edit]

When I was still using ReFill I made some improvements to the source code, particularly regarding website title retrevial. I think I added a better way to get the website titles, and also a title blacklist (so known bad titles ). I can upload the git patch diffs on some pastebin site or on here. It could be useful to you. Rlink2 (talk) 00:38, 8 February 2022 (UTC)[reply]

@Rlink2. I think the repo is located at https://github.com/CurbSafeCharmer/refill if you want to look into submitting PRs. –Novem Linguae (talk) 02:00, 25 May 2022 (UTC)[reply]

CI/CD[edit]

@TheresNoTime and SD0001: it would be great if we could get a CI/CD pipeline working on reFill. Anyone know how it is meant to work? Curb Safe Charmer (talk) 12:31, 6 June 2023 (UTC)[reply]

GitHub Actions can be used – it would require any one maintainer's SSH key to be stored in GitHub Secrets, used to login to toolforge and run the deploy commands every time a commit is pushed to master branch. c.f. https://github.com/siddharthvp/SDZeroBot/blob/master/.github/workflows/toolforge-deploy.yml – SD0001 (talk) 14:54, 6 June 2023 (UTC)[reply]

@SD0001: how does this relate to the publishing of docker images? Curb Safe Charmer (talk) 10:30, 7 June 2023 (UTC)[reply]

@Curb Safe Charmer The docker images are just for the local dev setup, right? Toolforge deployments run on the standard python/node16 images only and the cluster doesn't allow custom-built images. CI/CD = CI (automated tests) + CD (deployment). I don't think there's much CI to be done as we haven't written any tests. CD is something that we can setup using a GitHub action. – SD0001 (talk) 11:16, 7 June 2023 (UTC)[reply]

Ah, ok, thanks for clarifying re Docker. I think there is some CI as tox is in use. I was expecting to see some remnants of automated deployment, perhaps disabled or broken, but in the absence of such, would you be willing to set up the action? Curb Safe Charmer (talk) 11:32, 7 June 2023 (UTC)[reply]

Yes. I still need to figure out some details of the setup though. I see the refill tool is running a php webservice. But the git repo doesn't contain any php code? – SD0001 (talk) 12:14, 7 June 2023 (UTC)[reply]

There's this separate repo. I think TheresNoTime found some other moving parts that weren't under source control recently, will see if I can find that too. Curb Safe Charmer (talk) 15:14, 7 June 2023 (UTC)[reply]

Worth noting that the labsconf repo has CD – it's configured with a webhook (I'm guessing - I don't have owner access to the repo to actually check). The CD script (~/public_html/hooks/commit.php) doesn't appear to be under version control - one of the reasons why I prefer GitHub actions instead. – SD0001 (talk) 17:07, 9 June 2023 (UTC)[reply]

Branches[edit]

We might also need to rebuild the venv. The venv is currently running on python 3.7. – SD0001 (talk) 15:01, 6 June 2023 (UTC)[reply]

@Curb Safe Charmer: Just FYI, refill on toolforge uses the labs-stable branch of https://github.com/CurbSafeCharmer/refill — any merges to master won't be reflected on https://refill.toolforge.org

I'm sure you know that already, but just in case... — TheresNoTime (talk • they/them) 12:20, 9 June 2023 (UTC)[reply]

@TheresNoTime: no, I was unaware of that. Sadly I had a total of zero minutes of handover of the project from Zhaofenli, and anything I know about the project has had to be worked out... Curb Safe Charmer (talk) 14:16, 9 June 2023 (UTC)[reply]

@TheresNoTime At least I didn't know that. Seeing as this branch is 624 commits ahead of master, is there a reason not to mark it as the default branch? Otherwise anyone looking at GitHub is looking at the wrong code. And yep, this branch does have the PHP code which I was looking for. – SD0001 (talk) 12:32, 9 June 2023 (UTC)[reply]

@SD0001: There are probably reasons, but I'm afraid I don't know them — for the most part, the master branch was once going to be a rewrite but was never completed. — TheresNoTime (talk • they/them) 12:36, 9 June 2023 (UTC)[reply]

@SD0001, I'm shocked myself :D That's one of some 30 branches there. We should add this note to the main page here. I've wasted quite some time exploring the wrong code :( Ponor (talk) 12:38, 9 June 2023 (UTC)[reply]

It kind of looks to me like the /ng the master branch and the / version the labs-stable branch? AManWithNoPlan (talk) 13:54, 9 June 2023 (UTC)[reply]

It is all pretty confusing. At Wikipedia:Refill/technical#Hosting I started listing what I am seeing in the production environment with a view to working out what is live and what is not, working out what can be deleted, and working out what folders on toolforge map to which bits in the repo. Curb Safe Charmer (talk) 14:41, 9 June 2023 (UTC)[reply]

Tracked in Phabricator
Task T340502

Tracked in Phabricator
Task T340503

Tracked in Phabricator
Task T340504

This tool appears to have multiple issue trackers (GitHub issues and Phabricator), multiple repos, and multiple branches. Getting everything into 1 ticket tracker, 1 repo, and 1 branch should probably be a high priority goal. –Novem Linguae (talk) 16:15, 9 June 2023 (UTC)[reply]

@Novem Linguae: agreed. What's best practice for issue tracking for tools? Curb Safe Charmer (talk) 16:29, 9 June 2023 (UTC)[reply]

Could go with either issue tracker. Maybe look at which one has more open tickets and go with that one? Less bugs to migrate over. If we go with Phab, I'd recommend disabling the GitHub Issues tab, which can be done in the repo settings. If we go with GitHub issues, might need to find a Phab admin to disable the tag. –Novem Linguae (talk) 16:34, 9 June 2023 (UTC)[reply]

My 2¢ is keeping it on phab, as it'll save people having to create a GitHub account to report bugs. Pretty sure all the GitHub issues can be closed. — TheresNoTime (talk • they/them) 16:36, 9 June 2023 (UTC)[reply]

Ok, Phabricator it is. I prefer a Kanban board anyway - more visual. I will trawl through the GitHub issues and close or migrate them, and disable the tab. Will also in due course do another trawl through the Wikipedia talk:reFill to try to make sure we've captured everything that users are telling us about - there are lots of duplicate reports, and plenty in the "won't fix" category unless anyone fancies writing a load of Zotero translators. Curb Safe Charmer (talk) 16:44, 9 June 2023 (UTC)[reply]

Alright, I made a bunch of phab tickets for these items today, so we can start planning out details. phab:T340502, phab:T340503, phab:T340504. Anyone up for moving the GitHub issues tickets over to phab? If not I might boldly do it soon so we can turn off GitHub issues and centralize around phab. –Novem Linguae (talk) 01:31, 27 June 2023 (UTC)[reply]

Nov 2023 update[edit]

I just submitted 3 PR's that will combine github.com/CurbSafeCharmer/refill#master, github.com/CurbSafeCharmer/refill#labs-stable, and github.com/refill-ng/refill-labsconf, into subfolders of github.com/CurbSafeCharmer/refill#master called undeployed-rewrite, deployed, and labs-config. Please review these PRs at: 1, 2, 3. That should close these two tickets: phab:T340503 and phab:T340504. Thanks. –Novem Linguae (talk) 09:02, 30 November 2023 (UTC)[reply]
I think that https://refill.toolforge.org/ng is the master branch, so it is deployed after all. With that in mind, I picked these 3 folder names: version1, version2ng, and tools-config.

The PRs numbered 2 and 3 above should be safe to merge, even though CI is failing in #2. The failure is a warning about the current version of Twig having a security vulnerability. We can address that in a follow up patch.

The PR numbered 1 above has something broken in .github/workflows/python.yml. Assistance with diagnosing that would be appreciated. –Novem Linguae (talk) 12:58, 30 November 2023 (UTC)[reply]
As discussed above, tonight I migrated all the https://github.com/CurbSafeCharmer/refill/issues to Phabricator. Should be everything. Sorry for all the email spam. I'd like to request that we disable and hide the Issues tab on the https://github.com/CurbSafeCharmer/refill/issues repo, so that folks use Phabricator. This will close ticket phab:T340502. Think we can do that? –Novem Linguae (talk) 11:46, 30 November 2023 (UTC)[reply]
Done Curb Safe Charmer (talk) 11:52, 30 November 2023 (UTC)[reply]

/ng, language support[edit]

@TheresNoTime et al., I've been asking but got no answer: what is https://refill.toolforge.org/ng/ and how's it different from https://refill.toolforge.org? Why does the latter not support all languages? I've tried to find the answers in the code, but those .vue files... I really don't know what to do with them. Ponor (talk) 12:26, 9 June 2023 (UTC)[reply]

@Ponor: From memory (and I could be misremembering), /ng is just a prettier/more basic frontend tailored towards the English Wikipedia (at the very least, jobs submitted from both https://refill.toolforge.org and https://refill.toolforge.org/ng both end up in the same queue, and are both processed in the same way) — TheresNoTime (talk • they/them) 12:35, 9 June 2023 (UTC)[reply]

I wouldn't mind using https://refill.toolforge.org, but it does not let me use the language(s) I need – and I don't see why it wouldn't because they all work at /ng.

This is a great tool and I'd like to advertise it to some other communities, but I need to make sure it's working properly. Ponor (talk) 12:41, 9 June 2023 (UTC)[reply]

Which languages would you need? I can have a look at how that works (it may be as simple as adding the option..) — TheresNoTime (talk • they/them) 14:23, 9 June 2023 (UTC)[reply]

@TheresNoTime, well thank you very much! I've tracked the options down to wikioptions.html but I have no idea where the "wikis" come from (and would like to learn). I searched for "commons", "war" and some others, but github found nothing. To answer your question: adding hrwiki would be great (it works fine), but why not make it available to all? Ponor (talk) 15:48, 9 June 2023 (UTC)[reply]

Ah, yes, configs are stored in refill-ng/refill-labsconf (the config was originally in an inaccessible repo, hence it got moved to an otherwise-unused GitHub org..) — the wiki array is defined in common.php there

As for why not make it available to all, I honestly don't know — there's probably no technical reason why we couldn't.. — TheresNoTime (talk • they/them) 16:01, 9 June 2023 (UTC)[reply]

@Ponor: [EDIT CONFLICT] They are in this separate repo which TheresNoTime created recently because previously those files were not under source control. However, I do not believe that adding language support is just as simple as adding one to the list, as otherwise as you say we could just add every language version of Wikipedia. The templates that reFill emits i.e. the CS1 format must also exist on that language Wikipedia. Curb Safe Charmer (talk) 16:06, 9 June 2023 (UTC)[reply]

Ah! Forgot about the templates! — TheresNoTime (talk • they/them) 16:09, 9 June 2023 (UTC)[reply]

Oh yes, of course. But many do have them d:Q15403807. Ponor (talk) 16:22, 9 June 2023 (UTC)[reply]

Editors are invited to test reFill 2, a new version of reFill under development. It offers a redesigned interface with real-time progress reporting while the tool is running, and resolves many issues affecting the current version of the tool. Userscript and bot developers may also take advantage of the APIs exposed by reFill 2 to complete bare references programmatically.
— User:Zhaofeng Li
— Wikipedia:ReFill 22:50, 1 July 2019 (UTC+1)

Starting from June 30, 2019, all requests to fix English Wikipedia articles will be redirected to reFill 2. Both versions will remain available for Wikipedias in other languages.
— User:Zhaofeng Li
— Wikipedia talk:ReFill 19:47, 30 June 2019 (UTC+1)

@Nomopbs: It's part of the migration to the new tool, and the redirect isn't working for raw wikitext input. While I fix the issue, you can use the new interface directly at https://tools.wmflabs.org/refill/ng. The new tool will hopefully completely replace the old one when support for other Wikipedias is improved.
— User:Zhaofeng Li
— Wikipedia talk:ReFill 23:53, 13 January 2019 (UTC+1)

Oh wow. So /ng is the newer/better version. So maybe that's the way for everyone to go? I like how it works! Ponor (talk) 15:52, 9 June 2023 (UTC)[reply]

@Ponor: only if you are navigating to https://refill.toolforge.org/ng/ to call the API through that front end. Most people run reFill through the reflinks gadget which adds a reFill link to the tools sidebar. They just navigate to the page they want to expand citations on and click that link. They don't see the old or new web form. Curb Safe Charmer (talk) 16:17, 9 June 2023 (UTC)[reply]

@Curb Safe Charmer: I'm having a hard time convincing my patrollers and users to install any scripts. IDK why. So I'm thinking a nice web interface is much better when you want to show them that things work, they only need to follow a link. I thought /ng was some old version (it's well hidden), but it turned out it's the opposite. Ponor (talk) 16:46, 9 June 2023 (UTC)[reply]

Understood. I've always assumed that the developer intended ng to mean next generation. Curb Safe Charmer (talk) 16:55, 9 June 2023 (UTC)[reply]

I was going to guess "new GUI", but I like your guess better :) –Novem Linguae (talk) 17:02, 9 June 2023 (UTC)[reply]

Who is GlazerMann?[edit]

Does anyone know who is contributing to the repo as GlazerMann? They seem committed and know what they're doing, but I have no idea who they are. I'd like to ask if they'd like to become a maintainer. Curb Safe Charmer (talk) 14:31, 9 June 2023 (UTC)[reply]

Have you tried asking them on one of those PRs/issues? (If they don't want to disclose their wiki username, I see no problem with that.) – SD0001 (talk) 17:12, 9 June 2023 (UTC)[reply]

It's User:AManWithNoPlan * Pppery * _{it has begun...} 04:12, 27 June 2023 (UTC)[reply]

Phab watchers[edit]

@SD0001. Feel free to become a watcher/member of the Tool-refill tag on Phab if you're interested. –Novem Linguae (talk) 01:11, 27 June 2023 (UTC)[reply]

Any updates on pull requests and issues on github?[edit]

AManWithNoPlan (talk) 22:30, 19 September 2023 (UTC)[reply]

Merging this page with the main talk page?[edit]

I propose cutting and pasting everything on this talk page to the main talk page over at Wikipedia talk:reFill. There are no archives on this talk page to move over. We can slow down archiving on the main page by doubling min-threads. I believe it is important to consolidate everything so that things do not get too spread out. Any objections? Thanks. –Novem Linguae (talk) 09:04, 30 November 2023 (UTC)[reply]

The reason I created /technical was to document the technical architecture and centralise information about the codebase. The associated talk page became a place to talk about questions and discussion about the architecture and codebase. It makes that discussion much easier to find than if it was interspersed with the rest of the content at Wikipedia talk:ReFill which tends to be where non-technical users of reFill go to report operational problems (reFill has stopped working) or how it has mishandled a particular bare URL. Having split it for what I think are good reasons, I am not at all keen on merging it back. I don't think splitting technical and non-technical discussion means things get spread out - no more than having WP:VP/T separate from Wikipedia:Village pump (all). Curb Safe Charmer (talk) 09:29, 30 November 2023 (UTC)[reply]