“ Also, in general, it often isn't an issue of MediaWiki being unable to handle whatever the issue is; but rather that when you have an extremely simplified syntax like wikimarkup, that's used to generate relatively complex things like full-blown html, you're going to run into limitations and tradeoffs. The lack of end tags in wikimarkup makes it impossible for the parser to function without inference, and when inference rules start stacking you can't easily tweak one without knock-on effects for others. ”

“ Every single time that content is rendered the numbering is generated anew, and thus every single time there is a potential that something can go wrong (changes in MediaWiki's parser, changes in web browser CSS engines, different web browsers, etc. etc.) . It also means it can change over time due to changes in the standards that define those algorithms. And algorithms are inherently more complex than hardcoding the content in the first place. CSS counters are close to programming complexity, but even HTML numbered lists share a lot of this type of fragility. ”

“ There is no general way to solve sidenotes without explicit software support in Mediawiki (which will probably require a CSS standard for it in order to be implemented) . Keep in mind that the dynamic layouts and page numbers is not actually a feature of Mediawiki: it is a local javascript that GOIII hacked together (and which hasn't been maintained since he retired) . There is no left and right margin that we can rely on to be there, and in which we can place a sidenote. The block vs. inline is also impossible to solve in the current software. ”

“ But I am hesitant to implement this request: when you know you should not do something, technical measures should not be needed to enforce it. In this case, it is by your own judgement that you should not edit in the Template: namespace so refraining from doing so should be easy to adhere to. If it will help you, I can add a strong admonishment to refrain from editing there: not because I've actually seen you make any edits that would merit that, but because it clearly causes you great frustration. ”

“ For simple cases with little need for manual page-fiddling and such, the computer does most of the work (I just remove the extra scan reference images at the beginning and end, and any botched images interspersed in the image series) . I can usually find the time to grab the download and set it processing even when I'm otherwise busy. Case in point, I've just set the computer to crunching this scan so it should have a DjVu ready for upload by the time I'll be sufficiently caffeinated tomorrow morning. ”

“ However, since we're here talking about parts that are works in themselves, it's entirely possible the community will accept them in spite of policy. It's not something I would—with my admin hat on—go out of my way to enforce, but I can't really predict with any certainty what the community's sentiment would be if it were to end up at WS:PD. The best would be if you could get a complete scan of the book as published (if you have a bunch of images I can generate a DjVu file from them; and might even get some half-decent OCR text too) and work from that. ”

“ Again I'm not sure what the specific problem you're seeing is (your testcase page contains lots of things that may or may not be the issue you're concerned about) . But for div inside list items, the most obvious issue isn't the div as such, but rather extraneous newlines in the template. Due to html whitespace rules these are usually effectively ignored, but in certain contexts newlines have semantics for the MediaWiki parser. Lists being one such context: list items in MediaWiki cannot contain raw newlines due to the simplified list syntax relative to html lists. ”

“ That DjVu (Libre) is free in all the ways that matter (licence, patents, open source, open specification, free-as-in-beer, etc.) where PDF has uncomfortable caveats on several points is a secondary but not unimportant concern.
And, ultimately, from what I can tell DjVu—because it is designed specifically for our use case—is much better suited for our needs at the format level. For example the ability to separate a page image into multiple layers, where a single-color solid background layer can be encoded efficiently as essentially a single pixel ”

“ This is something else that might be needed, is identifying works on Wikisource/Commons, that a Djvu copy exists on WMF servers, but not a PDF. The intent would be to eventually have one "good" copy, and avoid duplicated uploads between (PDF, DJVU) . Detection of identical editions (but differently SHA-1'ed) scans is beyond the ability of automated tools. Of course if there are any major collections that are Commons compatible, and which aren't under consideration, the page I linked has a section for suggesting possible future 'batches'. ”

“ We could forbid multi-line use of {{center}} (so each of your header lines was wrapped in a separate {{center}} template) and go back and change every single existing use to conform to that. That's obviously not a workable solution (it can't be automated, and is obviously way too much effort for extremely little gain) . So the only alternative is to say that {{center}} cannot be used inside pre-like contexts (anywhere whitespace is significant) , and to create a separate centering template to cover this use case. ”

“ I am not aware of any problem with image quality in DjVus in MediaWiki but it is extracting the IW44-encoded image in the DjVu and reencoding it in JPEG for the thumbnails so some degradation will always be present. I also don't really see any egregious problem with the scan image quality in Page:A Short Account of the Botany of Poole.djvu/25: it is not nearly perfect, and poorer than what you see at IA, but that is due to excessive compression in whatever process generated that .djvu. It doesn't appear to be so bad the proofreading is impossible. ”

“ Essentially, IA is returning plain incorrect information about page numbers and ia-upload relies on that page order being correct in order to extract OCR text from the XML file at IA and associating it with the right pages in the DjVu. Once one page is incorrect every subsequent page will be too, and multiple such errors will compound. ”

“ When ProofreadPage's tag transcludes pages from the Page: namespace, it joins them together by removing any whitespace (modulo hyphenated words which are a special case) . If you put {{nopt}} at the end of the first page, and Mediawiki table syntax (row start, say: |-) at the start of the second, the transclusion will lead to the following construct: |-. Since the row start markup has to be at the beginning of the line this will not work. ”

“ That they are also imperfect and end up getting blinders regarding any problem that does not affect the Wikipedias is annoying, but very very human.
Once it becomes clear that a given issue cannot reasonably be solved on-wiki, the best course is to just drop it; either by finding a workaround that, while wrong, is at least a reasonable substitute (using headings instead of sidenotes for example) , or by dropping the issue and finding something that is actually solvable to do (lint errors in ref tags is probably in the "unsolvable" category) . ”

“ Are you saying you prefer to use the template in contravention of its documentation and the semantics of that parameter in its code? If so, what is the effect you are trying to achieve? --Xover (talk) 13:50, 22 February 2020 (UTC) I am explaining what the template does, if the documentation does not match the action then the documentation is wrong. The template takes the year of expiry, not the last year of life of copyright as the PD-old-nn series takes. ”

“ That "a certain other contributor" seems to be getting as frustrated by your perseverance and consequent frustration as your own frustration with the software, is, not to put too fine a point on it, not your fault. Do, by all means, try to avoid knowingly stepping on people's toes, of course. ”

“ People may wonder why policy change discussions can get bitter, when the boundaries are moved, and stretched and moved again. Remind me next time that this is what is going to happen, and I will be vociferous about opposing such initial changes as it seems that the defence needs to be made at the very beginning. ”

“ It is possible to change the precedence of the two scripts for each page, and it may give better results on pages in Hangul. There are also variant settings for horizontal scripts, and we can also try to specify the languages instead of the scripts (it can sometimes give better results) . There are also a couple other settings I can tweak to try to improve it if it's too bad to use.
In any case, no OCR is going to be perfect; it's always a matter of "degrees of awful". ”

“ There are fundamental limitations of a wiki as a medium, and other limitations that stem from the fact that the WMF does not have infinite resources but nearly infinite competing interests and requests on those resources. ”

“ If you give it --detail=page you'll get the same output MediaWiki uses. If you give it --detail=word you'll get the full detailed text structure, down to the individual word level, in sexpr-format. --page=pagenum specifies which page in the .djvu you want to dump, otherwise you get the whole file. djvused—essentially an interpreter for a private little scripting language for DjVu files. Very powerful, but for advanced users only. djvudump filename.djvu—dumps the internal structure of .djvu files. ”

“ Also, once a scan exists, it is possible there would be people interested in helping with various bits at the Korean Wikisource. The essence of crowdsourcing is that many hands make light loads; or, don't work yourself to death trying to tackle this job alone if there are others that could be induced to help. :) --Xover (talk) 08:03, 6 June 2020 (UTC) The institution made a single PDF file of the full scanned text for me, though I did need to pay. I've uploaded the file on commons:File:朝鮮巫俗の研究 上券.pdf for if you want to run OCR on them, though I'm not sure how well they'll work. ”

“ I can't recall—that wrapped almost every other word in a template with multiple arguments, resulting in the whole page being just a soup of markup. If you've ever seen raw PostScript data... That's a perfect (extreme) example of why this is a problem.
That doesn't mean we can't use these approaches at all, but it does mean we need to be careful to not fall into the trap of making the solution so fancy that it defeats the purpose. --Xover (talk) 06:13, 13 May 2020 (UTC) Play by Synge Could you please create a DjVu for J. M. Synge's play The Playboy of the Western World from (external scan) ? ”

“ The tricky bit in this specific case is that while your creation myths would seem to qualify as distinct works in themselves, the commentary, footnotes, etc. are not a complete work without the context of the remaining 600 pages. ”

“ The tools are what they are. We can do what we can within the limits of what they allow, or we can keep fighting them and being perennially frustrated. I heartily recommend the former approach. There are plenty of things to do here that do not involve templates or numbered paragraphs at all. --Xover (talk) 13:53, 8 February 2020 (UTC) BTW My reasons for redoing this is so that eventually I don't have to implement a new template for each 'different' level, I just add the relevant classes to the stylesheet. ”

“ People are who they are, and they will either pick up on what needs amending on their own (possibly with a little friendly and informal help) , or they will not (which on enWP usually means they will eventually end up blocked) regardless of any mentoring. ”

“ As this is the only good first edition scan at IA, I'd rather have it corrected than resort to using a later edition or poorer scan. If you can simply swap those two pages and upload the corrected file to Commons, that should be all this file needs. --EncycloPetey (talk) 16:34, 8 January 2020 (UTC) @EncycloPetey: Done. Swapping two pages is indeed easy: the only hard part is keeping straight in your head what goes where. :) --Xover (talk) 17:30, 8 January 2020 (UTC) Thanks! I also see that Yale Shakespeare volumes have been unlocked now, but the Coriolanus does not have a DjVu file available. ”

“ PS. Keep in mind that I can also regenerate a DjVu from the original page scan images, including generating a new OCR text layer. A lot of our "text layer is offset" problems with DjVu files are caused by phab:T219376 and these can be fixed by regenerating them from scans (my scripts have armoring to prevent triggering this MediaWiki bug) . --Xover (talk) 05:38, 8 January 2020 (UTC) Great! Then this should be a straightforward fix. The File:Aeneid (Conington 1866) .djvu (on Commons) came from IA, and looked to be just fine but pages 81 and 82 turned out to be swapped in the scan. These are pp. ”

“ We have endless backlogs of all sorts so there should be plenty of other things to do that will bring you pleasure rather than frustration. --Xover (talk) 19:21, 6 February 2020 (UTC) Page:UKSI19810859.pdf/31 This seems to be another instance of P wrapping weridness.. There should be a normal (paragraph spacing) between the end of the paragraph at the top of the page and the continuation paragraph following it. ”