DjVu

“ That DjVu (Libre) is free in all the ways that matter (licence, patents, open source, open specification, free-as-in-beer, etc.) where PDF has uncomfortable caveats on several points is a secondary but not unimportant concern.
And, ultimately, from what I can tell DjVu—because it is designed specifically for our use case—is much better suited for our needs at the format level. For example the ability to separate a page image into multiple layers, where a single-color solid background layer can be encoded efficiently as essentially a single pixel ”

“ If you look at the IA JPG, compared to the DJVU version, you can see the huge loss of detail caused by the DJVU compression which is optimised for text, not images. Can I recommend that you take your initial image from the IA directly, rather than from the compressed DJVU? However, I think it is certainly possible to: Automatically download a file (the name is directly related the image path, and simple scripting tools can grab images from a web address) Run a script on the file (you have that done already) Reupload (pywikipedia has an upload script that can overwrite files) . ”

“ However, illustrations may well be badly degraded by the DJVU compression, so you may also want to upload the JPGs to Commons as well, so that full-quality pictures can be extracted. People with bots (me for example) can help you batch-upload large numbers of files if this will be a problem. If the text is just text however, the DJVU alone generally suffices. I can also create DJVU files out of sets of JPGs for you, or tell you how to do it yourself quickly and easily with a script. Headers, footers, page numbers etc are added to the page-by-page proofread text, but not to the overall text. ”

“ What worries me is that I can't see how this would have happened unless the Internet Archive's DjVu encoder knows something about what kind of glyphs to expect to find on a page, and is willing to take a guess as to which one is correct—a process tantamount to low-level OCR. If it is the case that the Internet Archive's DjVu processing is guessing glyphs rather than faithfully reproducing whatever it sees, then this casts serious doubt upon how we do our work here. What is the point of using scans to ensure fidelity, if the scans themselves lack fidelity? ”

“ Let me know if there's any help I can provide on the revised deletion/reloading based on LA2-bot being the most recent updater of the page. Dictioneer (talk) 20:59, 16 May 2015 (UTC) I'll try to upload the latest text-layer, I need to customise a script first. But trust me, updating the text layer in the djvu file is a viable option if and only if a new OCR-process will be reapplied to the file. All the rest can be done working directly on text files, without bothering changing the djvu. And also dividing header, body and footer is not trivial. ”

“ Cygnis insignis (talk) 09:26, 22 January 2010 (UTC)
I reckon that the name of the "source" field is confusing. Typing "djvu" or "pdf" in that field, only to display a link to the djvu file might seem a bit useless. To understand why this field is there, remember how index pages used to work previously, before we supported the djvu format. Old index pages used to link together individual jpeg images. In that case, the source field was supposed to be filled with a link to a category where the scans belong. Now that the extension supports djvu files, this field is less useful. ”

“ A direct conversion from PNG to DjVu will yield the highest possible fidelity. To unnecessarily convert via an intermediate format means losing fidelity for no real reason. Moreover, the works that I DjVu typically contain images as well as text. Having said all that, I have no objection to your rearrange at Help:DjVi files; I'm just explaining why I'll continue to seek the highest quality scan possible. ”

“ I haven't figured out how to add text to the djvu's hidden layer, so the files do not have a text layer yet. Is Hathi's text preferable to just running it through OCR (e.g. any2djvu or tesseract) ? Is it worthwhile to try to sync the text of EOs you already have on WS to the pages in these compilations? I will look for a guide on adding a text layer when I have a chance to tinker with it, and I can upload the Hathi text layer somewhere if that is helpful for you. Also, it is relatively simple to remove pages from the djvu, so I can take care of that when we get the other big issue solved. ”

“ CONTENT="SUPÉRIEUR"/> As part of our processing, we could easily translate this into whatever syntax the DjVu needs. Our problem is that we do not what it is. As you seem to be quite knowledgeable on this, could you please give us some pointers on the DjVu text layer format ? ”

“ Nevertheless, it can be easier to work with smaller files because various processes (extracting images etc.) are quicker. Tried and tested: DjVus have been in use on Wikisource for longer than PDFs. It is more likely that any problems that can occur have already occurred and have been solved. DjVu files are less likely to cause problems with the ProofreadPage Extension or Wikisource in general. Disadvantages Lower resolution: DjVu files have a lower resolution than PDFs. ”

“ There is no hurry. Good night. — Ineuw talk 09:50, 23 September 2012 (UTC)
Then you should have no trouble with this... First thing to keep in mind is that the DjVu source files we deal with on en.WS are 'bundled' Djvu Files. A bundled DjVu file is a collection of 'indirect' DjVu files - usually one indirect DjVu file being a single scanned page of the original source document. All these seemingly disconnected indirect DjVu files are organized for viewing by a central DjVu file - designated the 'index' DjVu file. ”

“ Sounds similar to the 'Xerox of a Xerox of a Xerox of a Xerox of type written page of text paradox' no? Downloaded the volume and stored it in the program directory. Using djvm.exe in command line mode, deleted the two damaged pages containing the fingers of the IA volunteers, and then verified the results with djvu in GUI mode (very fast) . Merged the two new (consecutive) pages using djvm.exe in command line mode, into one file and verified the results with djvu in GUI mode before proceeding to the next step. ”

“ It’s important to emphasize that this is a consequence of the specific compression settings IA has chosen for their djvu encoder. More reasonable settings can produce better results. I grabbed the topmost png image from this post, converted to PAM, and ran it through the c44 djvu encoder at the default settings. ”

“ So, I'll think that their best location is into a subpage of the bot itself: I'll post them into User:Alex brolloBot/Python scripts, "as they are". --Alex brollo (talk) 21:31, 14 May 2010 (UTC) Request for gif --> djvu conversion I have about 35 individual .gif images, one for each page of a pamphlet published in the 1850s. My skills when it comes to djvu conversions are limited to pdf --> djvu conversion, so if anyone has a more robust tool and the ability/time to convert a collection of gifs into a single djvu file and send it to me, please let me know and I'll email you the files. Thanks. ”

“ PS. Keep in mind that I can also regenerate a DjVu from the original page scan images, including generating a new OCR text layer. A lot of our "text layer is offset" problems with DjVu files are caused by phab:T219376 and these can be fixed by regenerating them from scans (my scripts have armoring to prevent triggering this MediaWiki bug) . --Xover (talk) 05:38, 8 January 2020 (UTC) Great! Then this should be a straightforward fix. The File:Aeneid (Conington 1866) .djvu (on Commons) came from IA, and looked to be just fine but pages 81 and 82 turned out to be swapped in the scan. These are pp. ”

“ Well after spining my wheels for an hour or two I have come to realize something isn't quite right with 1938 the first. Besides the Trust's scan itself duplicating scan page numbers 158 (djvu 172) & 159 (djvu 173) again in djvu pages 174 & 175 before returning to normal in scan page number 160 (djvu 176) , something to do with first djvu page itself and its resolution of 1 × 1 causes the thumbnails normally displayed in both edit-view and view-view modes in the Page: namspace not to appear at all; they go to all black instead. ”

“ If say a "double-entry" exists, I've seen this throw off the layer to thumbnail marriages before. You'd then need to use DjVused.exe to properly extract, repair and merge the outline in the DjVu file. I could not find your command. What I did is looking into: DjView->View->Information but could not find any double or missing entry in the sequence. If you extracted the text-layer to a .txt file using DjVused.exe, run a 'find' against it using page 0 0 (thats the word page, space, zero, space, zero, space) . ”

“ This only works if the PDF file is uploaded to the Internet Archive. IA doesn't add the text layer to an existing DJVU. Yann (talk) 09:42, 16 October 2009 (UTC) as I already explained to Billinghurst, I recently added a "match" command to ThomasBot. This command compares the text layer of an existing djvu (supposed to contain raw ocr text) to a page of the wiki (supposed to contain a proofread text) , in order to find out the page positions. once the page positions have been found, the "split" command can be used to migrate the text to the page namespace. ”

“ Item (4) is less likely, unless Commons doesn't like DjVu files with pages named "page_####.djvu" (digits represented as "#") for some insane reason, because the new IA scan has even longer internal file names. The DjVuLibre tools here have no problem with any of the files, so I suspect something is wrong with the handling of .dvju files in MediaWiki. Pathore (talk) 03:52, 14 February 2015 (UTC) I really don't get what is going on here; the same file as File:Vol118.djvu but with the text-layer completely removed (File:2vol118.djvu) seems to process and render just fine. ”

“ Does it work with djvu only or pdf also? Thank you in advance, Chaojoker (talk) 15:57, 28 July 2012 (UTC)
re: removing watermarks: I would like to know too. Kind regards, --William Maury Morris II (talk) 16:37, 28 July 2012 (UTC) Unfortunately, the only software that works everytime is Adobe's Acrobat Professional v9 (or higher) and even then it is a labor intensive process. I've searched high and low on this for some time now and nothing [free] even comes close to AcroPro. The only reason I converted the finished PDF file to DjVu was to easily add a text-layer in the process. ”

“ Thinking further ahead, maybe we can improve mw:Extension:ScanSet to act like a DJVU file. i.e. if we create a dummy [ [Image:EB1911_tiff.scanset] ] and it understands [ [Image:EB1911_tiff.scanset/Vol15/EC5A664] ] (and converts the TIFF to PNG) . John Vandenberg 01:35, 16 January 2008 (UTC)
i would prefer the second option, because tiff suport is not widespread. also, a scanset is naturally ordered, which means that no index page is needed to create navigation buttons (like in pdf/djvu files) . ”

“ Index AL? % text [1] Best scan [2] Offset [3] Glitches [4] Comment [5] 1 index 100 (poor) 14 Replaced with djvu from Google Books PDF. Good condition. Like the scan it replaced, without errata. 2 index 100 (good) 12 New djvu, 100 dpi grayscale. Stray note stuck to one of the pages gone, but recorded. (may have had copyrighted material) No errata. Index restored 3 index 100 (good) 6 New djvu file, while readable throughout, contains isolated blurred text. March 25, 3rd djvu version introduced replacing previous one ”

“ Very nice; would you like me to upload a DjVu of page scans so you can proof against them? Hesperian 23:22, 14 October 2008 (UTC)
Please. Then I can see how it is done. Nice bit of analysis on your quote selection. Cygnis insignis (talk) 12:50, 18 October 2008 (UTC) I've made a note to do it on Monday. Unless some of the pages desperately need cropping, the file I produce won't be any better than if you downloaded the PDF from http://ndpbeta.nla.gov.au/ and used http://any2djvu.djvuzone.org/ to convert it to DjVu. So there's no reason you can't do it yourself if it takes your fancy to do so. ”

“ As I have put "anchor+" against the alpha-letter AND we can use "compactTOCalpha" to go to an alphabet area, my decision, reluctant as it is, is to ask for a subject breakout. We currently have no way of getting to say the subject of "Meekness" or "Merit", both of which have 2 quotes each versus the subject "Christ" which has 48 pages of quotes. I did put in DJVU page links but see it doesn't transfer to transclusion, which is not a good thing. (There must be a reason why it doesn't; then why have it? Is there not something that does work?) ”

“ It is quite good exercise. I am done with chapter 1 and all the graphics pages. Do you think match and split would work for the chapters 2-5 only? I am still a little blurry about what it does and how. I also have some thinks I not sure what is the best way to resolve: pages 28 and 29 are missing from djvu files but we do have the text. I added it to page 27. but I am not sure what is the best way. pages 11->12 transition splits word "Yesterday" into "yes-" and "terday", and the display program adds a space in between. ”

“ EDIT: an exact copy of the current .php file dealing with DjVu "processing" is --> DjVuImage. Line 40 jumps out at me for starters. -- George Orwell III (talk) 23:33, 16 February 2015 (UTC) That is a ~300MB memory limit. The djvutxt tool doesn't use anywhere near that much; 40M is adequate for processing all three versions of EB1911 volume 25 here. Any chance we could get debugging logs from processing some of these files? There's a bunch of wfDebug calls in that code. Pathore (talk) 01:00, 17 February 2015 (UTC) Beats me -- maybe Mpaa knows how? ”