Friday, May 17, 2024
Homepage · command-line
 Popular · Latest · Hot · Upcoming
5
rated 0 times [  5] [ 0]  / answers: 1 / hits: 14450  / 3 Years ago, sun, september 12, 2021, 12:15:47

I have thousand of documents and some of them are scanned. So I need a script to test all PDF files that belong to a directory. Is there a simple way to do that?




  1. Most PDFs are reports. Thus they have a lot of text.

  2. They are very different, but the scanned ones as mentioned below one can find some text due to a precarious OCR process coupled to the scan.




  3. The proposal due to Sudodus in the comments below seems to be very interesting. Look at the difference between a scanned to a not-scanned PDF:




Scanned:



grep --color -a 'Image' AR-G1002.pdf

<</BitsPerComponent 8/ColorSpace/DeviceRGB/Filter[/DCTDecode]/Height 2197/Length 340615/Name/Obj13/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 1/ColorSpace/DeviceGray/DecodeParms<</Columns 1698/K -1>>/Filter/CCITTFaxDecode/Height 2197/Length 40452/Name/Obj18/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 1/ColorSpace/DeviceGray/DecodeParms<</Columns 1698/K -1>>/Filter/CCITTFaxDecode/Height 2197/Length 41680/Name/Obj23/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 1/ColorSpace/DeviceGray/DecodeParms<</Columns 1698/K -1>>/Filter/CCITTFaxDecode/Height 2197/Length 41432/Name/Obj28/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 1/ColorSpace/DeviceGray/DecodeParms<</Columns 1698/K -1>>/Filter/CCITTFaxDecode/Height 2197/Length 59084/Name/Obj33/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 8/ColorSpace/DeviceRGB/Filter[/DCTDecode]/Height 2197/Length 472681/Name/Obj38/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 8/ColorSpace/DeviceRGB/Filter[/DCTDecode]/Height 2197/Length 469340/Name/Obj43/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 8/ColorSpace/DeviceRGB/Filter[/DCTDecode]/Height 2197/Length 371863/Name/Obj48/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 8/ColorSpace/DeviceRGB/Filter[/DCTDecode]/Height 2197/Length 344092/Name/Obj53/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 1/ColorSpace/DeviceGray/DecodeParms<</Columns 1698/K -1>>/Filter/CCITTFaxDecode/Height 2197/Length 59416/Name/Obj58/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 1/ColorSpace/DeviceGray/DecodeParms<</Columns 1698/K -1>>/Filter/CCITTFaxDecode/Height 2197/Length 48308/Name/Obj63/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 1/ColorSpace/DeviceGray/DecodeParms<</Columns 1698/K -1>>/Filter/CCITTFaxDecode/Height 2197/Length 51564/Name/Obj68/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 1/ColorSpace/DeviceGray/DecodeParms<</Columns 1698/K -1>>/Filter/CCITTFaxDecode/Height 2197/Length 63184/Name/Obj73/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 1/ColorSpace/DeviceGray/DecodeParms<</Columns 1698/K -1>>/Filter/CCITTFaxDecode/Height 2197/Length 40824/Name/Obj78/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 1/ColorSpace/DeviceGray/DecodeParms<</Columns 1698/K -1>>/Filter/CCITTFaxDecode/Height 2197/Length 23320/Name/Obj83/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 1/ColorSpace/DeviceGray/DecodeParms<</Columns 1698/K -1>>/Filter/CCITTFaxDecode/Height 2197/Length 31504/Name/Obj93/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 1/ColorSpace/DeviceGray/DecodeParms<</Columns 1698/K -1>>/Filter/CCITTFaxDecode/Height 2197/Length 18996/Name/Obj98/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 8/ColorSpace/DeviceRGB/Filter[/DCTDecode]/Height 2197/Length 292932/Name/Obj103/Subtype/Image/Type/XObject/Width 1698>>stream

<</BitsPerComponent 1/ColorSpace/DeviceGray/DecodeParms<</Columns 1698/K -1>>/Filter/CCITTFaxDecode/Height 2197/Length 27720/Name/Obj108/Subtype/Image/Type/XObject/Width 1698>>stream

               <rdf:li xml:lang="x-default">Image</rdf:li>

               <rdf:li xml:lang="x-default">Image</rdf:li>


Not Scanned:



grep --color -a 'Image' AR-G1003.pdf

<</Lang(en-US)/MarkInfo<</Marked true>>/Metadata 167 0 R/Pages 2 0 R/StructTreeR<</Contents 4 0 R/Group<</CS/DeviceRGB/S/Transparency/Type/Group>>/MediaBox[0 0 612 792]/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 7 0 R/F3 9 0 R/F4 11 0 R/F5 13 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC/ImageI]>>/StructParents 0/Tabs/S/Type/<</Filter/FlateDecode/Length 5463>>stream

<</BaseFont/Times#20New#20Roman,Bold/Encoding/WinAnsiEncoding/FirstChar 32/FontD<</Ascent 891/AvgWidth 427/CapHeight 677/Descent -216/Flags 32/FontBBox[-558 -216 2000 677]/FontName/Times#20New#20Roman,Bold/FontWeight 700/ItalicAngle 0/Leadi<</BaseFont/Times#20New#20Roman/Encoding/WinAnsiEncoding/FirstChar 32/FontDescri<</Ascent 891/AvgWidth 401/CapHeight 693/Descent -216/Flags 32/FontBBox[-568 -216 2000 693]/FontName/Times#20New#20Roman/FontWeight 400/ItalicAngle 0/Leading 42<</BaseFont/Arial,Bold/Encoding/WinAnsiEncoding/FirstChar 32/FontDescriptor 10 0<</Ascent 905/AvgWidth 479/CapHeight 728/Descent -210/Flags 32/FontBBox[-628 -210 2000 728]/FontName/Arial,Bold/FontWeight 700/ItalicAngle 0/Leading 33/MaxWidth<</BaseFont/Times#20New#20Roman,Italic/Encoding/WinAnsiEncoding/FirstChar 32/FontDescriptor 12 0 R/LastChar 118/Name/F4/Subtype/TrueType/Type/Font/Widths 164 0 <</Ascent 891/AvgWidth 402/CapHeight 694/Descent -216/Flags 32/FontBBox[-498 -216 1333 694]/FontName/Times#20New#20Roman,Italic/FontWeight 400/ItalicAngle -16.4<</BaseFont/Arial/Encoding/WinAnsiEncoding/FirstChar 32/FontDescriptor 14 0 R/La<</Ascent 905/AvgWidth 441/CapHeight 728/Descent -210/Flags 32/FontBBox[-665 -210 2000 728]/FontName/Arial/FontWeight 400/ItalicAngle 0/Leading 33/MaxWidth 2665<</Contents 16 0 R/Group<</CS/DeviceRGB/S/Transparency/Type/Group>>/MediaBox[0 0 612 792]/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 7 0 R/F5 13 0 R>>/ProcSet[<</Filter/FlateDecode/Length 7534>>streamarents 1/Tabs/S/Type/Page>>

<</Contents 18 0 R/Group<</CS/DeviceRGB/S/Transparency/Type/Group>>/MediaBox[0 0 612 792]/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 7 0 R/F5 13 0 R>>/ProcSet[<</Filter/FlateDecode/Length 6137>>streamarents 2/Tabs/S/Type/Page>>

<</Contents 20 0 R/Group<</CS/DeviceRGB/S/Transparency/Type/Group>>/MediaBox[0 0 612 792]/Parent 2 0 R/Resources<</Font<</F1 5 0 R/F2 7 0 R/F5 13 0 R/F6 21 0 R><</Filter/FlateDecode/Length 6533>>stream>>/StructParents 3/Tabs/S/Type/Page>>

<</BaseFont/Times#20New#20Roman/DescendantFonts 22 0 R/Encoding/Identity-H/Subty<</BaseFont/Times#20New#20Roman/CIDSystemInfo 24 0 R/CIDToGIDMap/Identity/DW 100<</Ascent 891/AvgWidth 401/CapHeight 693/Descent -216/Flags 32/FontBBox[-568 -216 2000 693]/FontFile2 160 0 R/FontName/Times#20New#20Roman/FontWeight 400/Italic<</Contents 27 0 R/Group<</CS/DeviceRGB/S/Transparency/Type/Group>>/MediaBox[0 0 612 792]/Parent 2 0 R/Resources<</ExtGState<</GS28 28 0 R/GS29 29 0 R>>/Font<</F1 5 0 R/F2 7 0 R/F3 9 0 R/F5 13 0 R/F6 21 0 R>>/ProcSet[/PDF/Text/ImageB/ImageC<</Filter/FlateDecode/Length 5369>>streamge>>


The number of images per page are much bigger (about one per page)!


More From » command-line
 Answers
3

Shellscript




  • If a pdf file contains an image (inserted in a document alongside text or as whole pages, 'scanned pdf'), the file often (maybe always) contains the string /Image/.


  • In the same way you can search for the string /Text to tell if a pdf file contains text (not scanned).




I made the shellscript pdf-text-or-image, and it might work in most cases with your files. The shellscript looks for the text strings /Image/ and /Text in the pdf files.



#!/bin/bash



echo "shellscript $0"

ls --color --group-directories-first

read -p "Is it OK to use this shellscript in this directory? (y/N) " ans

if [ "$ans" != "y" ]

then

 exit

fi



mkdir -p scanned

mkdir -p text

mkdir -p "s-and-t"



for file in *.pdf

do

 grep -aq '/Image/' "$file"

 if [ $? -eq 0 ]

 then

  image=true

 else

  image=false

 fi

 grep -aq '/Text' "$file"

 if [ $? -eq 0 ]

 then

  text=true

 else

  text=false

 fi





 if $image && $text

 then

  mv "$file" "s-and-t"

 elif $image

 then

  mv "$file" "scanned"

 elif $text

 then

  mv "$file" "text"

 else

  echo "$file undecided"

 fi

done


Make the shellscript executable,



chmod ugo+x pdf-text-or-image


Change directory to where you have the pdf files and run the shellscript.



Identified files are moved to the following subdirectories




  • scanned

  • text

  • s-and-t (for documents with both [scanned?] images and text content)



Unidentified file objects, 'UFOs', remain in the current directory.



Test



I tested the shellscript with two of your files, AR-G1002.pdf and AR-G1003.pdf, and with some own pdf files (that I have created using Libre Office Impress).



$ ./pdf-text-or-image

shellscript ./pdf-text-or-image

s-and-t                                 mkUSB-quick-start-manual-11.pdf    mkUSB-quick-start-manual-nox-11.pdf

scanned                                 mkUSB-quick-start-manual-12-0.pdf  mkUSB-quick-start-manual-nox.pdf

text                                    mkUSB-quick-start-manual-12.pdf    mkUSB-quick-start-manual.pdf

AR-G1002.pdf                            mkUSB-quick-start-manual-74.pdf    OBI-quick-start-manual.pdf

AR-G1003.pdf                            mkUSB-quick-start-manual-75.pdf    oem.pdf

DescriptionoftheOneButtonInstaller.pdf  mkUSB-quick-start-manual-8.pdf     pdf-text-or-image

GrowIt.pdf                              mkUSB-quick-start-manual-9.pdf     pdf-text-or-image0

list-files.pdf                          mkUSB-quick-start-manual-bas.pdf   README.pdf

Is it OK to use this shellscript in this directory? (y/N) y



$ ls -1 *

pdf-text-or-image

pdf-text-or-image0



s-and-t:

DescriptionoftheOneButtonInstaller.pdf

GrowIt.pdf

mkUSB-quick-start-manual-11.pdf

mkUSB-quick-start-manual-12-0.pdf

mkUSB-quick-start-manual-12.pdf

mkUSB-quick-start-manual-8.pdf

mkUSB-quick-start-manual-9.pdf

mkUSB-quick-start-manual.pdf

OBI-quick-start-manual.pdf

README.pdf



scanned:

AR-G1002.pdf



text:

AR-G1003.pdf

list-files.pdf

mkUSB-quick-start-manual-74.pdf

mkUSB-quick-start-manual-75.pdf

mkUSB-quick-start-manual-bas.pdf

mkUSB-quick-start-manual-nox-11.pdf

mkUSB-quick-start-manual-nox.pdf

oem.pdf


Let us hope that




  • there are no UFOs in your set of files

  • the sorting is correct concerning text versus scanned/images


[#6727] Sunday, September 12, 2021, 3 Years  [reply] [flag answer]
Only authorized users can answer the question. Please sign in first, or register a free account.
irripri
Add To Favorites
Follow
Total Points: 164
Total Questions: 111
Total Answers: 107
Location: South Georgia
Member since Sun, Aug 8, 2021
3 Years ago
;