Question

10

how to separate each content in the file?

rated 0 times [ 10] [ 0] / answers: 1 / hits: 921 / 2 Years ago, sat, january 15, 2022, 4:45:45

I have a text file (more then 1GB in size) and it contains lines like these:

1083021106e581c71003b987a75f18543cf5858b9fcfc5e04c0dddd79cd18764a865ba86d027de6d1900dc171e4d90a0564abbce99b812b821bd0d7d37aad72ead19c17

10840110dbd43121ef0c51a8ba62193eac247f57f1909e270eeb53d68da60ad61519f19cfb0511ec2431ca54e2fcabf6fa985615ec06def5ba1b753e8ad96d0564aa4c

1084011028375c62fd132d5a4e41ffef2419da345b6595fba8a49b5136de59a884d878fc9789009843c49866a0dc97889242b9fb0b8c112f1423e3b220bc04a2d7dfbdff

10880221005f0e261be654e4c52034d8d05b5c4dc0456b7868763367ab998b7d5886d64fbb24efd14cea668d00bfe8048eb8f096c3306bbb31aaea3e06710fa8c0bb8fca71

108501103461fca7077fc2f0d895048606b828818047a64611ec94443e52cc2d39c968363359de5fc76df48e0bf3676b73b1f8fea5780c2af22c507f83331cc0fbfe6ea9

1085022100a4ce8a09d1f28e78530ce940d6fcbd3c1fe2cb00e7b212b893ce78f8839a11868281179b4f2c812b8318f8d3f9a598b4da750a0ba6054d7e1b743bb67896ee62

1086022100638681ade4b306295815221c5b445ba017943ae59c4c742f0b1442dae4902a56d173a6f859dc6088b6364224ec17c4e2213d9d3c96bd9992b696d7c13b234b50

all strings staring with like below, .....

i need separate 8 files, how to do with sed command

Answers

Only authorized users can answer the question. Please sign in first, or register a free account.

kilusy

Add To Favorites

Follow

Total Points: 171

Total Questions: 110

Total Answers: 128

Location: Cayman Islands

Member since Sat, Dec 5, 2020

3 Years ago

kilusy questions

1 Asus rog strix z690-a gaming wifi d4 No sound in ubuntu 20.04

Sun, Dec 18, 22, 08:50, 1 Year ago

1 How to check for a specific string with linebreaks in a file with grep?

Tue, May 2, 23, 23:46, 1 Year ago

1 Ubuntu 20.04 no driver loaded for Intel Iris Xe Graphics

Tue, Aug 10, 21, 16:38, 3 Years ago

1 Restrict SSH-login with public-key only to selected users

Thu, Jan 27, 22, 03:19, 2 Years ago

1 pip3 not working on ubuntu 20.04: "has no attribute 'SourceFileLoader'"

Mon, Apr 11, 22, 08:39, 2 Years ago

View All

answered 2 Years ago kilusy · Accepted Answer

You could use sed to turn your file of prefixes into a file of sed commands, then use that in a sed command to process the large file - this will almost certainly be more efficient than using a shell loop to run sed (or grep) multiple times on the same (large) file. Ex. given

$ cat file2

10830110

1083021

10840110

10840110

1088022100

10850110

1085022100

1086022100

then

$ sed 's:.*:/^&/w&.txt:' file2

/10830110/w10830110.txt

/1083021/w1083021.txt

/10840110/w10840110.txt

/10840110/w10840110.txt

/1088022100/w1088022100.txt

/10850110/w10850110.txt

/1085022100/w1085022100.txt

/1086022100/w1086022100.txt

so that

$ sed 's:.*:/^&/w&.txt:' file2 | sed -n -f - file1

produces

$ head 108*.txt

==> 10830110.txt <==



==> 1083021.txt <==

1083021106e581c71003b987a75f18543cf5858b9fcfc5e04c0dddd79cd18764a865ba86d027de6d1900dc171e4d90a0564abbce99b812b821bd0d7d37aad72ead19c17



==> 10840110.txt <==

10840110dbd43121ef0c51a8ba62193eac247f57f1909e270eeb53d68da60ad61519f19cfb0511ec2431ca54e2fcabf6fa985615ec06def5ba1b753e8ad96d0564aa4c

10840110dbd43121ef0c51a8ba62193eac247f57f1909e270eeb53d68da60ad61519f19cfb0511ec2431ca54e2fcabf6fa985615ec06def5ba1b753e8ad96d0564aa4c

1084011028375c62fd132d5a4e41ffef2419da345b6595fba8a49b5136de59a884d878fc9789009843c49866a0dc97889242b9fb0b8c112f1423e3b220bc04a2d7dfbdff

1084011028375c62fd132d5a4e41ffef2419da345b6595fba8a49b5136de59a884d878fc9789009843c49866a0dc97889242b9fb0b8c112f1423e3b220bc04a2d7dfbdff



==> 10850110.txt <==

108501103461fca7077fc2f0d895048606b828818047a64611ec94443e52cc2d39c968363359de5fc76df48e0bf3676b73b1f8fea5780c2af22c507f83331cc0fbfe6ea9



==> 1085022100.txt <==

1085022100a4ce8a09d1f28e78530ce940d6fcbd3c1fe2cb00e7b212b893ce78f8839a11868281179b4f2c812b8318f8d3f9a598b4da750a0ba6054d7e1b743bb67896ee62



==> 1086022100.txt <==

1086022100638681ade4b306295815221c5b445ba017943ae59c4c742f0b1442dae4902a56d173a6f859dc6088b6364224ec17c4e2213d9d3c96bd9992b696d7c13b234b50



==> 1088022100.txt <==

10880221005f0e261be654e4c52034d8d05b5c4dc0456b7868763367ab998b7d5886d64fbb24efd14cea668d00bfe8048eb8f096c3306bbb31aaea3e06710fa8c0bb8fca71

You may want to de-duplicate the pattern file first - and possibly sort it numerically and modify the second sed command to break after the first match, so that you only match the longest prefix:

$ sort -nru file2 | sed 's:.*:/^&/{w&.txt

b

}:' | sed -n -f - file1

giving

$ head 108*.txt

==> 10830110.txt <==



==> 1083021.txt <==

1083021106e581c71003b987a75f18543cf5858b9fcfc5e04c0dddd79cd18764a865ba86d027de6d1900dc171e4d90a0564abbce99b812b821bd0d7d37aad72ead19c17



==> 10840110.txt <==

10840110dbd43121ef0c51a8ba62193eac247f57f1909e270eeb53d68da60ad61519f19cfb0511ec2431ca54e2fcabf6fa985615ec06def5ba1b753e8ad96d0564aa4c

1084011028375c62fd132d5a4e41ffef2419da345b6595fba8a49b5136de59a884d878fc9789009843c49866a0dc97889242b9fb0b8c112f1423e3b220bc04a2d7dfbdff



==> 10850110.txt <==

108501103461fca7077fc2f0d895048606b828818047a64611ec94443e52cc2d39c968363359de5fc76df48e0bf3676b73b1f8fea5780c2af22c507f83331cc0fbfe6ea9



==> 1085022100.txt <==

1085022100a4ce8a09d1f28e78530ce940d6fcbd3c1fe2cb00e7b212b893ce78f8839a11868281179b4f2c812b8318f8d3f9a598b4da750a0ba6054d7e1b743bb67896ee62



==> 1086022100.txt <==

1086022100638681ade4b306295815221c5b445ba017943ae59c4c742f0b1442dae4902a56d173a6f859dc6088b6364224ec17c4e2213d9d3c96bd9992b696d7c13b234b50



==> 1088022100.txt <==

10880221005f0e261be654e4c52034d8d05b5c4dc0456b7868763367ab998b7d5886d64fbb24efd14cea668d00bfe8048eb8f096c3306bbb31aaea3e06710fa8c0bb8fca71