How to search fast in a big .TXT file
by Spaghetih - July 08, 2021 at 06:28 AM
#1
Hello, I would like to know which and the fastest solution to find a word in a large text file (3GB)

I found some shit but it's not fast
Reply
#2
Split it into multiple thousands of files.
Reply
#3
(July 08, 2021 at 06:29 AM)moot Wrote: Split it into multiple thousands of files.

that makes me 1200 txt file
Reply
#4
Going to need a bit more info to help

1. What OS are you using
2. Do you know what your searching for
3. How is the file structured e.g random data or a CSV file
4. HDD or SSD

For my purposes I use Linux with a SSD/RAM disk for storing the files so I can read it quickly without being limited by drive speed, sift (faster grep alternative) for text files and for structured stuff like CSV files I'll usually import them into something like MySQL so I can would with them easier.
Reply
#5
(July 08, 2021 at 04:06 PM)UnknownPhatom Wrote: Going to need a bit more info to help

1. What OS are you using
2. Do you know what your searching for
3. How is the file structured e.g random data or a CSV file
4. HDD or SSD

For my purposes I use Linux with a SSD/RAM disk for storing the files so I can read it quickly without being limited by drive speed, sift (faster grep alternative) for text files and for structured stuff like CSV files I'll usually import them into something like MySQL so I can would with them easier.

1) I'am using Windows but if Linux is needed i can download it
2) Yes i know what i am serching
3) Random data it's a dumped database
4) SSD

Thank you

(July 08, 2021 at 04:06 PM)UnknownPhatom Wrote: Going to need a bit more info to help

1. What OS are you using
2. Do you know what your searching for
3. How is the file structured e.g random data or a CSV file
4. HDD or SSD

For my purposes I use Linux with a SSD/RAM disk for storing the files so I can read it quickly without being limited by drive speed, sift (faster grep alternative) for text files and for structured stuff like CSV files I'll usually import them into something like MySQL so I can would with them easier.
Which linux are you using?
Reply
#6
If you're using Windows, have you checked out Agent Ransack or AstroGrep?
Reply
#7
(July 10, 2021 at 11:24 PM)viddied Wrote: If you're using Windows, have you checked out Agent Ransack or AstroGrep?

Thank you didn't know that
Reply
#8
Use grep on Linux.
So much better.
Reply
#9
notepad++ does pretty well with large datasets
that said cli tools will likely be the fastest but I'd use that if you want a gui
Reply
#10
@Spaghetih take a look at ultraedit.com products Shy
Reply
#11
linux grep is good to searching big file (I try search 20G)
Reply
#12
you can use Glogg to view, or GVim, if you need to edit.
Reply

 Users browsing this thread: 1 Guest(s)