I'm working with several huge CSV files - over 5 million rows total - and I ran into a problem: it turned out there were a few lines in those files that imported incorrectly because they were not correctly escaped.
\nHere's an example of an invalid line:
\nSAI Exempt,\"Patty B\"s Hats & Tees,LLC\",,26 Broad St\nThe apostrophe in Patty B's Hats & Tees is incorrectly represented here as a double quote, and since that's in a double quoted string it breaks that line of CSV.
I decided to filter out any rows that had an odd number of quotation marks in them - saving those broken lines to try and clean up later.
\nStackOverflow offered this regular expression for finding lines with an odd number of quotation marks:
\n[^\"]*\" # Match any number of non-quote characters, then a quote\n(?: # Now match an even number of quotes by matching:\n [^\"]*\" # any number of non-quote characters, then a quote\n [^\"]*\" # twice\n)* # and repeat any number of times.\n[^\"]* # Finally, match any remaining non-quote characters\nI translated this into a ripgrep expression, adding ^ to the beginning and $ to the end in order to match whole strings.
This command counted the number of invalid lines:
\nrg '^[^\"]*\"(?:[^\"]*\"[^\"]*\")*[^\"]*$' --glob '*.csv' --count\n\n04.csv:52\n03.csv:42\n02.csv:24\n01.csv:29\nAdding --invert-match showed me the count of lines that did NOT have an odd number of quotes:
rg '^[^\"]*\"(?:[^\"]*\"[^\"]*\")*[^\"]*$' --glob '*.csv' --count --invert-match\n\n05.csv:2829\n04.csv:812351\n03.csv:961311\n02.csv:994265\n01.csv:995404\nThis shows that the invalid lines are a tiny subset of the overall files.
\nRemoving --count shows the actual content.
I used this for loop to import only the valid lines into a SQLite database:
\nfor file in *.csv;\n do rg --invert-match '^[^\"]*\"(?:[^\"]*\"[^\"]*\")*[^\"]*$' $file | \\\n sqlite-utils insert my.db rows - --csv;\ndone;To save the lines that contained odd numbers of double quotes I used this command:
\nrg '^[^\"]*\"(?:[^\"]*\"[^\"]*\")*[^\"]*$' \\\n --glob '*.csv' \\\n --no-line-number \\\n --no-filename > saved.txtSince I don't actually care which file they lived in - all of these CSV files share the same structure - I used --no-filename to omit the filename from the results and --no-line-number to omit the line number. The result is a saved.txt file containing just the raw CSV data that I skipped from the import.