Python Read Huge Csv File Line by Line

What is a CSV file?

A CSV file is a simple type of apparently text file which uses a specific structure to suit tabular data. The standard format of a CSV file is divers past rows and columns data where a newline terminates each row to begin the next row, and each cavalcade is separated by a comma within the row.

CSV is a common format for data interchange as it is compact, simple, and general. Many online services allow their users to consign tabular data from the website into a CSV file. Files of CSV will open into Excel, and near all databases have a tool to let import from CSV files.

In this tutorial, you will learn:

• What is a CSV file?
• CSV Sample File
• Python CSV Module
• CSV Module Functions
• How to Read a CSV File in Python
• How to read a CSV file into a Dictionary in Python
• How to write CSV File in Python
• Read CSV File using Pandas
• Write CSV File using Pandas

CSV Sample File

Data in the form of tables is also called CSV (comma separated values) – literally "comma-separated values." This is a text format intended for the presentation of tabular information. Each line of the file is ane line of the table. The values of individual columns are separated by a separator symbol – a comma (,), a semicolon (;) or another symbol. CSV can be easily read and processed by Python.

Consider the following Tabular array

Table Information

Programming language	Designed past	Appeared	Extension
Python	Guido van Rossum	1991	.py
Java	James Gosling	1995	.java
C++	Bjarne Stroustrup	1983	.cpp

You can represent this table in csv as below.

CSV Data

Programming language, Designed by, Appeared, Extension

Python, Guido van Rossum, 1991, .py

Coffee, James Gosling, 1995, .java

C++, Bjarne Stroustrup,1983,.cpp

Every bit yous can see each row is a new line, and each column is separated with a comma. This is an example of how a CSV file looks similar.

Download CSV Information

Python CSV Module

Python provides a CSV module to handle CSV files. To read/write data, you demand to loop through rows of the CSV. Y'all need to use the split method to get data from specified columns.

CSV Module Functions

In CSV module documentation you lot can find post-obit functions:

• csv.field_size_limit – return maximum field size
• csv.get_dialect – get the dialect which is associated with the name
• csv.list_dialects – show all registered dialects
• csv.reader – read data from a csv file
• csv.register_dialect – associate dialect with proper noun
• csv.author – write information to a csv file
• csv.unregister_dialect – delete the dialect associated with the proper noun the dialect registry
• csv.QUOTE_ALL – Quote everything, regardless of type.
• csv.QUOTE_MINIMAL – Quote fields with special characters
• csv.QUOTE_NONNUMERIC – Quote all fields that aren't numbers value
• csv.QUOTE_NONE – Don't quote anything in output

In this tutorial, nosotros are going to focus only on the reader and writer functions which allow you to edit, modify, and manipulate the data in a CSV file.

How to Read a CSV File in Python

Beneath are steps to read CSV file in Python.

Step 1) To read data from CSV files, you lot must utilize the reader function to generate a reader object.

The reader function is developed to take each row of the file and make a listing of all columns. Then, you have to cull the column you lot want the variable data for.

It sounds a lot more than intricate than it is. Let's have a look at this Python lawmaking to read CSV file, and we volition find out that working with csv file isn't so hard.

#import necessary modules import csv with open('X:\data.csv','rt')equally f:   data = csv.reader(f)   for row in data:         print(row)            

Stride 2) When you execute the plan higher up, the output will exist:

['Programming language; Designed by; Appeared; Extension'] ['Python; Guido van Rossum; 1991; .py'] ['Java; James Gosling; 1995; .java'] ['C++; Bjarne Stroustrup;1983;.cpp']            

How to read a CSV file into a Dictionary in Python

Y'all can also you lot use DictReader to read CSV files. The results are interpreted every bit a dictionary where the header row is the key, and other rows are values.

Consider the post-obit code

#import necessary modules import csv  reader = csv.DictReader(open("file2.csv")) for raw in reader:     print(raw)          

The result of this lawmaking is:

OrderedDict([('Programming language', 'Python'), ('Designed by', 'Guido van Rossum'), (' Appeared', ' 1991'), (' Extension', ' .py')]) OrderedDict([('Programming language', 'Java'), ('Designed past', 'James Gosling'), (' Appeared', ' 1995'), (' Extension', ' .java')]) OrderedDict([('Programming language', 'C++'), ('Designed by', ' Bjarne Stroustrup'), (' Appeared', ' 1985'), (' Extension', ' .cpp')])

And this way to read data from CSV file is much easier than earlier method. All the same, this is non isn't the best way to read data.

How to write CSV File in Python

Here is how to write a CSV file in Python:

When you lot have a gear up of data that you would like to store in a CSV file you accept to use writer() function. To iterate the data over the rows(lines), you take to utilise the writerow() function.

Consider the following case. We write information into a file "writeData.csv" where the delimiter is an apostrophe.

#import necessary modules import csv  with open('X:\writeData.csv', mode='w') as file:     author = csv.writer(file, delimiter=',', quotechar='"', quoting=csv.QUOTE_MINIMAL)      #way to write to csv file     writer.writerow(['Programming language', 'Designed by', 'Appeared', 'Extension'])     author.writerow(['Python', 'Guido van Rossum', '1991', '.py'])     author.writerow(['Coffee', 'James Gosling', '1995', '.java'])     writer.writerow(['C++', 'Bjarne Stroustrup', '1985', '.cpp'])            

Result in csv file is:

Programming linguistic communication, Designed by, Appeared, Extension  Python, Guido van Rossum, 1991, .py Java, James Gosling, 1995, .java C++, Bjarne Stroustrup,1983,.cpp

Read CSV File using Pandas

Pandas is an opensource library that allows to you import CSV in Python and perform data manipulation. Pandas provide an easy style to create, manipulate and delete the information.

You must install pandas library with command <lawmaking>pip install pandas</code>. In Windows, you will execute this command in Command Prompt while in Linux in the Final.

Reading the CSV into a pandas DataFrame is very quick and easy:

#import necessary modules import pandas result = pandas.read_csv('X:\data.csv') print(result)          

Event of the read CSV Pandas example:

Programming language, Designed by, Appeared, Extension  0    Python, Guido van Rossum, 1991, .py 1    Java, James Gosling, 1995, .java ii    C++, Bjarne Stroustrup,1983,.cpp

Very useful library. In just three lines of code you the same event as before. Pandas know that the first line of the CSV contained cavalcade names, and it volition utilise them automatically.

Write CSV File using Pandas

Writing to CSV file with Pandas is as easy as reading. Hither yous can convince in it. First you must create DataFrame based on the post-obit Python write to CSV lawmaking.

from pandas import DataFrame C = {'Programming language': ['Python','Java', 'C++'],         'Designed by': ['Guido van Rossum', 'James Gosling', 'Bjarne Stroustrup'],         'Appeared': ['1991', '1995', '1985'],         'Extension': ['.py', '.java', '.cpp'],     } df = DataFrame(C, columns= ['Programming linguistic communication', 'Designed past', 'Appeared', 'Extension']) export_csv = df.to_csv (r'X:\pandaresult.csv', alphabetize = None, header=Truthful) # hither y'all take to write path, where result file will be stored print (df)          

Here is the output

Programming language, Designed by, Appeared, Extension 0    Python, Guido van Rossum, 1991, .py 1    Coffee, James Gosling, 1995, .java 2    C++, Bjarne Stroustrup,1983,.cpp

And CSV file is created at the specified location.

Conclusion

So, now you know how apply method 'csv' and likewise read and write data in CSV format. CSV files are widely used in software applications because they are easy to read and manage, and their small size makes them relatively fast for processing and transmission.

The csv module provides diverse functions and classes which allow you to read and write easily. You lot can look at the official Python documentation and find some more interesting tips and modules. CSV is the best style for saving, viewing, and sending information. Really, it isn't so difficult to acquire as it seems at the beginning. Only with a lilliputian practise, you'll main it.

Pandas is a groovy alternative to read CSV files.

As well, in that location are other ways to parse text files with libraries similar ANTLR, PLY, and PlyPlus. They tin can all handle heavy-duty parsing, and if simple String manipulation doesn't piece of work, there are regular expressions which you can use.

williamsforneirdis1961.blogspot.com

Source: https://www.guru99.com/python-csv.html