77353

Largest number of rows in a csv python can handle?

Question:

import csv with open("C:\Anaconda3\FalkParameters\AllModels.csv","r") as f: reader = csv.reader(f) listofModels = list(reader)

AllModels is a huge csv file (2.4 GB). I am not sure how many rows are in there, because I can't open it with any application (notepad, notepad++, excel, etc.).

The code above runs okay, but if I try to print "listofModels", the console starts printing, but at some point I get a "ValueError: I/O operation on closed file". Printing stops at different rows each time.

I was wondering if there is a way I can deal with huge csv files in python.

Answer1:

Don't convert it to list. Rather use generator instead. print one value/line at a time.

for line in reader: print line

Hope this helps !

Recommend

  • TextPad “find in files” not matching on simple OR regex
  • PE file - what's missing?
  • Easier way to do non-recursive list on gcs bucket outside appengine
  • Printing out Japanese (Chinese) characters
  • VBScript to check for open process by user
  • @Autowired for @ModelAttribute
  • subprocess.Popen('start') fails
  • How to package a jar and all dependencies within a new jar with maven
  • PHP in userdir not working
  • VB.Net Double comparison after some additions
  • Extracting individual digits from a float
  • Creating UDF with VSTO in Excel
  • VBA Dir function not working on Excel 2010
  • Outlook to Excel hyperlink issue
  • C# Excel interop - how to test if interop object is still working and performing a task?
  • rewrite uppercase url to lowercase url htaccess
  • Access user's phone number on iOS 7
  • Memory error in python- how to use more memory
  • Sonar maven jacoco code coverage for Multimodule project
  • ViewController With Transparent Background Entering Current ViewController With Push Transition
  • C# - Is there a limit to the size of an httpWebRequest stream?
  • Is my CUDA kernel really runs on device or is being mistekenly executed by host in emulation?
  • Optimizing database types to compact database (SQLite)
  • C# - Serializing and deserializing static member
  • Sending data from AppleScript to FileMaker records
  • Perl system calls when running as another user using sudo
  • vba code to select only visible cells in specific column except heading
  • Do I've to free mysql result after storing it?
  • Run Powershell script from inside other Powershell script with dynamic redirection to file
  • Transpose CSV data with awk (pivot transformation)
  • How to include full .NET prerequisite for Wix Burn installer
  • Load html files in TinyMce
  • How can I get HTML syntax highlighting in my editor for CakePHP?
  • Acquiring multiple attributes from .xml file in c#
  • coudnt use logback because of log4j
  • How to CLICK on IE download dialog box i.e.(Open, Save, Save As…)
  • How can I remove ASP.NET Designer.cs files?
  • Is it possible to post an object from jquery to bottle.py?
  • Sorting a 2D array using the second column C++
  • java string with new operator and a literal