26032

Simple Way of NOT reading last N lines of a file in Python

I'd like to read a file line by line, except for the last N lines. How do I know where to stop, without reaching the end of the file and back tracking / discarding the last N lines, in Python? Is asking for # lines = X, and looping (X-N) a good way to go about this?

What's the simplest / most Pythonic way of doing this?

Answer1:

Three different solutions:

<strong>1) Quick and dirty</strong>, see John's answer:

with open(file_name) as fid: lines = fid.readlines() for line in lines[:-n_skip]: do_something_with(line)

The disadvantage of this method is that you have to read all lines in memory first, which might be a problem for big files.

<strong>2) Two passes</strong>

Process the file twice, once to count the number of lines n_lines, and in a second pass process only the first n_lines - n_skip lines:

# first pass to count with open(file_name) as fid: n_lines = sum(1 for line in fid) # second pass to actually do something with open(file_name) as fid: for i_line in xrange(n_lines - n_skip): # does nothing if n_lines <= n_skip line = fid.readline() do_something_with(line)

The disadvantage of this method is that you have to iterate over the file twice, which might be slower in some cases. The good thing, however, is that you never have more than one line in memory.

<strong>3) Use a buffer</strong>, similar to Serge's solution

In case you want to iterate over the file just once, you only know for sure that you can process line i if you know that line i + n_skip exists. This means that you have to keep n_skip lines in a temporary buffer first. One way to do this is to implement some sort of FIFO buffer (e.g. with a generator function that implements a circular buffer):

def fifo(it, n): buffer = [None] * n # preallocate buffer i = 0 full = False for item in it: # leaves last n items in buffer when iterator is exhausted if full: yield buffer[i] # yield old item before storing new item buffer[i] = item i = (i + 1) % n if i == 0: # wrapped around at least once full = True

Quick test with a range of numbers:

In [12]: for i in fifo(range(20), 5): ...: print i, 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14

The way you would use this with your file:

with open(file_name) as fid: for line in fifo(fid, n_skip): do_something_with(line)

Note that this requires enough memory to temporary store n_skip lines, but this is still better than reading all lines in memory as in the first solution.

Which one of these 3 methods is the best is a trade-off between code complexity, memory and speed, which depends on your exact application.

Answer2:

Unless you have a way to know in advance the actual number of lines, you will have to read the whole file.

But as I assume you want to process the file line by line except the N last line, you can do it without loading all the file in memory, and keeping only a list of N lines :

with open(file) as fd: lines = [] try: for i in range(N): lines.append(next(fd)) i = 0 for line in fd: # process lines[i] print (lines[i].rstrip()) lines[i] = line i = (i + 1) % N except StopIteration: print "less than %d lines" % (N,)

Answer3:

To read all lines up to the last X lines you need to know where the last X lines begin. You will need this information somewhere. There are several ways to get this information.

<ol> <li>When you write the file save the position of the last X lines. Stop reading when reaching that position.</li> <li>Store the positions of the line beginnings somewhere, this allows appending to the file.</li> <li>You know the size of the lines. <ol> <li>Each line could have the same size and you compute it out of the file size</li> <li>Each line has at least one character so you do not need to read the last X characters.</li> </ol></li> </ol>

Answer4:

Given we know the file must be read to the end to determine how many lines there are, here's my attempt at the "simplest / most Pythonic way" of reading up to the last n lines:

with open(foo, 'r') as f: lines = f.readlines()[:-n]

Recommend

  • C++ Gnuplot pipe input from C++ defined variables
  • Remove weekend gaps in gnuplot for candlestick chart
  • Crop image, change black area if not not enough photo region in white
  • Webdriver Xpath Performance
  • Why are Pickle files in Pickle protocol 4 twice as large as those in protocol 3 without having any g
  • OpenMP for dependent variables
  • How to read data from socket connection - android
  • C# where to add a method
  • How do references in functions work?
  • vi mode to emacs mode while on R
  • Crafting a LINQ based solution to determine if a set of predicates are satisfied for a pair of colle
  • how to remove a div with same ids but display='block' and display='none' in JAVa
  • Using an array for autofilter criteria
  • Is there any way to use wpdb prepare statements for array implode(' OR ', $myArray)?
  • Classic ASP URL Rewriting
  • Count from each distinct date, fill in missing dates with zero
  • Opaque reference instead of PImpl. Is it possible?
  • Count New Lines in Text File
  • Creating NSCollectionView with datasource programatically
  • SQL: Getting the physical size of a subset of a table
  • R convert summary result (statistics with all dataframe columns) into dataframe
  • Add dynamic data to line chart from mysql database with highcharts
  • msbuild create itemgroup from property group
  • Approximate Order-Preserving Huffman Code
  • Updating both a ConcurrentHashMap and an AtomicInteger safely
  • Using Sax parsing to edit and write XML in VB6
  • Reading a file into a multidimensional array
  • Grails calculated field in SQL
  • Django: Count of Group Elements
  • Is there a javascript serializer for JSON.Net?
  • Is possible to count alias result on mysql
  • Sending data from AppleScript to FileMaker records
  • MySQL WHERE-condition in procedure ignored
  • Where to put my custom functions in Wordpress?
  • Buffer size for converting unsigned long to string
  • costura.fody for a dll that references another dll
  • Binding checkboxes to object values in AngularJs
  • Observable and ngFor in Angular 2
  • UserPrincipal.Current returns apppool on IIS
  • java string with new operator and a literal