I'd like to read a file line by line, except for the last N lines. How do I know where to stop, without reaching the end of the file and back tracking / discarding the last N lines, in Python? Is asking for # lines = X, and looping (X-N) a good way to go about this?
What's the simplest / most Pythonic way of doing this?
Three different solutions:
<strong>1) Quick and dirty</strong>, see John's answer:
with open(file_name) as fid: lines = fid.readlines() for line in lines[:-n_skip]: do_something_with(line)
The disadvantage of this method is that you have to read all lines in memory first, which might be a problem for big files.
<strong>2) Two passes</strong>
Process the file twice, once to count the number of lines
n_lines, and in a second pass process only the first
n_lines - n_skip lines:
# first pass to count with open(file_name) as fid: n_lines = sum(1 for line in fid) # second pass to actually do something with open(file_name) as fid: for i_line in xrange(n_lines - n_skip): # does nothing if n_lines <= n_skip line = fid.readline() do_something_with(line)
The disadvantage of this method is that you have to iterate over the file twice, which might be slower in some cases. The good thing, however, is that you never have more than one line in memory.
<strong>3) Use a buffer</strong>, similar to Serge's solution
In case you want to iterate over the file just once, you only know for sure that you can process line
i if you know that line
i + n_skip exists. This means that you have to keep
n_skip lines in a temporary buffer first. One way to do this is to implement some sort of FIFO buffer (e.g. with a generator function that implements a circular buffer):
def fifo(it, n): buffer = [None] * n # preallocate buffer i = 0 full = False for item in it: # leaves last n items in buffer when iterator is exhausted if full: yield buffer[i] # yield old item before storing new item buffer[i] = item i = (i + 1) % n if i == 0: # wrapped around at least once full = True
Quick test with a range of numbers:
In : for i in fifo(range(20), 5): ...: print i, 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
The way you would use this with your file:
with open(file_name) as fid: for line in fifo(fid, n_skip): do_something_with(line)
Note that this requires enough memory to temporary store
n_skip lines, but this is still better than reading all lines in memory as in the first solution.
Which one of these 3 methods is the best is a trade-off between code complexity, memory and speed, which depends on your exact application.
Unless you have a way to know in advance the actual number of lines, you will have to read the whole file.
But as I assume you want to process the file line by line except the N last line, you can do it without loading all the file in memory, and keeping only a list of N lines :
with open(file) as fd: lines =  try: for i in range(N): lines.append(next(fd)) i = 0 for line in fd: # process lines[i] print (lines[i].rstrip()) lines[i] = line i = (i + 1) % N except StopIteration: print "less than %d lines" % (N,)
To read all lines up to the last X lines you need to know where the last X lines begin. You will need this information somewhere. There are several ways to get this information.<ol> <li>When you write the file save the position of the last X lines. Stop reading when reaching that position.</li> <li>Store the positions of the line beginnings somewhere, this allows appending to the file.</li> <li>You know the size of the lines. <ol> <li>Each line could have the same size and you compute it out of the file size</li> <li>Each line has at least one character so you do not need to read the last X characters.</li> </ol></li> </ol>
Given we know the file must be read to the end to determine how many lines there are, here's my attempt at the "simplest / most Pythonic way" of reading up to the last
with open(foo, 'r') as f: lines = f.readlines()[:-n]