1993

Python help to debug an error when modify a file in a stream mode

Question:

I have the following problem. I am reading a file x,y,z as:

481492.93 6244326.24 26.56 481493.03 6244325.60 25.06 481493.17 6244324.68 22.89 481493.50 6244322.52 17.80 481492.84 6244327.05 27.84 481492.90 6244326.66 26.90 481492.86 6244327.16 27.45 481493.48 6244323.08 17.79 481492.80 6244327.80 28.30 481492.94 6244326.84 26.04 ..........................

i wish to read, modify, and write on the same file (without create a back-up file because the originals file is more than 10GB)

481492.93 6244326.24 26.56 (375, 2902) 481493.03 6244325.60 25.06 (376, 2902) 481493.17 6244324.68 22.89 (377, 2902) 481493.50 6244322.52 17.80 (379, 2903) 481492.84 6244327.05 27.84 (375, 2902) 481492.90 6244326.66 26.90 (375, 2902) 481492.86 6244327.16 27.45 (374, 2902) 481493.48 6244323.08 17.79 (379, 2903) 481492.80 6244327.80 28.30 (374, 2902) 481492.94 6244326.84 26.04 (375, 2902) ..........................

i wrote the following approach

def get_point_grid_id(x,y,x_min,y_max,x_dist,y_dist): col = int((x - x_min)/x_dist) row = int((y_max - y)/y_dist) return (row, col) with open(file_temp, "r+") as f: for line in open(file_temp): x,y,z = line.split() id = get_point_grid_id(float(x),float(y),origin[0],origin[1],1,1) element = [x,y,z,id] newelement = " ".join([str(e) for e in element])+ "\n" f.write(newelement)

when i run the function i got the following error

Traceback (most recent call last): File "<editor selection>", line 3, in <module> ValueError: too many values to unpack

i suppose it's a connection problem with the original file

the error appears

>>> x,y,z = line.split() Traceback (most recent call last): File "<interactive input>", line 1, in <module> ValueError: too many values to unpack

where line is strange

'481499.82 6244470.31 29.23 (231, 2909)\n'

instead of '481499.82 6244470.31 29.23\n'

using print line after for line in open(file_temp): i got this print after run from a new file

481499.98 6244494.02 34.14 481499.98 6244494.02 34.14 (208, 2909) 481499.96 6244471.05 33.39 481499.96 6244471.05 33.39 (231, 2909) 481499.95 6244471.27 33.46 481499.95 6244471.27 33.46 (230, 2909) 481499.98 6244473.84 32.72 481499.98 6244473.84 32.72 (228, 2909) 481499.98 6244474.07 32.70 481499.98 6244474.07 32.70 (228, 2909) 481499.97 6244474.28 32.93 481499.97 6244474.28 32.93 (227, 2909) 481499.88 6244474.40 34.35 481499.88 6244474.40 34.35 (227, 2909)

Answer1:

opening the line in r+ mode means that you read a line, ie 38 characters read. Then you modify those 38 characters Then, at the current file position (character 39) you over write the existing data

I would guess this is not what you want

Hope this helps

Answer2:

This just isn't going to work. As Martijn said,

<h2>I</h2>

file objects have a buffer position. Every time you read a character the buffer position advances by 1. Suppose you read a line that's 10 characters long:

>>> myfile = open('some_file.txt') >>> myfile.tell() #gets the buffer position 0 >>> myfile.readline() '012345678\n'

Now the buffer position is advanced by len(line) characters:

>>> myfile.tell() 10

This means that when you call myfile.write(), it starts writing at position 10.

<h2>II</h2>

You simply can't "insert" characters into a file without overwriting something, or appending characters to the end (assuming that the buffer position is at the end of the file).

<h2>So what do you do?</h2>

You can create a temporary file, and simultaneously read from your input file, and write to your temp file. Afterwards (if you should wish), you can replace your original file with the temporary one:

with open(input_file) as infile, open(output_temp_file, "w") as outfile: for line in infile: x, y, z = line.split() new_line = ' '.join([x, y, z] + [function_of_xyz(x, y, z)]) + '\n' outfile.write(new_line)

You should also check out the <a href="http://docs.python.org/2/library/csv.html" rel="nofollow">csv module</a>.

Recommend

  • how can convert $files contain to array?
  • python read a file, save a new column for each line ad save the same file
  • Compare a column between 2 csv files and write differences using Python
  • href inside href [duplicate]
  • Scala: Function returning an unknown type
  • Google Maps api v3 get start and end coordinates of a street
  • gspread or such: help me get cell coordinates (not value)
  • Groovy: Unexpected token “:”
  • Do I need to reset a Perl hash index?
  • Magento Fatal error: Maximum execution error solution, on WAMP
  • Word Open XML Mail Merge
  • Bad request using file_get_contents for PUT request in PHP
  • Python CGI os.system causing malformed header
  • Spring Data JPA custom method causing PropertyReferenceException
  • FB SDK and cURL: Unknown SSL protocol error in connection to graph.facebook.com:443
  • Using $this when not in object context
  • How do I fake an specific browser client when using Java's Net library?
  • Opengl-es onTouchEvents problem or a draw problem? [closed]
  • How reduce the height of an mschart by breaking up the y-axis
  • How to redirect a user to a different server and include HTTP basic authentication credentials?
  • Perl system calls when running as another user using sudo
  • Can I make an Android app that runs a web view in Chrome 39?
  • align graphs with different xlab
  • Unanticipated behavior
  • using conditional logic : check if record exists; if it does, update it, if not, create it
  • Rails 2: use form_for to build a form covering multiple objects of the same class
  • How get height of the a view with gone visibility and height defined as wrap_content in xml?
  • FormattedException instead of throw new Exception(string.Format(…)) in .NET
  • How to stop GridView from loading again when I press back button?
  • LevelDB C iterator
  • Linking SubReports Without LinkChild/LinkMaster
  • Can't mass-assign protected attributes when import data from csv file
  • Django query for large number of relationships
  • Sorting a 2D array using the second column C++
  • Why is Django giving me: 'first_name' is an invalid keyword argument for this function?
  • How can I use `wmic` in a Windows PE script?
  • Unable to use reactive element in my shiny app
  • git trying to push non-existent file … after clearing cache
  • java string with new operator and a literal
  • How to push additional view controllers onto NavigationController but keep the TabBar?