78672

Convert list of pyodbc.rows to pandas Dataframe takes very long time

Question:

Is there a faster way to convert pyodbc.rows object to pandas Dataframe? It take about 30-40 minutes to convert a list of 10 million+ pyodbc.rows objects to pandas dataframe.

import pyodbc import pandas server = <server_ip> database = <db_name> username = <db_user> password = <password> port='1443' conn = pyodbc.connect('DRIVER={SQL Server};SERVER='+server+';PORT='+port+';DATABASE='+database+';UID='+username+';PWD='+ password) #takes upto 12 minutes rows = cursor.execute("select top 10000000 * from [LSLTGT].[MBR_DIM] ").fetchall() #Read cursor data into Pandas dataframe.....Takes forever! df = pandas.DataFrame([tuple(t) for t in rows])

Answer1:

You might get some improvement by using a generator expression rather than a list comprehension:

df = pandas.DataFrame((tuple(t) for t in rows))

Recommend

  • Executing a batch file containing an exe path remotely via wmic
  • Sending ByteArray using Java Sockets (Android programming)
  • Yii project not working on server
  • Is there a better way to get the public “properties” of a Ruby object?
  • PHP MYSQL : way to list table in ascending order
  • Error when submitting checkbox to MySQL
  • MySQLi using an IN in a where
  • Possible to get the line number of the currently executing sproc in SQL Server?
  • How to set an entity field that does not exist on the table but does exists in the raw SQL as an ali
  • It is possible use the same sql azure instance from two different cloud service of two different sub
  • TSQL Rolling Average of Time Groupings
  • distinct values from multiple fields within one table ORACLE SQL
  • What is corresponding c++ data type to SQL numeric(18,0) data type?
  • How can Delete be both a DDL and a DML statement
  • How to 'create temp table as select' in Slick?
  • pip in virtualenv gets ConnectTimeoutError
  • pyodbc doesn't report sql server error
  • nonblocking BIO_do_connect blocked when there is no internet connected
  • Redux Form - Not able to type anything in input
  • Get history of file changes from TFS to implement custom “blame”-behaviour of exceptions
  • PHPUnit_Framework_TestCase class is not available. Fix… - Makegood , Eclipse
  • PHP - How to update data to MySQL when click a radio button
  • Counter field in MS Access, how to generate?
  • Obtain ObjectIdHex value from mgo query
  • How to redirect a user to a different server and include HTTP basic authentication credentials?
  • How to extract text from Word files using C#?
  • How to check if every primary key value is being referenced as foreign key in another table
  • Sending data from AppleScript to FileMaker records
  • 'TypeError' while using NSGA2 to solve Multi-objective prob. from pyopt-sparse in OpenMDAO
  • using conditional logic : check if record exists; if it does, update it, if not, create it
  • Codeigniter doesn't let me update entry, because some fields must be unique
  • Getting error when using KSoap library to consume .NET web services
  • IndexOutOfRangeException on multidimensional array despite using GetLength check
  • How to get NHibernate ISession to cache entity not retrieved by primary key
  • costura.fody for a dll that references another dll
  • Observable and ngFor in Angular 2
  • How to Embed XSL into XML
  • UserPrincipal.Current returns apppool on IIS
  • Conditional In-Line CSS for IE and Others?
  • java string with new operator and a literal