40880

apply function to two pandas dataframes in python (scipy.stats.spearmanr for each row from two dataf

Question:

I have two panda dataframe: price and sales dataframe.

price dataframe records price for each product (columns) in each year (index)

|a |b |c |d |e | 2018|3.2|4.5|5.6|7.8|8.1| 2017|6.2|1.5|2.6|7.8|2.1| 2016|2.2|9.5|0.6|6.8|4.1| 2015|2.2|6.5|7.6|7.8|2.1|

sales dataframe (see below) records sales for each product (columns) in each year (index)

|a |b |c |d |e | 2018|101|405|526|108|801| 2017|601|105|726|308|201| 2016|202|965|856|408|411| 2015|322|615|167|458|211|

I would like to calculate spearman correlation between price and sales for each year. I know scipy.stats.spearmanr function does the similar job, but I need to apply scipy.stats.spearmanr fucction for each row in the two dataframes.

For example, for 2018, i need to calculate the spearman correlation between

|a |b |c |d |e | 2018|3.2|4.5|5.6|7.8|8.1|

and

|a |b |c |d |e | 2018|101|405|526|108|801|

May I know what is the best to do that? The results i want a output like below:

2018|spearman cor btw price and sales in 2018 2017|spearman cor btw price and sales in 2017 2016|spearman cor btw price and sales in 2016

Answer1:

Guess you could do

import scipy.stats as st >>> pd.Series(map(lambda k: st.spearmanr(k[0], k[1])[0], zip(df.values, df2.values)), index=df.index) 2018 0.7 2017 0.6 2016 0.3 2015 0.2 dtype: float64

Recommend

  • Reading CSV file in matlab [duplicate]
  • How to create a histogram based on true or false in R?
  • R: index() or index.xts() changes the values of Date of a time series, why?
  • Make a numpy array monotonic without a Python loop
  • dropdown display just js
  • AVAssetExportSession no audio (iPhone), works on iPad
  • vectorizing an R-loop with backward dependency
  • R: Random sampling an even number of observations from a range of categories
  • XSLT autocomplete with NetBeans 6.8
  • IndexAxisValueFormatter not working as expected
  • Not able to send json tweets events to Kafka topic/producer using kafka command line
  • Can't get JSON data from Rails API
  • Comparing floating point numbers with numpy and scipy
  • Dendrogram or Other Plot from Distance Matrix
  • Spring Web Security locks Neo4j embedded database
  • Android Error: Binary XML file line #16: Error inflating class fragment when adding map fragment
  • “Yesod devel” fails with fromJust in devel.hs
  • Greek letters in a GUI - PYTHON
  • Matplotlib rotate image file by X degrees
  • LyX — Title, Abstract, Keywords Missing When Using Elsevier Class in View
  • Spring Integration debounce/deduplicate
  • What's an elegant way of accessing parent controller's member from child controller?
  • NHibernate proxyexception
  • MongoDB GeoJSON “Can't extract geo keys from object, malformed geometry?” when inserting type P
  • RabbitMQ java client stops consuming messages
  • Couchbase: Connection refused
  • Cannot find “Grammar.txt” in python-sphinx
  • Suqueries in select clause with JPA
  • Calculating ratio of reciprocated ties for each node in igraph
  • During installation of Django, why do I keep getting ImportError: No module named django?
  • where do I find the xml.dom python package for the python-2.6.0-8.9.28 and I have a suse/x86_64 vers
  • zope_i18n_compile_mo_files doesn't work on a Zeo configuration
  • Installing iPhone App to iPhone
  • Mysterious problem with floating point in LISP - time axis generation
  • JSON response opens as a file, but I can't access it with JavaScript
  • Could not find rake using whenever rails
  • Cannot resolve symbol 'MyApi'
  • If I include Java 8 in my Android app does that affect which devices it will work on?
  • Cannot connect to cassandra from Spark
  • Matplotlib draw Spline from multiple points