Elegantly adding data to a pandas.Panel within a running simulation



This is a follow-up question from <a href="https://stackoverflow.com/questions/43519643/plotting-the-same-column-from-various-dataframes-in-a-panel" rel="nofollow">here</a>, where I had a pandas.Panel with several items consisting of pandas.DataFrames. I wanted to plot a certain column in my DataFrame (minor_axis in the Panel) from each item in only one command, avoiding a code cluster like plt.plot(x, DataFrame1[y1]) plt.plot(x, DataFrame2[y1]) ... It was brought as an answer that I could switch my axes in the Panel so that instead of one item containing all the information of one dataset (of a simulation with a certain starting parameter), but rather just one information (e.g. yvalue y1) for all the different simulations an storing other parameters in other items (DataFrames).

<hr />

My basic simulation code

Even though my code is to simulate the behaviour of a pendulum I'll break it down to a general simulation code with returned values y1-y3 instead of the real physical parameters. This simulation will be done for 2 different starting parameters k.

import pandas as pd data = pd.Panel(major_axis=[], minor_axis=['x', 'sim1', 'sim2']) # some kind of simulation resulting in 3 simulated values and with a # starting parameter for different simulation "strengths" # not sure whether to use a list or dict here ks = {'sim1' = 0.5, 'sim2' = 1.0} for k in ks: x, y1, y2, y3 = 0, 0, 0, 0 while x<100: x += 1 y1 += 1*ks[k]*x y2 += 2*ks[k]*x y3 += 3*ks[k]*x ... # for example the y2 value for the different k values should be plottable like this data['y2'].plot() <hr />


My question now is how to elegantly (as few lines of code as possible) add/append each value for each simulation to data, considering there could be 5 or more simulations with 10 or more values for each simulation step?

E.g. in my problem <a href="https://stackoverflow.com/questions/43519643/plotting-the-same-column-from-various-dataframes-in-a-panel" rel="nofollow">mentioned before</a> I'd create a new DataFrame and append it to my existing dataset for the given simulation - something like data.append(pd.DataFrame([[x, y1, y2, y3]], columns=['x', 'y1', 'y2', 'y3'])). But from there I couldn't plot properly with a single command but rather had to add a new graph for each simulation manually.

I'd be very happy if someone could help me understand how to build a Panel like this "on the run" - from my previous question I already know how to plot one :)

<hr /><strong>UPDATE</strong> I was asked for some example data, but since I want to consecutively add my simulated values into an Panel/item instead of generating a list first, I can only show how the data should look like in the end. In the beginning the Panel should look like this: In [1]: print(data) Out[1]: <class 'pandas.core.panel.Panel'> Dimensions: 2 (items) x 0 (major_axis) x 3 (minor_axis) Items axis: y1 to y2 Major_axis axis: None Minor_axis axis: x to sim2

In the following is shown how the simulations works and how for example the y1-item should look like in the end

In [2]: ks = {'sim1' : 0.5, 'sim2' : 1.0} Out[2]: {'sim1': 0.5, 'sim2': 1.0} In [3]: for k in ks: x, y1, y2 = 0, 0, 0 while x<3: x += 1 y1 += 1*ks[k]*x y2 += 2*ks[k]*x # HERE is missing what I'm looking for # it should append e.g. the y1 value to data['y1'] for both k Out[3]: ... In [4]: print(data['y1']) Out[4]: x sim1 sim2 0 1 0.5 1.0 1 2 1.5 3.0 2 3 3.0 6.0

I hope through this it's clearer now what I'm looking for - if not let me know


I think the easies way to build a Pandas.Panel would be to build a dictionary of the following form:

d = { 'items_axis_element0': DataFrame0, 'items_axis_element1': DataFrame1, 'items_axis_element2': DataFrame2, ... }

now you can easily build up a Panel:

p = pd.Panel(d)

You may find some usefull examples in <a href="http://pandas.pydata.org/pandas-docs/stable/cookbook.html#panels" rel="nofollow">Pandas Cookbook</a>

<hr />

<strong>UPDATE:</strong> here is slightly modified example from Pandas Cookbook:

rng = pd.date_range('1/1/2013',periods=100,freq='D') data = np.random.randn(100, 4) cols = ['A','B','C','D'] df1, df2, df3 = pd.DataFrame(data, rng, cols), pd.DataFrame(data, rng, cols), pd.DataFrame(data, rng, cols) pf = pd.Panel({'df1':df1,'df2':df2}) In [21]: pf Out[21]: <class 'pandas.core.panel.Panel'> Dimensions: 2 (items) x 100 (major_axis) x 4 (minor_axis) Items axis: df1 to df2 Major_axis axis: 2013-01-01 00:00:00 to 2013-04-10 00:00:00 Minor_axis axis: A to D

now we can add df3 as follows:

In [22]: pf.join(pd.Panel({'df3':df3})) Out[22]: <class 'pandas.core.panel.Panel'> Dimensions: 3 (items) x 100 (major_axis) x 4 (minor_axis) Items axis: df1 to df3 Major_axis axis: 2013-01-01 00:00:00 to 2013-04-10 00:00:00 Minor_axis axis: A to D


  • Use dictionary to replace a string within a string in Pandas columns
  • Python - How to convert JSON File to Dataframe
  • Linear regression analysis with string/categorical features (variables)?
  • Is the memory allocated by new operated consecutive?
  • Can't create a database in SQL Server 2012
  • How list all tables with data changes in the last 24 hours?
  • iOS: how to respond to up/down/left/right arrow event from physical keyboard?
  • Get Physical Keyboard Layout Programmatically
  • Piping output from aplay to arecord in centos
  • What to return when a route/url is found but not the resource behind it?
  • Invalidate JWT Token in NodeJS
  • Apache Spark selects all rows
  • Bigcommerce Python API, how do I create a product with an image?
  • UniVerse SQL View All Tablenames
  • Problems in creating unit test for ASP .NET MVC
  • Projecting Objects Out Of Collision
  • WCF and n-tier architecture and serialization performance
  • Using SSL to connect to mySQL from PHP
  • Use SendInput to lock the computer
  • Drop packet with libpcap
  • printk - showing in message log but not in terminal for any kernel log level
  • SQL Service Broker: Collecting data from distributed sources.
  • Root element minOccurs or maxOccurs
  • Linux over commit heuristic
  • Scala split a multi line string by lines that contain all hyphens
  • Insert records if not exist SQL Server 2005
  • How to filter on year and quarter in pandas
  • Color time-series based on column values in pandas
  • Android onKey w/ virtual keyboard
  • R convert summary result (statistics with all dataframe columns) into dataframe
  • Cannot upload to OneDrive using the new SDK
  • Error when parsing timestamp with pandas read_csv
  • Yii2: Config params vs. const/define
  • Ajax Loaded meta Tags
  • Android screen density dpi vs ppi
  • Why winpcap requires both .lib and .dll to run?
  • Return words with double consecutive letters
  • Python: how to group similar lists together in a list of lists?
  • KeystoneJS: Relationships in Admin UI not updating
  • Busy indicator not showing up in wpf window [duplicate]