84547

How can I remove indices of non-max values that correspond to duplicate values of separate list from

I have two lists, the first of which represents times of observation and the second of which represents the observed values at those times. I am trying to find the maximum observed value and the corresponding time given a rolling window of various length. For example-sake, here are the two lists.

# observed values linspeed = [280.0, 275.0, 300.0, 475.2, 360.1, 400.9, 215.3, 323.8, 289.7] # times that correspond to observed values time_count = [4.0, 6.0, 8.0, 8.0, 10.0, 10.0, 10.0, 14.0, 16.0] # actual dataset is of size ~ 11,000

The missing times (ex: 3.0) correspond to an observed value of zero, whereas duplicate times correspond to multiple observations to the floored time. Since my window will be rolling over the time_count (ex: max value in first 2 hours, next 2 hours, 2 hours after that; max value in first 4 hours, next 4 hours, ...), I plan to use an array-reshaping routine. However, it's important to set up everything properly before, which entails finding the maximum value given duplicate times. To solve this problem, I tried the code just below.

def list_duplicates(data_list): seen = set() seen_add = seen.add seen_twice = set(x for x in data_list if x in seen or seen_add(x)) return list(seen_twice) # check for duplicate values dups = list_duplicates(time_count) print(dups) >> [8.0, 10.0] # get index of duplicates for dup in dups: print(time_count.index(dup)) >> 2 >> 4

When checking for the index of the duplicates, it appears that this code will only return the index of the first occurrence of the duplicate value. I also tried using OrderedDict via module collections for reasons concerning code efficiency/speed, but dictionaries have a similar problem. Given duplicate keys for non-duplicate observation values, the first instance of the duplicate key and corresponding observation value is kept while all others are dropped from the dict. Per this SO post, my second attempt is just below.

for dup in dups: indexes = [i for i,x in enumerate(time_count) if x == dup] print(indexes) >> [4, 5, 6] # indices correspond to duplicate time 10s but not duplicate time 8s

I should be getting [2,3] for time in time_count = 8.0 and [4,5,6] for time in time_count = 10.0. From the duplicate time_counts, 475.2 is the max linspeed that corresponds to duplicate time_count 8.0 and 400.9 is the max linspeed that corresponds to duplicate time_count 10.0, meaning that the other linspeeds at leftover indices of duplicate time_counts would be removed.

I'm not sure what else I can try. How can I adapt this (or find a new approach) to find all of the indices that correspond to duplicate values in an efficient manner? Any advice would be appreciated. (PS - I made numpy a tag because I think there is a way to do this via numpy that I haven't figured out yet.)

Answer1:

Without going into the details of how to implement and efficient rolling-window-maximum filter; reducing the duplicate values can be seen as a grouping-problem, which the numpy_indexed package (disclaimer: I am its author) provides efficient and simple solutions to:

import numpy_indexed as npi unique_time, unique_speed = npi.group_by(time_count).max(linspeed)

For large input datasets (ie, where it matters), this should be a lot faster than any non-vectorized solution. Memory consumption is linear and performance in general NlogN; but since time_count appears to be sorted already, performance should be linear too.

Answer2:

OK, if you want to do this with numpy, best is to turn both of your lists into arrays:

l = np.array(linspeed) tc = np.array(time_count)

Now, finding unique times is just an np.unique call:

u, i, c = np.unique(tc, return_inverse = True, return_counts = True) u Out[]: array([ 4., 6., 8., 10., 14., 16.]) i Out[]: array([0, 1, 2, 2, 3, 3, 3, 4, 5], dtype=int32) c Out[]: array([1, 1, 2, 3, 1, 1])

Now you can either build your maximums with a for loop

m = np.array([np.max(l[i==j]) if c[j] > 1 else l[j] for j in range(u.size)]) m Out[]: array([ 280. , 275. , 475.2, 400.9, 360.1, 400.9])

Or try some 2d method. This could be faster, but it would need to be optimized. This is just the basic idea.

np.max(np.where(i[None, :] == np.arange(u.size)[:, None], linspeed, 0),axis = 1) Out[]: array([ 280. , 275. , 475.2, 400.9, 323.8, 289.7])

Now your m and u vectors are the same length and include the output you want.

Recommend

  • PHP: certain time/date values not being marked as conflicts (most others are though)
  • Python - Skip header row with csv.reader
  • Tensorflow batch_join's allow_smaller_final_batch doesn't work?
  • Drawing a correlation graph in matplotlib
  • how does “not x” in python set method works?
  • Table generated in by Javascript not displaying
  • Converting a R ListVector to a Python list with rpy2
  • dividing duplicates by the counter of duplicates
  • Fetch all the duplicate records from a list without quadratic time complexity?
  • Efficient User-Agent Regex to find Safari in Python
  • pickle.PicklingError: args[0] from __newobj__ args has the wrong class with hadoop python
  • Scanner nextInt() and hasNextInt() problems
  • Is it possible to run clang with llc flags
  • Cannot get text from text area
  • Getting short path in python
  • Getting different value with placeholder over CPU/GPU
  • there is no graph with tensorboard
  • Group list of tuples by item
  • Redirect STDERR in OPEN pipe comand. Perl Linux
  • Get specific string
  • How to remove a SwiftyJSON element?
  • Set matplotlib plot axis to be the dataframe column name
  • Appending Character to Character Array In C
  • Do I need to reset a Perl hash index?
  • Django rest serializer Breaks when data exists
  • Is it possible to access block's scope in method?
  • Checking free space on FTP server
  • Can I check if a recipient has an automatic reply before I send an email?
  • align graphs with different xlab
  • Return words with double consecutive letters
  • Angular 2 constructor injection vs direct access
  • Java static initializers and reflection
  • Android Google Maps API OnLocationChanged only called once
  • Django query for large number of relationships
  • Why is Django giving me: 'first_name' is an invalid keyword argument for this function?
  • Reading document lines to the user (python)
  • How can I use `wmic` in a Windows PE script?
  • UserPrincipal.Current returns apppool on IIS
  • Python/Django TangoWithDjango Models and Databases
  • How to push additional view controllers onto NavigationController but keep the TabBar?