I have two lists, the first of which represents times of observation and the second of which represents the observed values at those times. I am trying to find the maximum observed value and the corresponding time given a rolling window of various length. For example-sake, here are the two lists.
# observed values linspeed = [280.0, 275.0, 300.0, 475.2, 360.1, 400.9, 215.3, 323.8, 289.7] # times that correspond to observed values time_count = [4.0, 6.0, 8.0, 8.0, 10.0, 10.0, 10.0, 14.0, 16.0] # actual dataset is of size ~ 11,000
The missing times (ex: 3.0) correspond to an observed value of zero, whereas duplicate times correspond to multiple observations to the floored time. Since my window will be rolling over the
time_count (ex: max value in first 2 hours, next 2 hours, 2 hours after that; max value in first 4 hours, next 4 hours, ...), I plan to use an array-reshaping routine. However, it's important to set up everything properly before, which entails finding the maximum value given duplicate times. To solve this problem, I tried the code just below.
def list_duplicates(data_list): seen = set() seen_add = seen.add seen_twice = set(x for x in data_list if x in seen or seen_add(x)) return list(seen_twice) # check for duplicate values dups = list_duplicates(time_count) print(dups) >> [8.0, 10.0] # get index of duplicates for dup in dups: print(time_count.index(dup)) >> 2 >> 4
When checking for the index of the duplicates, it appears that this code will only return the index of the first occurrence of the duplicate value. I also tried using
OrderedDict via module
collections for reasons concerning code efficiency/speed, but dictionaries have a similar problem. Given duplicate keys for non-duplicate observation values, the first instance of the duplicate key and corresponding observation value is kept while all others are dropped from the dict. Per this SO post, my second attempt is just below.
for dup in dups: indexes = [i for i,x in enumerate(time_count) if x == dup] print(indexes) >> [4, 5, 6] # indices correspond to duplicate time 10s but not duplicate time 8s
I should be getting
time in time_count = 8.0 and
time in time_count = 10.0. From the duplicate time_counts,
475.2 is the
max linspeed that corresponds to duplicate
time_count 8.0 and
400.9 is the
max linspeed that corresponds to duplicate
time_count 10.0, meaning that the other linspeeds at leftover indices of duplicate time_counts would be removed.
I'm not sure what else I can try. How can I adapt this (or find a new approach) to find all of the indices that correspond to duplicate values in an efficient manner? Any advice would be appreciated. (PS - I made numpy a tag because I think there is a way to do this via numpy that I haven't figured out yet.)
Without going into the details of how to implement and efficient rolling-window-maximum filter; reducing the duplicate values can be seen as a grouping-problem, which the numpy_indexed package (disclaimer: I am its author) provides efficient and simple solutions to:
import numpy_indexed as npi unique_time, unique_speed = npi.group_by(time_count).max(linspeed)
For large input datasets (ie, where it matters), this should be a lot faster than any non-vectorized solution. Memory consumption is linear and performance in general NlogN; but since time_count appears to be sorted already, performance should be linear too.
OK, if you want to do this with numpy, best is to turn both of your lists into arrays:
l = np.array(linspeed) tc = np.array(time_count)
Now, finding unique times is just an
u, i, c = np.unique(tc, return_inverse = True, return_counts = True) u Out: array([ 4., 6., 8., 10., 14., 16.]) i Out: array([0, 1, 2, 2, 3, 3, 3, 4, 5], dtype=int32) c Out: array([1, 1, 2, 3, 1, 1])
Now you can either build your maximums with a
m = np.array([np.max(l[i==j]) if c[j] > 1 else l[j] for j in range(u.size)]) m Out: array([ 280. , 275. , 475.2, 400.9, 360.1, 400.9])
Or try some 2d method. This could be faster, but it would need to be optimized. This is just the basic idea.
np.max(np.where(i[None, :] == np.arange(u.size)[:, None], linspeed, 0),axis = 1) Out: array([ 280. , 275. , 475.2, 400.9, 323.8, 289.7])
u vectors are the same length and include the output you want.