How to remove all dupes from an array of objects?

// This is a large array of objects, e.g.: let totalArray = [ {"id":"rec01dTDP9T4ZtHL4","fields": {"user_id":170180717,"user_name":"abcdefg","event_id":516575, }] let uniqueArray = []; let dupeArray = []; let itemIndex = 0 totalArray.forEach(x => { if(!uniqueArray.some(y => JSON.stringify(y.fields.user_id) === JSON.stringify(x.fields.user_id))){ uniqueArray.push(x) } else(dupeArray.push(x)) }) node.warn(totalArray); node.warn(uniqueArray); node.warn(dupeArray); return msg;

I'm successfully deduping the array to produce only unique values. Problem is, I need to remove both duplicates, e.g.: if there are 2 objects with the same user_id key, I want to remove both of the objects from the array, not just one.


One option is to iterate over the array and put the current object being iterated over at a user_id property on the object. If the property already exists there, reassign it to null instead. At the end, take the values of the object and remove the null values:

<pre class="snippet-code-js lang-js prettyprint-override">const totalArray = [{ "id": "rec01dTDP9T4ZtHL4", "fields": { "user_id": 170180717, "user_name": "abcdefg", "event_id": 516575, } }, { "id": "rec01dTDP9T4ZtHL4", "fields": { "user_id": 170180717, "user_name": "abcdefg", "event_id": 516575, } }, { "id": "unique", "fields": { "user_id": 1234, "user_name": "abcdefg", "event_id": 516575, } }]; const uniques = {}; for (const item of totalArray) { const prop = item.fields.user_id; uniques[prop] = uniques.hasOwnProperty(prop) ? null : item; } const output = Object.values(uniques).filter(Boolean); console.log(output);

You could first group the objects based on user_id. Then partition them based on group size (if a group only contains a single record it is unique). Then flatten the groups so it'll end up as a normal array of records and not an array of groups of records.

<pre class="snippet-code-js lang-js prettyprint-override">function groupBy(iterable, fn) { const groups = new Map(); for (const item of iterable) { const key = fn(item); if (!groups.has(key)) groups.set(key, []); groups.get(key).push(item); } return groups; } function partition(iterable, fn) { const truthy = [], falsy = []; for (const item of iterable) { (fn(item) ? truthy : falsy).push(item); } return [truthy, falsy]; } const totalArray = [{ "id": "rec01dTDP9T4ZtHL4", "fields": { "user_id": 170180717, "user_name": "abcdefg", "event_id": 516575, } }, { "id": "rec01dTDP9T4ZtHL4", "fields": { "user_id": 170180717, "user_name": "abcdefg", "event_id": 516575, } }, { "id": "unique", "fields": { "user_id": 1234, "user_name": "abcdefg", "event_id": 516575, } }]; const [uniqueArray, dupeArray] = partition( groupBy(totalArray, record => record.fields.user_id).values(), group => group.length == 1 ) .map(groups => groups.flat(1)); console.log("uniqueArray =", uniqueArray); console.log("dupeArray =", dupeArray);

You can use Array.reduce && Array.some like this:

<pre class="snippet-code-js lang-js prettyprint-override">let totalArray = [ { id: "rec01dTDP9T4ZtHL4", fields: { "user_id":170180717, "user_name":"abcdefg", "event_id":516575, } }, { id:"rec01dTDP9T4ZtHL4", fields: { "user_id":170180717, "user_name":"abcdefg", "event_id":516575, } }, { id:"rec01dTDP9T4ZtHL4", fields: { "user_id":1470107417, "user_name":"abcdefg", "event_id":516575, } }, { id:"rec01dTDP9T4ZtHL4", fields: { "user_id":1470107417, "user_name":"abcdefg", "event_id":516575, } }, { id:"rec01dTDP9T4ZtHL4", fields: { "user_id":14701073417, "user_name":"abcdefg", "event_id":516575, } } ]; let removeDups = totalArray.reduce((result,item,index,original)=>{ if (original.some((o,i)=>o.fields.user_id===item.fields.user_id && i!==index)) return result; result.push(item); return result; },[]) console.log(removeDups);



  • how to create listview to a grid programmatically
  • Firefox : There is no such field called inputType in keydown event object
  • Apache Flink Streaming window WordCount
  • Why is the line of wss-plot (for optimizing the cluster analysis) looks so fluctuated?
  • Java Regex Finding digits in a String
  • How application get update from apple store?
  • changing src attributes
  • Iframe onLoad won't fire when dumping a file?
  • Matplotlib: subplot
  • How to remove item from Panel
  • swift UIView animateWithDuration with repeat and autoreverse
  • When scaling and drawing an image to canvas in iOS Safari, width is correct but height is squished
  • Keep node's X and Z axes parallel to ground while rotating Y axis to face camera
  • Upload synthesized speech from firebase function node.js server's tmp directory
  • getting the status code of a HTTP redirected page
  • no endpoints available for service \\“kubernetes-dashboard\\”
  • Getting CKEditor to work with Flask Admin
  • Specify the _id field using Bulk.IndexMany in ElasticSearch
  • wxPython UltimateListCtrl check (tick) listitem programatically
  • How to implement JQuery confirm dialog with JSF
  • Nested AJAX Calls using .done
  • How to run Daphne Server (Django Channels) & workers in the background?
  • How to process remote XML files with XSLT
  • Query regarding com.jcraft.jsch.JSchException: UnknownHostKey: x.y.com. DSA key fingerprint is “ac:e
  • System.Drawing.DrawString() weird wrapping of long string
  • Exclude Classes from Being Included in Shaded Jar
  • Facebook Error (#200) The user hasn't authorized the application to perform this action (PHP)
  • How to load dynamic images in custom ListView
  • Ruby on Rails: Get mediaplayer information (iTunes, TRAKTOR, Cog; current song + playlist)
  • What is the difference between dynamically creating a script tag and statically embed a script tag?
  • openpyxl - adding new rows in excel file with merged cell existing
  • r - input value by user to dataframe via shiny
  • VSTS work items list through REST API
  • Cross compile glibc for arm, got undefined reference to some unwind functions
  • multiple button click in asp.net MVC 3
  • How do I use TagLib-Sharp to write custom (PRIV) ID3 frames?
  • CAS 4 - Not able to retrieve the LDAP groups after successful authentication
  • How to mutate multiple variables without repeating codes?
  • JavaScript RegExp Replace