65851

Parallel Producing and Consuming in Kafka

Question:

<strong>1. Consuming concurrently on the same topic and same partition</strong>

Suppose I have 100 partitions for a given topic (e.g. Purchases), I can easily consume these 100 partitions (e.g. Electronics, Clothing, and etc...) in parallel using a consumer group with 100 consumers in it.

However, that is assigning one consumer to each subset of the total data on Purchases. What if I want just want to consume one subset of data with 100 consumers concurrently? For example, for all of my consumers, they just want to know Electronics partition of the Purchases topic.

<em>Is there way they can consume this partition concurrently?</em>

In general I just want all my consumers to receive <strong>the same data set concurrently</strong>.

From the information I've gathered, it seems to me that consumers CANNOT consume from replicas: <a href="https://stackoverflow.com/questions/37803376/consuming-from-a-replica" rel="nofollow">Consuming from a replica</a>

<em>Can I produce the same data to multiple topics, like Purchase-1[Electronics] and Purchase-2[Electronics] so then I can consume them concurrently? Is this a recommended approach?</em>

<strong>2. Producing concurrently on the same topic and same partition</strong>

When multiple producers are producing to the same topic and same partition, since we can only write to the partition leader and replicas are only there for fault-tolerance, <em>does this mean there isn't any concurrency?</em> (i.e. each commit must wait in line.)

Answer1:

<ol><li>If those 100 consumers belong to different consumer groups, they can consume from the same topic and partition simultaneously. In that case, you need to make sure each consumer is able to handle the load from the 100 partitions.</li> <li>Producers can produce to the same topic partition at the same time, but the actual order of messages written to the partition is determined by the partition leader.</li> </ol>

Recommend

  • Resizing ToolStripButtons to fit complete BackGround image
  • Update two sets of radiobuttons - shiny
  • pandas parse csv with left and right quote chars
  • Delphi: Form becomes Frozen while assigning strings in thread
  • setTimeOut and local function
  • What are zone turns?
  • Edit assembly language code in Visual Studio while stepping through each statement
  • Find unique tuples in a relation represented by a BDD
  • Jhipster: How to create relationships with User entity using supplied tools?
  • How do I compile a C/C++ program through windows command prompt?
  • javascript variables, What does var x = a = {} do?
  • python - calculate orthographic similarity between words of a list
  • Azure table query partial partitionkey guid match
  • FTS3 searches in ORMLite?
  • read values from form post in jquery or javascript
  • SQL: Getting the physical size of a subset of a table
  • apply a javascript function to draggable copy
  • how to solve invalid conversion specifier warning in iphone app
  • How can I get the full list of running processes on a Mac from a python app
  • Consuming a WCF service in a Java Client using wsHttpBinding
  • Exception gevent.hub.LoopExit: LoopExit('This operation would block forever',)
  • there is no graph with tensorboard
  • pip in virtualenv gets ConnectTimeoutError
  • Multiple producers single consumer locking schema
  • Merging rows to columns
  • Access Android Market through SSH tunnel
  • Marklogic : Query response time is very high
  • Can you perform a UNION without a subquery in SQLAlchemy?
  • ImageMagick, replace semi-transparent white with opaque white
  • Asynchronous UI Testing in Xcode With Swift
  • How can I use Kendo UI with Razor?
  • ActionScript 2 vs ActionScript 3 performance
  • Hazelcast - OperationTimeoutException
  • File upload with ng-file-upload throwing error
  • What are the advantages and disadvantages of reading an entire file into a single String as opposed
  • embed rChart in Markdown
  • apache spark aggregate function using min value
  • How to get NHibernate ISession to cache entity not retrieved by primary key
  • How can I use `wmic` in a Windows PE script?
  • Unable to use reactive element in my shiny app