54011

How to connect Spark Streaming to standalone Solr on windows?

Question:

I want to integrate Spark Streaming with Standalone Solr. I am using Spark 1.6.1 and Solr 5.2 standalone on windows with no Zookeeper configuration. I am able to find some solution where they are connecting to Solr from spark by passing the Zookeeper config. How can I connect my spark program to standalone Solr?

Answer1:

Please see if this example is helpful <a href="http://spark.apache.org/docs/latest/streaming-programming-guide.html#design-patterns-for-using-foreachrdd" rel="nofollow">http://spark.apache.org/docs/latest/streaming-programming-guide.html#design-patterns-for-using-foreachrdd</a>

From example, you will need to write your own Connection class which wraps object of <a href="https://lucene.apache.org/solr/5_3_1/solr-solrj/org/apache/solr/client/solrj/impl/HttpSolrClient.html" rel="nofollow">HttpSolrClient</a> or <a href="https://lucene.apache.org/solr/5_3_1/solr-solrj/org/apache/solr/client/solrj/impl/ConcurrentUpdateSolrClient.html" rel="nofollow">ConcurrentUpdateSolrClient</a>. You need to also write your own ConnectionPool class which will implement pool of your own Connection objects (or if its thread safe, just return same singleton object).

Recommend

  • new spark.sql.shuffle.partitions value not used after checkpointing
  • SQL Server Integrated Security from an Azure Web Site
  • detecting connection lost in spark streaming
  • New-PSSession in an Azure-runbook (ARM)
  • qt how to know that a pushbutton is clicked?
  • How do I check if System::Collections:ArrayList is empty / nullptr / null?
  • Creating a C++ function that calls other Lua function
  • File extension of zlib zipped html page?
  • iOS Cordova first plugin - plugin.xml to inject a feature
  • How to use function wrapper in mustache.php?
  • Dynamically switching connect in Modelica
  • Memory error in python- how to use more memory
  • Differences in dis-assembled C code of GCC and Borland?
  • nonblocking BIO_do_connect blocked when there is no internet connected
  • Redux Form - Not able to type anything in input
  • Does Mobilefirst provide a provision to access web services directly?
  • Get history of file changes from TFS to implement custom “blame”-behaviour of exceptions
  • Parsing a CSV string while ignoring commas inside the individual columns
  • Problem deserializing objects from cache on MyBatis 3/Java
  • Spark fat jar to run multiple versions on YARN
  • Read a local file using javascript
  • ImageMagick, replace semi-transparent white with opaque white
  • Algorithm for a smudge tool?
  • Cannot connect to cassandra from Spark
  • Excel - Autoshape get it's name from cell (value)
  • Cross-Platform Protobuf Serialization
  • Check if a string to interpolate provides expected placeholders
  • Statically linking a C++ library to a C# process using CLI or any other way
  • Alternatives to the OPTIONAL fallback SPARQL pattern?
  • RestKit - RKRequestDelegate does not exist
  • Invalid access key error using credentials redeemed from an amazon open id token
  • Circular dependency while pushing http interceptor
  • Traverse Array and Display in markup
  • bootstrap to use multiple ng-app
  • How to get icons for entities from eclipse?
  • InvalidAuthenticityToken between subdomains when logging in with Rails app
  • Turn off referential integrity in Derby? is it possible?
  • JaxB to read class hierarchy
  • Checking variable from a different class in C#
  • UserPrincipal.Current returns apppool on IIS