74451

Text classification extract tags from text

I have a lucene index with a lot of text data, each item has a description, I want to extract the more common words from the description and generate tags to classify each item based on the description, is there a lucene.net library for doing this or any other library for text classification?

Answer1:

No, lucene.net can make search, index, text normalization, "find more like this" funtionalty, but not a text classification.

What to suggest to you depends from your requirements. So, maybe more description needed. But, generally, easiest way try to use external services. All external services have REST API, and it's very easy to interact with it using C#.

From external services:

    <li>Open Calais </li> <li>uClassify </li> <li>Google Prediction API </li> <li>Text Classify </li> <li>Alchemy API </li> </ul>

    Also there good Java SDK like Mahout. As I remember interactions with Mahout could be also done like with service, so integration with it is not a problem at all.

    I had similar "auto tagging" task using c#, and I've used for that Open Calais. It's free to make 50,000 transactions per day. It was enough for me. Also uClassify has good pricing, as example "Indie" license 99$ per year.

    But maybe external services and Mahout is not your way. Than take a look at DBpedia project and RDF. And the last, you can use some implementations of Naive Bayes algorithm, at least. It's easy, and all will be under your control.

    Answer2:

    This is a very hard problem but if you don't want to spend time on it you can take all words which have between 5% and 10% frequency in the whole document. Or, you simply take the most common 5 words.

    Doing tag extraction well is very very hard. It is so hard that whole companies live from webservices exposing such an API.

    You can also do stopword removal (using a fixed stopword list obtained from the internet).

    And you can find common N-grams (for example pairs) which you can use to find multi-word tags.

Recommend

  • DevExpress: How do get an instance of a control client-side and access its client-side members?
  • jOOQ: Allowed-Character constraints?
  • How do i install the cakephp forum plugin by Miles J?
  • Convert adjacency matrix to a csv file
  • Working with big files in classic ASP
  • Calling PowerShell's where-object from C#
  • Sql indexes vs full table scan
  • Class implementation in a header file == bad style? [duplicate]
  • Netezza Incremental load from Sql server using SSIS
  • Get all existing pointers to an object
  • composer dependency stating in doesn't have php-xsl
  • LNK1104: cannot open file 'kernel32.lib'
  • Extending the Django 1.11 User Model
  • Request response issues in biztalk
  • Access user's phone number on iOS 7
  • cordova is not defined - cordova.js has already been loaded :: Ionic
  • Clear fused location provider's location for testing
  • Android changing fragment order inside FragmentPagerAdapter
  • Jquery popup on mouse over of calendar control
  • Reading a file into a multidimensional array
  • Authentication in Play! and RestEasy
  • Object and struct member access and address offset calculation
  • jQuery ready not fired after rails link_to is clicked
  • How do I get HTML corresponding to current DOM tree?
  • azure media services - The request body is too large and exceeds the maximum permissible limit
  • Hardware Accelerated Image Scaling in windows using C++
  • Magento Fatal error: Maximum execution error solution, on WAMP
  • How to attach a node.js readable stream to a Sendgrid email?
  • Bad request using file_get_contents for PUT request in PHP
  • How to create a file in java without a extension
  • How to delay loading a property with linq to sql external mapping?
  • How to redirect a user to a different server and include HTTP basic authentication credentials?
  • Can I make an Android app that runs a web view in Chrome 39?
  • Hazelcast - OperationTimeoutException
  • Timeout for blocking function call, i.e., how to stop waiting for user input after X seconds?
  • RestKit - RKRequestDelegate does not exist
  • Matrix multiplication with MKL
  • Revoking OAuth Access Token Results in 404 Not Found
  • LevelDB C iterator
  • Linking SubReports Without LinkChild/LinkMaster