Concept Extraction Plug-in (CES)

Overview

As we have already provided means to annotate documents through SPARQL, now it makes sense to continue the trend and expose more concept extraction oriented functionalities in the same fashion. This page provides a comprehensive definition of the embedded CES API service including control mechanisms for its administration and configuration.

Semantic annotation

The document annotation is executed through a specially crafted SELECT SPARQL query. It takes a single triple pattern, which consists of a binding variable, a special predicate, and an RDF Collection holding the parameters.

The following example will get you started:

Where:

• <http://www.ontotext.com/owlim/ces#annotate> is a special predicate, which means that GraphDB listens for it and knows how to interpret it;
• "content=China economy on the rise" is the text of the document. However, human readable text form is accepted only if it is a single line (useful to show the idea);
• "domain-name=http://www.ontotext.com/owlim/ces#default" is a domain identifier - it explicitly denotes which extraction algorithm should be used. Different domains usually require different extraction techniques;
• "content type=text/plain" is the MIME type of the document, which is just a plain text in this example.

Parameter reference

Parameter	Required	Supported values	Default value	Comment
content	true	XML/JSON The input should already be validated, no validation is performed at query parsing and processing level.	none	The (de)serialising is done with the URL-safe flag turned on. See http://en.wikipedia.org/wiki/Base64#URL_applications and http://commons.apache.org/codec/apidocs/org/apache/commons/codec/binary/Base64.html
domain	true	URN	none	eg: http://www.ontotext.com/owlim/ces#default
content-type	false	"application/vnd.ontotext.ces.document+xml", "application/vnd.ontotext.ces.document+json"	"application/vnd.ontotext.ces.document+xml"	This is the type of the encoded document. A markup aware GATE document is created from it.
accept-type	false	"application/vnd.ontotext.ces.document+xml", "application/vnd.ontotext.ces.document+json"	"application/vnd.ontotext.ces.document+xml"	This parameter serves to indicate the preferred result type - XML or JSON.
annotation-sets-to-preserve	false	Comma separated list of the internal annotation set names (strings). I.e. to preserve the annotation set you need to add the following parameter An internal name is in the format <annotation set name>|<ref id>	none	This parameter allows you to specify which annotation sets should be preserved. Trimming is performed, so no white spaces are allowed at the start and end of a name.

Results

The result of the semantic annotation is a JSON/XML document with annotations. The document is a valid instance of Ontotext's generic schema definition.

Re-training

Not defined yet. It depends on the pipeline whether or not it contains re-trainable machine learning components.

Administration and configuration

Concept extraction is disabled by default. Start/stop is achieved through SPARQL Update queries.

Start

• The first time initialisation DOES NOT include gazetteer cache loading - see Reload dictionary.
• On subsequent starts it loads the cache from the file system.

To start ALL registered concept extraction pipelines, use the following query:

To start a specific pipeline, include its specific name graph in the query. See an example with the default pipeline:

Stop

To stop ALL concept extraction pipelines, use:

To stop a specific pipeline, use the named graph of the pipeline:

Reload dictionary

Reload dictionary cleans the gazetteer cache from the file system and loads it again from the repository.

In case the concept extraction service is not started, this SPARQL update operation will NOT schedule a dictionary reload (unlike before).

The following query initiates a dictionary reload on all running pipelines. To specify a particular pipeline, use a named graph as shown in the Start/Stop sections of this page.

Add/remove gazetteer configuration

Add/remove gazetteer configuration registers template queries for the different entity types via INSERT/DELETE DATA.

• <http://www.ontotext.com/owlim/ces#gazetteerConfig> is a special (interpretable) predicate that denotes a gazetteer template query entry;
• Each gazetteer configuration should be added in a separate named graph (per domain), i.e. the default pipeline uses <http://www.ontotext.com/owlim/ces#default>;
• The template queries are also executed for all sub-classes of the defined class;
• The configuration is stored as regular triples in the repository and is loaded on the concept extraction initialisation.

Example configuration that indicates how to load all rdfs:labels of all Agents, Locations and EconomicConcepts into the gazetteer dictionary.

Adding/removing gazetteer configuration does not take full effect immediately. For example, the result of adding a new template query, is that the CES plugin starts to listen for entities of its corresponding type. However, it does not load already existing entities of the same type. In order to do this, you should trigger a dictionary reload.

FAQ

How to deploy a pipeline?

Just unpack your pipeline into ${info.aduna.platform.appdata.basedir}/repositories/${repository.name}/storage/ces/pipelines/ and it will be discovered automatically. You can confirm it is discovered by finding an MBean, called PipelineManager, and checking its AvailablePipelines property, which lists the URIs of all the deployed pipelines. Note that in order to usе the pipeline you need to start it.

How to preserve annotation sets?

Check out the annotation-sets-to-preserve parameter of the annotation query above. No annotation sets are preserved by default.