View Source

{toc}

h2. Overview

We have already provided means to annotate documents through SPARQL and it makes sense to continue the trend and expose more concept extraction oriented functionality in the same fashion. The idea is to expose control mechanisms for administration and configuration of the embedded CES service. This page provides a comprehensive definition of this API.


h2. Semantic annotation

The document annotation is executed through a specially crafted SELECT SPARQL query. It takes a single triple pattern, which consists of a binding variable, a special predicate and an [RDF Collection|http://www.w3.org/TR/rdf-sparql-query/#collections] holding the parameters.

We will jump ahead with an example to get started:
{code:language=html/xml}SELECT * WHERE {
?s <http://www.ontotext.com/owlim/ces#annotate> (
"""content=<?xml version="1.0" encoding="UTF-8"?>
<tns:document id="http://ontotext.com/publishing/document/215351"
xmlns:tns="http://www.ontotext.com/DocumentSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">

<tns:feature-set>
<tns:feature>
<tns:name type="xs:string">sourceUrl</tns:name>
<tns:value type="xs:string"></tns:value>
</tns:feature>
</tns:feature-set>

<tns:document-parts>
<tns:document-part id="1" part="TITLE">
<tns:content>
China economy on the rise
</tns:content>
</tns:document-part>
</tns:document-parts>
</tns:document>"""
"domain=urn:my.domain:namespace"
"content-type=application/vnd.ontotext.ces.document+xml"
"accept-type=application/vnd.ontotext.ces.document+xml") .
}
{code}
Here's the intuition behind the query:
* <[http://www.ontotext.com/owlim/ces#annotate]> is a special predicate, which means that OWLIM listens for it and knows how to interpret it.
* "content=China economy on the rise" is the text of the document. However, human readable text form is accepted only if it is a single line (useful to show the idea).
* "domain-name=http://www.ontotext.com/owlim/ces#default" is a domain identifier - it explicitly denotes which extraction algorithm should be used. Different domains usually require different extraction techniques.
* "content type=text/plain" is the MIME type of the document, which is just plain text in this example.

h3. Parameter reference

|| Parameter || Required || Supported values || Default value || Comment ||
| content | true | XML/JSON \\ {warning}We expect already validated input, so no validation is performed at query parsing and processing level.{warning} | none | {note}We are (de)serialising with the URL-safe flag turned on \\
See [http://en.wikipedia.org/wiki/Base64#URL_applications] \\
and [http://commons.apache.org/codec/apidocs/org/apache/commons/codec/binary/Base64.html]\\
{note} |
| domain | true | URN | none | eg: http://www.ontotext.com/owlim/ces#default \\ |
| content-type | false | "application/vnd.ontotext.ces.document+xml", "application/vnd.ontotext.ces.document+json" | "application/vnd.ontotext.ces.document+xml" | This is the type of the encoded document. We create a markup aware GATE document from it. |
| accept-type | false \\ | "application/vnd.ontotext.ces.document+xml", "application/vnd.ontotext.ces.document+json" | "application/vnd.ontotext.ces.document+xml" | This parameter serves to indicate the preferred result type - XML or JSON. \\ |
| annotation-sets-to-preserve | false \\ | Comma separated list of the internal annotation set names (strings). I.e. to preserve the annotation set {code}<tns:annotation-set name="brendan" ref="2">{code} you need to add the following parameter {code}"annotation-sets-to-preserve=brendan|2"{code} An internal name is in the format <annotation set name>\|<ref id> \\ | none \\ | This parameter allows you to specify which annotation sets should be preserved. \\
Trimming is performed, so no whitespaces are allowed at the start and end of a name. \\ |

h3. Results

Return a JSON/XML document with annotations. The document is a valid instance of Ontotext's generic schema definition.


h2. Re-training

Not defined yet. Depends on the pipeline as well, whether or not it contains re-trainable machine learning components.


h2. Administration and configuration

Concept extraction is disabled by default. Start/stop is achieved through SPARQL Update queries.

h3. Start

* First time initialization *DOES NOT*&nbsp;include Gazetteer cache loading - see [Reload dictionary|#Reloaddictionary]
* On subsequent starts will load the cache from the file system

To start&nbsp;*ALL*&nbsp;registered concept extraction pipelines use the following query:


{code:language=html/xml}
INSERT DATA {
[] <http://www.ontotext.com/owlim/ces#start> [].
}
{code}

To start a specific pipeline, include it's specific name graph in the query. See an example with the default pipeline:

{code:lang=xml}
INSERT DATA {
GRAPH <http://www.ontotext.com/owlim/ces#default> {
[] <http://www.ontotext.com/owlim/ces#start> [].
}
}
{code}

h3. Stop

Stops *ALL* concept extraction service, nothing special here.

{code:language=html/xml}
INSERT DATA {
[] <http://www.ontotext.com/owlim/ces#stop> [].
}
{code}

Again, to stop a specific pipeline use the named graph of the pipeline:
{code:lang=xml}
INSERT DATA {
GRAPH <http://www.ontotext.com/owlim/ces#default> {
[] <http://www.ontotext.com/owlim/ces#stop> [].
}
}
{code}

h3. Reload dictionary

Cleans the Gazetteer cache from the file system and loads it again from the repository.

{note}In case the concept extraction service is not started, this SPARQL update operation will *NOT* schedule a dictionary reload (unlike before).{note}
The following query will initiate a dictionary reload on all running pipelines. To specify a particular pipeline use a named graph like shown in the Start/Stop sections of this page.

{code:language=html/xml}
INSERT DATA {
[] <http://www.ontotext.com/owlim/ces#reloadDictionary> [].
}
{code}

{code:language=html/xml}
INSERT DATA {
GRAPH <http://www.ontotext.com/owlim/ces#default> {
[] <http://www.ontotext.com/owlim/ces#reloadDictionary> [].
}
}
{code}

h3. Add/remove Gazetteer configuration

Registers template queries for different entity types via INSERT/DELETE DATA.
* <[http://www.ontotext.com/owlim/ces#gazetteerConfig]> is a special (interpretable) predicate, which denotes a Gazetteer template query entry.
* Each Gazetteer configuration should be added in a separate named graph (per domain), i.e. the default pipeline uses <[http://www.ontotext.com/owlim/ces#default]>
* The template queries are also executed for all sub-classes of the defined class
* The configuration is stored as regular triples in the repository and is loaded on concept extraction initialization

Example configuration which indicates to load all rdfs:labels of all Agents, Locations and EconomicConcepts into the Gazetteer dictionary.
{code:language=html/xml}
INSERT DATA { GRAPH <http://www.ontotext.com/owlim/ces#default> {
<http://ontotext.com/ontologies/core/Agent> <http://www.ontotext.com/owlim/ces#gazetteerConfig> "select ?label ?inst where {<%s> a <http://ontotext.com/ontologies/core/Agent> . <%s> rdfs:label ?label.}" .

<http://ontotext.com/ontologies/location/Location> <http://www.ontotext.com/owlim/ces#gazetteerConfig> "select ?label ?inst where {<%s> a <http://ontotext.com/ontologies/location/Location> . <%s> rdfs:label ?label.}" .

<http://ontotext.com/ontologies/economy/EconomicConcept> <http://ontotext.com/owlim/ces#gazetteerConfig> "select ?label ?inst where {<%s> a <http://ontotext.com/ontologies/economy/EconomicConcept> . <%s> rdfs:label ?label.}" . }
}
{code}

{info}Adding/removing Gazetteer configuration doesn't take full effect immediately. For example, adding a new template query results in the CES plugin starting to listen for entities of its corresponding type. However, it does not load already existing entities of the same type. In order to achieve that you should trigger a dictionary reload.{info}

h2. FAQ

h3. How to deploy a pipeline?

Just unpack your pipeline package into _$\{info.aduna.platform.appdata.basedir\}/repositories/$\{repository.name\}/storage/ces/pipelines/_ and the it will be discovered automatically. You can confirm it is discovered by finding an MBean called _PipelineManager_ and checking its _AvailablePipelines_ property, which lists the URIs of all the deployed pipelines. Note that in order to start using the pipeline you need to start it.

h3. How to preserve annotation sets?

Check out the _annotation-sets-to-preserve_ parameter of the annotation query above. No annotation sets are preserved by default.