Difference between revisions of "Annotator Dataset Workflow Howto"

From NCBO Wiki
Jump to navigation Jump to search
(Replaced content with "As of Virtual Appliance v2.2, populating the Annotator Dataset is done automatically when an ontology is processed.")
 
Line 1: Line 1:
−
== Introduction ==
+
As of Virtual Appliance v2.2, populating the Annotator Dataset is done automatically when an ontology is processed.
−
 
 
−
Once you have submitted some ontologies into the BioPortal Ontology Services (BioPortal Core), you can use these to populate the backend data sets required by the Annotator Web Service. These datasets are collectively referred to as the Annotator datasets and comprise: The hierarchy database (formerly known as "OBS"), a dictionary file, and (optionally) a mapping database. The population process uses classes, terms, relations, and semantic types from the ontologies. The population is done in two major steps: 1) Synchronize the hierarchy database with BioPortal Core and 2) Create the dictionary file for use with MGREP. A third step, populating mapping information, should be done if there are mappings available.
 
−
 
 
−
== Synchronize Hierarchy Database with Ontology Services ==
 
−
 
−
<p><font color="red"><b>Note:</b></font> All NCBO REST Web services will be required to contain the parameter "apikey=YourApiKey" starting June 2011. As a result, the "ObsApiKey" parameter should now be appended to all the OBS worklflow execution urls.</p>
 
−
The ontologies and related data that will be used by the Annotator are gathered from the Ontology Services (part of BioPortal Core). This process should be run any time a new ontology (or a new version of an existing ontology) is added to the Ontology Services, though it could theoretically be run from a cron script or scheduled job.
 
−
 
 
−
<ol>
 
−
<li>Remove out-dated ontologies from hierarchy database (e.g. older version of ontologies that does not in BioPortal anymore). By invoking this restlet, it will remove all the outdated ontology data and the associated entities such as concepts, terms, relations, semantic types and hierarchy information.
 
−
<p><code>
 
−
See the list of ontologies/versions to be removed:
 
−
http://example/obs/admin/ontologies/list/old?apikey=ObsApiKey<br/>
 
−
Remove old ontologies:
 
−
http://example/obs/admin/ontologies/remove?apikey=ObsApiKey
 
−
</code></p></li>
 
−
 
 
−
<li>Add new ontologies from BioPortal to the hierarchy database. By invoking this restlet, it will add all the new ontology data and the associated entities such as concepts, terms, relations, semantic types and hierarchy information.
 
−
<p><code>
 
−
See the list of ontologies/versions to be added:
 
−
http://example/obs/admin/ontologies/list/new?apikey=ObsApiKey<br/>
 
−
Add new ontologies:
 
−
http://example/obs/admin/ontologies/add?apikey=ObsApiKey
 
−
</code></p></li>
 
−
 
 
−
<li>Populate Concepts (For details, please refer to Chapter 2.1)
 
−
<p><code>http://example/obs/loaderBigConcepts/all?apikey=ObsApiKey</code></p></li>
 
−
<li>Populate Hierarchy (For details, please refer to Chapter 2.2)
 
−
<p><code>http://example/obs/loaderBigPaths/all?apikey=ObsApiKey</code></p>
 
−
 
 
−
<p>To monitor the progress and view any errors, refer to:</p>
 
−
<ol>
 
−
<li>The "status" field in the table obs_ontology in Annotator DB (obs_hibernate database).
 
−
<li>Check the Tomcat log (/var/logs/tomcat6/catalina.out)</li>
 
−
</ol></li>
 
−
</ol>
 
−
 
 
−
== Create Dictionary File ==
 
−
 
 
−
<p>All the terms will be created as dictionary file. (Data is coming from the obs_term table)</p>
 
−
<ul>
 
−
<li>Location : The directory location is specified in build.properties</li>
 
−
<pre>
 
−
# Dictionary File path
 
−
obs.dictionary.path=/srv/ncbo/bioportal_resources/annotator/dictionary/dict
 
−
</pre>
 
−
<li>The dictionary file is generated in several chunks to avoid memory problems. There is a 1,000,000 term limit per file.</li>
 
−
<li>Generate the dictionary file:</li>
 
−
<pre>
 
−
http://example/obs/createDictionary/0?apikey=ObsApiKey
 
−
</pre>
 
−
<li>After the files are generated, you can run the command <code>ncborestart</code> to concatenate the resulting dictionary files and to restart the MGREP server.</li>
 
−
</ul>
 
−
 
 
−
== Mapping Data Population ==
 
−
<p>Mappings between ontologies can be used in the Annotator Web Service to find related terms. Loading the mapping information is currently a manual process, though this will be automated in the future. If you have mapping data you would like to include in annotator, please [mailto:support@bioontology.org contact NCBO].</p>
 
−
 
 
−
== Appendix ==
 
−
 
 
−
=== Data Population - Concepts and Hierarchy ===
 
−
<p><b>Introduction:</b> the Annotator hierarchy database population application pulls the ontology and the concept data from BioPortal Core via the REST services, then extracts and computes the hierarchy information, and finally stores the information in the hierarchy database. Then this computed data is accessible via the Annotator REST services.</p>
 
−
 
 
−
<p>Data population is divided into two parts: 1) Concepts and 2) Hierarchy. Please see below for details on these two population procedures.</p>
 
−
 
 
−
==== Concepts and Direct Relation (level == 1) ====
 
−
===== Pre-requisite in "status": =====
 
−
<p>The ontology should be in valid status (<font color="red"><b>"status = 3"</b></font>) in the obs_ontology table in Hierarchy Database to start this process (i.e. This is the initial status from BioPortal Core when ontology is successfully parsed). This is also serves as a safety lock to avoid launching population again for ontologies that are already in the process of population or already populated.</p>
 
−
 
 
−
===== REST calls available =====
 
−
<ol>
 
−
<li>For all ontologies: populate all ontologies with valid status ("status = 3") currently stored in Hierarchy DB (ontologies are added during the synchronization process).<br/>
 
−
<code>http://example/obs/loaderBigConcepts/all?apikey=ObsApiKey
 
−
</code></li>
 
−
<li>For a specific ontology: populate the given ontology if "status = 3". Otherwise the data population process will complain that the status is invalid in the Tomcat log and exit.<br/>
 
−
<code>http://example/obs/loaderBigConcepts/{ontology_versoin_id}?apikey=ObsApiKey
 
−
</code></li>
 
−
</ol>
 
−
 
 
−
===== Troubleshooting =====
 
−
<p>Errors are logged both in Tomcat catalina.out and DB (obs_error_queue table).</p>
 
−
<ol>
 
−
<li><b>Case 1: BioPortal REST Service is Down</b>
 
−
<p>When you kick off the process using the Annotator REST call - either via web browser or shell script - it checks first if BioPortal REST service is alive. If the BioPortal REST service is down – the tomcat log will generate an error message about BioPortal REST service being down (But it does not change the ontology "status" field. Just simply kick off the process again when BioPortal REST service is back up. If BioPortal REST service is down, no change in Annotator database, therefore no need to clean up. See "Error Handling" for the "status" change scenario).</p></li>
 
−
 
 
−
<li><b>Case 2: Critical Error</b>
 
−
<p>If the error is critical – RunTimeException and etc – the "status" the ontology in obs_ontology table becomes "99" and the data population process halts.</p></li>
 
−
 
 
−
<li><b>Case 3: Non-critical Error</b>
 
−
<p>If the error is not critical, the "status" the ontology in obs_ontology table becomes "99" but the data population process still continues to populate the rest of the data.  An example for a non-critical error is a discrepancy between the data from two different BioPortal REST calls – i.e. Some of the root concepts from getRootConcepts call are missing in BioPortal (The list of concepts from getAllConcepts does not have some of the root concepts).</p></li>
 
−
</ol>
 
−
<p>In the case of case 2 & 3, data clean up may be necessary. Please see "Data Clean-Up".</p>
 
−
 
 
−
==== Indirect Relation Hierarchy (level > 1) ====
 
−
===== Pre-requisite in "status": =====
 
−
<p>The ontology should be in valid status (<font color="red"><b>"status = 14"</b></font>) in obs_ontology table in Hierarchy Database to start this process (i.e. "loaderBigConcepts" should have been completed for the given ontology). This is a safety lock to avoid launching population again for the ontologies already in the process of population or already populated.</p>
 
−
 
 
−
===== REST calls available: =====
 
−
<ol>
 
−
<li>For all ontologies: populate all ontologies with "status = 14"<br/>
 
−
<code>http://example/obs/loaderBigPaths/all?apikey=ObsApiKey</code></li>
 
−
<li>For a specific ontology: populate the given ontology if "status = 14". Otherwise the data population process will complain that the status is invalid in the Tomcat log and exit.<br/>
 
−
<code>http://example/obs/loaderBigPaths/{ontology_version_id}?apikey=ObsApiKey<br/>
 
−
e.g. http://example/obs/loaderBigPaths/40671?apikey=ObsApiKey</code></li>
 
−
</ol>
 
−
 
 
−
==== Monitoring the Progress ====
 
−
<p>To monitor the progress, please see "status" field in obs_ontology table.</p>
 
−
<table cellpadding="5" cellspacing="1">
 
−
<tr bgcolor="#aaaaaa">
 
−
<th>Restlets</th>
 
−
<th>Status Required (valid status required to begin the process)</th>
 
−
<th>Status Start -> Finish</th>
 
−
</tr>
 
−
<tr bgcolor="#eeeeee">
 
−
<td>loaderBigConcepts</td>
 
−
<td>3</td>
 
−
<td>11 -> 14</td>
 
−
</tr>
 
−
<tr bgcolor="#eeeeee">
 
−
<td>loaderBigPaths</td>
 
−
<td>14</td>
 
−
<td>21 -> 28</td>
 
−
</tr>
 
−
<tr bgcolor="#eeeeee">
 
−
<td> </td>
 
−
<td> </td>
 
−
<td>(status value in obs_ontology table changes as progress continues)</td>
 
−
</tr>
 
−
</table>
 
−
 
 
−
===== Error Handling: =====
 
−
<ul>
 
−
<li>If there is any <font color="red"><b>error</b></font>, the status will be set to <font color="red"><b>99</b></font></li>
 
−
<li>If critical error (e.g. BioPortal REST services down) occurs, the population process will stop and exit. But if it is NOT a critical error (e.g. if the semantic type description is not found from semantic look up table etc), it will mark the status to 99, log the error (both Tomcat log and DB obs_error_queue), but continue with the population.</li>
 
−
<li>To restart the process, just run the script to clean up. The script will reset the status either to "3" or "14" depending on Concept clean up or Hierarchy clean up. (See Section II. Data Clean up)</li>
 
−
</ul>
 
−
 
 
−
=== Data Clean-Up ===
 
−
<p>The source for several SQL commands that can be used to clean up data after an aborted population attempt is located in <b>/ncbo/sources/annotator/2004/db/sql/obs_db_cleanup.sql</b> – <b><i>DO NOT RUN</i></b> the entire script since this is just compiled list of commands.</p>
 
−
 
 
−
<pre>
 
−
/*
 
−
-----------------------------------------------------------------------
 
−
to clean up or undo everything on one ontology, just run #1 and #2
 
−
-----------------------------------------------------------------------
 
−
1. Rollback BPConceptManager.
 
−
Run this to rollback to initial state - to undo 'loaderConcepts' restlet
 
−
*/
 
−
 
 
−
set @var_ontology_version_id := '39545';
 
−
 
 
−
delete a.* from obs_relation a, obs_concept b, obs_ontology c
 
−
where a.level = 1 and a.concept_id = b.id and b.ontology_id = c.id
 
−
and c.local_ontology_id = @var_ontology_version_id;
 
−
 
 
−
delete a.* from obs_term a, obs_concept b, obs_ontology c
 
−
where a.concept_id = b.id and b.ontology_id = c.id
 
−
and c.local_ontology_id = @var_ontology_version_id;
 
−
 
−
delete a.* from obs_semantic_type a, obs_concept b, obs_ontology c 
 
−
where a.concept_id = b.id and b.ontology_id = c.id
 
−
and c.local_ontology_id = @var_ontology_version_id;
 
−
 
 
−
delete a.* from obs_concept a, obs_ontology b
 
−
where a.ontology_id = b.id
 
−
and b.local_ontology_id = @var_ontology_version_id;
 
−
 
−
UPDATE obs_ontology set status = 3
 
−
where local_ontology_id = @var_ontology_version_id;
 
−
 
 
−
/*
 
−
-----------------------------------------------------------------------
 
−
2. Rollback BPPathManager.
 
−
Run this to rollback - to undo 'loaderPaths' restlet 
 
−
*/
 
−
 
 
−
set @var_ontology_version_id := '39545';
 
−
 
 
−
delete a.* from obs_relation a, obs_concept b, obs_ontology c
 
−
where a.concept_id = b.id and b.ontology_id = c.id
 
−
and c.local_ontology_id = @var_ontology_version_id and a.level > 1;
 
−
 
−
delete a.* from obs_path_to_root a, obs_concept b, obs_ontology c
 
−
where a.concept_id = b.id and b.ontology_id = c.id
 
−
and c.local_ontology_id = @var_ontology_version_id;
 
−
 
−
delete a.* from obs_path_to_leaf a, obs_concept b, obs_ontology c
 
−
where a.concept_id = b.id and b.ontology_id = c.id
 
−
and c.local_ontology_id = @var_ontology_version_id;
 
−
 
−
UPDATE obs_ontology set status = 14
 
−
where local_ontology_id = @var_ontology_version_id;
 
−
</pre>
 
−
 
 
−
=== Useful SQL Queries for Monitoring and Validation ===
 
−
The source for several queries that can be used for monitoring and data validation is located in <b>/ncbo/sources/annotator/2004/db/sql/obs_db_cleanup.sql</b> (In the same file as the Data Clean-Up)
 
−
 
 
−
== To check if there are multiple entries for an Ontology ==
 
−
<pre>
 
−
select name, virtual_ontology_id, local_ontology_id
 
−
from obs_ontology
 
−
group by virtual_ontology_id
 
−
having count(virtual_ontology_id)>1;
 
−
</pre>
 
−
 
 
−
== Number of concepts for a specific Ontology ==
 
−
<pre>
 
−
select OT.name, OT.local_ontology_id AS local_ontology_id,
 
−
count(CT.id) AS concepts, OT.status  from obs_ontology OT
 
−
left join obs_concept CT ON CT.ontology_id=OT.id
 
−
WHERE OT.local_ontology_id in ('40133')
 
−
GROUP BY OT.local_ontology_id;
 
−
</pre>
 
−
 
 
−
== Number of paths to roots for a specific Ontology ==
 
−
<pre>
 
−
select OO.local_ontology_id, OO.name,
 
−
count(OPR.id) as path_to_roots, OO.status
 
−
from obs_ontology OO, obs_concept OC, obs_path_to_root OPR
 
−
where OO.id=OC.ontology_id
 
−
and OC.id = OPR.concept_id
 
−
and OO.local_ontology_id ="40133";
 
−
</pre>
 
−
 
 
−
== Number of total relations or path_to_root/leaf for a specific ontology (to see the progress) – by looking at how fast the number is growing ==
 
−
<pre>
 
−
select count(*) from obs_relation a, obs_concept b, obs_ontology c
 
−
where a.concept_id = b.id and b.ontology_id = c.id
 
−
and c.local_ontology_id = '40133' and a.level > 1;
 
−
 
 
−
select a.*, b.local_concept_id from obs_path_to_root a,
 
−
obs_concept b, obs_ontology c
 
−
where a.concept_id = b.id and b.ontology_id = c.id
 
−
and c.local_ontology_id = '40483';
 
−
</pre>
 
−
 
 
−
== Ontologies that have the status "Ready" (28) but have no concepts in obs_concept table ==
 
−
<pre>
 
−
SELECT o.*, c.ontology_id
 
−
FROM obs_ontology o
 
−
LEFT OUTER JOIN (
 
−
    SELECT DISTINCT ontology_id FROM obs_concept
 
−
) c ON o.id = c.ontology_id
 
−
WHERE o.status = 28 AND c.ontology_id IS NULL;
 
−
</pre>
 
−
 
 
−
[[Category:NCBO Virtual Appliance]]
 

Latest revision as of 16:06, 14 January 2015

As of Virtual Appliance v2.2, populating the Annotator Dataset is done automatically when an ontology is processed.

Retrieved from "https://www.bioontology.org//mediawiki/index.php?title=Annotator_Dataset_Workflow_Howto&oldid=12711"