Extract and Link Information

Analytics Server's memory consumption includes the JVM heap and memory mapping of on-disk files. The size of these vary depending on the endpoint(s) enabled in the instance.

You should have sufficient amount of free space to unpack the application and data from your shipment. This could range from 1GB to 90GB. The package size will grow as product updates are released. The amount required is the unpacked directories and files from all *.tar.gz in the shipment package.

Analytics Server also requires space to hold logs and other temporary files. Logs can grow depending on the log level and the number of calls. If you have a log rotation mechanism in place, a couple of GBs should be sufficient. Otherwise, experiment with your call patterns and plan for growth accordingly.

You will receive an email containing all the files needed to install Analytics Server, for multiple operating systems and also for using Docker. The files you download depends on your operating system and type of install.

The component files (roots) contain the endpoint specific models and data. Each shipment contains one or more root files, based on your license. You may receive roots that you did not order because of Root dependencies.

Once the files are downloaded and installed, Docker can be run without being connected to the internet. To install offline:

To download the volumes directly, you must have an internet connection.

echo "COMPOSE_HTTP_TIMEOUT=300" >> .env

The following configuration options can be changed by editing the environment section of the file.  

Uncomment the line for the variable and change the value.

environment:    
# - ROSETTE_JVM_MAX_HEAP=4  # max Java heap size in GB, default is 4, must be >=4;
                            # to run all endpoints the recommended minimum is 16
# - ROSETTE_WORKER_THREADS=2  # number of worker threads, default is 2, must be >=1
# - ROSETTE_PRE_WARM=false  # pre-warm the server on startup, default is false, 
                            # valid values are true|false
# - ROSETTE_DOC_HOST_PORT=localhost:8181  # hostname should be accessible on the network, 
                                          # port value should match mapped port above
#      - ROSETTE_APIKEY_SECURITY=true     # Whether to enable API key security for the Rosette Server endpoints. Can be true or false. Default is false.
#      - ROSETTE_APIKEY_SECURITY_AUTHENTICATION_ONLY_MODE=true # Whether API key security is used in authentication only mode. 
                                                               # Needs API key security to be enabled. Can be true or false. Default is true.
#      - ROSETTE_APIKEY_SECURITY_UNSECURED_ENDPOINTS=doc/**,v1/info,v1/ping # Comma separated list of endpoints that should not require API keys. 
                                                                            # Needs API key security to be enabled. Default is doc/**,v1/info,v1/ping

You can specify your own volume, for example, backed by a different volume driver.

volumes:  
# if a local volume is not desirable, change this to suit your needs  
rosette-roots-vol:

The default docker configuration uses port 8181 for the Analytics endpoints.  To change this, modify the ports section. 

ports:      
- "8181:8181"

Only the first value in the port statement should be changed.  The port statement and the ROSETTE_DOC_HOST_PORT value must match.

ports:      
- "4444:8181"

environment:
- ROSETTE_DOC_HOST_PORT=localhost:4444

If you're accessing the documentation from a different machine, change local host to the documentation machine network accessible host name.

There are times you may need to modify and/or add files to the Analytics Server installation. For example, to add an English gazetteer to entity extraction, you must add the file to the installDirectory/roots/rex/<version>/data/gazetteer/eng/accept directory.

To access the installation directories within the Docker volumes:

The response time is much slower when indoc coref is enabled. For production work, we strongly recommended that the indoc coref server is installed on a system with a GPU.

Requirements:

To install on the GPU machine:

The following endpoints must be installed and licensed in Analytics Server to support event extraction.

The script will prompt you for the following information:

Event extraction requires specific Entity Extractor configuration parameters. The install scripts install a version of the rex-factory-config.yaml file containing the correct values for the parameters. The parameters added or modified by the install scripts are in the table below.

Ping the server to test that Server is running and you can connect to it.

This should return:

{"message":"Rosette at your service","time":1467912784915}

This should return:

{
  "name": "Rosette",
  "version": "1.28.0",
  "buildNumber": "a8ea5010",
  "buildTime": "20231218215507",
  "licenseExpiration": "Perpetual"
}

Test an endpoint that you have a license for.  For example, the following code tests the entities endpoint.

This will return a list of extracted entities in JSON format.  

These are the configuration files for overall service operation. The perceptive reader may notice that there are endpoint specific configuration files here as well, e.g. dedupe, rni, rnt.

Location: server/launcher/config

The following files are deprecated as of Release 1.28.0 (December 2023):

The following files contain configuration parameters for Analytics Server, individual endpoints, and transport rules.

Location: /server/launcher/config/rosapi 

The following file is deprecated as of Release 1.28.0 (December 2023):

The worker-config.yaml file details component factories and the pipelines for each endpoint. A single endpoint may use multiple factories. Use this file to determine which factories you may have to modify to set the configuration values for a task. Some factories, such as rbl-factory-config.yaml are used by multiple endpoints.

Typically, all endpoints which you have active licenses for will load and run when called. Use these instructions to disable specific endpoints.

If you call a disabled endpoint, you will receive an "unknownError". For example, if you are licensed for the /topics endpoint, but disable it, a call to the endpoint will return the following JSON:

{
  "code": "unknownError",
  "message": "Worker unsupported target Endpoint{path=/topics}/eng",
  "stack": null
}

The /ping and /info endpoints are always enabled. They do not have to be listed in the override-endpoints.yaml file.

There are two .yaml files located in the installDirectory/roots/rni-rnt/<version>/rlpnc/data/etc directory to guide you in configuring the name-similarity endpoint, parameter_defs.yaml and parameter_profiles.yaml. The parameter_defs.yaml file lists the default value for all parameters, along with a short description. This file should not be modified.

To configure the name-similarity results, change the values of the parameters in the parameter_profiles.yaml file. Parameter values can be for all language pairs, or for a specific language pair. If the change is for all languages, use the any: profile. If a parameter change is for a specific language pair, use the appropriate language code pair. The two language codes are always written in alphabetical order, except for eng, which always comes last.

Analytics Server contains a configurable health check endpoint to report the health information of servers connected to Analytics. These servers include:

The health status supports the following states for each service:

The configuration is in the file config/com.basistech.ws.fe.health.cfg. It is disabled by default.

# Should the health/services endpoint list external services' health
# Default is false
#showExternalServicesHealth=true

Timeout for the health requests towards the external services can be set with the asyncResponseTimeoutMs field in the config/com.basistech.ws.fe.health.cfg file.

$ curl -s http://localhost:31818/rest/v1/health/services | jq '.'
{
  "events-training-server": "UP",
  "indoc-coref-server": "DOWN",
  "rex-training-server": "UP"
}

Analytics uses Log4j to configure and control logging. The file /conf/log4j2.xml configures logging of the Analytics Server instance. This file can be customized to provide the level of logging preferred by your organization.

By default, Analytics generates the following log files in the /logs/ directory:

The current default logging is deliberately quiet on successful calls and client-side (4xx) errors. It includes some startup and configuration messaging, along with errors from server-side (5xx) problems. You may require additional logging for diagnostics, monitoring, and analysis.

Analytics Server includes the ability to turn on CXF logging to provide additional message visibility for debugging. By default, this feature is disabled. To enable CXF logging, modify the following files.

At a very high level Log4J has the concept of a Logger which references one or more Appenders. Appenders are the objects that write to logs. Types of Appenders include:

Each Appender is configured slightly differently either programmatically or using an XML configuration file. An Appender will include a Layout which indicates what is written to the log. Here you can find the specification for the plain text Pattern, known as PatternLayout.

Another important concept in logging is the log level, which is an indication of the severity of the message being logged. Severity ranges from low priority INFO to WARN to ERROR. When a logger has its log level set to WARN then only log messages of severity WARN or ERROR are logged when using that logger. When set to INFO then messages of INFO or higher are logged. Loggers can be enabled for all messages in the system. These loggers are known as Root loggers. They can also be associated with specific classes by using the class's name. For example, a Logger named com.basistech.ws.logrequesttracker would only be used when a class with a name starting with com.basistech.ws.logrequesttracker outputs a log message.

There are many types of Appenders. You need to identify the Appender defined for you need. For example, HTTP Appenders send logs of HTTP, Syslog Appenders write to the system log, etc. A list of built-in Appenders can be found here.

Loggers can also have different types of layouts in addition to text. Common layouts include CSV, JSON, and XML. The list of built-in Layouts can be found here.

Logging in Analytics Server is configured through an XML configuration file named log4j2.xml, found in the server/conf directory. The configuration file defines four loggers, one of which, com.basistech.ws.logrequesttracker, is commented out (disabled) by default.

  <Loggers>
       <Root level="warn">
           <AppenderRef ref="File-Appender" level="warn"/>
           <AppenderRef ref="Console-Appender" level="warn"/>
       </Root>
       <Logger name="org.eclipse.jetty.server.handler" level="error"/>
<!--
       <Logger name="com.basistech.ws.logrequesttracker" level="info" additivity="false">
           <AppenderRef ref="Request-Appender"/>
       </Logger>
-->
       <Logger name="WEBSITE.WS - Your Internet Address For Life™ " level="info">
           <AppenderRef ref="500-Exception-Appender" level="info"/>
       </Logger>
   </Loggers>

[%-5level] %d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %c - %msg%n

[INFO ] 2024-05-30 18:56:07.729 [rosapi-worker-0]
com.basistech.rosette.tvec.EmbeddingMetadata - Vector metadata:

<Appenders>
       <Console name="Console-Appender" target="SYSTEM_OUT">
           <PatternLayout>
               <pattern>
                   [%-5level] %d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %c - %msg%n
               </pattern>>
           </PatternLayout>
       </Console>
       <RollingFile name="File-Appender"
                    fileName="${log-path}/rosapi.log"
                    filePattern="${archive}/rosapi.log.%d{yyyy-MM-dd-hh-mm}.gz">
           <PatternLayout pattern="[%-5level] %d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %c - %msg%n"/>
           <Policies>
               <SizeBasedTriggeringPolicy size="30 MB"/>
           </Policies>
           <DefaultRolloverStrategy max="30"/>
       </RollingFile>
<!--
       <RollingFile name="Request-Appender"
                    fileName="${log-path}/request-tracker.log"
                    filePattern="${archive}/request-tracker.log.%d{yyyy-MM-dd-hh-mm}.gz">
           <PatternLayout pattern="[%-5level] %d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %c - %msg%n"/>
           <Policies>
               <SizeBasedTriggeringPolicy size="30 MB"/>
           </Policies>
           <DefaultRolloverStrategy max="30"/>
       </RollingFile>
-->
       <RollingFile name="500-Exception-Appender"
                    fileName="${log-path}/500-exception.log"
                    filePattern="${archive}/500-exception.log.%d{yyyy-MM-dd-hh-mm}.gz">
           <PatternLayout pattern="[%-5level] %d{yyyy-MM-dd HH:mm:ss.SSS} [%t] %c - %msg%n"/>
           <RegexFilter regex=".*Exception processing ticket.*" onMatch="ACCEPT" onMismatch="DENY"/>
           <Policies>
               <SizeBasedTriggeringPolicy size="30 MB"/>
           </Policies>
           <DefaultRolloverStrategy max="30"/>
       </RollingFile>
   </Appenders>

To disable writing to files and to instead log all messages to STDOUT or STDERRR:

For example, to only write to STDOUT:

  <Loggers>
       <Root level="warn">
<!--
           <AppenderRef ref="File-Appender" level="warn"/> 
-->
           <AppenderRef ref="Console-Appender" level="warn"/>
       </Root>
       <Logger name="org.eclipse.jetty.server.handler" level="error"/>
<!-- 
      <Logger name="com.basistech.ws.logrequesttracker" level="info" additivity="false">
           <AppenderRef ref="Request-Appender"/>
       </Logger>
-->
       <Logger name="WEBSITE.WS - Your Internet Address For Life™ " level="info">
<!--            <AppenderRef ref="500-Exception-Appender" level="info"/> -->
           <AppenderRef ref="Console-Appender" level="info"/>
       </Logger>
   </Loggers>

There is not a single one size fits all number here. The best value for max heap size depends on a number of factors:

Our recommendation is to follow directions from well-known sources, such as this to experiment with heap settings by testing your usage of Analytics Server in order to identify the ideal settings that suits you the best.

Please note that it’s not recommended setting the max heap to the amount of physical RAM in the system. More heap doesn’t always translate to better performance, especially depending on your garbage collection settings. Also, we do require sufficient amount of free memory for memory mapped files.

Use this table to estimate the minimum heap required based on your selection of endpoints. Note that endpoints may have implicit code dependencies on other endpoints, so the dependencies' heap needs to be added if they have not been accounted for.

On macOS/Linux or Windows:

With Docker:

Multiple worker threads allow you to implement parallel request processing. Generally, we recommend that the number of threads should be less than the number of physical cores or less than the total number of hyperthreads, if enabled.

You can experiment with 2-4 worker threads per core. More worker threads may improve throughput a bit, but typically won't improve latency. The default value of worker threads is 2.

If the URL for all licensed endpoints are set to local: (not distributed):

If using transport rules in a distributed deployment on macOS/Linux or Windows:

If using Docker, only the docker-compose.yml file must be modified:

To speed up first call response time, Analytics Server can be pre-warmed by loading data files at startup at the cost of a larger memory footprint.

Most components load their data lazily, meaning that the data required for processing will only be loaded into memory when an actual call hits. This is particularly true for language-specific data. The consequence is that when the very first call with text in a given language arrives at a worker, the worker can take a quite a bit of time loading data before it can process the request.

Pre-warming is Analytics Server's attempt to address the 1st-call penalty by hitting the worker with text in every licensed language it supports at boot time. Then, when an actual customer request comes in, all data will have already been memory mapped and you won't experience a first call delay as the data is loaded. Only languages licensed for your installation will be pre-warmed.

The default is set to false, pre-warm is not enabled.

To set Analytics Server to warm up the worker upon activation

On macOS/Linux or Windows:

With Docker:

If the language of the input text is known, you can add the language parameter to bypass the language identification step in the processing pipeline, speeding up the processing time and increasing throughput.

Each document endpoint accepts an optional language parameter:

{"content": "your_text_here", "language":"eng"}

If the data consists of many relatively small individual files, concatenating them will improve the throughput. But you must be aware that this can impact the accuracy of the model. The statistical model includes a consistency feature which reflects a tendency of the model to label recurring tokens with the same type. This may cause entities to be labelled incorrectly when concatenating text samples that don't share the same context.

Regular Expressions

Regular expressions (regexes) are used for finding entities which follow a strict pattern with a rigid form and infinite combinations, such as URLs and credit card numbers. In the default Entity Extractor installation the regex files are:

Regular expressions can decrease throughput performance. The /entities endpoint is pre-configured with a set of regular expressions. You can improve performance by removing unused expressions by:

The supplemental regular expressions are configured in the rex-factory-config.yaml file. Remove or comment out values from the supplementalRegularExpressionPaths parameter to remove unused supplemental regex files.

Disable Entity linking. By default, entity linking is disabled, but enabling it can slow down the response time of Analytics Server.

Disable Pronominal resolution By default, pronominal resolution is disabled, but enabling it can slow down the response time of Analytics Server.

Disable In-document Coreference Documents often contain multiple references to a single entity. In-document coreference (indoc coref) chains together all mentions to the same entity. By default, indoc coref is disabled (NULL).

The limits for the input parameters are in the file /rosapi/constraints.yaml. Modify the values in this file to increase the limits on the maximum input character count and maximum input payload per call.  You can also increase the number of names per list for each call to the name deduplication endpoint.

The default values were determined as optimal during early rounds of performance tests targeting < 2 second response times.  Larger values may cause degradation of system performance. 

To modify the input constraints:

Most endpoints can take either a text block, a file, or a link to a webpage as the input text.  The webpage link is in the form of a URI. To enable passing a URI to an endpoint, the enableDTE flag must be set in the file com.basistech.ws.worker.cfg.

By default, the flag is set to True; URI passing is enabled.

#download and text 
extractorenableDte=true

The default installation uses port 8181 for the Analytics endpoints. To change the default port, edit the file conf/wrapper.conf; uncomment and modify the port value.

wrapper.java.additional.301=-Drosapi.port=8181

For example, change 8181 to 9191.

When changing the port, update the documentation hostname as well:

If you want to change the default port to execute the interactive documentation:

The transport rules provide a means of mapping an endpoint along with some defined options (language, linked-entities, length) to a processing URL.  Transport rules allow you to define a distributed deployment routing a subset of calls to different machines, balancing the load by routing high demand calls to separate machines. 

Location:  <version>/server/launcher/config/rosapi/transport-rules.tsv

The basic format of an entry is:

endpoint [tab] options [tab] URL

where:

An example of multiple endpoint entries:

/entities lang=eng|spa, http://localhost:${rosapi.port}/rest/worker/process
/entities linkEntities=false  http://localhost:${rosapi.port}/rest/worker/process
/entities lang=ara|eng|jpn|spa|zho  http://localhost:${rosapi.port}/rest/worker/process
/entities * http://localhost:${rosapi.port}/rest/worker/process
/language       *       local:

To improve performance, especially when invoking deep neural network based models, you may choose to run on a GPU.

Download TensorFlow 

To make use of GPUs on your Linux system, you should download libtensorflow_jni_gpu-1.14.0.jar Platform-dependent native code with GPU (CUDA) support for the TensorFlow Java library from http://repo1.maven.org/maven2/org/tensorflow/libtensorflow_jni_gpu/1.14.0/libtensorflow_jni_gpu-1.14.0.jar.

You can also compile your own version of libtensorflow_jni, which may provide better performance than the pre-compiled version.

cd $ROSAPI
mkdir tf_jni_gpu
jar xf libtensorflow_jni_gpu-1.14.0.jar -C tf_jni_gpu

Edit conf/wrapper.conf to modify java.library.path to the following:

wrapper.java.library.path.1=../tf_jni_gpu/org/tensorflow/native/linux-x86_64
wrapper.java.library.path.2=../lib

Verify TensorFlow GPU Support 

To verify TensorFlow GPU support, run Analytics Server

cd $ROSAPI/bin
./launch.sh console

And in an entity request with DNN modelType options:

curl --request POST \
--url http://localhost:8181/rest/v1/entities \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--data '{"content": "Barack Obama was born in Hawaii", \
  "options": { "modelType": "DNN" }}'

Verify that you see Tensorflow found and create a GPU in your console output

jvm 1    | 2018-04-18 18:38:15.346273: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1344] Found device 0 with properties:
jvm 1    | name: Tesla K80 major: 3 minor: 7 memoryClockRate(GHz): 0.8235
jvm 1    | pciBusID: 0000:00:1e.0
jvm 1    | totalMemory: 11.17GiB freeMemory: 11.10GiB
jvm 1    | 2018-04-18 18:38:15.346291: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1423] Adding visible gpu devices: 0
jvm 1    | 2018-04-18 18:38:15.617319: I tensorflow/core/common_runtime/gpu/gpu_device.cc:911] Device interconnect StreamExecutor with strength\
 1 edge matrix:
jvm 1    | 2018-04-18 18:38:15.617346: I tensorflow/core/common_runtime/gpu/gpu_device.cc:917]      0
jvm 1    | 2018-04-18 18:38:15.617351: I tensorflow/core/common_runtime/gpu/gpu_device.cc:930] 0:   N
jvm 1    | 2018-04-18 18:38:15.617609: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1041] Created TensorFlow device (/job:localhost/repli\
ca:0/task:0/device:GPU:0 with 10764 MB memory) -> physical GPU (device: 0, name: Tesla K80, pci bus id: 0000:00:1e.0, compute capability: 3.7)
jvm 1    | 2018-04-18 18:38:15.941174: I tensorflow/cc/saved_model/loader.cc:161] Restoring SavedModel bundle.
jvm 1    | 2018-04-18 18:38:16.094135: I tensorflow/cc/saved_model/loader.cc:196] Running LegacyInitOp on SavedModel bundle.
jvm 1    | 2018-04-18 18:38:16.131329: I tensorflow/cc/saved_model/loader.cc:291] SavedModel load for tags { serve }; Status: success. Took 986\
397 microseconds.

Known Limitations 

Tensorflow does not provide GPU support for Java macOS as of Tensorflow 1.14

Enable the feature

Select the authentication and authorization setting

Edit the file launcher/config/com.basistech.ws.apikeys.cfg. By default, only authentication is enabled. To enable both authentication and authorization, uncomment the following line and set the value to false.

#authenticationOnly=true 

If installing using Docker, there are 3 additional environment variables:

# Whether to enable API key security for the Rosette Server endpoints. 
# Can be true or false. Default is false.
#     - ROSETTE_APIKEY_SECURITY=true 

# Whether API key security is used in authentication only mode. 
# Needs API key security to be enabled. Can be true or false. Default is true.
#     - ROSETTE_APIKEY_SECURITY_AUTHENTICATION_ONLY_MODE=true 

# Comma separated list of endpoints that should not require API keys. 
#Needs API key security to be enabled. Default is doc/**,v1/info,v1/ping 
#     - ROSETTE_APIKEY_SECURITY_UNSECURED_ENDPOINTS=doc/**,v1/info,v1/ping

The key is passed using the header, X-BabelStreetAPI-Key.  This header is pre-defined in our programming language bindings, as it is the same header used for accessing Babel Street Hosted Services.

401:  the key is expired, it has been disabled, it doesn't exist in the database, or, no key was included in the request. 

403:  the key is valid (enabled, unexpired, exists) but does not have authorization for the endpoint being called.

Other configuration parameters are set in the file. These options are exposed to allow you to customize the configuration for your implementation. To change any of the parameters, uncomment the parameter and change the value. You must then restart the server.

launcher/config/com.basistech.ws.apikeys.cfg

Database connection mode

A database is required to store the authentication and authorization users, keys, and other data. The database can be on your local file system or a separate H2 database server.

Unsecured endpoints

This is a list of endpoints that are accessible to unauthenticated users. The default list includes the browsable documentation as well as /info and /ping.

Key expiration

On creation, keys will never expire unless an expiration is explicitly defined.

You can explicitly define the number of days a key should be valid from it's creation, by providing a positive integer to the --expiryDays parameter of the create key command.

Providing 0 will make the command use the default value set in the file. If no default value is defined in the file, then it will be set to 90 days.

Updates to this default are only registered during the console startup.  If you change the value make sure to restart the API Key Management Console.

rosette-apikeys is a shell tool to manage the keys. To launch the tool:

Linux/macOS

./bin/rosette-apikeys  

Windows

.\bin\rosette-apikeys.bat 

Once in the shell, use help to see the available commands:

help 

To see the usage for a command:

help <command> 

or

<command> --help

To run commands in non-interactive mode, add them after the script. Example:

./bin/rosette-apikeys list keys

The key management console can communicate with an SSL secured database server. It uses the same launcher/config/com.basistech.ws.apikeys.cfg file as Server to connect to the database server. A custom truststore can be provided for the key management console by setting the RS_TRUSTSTORE and RS_TRUSTSTORE_PASSWORD environment variables.

For example, if you have your truststore located in launcher/config, set your environment variables:

export RS_TRUSTSTORE=/rosette/server/launcher/config/my_custom_truststore.jks 
export RS_TRUSTSTORE_PASSWORD=mycustomtruststorepassword 

Calling Analytics Server 

The API keys are sent in headers and are vulnerable in transfer if unencrypted. To secure them, enable SSL/TLS for Analytics Server.

If the database is in server mode:

The following table shows the endpoint(s) that correspond to each assigned authority. When you add the authority, you may get access to multiple endpoints.

Edit the keystore and truststore file properties and passwords in launcher/config/jetty-ssl-config.xml.

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xmlns:http="http://cxf.apache.org/transports/http/configuration"
       xmlns:httpj="http://cxf.apache.org/transports/http-jetty/configuration"
       xmlns:sec="http://cxf.apache.org/configuration/security"
       xsi:schemaLocation="
        http://www.springframework.org/schema/beans                 http://www.springframework.org/schema/beans/spring-beans.xsd
        http://cxf.apache.org/transports/http/configuration         http://cxf.apache.org/schemas/configuration/http-conf.xsd
        http://cxf.apache.org/transports/http-jetty/configuration   http://cxf.apache.org/schemas/configuration/http-jetty.xsd
        http://cxf.apache.org/configuration/security                http://cxf.apache.org/schemas/configuration/security.xsd">


    <httpj:engine-factory id="rosette-server-engine-config">
        <httpj:engine port="#{ systemProperties['rosapi.port'] }">
            <httpj:tlsServerParameters>
                <sec:clientAuthentication required="false" />
                <sec:keyManagers keyPassword="[key-pass]">
                    <sec:keyStore type="JKS" password="[keystore-pass]"
                                  file="path/to/keystore.jks"/>
                </sec:keyManagers>
                <sec:trustManagers>
                    <sec:keyStore type="JKS" password="[truststore-pass]"
                                  file="path/to/truststore.jks"/>
                </sec:trustManagers>
            </httpj:tlsServerParameters>
        </httpj:engine>
    </httpj:engine-factory>

</beans>

Change http to https in /launcher/config/com.basistech.ws.cxf.cfg.

urlBase=https://0.0.0.0:${rosapi.port}/rest

To use remote workers, the certificate needs to be trusted.

For testing, import the certificate to the truststore file, cacerts.jks, as trusted.

This example is for evaluation purposes only, continuing using the previously generated key.  Please work with your appropriate internal group to acquire your keys for production usage.  If your key is acquired from a trusted certificate authority, no further configuration may be required.  As this example uses self-signed certificates, the following steps are necessary.

When you call the API, add "profileId" = "myProfileId" to the body of the call.

{"content": "The black bear fought the white tiger at London Zoo.",
 "profileId": "group1"
}

https://localhost:8181/rest/v1/custom-profiles

The /custom-profiles endpoint returns a list of all custom profiles on the server.

curl -s http://localhost:8181/rest/v1/custom-profiles

If the call includes an app-id in the request header, the custom-profiles endpoint returns all profiles under the specified app-id.

curl -s http://localhost:8181/rest/v1/custom-profiles -H "X-RosetteAPI-App-Id: app-id"

New profiles are automatically loaded in Analytics Server. You do not have to bring down or restart the instance to add new models or data to Analytics Server.

When editing an existing profile, you may need to restart Analytics Server. If the profile has been called since Analytics Server was started, the Server must be restarted for the changes to take effect. If the profile has not been called since Analytics Server was started, there is no need to restart.

To add or update models or data, assuming the custom profile root rosette-users and profiles group1 and group2.

The configurations for each endpoint are contained in the factory configuration files. The worker-config.yaml file describes which factory configuration files are used by each endpoint as well as the pipelines for each endpoint. To modify parameter values or any other configuration values, copy the factory configuration file into the profile path and modify the values.

Each profile can include custom data sets. For example, the entities endpoint includes multiple types of data files, including regex and gazetteers. These files can be put into their own directory for entities, known as an overlay directory. This is an additional data directory which takes priority over the default entities data directory.

You can train and deploy a custom model to the entities endpoint for entity extraction. You can either:

In this example, we're going to add the entity types COLORS and ANIMALS to the entities endpoint, using a regex file.

To access the statistics, call the usage endpoint:

curl http://localhost:8181/rest/usage

where localhost:8181 is the location of the Analytics installation.

Sample Response

{"no-app-id": {
   "no-profile-id": {
      "/rest/v1/tokens": {
        "eng": {
            "calls": 1
        },
        "zho": {
            "calls": 1
        }
      },
      "/rest/v1/categories": {
        "eng": {
            "calls": 1
        }
      },
      "/rest/v1/language": {
        "xxx": {
            "calls": 1
        }
      }
    }
  }
}

You can also aggregate usage data by having Prometheus pull metrics from multiple instances using the /metrics endpoint. A single call returns all endpoints.

curl http://localhost:8181/rest/metrics

where localhost:8181 is the location of the Analytics installation.

Sample Response

# HELP rosette_http_requests_total Total number of Rosette Enterprise requests processed.
# TYPE rosette_http_requests_total counter
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/tokens",lang="zho",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/semantics/vector",lang="eng",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/morphology/compound-components",lang="deu",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/syntax/dependencies",lang="eng",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",            
    endpoint="/rest/v1/morphology/lemmas",lang="eng",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/topics",lang="eng",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/transliteration",lang="eng",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/sentences",lang="eng",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/address-similarity",lang="xxx",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/name-deduplication",lang="xxx",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/morphology/complete",lang="eng",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/entities",lang="eng",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/name-translation",lang="xxx",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/morphology/parts-of-speech",lang="eng",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/semantics/similar",lang="eng",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/language",lang="xxx",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/sentiment",lang="eng",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/categories",lang="eng",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/name-similarity",lang="xxx",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/morphology/han-readings",lang="zho",} 1.0
rosette_http_requests_total{app_id="no-app-id",profile_id="no-profile-id",
    endpoint="/rest/v1/relationships",lang="eng",} 1.0

The configuration parameters for usage tracking is in the file launcher/config/com.basistech.ws.local.usage.tracker.cfg.

To reset the counter:

No authorization is required when using an on-premises installation of Analytics. You may, however, want to track Analytics calls by groups within your organization. To do this, include an application id (app-id) in the request header of all calls. This allows Analytics to track usage by app-id.

An application id is:

Example: 

curl -s -X POST \
    -H "X-RosetteAPI-App-Id: usergroup1" \
    -H "Content-Type: application/json" \
    -H "Accept: application/json" \
    -H "Cache-Control: no-cache" \
    -d '{"content": "Por favor Señorita, says the man." }' \
    "https://localhost:8181/rest/v1/language"

wrapper.java.additional.250=-Drosapi.feature.CUSTOM_PROFILE_UNDER_APP_ID

When tracking usage in installations which utilize custom profiles, you can require that the custom profiles must be installed in subdirectories under each application group.

Example:

The app-id identifies the group making the call. The app-id is the value of X-RosetteAPI-App-ID in the call header. If no application id is provided in the call header, the calls are allocated to the no-app-id group.

The ProfileId identifies the customized components

<app-id>/<ProfileID>/config
<app-id>/<ProfileID>/config/rosapi
<app-id>/<ProfileID>/custom-rex

If you have multiple groups (app-id) using the same set of configuration and customizations, you can create symbolic links (ln -s <source> <link> to save disk space. Otherwise, you will have to install the customizations under each app-id.

Figure 3. Example directories and links

The custom endpoint architecture includes two platform-independent components:

The reverse proxy's configuration file routes all incoming requests:

All shipments contain the file rosent-custom-endpoint-installer-<version>.tar.gz containing an installation script and the files for the reverse proxy and application server. The installation script is currently for Linux and macOS only.

You can install the custom endpoints while Analytics Server is running, or start it after you complete the custom endpoints installation.

The application.yml file configures most aspects of the proxy including ports, certificates, routing, and timeouts. If you move the application.yml file, the ./conf/proxy-wrapper.conf must be updated with the new location.

wrapper.app.parameter.2=--spring.config.location=<new file location>

The proxy can be configured using the values in the application.yml file directly or by setting the environment variables in ./conf/proxy-wrapper.conf. 

If your custom code is in Java, put your WAR file into the webapps directory of the Tomcat installation directory. To use other languages, see Adding non-Java custom code.

Your Java web application should define a URI to access the code. Create a rule in the proxy definition to adjust the final path to access the custom code.

Each component, including Analytics Server, has its own, independent, Tanuki Java Wrapper application. Tanuki monitors the health of the application and relaunches if the Java application crashes or gets into a bad state.

In addition, the proxy application exposes two endpoints that can be used to access the proxy health:

The custom endpoint application includes an example application, matchSentences , which calculates the relevance of documents and sentences against a list of keywords. It combines calls to the /sentences and /semantics/vector endpoints with custom code. The custom code uses the cosine similarity function to calculate similarity scores between the sentences and the keywords, as well as comparing the scores to the input threshold value.

The input is a list of keywords, a document or the URL to a document, and a threshold.

The response is the cosine similarity scores and a true/false value indicating if the score is above the threshold for each sentence, and for the entire document.

{
    "keywords": [
       "USA",
       "Iran",
       "attack",
       "protest"
    ],    
    "document": "Iran was planning attacks on    four US embassies 
     when its top general was killed, President Donald Trump says. 
     When asked what threat led to last Friday's US drone strike, 
     he told Fox News: \"I can reveal that I believe it probably
     would've been four embassies.\" \"The killing of Gen Qasem Soleimani, 
     a national hero, came after days of protests at the US embassy in Baghdad.",    
   "threshold": 0.25
}

curl -s http://localhost:8182/rest/v1/match-sentences -XPOST \
-H 'Content-Type: application/json; charset=utf-8' -d \
@matchSentences.json 

spring:
  cloud:
    gateway:
      routes:
        # Route to the custom endpoint
        - id: sample_app_route
          uri: ${SAMPLE_ENDPOINT_APP_HOST:http://localhost:8183}
          predicates:
            - Path=/rest/v1/match-sentences/**,matchTrailingSlash=false
          filters:
# When running the custom endpoint as a stand-alone spring boot application
#           - RewritePath=/rest/v1/match-sentences/info(?<segment>.*), /service/info$\{segment}
#           - RewritePath=/rest/v1/match-sentences/health(?<segment>.*), /service/health$\{segment}
#           - RewritePath=/rest/v1/match-sentences/prometheus(?<segment>.*), /service/prometheus$\{segment}
#           - RewritePath=/rest/v1/match-sentences/env(?<segment>.*), /service/env$\{segment}
#           - RewritePath=/rest/v1/match-sentences(?<segment>.*), /service/matchSentences/$\{segment}
# When running the custom endpoint as a war file in tomcat
#           - RewritePath=/rest/v1/match-sentences/info(?<segment>.*), /custom-endpoint/info$\{segment}
#           - RewritePath=/rest/v1/match-sentences/health(?<segment>.*), /custom-endpoint/health$\{segment}
#           - RewritePath=/rest/v1/match-sentences/prometheus(?<segment>.*), /custom-endpoint/prometheus$\{segment}
#           - RewritePath=/rest/v1/match-sentences/env(?<segment>.*), /custom-endpoint/env$\{segment}
#           - RewritePath=/rest/v1/match-sentences(?<segment>.*), /custom-endpoint/matchSentences/$\{segment}
     # Calls to Analytics Server Endpoints
     - id: rosette_pass_thru
       uri: ${ROSETTE_SERVER_HOST:http://localhost:8181}
       predicates:
         - Path=/rest/v1/**,matchTrailingSlash=false
         - Path=/rest/**,matchTrailingSlash=false

The examples and instructions here assume you are writing your custom code in Java. If you are using a different language, you will need to set up your own application server that is specific for that language. Then you add your own routing rules to the proxy so that the traffic is directed to the correct place.

Let's assume you'd like to add a python file named python_example.py,and you want to execute it from the url http://localhost:8182/rest/v1/python_example.

There are many ways to configure Match to better fit your use case and data. The two primary mechanisms are by modifying match parameters and editing overrides. You can also train a custom language model.

To modify the default configuration of the /entities endpoint, edit the file /launcher/config/rosapi/rex-factory-config.yaml. The file contains the following parameters. To change the value of a parameter, uncomment the parameter and set the new value.

Within a document, there may be multiple references to a single entity. In-document coreference (indoc coref) chains together all mentions to an entity.

You can use the API to dynamically add gazetteer entries to the /entities endpoint. The REST endpoint is:

https://localhost:8181/rest/v1/entities/configuration/gazetteer/add 

Parameters:

curl --request POST \
--url http://localhost:8181/rest/v1/entities/configuration/gazetteer/add \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--data '{"language": "xxx", "configuration":{"entities":{ "COMPANY": ["New Corp", "Best Business"]}}}'

curl --request POST \
--url http://localhost:8181/rest/v1/entities/configuration/gazetteer/add \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--data '{"language": "xxx", \
"configuration":{"entities":{ "COMPANY": ["New Corp", "Best Business"]}}, "profileId": "group1"}'

{"language": "xxx", "configuration": {"entities":{ "COMPANY": ["New Corp", "Best Business"] } } } 

curl --request POST \
--url http://localhost:8181/rest/v1/entities/configuration/gazetteer/add \
--header 'accept: application/json' \
--header 'content-type: application/json' \
--data '@new_companies.json'

If your project has a set of unique data files that you would like to keep separate from other data files, you can put them in their own directory, also known as an overlay directory. This is an additional data directory, which takes priority over the default Entity Extractor data directory.

The overlay directory must have the same directory tree as the provided data directory. If an overlay directory is set, Entity Extractor searches both it and the default data directory.

To specify the overlay directory use:

my-data/gazetter/deu/reject/reject-names.txt

The default configuration of the /entities endpoint uses the same default values as the Entity Extractor Java SDK. It is optimized to be more performance-oriented, with fewer options enabled, than the configuration of Babel Street Hosted Services, which is configured to provide a fully-functional demonstration environment. ^[1]

kbs:
    - /customKBs/kb1
    - /customKBs/kb2
    - /rosette/server/roots/rex/7.44.1.c62.2/data/flinx/data/kb/basis

#The option to add supplemental regex files, usually for entity types that are excluded by
#default. The supplemental regex files are located at data/regex/<lang>/accept/supplemental and
#are not used unless specified.
supplementalRegularExpressionPaths:
- "${rex-root}/data/regex/eng/accept/supplemental/date-regexes.xml"
- "${rex-root}/data/regex/eng/accept/supplemental/geo-regexes.xml"

#The option to resolve pronouns to person entities.
resolvePronouns: true

#The capitalization (aka 'case') used in the input texts. Processing standard documents
#requires caseSensitive, which is the default. Documents with all-caps, no-caps or headline
#capitalization may yield higher accuracy if processed with the caseInsensitive value.
caseSensitivity: automatic

In cases where a document or part of a document contains tables and lists, instead of sentences, the /sentences endpoint can detect fragment boundaries as sentence boundaries. One way fragment boundaries are identified is by encountering fragment delimiters. A delimiter is restricted to one character and the default delimiters are U+0009 (tab), U+000B (vertical tab), and U+000C (form feed).

You can modify the set of recognized delimiters:

In addition to the fragment delimiters, the fragment boundary detector automatically inserts a break:

By default, fragment boundary detection is turned on. To turn off fragment boundary detection:

The Analytics Classification Field Training Kit allows users to train their own classification models for the /categories and /sentiment endpoints. Reasons for training a new model include:

See the Training Classification Models with Rosette publication for more information.

 > mkdir ${tcat-root}/models/<lang>/unused
 > mv ${tcat-root}/models/<lang>/combined-iab-qag/* ${tcat-root}/models/<lang>/unused

The sentiment analysis endpoint can be configured to return document-level sentiment analysis only, by turning off entity-level sentiment analysis. This requires modifying the worker-config.yaml file to remove the entity extraction step from the process. This will speed up document-level sentiment analysis.

The following edits are made to the worker-config.yaml file. The shipped version of the file is:

# sentiment
- endpoint: /sentiment
  languages: [ 'ara', 'eng', 'fas', 'fra', 'jpn', 'spa' ]
  steps:
  - componentName: entity-extraction
  - componentName: sentiment

Change the above block to:

# sentiment
- endpoint: /sentiment
  languages: [ 'ara', 'eng', 'fas', 'fra', 'jpn', 'spa' ]
  steps:
  - componentName: base-linguistics
    factoryName: tokenize
  - componentName: sentiment

The entity extraction endpoint must be replaced by the tokenization endpoint in the pipeline.