Chapter 3 Answer Formats for Queries

The ConceptBase server provides an ASK command in its interface, which allows to specify in which text-based format the answer should be returned. There are two pre-defined formats: one for returning a list of object names, and one for returning a list of object frames. These two formats can be extended by user-defined answer formats.

Examples for answer formats are available from the CB-Forum, see link.

To understand ConceptBase answer formats, we first look at the syntax of the ASK command at the programming interface. We use here the syntax of CBshell (section 7) since it can be tested directly in a command terminal:

   ask "<querydefinition>" FRAMES <answer-format> <roll-backtime>
   ask <querycall> OBJNAMES <answer-format> <roll-backtime>

The first variant is for queries where the query is provided as a Telos frame, the second one is for parameterized query calls like Q[abc/param1]. If the query call contains special characters or is computes of several query calls then surround them by double quotes. The rollback time is typically Now, i.e. the query refers to the current database state. There are four options for the answer format:

LABEL:: the answer shall be returned as a comma-separated list of object names, e.g. bill, mary, john
FRAME:: the answer shall be returned as a list of Telos frames.
JSONIC:: the answer shall be returned as a list of JSON-like frames.
answer-format-name:: Finally, the name of a user-defined answer format can be provided, like MyFormat. This will override any answer format that is assigned via the forQuery attribute for the given query. Hence, one can maintain several answer formats for the same query.
default:: This value is leaving the choice to the ConceptBase server. For function calls, the answer format LABEL is selected, for other query calls, ConceptBase first checks if an answer format was defined for the query (attribute forQuery). If that exists, it is selected, otherwise the format FRAME is selected. If there are more than one, the first one is selected.

The command ask <querycall> is a shortcut for

   ask <querycall> OBJNAMES LABEL Now

Assume, we want to execute a query call Q[abc/param1] on the current database and have the answer formatted according as frames, we would issue:

   ask Q[abc/param1] OBJNAMES FRAME Now

We assume here that there is no user-defined answer format for query Q. If there is a user-defined answer-format like MyFormatA and MyFormatB, but the answer formats are not assigned via forQuery to Q, we can call:

   ask Q[abc/param1] OBJNAMES MyFormatA Now

and also

   ask Q[abc/param1] OBJNAMES MyFormatB Now

If we assign MyFormatB to Q via the forQuery attribute, the last call is equivalent to the first call using default as answer format name.

Subsequently, the specification of such user-defined answer formats is presented.

3.1 Basic definitions

By default, ConceptBase displays answers to queries in the FRAME format (see ‘A’ and ‘B’ below). For many applications, other answer representations are more useful. For example, relational data is more readable in a table structure. Another important example are XML data. If ConceptBase is integrated into a Web-based information system, then answers in HTML format are quite useful. For this reason, answer format definitions are provided.

Answer formats in ConceptBase are based on term substitutions where terms are evaluated against substitution rules defined by the answers to a query. A substitution rule has the form L —→ R with the intended meaning that a substring L in a string is replaced by the substring R. The object of a term substitution is a string in which terms may occur, for example:

this is a string called {a} with a term {b}

Assume the substitution rules:

{a} —→ string no. {x}
{b} —→ that was subject to substitution
{x} —→ 123

The derivation of a string with terms proceeds from left to right. First, the term occurence {a} is dealt with. The next term in the string is then {x} which is evaluated to 123. Finally, {b} is substituted and the result string is this is a string called string no. 123 with a term that was subject to substitution.

We denote a single derivation step of a string S₁ to a string S₂ by S₁ =⇒ S₂. It is defined when there occurs a substring L in S₁, i.e. S₁=V+L+W and a substitution rule L —→ R and S₂=V+R+W. The substrings V and W may be empty. A string S is called ground when no substition rule can be applied. A sequence S =⇒ S₁ =⇒ … =⇒ S_n is called a derivation of S. A complete derivation of S ends with a ground string. In our example, the complete derivation is:

	this is a string called `{a}` with a term `{b}`.
=⇒	this is a string called string no. `{x}` with a term `{b}`.
=⇒	this is a string called string no. 123 with a term that was
	subject to substitution.

An exception to the left-to-right rule are complex terms like {do({y})}. Here, the inner term {y} is first evaluated (e.g. to 20) and then the result {do(20)} is evaluated.

In general, term substitution can result in infinite loops. This looping can be prevented either by restricting the structure of the substitution rule or by terminating the substitution process after a finite number of steps. The end result of a substitution process of a string is called its derivation. In ConceptBase, the substitution rules are guaranteeing termination except for the case of external procedures. The problem with the exception is solved by prohibiting cyclic calls of the same external procedure during the substitution of a call. A cyclic call is a call that has the same function name (e.w. query class) and the same arguments (expressed as parameter substitutions).

In ConceptBase, an answer format is an instance of the new pre-defined class ‘AnswerFormat’.

Individual AnswerFormat in Class with
  attribute
     forQuery : QueryClass;
     order : Order;
     orderBy : String;
     head : String;
     pattern : String;
     tail : String;
     fileType: String
end

The first attribute assigns an answer format to a query class (a query may have at most one answer format). The second and third attribute specify the sorting order of the answer, i.e. one can specify by which field an answer is sorted, and whether the answer objects are sorted ‘ascending’ or ‘descending’ (much like in SQL). The ’orderBy’ attribute specifies the property by which the answer shall be sorted. The most common value is the expression "this", i.e. sort the answer by the name of the objects in the answer. You can also specify an attribute expression such as "this.name" referring to an answer variable. If you specify "none" for ’orderBy’, then the answer is not sorted. If the number of objects in an answer exceeds 5000, then no sorting is applied due to memory limitations.

The ‘head’, ‘pattern’, and ‘tail’ arguments are strings that define the substring substitution when the answer is formatted. They contain substrings of type expr that are replaced. The head and tail strings are evaluated once, independent form the answer to the query. Usually, they do not contain expressions but only text. The response to a query is a set of answer objects A1,A2,.... The pattern string is evaluated against each answer object. For each answer object, the derivation of the pattern is put into the answer text. Hence, the complete answer consists of

	derivation of head string
+	derivation of pattern string for answer object A1
+	derivation of pattern string for answer object A2
	…
+	derivation of tail string

The fileType attribute is explained in section 3.4.

In the next sections, we will explain more details about answer formats using the following example: An answer object A to a query class QC has by default a ‘frame’ structure

   A in QC with
     cat1
        label11: v11;
        label12: v12
        [...]
     cat2
        label21: v21
     [...]
   end

In case of a complex view definition VC, the values vij can be answer objects themselves, e.g.

   B in VC with
     cat1
        label11: v11;
        label12: v12
        [...]
     cat2
        label21: v21 with
                   cat21
                     label211: v211
                 [...]
                 end
     [...]
   end

3.2 Constructs in answer formats

3.2.1 Simple expressions in patterns

We first concentrate on the pattern attribute, i.e. they are not applicable in the head or tail attribute of an answer format. The pattern of an answer format is applied to each answer object of a given query call, effectively transforming it according to the pattern. The following expressions are allowed. Capital letters in the list below indicate that the term is a placeholder for some label occuring in the query definition or the answer objects.

{this}: denotes the object name of an answer object (e.g. ‘A’). The syntax for object names is defined in section 2.1. In particular, attribution objects have names like mary!earns.
{this.ATTRIBUTE_CAT}: denotes the value(s) of the attribute ‘ATTRIBUTE_CAT’ of the current answer object ‘this’. The attribute must be defined in the query class (retrieved_attribute, computed_attribute) or view (inherited_attribute). Note that some attributes are multi-valued. For the answer object ‘A’, {this.cat1} evaluates to v11,v12 and {this.cat2} evaluates to v21. For the complex object ‘B’, a path expression like {this.cat2.cat21} is allowed and yields v21. Note that all such expressions are set-valued.
{this^ATTRIBUTE_LABEL}: denotes the value of the attribute ‘ATTRIBUTE_LABEL’ of the current answer object ‘this’. The attribute_label is at the level of the answer object, i.e. not at the class level but at the instance level. Therefore, this expression is rarely used because the instance level attribute labels are usually unknown at query definition time. Example A and B: {this^label12} evaluates to v12.
{this∣ATTRIBUTE_CAT}: denotes all attribute labels of the answer object ‘this’ that are grouped under the category ‘ATTRIBUTE_CAT’ (defined in the query class). Example: {this|cat1} evaluates to label11,label12.

The derivation of pattern is performed for each answer object that is in the result set of a query. The answer object ‘A’ induces the following substition rules:

`{this}`	—→	A
`{this.cat1}`	—→	v11,v12
`{this.cat2}`	—→	v21
`{this^label11}`	—→	v11
`{this^label12}`	—→	v12
`{this^label21}`	—→	v21
`{this\|cat1}`	—→	label1,label2
`{this\|cat2}`	—→	label21

Extended examples for these simple expressions are given in simple_answerformats1.sml and simple_answerformats2.sml in the directory $CB_HOME/examples/AnswerFormat/.

Note that only objects that match the query constraint are in the answer. Particularly, the categories computed_attribute and retrieved_attribute require that at least one filler for the respective attribute is present in an answer object! Use ConceptBase views (‘View’) and inherited_attribute in case that zero fillers are also allowed for answers.

There are few other simple expressions that may be useful. They just list attributes without having to refer to specific attributes.

{this.attrCategory}: denotes all attribute categories that are present in an answer object. Example: For answer object ‘A’ {this.attrCategory} evaluates to cat1,cat2.
{this∣attribute}: lists all attribute labels occuring in an answer object. Example: for answer object ‘A’, {this|attribute} evaluates to label11,label12,label21.
{this.attribute}: lists all attribute values occuring in an answer object. Example: for answer object ‘A’, {this.attribute} evaluates to v11,v12,v21.
{this.oid}: displays the internal object identifier of the answer object {this}.

For the answer object A, these expressions induce the following additional substitution rules:

`{this.attrCategory}`	—→	cat1,cat2
`{this\|attribute}`	—→	label11,label12,label21
`{this.attribute}`	—→	v11,v12,v21

3.2.2 Pre-defined variables

The following variables can be used in the head, tail, and pattern of an answer format. They do not refer to the variable this.

{user}: outputs the user name of the current transaction; typically has the structure name@address
{transactiontime}: the time when the current transaction was started; has format YYYY-MM-DD hh:mm:ss and is based on Coordinated Universal Time (UTC), formerly known as Greenwhich Mean Time
{cb_version}: the version number of ConceptBase
{cb_date_of_release}: the version number of ConceptBase
{currentmodule}: is expanded to the name of the current module.
{currentpath}: is expanded to the complete module path (starting with root module System) that was active when starting the transaction
{database}: is expanded to relative or absolute path of the database that was specified with the -d option of the CBserver (section 6); if no database was specified, the variable is expanded to <none>

ConceptBase also adds those command line parameters that deviate from their defaults to the set of pre-defined variables. The most common ones are (see also section 5.11):

{loadDir}: : directory from which the CBserver loads Telos source files at start-up; command-line parameter -load
{saveDir}: : directory into which the CBserver saves Telos sources of modules at shut-down or client logout; command-line parameter -save
{viewDir}: : directory into which the CBserver materializes results of certain queries command-line parameter -views

Moreover, if the current transaction was a call of a query like MyQuery[v1/param1,...] then {param1} will be evaluated to v1. This makes all parameter substitutions of a query call available to answer formatting.

3.2.3 Iterations over expressions

In case of expressions with multiple values, the user may want to generate a complex text that uses one value after the other as a parameter. This is in particular useful to transform multiple attribute values like this.cat1. The ‘Foreach’ construct has the format:

{Foreach( (expr1,expr2,...), (x1,x2,...), expr )}

The expression expr1 is evaluated yielding each a list of solutions s11,s12,... The same is applied to expr2 yielding a list s21,s22,... Then, the variables x1,x2,... are matched against the first entries of all lists, i.e. x1=s11,x2=s21,... This binding is then applied to the expression expr which should contain occurences of {x1}, {x2}, ... This replacement is continued with the second entries in all lists yielding bindings x1=s12,x2=s22,... This is continued until all elements of all lists are iterated. If some lists are smaller than others, the missing entries are replaced by NULL.

During each iteration, the new bindings induce substitution rules for the binding

Iteration 1:

`{x1}`	—→	s11
`{x2}`	—→	s21
...

Iteration 2:

`{x1}`	—→	s12
`{x2}`	—→	s22
...

Note that the third argument expr may contain other subexpressions, even a nested ‘Foreach’. An example for iterations is given in $CB_HOME/examples/AnswerFormat/iterations.sml.

The Foreach construct contains three arguments separated by commas. These two commas are used by ConceptBase to parse the arguments of the Foreach-construct and similar answer formatting expressions. They are not printed to the answer stream.

3.2.4 Special characters

If one wants a comma inside an expresssion that shall be visible in the answer, then one has to escape it ‘\,’. The same holds for the other special characters like ‘(’, ‘)’ etc. Here is a short list of supported special characters. Some require a double backslash.

\\n	:	new line (ASCII character 10)
\\t	:	tab character
\\b	:	backspace character
\0	:	empty string (no character)
\(	:	left parenthesis
\)	:	right parenthesis
\,	:	comma

In principal, any non-alphanumerical character like ’(’,’)’,’{’,’}’, ’[’, ’]’ can be referred to by the backslash operator. Note that the vanilla versions of these characters are used to denote expressions in answer formats. Hence, we need to ’escape’ them by the backslash if they shall appear in the answer.

3.2.5 Function patterns

The substitution mechanism for answer formats recognizes patterns such as

{F(expr1,expr2,...)}

as function calls. An example is the ASKquery construct from from section 3.2.6. The mechanism is however very general and can be used to realize almost arbitrary substititions. The parentheses and the commas separating the arguments of F are parsed by ConceptBase and not placed on the output. The following simple function patterns are pre-defined:

{QT(expr)} puts the expr into double quotes if not already quoted.
{UQ(expr)} removes double quotes from expr if present.
{ALPHANUM(expr)} outputs an alphanumeric transcription of expr. This is useful when an object names contains special characters but the output format requires an alphanumeric label. If expr already evaluates to an alphanumeric label, then it is inserted unchanged.
{From(expr)} inserts the source object of expr. The source object is computed by the predicate From(x,o) of section 2.2. This pattern is useful for printing attributes. ConceptBase will first apply pattern substitution to expr and then to the whole term {From(expr)}.
{To(expr)} inserts the destination object of expr. The destination object is computed by the predicate To(x,o) of section 2.2.
{Label(expr)} inserts the label of the object referenced by expr. The label is computed by the predicate Label(x,l) of section 2.2.
{Oid(expr)} inserts the identifier of the object referenced by expr.

Note that the argument expr can be another pattern such as {this}. Specifically, the expression {Oid({this})} is equivalent to {this.oid}. However, the Oid pattern is also applicable to patterns not including {this}.

Examples are available from the CB-Forum, see link.

The above set of patterns can be extended by user-defined functions via LPI plugins. In principle, any routine that can be called from the ConceptBase server, can also be called in an answer format. The programming interface is not documented here since this requires extensive knowledge of the ConceptBase server source code. For the experienced user, we provide an example in the subdirectory examples/AnswerFormat of the ConceptBase installation directory, see files externalcall.sml and externalcall.swi.lpi (externalcall.bim.lpi for BIM-Prolog variant). Note that one has to create a persistent database, load the model externalcall.sml into it, then terminate the ConceptBase server, and then copy the file externalcall.swi.lpi or externalcall.bim.lpi into the database directory (see also appendix F). Thereafter, restart ConceptBase and call the query EmpDept.

3.2.6 Calling queries in answer formats

A query call within an answer format is an example of a so-called external procedure. The pattern as well as head and tail of an answer format may contain the call to a query (possibly the same for which the answer format was defined for). This allows to generate arbitrarily complex answers.

{ASKquery(Q[subst1,subst2,...],formatname)}

The argument Q is the name of a query. The arguments subst1,subst2 are parameter substitutions (see section 2.3). The argument formatname specifies the answer format for the query call Q[subst1,subst2,...]. The answer format may also have parameters (see below). If you use default as answer format name, then the ConceptBase server will pick the default format. This is LABEL (list of object names) for function calls. For other queries, it is the first answer format that list the query Q in its forQuery attribute. If no such answer format exists, the format FRAME (Telos frames) is chosen.

The effect of ASKquery in an answer format is that the above query call is evaluated and the ASKquery expression is replaced by the complete answer to the query. In terms of the substitution, the following rule is applied:

{ASKquery(Q[subst1,subst2,...],default)} —→ X

where X is the result of the query call Q[subst1,subst2,...] after derivation of the arguments subst1, subst2 etc. This sequencing is important since an ASKquery call can contain terms that are subject to substitution, e.g.
{ASKquery(MyQuery[{this.cat1}/param1,{this.{x1}}/{x2}],default)}.
ConceptBase will always start to evaluate left to right and the innermost terms before evaluating the terms that contain inner terms. Hence, the derivation sequence is

	`{ASKquery(MyQuery[{this.cat1}/param1,{this.{x1}}/{x2}],default)}`
=⇒	`{ASKquery(MyQuery[alpha/param1,{this.{x1}}/{x2}],default)}`	(r2)
=⇒	`{ASKquery(MyQuery[alpha/param1,{this.name}/{x2}],default)}`	(r1)
=⇒	`{ASKquery(MyQuery[alpha/param1,"smith"/{x2}],default)}`	(r3)
=⇒	`{ASKquery(MyQuery[alpha/param1,"smith"/param2],default)}`	(r4)
=⇒	`The answer is ...`	(r5)

where we assume the following example substitution rules

r1:	`{x1}`	—→	name
r2:	`{this.cat1}`	—→	alpha
r3:	`{this.name}`	—→	`"smith"`
r4:	`{x2}`	—→	param2
r5:	`{ASKquery(MyQuery[...],default)}`	—→	The answer is ...

This guarantees that the query call is ‘ground’, i.e. does not contains terms which are subject to substitution.

The ASKquery construct allows to introduce recursive calls during the derivation of a query since there can (and should) be an answer format for Q which may contain expressions ASKquery itself. In principle, this allows infinite looping. However, the answer format evaluator prevents such loops by halting the expansion when a recursive call with same parameters has occured. The answer then contains an ‘' character at the position where the loop was detected. Additionally, an error message is written on the console window of the ConceptBase server (tracemode must be at least low).

A simple example for use of ASKquery is given in recursive-answers.sml. The example uses a view instead of a query class in order to include also answers into the solution which not have a filler for the requested attribute, i.e. hasChild is inherited_attribute, not retrieved_attribute.

It is common practice to combine the ASKquery construct with ‘Foreach’ in order to display an iteration of objects in the same way. The user should define an answer format for the iterated query Q as well.

Do not mix the use of ASKquery with the view definitions in ConceptBase! The nesting depth of a view is determined by the view definition. The nesting depth of an answer generated by expansion of ASKquery is only limited by the complexity of the database. For example, one can set up an ancestor database and display all descendants of a person and recursivley their descendants in a single answer string for that person. The nested ASKquery inside an answer format usually results in the unfolding also using an answer format (possibly the same as used for the original query). This feature allows the user to specify very complex structured answers that might even contain the complete database. In particular, complex XML representations can be constructed in this way.

3.2.7 Expressions in head and tail

The features ‘head’ and ‘tail’ are similar to pattern. The difference is that any expression using ‘this’ (the running variable for answer objects) is disallowed. This only leaves function patterns such as ASKquery expressions and pre-defined patterns such as {user}. Of course, the head and tail strings can contain multiple occurences of ASKquery or other function patterns.

3.2.8 Conditional expressions

Conditional expressions allow to expand a substring based on the evaluation of a condition. The syntax is:

{IFTHENELSE(predicate,thenstring,elsestring)}

The ‘predicate’ can be one of

{GREATER(expr1,expr2)}
{LOWER(expr1,expr2)}
{EQUAL(expr1,expr2)}
{AND(expr1,expr2)}
{OR(expr1,expr2)}
{ISFIRSTFRAME()}
{ISLASTFRAME()}

Example: {GREATER({this.salary},10000)}. Note that the arguments may also contain expressions. The predicate {ISFIRSTFRAME()} is true, when ConceptBase starts with processing answer frames. It is false, when the first frame has been processed. The predicate {ISlASTFRAME()} is true when ConceptBase starts with processing the last answer frame for a given query. Otherwise, it is false. An example for conditional expressions is provided in the CB-Forum, see file "csv.sml" in link. In most cases, the IFTHENELSE construct can be avoided by a more elegant query class formulation.

3.2.9 Views and path expressions

If the answer format is defined for a complex view, then path expressions like this.cat2.cat21... for the parts of the complex answer can be defined. An example for use of answer formats for views is given in views.sml.

The reader should note that complex path expressions can only refer to components that were defined as retrieved, computed, or inherited attributes in the view definition. For example, one cannot refer to this.dept.budget in the example view EmpDept in views.sml since it is not a retrieved attribute of the dept component of the view definition EmpDept. The second expression of the answer format EmpDeptFormat uses the builtin procedure UQ. It removes the quotes ‘"’ from a string. Analogously, a procedure QT can be used to put quotes around a term.

3.3 Parameterized answer formats

The general way to use an answer format for a query is to define the attribute forQuery. Another possibility is to specify the answer format for a query is to use the answer representation field of the ASK method in the IPC interface.

The following code is an example for specifying a user-defined answer in the ASK method. This example is written in Java and uses the standard Java API of ConceptBase (see the Programmer’s Manual for details).

import i5.cb.api.*;

public class CBAnswerFormat {

    public static void main(String[] argv) throws Exception {

        CBclient cb=new CBclient("localhost",4001,null,null);

        CBanswer ans=cb.ask("find_specializations[Class/class,TRUE/ded]",
                        "OBJNAMES","AFParameter[bla/somevar]","Now");
        System.out.println(ans.getResult());
        cb.cancelMe();
    }
}

In the example, a connection is made to a ConceptBase server on localhost listening on port 4001. The ask-method of the CBclient class sends a query to the server. The first argument is the query, the second argument is the format of the query (in this example, it is just one object name), the third argument is the answer representation, and the last argument is the rollback time.

The third argument, is the the answer representation. There are four predefined answer representations. FRAME returns the answers as Telos frames, including retrieved and computed attributes. LABEL returns only the names of the answer objects as a comma-separated list. Thirdly, the format JSONIC returns the answer in JSON-like frames. Finally, default lets the CBserver choose between LABEL (for function calls), the explicit answer format assigned for a query (attribute forQuery), and FRAME (otherwise). Besides these pre-defined answer representations, one can specify user-defined answer formats. This is also the preferred way. In our case, it is a parameterized answer format: AFParameter[somevalue/somevar]. This means that the result of the query will be formatted according to the answer format AFParameter and the variable somevar will be replaced with somevalue. The variable can be used like any other expression, i.e. it must be enclosed in {}.

The following definition of AFParameter is an example, how the parameter can be used in the pattern. If the parameter is not specified, the string {somevar} will not be replaced.

Individual AFParameter in AnswerFormat with
  head hd : "<result>"
  tail tl : "</result>"
  pattern
     p : "
<object>
  <type>{somevar}</type>
  <name>{this}</name>
</object>"
end

Note, that you can use any answer format (with or without parameters) as answer representation in the ASK method.

3.4 File type of answer formats

The optional fileType attribute of answer formats is used by the server-side materialization of query results (section 5.11). ConceptBase will use the specified file type when storing the query results in the file system. The default value is "txt". The attribute is single-valued though single-valuedness is not enforced.

3.5 Bulk query calls

It is sometimes useful to call the same query class with multiple arguments in a single call rather than in a sequence of calls. Each individual call from a ConceptBase client to the server comes with a certain latency time. Thus, if one would have to call the same query for dozens of arguments in a sequence, most of the answer time would actually be the latency time.

To address this problem, the CBserver offers a query call pattern for bulk queries:

bulk[q,x1,x2,x3,...]

The query q stands for a query class with a single parameter. The bulk query is converted by the CBserver into the following sequence of query calls:

q[x1],q[x2],q[x3],...

The answers to the query call are collected into a single answer set and then transformed with the answer format of the query call. Hence, the answer format is applied to the whole answer and not to the part answers.

Example:

   ask bulk[Q,abc,def] OBJNAMES MyFormatA Now

Sorting of answers is disabled for bulk queries in order to return the answers in the sequence indicated by the arguments of the bulk query call. Here, the answer to the argument abc shall precede the answer to argument def. Arguments that do not reference an existing object are removed from the argument list by the CBserver before answering the query. Bulk queries are only supported for generic query classes with a single parameter. The query class may not be a builtin query class.

The main prupose of bulk queries is to speed up the interaction between the ConceptBase clients, such as CBGraph, and the CBserver. You can however use them with the CBShell.