Log Standard Implementation of DMA Metrics

Contents

• Introduction
• DMA Metrics
• Task Completion
• Time to Completion
• Turns to Completion
• Mean User Words/Turn
• Mean System Words/Turn
• Mean User Concepts/Turn
• Mean Concept Efficiency
• State of Itinerary
• Error Messages
• Help Messages
• Response Latency
• User Words to Completion
• System Words to Completion
• User Repeats
• System Repeats/Reprompts
• Word Error
• Mean Length of System Utterance
• Mean Length of System Turn
• Summary of Attributes and Metrics

Introduction

---

DMA Metrics

Task Completion

• Requires human annotation
• Human annotations are stored in a separate XML file (to facilitate multiple annotations and measurement of interannotator agreement)
• Example:

<GC_LOG_ANNOTATIONS>
   <GC_SESSION  id="199.94.106.6:20300:0" stime="941473394.650000" etime="941473494.820000">
      <GC_ANNOT task_completion="1"/>
       ...
   </GC_SESSION>
</GC_LOG_ANNOTATIONS>

Time to Completion

• Relies on two tag attributes:
• the type_start_task attribute has a value of either total or task
• the total value is used when time to completion indicates total call time (attribute placed on a tag at the start of the first system utterance)
• the task value is used when time to completion indicates on-task time (attribute placed on a tag at the start of the user's first utterance)
• the type_end_task attribute has a value of true
• The attributes may be placed on GC_OPERATION, GC_MESSAGE, or GC_EVENT tags
• Example:
• Start:

<GC_OPERATION type_start_task="task" name="nop" server="Builtin" location="Builtin:-1" turnid="0" stime="941473406.850000" etime="941473406.850000" tidx="13">
   <GC_DATA key=":listening_has_begun" dtype="unknown"/>
</GC_OPERATION>
• End:

<GC_OPERATION type_end_task="true" name="nop" server="Builtin" location="Builtin:-1" turnid="5" stime="941473494.520000" etime="941473494.520000" tidx="315">
   <GC_DATA key=":playing_has_ended" dtype="unknown"/>
</GC_OPERATION>
• Time to completion is calculated by subtracting the value of stime of the tag with a type_start_task attribute from the value of etime of the tag with a type_end_task attribute

Turns to Completion

• Relies on type_new_turn attributes which have values of user or system
• The attributes may be placed on GC_OPERATION, GC_MESSAGE, or GC_EVENT tags
• Examples:
• System:

<GC_OPERATION type_new_turn="system" name="speak_output" server="tts" location="localhost:15020" turnid="-1" stime="941473394.700000" etime="941473394.700000" tidx="3">
   <GC_DATA type_utt_text="system" key=":reply_string" dtype="string">
      Hi! Welcome to Mitre's Travel demonstration. This call is being recorded for system development. You may hang up or ask for help at any time. How can I help you?
   </GC_DATA>
</GC_OPERATION>
• User:

<GC_OPERATION type_new_turn="user" name="create_frame" server="nl" location="localhost:11000" turnid="0" stime="941473415.560000" etime="941473415.620000" tidx="50">
   <GC_DATA type_utt_text="user" key=":input_string" dtype="string">
      i would like to fly to newark
   </GC_DATA>
   ...
</GC_OPERATION>
• After collapsing all consecutive "sub-turns" from the same side of the dialogue into single turns (as outlined in the DMA specification), turns to completion may be calculated by simply counting

Mean User Words/Turn

• Requires human annotations (transcription), which are stored in a separate XML file external to the main logfile
• Relies on type_utt_text attribute with a value of transcription
• The type_utt_text attribute is placed on GC_DATA tags
• Example:

<GC_DATA type_utt_text="transcription" dtype="string">
   I want a flight to Newark
</GC_DATA>
• Count the words in each element with a type_utt_text="transcription" attribute, sum them all and divide by the number of user turns

Mean System Words/Turn

• Relies on type_utt_text attribute with a value of system
• The type_utt_text attribute is placed on GC_DATA tags
• Example:

<GC_DATA type_utt_text="system" key=":reply_string" dtype="string">
   U S Airways flight 1820 departs Philadelphia at three thirty p m  and arrives Newark at four thirty p m . What else would you like to know?
</GC_DATA>
• Count the words in each element with a type_utt_text="system" attribute, sum them all and divide by the number of system turns

Mean User Concepts/Turn

• Deferred until concept labeling is defined

Mean Concept Efficiency

• Deferred until concept labeling is defined

State of Itinerary

• Deferred

Error Messages

• Relies on type_error_msg attribute (with a value of true)
• The type_error_msg attribute is placed on GC_DATA tags that also have type_utt_text="system" attributes
• Example:

<GC_DATA type_utt_text="system" type_error_msg="true" key=":reply_string" dtype="string">
   I was unable to parse your utterance.
</GC_DATA>

Help Messages

• Relies on type_help_msg attribute (with a value of true)
• The type_help_msg attribute is placed on GC_DATA tags that also have type_utt_text="system" attributes
• Example:

<GC_DATA type_utt_text="system" type_help_msg="true" key=":reply_string" dtype="string">
   I can provide worldwide travel information.
</GC_DATA>

Response Latency

• Relies on type_start_utt and type_end_utt attributes with values of user or system
• End of user utterances are indicated by a time stamped tag with a type_end_utt="user" attribute
• Start of system utterances are indicated by a time stamped tag with a type_start_utt="system" attribute
• The attributes may be placed on GC_OPERATION, GC_MESSAGE or GC_EVENT tags
• Example:
End of user utterance:
<GC_OPERATION type_end_utt="user" name="nop" server="Builtin" location="Builtin:-1" turnid="1" stime="941473421.440000" etime="941473421.440000" tidx="78">
   <GC_DATA key=":recording_has_ended" dtype="unknown"/>
</GC_OPERATION>
Start of system utterance:
<GC_OPERATION type_start_utt="system" name="nop" server="Builtin" location="Builtin:-1" turnid="1" stime="941473425.140000" etime="941473425.140000" tidx="91">
   <GC_DATA key=":playing_has_begun" dtype="unknown"/>
</GC_OPERATION>
• Note that the type_end_utt="user" and type_start_utt="system" type attributes will likely be on tags other than those in the example above (for better accuracy, the audio server could issue messages indicating the utterance boundaries, in which case the attributes would appear on <GC_MESSAGE> tags)
• Response latency is calculated by subtracting value of the etime attribute of the tag with type_end_utt="user" from the value of the stime attribute of the tag with the type_start_utt="system" attribute

User Words to Completion

• Requires human annotations (transcription), which are stored in a separate XML file external to the main logfile
• Relies on several attributes
• type_utt_text attributes with a values of transcription are placed on GC_DATA tags
• the type_start_task attribute has a value of either total or task
• the total value is used when time to completion indicates total call time (attribute placed on a tag at the start of the first system utterance)
• the task value is used when time to completion indicates on-task time (attribute placed on a tag at the start of the user's first utterance)
• the type_end_task attribute has a value of true
• Example:

<GC_DATA type_utt_text="transcription" dtype="string">
   I want a flight to Newark
</GC_DATA>
• Collect all of the <GC_DATA type_utt_text="transcription"> elements for utterances that exist between the tag with the type_start_task attribute (be it a <GC_OPERATION> or <GC_MESSAGE>) and the tag with the type_end_task attribute
• Count the words in each of these elements to calculate user words to completion

System Words to Completion

• Relies on type_start_task and type_end_task attributes
• Collect all of the <GC_DATA type_utt_text="system" key=":reply_string" dtype="string"> elements that exist between the tag with the type_start_task attribute (be it a <GC_OPERATION> or <GC_MESSAGE>) and the tag with the type_end_task attribute
• Example:

<GC_DATA type_utt_text="system" key=":reply_string" dtype="string">
   U S Airways flight 1820 departs Philadelphia at three thirty p m  and arrives Newark at four thirty p m . What else would you like to know?
</GC_DATA>
• Count the words in each of these elements to calculate system words to completion

User Repeats

• Deferred until concept labeling is defined

System Repeats/Reprompts

• Relies on type_prompt attributes
• Example:

<GC_OPERATION type_prompt="leg1-dep_city" name="speak_output" server="tts" location="localhost:15020" turnid="0" stime="941473413.840000" etime="941473413.840000" tidx="34">
   <GC_DATA key=":reply_string" dtype="string">
      What city does the flight depart from?
   </GC_DATA>
</GC_OPERATION>
• A postprocess will look for consecutive instances of the same value for the type_prompt attribute

Word Error

• Requires human annotations (transcription), which are stored in a separate XML file external to the main logfile
• Relies on type_utt_text attributes with values of transcription or asr
• Produce transcriptions as <GC_DATA> tags with type_utt_text="transcription" attributes
• Speech recognition hypotheses are repesented as <GC_DATA> tags with type_utt_text="asr" attributes
• Example:
• Hypotheses:

<GC_DATA type_utt_text="asr" dtype="string">
   I want any flight to Newark
</GC_DATA>
• Transcription:

<GC_DATA type_utt_text="transcription" dtype="string">
   I want a flight to Newark
</GC_DATA>
• The standard NIST metric is used to calculate word error rate

Mean Length of System Utterance

• Relies on type_start_utt="system" and type_end_utt="system"
• Example:
Start:
<GC_OPERATION type_start_utt="system" name="nop" server="Builtin" location="Builtin:-1" turnid="-1" stime="941473396.190000" etime="941473396.190000" tidx="5">
   <GC_DATA key=":playing_has_begun" dtype="unknown"/>
</GC_OPERATION>
End:
<GC_OPERATION type_end_utt="system" name="nop" server="Builtin" location="Builtin:-1" turnid="-1" stime="941473406.570000" etime="941473406.570000" tidx="11">
   <GC_DATA key=":playing_has_ended" dtype="unknown"/>
</GC_OPERATION>
• Tags with type_start_utt="system" are paired with the corresponding tag with type_end_utt="system" in the same utterance
• Length of an individual system utterance is calculated by subtracting the value of the stime attribute of the tag with type_start_utt="system" from the value of the etime attribute of the tag with type_end_utt="system"
• Mean length of system utterance is calculated by summing the individual utterance lengths and dividing by the number of system utterances

Mean Length of System Turn

• Like Mean Length of System Utterance, this metric relies on time stamped tags with type_start_utt="system" and type_end_utt="system" attributes
• For each system turn, the first tag with type_start_utt="system" is paired with the last tag with type_end_utt="system" in the same turn
• Length of an individual system turn is calculated by subtracting the value of the stime attribute of the tag with type_start_utt="system" from the value of the etime attribute of the tag with type_end_utt="system"
• Mean length of system turn is calculated by summing the individual turn lengths and dividing by the number of system turns

---

Summary of Attributes and Metrics

• The Attribute column names the attribute
• The Values column lists the possible values of the attribute
• The Metrics column lists the metrics that use the attribute (with the appropriate attribute values in parentheses)
• The Tags column lists the tags on which the attribute may appear