Spoken Dialogue Systems CS 4705

Spoken Dialogue Systems CS 4705

Boston Directions Corpus Erratum from last week: speakers did give directions to a silent confederate, a Harvard student

Talking to a Machine .and (often) Getting an Answer Today’s spoken dialogue systems make it possible to accomplish real tasks without talking to a person – Could Eliza do this? – What do today’s systems do better? – Do they actually embody human intelligence? Key advances – – – – Stick to goal-directed interactions in a limited domain Prime users to adopt the vocabulary you can recognize Partition the interaction into manageable stages Judicious use of system vs. mixed initiative

Today Utterances Turn-taking Initiative Strategies Grounding Confirmation Strategies The Waxholm Project: Word and Topic Prediction Evaluating Spoken Dialogue Systems Predicting System Errors and User Corrections More Examples

Dialogue vs. Monologue Monologue and dialogue both involve interpreting – – – – Information status Coherence issues Reference resolution Speech acts, implicature, intentionality Dialogue involves managing – Turn-taking – Grounding and repairing misunderstandings – Initiative and confirmation strategies

Segmenting Speech into Utterances What is an utterance’? – Why is EOU detection harder than EOS? – How does speech differ from text? – Single syntactic sentence may span several turns A: We've got you on USAir flight 99 B: Yep A: leaving on December 1. – Multiple syntactic sentences may occur in single turn A: We've got you on USAir flight 99 leaving on December. Do you need a rental car? – Intonational definitions: intonational phrase, breath group, intonation unit

Turns and Utterances Dialogue is characterized by turn-taking: who should talk next, and when they should talk How do we identify turns in recorded speech? – Little speaker overlap (around 5% in English --although depends on domain) – But little silence between turns either How do we know when a speaker is giving up or taking a turn? Holding the floor? How do we know when a speaker is interruptable?

Simplified Turn-Taking Rule (Sacks et al) At each transition-relevance place (TRP) of each turn: – If current speaker has selected A as next speaker, then A must speak next – If current speaker does not select next speaker, any other speaker may take next turn – If no one else takes next turn, the current speaker may take next turn TRPs are where the structure of the language allows speaker shifts to occur

Adjacency pairs set up next speaker expectations – – – – GREETING/GREETING QUESTION/ANSWER COMPLIMENT/DOWNPLAYER REQUEST/GRANT ‘Significant silence’ is dispreferred A: Is there something bothering you or not? (1.0s) A: Yes or no? (1.5s) A: Eh? B: No.

Intonational Cues to Turntaking Continuation rise (L-H%) holds the floor H-H% requests a response – L*H-H% (ynq contour) – H* H-H% (highrise question contour) Intonational contours signal dialogue acts in adjacency pairs

Timing and Turntaking How should we time responses in a SDS? – Japanese studies of aizuchi (backchannels) (Koiso et al ‘98, Takeuchi et al ‘02) in natural speech – Lexical information: particles ne and ka ending preceding turn or (in telephone shopping) product names – Length of preceding utterance, f0, loudness, and pause after even more important in predicting turntaking

Turntaking and Initiative Strategies System Initiative S: Please give me your arrival city name. U: Baltimore. S: Please give me your departure city name . User Initiative S: How may I help you? U: I want to go from Boston to Baltimore on November 8. Mixed’ initiative S: How may I help you? U: I want to go to Boston. S: What day do you want to go to Boston?

Grounding (Clark & Shaefer ‘89) Conversational participants don’t just take turns speaking .they try to establish common ground (or mutual belief) H must ground a S's utterances by making it clear whether or not understanding has occurred How do hearers do this? S: I can upgrade you to an SUV at that rate. – Continued attention (U gazes appreciatively at S) – Relevant next contribution U: Do you have a RAV4 available?

– Acknowledgement/backchannel U: Ok/Mhmmm/Great! – Demonstration/paraphrase U: An SUV. – Display/repetition U: You can upgrade me to an SUV at the same rate? – Request for repair U: I beg your pardon?

Detecting Grounding Behavior Evidence of system misconceptions reflected in user responses (Krahmer et al ‘99, ‘00) – Responses to incorrect verifications contain more words (or are empty) show marked word order (especially after implicit verifications) contain more disconfirmations, more repeated/corrected info – ‘No’ after incorrect verifications vs. other ynq’s has higher boundary tone wider pitch range longer duration longer pauses before and after more additional words after it

User information state reflected in response (Shimojima et al ’99, ‘01) – Echoic responses repeat prior information – as acknowledgment or request for confirmation S1: Then go to Keage station. S2: Keage. – Experiment: Identify ‘degree of integration’ and prosodic features (boundary tone, pitch range, tempo, initial pause) Perception studies to elicit ‘integration’ effect – Results: fast tempo, little pause and low pitch signal high integration

Grounding and Confirmation Strategies U: I want to go to Baltimore. Explicit S: Did you say you want to go to Baltimore? Implicit S: Baltimore. (H* L- L%) S: Baltimore? (L* H- H%) S: What time do you want to leave Baltimore? No confirmation

The Waxholm Project at KTH tourist information Stockholm archipelago time-tables, hotels, hostels, camping and dining possibilities. mixed initiative dialogue speech recognition multimodal synthesis graphic information pictures, maps, charts and time-tables Demos at http://www.speech.kth.se/multimodal

Dialogue control state prediction Dialog grammar specified by a number of states Each state associated with an action database search, system question Probable state determined from semantic features Transition probability from one state to state Dialog control design tool with a graphic interface

Waxholm Topics TIME TABLE Task: get a time-table. Example: När går båten? (When does the boat leave?) SHOW MAP Task : get a chart or a map displayed. Example: Var ligger Vaxholm? (Where is Vaxholm located?) EXIST Task : display lodging and dining possibilities. Example: Var finns det vandrarhem? (Where are there hostels?) OUT OF DOMAIN Task : the subject is out of the domain. Example: Kan jag boka rum. (Can I book a room?) NO UNDERSTANDING Task : no understanding of user intentions. Example: Jag heter Olle. (My name is Olle) END SCENARIO Task : end a dialog. Example: Tack. (Thank you.)

Topic selection FEATURES TOPIC EXAMPLES TIME TABLE SHOW MAP FACILITY NO UNDEROUT OF STANDING DOMAIN OBJECT QUEST-WHEN QUEST-WHERE FROM-PLACE AT-PLACE .062 .188 .062 .250 .062 .312 .031 .688 .031 .219 .073 .024 .390 .024 .293 .091 .091 .091 .091 .091 .067 .067 .067 .067 .067 TIME.312 PLACE OOD.062 END.062 HOTEL HOSTEL ISLAND PORT.125 MOVE .031 .091 .031 .031 .062 .062 .333 .750 .875 .024 .200 .122 .024 .031 .031 .556 .244 .031 .091 .500 .091 .091 .488 .122 .062 .091 .098 .067 .091 .933 .067 .091 .091 .091 .067 .091 .091 .067 .091 .909 .067 .067 .067 .091 .067 { p(ti F )} argmax i END .091 .091 .091 .091 .091 .091 .091 .091 .091 .091

Topic prediction results % Errors 15 12,9 8,8 10 5 12,7 8,5 “no understanding” excluded 3,1 2,9 0 complete parse All raw data no extra linguistic sounds

User answers to questions? The answers to the question: “What weekday do you want to go?” (Vilken veckodag vill du åka?) 22% 11% 11% 7% 6% - Friday (fredag) I want to go on Friday (jag vill åka på fredag) I want to go today (jag vill åka idag) on Friday (på fredag) I want to go a Friday (jag vill åka en fredag) are there any hotels in Vaxholm? (finns det några hotell i Vaxholm)

User answers to questions? The answers to the question: “What weekday do you want to go?” (Vilken veckodag vill du åka?) 22% 11% 11% 7% 6% - Friday (fredag) I want to go on Friday (jag vill åka på fredag) I want to go today (jag vill åka idag) on Friday (på fredag) I want to go a Friday (jag vill åka en fredag) are there any hotels in Vaxholm? (finns det några hotell i Vaxholm)

Examples of questions and answers Hur ofta åker du utomlands på semestern? jag åker en gång om året kanske Hur ofta reser du utomlands på semestern? jag reser en gång om året utomlands jag reser inte ofta utomlands på semester det blir mera i jag åker ganska sällan utomlands på semester jag åker nästan alltid utomlands under min semesterarbetet jag reser reser utomlands på semestern vartannat år jag åker ungefär 2 gånger per år utomlands på semester jag reser utomlands en gång per semester jag reser utomlands på semester ungefär en gång per år jag åker utomlands nästan varje år jag åker utomlands på semestern varje år jag brukar resa utomlands på semestern åtminståne en gång i året en gång per år kanske jag åker utomlands ungefär en gång om året jag är nästan aldrig utomlands en eller två gånger om året en gång per semester kanske en gång per år ungefär en gång per år åtminståne en gång om året nästan aldrig en gång vart annat år varje år vart tredje år ungefär nu för tiden inte så ofta varje år brukar jag åka utomlands

Results no no reuse answer 4% 2% other 24% reuse 52% 18% ellipse

The Waxholm system There Information Information Which areWhen IIs Ilots Waxholm am Which think day it This Where looking possible of Ido about of want Where about boats Iishotels the want the Thank a is can Thank table hotels to the evening The for week the from to shown isto go are Ieat restaurants boats Waxholm? hotels you find city go of tomorrow you is Stockholm do in in the to on shown boats too Waxholm? hotels? you Waxholm? toWaxholm in boats. this Waxholm Waxholm want depart? in map inWaxholm to this toWaxholm go? table is on a Friday, FromisAt shown where shown whatin do time inthis you this do table want table youto want go to go?

How do we evaluate Dialogue Systems? PARADISE framework (Walker et al ’00) “Performance” of a dialogue system is affected both by what gets accomplished by the user and the dialogue agent and how it gets accomplished Maximize Task Success Minimize Costs Efficiency Measures Qualitative Measures

What metrics should we use? Efficiency of the Interaction:User Turns, System Turns, Elapsed Time Quality of the Interaction: ASR rejections, Time Out Prompts, Help Requests, Barge-Ins, Mean Recognition Score (concept accuracy), Cancellation Requests User Satisfaction Task Success: perceived completion, information extracted 05/13/24 29

User Satisfaction: Sum of Many Measures Was Annie easy to understand in this conversation? (TTS Performance) In this conversation, did Annie understand what you said? (ASR Performance) In this conversation, was it easy to find the message you wanted? (Task Ease) Was the pace of interaction with Annie appropriate in this conversation? (Interaction Pace) In this conversation, did you know what you could say at each point of the dialog? 05/13/24 (User Expertise) How often was Annie sluggish and slow to reply to you in this conversation? (System Response) Did Annie work the way you expected her to in this conversation? (Expected Behavior) From your current experience with using Annie to get your email, do you think you'd use Annie regularly to access your mail when you are away from your desk? (Future Use) 30

Performance Model Weights trained for each independent factor via multiple regression modeling: how much does each contribute to User Satisfaction? Result useful for system development – Making predictions about system modifications – Distinguishing ‘good’ dialogues from ‘bad’ dialogues But can we also tell on-line when a dialogue is ‘going wrong’

Identifying Misrecognitions, Awares and User Corrections Automatically (Hirschberg, Litman & Swerts) Collect corpus from interactive voice response system Identify speaker ‘turns’ incorrectly recognized where speakers first aware of error that correct misrecognitions Identify prosodic features of turns in each category and compare to other turns Use Machine Learning techniques to train a classifier to make these distinctions automatically

Turn Types TOOT: Hi. This is AT&T Amtrak Schedule System. This is TOOT. How may I help you? User: Hello. I would like trains from Philadelphia to New York leaving on Sunday at ten thirty in the evening. misrecognition TOOT: Which city do you want to go to? User: New York. correction aware site

Results Reduced error in predicting misrecognized turns to 8.64% Error in predicting ‘awares’ (12%) Error in predicting corrections (18-21%)

Percentage of all repetitions Features in repetition corrections (KTH) 50 40 adults children 30 20 10 0 more increased shifting of clearly focus loudness articulated

The August system People IIYes, Icall Over can The How come Strindberg The Perhaps myself answer that information who amany Royal from million was live we Strindberg, questions people was the Institute ain will people smart department married glass is meet live shown thing about of houses live but in I Yes, What Strindberg When it What Do You do might you Thank you Good were are iswas like your be do welcome! bye! you you! that born for itname? here? born? we ainliving? 1849 will! Strindberg, ofdon’t Speech, should in the really Technology! three Stockholm? on soon not Stockholm KTH Music tothe have throw say! times! again! map and and a stones surname Stockholm area Hearing

Initial experiments Studies on human-human conversation The Higgins domain (similar to Map Task) Using ASR in one direction to elicit error handling behaviour User Speaks ASR Listens Vocoder Reads Speaks Operator

Non-Understanding Error Recovery (Skantze ’03) Humans tend not to signal non-understanding: – O: – U: – O: Do you see a wooden house in front of you? ASR: YES CROSSING ADDRESS NOW (I pass the wooden house now) Can you see a restaurant sign? This leads to – Increased experience of task success – Faster recovery from non-understanding

Conclusions Spoken dialogue systems presents new problems -- but also new possibilities – Recognizing speech introduces a new source of errors – Additional information provided in the speech stream offers new information about users’ intended meanings, emotional state (grounding of information, speech acts, reaction to system errors) Why spoken dialogue systems rather than webbased interfaces?