Speech Synthesis & Speech Recognition: Using SAPI 5.1
This article looks at adding support for speech capabilities to Microsoft Windows applications written in Delphi, using the Microsoft Speech API version 5.1 (SAPI 5.1). For an overview on the subject of speech technology please click here.
There is also coverage on using SAPI 4 to build speech-enabled applications. Information on using the SAPI 4 high level interfaces can be found by clicking here, whilst discussion of the low level interfaces can be found by clicking here.
SAPI 5.1 exposes most of the important interfaces, types and constants through a registered type library (SAPI 5.0 did not do this, making it difficult to use in Delphi without someone writing the equivalent of the JEDI import unit for SAPI 5). This means that you can access SAPI 5.1 functionality through late bound or early bound Automation. We will focus our attention on early bound Automation, which requires you to import the type library.
Choose Project | Import Type Library... and locate the type library described as Microsoft Speech Object Library (Version 5.1) in the list. Now ensure the Generate Component Wrapper checkbox is checked so the type library import unit will include component wrapper classes for each exposed Automation object. These components will go on theActiveX page of the Component Palette by default, but you may wish to specify a more appropriate page, such as SAPI 5.1.
Now press Install... so the type library will be imported and the generated components will be installed onto the Component Palette (pressing Create Unit would also generate the type library import unit, but would require us to install it manually).
The generated import unit is called SpeechLib_TLB.pas and will be installed in a package. You can either select the default package offered (the Borland User Components package by default), choose to open a different package or even create a new one. When the package is compiled and installed you will get a whopping set of 19 new components on the SAPI 5.1page of the Component Palette.
Each component is named after the primary interface it implements. So for example, the TSpVoice component implements the SpVoice interface. You can find abundant documentation on all these interfaces in the SAPI 5.1 SDK documentation.
Ready made SAPI 5.1 packages containing Automation components for Delphi 5, 6 and 7 can be found in appropriately named subdirectories under SAPI 5.1 in the accompanying files.
That's the simple example out of the way, but what can we achieve if we dig a little deeper and get our hands a little dirtier? The next project, which holds the answers to these questions, can be found as TextToSpeech.dpr in this article's files. You can see it running in the screenshot below; notice that as the text is spoken, the current sentence is italicised and the current word is displayed selected and also the phonemes spoken are written to a memo.
The following sections describe the important parts of the code from this project.
In order to keep track of the voices, these ISpeechObjectToken interfaces are added, along with a description, to the combobox's Items (the description in the Strings array and the interfaces in the Objects array).
Storing an interface reference in an object reference is possible as long as we remember exactly what we stored, and we don't make the mistake of accessing it as an object reference. Also, since the interface reference is stored using an inappropriate type, it is important to manually increment its reference count to stop it being destroyed when the RTL code decrements the reference count at the end of the method.
The OnDestroy event handler tidies up these descriptor objects by decrementing their reference counts, thereby allowing them to be destroyed.
Additionally (among others) an OnAudioLevel event allows a progress bar to be used as a VU meter for the spoken text. However it is important to note that for some events to fire you must set the EventInterests property accordingly; to receive the OnAudioLevel event you should set EventInterests to SVEAudioLevel or SVEAllEvents.
If you have some artistic flair and can draw a mouth in each position represented by the visemes you could use this event to provide a simple animated representation of speech.
The SAPI 5.1 SDK comes with a C++ example called TTSApp, which displays an animated cartoon microphone whose mouth is drawn to represent each viseme. The microphone is made up from a number of separate images that can all be loaded into an image list. The additional demo program TextToSpeechAnimated.dpr makes use of these images to show how the effect can be achieved.
You can take this idea further if you need, by using images of a person's face saying each of the 22 visemes (for real people it seems to work best if you use 22 images, rather than 13). This way you can animate a real person's face in sync with the spoken text quite trivially.
A statement such as:
When you need SR you can either use a shared recogniser (TSpSharedRecognizer) or an in-process recogniser (TSpInprocRecognizer). The in-process recogniser is more efficient (it resides in your process address space) but means that no other SR applications can receive input from the microphone until it is closed down. On the other hand the shared recogniser can be used by multiple applications, and each one can access the microphone. It is more common to use the shared recogniser in typical SAPI applications.
The recogniser uses the notion of a recognition context to identify when it will be active (not to be confused with the use of context in a context-free grammar or CFG). A context is represented by the SpInprocRecoContext or SpSharedRecoContext interfaces. An application may use one context for each form that will use SR, or several contexts for different application modes (Office XP has a dictation mode for adding text to a document and a control mode for executing menu commands).
Recognition contexts enable you to start and stop recognition, set up the grammar and receive important recognition notifications.
In the case of dictation, a grammar can be used to indicate some words that are likely to be spoken. It is not feasible to try and represent the entire spoken English language as a grammar so the recogniser uses its own rules and context analysis, with any help from a grammar you might supply.
With Command and Control, the words that are understood are limited to the supported commands defined in the grammar. The grammar defines various rules that dictate what will be said and this makes the recogniser's job much easier. Rather than trying to understand anything spoken, it only needs to recognise speech that follows the supplied rules. A simple grammar that recognises three colours might look like this:
Grammars support lists to make implementing many similar commands easy and also support optional sections. For example this grammar will recognise any of the following:
The main event is OnRecognition, which is called when the SR engine has decided what has been spoken. Whilst working it out, it will likely call the OnHypothesis event several times. Finished phrases are added to a memo on the form and whilst a phrase is being worked out the hypotheses are added to a list box so you can see how the SR engine made its decision. Each time a new phrase is started, the hypothesis list is cleared.
You can see the list of hypotheses building up in this screenshot of the program running.
Both OnRecognition and OnHypothesis are passed a Result parameter; this is an ISpeechRecoResult results object. In Delphi 7 this is declared using the correctISpeechRecoResult interface type, but in earlier versions this was just declared as an OleVariant (which contained the ISpeechRecoResult interface).
This code can be used in Delphi 6 and earlier to access the text that was recognised:
The colour rule defines one of three colours that can be spoken, each of which has a value defined for it. Again, this value will be accessible thanks to the list being defined as a property (colourvalue).
This grammar is stored in an XML file and loaded in the OnCreate event handler.
The OnRecognition event handler does the work of locating the chosencolour property and then finding the nested colourvalue property. Its value is used to change the form colour at the user's request, for example with phrases such as:
There is much to Speech API that we have not looked at in this paper but hopefully the areas covered will be enough to whet your appetite and get you exploring further on your own.
Besides authoring a Borland Pascal problem-solving book published in 1994, Brian is a regular columnist in The Delphi Magazine and has had numerous articles published in Developer's Review, Computing, Delphi Developer's Journal and EXE Magazine. He was nominated for the Spirit of Delphi 2000 award and was voted Best Speaker at Borland's BorCon 2002 conference in Anaheim, California by the conference delegates.
There are a growing number of conference papers and articles available on Brian's Web site, so feel free to have a browse.
In his spare time (and waiting for his C++ programs to compile) Brian has learnt the art of juggling and making inflatable origami paper frogs.
There is also coverage on using SAPI 4 to build speech-enabled applications. Information on using the SAPI 4 high level interfaces can be found by clicking here, whilst discussion of the low level interfaces can be found by clicking here.
SAPI 5.1 exposes most of the important interfaces, types and constants through a registered type library (SAPI 5.0 did not do this, making it difficult to use in Delphi without someone writing the equivalent of the JEDI import unit for SAPI 5). This means that you can access SAPI 5.1 functionality through late bound or early bound Automation. We will focus our attention on early bound Automation, which requires you to import the type library.
Choose Project | Import Type Library... and locate the type library described as Microsoft Speech Object Library (Version 5.1) in the list. Now ensure the Generate Component Wrapper checkbox is checked so the type library import unit will include component wrapper classes for each exposed Automation object. These components will go on theActiveX page of the Component Palette by default, but you may wish to specify a more appropriate page, such as SAPI 5.1.
Now press Install... so the type library will be imported and the generated components will be installed onto the Component Palette (pressing Create Unit would also generate the type library import unit, but would require us to install it manually).
The generated import unit is called SpeechLib_TLB.pas and will be installed in a package. You can either select the default package offered (the Borland User Components package by default), choose to open a different package or even create a new one. When the package is compiled and installed you will get a whopping set of 19 new components on the SAPI 5.1page of the Component Palette.
Each component is named after the primary interface it implements. So for example, the TSpVoice component implements the SpVoice interface. You can find abundant documentation on all these interfaces in the SAPI 5.1 SDK documentation.
Ready made SAPI 5.1 packages containing Automation components for Delphi 5, 6 and 7 can be found in appropriately named subdirectories under SAPI 5.1 in the accompanying files.
If you are using Delphi 6 you will encounter a problem that is still present even with Update Pack 2 installed. The type library importer has a bug where the parameters to Automation events are incorrectly dispatched (they are sent in reverse order) meaning that all the Automation events operate incorrectly (if at all). You can avoid this by importing the type library in Delphi 5 or 7 and using the generated type library import unit in Delphi 6. A Delphi 6 compatible package is supplied with this article's files (it uses a Delphi 5 generated type library import unit).
The Delphi 7 type library importer has been improved to produce more accurate Pascal representations of items in the type library than Delphi 5 did (and than Delphi 6 tried to). As a result of this, the event handlers will often have different parameter lists in the Delphi 7 imported type library. This means that the sample programs won't compile with Delphi 7 with the true Delphi 7 SAPI type library import unit.
If you wish, you can write late bound Automation that calls CreateOleObject to instantiate the Automation objects. In the case of the SpVoice interface, you would execute:var
SpVoice: Variant;
...
SpVoice := CreateOleObject('SAPI.SpVoice')Speech Synthesis
At its simplest level, all you need to do to get your program to speak is to use a TSpVoice Automation object and call the Speak method. A trivial application that does this can be found in the TextToSpeechSimple.dpr project in the files associated with this article. The code looks like this:procedure TfrmTextToSpeech.Button1Click(Sender: TObject);
begin
SpVoice1.Speak(memText.Text, SVSFDefault)
end;- The first is the text to speak, passed as a PChar. Because of the second parameter, this call will be synchronous and so will not return until the text has been spoken.
- The second parameter represents some flags that indicate how to use the first parameter (you can combine multiple flags with the or operator). For example:
- SVSFDefault means the Speak method will be synchronous
- SVSFlagAsync makes the Speak method asynchronous and so it returns immediately (you can use events to find out when speech terminates, or call theWaitUntilDone method, or call SpeakCompleteEvent to receive a Win32 event handle, which can be passed to WaitForSingleObject).
Note that the Speak method returns a stream number. When queuing several asynchronous voice streams, the stream number allows you to identify them; each voice event passes the stream number to which it relates as a parameter. - SVSFPurgeBeforeSpeak means any text being spoken and any text queued to speak will be immediately cancelled.
- SVSFNLPSpeakPunc means punctuation marks are read out by their names, rather than being used as punctuation (so ? is read out as question mark)
- SVSFIsFilename means the first parameter is a file name containing text to speak.
- SVSFIsXML means the text includes XML tags to alter attributes of the spoken text. For example this text controls the pitch, rate, volume, emphasis and pronunciation of the spoken text:
<EMPH>Hello</EMPH>
<PRON SYM="d eh l f y">Delphi</PRON> developers!
<VOLUME LEVEL="70">
I can speak <PITCH MIDDLE="+10">high</PITCH> and <PITCH MIDDLE="-10">low</PITCH>.
I can speak <RATE SPEED="+10">very quickly</RATE> and <RATE SPEED="-10">very slowly</RATE>.
I can speak <VOLUME LEVEL="40">quietly</VOLUME> and <VOLUME LEVEL="100">loudly</VOLUME>.
</VOLUME>
Enumerating Voices
The first thing the program does is to add a list of all the available voices to the combobox and set the rate and volume track bar positions. The latter part of this is trivial as the voice rate and volume are always within predetermined ranges (the volume is in the range 0 to 100 and the rate is in the range -10 to 10).procedure TfrmTextToSpeech.FormCreate(Sender: TObject);
var
I: Integer;
SOToken: ISpeechObjectToken;
SOTokens: ISpeechObjectTokens;
begin
SendMessage(lstProgress.Handle, LB_SETHORIZONTALEXTENT, Width, 0);
//Ensure all events fire
SpVoice.EventInterests := SVEAllEvents;
Log('About to enumerate voices');
SOTokens := SpVoice.GetVoices('', '');
for I := 0 to SOTokens.Count - 1 do
begin
//For each voice, store the descriptor in the TStrings list
SOToken := SOTokens.Item(I);
cbVoices.Items.AddObject(SOToken.GetDescription(0), TObject(SOToken));
//Increment descriptor reference count to ensure it's not destroyed
SOToken._AddRef;
end;
if cbVoices.Items.Count > 0 then
begin
cbVoices.ItemIndex := 0; //Select 1st voice
cbVoices.OnChange(cbVoices); //& ensure OnChange triggers
end;
Log('Enumerated voices');
Log('About to check attributes');
tbRate.Position := SpVoice.Rate;
lblRate.Caption := IntToStr(tbRate.Position);
tbVolume.Position := SpVoice.Volume;
lblVolume.Caption := IntToStr(tbVolume.Position);
Log('Checked attributes');
end;In order to keep track of the voices, these ISpeechObjectToken interfaces are added, along with a description, to the combobox's Items (the description in the Strings array and the interfaces in the Objects array).
Storing an interface reference in an object reference is possible as long as we remember exactly what we stored, and we don't make the mistake of accessing it as an object reference. Also, since the interface reference is stored using an inappropriate type, it is important to manually increment its reference count to stop it being destroyed when the RTL code decrements the reference count at the end of the method.
The OnDestroy event handler tidies up these descriptor objects by decrementing their reference counts, thereby allowing them to be destroyed.
procedure TfrmTextToSpeech.FormDestroy(Sender: TObject);
var
I: Integer;
begin
//Release all the voice descriptors
for I := 0 to cbVoices.Items.Count - 1 do
ISpeechObjectToken(Pointer(cbVoices.Items.Objects[I]))._Release;
end;procedure TfrmTextToSpeech.cbVoicesChange(Sender: TObject);
var
SOToken: ISpeechObjectToken;
begin
with lstEngineInfo.Items do
begin
Clear;
SOToken := ISpeechObjectToken(Pointer(
cbVoices.Items.Objects[cbVoices.ItemIndex]));
SpVoice.Voice := SOToken;
Add(Format('Name: %s', [SOToken.GetAttribute('Name')]));
Add(Format('Vendor: %s', [SOToken.GetAttribute('Vendor')]));
Add(Format('Age: %s', [SOToken.GetAttribute('Age')]));
Add(Format('Gender: %s', [SOToken.GetAttribute('Gender')]));
Add(Format('Language: %s', [SOToken.GetAttribute('Language')]));
Add(Format('Reg key: %s', [SOToken.Id]));
end
end;Making Your Computer Talk
There are different calls to start speech and to continue paused speech, so a helper flag is employed to record whether pause has been pressed. This allows the play button to start a fresh speech stream as well as continue a paused speech stream. The text to speak is taken from a richedit control and is spoken asynchronously thanks to the SVSFlagsAsync flag being used.procedure TfrmTextToSpeech.btnPlayClick(Sender: TObject);
begin
if not BeenPaused then
SpVoice.Speak(reText.Text, SVSFlagsAsync)
else
begin
SpVoice.Resume;
BeenPaused := False
end
end;
procedure TfrmTextToSpeech.btnPauseClick(Sender: TObject);
begin
SpVoice.Pause;
BeenPaused := True
end;
procedure TfrmTextToSpeech.btnStopClick(Sender: TObject);
begin
SpVoice.Skip('Sentence', MaxInt)
end;type
TfrmVTxtAutoLateBound = class(TForm)
...
private
MSWord: Variant;
end;
...
procedure TfrmTextToSpeechReadWordDoc.FormCreate(Sender: TObject);
begin
MSWord := CreateOleObject('Word.Application');
end;
procedure TfrmTextToSpeechReadWordDoc.btnReadDocClick(Sender: TObject);
const
// Constants for enum WdUnits
wdCharacter = $00000001;
wdParagraph = $00000004;
// Constants for enum WdMovementType
wdExtend = $00000001;
var
Moved: Integer;
Txt: String;
begin
(Sender as TButton).Enabled := False;
Stopped := False;
if dlgOpenDoc.Execute then
begin
MSWord.Documents.Open(FileName := dlgOpenDoc.FileName);
Moved := 2;
while (Moved > 1) and not Stopped do
begin
//Select next paragraph
Moved := MSWord.Selection.EndOf(Unit:=wdParagraph, Extend:=wdExtend);
if Moved > 1 then
begin
Txt := Trim(MSWord.Selection.Text);
if Length(Txt) > 0 then
SpVoice.Speak(Txt, SVSFlagsAsync);
Application.ProcessMessages;
//Move to start of next paragraph
MSWord.Selection.MoveRight(Unit := wdCharacter);
end
end;
end;
MSWord.ActiveDocument.Close;
TButton(Sender).Enabled := True;
end;
procedure TfrmTextToSpeechReadWordDoc.btnStopClick(Sender: TObject);
begin
SpVoice.Skip('Sentence', Maxint);
Stopped := True;
end;
procedure TfrmTextToSpeechReadWordDoc.FormDestroy(Sender: TObject);
begin
btnStop.Click;
MSWord.Quit;
MSWord := Unassigned;
end;Voice Events
The SpVoice object has a variety of events that fire during speech. Each block of speech starts with an OnStartStream event and ends with OnEndStream. OnStartStream identifies the speech stream, and all the other events pass the stream number to which they pertain. As each sentence is started an OnSentence event fires and there is also an OnWord event that triggers at the start of each spoken word.Additionally (among others) an OnAudioLevel event allows a progress bar to be used as a VU meter for the spoken text. However it is important to note that for some events to fire you must set the EventInterests property accordingly; to receive the OnAudioLevel event you should set EventInterests to SVEAudioLevel or SVEAllEvents.
const
Phonemes: array[1..49] of String = (
'-', '!', '&', ',', '.', '?', '_',
'1', '2', 'aa', 'ae', 'ah', 'ao', 'aw',
'ax', 'ay', 'b', 'ch', 'd', 'dh', 'eh',
'er', 'ey', 'f', 'g', 'h', 'ih', 'iy',
'jh', 'k', 'l', 'm', 'n', 'ng', 'ow',
'oy', 'p', 'r', 's', 'sh', 't', 'th',
'uh', 'uw', 'v', 'w', 'y', 'z', 'zh'
);
procedure TfrmTextToSpeech.SpVoicePhoneme(Sender: TObject;
StreamNumber: Integer; StreamPosition: OleVariant; Duration: Integer;
NextPhoneId: Smallint; Feature: TOleEnum; CurrentPhoneId: Smallint);
begin
if CurrentPhoneId <> 7 then //Display phonemes, except silence
memEnginePhonemes.Text :=
memEnginePhonemes.Text + Phonemes[CurrentPhoneId] +'-'
end;Animating Speech
An OnViseme event is triggered for each recognised viseme (a portion of speech requiring the mouth to move into a visibly different position); there are 22 different visemes generated by English speech and these are based on the Disney 13 visemes (cartoons have less granularity and Disney animators discovered many years ago that only 13 cartoon mouth shapes are required to represent all English phonemes).If you have some artistic flair and can draw a mouth in each position represented by the visemes you could use this event to provide a simple animated representation of speech.
The SAPI 5.1 SDK comes with a C++ example called TTSApp, which displays an animated cartoon microphone whose mouth is drawn to represent each viseme. The microphone is made up from a number of separate images that can all be loaded into an image list. The additional demo program TextToSpeechAnimated.dpr makes use of these images to show how the effect can be achieved.
const
Visemes: array[0..21] of Byte = (
0, // SP_VISEME_0 = 0, // Silence
11, // SP_VISEME_1, // AE, AX, AH
11, // SP_VISEME_2, // AA
11, // SP_VISEME_3, // AO
10, // SP_VISEME_4, // EY, EH, UH
11, // SP_VISEME_5, // ER
9, // SP_VISEME_6, // y, IY, IH, IX
2, // SP_VISEME_7, // w, UW
13, // SP_VISEME_8, // OW
9, // SP_VISEME_9, // AW
12, // SP_VISEME_10, // OY
11, // SP_VISEME_11, // AY
9, // SP_VISEME_12, // h
3, // SP_VISEME_13, // r
6, // SP_VISEME_14, // l
7, // SP_VISEME_15, // s, z
8, // SP_VISEME_16, // SH, CH, JH, ZH
5, // SP_VISEME_17, // TH, DH
4, // SP_VISEME_18, // f, v
7, // SP_VISEME_19, // d, t, n
9, // SP_VISEME_20, // k, g, NG
1 // SP_VISEME_21, // p, b, m
);
procedure TfrmTextToSpeech.SpVoiceViseme(Sender: TObject;
StreamNumber: Integer; StreamPosition: OleVariant; Duration: Integer;
NextVisemeId, Feature, CurrentVisemeId: TOleEnum);
const
EyesNarrow = 14;
EyesClosed = 15;
begin
imgsMic.Draw(pbMic.Canvas, 0, 0, Visemes[CurrentVisemeId]);
if Visemes[CurrentVisemeId] mod 6 = 2 then
imgsMic.Draw(pbMic.Canvas, 0, 0, EyesNarrow)
else
if Visemes[CurrentVisemeId] mod 6 = 5 then
imgsMic.Draw(pbMic.Canvas, 0, 0, EyesClosed);
end;
procedure TfrmTextToSpeech.pbMicPaint(Sender: TObject);
begin
imgsMic.Draw(pbMic.Canvas, 0, 0, 0);
end;Keeping Track Of Spoken Text
We can use OnWord and OnSentence to highlight the currently spoken work or sentence, as the events provide the character offset and length of the pertinent characters in the text. So when a sentence is started, the OnSentence event tells you which character in the text is the start of the sentence, and also how long the sentence is.procedure TfrmTextToSpeech.SetTextHilite(FirstChar, Len: Integer);
begin
reText.SelStart := FirstChar; //highlight word
reText.SelLength := Len;
end;
procedure TfrmTextToSpeech.SetTextStyle(FirstChar, Len: Integer; Styles: TFontStyles);
begin
with reText do
begin
Lines.BeginUpdate;
try
SelStart := FirstChar; //highlight word
SelLength := Len;
SelAttributes.Style := Styles; //apply requested style
SelLength := 0; //unhighlight word
finally
Lines.EndUpdate
end
end
end;
procedure TfrmTextToSpeech.SpVoiceSentence(Sender: TObject;
StreamNumber: Integer; StreamPosition: OleVariant; CharacterPosition,
Length: Integer);
begin
Log('OnSentence: stream %d, position: %s, char. pos. %d, length %d',
[StreamNumber, String(StreamPosition), CharacterPosition, Length]);
SetTextStyle(OldSentencePos, OldSentenceLen, []);
if Length > 0 then
begin
SetTextStyle(CharacterPosition, Length, [fsItalic]);
OldSentencePos := CharacterPosition;
OldSentenceLen := Length;
end;
if not StreamJustStarted then
memEnginePhonemes.Text := memEnginePhonemes.Text + #13#10;
StreamJustStarted := False;
end;
procedure TfrmTextToSpeech.SpVoiceWord(Sender: TObject;
StreamNumber: Integer; StreamPosition: OleVariant; CharacterPosition,
Length: Integer);
begin
Log('OnWord: stream %d, position: %s, char. pos. %d, length %d',
[StreamNumber, String(StreamPosition), CharacterPosition, Length]);
SetTextHilite(CharacterPosition, Length);
end;The comment in the OnSentence event handler points out that the last OnSentence event for some text has the character position set to the last character and the length set to the negative equivalent. This gives an opportunity to reset all the text formatting back to the default styles. However it is only true if the text ends with a full stop; if not you can use theOnEndStream event for tidying up.
Speaking Dialogs
As an example of using speech synthesis you can make all your VCL dialogs talk to you using this small piece of code.uses
ComObj;
var
Voice: Variant;
procedure TForm1.FormCreate(Sender: TObject);
begin
Screen.OnActiveFormChange := ScreenFormChange;
end;
procedure TForm1.ReadVCLDialog(Form: TCustomForm);
var
I: Integer;
ButtonCaptions, LabelCaption, DialogText: string;
const
SVSFlagsAsync = 1;
begin
try
if VarType(Voice) <> varDispatch then
Voice := CreateOleObject('SAPI.SpVoice');
for I := 0 to Form.ComponentCount - 1 do
if Form.Components[I] is TLabel then
LabelCaption := TLabel(Form.Components[I]).Caption
else
if Form.Components[I] is TButton then
ButtonCaptions := Format('%s%s, ',
[ButtonCaptions, TButton(Form.Components[I]).Caption]);
ButtonCaptions := StringReplace(ButtonCaptions,'&','', [rfReplaceAll]);
DialogText := Format('%s.%s%s.%s%s',
[Form.Caption, sLineBreak, LabelCaption, sLineBreak, ButtonCaptions]);
Memo1.Text := DialogText;
Voice.Speak(DialogText, SVSFlagsAsync)
except
//pretend everything is okay
end
end;
procedure TForm1.ScreenFormChange(Sender: TObject);
begin
if Assigned(Screen.ActiveForm) and
(Screen.ActiveForm.ClassName = 'TMessageForm') then
ReadVCLDialog(Screen.ActiveForm)
end;A statement such as:
MessageDlg('Save changes?', mtConfirmation, mbYesNoCancel, 0)Confirm.
Save changes?.
Yes, No, Cancel,Speech Recognition
Continuous dictation is easy to set up as no specific grammar is required, but Command and Control recognition will need a grammar to educate the recogniser as to the permissible commands.When you need SR you can either use a shared recogniser (TSpSharedRecognizer) or an in-process recogniser (TSpInprocRecognizer). The in-process recogniser is more efficient (it resides in your process address space) but means that no other SR applications can receive input from the microphone until it is closed down. On the other hand the shared recogniser can be used by multiple applications, and each one can access the microphone. It is more common to use the shared recogniser in typical SAPI applications.
The recogniser uses the notion of a recognition context to identify when it will be active (not to be confused with the use of context in a context-free grammar or CFG). A context is represented by the SpInprocRecoContext or SpSharedRecoContext interfaces. An application may use one context for each form that will use SR, or several contexts for different application modes (Office XP has a dictation mode for adding text to a document and a control mode for executing menu commands).
Recognition contexts enable you to start and stop recognition, set up the grammar and receive important recognition notifications.
Grammars
Part of the process of speech recognition involves deciding what words have actually been spoken. Recognisers use a grammar to decide what has been said, where possible. SAPI 5.x represents grammars in XML.In the case of dictation, a grammar can be used to indicate some words that are likely to be spoken. It is not feasible to try and represent the entire spoken English language as a grammar so the recogniser uses its own rules and context analysis, with any help from a grammar you might supply.
With Command and Control, the words that are understood are limited to the supported commands defined in the grammar. The grammar defines various rules that dictate what will be said and this makes the recogniser's job much easier. Rather than trying to understand anything spoken, it only needs to recognise speech that follows the supplied rules. A simple grammar that recognises three colours might look like this:
<GRAMMAR LANGID="809">
<!-- "Constant" definitions -->
<DEFINE>
<ID NAME="RID_start" VAL="1"/>
</DEFINE>
<!-- Rule definitions -->
<RULE NAME="start" ID="RID_start" TOPLEVEL="ACTIVE">
<L>
<P>red</P>
<P>blue</P>
<P>green</P>
</L>
</RULE>
</GRAMMAR>Grammars support lists to make implementing many similar commands easy and also support optional sections. For example this grammar will recognise any of the following:
- colour red
- colour red please
- colour blue
- colour blue please
- colour green
- colour green please
<GRAMMAR LANGID="809">
<DEFINE>
<ID NAME="RID_start" VAL="1"/>
</DEFINE>
<RULE NAME="start" ID="RID_start" TOPLEVEL="ACTIVE">
<P>colour</P>
<RULEREF NAME="colour" />
<O>please</O>
</RULE>
<RULE NAME="colour">
<L>
<P>red</P>
<P>blue</P>
<P>green</P>
</L>
</RULE>
</GRAMMAR>Continuous Dictation Recognition
Thankfully this is quite straightforward to use. We need to set up a recognition context object for the shared recogniser so drop a TSpSharedRecContext component on the form.The recogniser will implicitly be set up if we do not create one specifically. This means you do not need to drop a TSpSharedRecognizer or a TSpInprocRecognizer on the form unless you need to use them directly.
The code below shows how you create a simple grammar that will satisfy the SR engine for continuous dictation. The grammar is represented by an ISpeechRecoGrammar interface and is used to start the dictation session. The code comes from the ContinuousDictation.dpr sample project. SRGrammar: ISpeechRecoGrammar;
...
procedure TfrmContinuousDictation.FormCreate(Sender: TObject);
begin
//OnAudioLevel event is not fired by default - this changes that
SpSharedRecoContext.EventInterests := SREAllEvents;
SRGrammar := SpSharedRecoContext.CreateGrammar(0);
SRGrammar.DictationSetState(SGDSActive)
end;Grammar Notifications
As the SR engine does its work it calls notification methods when certain things happen, such as a phrase having been finished and recognised. These notifications are available as standard Delphi events in the Delphi Automation object component wrappers. This greatly simplifies the job of responding to the notifications.The main event is OnRecognition, which is called when the SR engine has decided what has been spoken. Whilst working it out, it will likely call the OnHypothesis event several times. Finished phrases are added to a memo on the form and whilst a phrase is being worked out the hypotheses are added to a list box so you can see how the SR engine made its decision. Each time a new phrase is started, the hypothesis list is cleared.
You can see the list of hypotheses building up in this screenshot of the program running.
This code can be used in Delphi 6 and earlier to access the text that was recognised:
procedure TfrmContinuousDictation.SpSharedRecoContextHypothesis(
Sender: TObject; StreamNumber: Integer; StreamPosition: OleVariant;
var Result: OleVariant);
begin
lstHypotheses.Items.Add(Result.PhraseInfo.GetText);
lstHypotheses.ItemIndex := lstHypotheses.Items.Count - 1
end;
procedure TfrmContinuousDictation.SpSharedRecoContextRecognition(
Sender: TObject; StreamNumber: Integer; StreamPosition: OleVariant;
RecognitionType: TOleEnum; var Result: OleVariant);
begin
memText.SelText := Result.PhraseInfo.GetText + #32
end;This code uses late bound Automation on the results object (so no Code Completion or Code Parameters), but you could use early bound Automation with:
procedure TfrmContinuousDictation.SpSharedRecoContextHypothesis(
Sender: TObject; StreamNumber: Integer; StreamPosition: OleVariant;
var Result: OleVariant);
var
SRResult: ISpeechRecoResult;
begin
SRResult := IDispatch(Result) as ISpeechRecoResult;
lstHypotheses.Items.Add(SRResult.PhraseInfo.GetText(0, -1, True));
lstHypotheses.ItemIndex := lstHypotheses.Items.Count - 1
end;
procedure TfrmContinuousDictation.SpSharedRecoContextRecognition(
Sender: TObject; StreamNumber: Integer; StreamPosition: OleVariant;
RecognitionType: TOleEnum; var Result: OleVariant);
var
SRResult: ISpeechRecoResult;
begin
SRResult := IDispatch(Result) as ISpeechRecoResult;
memText.SelText := SRResult.PhraseInfo.GetText(0, -1, True) + #32
end;The code here does not check if a valid IDispatch reference is in the Variant but probably should.
In Delphi 7 the code should look like this:procedure TfrmContinuousDictation.SpSharedRecoContextHypothesis(
ASender: TObject; StreamNumber: Integer; StreamPosition: OleVariant;
const Result: ISpeechRecoResult);
begin
lstHypotheses.Items.Add(Result.PhraseInfo.GetText(0, -1, True));
lstHypotheses.ItemIndex := lstHypotheses.Items.Count - 1
end;
procedure TfrmContinuousDictation.SpSharedRecoContextRecognition(
ASender: TObject; StreamNumber: Integer; StreamPosition: OleVariant;
RecognitionType: TOleEnum; const Result: ISpeechRecoResult);
begin
memText.SelText := Result.PhraseInfo.GetText(0, -1, True) + #32
end;Engine Dialogs
The buttons on the form allow various engine dialogs to be invoked (if supported). This support is all attained through a couple of methods of the recogniser object.const
SPDUI_EngineProperties = 'EngineProperties';
SPDUI_AddRemoveWord = 'AddRemoveWord';
SPDUI_UserTraining = 'UserTraining';
SPDUI_MicTraining = 'MicTraining';
SPDUI_RecoProfileProperties = 'RecoProfileProperties';
SPDUI_AudioProperties = 'AudioProperties';
SPDUI_AudioVolume = 'AudioVolume';
procedure TfrmContinuousDictation.btnEnginePropsClick(Sender: TObject);
begin
InvokeUI(SPDUI_EngineProperties, 'Engine Properties')
end;
procedure TfrmContinuousDictation.btnUserSettingsClick(Sender: TObject);
begin
InvokeUI(SPDUI_RecoProfileProperties, 'User Settings')
end;
procedure TfrmContinuousDictation.btnLexiconClick(Sender: TObject);
begin
InvokeUI(SPDUI_AddRemoveWord, 'Add/Remove Word')
end;
procedure TfrmContinuousDictation.btnTrainGeneralClick(Sender: TObject);
begin
InvokeUI(SPDUI_UserTraining, 'Speaker Training')
end;
procedure TfrmContinuousDictation.btnTrainMicClick(Sender: TObject);
begin
InvokeUI(SPDUI_MicTraining, 'Microphone Setup')
end;
procedure TfrmContinuousDictation.btnAudioPropsClick(Sender: TObject);
begin
InvokeUI(SPDUI_AudioProperties, 'Audio Properties')
end;
procedure TfrmContinuousDictation.btnAudioVolClick(Sender: TObject);
begin
InvokeUI(SPDUI_AudioVolume, 'Audio Volume')
end;
procedure TfrmContinuousDictation.InvokeUI(const TypeOfUI, Caption: WideString);
var
U: OleVariant;
begin
U := Unassigned;
if SpSharedRecoContext.Recognizer.IsUISupported(TypeOfUI, U) then
SpSharedRecoContext.Recognizer.DisplayUI(Handle, Caption, TypeOfUI, U)
end;Command and Control Recognition
For C and C recognition we will need a grammar to give the SR engine rules by which to recognise the commands. This grammar is used by a sample project called CommandAndControl.dpr in the files that accompany this article.<GRAMMAR LANGID="809">
<!-- "Constant" definitions -->
<DEFINE>
<ID NAME="RID_start" VAL="1"/>
<ID NAME="PID_chosencolour" VAL="2"/>
<ID NAME="PID_colourvalue" VAL="3"/>
</DEFINE>
<!-- Rule definitions -->
<RULE NAME="start" ID="RID_start" TOPLEVEL="ACTIVE">
<O>colour</O>
<RULEREF NAME="colour" PROPNAME="chosencolour" PROPID="PID_chosencolour" />
<O>please</O>
</RULE>
<RULE NAME="colour">
<L PROPNAME="colourvalue" PROPID="PID_colourvalue">
<P VAL="1">red</P>
<P VAL="2">blue</P>
<P VAL="3">green</P>
</L>
</RULE>
</GRAMMAR>The colour rule defines one of three colours that can be spoken, each of which has a value defined for it. Again, this value will be accessible thanks to the list being defined as a property (colourvalue).
This grammar is stored in an XML file and loaded in the OnCreate event handler.
procedure TfrmCommandAndControl.FormCreate(Sender: TObject);
begin
//OnAudioLevel event is not fired by default - this changes that
SpSharedRecoContext.EventInterests := SREAllEvents;
SRGrammar := SpSharedRecoContext.CreateGrammar(0);
SRGrammar.CmdLoadFromFile('C and C Grammar.xml', SLODynamic);
SRGrammar.CmdSetRuleIdState(0, SGDSActive)
end;The OnRecognition event handler does the work of locating the chosencolour property and then finding the nested colourvalue property. Its value is used to change the form colour at the user's request, for example with phrases such as:
- red please
- colour green
- colour blue please
- red
procedure TfrmCommandAndControl.SpSharedRecoContextRecognition(
ASender: TObject; StreamNumber: Integer; StreamPosition: OleVariant;
RecognitionType: TOleEnum; const Result: ISpeechRecoResult);
begin
with Result.PhraseInfo do
begin
Log('OnRecognition: %s', [GetText(0, -1, True)]);
case GetPropValue(Result, ['chosencolour', 'colourvalue']) of
1: Color := clRed;
2: Color := clBlue;
3: Color := clGreen;
end
end
end;function GetProp(Props: ISpeechPhraseProperties;
const Name: String): ISpeechPhraseProperty; overload;
var
I: Integer;
Prop: ISpeechPhraseProperty;
begin
Result := nil;
for I := 0 to Props.Count - 1 do
begin
Prop := Props.Item(I);
if CompareText(Prop.Name, Name) = 0 then
begin
Result := Prop;
Break
end
end
end;
function GetPropValue(SRResult: ISpeechRecoResult;
const Path: array of String): OleVariant;
var
Prop: ISpeechPhraseProperty;
PathLoop: Integer;
begin
for PathLoop := Low(Path) to High(Path) do
begin
if PathLoop = Low(Path) then //top level property
Prop := GetProp(SRResult.PhraseInfo.Properties, Path[PathLoop])
else //nested property
Prop := GetProp(Prop.Children, Path[PathLoop]);
if not Assigned(Prop) then
begin
Result := Unassigned;
Exit;
end
end;
Result := Prop.Value
end;Speech Recognition Troubleshooting
If you get issues of SR stopping (or not starting) unexpectedly, or other weird SR issues, check your recording settings have the microphone enabled.- Double-click the Volume icon in your Task Bar's System Tray. If no Volume icon is present, choose Start | Programs | Accessories | Entertainment | Volume Control.
- If you see a Microphone column, ensure it has its Mute checkbox checked
- Choose Options | Properties, click Recording, ensure the Microphone option is checked and press OK.
- Now ensure the Microphone column has its Select checkbox enabled, if it has one, or that its Mute checkbox is unchecked, if it has one.
SAPI 5.1 Deployment
When distributing SAPI 5.1 applications you will need get hold of the redistributable components package available as SpeechSDK51MSM.exe fromhttp://www.microsoft.com/speech/download/SDK51 (a colossal file, weighing in at 132 Mb) contains Windows Installer merge modules for all the SAPI 5.1 components (the main DLLs, the TTS and SR engines, the Control Panel applet) and the SDK documentation includes a white paper on how to use all these components from within a Windows Installer compatible installation building tool.Summary
Adding various speech capabilities into a Delphi application does not take an awful lot of work, particularly if you do the background work to understand the SAPI concepts.There is much to Speech API that we have not looked at in this paper but hopefully the areas covered will be enough to whet your appetite and get you exploring further on your own.
References/Further Reading
The following is a list of useful articles and papers that I found on SAPI 5.1 development during my research on this subject.- Speech Part 1 - How to Add "Text to Speech" (Speech Synthesis) to your Delphi Apps by Alec Bergamini, Delphi 3000.
This discusses installing the SAPI 5.1 SDK and getting simple speech. - Speech Part 2 - How to Add Simple Dictation speech recognition to your Delphi Apps by Alec Bergamini, Delphi 3000.
This looks at simple dictation SR.
About Brian Long
Brian Long used to work at Borland UK, performing a number of duties including Technical Support on all the programming tools. Since leaving in 1995, Brian has been providing training and consultancy on Borland's RAD products ever since, and is now moving into the .NET world.Besides authoring a Borland Pascal problem-solving book published in 1994, Brian is a regular columnist in The Delphi Magazine and has had numerous articles published in Developer's Review, Computing, Delphi Developer's Journal and EXE Magazine. He was nominated for the Spirit of Delphi 2000 award and was voted Best Speaker at Borland's BorCon 2002 conference in Anaheim, California by the conference delegates.
There are a growing number of conference papers and articles available on Brian's Web site, so feel free to have a browse.
In his spare time (and waiting for his C++ programs to compile) Brian has learnt the art of juggling and making inflatable origami paper frogs.
Update Contact :
No Wa/Telepon (puat) : 085267792168
No Wa/Telepon (fajar) : 085369237896
Email : Fajarudinsidik@gmail.com
No Wa/Telepon (puat) : 085267792168
No Wa/Telepon (fajar) : 085369237896
Email: Fajarudinsidik@gmail.com
atau Kirimkan Private messanger melalui email dengan klik tombol order dibawah ini :
