New Text to Speech experiment

[dkotter]

Note

This is dependent on changes coming to the OpenAI Provider (see PR 42) or Google Provider (see PR 31)

What?

Adds a new experiment, Text to Speech, that allows editors the ability to on-demand generate speech for a post, including the title and post content. This is then displayed as a player on the front-end or that can be turned off on a post by post basis.

Why?

Adding support for Text to Speech allows sites to provide their users with the ability to listen to content instead of having to read the content.

How?

• Registers two new Abilities, ai/speech-generation and ai/speech-import. These can be used to generate speech from text and import that speech as an audio file into the Media Library
• Add shared functionality across multiple classes that can take in some text content, chunk that content down (if needed, to stay under API limits. Note only works if the format is MP3) and send that content off to generate speech for. It then combines all the returned speech data into a single audio file which we import into the Media Library. This functionality is used both if someone directly uses the Abilities above and when audio generation is manually triggered in the admin
• Add a REST endpoint that is used to track audio progress, allowing audio generation to happen via cron events. This allows longer running processes and processing of audio even if someone navigates away

Use of AI Tools

AI assistance: Yes
Tool(s): Claude Code
Model(s): Opus 4.8
Used for: Help with initial planning and implementation. Refinement, review and testing done by me

Testing Instructions

1. Requires an AI Provider that supports Text to Speech. Easiest way to test for now is to download the OpenAI Provider plugin from this PR or Google Provider plugin from this PR
2. Turn on the Text to Speech experiment
3. Go to a post and find the Text to Speech sidebar panel. Click on the Generate Audio button
4. Audio generation will take a bit but is run in the background via cron jobs. Easiest is to just stay on the page until it is done but you can navigate away and come back later
5. Once finished, you should see an audio player render in that panel, along with a toggle to display the audio player on the front-end and a button to regenerate audio
6. View the front-end and ensure the audio player shows there

Screenshots

Changelog Entry

Added - New experiment, Text to Speech, that allows you to generate audio for a specific post and then render audio controls on the frontend.

[github-actions]

The following accounts have interacted with this PR and/or linked issues. I will continue to update these lists as activity occurs. You can also manually ask me to refresh this list by adding the props-bot label.

Unlinked Accounts

The following contributors have not linked their GitHub and WordPress.org accounts: @drzraf.

Contributors, please read how to link your accounts to ensure your work is properly credited in WordPress releases.

If you're merging code through a pull request on GitHub, copy and paste the following into the bottom of the merge commit message.

Unlinked contributors: drzraf.

Co-authored-by: dkotter <dkotter@git.wordpress.org>
Co-authored-by: saarnilauri <laurisaarni@git.wordpress.org>
Co-authored-by: jeffpaul <jeffpaul@git.wordpress.org>

To understand the WordPress project's expectations around crediting contributors, please review the Contributor Attribution page in the Core Handbook.

[codecov]

Codecov Report

❌ Patch coverage is 77.15134% with 231 lines in your changes missing coverage. Please review.
✅ Project coverage is 74.77%. Comparing base (f5bade4) to head (e7d4b51).

Files with missing lines	Patch %	Lines
...udes/Experiments/Text_To_Speech/Text_To_Speech.php	69.18%	49 Missing ⚠️
...es/Experiments/Text_To_Speech/Speech_Generator.php	34.37%	42 Missing ⚠️
includes/Abilities/Speech/Generate_Speech.php	78.35%	29 Missing ⚠️
includes/Abilities/Speech/Import_Base64_Audio.php	83.92%	27 Missing ⚠️
...ncludes/Experiments/Text_To_Speech/Job_Manager.php	87.64%	22 Missing ⚠️
...udes/Experiments/Text_To_Speech/Voice_Resolver.php	76.34%	22 Missing ⚠️
...des/Experiments/Text_To_Speech/REST_Controller.php	85.57%	15 Missing ⚠️
includes/helpers.php	71.42%	10 Missing ⚠️
...udes/Experiments/Text_To_Speech/Audio_Combiner.php	77.14%	8 Missing ⚠️
...des/Experiments/Text_To_Speech/Content_Chunker.php	89.18%	4 Missing ⚠️
... and 1 more

Additional details and impacted files

@@              Coverage Diff              @@
##             develop     #888      +/-   ##
=============================================
+ Coverage      74.57%   74.77%   +0.19%     
- Complexity      3132     3355     +223     
=============================================
  Files            132      141       +9     
  Lines          12213    13224    +1011     
=============================================
+ Hits            9108     9888     +780     
- Misses          3105     3336     +231     

Flag	Coverage Δ
unit	74.77% <77.15%> (+0.19%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[jeffpaul]

@dkotter should we work to get those two provider plugin PRs merged & into a release before pressing much further ahead on this PR?

[dkotter]

@dkotter should we work to get those two provider plugin PRs merged & into a release before pressing much further ahead on this PR?

Yeah, not much point in releasing this here until we have at least one provider that supports it. I've bumped this to the next milestone so hopefully by then the upstream PRs are merged and released

[saarnilauri]

I have been workin on the Elevenlabs provider. Would love to hear how it works with this experiment.

https://github.com/saarnilauri/ai-provider-for-elevenlabs

[dkotter]

I have been workin on the Elevenlabs provider. Would love to hear how it works with this experiment.

https://github.com/saarnilauri/ai-provider-for-elevenlabs

Thanks for pointing me at that. Tested that quickly and it currently does not work due to two issues:

1. The is_supported_for_text_to_speech_conversion fails for ElevenLabs as it doesn't declare support for the inline output file type (simple change, see https://github.com/WordPress/ai-provider-for-openai/pull/42/changes#diff-6b896bf241058a261e3e1bbf2d83114e2f8ed5273bc0402f0e01fdcc1ed6f08dR189)
2. I added that and was then getting the following error: The outputSpeechVoice option is required for ElevenLabs text-to-speech.

Seems ElevenLabs requires you to pass in a voice ID which we don't do in this PR. We do support passing in a custom voice option though we don't surface a setting for that anywhere. I temporarily added this line to our code $prompt_builder->as_output_speech_voice( 'JBFqnCBsd6RMkjVDRZzb' ) and it then worked

Will need to decide here whether we surface a setting that allows a user to choose a voice (ideally dynamically pulling in the voice options the provider supports and probably defaulting to the first in the list in case someone doesn't care to choose) or that's also a change you could make on your end, hardcoding a voice ID if one isn't passed (though I don't know enough about ElevenLabs to know if voice IDs are unique per account)

[saarnilauri]

@dkotter Following up on the ElevenLabs voice ID discussion above:

Provider side: I've released ElevenLabs provider 0.3.0 which fixes both issues you found. The TTS (and sound generation) models now declare inline outputFileType support, and when no outputSpeechVoice is configured the provider falls back to the premade "George" voice (JBFqnCBsd6RMkjVDRZzb, premade voice IDs are shared across all accounts, so it always works). The default can be overridden via an env var/constant, an option, or a filter, and an explicitly passed voice always wins. So this PR now works with ElevenLabs out of the box, no changes needed here.

Plugin side: I also took a stab at the "dynamically pull voices and default to the first one" idea and opened dkotter#5 against your feature/text-to-speech branch. It adds a Voice_Resolver that reads voices from the resolved model's metadata (supportedValues on the outputSpeechVoice option), renders the Voice setting as a select when a provider declares them, defaults to the first voice at generation time, and adds a wpai_tts_default_voice filter as an escape hatch for providers that declare no values yet. Since no provider currently declares voice values, it degrades to the existing free-text field, but it establishes the contract providers can adopt. Feel free to merge, adapt, or ignore it.

[drzraf]

$pre = apply_filters( 'wpai_tts_pre_generate_chunk', null, $text, $voice );

Most TTS (for accessibility etc...) distinguish between voice and language. If you choose a voice, you generally also have a distinct language setting alongside. (Both are important to generate audio output and inferring a language from a voice or vice-versa may not be obvious)

To take one example, the speechd Speech Synthesis Interface Protocol considers:

SET { all | self | id } LANGUAGE language-code
SET { all | self | id } VOICE_TYPE name

(not to mention the speed, which is different from the playback speed)

[drzraf]

I would also suggest one missing feature: the ability to easily filter by post-type (or just pass the post to allow for more granular control [sticky/date/...]). The vast majority of websites would likely want to selectively use it based on specific post type.

[dkotter]

Feel free to merge, adapt, or ignore it.

Thanks @saarnilauri! Made a few changes but looked good so I've merged that in now

[dkotter]

Most TTS (for accessibility etc...) distinguish between voice and language. If you choose a voice, you generally also have a distinct language setting alongside

@drzraf I don't believe OpenAI supports passing in a language (or even Google), they just infer the language from the text you send. ElevenLabs does allow you to pass the language code but it also will default to using the language of the text you send.

I guess I see no reason to complicate this further and introduce a setting to set the language when we can just rely on the LLM to match the language of the content given.

I would also suggest one missing feature: the ability to easily filter by post-type (or just pass the post to allow for more granular control [sticky/date/...]). The vast majority of websites would likely want to selectively use it based on specific post type.

I guess not sure what the request is here? Right now you have to manually trigger TTS, so you as a user choose what post that is run on. Is the thought here to provide a way for a user to limit the output of that generate button?

[drzraf]

I guess I see no reason to complicate this further and introduce a setting to set the language when we can just rely on the LLM to match the language of the content given.

Assume the LLM fails the language detection for my post. What would be the steps to follow to overcome this (or would I be stuck, unable to associate a spoken version of the post?)

[dkotter]

Assume the LLM fails the language detection for my post. What would be the steps to follow to overcome this (or would I be stuck, unable to associate a spoken version of the post?)

If this happens then yes, the audio generated wouldn't be what you want (or if the LLM errors instead, you wouldn't have any audio). This feels fairly unlikely to me that an LLM supports your language but can't generate speech without you telling it the language but I guess that may happen. Curious if this is a scenario you've run into before?

I personally still lean towards not complicating the UI by adding an additional setting a user has to consider and just assuming/hoping each AI service can detect the language properly. If we get actual user reports that run into problems with that, we can then look to add this in, though will need to figure out how to allow someone to set a language but not have that break integrations (like OpenAI) that don't support passing in a language.

[drzraf]

I'm completely with you about not complicating the UI.

What I'm a bit worried about is the limited internal API/hooks. Because if UI/network/LLM/service/quota/payment fails, that's what one would use (wp eval or whatever or read-out myself the article with the microphone) to resolve/workaround the problem. Simple UI but flexible API that also brings foundations for other plugins enhancements to build upon.

There are many ways language could be badly interpreted. For example, in STT (whisper), the understanding is language-neutral (because of the way training is done for natively mulilingual dataset/STT). Language acts as a "hint" but also condition the text output language. English with language=french would output the English transcribed text translated in French.

Non-English users may also have posts containing English expressions mixed up with their native language and I'm not sure these are situation where auto-detection is 100% reliable.

Basically, if install my local WordPress instance + AI endpoint + piper TTS + couple of custom voices, could I expect to manage language & LLM failure. Maybe that day I've to read-out myself the article using my microphone :)

Making language selection at the hook/filter level + audio attachment decoupled from LLM generation itself may make the whole workflow more resilient for unexpected cases. For a component relying so much on network/3rd-party service/non-deterministic process, it may be desirable.