Repository navigation
feat(convozen): add Akshara STT and Ragini TTS plugin - #7319
aminaashrafch wants to merge 13 commits into
Conversation
Adds livekit-plugins-convozen for ConvoZen's Akshara speech-to-text and Ragini text-to-speech, covering nine Indian languages including code-mixed speech. Akshara returns one transcript per request and emits no interim results, so the STT declares streaming=False and Agent.stt_node wraps it in stt.StreamAdapter using the session VAD to segment utterances. Ragini synthesizes a whole request at a time, so the TTS declares streaming=False and is wrapped by tts.StreamAdapter. Neither has a websocket path. Ragini returns a chunked WAV container whose RIFF length fields are placeholders, so the package depends on livekit-agents[codecs] and lets AudioEmitter decode it rather than parsing headers itself. Keyterms configured on the AgentSession are forwarded to Akshara's keywords field. Akshara also returns a score, which is an unbounded log-probability rather than a [0, 1] confidence; it is exposed through SpeechData.metadata and confidence is left at its default.
|
|
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
There was a problem hiding this comment.
Note
This report is out of date. Scroll down for Devin Review's latest report on this PR.
Devin Review found 2 potential issues.
1 flag not posted on this PR by your GitHub settings — view it in Devin Review. (Configure)
| if res.status != 200: | ||
| body = (await res.text())[:_MAX_ERROR_BODY] | ||
| raise APIStatusError( | ||
| message=f"ConvoZen Akshara returned {res.status}: {body}", | ||
| status_code=res.status, | ||
| request_id=None, | ||
| body=body, | ||
| ) |
There was a problem hiding this comment.
🟨 STT errors expose provider content
A non-200 response embeds body in APIStatusError. Retry logs and exception telemetry can expose provider content without PII tagging.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
In 5e45276 I set aligned_transcript=False explicitly, documented that word_timestamps values are relative to each recognized utterance (docstring and README), and added a test pinning the capability with word_timestamps=True. If you'd like batch STTs to support alignment, I'm happy to follow up with a StreamAdapter change that propagates the capability and offsets forwarded timings by each segment's position in the stream.
Akshara returns word timings relative to each recognized utterance, and stt.StreamAdapter forwards them without offsetting them onto the audio stream, so they do not meet the aligned_transcript contract. Declare aligned_transcript=False explicitly, document the timing reference in the word_timestamps docstring and README, and pin the capability in tests.
# Conflicts: # livekit-agents/pyproject.toml
There was a problem hiding this comment.
Note
Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.
Devin Review found 2 new potential issues.
2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| if is_given(language) or is_given(lang_tags): | ||
| # Re-derive so that changing the language alone also moves the hint, | ||
| # and so an explicit tag list is validated the same way as at __init__. | ||
| self._opts.lang_tags = _resolve_lang_tags( | ||
| self._opts.language, | ||
| lang_tags if is_given(lang_tags) else NOT_GIVEN, | ||
| ) |
There was a problem hiding this comment.
🟡 Explicit language hints are discarded
Calling update_options(language=...) replaces explicit lang_tags with one derived tag. Later code-mixed recognition loses its configured language hints.
Learn more
lang_tags can either be explicit configuration or a default derived from language. The current options object stores only the resolved list, so update_options cannot distinguish those cases. A language-only update always resolves a fresh list and overwrites explicit code-mixing hints.
Example: Construct STT(language="hi", lang_tags=["hi", "en"]), then call update_options(language="ta"). The next request sends ["ta"] instead of retaining the explicitly configured ["hi", "en"].
Recommended fix: Track whether lang_tags is explicit or derived. Recompute it on a language-only update only when it was derived; update that state whenever lang_tags itself is supplied.
Was this helpful? React with 👍 or 👎 to provide feedback.
There was a problem hiding this comment.
Note
Newer findings are available below. Devin Review posted a newer report on this PR, in addition to the findings presented here.
Devin Review found 3 new potential issues.
4 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| if is_given(language): | ||
| self._opts.language = language | ||
| if is_given(language) or is_given(lang_tags): | ||
| # Re-derive so that changing the language alone also moves the hint, | ||
| # and so an explicit tag list is validated the same way as at __init__. | ||
| self._opts.lang_tags = _resolve_lang_tags( | ||
| self._opts.language, | ||
| lang_tags if is_given(lang_tags) else NOT_GIVEN, | ||
| ) |
There was a problem hiding this comment.
🟡 Rejected tag updates still change recognition options
Passing an invalid lang_tags value raises after update_options has changed language and other options. Later requests use the rejected update's partially applied settings.
Learn more
The recognizer remains usable after update_options raises. The method writes several options first, then validates language tags through _resolve_lang_tags. An invalid tag raises without reverting earlier writes, so subsequent recognition no longer uses the configuration from before the rejected update.
Example: Start with language hi, then call update_options(language="ta", lang_tags=["xx"]). The call raises, yet a later request reports ta while retaining the old tags.
Recommended fix: Build and validate a complete new options object before replacing self._opts, so failed updates leave the existing configuration intact.
Was this helpful? React with 👍 or 👎 to provide feedback.
| if resp.status != 200: | ||
| body = (await resp.text())[:_MAX_ERROR_BODY] | ||
| raise APIStatusError( | ||
| message=f"ConvoZen Ragini returned {resp.status}: {body}", | ||
| status_code=resp.status, | ||
| request_id=None, | ||
| body=body, |
There was a problem hiding this comment.
🟥 Provider error bodies leak into TTS retry logs
When Ragini returns a retryable error containing synthesis text or credentials, APIStatusError includes its body in the message. TTS retries log that message without a PII marker, exposing the value.
Was this helpful? React with 👍 or 👎 to provide feedback.
The merge with main left both the pre-release and the 1.8.3 optional dependency tables in livekit-agents/pyproject.toml, which made the file invalid TOML and failed every job that starts with uv sync. Keep the 1.8.3 table with the convozen extra, and align the plugin's version and livekit-agents floor with the release.
# Conflicts: # livekit-agents/pyproject.toml # uv.lock
There was a problem hiding this comment.
Devin Review found 1 new potential issue.
3 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)
| cartesia = ["livekit-plugins-cartesia>=1.8.5"] | ||
| cerebras = ["livekit-plugins-cerebras>=1.8.5"] | ||
| clova = ["livekit-plugins-clova>=1.8.5"] | ||
| convozen = ["livekit-plugins-convozen>=1.8.5"] |
There was a problem hiding this comment.
🔴 ConvoZen extra cannot resolve its plugin
Installing the convozen extra requires plugin version 1.8.5, but the workspace plugin declares 1.8.3. Dependency resolution fails for the documented installation path.
Learn more
The optional dependency names a plugin from this workspace. Its declared version is 1.8.3, below the new 1.8.5 minimum. The workspace source is registered, so local resolution cannot choose a compatible workspace release. Other extras track the released framework version, but this plugin has not reached that version.
Example: A developer checks out the PR and installs livekit-agents[convozen] from the workspace. The resolver sees ConvoZen 1.8.3 and the >=1.8.5 requirement, so it rejects the dependency rather than installing the plugin.
Recommended fix: Align the ConvoZen extra's lower bound with the plugin version available at installation time. Coordinate the next version bump with the release process, which owns version.py edits.
Was this helpful? React with 👍 or 👎 to provide feedback.
This branch has not been deployed
2 participants
Adds
livekit-plugins-convozen, integrating ConvoZen's Akshara speech-to-text and Ragini text-to-speech models. Both cover nine Indian languages (Bengali, English, Gujarati, Hindi, Kannada, Malayalam, Marathi, Tamil, Telugu), including code-mixed speech.Install with
pip install "livekit-agents[convozen]".Design
STTdeclaresstreaming=False.Agent.stt_nodewraps it instt.StreamAdapter, so a VAD is required on theAgentSession.TTSdeclaresstreaming=Falseand is wrapped bytts.StreamAdapter. Audio streams back as a chunked WAV, decoded byAudioEmitter(hence thelivekit-agents[codecs]dependency).keywordsfield (keyterms=True).scoreis an unbounded log-probability rather than a[0, 1]confidence, so it's exposed viaSpeechData.metadataandconfidenceis left unset.from Noneand report only the exception type, since aiohttp errors carry request headers including the API key.Testing
tests/test_plugin_convozen.py: 29 hermetic unit tests (pytest --unit)make format-checkandmake lintpass;livekit.plugins.convozentype-checks undermake type-checkAgentSessionwith Silero VAD against a local LiveKit server: caller speech → STT → agent turn → TTS audio back to the callerNotes
uv.lockisn't included; regenerating it locally changed unrelated Python version markers, and CI'suv syncresolves it.