fix: Gemini Omni video generation gaps and session-expiry data loss - #261
Open
HSbedi87 wants to merge 20 commits into
Open
fix: Gemini Omni video generation gaps and session-expiry data loss#261HSbedi87 wants to merge 20 commits into
HSbedi87 wants to merge 20 commits into
Conversation
Editing an image repeatedly produced the same result, with no way to ask for more variation. Temperature was never sent, so every request used the model's default. The DTO now accepts an optional temperature and threads it into GenerateContentConfig at both call sites - text-to-image and image-to-image. Omitting it keeps the model default rather than substituting one of ours. The range is 0.0 to 2.0, which the Gemini image model cards document and a live check confirmed: 2.0 is accepted and 2.5 is rejected. Imagen and Gemini Omni do not take the parameter and do not receive it.
Extracted from a later commit that bundled it with unrelated Edit Video changes when this branch was first split. Adds the model-capability flag, the slider in the settings popup, and a status chip showing the current value so it does not require opening the popup to check.
Users reported being signed out unexpectedly and losing what they had set up. Four separate defects contributed. No token refresh existed. getValidIdentityPlatformToken$ only read a cached token despite a comment claiming it would "trigger a silent refresh", so once the hour-long Firebase token lapsed every request failed. It now calls getIdToken(), which refreshes as expiry approaches. Requests were dropped silently. The no-session branch returned of() - an observable that completes without emitting - so the request was never sent and the caller received neither a response nor an error. It now raises SessionExpiredError. isLoggedIn() navigated as a side effect. A predicate named isLoggedIn redirected to /login, and it was consulted on every HTTP request, so any call made after the cached token aged out bounced the user off the page they were on. It is now pure; both route guards already redirected. Firebase restores the signed-in user asynchronously, so currentUser is briefly null after a page load. Treating that as "no session" logged the user out on refresh whenever the cache had aged out - precisely the case Firebase would have recovered from moments later. The token path now waits for the first definitive auth state before concluding anything. The interceptor logged out on any error that was not an HttpErrorResponse, which includes a dropped connection during refresh. It now signs out only on SessionExpiredError. Separately, the video page discarded reference images, video and audio on every save and restore, so a reload silently cleared whatever the user had attached - and an unexpected logout took it too. No file data is involved: those entries hold asset IDs plus a presigned preview URL. The IDs are durable and are what generation needs, so they are persisted and entries lacking one are dropped on restore. The existing spec asserted the discarding behaviour as correct and has been rewritten.
The Omni branch called interactions.create with only model, input and sometimes previous_interaction_id. Everything else the Interactions API accepts was left unset, so the model had to infer intent from an untagged, order-dependent list of media parts while the UI silently discarded the mode, aspect ratio and duration the user had chosen. Verified against the live Vertex API rather than docs alone; several of the constraints below are undocumented and were found by testing. Sent now: - generation_config.video_config.task, derived from the request. Without it the model guesses, which is why a reference image was rendered as a closing frame. - response_format.aspect_ratio. Selecting 9:16 previously produced 16:9. - response_format.duration. Confirmed honoured: 3s/6s/10s requests returned 3.01s/6.02s/10.01s. - delivery=uri with gcs_uri, so Vertex writes straight to the bucket. Inline base64 is capped near 4MB, which an 8s 720p clip exceeds. Multi-turn editing never ran. raw_data was written nested and read flat, so the turn-2 branch could not be reached; the replayed payload also carried a "thought" content part, which is not a Content variant, and re-uploaded the parent video. Vertex additionally rejects previous_interaction_id for this model, and rejects it outright when a task is also set. Editing now takes one of two verified paths: replaying the prior interaction's steps, or sending the clip by URI with task=edit for clips that predate step storage. Both aspect_ratio and duration are omitted on edits, which the API also rejects. Output is read via interaction.output_video instead of assuming the video is the first content item, and number_of_media now drives the clip count instead of being pinned to 1. Limits are resolved per model rather than as static field constraints: Omni allows 3-10s and 7 references, Veo allows 4/6/8s and 3. The old bounds accepted 5s and 7s for Veo, which it does not offer. Inputs Omni cannot process - video extension, first+last frame interpolation and audio references - are rejected rather than accepted and ignored. gemini-omni-generate-preview is moved to the deprecated block: Vertex answers "Unsupported model interaction". It is retained so historical rows still deserialize.
The video page hardcoded its mode list, aspect ratios, duration rules and output count, so the UI advertised whatever Veo happened to support. With Gemini Omni as the default model that meant offering video extension and first+last frame interpolation, neither of which it can do, and hiding multi-output, which it can. A ModelCapability system already existed and was honoured in home.component, flow-prompt-box and the workflow editor. The video page was the outlier. These options now come from the same source. Capabilities added: supportsLastFrame, supportsVideoReference, supportsAudioReference, maxOutputs, supportsRoleTags and restrictsDurationOutsideTextToVideo. GenerationMode gains Extend Video, Concatenate Video and Edit Video, which previously could not be expressed at all. Three hardcoded model checks are replaced by capability lookups. Each was invisible until a model diverged from Veo: - The duration rule pinned every non-text-to-video mode to the longest length. That is true for Veo but not Omni, which left Omni stuck at 10s in Frames to Video and Ingredients to Video. - The output dropdown hid x2-x4 for Omni, making multi-output unreachable. - The audio reference slot was offered for Omni. Testing four input shapes - audio part by URI, audio part inline, and document part with and without an explicit mime - the API rejects all of them with "This model does not support audio input", so the control is now hidden. Editing an existing clip becomes a first-class mode. Extend Video appends footage and Omni cannot do it; editing modifies a clip and Omni can, so they need separate fields rather than sharing source_video_asset_id. Edit with Omni now routes here and passes the selected clip index, so editing the third clip of a job continues that clip's conversation. Reference thumbnails show a clickable index that inserts <IMAGE_REF_N> at the caret. The binding is real: tagged prompts produced the requested pairing in 4 of 4 runs against a 1 in 4 untagged baseline. Prompts are passed through untouched so user-authored tags survive. The workflow editor's mode list is filtered separately, since its request body cannot express Extend or Concatenate.
The backend applies pending migrations automatically on startup, so deploying a newer image runs every migration the environment is behind on - not only those belonging to the change being deployed. Some of them drop tables. Nothing documented that, so an operator could reasonably deploy without knowing what was about to run against their database. Covers checking pending migrations before deploying, taking a restorable backup, deploying both services via Cloud Build, and rolling back. Notes that Cloud Run rollback is lossless while a database rollback is not, so the backup is the real safeguard. Also records the compatibility notes for the Gemini Omni changes: which previously accepted requests now return 400, why saved workflows pairing Omni with an end frame need auditing first, and confirmation that no schema change or data rewrite is involved.
Generated video output and character reference sheets used for local content-policy testing. Both are large binaries that should never reach a commit, and _review/ in particular can run to tens of megabytes.
<IMAGE_REF_N> is positional: it counts the reference images in the order they reach the model. The edit branch added each one with insert(1, ...) inside a loop, which reverses them, so <IMAGE_REF_0> bound to the last image the user attached and every tag in the prompt addressed the wrong character. With two references it is a straight swap. References are now appended in order, giving text, images, video. Verified against the live API. With three references, naming a single index selected the right character in every run. With two tags bound in one prompt, exchanging the indices flipped the composited arrangement each time - and the test used a character absent from the source clip and a pose the source does not contain, so neither result can be explained by the model copying the source. The existing test used a single reference and so could not detect a reversal. The new one uses three and asserts the URIs in sequence. Also adds the audio strip helper the edit path needs: Omni refuses to edit a clip containing speech when reference images are present, and its own output always carries audio, so editing one of its clips would otherwise fail.
Driving the UI from model capabilities silently disabled several paths that previously worked. Reference images were never sent in Edit Video. The payload gated them on 'Ingredients to Video', so they shipped as undefined and the <IMAGE_REF_N> tags in the prompt had nothing to bind to. This is why compositing a character into a clip appeared to ignore the tags entirely. The gallery "Edit with Omni" button opened an empty screen. It still emitted the clip as a referenceVideo, and selectModel now nulls that for any model declaring supportsVideoReference: false, which Omni does. applyRemixState had no editSource handling at all. Both handlers now go through a shared applyEditSource, and the gallery emits the new contract. "Use as end frame" started failing outright. That hand-off carries no model, so it lands on the persisted default - Omni, which cannot interpolate - and the new backend validator rejects the request. It now moves to Veo 3.1 with a notice, as the reference-image path already did. The purge on model switch also missed end frames that arrive as media items rather than uploaded assets. Extend Video and Concatenate Video disappeared from the mode menu on the default model, with no explanation. Extending on Omni never worked - it produced an unrelated clip rather than an extension - but removing the entries was the wrong remedy. Every mode is offered again; choosing one the active model cannot do switches to a model that can. An over-limit set of reference images survived a model switch, so every generate failed validation until the user worked out which to remove. They are now clamped, with a notice saying how many were dropped. Edit Video mode was persisted while the clip being edited was not, so returning to the page restored the mode with an empty slot and generating made a new video instead of an edit. editSource and stripSourceAudio are now persisted, and a restored Edit Video without a clip falls back. Also adds the stripSourceAudio field to VeoRequest, which the edit payload needs and which had only ever existed alongside temperature in a later commit now split into a separate PR.
Two findings from testing against the live API that callers cannot infer from the request shape. Role tags are positional over the reference images in attachment order. Edit prompts that scope the change with a specific preservation clause are frequently rejected with "This model does not support video extension", despite asking for task=edit. Across 116 calls the closing sentence decided the outcome while reference count, tags versus plain names and the staging described made no difference: "Keep the room and lighting the same." failed 13/13 on one prompt, and the same request ending "Keep everything else the same." succeeded 9/10. Whether the service genuinely reclassifies the request is unverified - that reading comes from the error text alone. What is established is that the wording controls it, so the guide states the rule and flags the mechanism as unconfirmed.
…idance Both were found by actually running the upgrade guide against a real deployment rather than just reading it. The manual frontend deploy pointed at frontend/cloudbuild.yaml, which only forwards _FIREBASE_PROJECT_ID before kicking off the real deploy build --async and reporting success immediately - it never confirms the deploy itself worked. cloudbuild-deploy.yaml, the file that actually injects secrets and pushes to Firebase Hosting, has no substitution defaults of its own, so following the doc as written would submit an incomplete build. Points at cloudbuild-deploy.yaml directly with the substitutions it needs. The post-deploy log check named only "Migrations applied successfully", which the code only logs when a migration actually runs. A deploy that carries no migration - confirmed against a live deploy - logs "Database is already up to date" instead, which the doc gave no indication was also a healthy outcome.
…Omni Anchoring a shot to a chosen first frame while attaching character sheets is the core consistency workflow for serialized drama, and it was unreachable: blocked in the DTO, unreachable in the UI, and mis-routed even if both were bypassed. The DTO rejected the pairing for every model. That rule is right for Veo, which types reference images separately from its input image and refuses both in one request, but Omni has no typed reference field - every image rides the multimodal input, so a frame plus character sheets is simply an ordered list. Verified live before relaxing it: task=image_to_video with a frame and a character sheet held the frame as frame 1 and still carried the likeness. An end frame or extension source alongside references stays rejected for every model, Omni included, since Omni can neither interpolate nor extend. The start-frame role arriving as a media item was tracked separately from the end-frame and extension roles so relaxing one does not relax the others. That comparison is by value, not identity: the field is a plain string, so an identity check against the enum silently never matches. resolve_omni_task now ranks a start image above references. Both can be sent, but only image_to_video treats the first image as the opening frame; reference_to_video would demote a deliberately chosen frame to one more reference and lose the anchor that is the entire reason for supplying it. The function had no test coverage at all, so its full truth table is now pinned. Frames to Video renders the reference strip for models declaring the new supportsFrameWithReferences capability, rather than inferring the model from unrelated flags. The payload sends both source-media lists in that mode: the frame slots carry the opening frame and the reference entries carry the cast, and sending only one silently dropped the other.
<IMAGE_REF_N> counts all images in the request and the opening frame is sent first, so the frame owns <IMAGE_REF_0> and the references start at 1. The prompt box numbered the reference strip from 0 regardless, which pointed every tag one image off as soon as a frame was attached - the same off-by-one that made role tags look broken in Edit Video, moved into the UI. The frame slot now carries its own 0 badge and the reference badges are offset, so what a user clicks is what the model receives. Established with a mirrored pair rather than assumed. Swapping which tag was told to act swapped which person acted, so the tags do bind - but inverted against the earlier reading, which is what exposed the numbering. The same test showed <FIRST_FRAME> is not recognised: it drove the wrong image. An earlier apparently-positive result for it was confounded by gendered pronouns in the prompt, which let the model resolve the roles without reading the tag at all. No badge is offered for it. Also stops offering the closing-frame slot to models that cannot interpolate. Omni declares supportsLastFrame: false and the backend rejects an end frame outright, so the slot only led to a rejected generate. Concatenate keeps its second slot, which is a clip rather than a frame.
… Omni Attaching the first reference wiped both frame slots and announced "Start/end frames and extension videos have been cleared to use reference images", so the combination this branch just enabled was impossible to reach from the UI: choosing a frame and then a character sheet threw the frame away. That clearing is a Veo constraint - it types reference images separately from its input image and refuses both in one request. Omni carries every image in one multimodal input, and a frame plus character sheets is the whole point of the pairing, so a still opening frame now survives for models that allow it. Cleared regardless, for every model: a closing frame, and any video in the slots. Omni can neither interpolate nor extend, and a video in slot 1 means Extend or Concatenate rather than an opening frame. When only the closing frame goes, the message says so instead of claiming the opening frame was cleared too.
…s on Omni The third copy of Veo's "not both" rule, and the one that actually produced "Reference images cannot be used at the same time as a start/end image or a source video." at generate time. The DTO and the prompt box were already fixed, so a request assembled correctly in the UI still failed in the worker. Veo types its reference images separately from its `image=` input and refuses both in one request; Omni carries every image in the multimodal input, so an opening frame with character sheets is valid and is how a shot is anchored while identities are held. An end frame or extension source stays rejected for Omni too, since it can neither interpolate nor extend. The guard had no test. This one fails with the old code on the exact error the user saw, and also pins what the fix is for: the request goes out as image_to_video rather than reference_to_video so the frame stays frame 1, and the frame is sent before the references so it owns <IMAGE_REF_0>.
Selecting 9:16 could still generate 16:9, and the model chip could read Gemini Omni while the request went to Veo. Both came from the same cause: display state and request state drifting apart, so the UI described one request and the payload carried another. The clear button called resetAllFilters(), which rebuilt searchRequest wholesale - forcing aspectRatio to 16:9 and generationModel to veo-3.0 - and never touched selectedAspectRatio or selectedGenerationModel. The chips kept the old values. Re-picking Omni afterwards fixed the model chip but left the ratio at 16:9 under a chip still reading 9:16, which is exactly the reported pair of symptoms. That function was unreachable on main; the clear button is what exposed it. It is replaced with clearPromptAndInputs(), which does what the control's tooltip promises: clears the prompt, negative prompt and every attachment, and leaves model, ratio, resolution and duration alone. Those are settings, not inputs, and silently switching a user's model on clear was never intended. The old function also never called resetInputs(), so attachments survived a "clear all" regardless. restoreState() had the same shape of hazard: it assigned the persisted model and ratio unconditionally but only updated the chips when a matching option existed, so stale storage could leave the two permanently out of step. Unrecognised values now fall back to a real option and update both halves together. Verified against the data rather than by reading: media item 80 recorded 16:9 and its clip really is 1280x720, item 81 recorded 9:16 and is 720x1280. The backend honours whatever it receives, so the wrong value was leaving the browser.
feat(images): expose temperature control for Gemini image models
# Conflicts: # frontend/src/app/common/components/flow-prompt-box/flow-prompt-box.component.ts # frontend/src/app/common/config/model-config.ts # frontend/src/app/home/home.component.html
…omni-first-frame-plus-refs # Conflicts: # frontend/src/app/common/config/model-config.ts
feat(video): allow an opening frame and reference images together on Omni
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #257
Fixes #258
Two issues are combined here rather than split further because the fixes
turned out to share code: the Omni edit-persistence work builds directly on
the session-state rewrite from the auth fix, both touching
video-state.service.ts. Splitting them would mean one PR silently dependson the other merging first with no way to express that other than a stacked
branch, which is worse for review than one coherent PR.
#257 - Gemini Omni video generation gaps
reach the model.
task=editmulti-turn editing support, matching thedocumented Vertex pattern of
[text, image(s), video].<IMAGE_REF_N>silently boundto the wrong image in edit mode (references were being inserted in
reverse order). Verified against the live API with 3 references and a
positional swap - regression test included, confirmed to fail against the
old code.
replaces it with proper Edit Video support.
hardcoded model checks, so unsupported inputs (audio reference,
interpolation, extension) aren't offered for models that reject them.
a misleading "does not support video extension" error depending on prompt
wording alone. Isolated via 116 live API calls varying one variable at a
time; full detail in
UPGRADING.md.gating change are fixed in the same PR (reference images not being sent
in Edit Video, the gallery "Edit with Omni" handoff, end-frame handling,
and Extend Video/Concatenate Video disappearing from the mode menu).
#258 - Session expiry and data loss
Four defects in frontend auth/session handling:
refreshed, so every request failed once the ~1 hour token expired.
surfacing an error.
isLoggedIn(), a predicate, had a navigation side effect that couldbounce users off whatever page they were on.
currentUserduring Firebase's async session restore wastreated as "logged out," logging users out on a plain page refresh.
Also fixes the video generation page discarding attached
images/video/audio on every save/restore cycle, which compounded the
perceived data loss whenever a spurious logout occurred.
pytestbackend: 384 passed;ng buildfrontend: clean)UPGRADING.mdcovers migration/compatibility notes,the edit-prompt wording gotcha, and the reference-ordering fix