Skip to content
This repository was archived by the owner on Apr 14, 2026. It is now read-only.

fix: Allow continously used caches to be scaled to 2 copies - #153

Open
haiminh2001 wants to merge 1 commit into
kserve:mainfrom
haiminh2001:main
Open

fix: Allow continously used caches to be scaled to 2 copies#153
haiminh2001 wants to merge 1 commit into
kserve:mainfrom
haiminh2001:main

Conversation

@haiminh2001

@haiminh2001 haiminh2001 commented Nov 23, 2024

Copy link
Copy Markdown

Motivation

  • The earlierUseIteration (i1) is updated when i1 are not in range, including two cases: i1 is smaller than the lower bound (40m ago) and larger than the upper bound (7m ago).
  • Therefore, if a model is used continously, the earlierUseIteration (i1) will always be larger than the upper bound and will never be in range. To be scaled to 2 copies, the model has to be used once, wait for 7 minutes WITHOUT ANY other usage, if any other usage appear, it will interrupt (update the earlierUseIteration to the lastUsedIteration).

Modifications

  • Only update the earlierUseIteration if i1 is smaller than the lower bound, not when i1 is larger than the upper bound.

Results:

  • A model that is continuously used will be able to two copies, maintain the HA of that model.
Motivation:

- The earlierUseIteration (i1) is updated when i1 are not in range, including two cases: i1 is smaller than the lower bound (40m ago) and larger than the upper bound (7m ago).
- Therefore, if a model is used continously, the earlierUseIteration (i1) will always be larger than the upper bound and will never be in range. To be scaled to 2 copies, the model has to be used once, wait for 7 minutes WITHOUT ANY other usage, if any other usage appear, it will interrupt (update the earlierUseIteration to the lastUsedIteration).

Modifications:

- Only update the earlierUseIteration if i1 smaller than the lower bound, not when i1 larger than the upper bound.

Results:
- A model that is continously used will be abled to two copies, maintain the HA of that model.

Signed-off-by: haiminh2001 <haiminhnguyen2001@gmail.com>
@Legion2

Legion2 commented Feb 19, 2025

Copy link
Copy Markdown
Contributor

Thanks for the fix, will this be part of the next release?

@spolti
spolti requested review from ckadner and rafvasq February 19, 2025 16:12
@haiminh2001

Copy link
Copy Markdown
Author

@Legion2 thank you for noticing my PR, it has been around for quite a while :)). I have also applied this fix in my company 2 months ago, it works just as what I expect. If you need further elaboration on this logic, please ask. I would be so happy if this fix can be part of the next release so that I will no longer have to maintain a custom build of modelmesh in my company.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

2 participants