feat: fs-cache #1645

daukadolt · 2025-10-23T15:27:19Z

Description

This PR implements filesystem-based cache for cognee short-term memory (#1545)

Type of Change

Bug fix (non-breaking change that fixes an issue)
New feature (non-breaking change that adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
Documentation update
Code refactoring
Performance improvement
Other (please specify):

Screenshots/Videos (if applicable)

Pre-submission Checklist

I have tested my changes thoroughly before submitting this PR
This PR contains minimal changes necessary to address the issue/feature
My code follows the project's coding standards and style guidelines
I have added tests that prove my fix is effective or that my feature works
I have added necessary documentation (if applicable)
All new and existing tests pass
I have searched existing PRs to ensure this change hasn't been submitted already
I have linked any relevant issues in the description
My commits have clear and descriptive messages

DCO Affirmation

I affirm that all code in every commit of this pull request conforms to the terms of the Topoteretes Developer Certificate of Origin.

pull-checklist · 2025-10-23T15:27:23Z

Please make sure all the checkboxes are checked:

I have tested these changes locally.
I have reviewed the code changes.
I have added end-to-end and unit tests (if applicable).
I have updated the documentation and README.md file (if necessary).
I have removed unnecessary code and debug statements.
PR title is clear and follows the convention.
I have tagged reviewers or team members for feedback.

coderabbitai · 2025-10-23T15:27:44Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

✨ Finishing touches

🧪 Generate unit tests (beta)

Create PR with unit tests
Post copyable unit tests in a comment
Commit unit tests in branch feat/fs-cache

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

daukadolt · 2025-10-23T16:08:15Z

Looks like redislite has issues with win 32

cognee/infrastructure/databases/cache/get_cache_engine.py

pyproject.toml

Vasilije1990 · 2025-10-28T07:57:31Z

@daukadolt do you need a review of this one?

… for Redis integration

daukadolt · 2025-10-28T12:14:08Z

Only distributed cognee is failing due to poetry.lock

cognee/infrastructure/databases/cache/get_cache_engine.py

cognee/infrastructure/databases/cache/fscache/FsCacheAdapter.py

hajdul88 · 2025-10-28T17:02:59Z

I tested it.

Problems I found:
-This is memory based session implementation not file based. Therefore session are not gonna persist. I think the objective was to create file based adapter.
-By setting this fs based solution the behavior of the following tests may change so please make sure they run with Redis and not with this solution
-Concurrent Subprocess Access Test
-Conversation sessions test

Also I think it would be a good idea to cover this adapter with the Conversation sessions tests so we test both Redis and fs behavior.

daukadolt · 2025-10-28T18:21:02Z

Just validated it works with kuzu multiprocessing test.

I'll make a copy of the test for FS caching

hajdul88 · 2025-10-28T21:45:12Z

I fixed the two redis tests. Tested sessions they don't work now.

… filesystem caching too

…dantic model

borisarzentar · 2025-10-30T11:08:49Z

cognee/infrastructure/databases/cache/fscache/FsCacheAdapter.py

+        data_root_directory = storage_config["data_root_directory"]
+        cache_directory = os.path.join(data_root_directory, ".cognee_fs_cache", lock_key)
+
+        os.makedirs(cache_directory, exist_ok=True)


Do we want this to always be in the local filesystem?

borisarzentar · 2025-10-30T11:11:06Z

cognee/infrastructure/databases/cache/fscache/FsCacheAdapter.py

+
+        self.lock.release()
+
+    async def add_qa(


Do we have this in redis adapter as well?

borisarzentar · 2025-10-30T11:12:11Z

cognee/tests/integration/caching/test_fs_caching.py

+
+
+def test_can_release_lock():
+    cache = FSCacheAdapter(timeout=5, blocking_timeout=10, lock_key="test_release")


This one is the same as previous.

borisarzentar · 2025-10-30T11:12:52Z

cognee/tests/integration/caching/test_fs_caching.py

+def test_releasing_non_acquired_lock():
+    try:
+        cache = FSCacheAdapter(timeout=5, blocking_timeout=10, lock_key="test_non_acquired")
+        cache.acquire_lock()


We acquire it, how is it non acquired?

borisarzentar · 2025-10-30T11:16:12Z

cognee/tests/integration/caching/test_fs_caching.py

+@pytest.mark.asyncio
+async def test_closing_connection():
+    cache = FSCacheAdapter(timeout=1, blocking_timeout=10, lock_key="test_closing")
+    await cache.close()


Why do we call cache.cache.close() above and cache.close() here?

daukadolt added 3 commits October 23, 2025 15:35

feat: init FsCacheAdapter

d8ca869

feat: functioning FsCacheAdapter

678b93a

add FsCacheAdapter into cache_engine

96e8cce

daukadolt requested a review from hajdul88 October 23, 2025 15:27

daukadolt marked this pull request as draft October 24, 2025 13:15

daukadolt added the core-team label Oct 24, 2025

daukadolt and others added 2 commits October 24, 2025 14:51

replace redislite with fakeredis

2e43e48

Merge branch 'dev' into feat/fs-cache

7694fbf

daukadolt changed the title ~~Feat/fs-cache (redislite)~~ Oct 27, 2025

daukadolt added 2 commits October 27, 2025 13:47

chore: install fakeredis with lua support

c315d5c

Migrate FsCacheAdapter.py from redislite to fakeredis

1556e68

daukadolt marked this pull request as ready for review October 27, 2025 13:49

daukadolt commented Oct 27, 2025

View reviewed changes

cognee/infrastructure/databases/cache/get_cache_engine.py Outdated Show resolved Hide resolved

daukadolt commented Oct 27, 2025

View reviewed changes

pyproject.toml Show resolved Hide resolved

daukadolt added 2 commits October 27, 2025 17:33

tests: FS based cache test

c61b9ef

chore: print whole qa in test_fs_cache_conversation_history.py

31b3c3d

daukadolt added 2 commits October 28, 2025 10:43

fix: update cache backend environment variable to use 'CACHE_BACKEND'…

465f307

… for Redis integration

Merge branch 'dev' into feat/fs-cache

9e8da7e

chore: update poetry.lock

bcf359d

hajdul88 reviewed Oct 28, 2025

View reviewed changes

cognee/infrastructure/databases/cache/get_cache_engine.py Outdated Show resolved Hide resolved

hajdul88 reviewed Oct 28, 2025

View reviewed changes

cognee/infrastructure/databases/cache/fscache/FsCacheAdapter.py Show resolved Hide resolved

daukadolt added 2 commits October 28, 2025 13:24

chore: add cache_backend into cache_config

41dfaea

chore: keep caching enabled by default

a20fc9b

feat: process agnostic filesystem based caching

40f308d

daukadolt added 5 commits October 28, 2025 17:52

tests: add unit test for validating that lock timeout works

8b6fc1c

chore: update poetry.lock

44afec0

chore: cleanup custom debug error, explicitly expire old key-vals

23690c8

expire on init as well

b46960e

config: do not enable caching by default

0574a66

hajdul88 added 2 commits October 28, 2025 22:33

chore: changes cache backend to redis instead of default

dd5ff44

Merge branch 'dev' into feat/fs-cache

0690a30

daukadolt added 3 commits October 28, 2025 22:50

typo fix and ensure lock key is not None

f5736c3

chore: don't raise exception when key doesn't exist since it's expected

bb23655

Validate that test_concurrent_subprocess_access.py can now run with…

9c22257

… filesystem caching too

daukadolt changed the title ~~Feat/fs-cache (fakeredis)~~ Oct 28, 2025

tests: update test_cache_config.py after new fields added to the Py…

dfd9c49

…dantic model

borisarzentar changed the title ~~Feat/fs-cache~~ Oct 30, 2025

borisarzentar reviewed Oct 30, 2025

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: fs-cache #1645

feat: fs-cache #1645

Uh oh!

daukadolt commented Oct 23, 2025 •

edited

Loading

pull-checklist bot commented Oct 23, 2025

coderabbitai bot commented Oct 23, 2025 •

edited

Loading

Review skipped

daukadolt commented Oct 23, 2025 •

edited

Loading

Uh oh!

Uh oh!

Vasilije1990 commented Oct 28, 2025

daukadolt commented Oct 28, 2025 •

edited

Loading

Uh oh!

Uh oh!

hajdul88 commented Oct 28, 2025

daukadolt commented Oct 28, 2025

hajdul88 commented Oct 28, 2025

borisarzentar Oct 30, 2025

borisarzentar Oct 30, 2025

borisarzentar Oct 30, 2025

borisarzentar Oct 30, 2025

borisarzentar Oct 30, 2025

Labels

5 participants



		def test_can_release_lock():
		cache = FSCacheAdapter(timeout=5, blocking_timeout=10, lock_key="test_release")

feat: fs-cache #1645

Are you sure you want to change the base?

feat: fs-cache #1645

Uh oh!

Conversation

daukadolt commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of Change

Screenshots/Videos (if applicable)

Pre-submission Checklist

DCO Affirmation

pull-checklist bot commented Oct 23, 2025

Please make sure all the checkboxes are checked:

coderabbitai bot commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

daukadolt commented Oct 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Vasilije1990 commented Oct 28, 2025

daukadolt commented Oct 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

hajdul88 commented Oct 28, 2025

daukadolt commented Oct 28, 2025

hajdul88 commented Oct 28, 2025

borisarzentar Oct 30, 2025

Choose a reason for hiding this comment

borisarzentar Oct 30, 2025

Choose a reason for hiding this comment

borisarzentar Oct 30, 2025

Choose a reason for hiding this comment

borisarzentar Oct 30, 2025

Choose a reason for hiding this comment

borisarzentar Oct 30, 2025

Choose a reason for hiding this comment

Labels

5 participants

daukadolt commented Oct 23, 2025 •

edited

Loading

coderabbitai bot commented Oct 23, 2025 •

edited

Loading

daukadolt commented Oct 23, 2025 •

edited

Loading

daukadolt commented Oct 28, 2025 •

edited

Loading