Skip to content

triedb/pathdb: introduce file-based state journal #32060

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 18 commits into
base: master
Choose a base branch
from
Open

triedb/pathdb: introduce file-based state journal #32060

wants to merge 18 commits into from

Conversation

jsvisa
Copy link
Contributor

@jsvisa jsvisa commented Jun 18, 2025

allow write trie journal into a local file instead of disk, used when the trie jounral is >= 4GB, ref gballet#546 (comment)
@jsvisa jsvisa requested a review from rjl493456442 as a code owner June 18, 2025 08:01
@rjl493456442 rjl493456442 self-assigned this Jun 18, 2025
rjl493456442
rjl493456442 reviewed Jun 19, 2025
View reviewed changes
jsvisa
jsvisa commented Jun 19, 2025
View reviewed changes
jsvisa
jsvisa commented Jun 19, 2025
View reviewed changes
@jsvisa jsvisa force-pushed the journal-into-file branch from 2ca455d to 2d222b2 Compare June 19, 2025 13:35
@rjl493456442 rjl493456442 mentioned this pull request Jun 20, 2025
Open
2 tasks
@jsvisa jsvisa force-pushed the journal-into-file branch from 2d222b2 to 9abf10a Compare June 23, 2025 09:14
@fjl
Copy link
Contributor

fjl commented Jun 24, 2025

The idea is good, but the path should not be configurable. We should just resolve the path in the datadir and store the file there unconditionally.
@fjl fjl removed the status:triage label Jun 24, 2025
@jsvisa jsvisa force-pushed the journal-into-file branch from 9abf10a to 516c1b3 Compare June 25, 2025 07:04
@rjl493456442
Copy link
Member

rjl493456442 commented Jun 27, 2025 •
edited
Loading

This pull request is backward-compatible:

  • Upgrade from old version, the journal will be resolved from database and persist into file
  • Downgrade to old version, Geth will attempt to resolve journal from database only and drop in-memory layers if not matched

The journal directory will be $DATADIR/triedb, with $DATADIR/triedb/merkle.journal for legacy merkle trie database and $DATADIR/triedb/verkle.journal for verkle respectively.
@rjl493456442
Copy link
Member

logs:

INFO [06-27|14:17:56.460] Load database journal from file          path=/home/gary/eth-sepolia/el/geth/triedb/merkle.journal

...

INFO [06-27|14:21:04.978] Persisting dirty state                   head=5,276,356 root=b0d1cb..19ba68 layers=1551
INFO [06-27|14:21:06.556] Persisted dirty state to file            path=/home/gary/eth-sepolia/el/geth/triedb/merkle.journal size=342.21MiB elapsed=1.578s
rjl493456442
rjl493456442 previously approved these changes Jun 27, 2025
View reviewed changes
Copy link
Member

@rjl493456442 rjl493456442 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sgtm
jsvisa and others added 15 commits July 1, 2025 10:09
@jsvisa @rjl493456442
all: dump trie journal into file (ethereum#546)
9db6f07 
* all: load/dump trie disklayer into file

Signed-off-by: jsvisa <delweng@gmail.com>

* cmd: set cache.trie.journal=trie-disklayer.rlp as default

Signed-off-by: jsvisa <delweng@gmail.com>

* journal: directly write into file

Signed-off-by: jsvisa <delweng@gmail.com>

---------

Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa @rjl493456442
cmd: no-legacy cache.trie.journal
bc358d5 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa @rjl493456442
disable by default
82d57f9 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa @rjl493456442
atomic file write
c7d2d30 
Signed-off-by: jsvisa <delweng@gmail.com>
@rjl493456442
cmd, core, eth, triedb/pathdb: improve journal file flushing
22e1c62 
Signed-off-by: jsvisa <delweng@gmail.com>
@rjl493456442
eth/ethconfig: regenerate config
ec518e3 
Signed-off-by: jsvisa <delweng@gmail.com>
@rjl493456442
cmd/geth: prevent double definition
4b5cf34 
Signed-off-by: jsvisa <delweng@gmail.com>
@rjl493456442
cmd, eth: fix journal path
f074797 
Signed-off-by: jsvisa <delweng@gmail.com>
@rjl493456442
triedb/pathdb: fix tests
d5da02a 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa @rjl493456442
Apply suggestions from code review
525aacb 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa @rjl493456442
dup assign
fef157d 
Signed-off-by: jsvisa <delweng@gmail.com>
@jsvisa @rjl493456442
set triedb.journal as a switch
3cc8e23 
Signed-off-by: jsvisa <delweng@gmail.com>
@rjl493456442
triedb/pathdb: change to the default datadir
6bb3dfc
@rjl493456442
eth, triedb: fix mkdir
7afd7d0
@rjl493456442
triedb/path: update logs
e9689c3
@rjl493456442 rjl493456442 changed the title trie: write trie journal into disk Jul 1, 2025
@rjl493456442 rjl493456442 added this to the 1.16.1 milestone Jul 1, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
3 participants
@jsvisa @fjl @rjl493456442