Fix process_large_node_recursively discarding real children on large … - #420
Fix process_large_node_recursively discarding real children on large …#420Ametist3d wants to merge 1 commit into
Conversation
…parent nodes. post_processing/list_to_tree give a parent node an end_index that only reaches its first child's start_index, not the full subtree span. When that preamble alone exceeded max_page_num_each_node/max_token_num_each_node, process_large_node_recursively re-ran meta_processor over just that prefix and overwrote node['nodes'] with the result, silently discarding the node's already-discovered children and everything under them. Skip the large-node split when the node already has children and recurse into the existing children instead. Adds a regression test asserting the original children survive and meta_processor is not called in that case.
|
Nice fix for the child-discarding bug. One issue: this also removes the only place the function recurses into newly created children. When node has no pre-existing nodes (the normal case — this is how splitting happens at all), the top if node.get('nodes') guard is skipped, the split below assigns node['nodes'], and the function returns without ever calling process_large_node_recursively on those new children. Any child that's itself still over max_page_num_each_node/max_token_num_each_node will never get split further — it'll stay an oversized leaf. I think the old bottom block (or an equivalent recursive call) still needs to run for the "just split in this call" path, guarded so it doesn't re-run for the "already had real children" early-return path. |
2 participants
Fix process_large_node_recursively discarding real children on large parent nodes
post_processing/list_to_tree give a parent node an end_index that only reaches its first child's start_index, not the full subtree span. When that preamble alone exceeded max_page_num_each_node/max_token_num_each_node, process_large_node_recursively re-ran meta_processor over just that prefix and overwrote node['nodes'] with the result, silently discarding the node's already-discovered children and everything under them.
Skip the large-node split when the node already has children and recurse into the existing children instead. Adds a regression test asserting the original children survive and meta_processor is not called in that case.