docs(cli): document all 106 CLI commands - #19799
Conversation
a11a0e2 to
70a311d
Compare
|
Is this a 2 part PR or? The description says part of. If this is a 2 part PR, please make it clear in the title this is Part 1 of N, where N should be clearly defined. |
Closes apache#15376 (HUDI-4689). hudi-cli exposes 106 commands across 26 command classes. cli.md covered 86 of them, and of those only 30 had their options documented anywhere; the other 56 existed on the page as a single line in the embedded help output. Twenty were absent entirely, including the whole locks audit family, both timeline commands and both diff commands. Every command name, option, default and behaviour below was read from the @ShellMethod and @ShellOption declarations on master, not from the help strings, which have drifted. The help listing now carries all 106 commands, inserted at their alphabetical positions rather than regenerated, so the diff stays reviewable. A new "Command reference" section documents every command, grouped into nine areas, with each command's options and defaults. It also records which entries are aliases rather than distinct commands: refresh, metadata refresh, commits refresh, cleans refresh and savepoints refresh are five names for one method, and temp query / temp_query, temp delete / temp_delete and temps show / temps_show are spelling variants. New worked sections for what had no documentation at all: Auditing Storage Locks locks audit enable / disable / status / validate / cleanup, the layout under .hoodie/.locks, and what validate reports as an error versus a warning Inspecting the Timeline timeline show active / incomplete and the two metadata timeline equivalents, with the output columns Diffing a File or Partition diff file, diff partition Repairing a Table rename partition, repair deprecated partition, repair show empty commit metadata Changing the Meta Fields Mode table set-meta-fields-mode and its two guards Four more are documented in the sections they belong to: trigger archival under Archived Commits, show restore / show restores under Savepoint and Restore, kerberos kdestroy under the Kerberos section, and metadata delete-record-index under Record Index Lookup. Three things the source read turned up that the help text does not tell you: - diff file and diff partition ignore --startTs and --endTs unless --includeArchivedTimeline is also true. CLIUtils#getTimelineInRange applies the bounds only on the archived branch and returns the whole active timeline otherwise, so a narrow range silently does nothing. - repair show empty commit metadata returns void and reports through log.warn, so it renders no result table. - rename partition physically removes the old partition directory after its delete_partition write; repair deprecated partition leaves the emptied directory. Angle brackets in generated text are escaped: the rename partition usage string contains <oldPartition>, which MDX otherwise parses as a JSX tag and fails the build on.
4e1cc4c to
2dd66c7
Compare
|
Hi @voonhous I have update the PR description and documentation changes. |
hudi-agent
left a comment
There was a problem hiding this comment.
Thanks for the docs update! This is a thorough and impressively accurate expansion of cli.md — I cross-checked the command names, options, defaults, and behavioral notes (diff range semantics, lock-audit paths and verdicts, delete-record-index backup path, set-meta-fields-mode guards, trigger archival, etc.) against the @ShellMethod/@ShellOption declarations on master and they line up. The only concern is a handful of reference entries that print an unresolved Java constant as the --sparkMaster default instead of its value. Please route to a Hudi committer/PMC member (e.g. @yihua) for a final pass before merge.
| - **`cleans show`** Show the cleans. | ||
| <br />Options: `--limit` (`-1`), `--sortBy`, `--startTs`, `--endTs`, `--includeArchivedTimeline` (`false`), `--desc` (`false`), `--headeronly` (`false`) | ||
| - **`clustering run`** Run Clustering. | ||
| <br />Options: `--sparkMaster` (`SparkUtil.DEFAULT_SPARK_MASTER`), `--sparkMemory` (`4g`), `--parallelism` (`1`), `--retry` (`1`), `--clusteringInstant`, `--propsFilePath`, `--hoodieConfigs` |
There was a problem hiding this comment.
🤖 This entry lists the --sparkMaster default as the raw Java constant SparkUtil.DEFAULT_SPARK_MASTER rather than its value. That constant resolves to yarn (SparkUtil.DEFAULT_SPARK_MASTER = "yarn"), so it would help to substitute the concrete value here — the way the other Spark commands in this reference already do (e.g. compaction/marker show local). The same unresolved constant appears on the clustering schedule / clustering scheduleAndExecute entries (lines 1177, 1179) and on metadata create / metadata init / metadata list-partitions (lines 1246, 1252, 1256). It might also be worth a one-line note that this default is yarn, since that differs from the local default used by the compaction and other Spark commands and could surprise a reader running these on a non-YARN setup.
Describe the issue this Pull Request addresses
Closes #15376 (JIRA: HUDI-4689, epic
HUDI-1388), "Add documentation for all CLI commands".
hudi-cliexposes 106 commands across 26 command classes.website/docs/cli.mdhad two gaps:discover they exist. The whole
locks auditfamily, bothtimelinecommands and bothdiffcommands were amongthem.
helpoutput, with their options documentednowhere on the page.
Summary and Changelog
One file,
website/docs/cli.md.1. The
helplisting now carries all 106 commands. Entries were inserted at their alphabetical positionsrather than the block being regenerated, so the diff stays reviewable.
2. A new
Command referencesection documents every command, grouped into nine areas, with each command'soptions and their defaults. It also records which entries are aliases rather than distinct commands:
refresh,metadata refresh,commits refresh,cleans refreshandsavepoints refreshare five names for one method thatreloads table metadata, and
temp query/temp_query,temp delete/temp_deleteandtemps show/temps_showare spelling variants of three commands.3. New worked sections for the commands that had no documentation at all:
Auditing Storage Lockslocks auditcommands, the layout under.hoodie/.locks, and whatvalidatereports as an error versus a warningInspecting the Timelinetimeline show active/incompleteand the twometadata timelineequivalents, with their output columnsDiffing a File or Partitiondiff file,diff partitionRepairing a Tablerename partition,repair deprecated partition,repair show empty commit metadataChanging the Meta Fields Modetable set-meta-fields-modeand its two guardsFour more are documented in the sections they belong to:
trigger archivalunder Archived Commits,show restore/show restoresunder Savepoint and Restore,kerberos kdestroyunder the Kerberos section, andmetadata delete-record-indexunder Record Index Lookup.Every command name, option, default and behaviour was read from the
@ShellMethodand@ShellOptiondeclarationson
master, not from the help strings, which have drifted.Things the source read turned up that the help text does not say
diff fileanddiff partitionignore--startTsand--endTsunless--includeArchivedTimelineis alsotrue.CLIUtils#getTimelineInRange(:46-62) applies the bounds only on the archived branch and returns thewhole active timeline otherwise, so a narrow range silently does nothing. The help text advertises the defaults
with no hint of the precondition.
repair show empty commit metadatareturnsvoidand reports throughlog.warn, so it renders no resulttable; its findings go to the CLI log.
rename partitionphysically removes the old partition directory after itsdelete_partitionwrite(
SparkMain#renamePartition), whilerepair deprecated partitionleaves the emptieddefaultdirectory. The twoare otherwise the same shape.
Verification
The extraction behind the command reference was validated against the source as well: the 18 commands recorded as
taking no options were each confirmed to declare none, and the captured option count reconciles exactly with the
404
@ShellOptionoccurrences in the tree once the two alias methods that double-count are accounted for.Sections and the reference were confirmed in the rendered
build/docs/next/cli/index.html, including that theescaped angle brackets in the
rename partitionusage string render as literal<and>.Impact
Documentation only; no code or config changes. The page is the current (
next) docs version, which tracksmaster, where these commands live. Versioned copies underwebsite/versioned_docs/are deliberately untouched,since several of these commands do not exist in those releases.
Risk Level
none
Documentation Update
This is the documentation update.
Contributor's checklist