Skip to content

Conversation

@DickTracyII
Copy link
Collaborator

@DickTracyII DickTracyII commented Sep 16, 2025

Add documentation for Invoke-ScubaCached

🗣 Description

This PR introduces documentation for the Invoke-ScubaCached command.
The new docs describe what the command does, its parameters, usage scenarios, and how it differs from Invoke-Scuba.

Documentation includes:

  • Purpose and functionality of Invoke-ScubaCached.
  • Examples of running scans using cached data.
  • Guidance on when to use cached vs. fresh scans.
  • Notes on limitations and environment considerations.

💭 Motivation and context

While Invoke-ScubaCached has been available as part of ScubaGear for some time, its functionality was not documented.
This created confusion for users on:

  • How to reuse cached data effectively.
  • Which scenarios benefit from cached scans (e.g., reducing API calls, offline reviews).
  • Differences between Invoke-Scuba (live scan) and Invoke-ScubaCached (cached evaluation).

Adding documentation addresses these gaps and supports user adoption, particularly for environments with strict API throttling or offline validation requirements. This documentation also provided guidance for Developers and those troubleshooting ScubaGear results.

This resolves [Issue #1516]
Closes #1516

🧪 Testing

Validation steps performed:

  • Reviewed command behavior by running Invoke-ScubaCached against existing cached scan data.
  • Verified that documented examples match actual command behavior.
  • Confirmed that invalid/missing cache paths trigger the expected error messages.

Steps for reviewers to test the docs:

  1. Run a fresh scan with Invoke-Scuba to generate cached data.
  2. Execute Invoke-ScubaCached using the cache output.
  3. Confirm results match documentation examples.
  4. Verify that error handling for invalid cache paths aligns with documented behavior.

✅ Pre-approval checklist

  • This PR has an informative and human-readable title.
  • PR targets the correct parent branch (e.g., main or release-name) for merge.
  • Changes are limited to a single goal - eschew scope creep!
  • Changes are sized such that they do not touch excessive number of files.
  • All future TODOs are captured in issues, which are referenced in code comments.
  • These code changes follow the ScubaGear content style guide.
  • Related issues these changes resolve are linked preferably via closing keywords.
  • All relevant type-of-change labels added.
  • All relevant project fields are set.
  • All relevant repo and/or project documentation updated to reflect these changes.
  • Unit tests added/updated to cover PowerShell and Rego changes.
  • Functional tests added/updated to cover PowerShell and Rego changes.
  • All relevant functional tests passed.
  • All automated checks (e.g., linting, static analysis, unit/smoke tests) passed.

✅ Pre-merge checklist

  • PR passed smoke test check.

  • Feature branch has been rebased against changes from parent branch, as needed

    Use Rebase branch button below or use this reference to rebase from the command line.

  • Resolved all merge conflicts on branch

  • Notified merge coordinator that PR is ready for merge via comment mention

  • Demonstrate changes to the team for questions and comments.
    (Note: Only required for issues of size Medium or larger)

✅ Post-merge checklist

  • Feature branch deleted after merge to clean up repository.
  • Verified that all checks pass on parent branch (e.g., main or release-name) after merge.
@DickTracyII DickTracyII added this to the Nemo milestone Sep 16, 2025
@DickTracyII DickTracyII self-assigned this Sep 16, 2025
@DickTracyII DickTracyII added the documentation This issue or pull request improves or adds to documentation label Sep 16, 2025
@DickTracyII DickTracyII requested a review from AConant September 18, 2025 14:46
@DickTracyII DickTracyII requested a review from buidav September 22, 2025 17:18
Copy link
Collaborator

@MichaelHicks-MSFT MichaelHicks-MSFT left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Went through all the examples in scubacached.md and verified everything worked as expected.

@mitchelbaker-cisa mitchelbaker-cisa modified the milestones: Nemo, Orca Oct 6, 2025
Copy link
Collaborator

@mitchelbaker-cisa mitchelbaker-cisa left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good overall. Could we add one more example for how Invoke-SCuBACached could be used when testing Rego on injected settings that can't be tested in a live tenant?

We could use something like the authentication policy migration settings for AAD 3.4 as an example which @MichaelHicks-MSFT previously worked on in PR #1778. Testing for this PR involved using Invoke-SCuBACached to modify the JSON output.

DickTracyII and others added 3 commits October 14, 2025 12:21
Co-authored-by: mitchelbaker-cisa <149098823+mitchelbaker-cisa@users.noreply.github.com>
@DickTracyII
Copy link
Collaborator Author

Looks good overall. Could we add one more example for how Invoke-SCuBACached could be used when testing Rego on injected settings that can't be tested in a live tenant?

We could use something like the authentication policy migration settings for AAD 3.4 as an example which @MichaelHicks-MSFT previously worked on in PR #1778. Testing for this PR involved using Invoke-SCuBACached to modify the JSON output.

@mitchelbaker-cisa I added a section for this. please review. @MichaelHicks-MSFT make sure my example makes sense too

- [Power Platform](docs/troubleshooting/power.md)
- [Microsoft Graph](docs/troubleshooting/graph.md)
- [Network & Proxy](docs/troubleshooting/proxy.md)
| Topic | Resource | Notes |
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the concept of a table but for the nested product-specific issues the format is confusing. Can we switch back to the list format?

Image
- [Multiple Tenants](docs/troubleshooting/tenants.md)
- [Product-Specific Issues](docs/troubleshooting/)
  - [Defender](docs/troubleshooting/defender.md)
  - [Exchange Online](docs/troubleshooting/exchange.md)
  - [Power Platform](docs/troubleshooting/power.md)
  - [Microsoft Graph](docs/troubleshooting/graph.md)
- [Network & Proxy](docs/troubleshooting/proxy.md)
- [Cached Execution](docs/execution/scubacached.md)
Invoke-SCuBACached -ProductNames aad -OutPath "C:\ScubaResults\AAD_Policy_Test"
```

2. Manually modify the `ProviderSettingsExport.json` file to inject test scenarios (example):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

While ProviderSettingsExport.json can be used, it is the legacy file option. Switch this to ScubaResults_<guid>.json instead.


When testing Rego policies against specific configuration scenarios that can't be replicated in a live tenant, you can modify the exported JSON data and use cached mode to test the policy logic.

For example, when testing AAD authentication policy migration settings
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We could define a couple different scenarios here.

For example, when testing AAD authentication policy migration settings:

// Update the "PolicyMigrationState" property to test with other values such as "preMigration", "migrationComplete", or "migrationInProgress"
{
  "authentication_method_policy":  {
    "AuthenticationMethodConfigurations@odata.context":  "https://graph.microsoft.com/beta/$metadata#policies/authenticationMethodsPolicy/authenticationMethodConfigurations",
    "PolicyVersion":  "1.5",
    "PolicyMigrationState":  null,
    "DisplayName":  "Authentication Methods Policy"
  }
}
Or when testing AAD conditional access policy settings:

// Update the "State" property to test with other values such as "Report only" or "disabled"
{
  "conditional_access_policies": [
    {
      "DisplayName": "Test Migration Policy",
      "State": "enabled",
      "Conditions": {
        "Users": {
          "IncludeUsers": ["All"]
        }
      },
      "GrantControls": {
        "BuiltInControls": ["mfa"]
      }
    }
  ]
}

`Invoke-SCuBACached` is a specialized version of the main ScubaGear function that allows you to run security assessments in two distinct modes:

1. **Full mode** - Works identically to `Invoke-SCuBA` (exports provider data, runs Rego analysis, and generates reports)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not exactly. One thing Invoke-Scubacached does different than Invoke-SCuBA even when run regularly is how the output artifacts are grouped together.

When Invoke-SCuBA is executed a directory folder of the form M365BaselinConformance_{datetimestamp} is generated in the file path specified by the OutPath parameter. The ScubaGear outputs (json, html) files are put under this folder.

When Invoke-ScubaCached is executed the directory folder (M365BaselineConformance) is NOT created and the SCuBA outputs (JSON, HTML files etc.) are put directly into the file path specified by OutPath.

The fact a M365BaselineConformance folder is not created by Invoke-ScubaCached was it to make it easier to do rego develpment without having to rerun the providers.
Run full mode once to pull in a provider JSON.
Then on each subsequent run just run in cached mode. The user doesn't need to mess with OutPath.

Comment on lines +229 to +233
### Development and Testing

- Use cached mode during development to speed up iteration cycles
- Keep a "golden" export for consistent testing across different configurations
- Use descriptive output folder names to organize different test scenarios
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Specifically for Rego development. This was what the function was orginally made for.
A developer would be able to work on the rego then run ScubaCached to quickly see their rego changes on the HTML report. Without having to wait for the entire provider to run.

Our own main use case for this this function right now is the functional test pipeline. The code for the pipeline makes HEAVY use of this function to manipulate cached JSON to test SCuBA baseline policies that can't be tested safely. Such as manipulating MFA conditional access policies.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation This issue or pull request improves or adds to documentation

5 participants