feat(access): the ADR5 grant/consent/visibility layer - #900
Open
jirhiker wants to merge 12 commits into
Open
Conversation
Answers PUB-D14 / PERM-D3 from the "Who May See What, and Who May Do What" whitepaper: two grant tables, one enforcement engine. Landowner publication consent and internal permission grants are stored separately, because they are decided by different people on different authority and revoked by different events. Both are evaluated by a single visibility layer and projected through a single field-allowlist chokepoint, because distributed enforcement has already drifted here: baba91fe5e83 fixed ogc_waterlevels publishing readings from draft and private wells -- with the well's name and coordinates attached -- while ogc_water_chemistry had gated on the parent thing since d9e0f1a2b3c4. Twelve views carry their own copy of the rule today. Leaves open, on purpose: data type granularity, whether field protection is removal or also transformation, where coarse group membership lives, what each audit event records, and who owns the never-public field list. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Two vocabulary fixes from ADR5. Neither changes an API response shape. The table named permission_history records a landowner's consent to physical site access -- sampling, equipment installation -- which is a domain fact, not authorization. It is now field_access_consent (model FieldAccessConsent). The permission_type and permission_allowed columns keep their names, since permission_type is lexicon-backed and both are published on Thing responses under the unchanged `permissions` key. db/permission.py's Permission and base.py's PermissionMixin are deleted. They were dead: never imported in db/__init__.py, no `permission` table in any migration, no callers. release_status was one column carrying two axes -- its lexicon lists draft, public, private, published and archived next to provisional and final -- so a record could not be public and provisional at once, which San Acacia data is. ReleaseMixin now carries data_maturity alongside it. That is not a new vocabulary: the lexicon category (provisional, in review, approved) and the column already existed on transducer_observation, whose duplicate declaration is removed. Nullable, NULL meaning not stated. No data moves. Rows already marked release_status='provisional' are left alone; reassigning them across the two axes needs a decision per row and is tracked in ADR5. Migration e7c1a9f4b2d8 renames the table (with its primary key constraint and sequence, so the old name stops appearing in errors) and adds data_maturity to the 34 other ReleaseMixin tables and the 7 continuum version tables. Verified down and back up against ocotilloapi_test. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The storage and evaluation half of ADR5. No existing endpoint changes behavior: the layer goes behind one service before it goes behind all of them. Four tables. permission_grant holds internal authorization -- principal x capability x scope, one row per data type. publication_consent holds what a landowner agreed to publish about their well, per destination and data type, so "levels yes, chemistry no" is a row rather than institutional memory. destination is the registry those consent rows point at. authorization_audit is append-only and is written in the same transaction as every change, because the first question after an exposure is who granted that, and when. Two tables rather than one because consent and clearance are decided by different people on different authority and revoked by different events. One evaluator rather than two because migration baba91fe5e83 already showed what independent filtering costs. domain/access.py holds the rules as plain functions over plain values, so they are testable without a database and cannot be restated differently by a second caller. Three invariants live there: default deny, no wildcard data type, and expiry compared at use rather than swept by a job that can be missed. An unrecognized scope type denies instead of raising, so a row written by a newer version of the code never reads as permission to an older one. services/visibility.py is the only place that answers "may this principal" and "what does this destination get". api/access.py is its one tenant: grant and consent administration, the destination registry, and a /access/decision introspection route. The prefix is /access because /publication is already the bibliography. Writes are Admin, reads are Viewer. The five vocabularies are lexicon categories seeded from core/lexicon.json, not enums in code, so adding a destination kind is not a deploy. Data types are deliberately coarse -- water level, water chemistry, well construction, site metadata -- and there is deliberately no term meaning "all". Not included: no pre-existing endpoint consults the layer, so the OGC views still publish on release_status; field projection is the next step; grants are read per request with no cache, since an unbounded one would defeat the immediate-revocation promise; and Authentik roles have no broad day-one grants yet, so /access/decision currently says no to everyone. 52 tests: 32 with no database over the rules, 20 through the routes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The attribute-level half of ADR5. Fields are published by allowlist, per
audience, at one chokepoint, because the Bureau's promises are field-shaped:
the owner's name and phone number sit on the same record as the water levels
the owner agreed to share.
core/field-allowlists.yml holds the allowlists, keyed by destination kind with
a per-slug override that replaces rather than extends. An audience with no
entry receives an empty record, not the whole one, so a column added next year
is invisible until someone lists it deliberately.
domain/field_projection.py holds the rules over plain dicts.
services/field_projection.py parses the configuration, validates it in full on
first read, and turns a model row into the record an audience receives. It is
called from services/visibility.py, below the routes: api/access.py never
projects anything itself, so a new route or output format cannot skip the rule
by forgetting to call it.
Validation is at load rather than at request time. An unknown field name, a
transform on a field nobody publishes, or an allowlist naming a never-public
field raises immediately, because the alternative is discovering a typo by
noticing data that should have been there -- or data that should not have been.
The never-public list wins over every allowlist and is applied twice, at load
and again at projection. It currently covers provenance columns, legacy AMPAPI
primary keys, and the free-text location and coordinate note columns, which
are where gate codes, lock combinations and candid landowner notes have
landed. Adding to that list is safe at any time; removing from it is a policy
decision with a named owner, and ADR5 records that nobody has been named yet.
Protection includes transformation, not only removal. `round` is the first
transform: the same well reaches a harvester at four decimal places and the
public web at two. Rounded rather than withheld, so protecting a landowner
does not mean dropping the well off the map.
/access/destination/{slug}/thing now returns the projected record and location
alongside the consented data types.
Not covered: the ogc_* views still select their own columns in SQL and gate on
release_status, so nothing on this path applies to them yet; only thing and
location are projectable entities; and field rules between internal roles --
contact information being AMP-only -- are not implemented.
Documented in docs/access-field-projection.md. 22 tests, most needing no
database.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
An empty grant table denies everyone, which is correct and useless: /access/decision said no to the people who administer the system. This writes down the access the Authentik roles already had, so nobody's access changes on the day the visibility layer starts being consulted (ADR5, 5.2). 68 grants: seven roles x their capabilities x four data types, all globally scoped. The mapping mirrors core/dependencies.py -- Viewer reads, Editor also enters and corrects, Admin also administers, the AMP family the same, and OGCInternal reads because the desktop-GIS mount has never written. Lexicon* is not seeded: it gates vocabulary, not data. AMP.Staging is not seeded because it gates a workbench that ships dark, and granting access to something still being validated is the one outcome nobody intended. The seeder will not resurrect a grant somebody revoked. Its skip check counts revoked rows as already-seeded, because narrowing this baseline is the whole point of writing it down, and a seeder that quietly undid a deliberate revocation would be worse than no seeder. Every row it writes carries granted_by = system:day-one-baseline and a reason naming the ADR, so a grant that exists because of institutional history is distinguishable from one somebody weighed. Each is audited as grant.created. Exposed as `oco seed-access-grants`, which previews by default and needs --apply to write. Grants are security state; creating them as a side effect of a process start would be the kind of thing nobody remembers is happening. It has not been run in any environment, so dev, staging and production still deny everyone until somebody runs it there. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four rules on how to approach changes in this repo: state assumptions before implementing, keep the solution minimal, keep diffs surgical, and turn tasks into verifiable goals. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The projection chokepoint only covered payloads built by services/visibility.py; the OGC views selected their own columns in SQL. This closes that gap for all 27 public collections. DescribedPostgreSQLProvider now looks its table up in the new ogc: block of core/field-allowlists.yml and hands the result to pygeoapi as the provider's `properties`, which is what _select_properties builds the SELECT from. An unlisted column is never read out of Postgres, so it cannot surface in a feature, in /schema, or in /queryables, and a filter cannot probe for it. get_fields is narrowed to match. A collection with no entry publishes nothing, and tests/test_ogc_projection.py fails on a missing entry so it shows up in CI rather than in production. The internal mount (ogc_internal_*) is not projected. The lists were generated from what each view published on 2026-08-24 minus the never-public fields, so no consumer lost a field it was using. Two leaks did close: eleven collections were publishing nma_pk_welldata and ogc_temp_depth_measurements was publishing entered_by, a staff member's name. Also fixes PRINCIPAL_API_KEY, which said "api_key" where the lexicon says "api key". The route validated against the constant, so every API-key grant was rejected with a 422. A new test pins the domain vocabularies to the lexicon enums so they cannot drift again. Drops domain.access.published_thing_ids, which had no callers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Design-canvas artboards, deck sources and their generated PDFs. Working material, not part of the API, and the PDFs are binary. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
GET /access/grant required principal_id; there was no way to browse every grant for an admin audit view. principal_id is now an optional filter, alongside new capability/data_type/scope_type filters -- the bare route lists everything. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
alembic/versions/79a3ab24627e_add_access_control_tables.py adds permission_grant etc. but deliberately does not seed the principal_type/capability/grant_scope_type/access_data_type lexicon terms those columns FK to -- that's core/lexicon.json via init_lexicon, kept out of the migration so a new capability is a data change, not a migration. entrypoint.sh only ran alembic upgrade head, so a fresh dev DB (e.g. after dropping the postgres_data_dev volume) has the tables but no terms, and any POST /access/grant 500s on a FK violation. Run `oco initialize-lexicon` right after migrations; it's idempotent (on_conflict_do_nothing), safe on every start. Dev-only -- staging/production deploy runs alembic directly and never calls this script. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Contributor
Coverage✅ 80.20% total — gate is 75%. Coverage for the Python files changed in this PR
|
The two A1 scenarios that need a pre-migration database downgraded to
PRE_A1_REVISION and then seeded through the ORM. Every column added by a
migration newer than that revision is mapped on the models but absent
from the downgraded schema, so the seed insert failed:
column "data_maturity" of relation "group" does not exist
data_maturity (e7c1a9f4b2d8) is only the current example -- any future
ReleaseMixin column would break these steps the same way.
Seed at head first, then downgrade. The downgrade drops the newer
columns but keeps the rows, which is the pre-migration state these
scenarios are asserting against. This is the order the already-passing
reversibility scenario uses.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Contributor
Coverage✅ 80.24% total — gate is 75%. Coverage for the Python files changed in this PR
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Storage and evaluator half of ADR5 -- two grant tables, one visibility
layer, and the first (only) tenant that consults it.
What lands
publication consent, one evaluator, one field-projection chokepoint
(field projection itself is not in this PR).
db/permission_grant.py,db/publication_consent.py,db/destination.py,db/authorization_audit.py-- the storage. Nowildcards: a grant names its
data_type, and nothing sweeps expiredrows -- every check compares against the date asked about.
services/visibility.py-- the only evaluator.may()answersinternal authorization;
published_things()answers "what does thisdestination get." Loads rows, calls
domain/access.py(no DB, noHTTP) for the rules.
api/access.py(/access) -- grants, destinations, consent, anda
/access/decisionintrospection route. Every write is logged toauthorization_auditin the same transaction.services/access_seed.py/oco seed-access-grants-- seeds oneglobal grant per (Authentik role, capability, data type) so today's
roles keep today's access on a fresh environment. Previews by
default,
--applyto write, idempotent, and it will not resurrect agrant somebody revoked.
b07d968b-- renamedPermissionHistorytoFieldAccessConsent(tablefield_access_consent): a landowner'sconsent to physical site access, which is not authorization, and
was easy to confuse with the new tables this PR adds.
GET /access/grantto list everygrant (principal_id now optional, plus capability/data_type/scope_type
filters) instead of requiring one principal; and a dev-only
entrypoint.shfix so a fresh docker-compose DB seeds the newlexicon categories automatically instead of 500ing on a FK violation.
What does not land here
No existing endpoint consults this layer yet --
release_statusstillgoverns what the OGC views publish.
field-allowlists.ymlandservices/field_projection.py(the actual field-by-field redaction)are a separate PR;
a2d9ef69only moves the public OGC collectionsonto the projection groundwork, not new redaction behavior.
Verification
uv run pytest tests/test_access.py tests/test_domain_access.py tests/test_access_seed.py-- all green./access/grant,/access/decision,/access/destination,/access/consentagainst a fresh dev DB(docker compose, lexicon seeded via
oco initialize-lexicon).🤖 Generated with Claude Code