Skip to content

transfers

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

transfers

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
backfill
backfill
 
 
data
data
 
 
README.md
README.md
 
 
__init__.py
__init__.py
 
 
aquifer_system_transfer.py
aquifer_system_transfer.py
 
 
asset_transfer.py
asset_transfer.py
 
 
associated_data.py
associated_data.py
 
 
chemistry_sampleinfo.py
chemistry_sampleinfo.py
 
 
contact_transfer.py
contact_transfer.py
 
 
export_nmw_csvs.py
export_nmw_csvs.py
 
 
field_parameters_transfer.py
field_parameters_transfer.py
 
 
geologic_formation_transfer.py
geologic_formation_transfer.py
 
 
group_transfer.py
group_transfer.py
 
 
hydraulicsdata.py
hydraulicsdata.py
 
 
link_ids_transfer.py
link_ids_transfer.py
 
 
logger.py
logger.py
 
 
major_chemistry.py
major_chemistry.py
 
 
metrics.py
metrics.py
 
 
migrate_nmbgmr_site_names.py
migrate_nmbgmr_site_names.py
 
 
minor_trace_chemistry_transfer.py
minor_trace_chemistry_transfer.py
 
 
ngwmn_views.py
ngwmn_views.py
 
 
nmw_mirror_transfer.py
nmw_mirror_transfer.py
 
 
nmw_sql_dump.py
nmw_sql_dump.py
 
 
permissions_transfer.py
permissions_transfer.py
 
 
profiling.py
profiling.py
 
 
radionuclides.py
radionuclides.py
 
 
reference_lexicon_transfer.py
reference_lexicon_transfer.py
 
 
relaxed_constraints.md
relaxed_constraints.md
 
 
seed.py
seed.py
 
 
seed_geothermal.py
seed_geothermal.py
 
 
sensor_transfer.py
sensor_transfer.py
 
 
smoke_test.py
smoke_test.py
 
 
soil_rock_results.py
soil_rock_results.py
 
 
stratigraphy_legacy.py
stratigraphy_legacy.py
 
 
stratigraphy_transfer.py
stratigraphy_transfer.py
 
 
surface_water_data.py
surface_water_data.py
 
 
surface_water_photos.py
surface_water_photos.py
 
 
tester.py
tester.py
 
 
thing_transfer.py
thing_transfer.py
 
 
transfer.py
transfer.py
 
 
transfer_geothermal.py
transfer_geothermal.py
 
 
transfer_results_builder.py
transfer_results_builder.py
 
 
transfer_results_specs.py
transfer_results_specs.py
 
 
transfer_results_types.py
transfer_results_types.py
 
 
transferable_wells.py
transferable_wells.py
 
 
transferer.py
transferer.py
 
 
util.py
util.py
 
 
waterlevels_transducer_transfer.py
waterlevels_transducer_transfer.py
 
 
waterlevels_transfer.py
waterlevels_transfer.py
 
 
waterlevelscontinuous_pressure_daily.py
waterlevelscontinuous_pressure_daily.py
 
 
weather_data.py
weather_data.py
 
 
weather_photos.py
weather_photos.py
 
 
well_transfer.py
well_transfer.py
 
 
well_transfer_util.py
well_transfer_util.py
 
 

README.md

Transfers

This directory contains legacy-to-target ETL transfer logic.

Status: deprecated

Both legacy migration drivers are frozen. Do not add new migrations to either:

  • transfers/transfer.py -- the NM_Aquifer (AMPAPI, SQL Server) driver.
  • transfers/transfer_geothermal.py -- the NM_Wells (geothermal) driver, plus its nmw_mirror_transfer.py, nmw_sql_dump.py, and export_nmw_csvs.py supporting modules.

Their top-level entry points raise DeprecationWarning. They are kept runnable because the tables they populate (NMA_*, NMW_*) are still read by live API routes, so backfills and re-runs must remain possible -- but they receive no new features.

No automated build or trigger

There is no deployed transfer job any more. The repo used to carry a Procfile (web: python3 -m transfers.transfer) that a Cloud Build trigger (nma-ocotillo-transfer, us-central1) built with buildpacks on every push; that trigger and the Procfile are gone. Nothing runs a transfer on its own -- the drivers are invoked by hand, by an engineer, for a backfill. Do not add a Procfile or a build trigger back.

Consequently their tests live in tests/transfers/ and do not gate CI (.github/workflows/tests.yml runs pytest with --ignore=tests/transfers), and transfers/* is omitted from the coverage total in pyproject.toml. Run them by hand with uv run pytest tests/transfers.

Still live and not deprecated:

  • services/scoped_transfer.py and the oco scoped-transfer command, which import the individual NM_Aquifer transferers directly.
  • transfers/seed_geothermal.py, a dev/test seeder that generates fake data rather than reading a legacy source.

Main orchestration

  • transfers/transfer.py (deprecated)

Important supporting modules

  • transfers/transferer.py: base transfer patterns
  • transfers/util.py: shared parsing/mapping helpers
  • transfers/logger.py: transfer logging
  • transfers/metrics.py: metrics capture

Performance rules

For high-volume tables, prefer Core batch inserts:

  • session.execute(insert(Model), rows)

Avoid ORM-heavy per-row object construction for bulk workloads.

Outputs

  • Logs: transfers/logs/
  • Metrics: transfers/metrics/

Transfer Auditing CLI

Use the transfer-auditing CLI to compare each source CSV against the current destination Postgres table.

Run

source .venv/bin/activate
set -a; source .env; set +a
oco transfer-results

Useful options

oco transfer-results --sample-limit 5
oco transfer-results --summary-path transfers/metrics/transfer_results_summary.md
  • --sample-limit: limits sampled key details retained internally per transfer result.
  • --summary-path: path to the markdown report.

If oco is not on your PATH, use:

python -m cli.cli transfer-results --sample-limit 5

Output

Default report file:

  • transfers/metrics/transfer_results_summary.md

Summary columns:

  • Source Rows: raw row count in the source CSV.
  • Agreed Rows: rows considered in-scope by transfer rules/toggles.
  • Dest Rows: current row count in destination table/model.
  • Missing Agreed: Agreed Rows - Dest Rows (positive means destination is short vs agreed source rows).