Skip to content

Perf optimization -- debugging tracker latency #1028

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
elijahbenizzy wants to merge 4 commits into
base: main
Choose a base branch
from
Draft

Perf optimization -- debugging tracker latency #1028

elijahbenizzy wants to merge 4 commits into from

Conversation

@elijahbenizzy
Copy link
Contributor

@elijahbenizzy elijahbenizzy commented Jul 15, 2024

--- PR TEMPLATE INSTRUCTIONS (1) ---

Looking to submit a Hamilton Dataflow to the sf-hamilton-contrib module? If so go the the Preview tab and select the appropriate sub-template:

Else, if not, please remove this block of text.

--- PR TEMPLATE INSTRUCTIONS (2) ---

[Short description explaining the high-level reason for the pull request]

Changes

How I tested this

Notes

Checklist

  • PR has an informative and human-readable title (this will be pulled into the release notes)
  • Changes are limited to a single goal (no scope creep)
  • Code passed the pre-commit check & code is left cleaner/nicer than when first encountered.
  • Any change in functionality is tested
  • New functions are documented (with a description, list of inputs, and expected output)
  • Placeholder code is flagged / future TODOs are captured in comments
  • Project documentation has been updated if adding/changing functionality.
skrawcz and others added 4 commits July 10, 2024 14:44
@skrawcz
Example showing inline data saver & loaders
2a53f32 
This is a proof of concept.

What needs to be actually done:

1. ideally we expand/wrap the function with the dataloader type appropriately,
to mirror the current process (I think that's what we want).

I made them classes to make it easy to add from_X functions to create
the metadata. Otherwise I don't type the metadata dictionaries --
so maybe we should do that / provide a way to push people to putting
standard things in it.

Otherwise I think this is more ergonomic for most people getting
started.
@skrawcz
Adds notebook -- proving it works
984d6cb
@skrawcz
Adds loader and saver decorators
794c2c4 
They enable one to annotate a function as loading or
saving data and then having that metadata available
for capture.

This also removes older code -- hopefully all of it...
@elijahbenizzy
perf profiling
3868da6
@elijahbenizzy elijahbenizzy marked this pull request as draft July 15, 2024 21:55
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

2 participants

@elijahbenizzy @skrawcz