Skip to content
Closed
Changes from 1 commit
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
bf584d4
aux: add egg-info to gitignore
rdnfn Mar 10, 2023
e49d4ac
aux: add vscode to gitignore
rdnfn Mar 10, 2023
8550aaa
aux: add cohere as separate extra of package
rdnfn Mar 10, 2023
6616b9a
feat: add rough sketch of cohere lm
rdnfn Mar 10, 2023
384a45a
docs: add more comprehensive description
rdnfn Mar 11, 2023
89ff0c3
docs: add description to reorderer
rdnfn Mar 11, 2023
6876c47
feat: update loglikelihood computation
rdnfn Mar 13, 2023
8b5f270
feat: implement more robust API call with backoff
rdnfn Mar 13, 2023
1f2c3cd
fix: ensure res list filled with answers
rdnfn Mar 13, 2023
f5c13a5
refactor: minor tweaks
rdnfn Mar 14, 2023
f3188ed
add cohere lm to model registry
rdnfn Mar 14, 2023
c018bd4
feat: implement greedy method of cohere lm
rdnfn Mar 14, 2023
7597f31
feat: add correct greedy check with secondary API call
rdnfn Mar 22, 2023
5de956b
feat: use cohere retry tools
rdnfn Mar 22, 2023
938af35
fix: ensure currect comparison made for is_greedy
rdnfn Mar 22, 2023
dfe032e
test: add tests for new cohere lm class
rdnfn Mar 22, 2023
ed15f7a
fix: add newline instead of empty context and fix old api call
rdnfn Mar 22, 2023
ab0c775
feat: update tokenizer to fully rely on API
rdnfn Mar 24, 2023
fac5035
performance: remove unused tokenisation in loglikelihood()
rdnfn Mar 24, 2023
fe5d8f2
fix: comply with pre-commit (add newline)
rdnfn Mar 24, 2023
ee012b1
fix: remove update huggingface.py for pre-commit error
rdnfn Mar 24, 2023
e2985c5
fix: codespell error in cohere_lm-py
rdnfn Mar 24, 2023
e85b3c7
fix: add cohere as extras requirement for build CI check
rdnfn Mar 24, 2023
07d63bb
fix: update truncate param to match cohere api
rdnfn Mar 25, 2023
3044f47
feat: allow to disable is_greedy computation
rdnfn Mar 25, 2023
92e32c8
test: add new checks for empty context
rdnfn Mar 25, 2023
0ceea05
feat: add custom loglikelihood_rolling implementation
rdnfn Apr 4, 2023
11b2368
text: add check for rolling loglikelihood in cohere lm
rdnfn Apr 4, 2023
5383692
Merge branch 'master' into master
StellaAthena Apr 23, 2023
8aafdda
fix: prevent too long contexts if prefix is added
rdnfn Jun 21, 2023
File filter

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
docs: add description to reorderer
  • Loading branch information
rdnfn committed Mar 11, 2023
commit 89ff0c31f1803ea4b5e489695cf0abf9ff74503d
14 changes: 14 additions & 0 deletions lm_eval/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -119,9 +119,23 @@ def make_disjoint_window(pair):
class Reorderer:
def __init__(self, arr, fn):
self.size = len(arr)

# create list of (index, value) pairs
# e.g [(0,arr[0]), (1,arr[1])]
arr = list(enumerate(arr))

# set arr to a list of lists of entries arr[i]s with the same fn(arr[i])
# e.g. [[(0,arr[0]), (1,arr[1])]]
arr = group(arr, lambda x: fn(x[1]))

# set arr to list of tuples of original list indices
# mapping to the same fn output and the first
# corresponding value of the original array
# e.g. [([0,1], arr[0])]
arr = [([y[0] for y in x], x[0][1]) for x in arr]

# sort the list by the value of fn applied to the
# first corresponding value of the original array
arr.sort(key=lambda x: fn(x[1]))

self.arr = arr
Expand Down