Commit Graph
5 Commits
Author SHA1 Message Date
Brendan HoarandGitHub 44afb48447 Linter fix - extraneous whitespace 2024-04-26 08:17:37 -04:00
Brendan HoarandGitHub c5ae225418 Better handling of embeddings with two rare, but not unusual, files in them
I have encountered pickled embeddings with a short byteorder file at the top-level, as well as a .data/serialization_id file.

Both load fine after allowing these files in the dataset.

I do not think it is likely adding them to the safe unpickle regular expression would be a security risk, but that's for the maintainers to decide.
2024-04-26 07:55:39 -04:00
Brendan HoarandGitHub c5b7559856 Better error handling when unable to extract contents of embedding/TI file 2024-04-26 06:57:32 -04:00
Brendan HoarandGitHub 8dc920228e Better error handling when unable to read metadata from safetensors file 2024-04-26 06:52:21 -04:00
Brendan HoarandGitHub 3902aa222b Better error handling to skip non-standard ss_tag_frequency content 2024-04-26 06:44:41 -04:00