9/19/2023 0 Comments Is gpu shark safe![]() ![]() SafeTensors is not zero-copy for the header. ![]() Truly be zero-copy, whereas on GPU there is not such disk cache, so a copy is always requiredīut you can bypass allocating all the tensors on CPU at any given point. On CPU, if the file is already in cache, then it can Zero-copy: No format is really zero-copy in ML, it needs to go from disk to RAM/GPU RAM (that takes time). Layout control seems possible but not trivial as buffers have limitations link. Cap'n'proto: Float16 support is not present link so using a manual wrapper over a byte-buffer would be necessary.MsgPack: No layout control to enable lazy loading (important for loading specific parts in distributed setting).SavedModel: Tensorflow specific (it contains TF graph information).Also 210k lines of code vs ~400 lines for this lib currently. On a very different level than pickle security-wise. Seems like a great fit otherwise actually. H5: Apparently now discouraged for TF/Keras.Necessary)? This is becoming increasingly important in the ML world. Bfloat16: Does the format support native bfloat16 (meaning no weird workarounds are.Flexibility: Can I save custom code in the format and be able to use it later with zero extra code ? (~ means we can store more than pure tensors, but no custom code).No file size limit: Is there a limit to the file size ?.Controlling the layout to keep fast access to single tensors is important. Layout control: Lazy loading, is not necessarily enough since if the information about tensors is spread out in your file, then even if the information is lazily accessible you might have to access most of your file to read the available tensors (incurring many DISK -> RAM copies).Some tensors in it without scanning the whole file (distributed setting) ? Lazy loading: Can I inspect the file without loading everything ? And loading only.Zero-copy: Does reading the file require more memory than the original file ?.Safe: Can I use a file randomly downloaded and expect not to run arbitrary code ?.This is my very personal and probably biased view: Format Let's take a look at alternatives and why this format is deemed interesting. There are other formats out there used by machine learning and more general Pickle on PyTorch which is used by default. The main rationale for this crate is to remove the need to use The byte buffer needs to be entirely indexed, and cannot contain holes.0-rank Tensors (tensors with shape ) are allowed, they are merely a scalar.They don't really bring a lot of values but are accepted since they are valid tensorsįrom traditional tensor libraries perspective (torch, tensorflow, numpy. They are not storing any data in the databuffer, yet retaining size in the header. Empty tensors (tensors with 1 dimension being 0) are allowed.Tensor values are not checked against, in particular NaN and +/-Inf could.To represent integer, newlines and escapes in utf-8 strings. Anything obscure might be modified at a later time, that odd ways In general the subset of JSON is implicitly decided by serde_json for.Arbitrary JSON is not allowed, all values must be strings. A special key _metadata_ is allowed to contain free form string-to-string map.With BEGIN as the starting offset and END as the one-past offset (so total tensor byte size = END - BEGIN). data_offsets point to the tensor data relative to the beginning of the byte buffer (i.e.Import torch from safetensors import safe_open from safetensors.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |