Introducing Google AI Edge Portal: Benchmark Edge AI at scale. Sign-up to request access during private preview.

tflite_support.metadata_writers.metadata_info.RegexTokenizerMd

View source on GitHub

A container for the Regex tokenizer 1 metadata information.

View aliases

Main aliases

tflite_support.metadata_writers.audio_classifier.metadata_info.RegexTokenizerMd, tflite_support.metadata_writers.audio_classifier.metadata_writer.metadata_info.RegexTokenizerMd, tflite_support.metadata_writers.bert_nl_classifier.metadata_info.RegexTokenizerMd, tflite_support.metadata_writers.bert_nl_classifier.metadata_writer.metadata_info.RegexTokenizerMd, tflite_support.metadata_writers.image_classifier.metadata_info.RegexTokenizerMd, tflite_support.metadata_writers.image_classifier.metadata_writer.metadata_info.RegexTokenizerMd, tflite_support.metadata_writers.image_segmenter.metadata_info.RegexTokenizerMd, tflite_support.metadata_writers.image_segmenter.metadata_writer.metadata_info.RegexTokenizerMd, tflite_support.metadata_writers.nl_classifier.metadata_info.RegexTokenizerMd, tflite_support.metadata_writers.nl_classifier.metadata_writer.metadata_info.RegexTokenizerMd, tflite_support.metadata_writers.object_detector.metadata_info.RegexTokenizerMd, tflite_support.metadata_writers.object_detector.metadata_writer.metadata_info.RegexTokenizerMd

tflite_support.metadata_writers.metadata_info.RegexTokenizerMd(
    delim_regex_pattern: str, vocab_file_path: str
)

Args
`delim_regex_pattern`	the regular expression to segment strings and create tokens.
`vocab_file_path`	path to the vocabulary file.

Methods

`create_metadata`

create_metadata() -> tflite_support.metadata_schema_py_generated.ProcessUnitT

Creates the Bert tokenizer metadata based on the information.

Returns
A Flatbuffers Python object of the Bert tokenizer metadata.

Except as otherwise noted, the content of this page is licensed under the Creative Commons Attribution 4.0 License, and code samples are licensed under the Apache 2.0 License. For details, see the Google Developers Site Policies. Java is a registered trademark of Oracle and/or its affiliates.

Last updated 2024-05-08 UTC.