GenInferencer¶

class openicl.icl_inferencer.icl_gen_inferencer.GenInferencer(model_name: str | None = 'gpt2-xl', tokenizer_name: str | None = None, max_model_token_num: int | None = None, model_config: PretrainedConfig | None = None, batch_size: int | None = 1, gen_field_replace_token: str | None = '', generation_kwargs={'do_sample': False, 'max_new_tokens': 100}, accelerator: Accelerator | None = None, output_json_filepath: str | None = './icl_inference_output', output_json_filename: str | None = 'predictions', api_name: str | None = None, model_parallel: bool | None = False, **kwargs)[source]¶

Generation In-context Learning Inferencer Class: In-context Learning Inferencer for Directly Generation.

model¶

Local PLM (loaded from Hugging Face), which can be initialized by name or a config class.

Type:: AutoModelForCausalLM, optional

tokenizer¶

Tokenizer for model.

Type:: AutoTokenizer or GPT2Tokenizer, optional

max_model_token_num¶

Maximum number of tokenized words allowed by the LM.

Type:: int, optional

batch_size¶

Batch size for the DataLoader.

Type:: int, optional

accelerator¶

An instance of the Accelerator class, used for multiprocessing.

Type:: Accelerator, optional

output_json_filepath¶

File path for output JSON file.

Type:: str, optional

output_json_filename¶

File name for output JSON file.

Type:: str, optional

api_name¶

Name of API service.

Type:: str, optional

call_api¶

If True, an API for LM models will be used, determined by api_name.

Type:: bool

gen_field_replace_token¶

Used to replace the generation field token when generating prompts.

Type:: str, optional

generation_kwargs¶

Parameters for the model.generate() method.

Type:: Dict, optional

inference(retriever: BaseRetriever, ice_template: PromptTemplate | None = None, prompt_template: PromptTemplate | None = None, output_json_filepath: str | None = None, output_json_filename: str | None = None) → List[source]¶

Perform In-Context Inference given a retriever and optional templates.

Parameters:

retriever (BaseRetriever) – An instance of a Retriever class that will be used to retrieve in-context examples
ice_template (PromptTemplate, optional) – A template for generating the in-context examples prompt. Defaults to None.
prompt_template (PromptTemplate, optional) – A template for generating the final prompt. Defaults to None.
output_json_filepath (str, optional) – The file path to save the results as a JSON file. Defaults to None.
output_json_filename (str, optional) – The file name to save the results as a JSON file. Defaults to None.

Raises:

NotImplementedError – If the function is not implemented in the subclass.

Returns:

List: A list of string, each representing the results of one inference.