6.1. Protocol

Protocols are a set of methods that are used to evaluate the performance of quantifiers. They are designed to provide a standardized way of measuring the performance of different quantifiers, and to allow for easy comparison between them. Two main protocols were defined in the literature: the APP (Artificial Prevalence Protocol) and the NPP (Natural Prevalence Protocol).

The base parameters for any protocol are:

Base parameters for any protocol

Parameter	Type
models	Union[List[Union[str, Quantifier]], str, Quantifier]
learner	BaseEstimator = None
n_jobs	int = 1
random_state	int = 32
verbose	bool = False
return_type	str = “predictions” (can be “table”)
measures	List[str] = None
columns	[“ITERATION”, “QUANTIFIER”, “REAL_PREVS”, “PRED_PREVS”, “BATCH_SIZE”]

The return_type parameter can be set to “predictions” or “table”. If set to “predictions”, the protocol will return 3 array-like objects: the name of the quantifiers, the real prevalences and the predicted prevalences. If set to “table”, the protocol will return a pandas DataFrame with the results of the evaluation, containing:

Evaluation output parameters for “table” return_type

Parameter	Type	Description
ITERATION	int	Iteration number
QUANTIFIER	str	Name of the quantifier
REAL_PREVS	array-like	Real prevalences of the batch, with size (n_classes)
PRED_PREVS	array-like	Pred prevalences of the batch, with size (n_classes)
BATCH_SIZE	int	Size of the batch
measurei	float, array-like	Value of the measurei to be applied to (REAL_PREVSi, PRED_PREVSi)

For the models, the examples of usage can be found in Artificial Prevalence protocol with general usage and Artificial Prevalence protocol with selected usage sections.

The columns parameter can be a list of column names if you want to change the default columns of the output table.

Note

If you want to make your own protocol, see Building a Protocol for more details.