ibeis.expt package¶

Submodules¶

ibeis.expt._grave_testres module¶

ibeis.expt.annotation_configs module¶

Definitions for common aid configurations

Rename to annot_cfgdef

ibeis.expt.annotation_configs.apply_qualcontrol(acfg)[source]¶

ibeis.expt.annotation_configs.apply_timecontrol(acfg, min_timedelta=u'6h', require_timestamp=True)[source]¶

ibeis.expt.annotation_configs.compress_acfg_list_for_printing(acfg_list)[source]¶

CommandLine:

python -m ibeis --tf compress_acfg_list_for_printing

Example

>>> from ibeis.expt.annotation_configs import *  # NOQA
>>> qcfg_list = [{'f': 1, 'b': 1}, {'f': 2, 'b': 1}, {'f': 3, 'b': 1, 'z': 4}]
>>> acfg_list = [{'qcfg': qcfg} for qcfg in qcfg_list]
>>> nonvaried_dict, varied_dicts = compress_acfg_list_for_printing(acfg_list)
>>> result = ('varied_dicts = %s\n' % (ut.list_str(varied_dicts),))
>>> result += ('nonvaried_dict = %s' % (ut.dict_str(nonvaried_dict),))
>>> print(result)

ibeis.expt.annotation_configs.compress_aidcfg(acfg, filter_nones=False, filter_empty=False, force_noncommon=[])[source]¶

Parameters:	acfg (dict) –
Returns:	acfg
Return type:	dict

CommandLine:

#python -m ibeis --tf compress_aidcfg
python -m ibeis.expt.annotation_configs --exec-compress_aidcfg --show

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.annotation_configs import *  # NOQA
>>> acfg = default
>>> acfg = compress_aidcfg(acfg)
>>> result = ('acfg = %s' % (ut.dict_str(acfg),))
>>> print(default)
>>> print(result)

ibeis.expt.annotation_configs.flatten_acfg_list(acfg_list)[source]¶

Returns a new config where subconfig params are prefixed by subconfig keys

ibeis.expt.annotation_configs.get_varied_acfg_labels(acfg_list, mainkey=u'_cfgname', checkname=False)[source]¶

>>> from ibeis.expt.annotation_configs import *  # NOQA

ibeis.expt.annotation_configs.partition_acfg_list(acfg_list)[source]¶

ibeis.expt.annotation_configs.print_acfg(acfg, expanded_aids=None, ibs=None, **kwargs)[source]¶

ibeis.expt.annotation_configs.print_acfg_list(acfg_list, expanded_aids_list=None, ibs=None, combined=False, **kwargs)[source]¶

Parameters:	acfg_list (list) – expanded_aids_list (list) – (default = None) ibs (IBEISController) – ibeis controller object(default = None) combined (bool) – (default = False)

CommandLine:

python -m ibeis.expt.annotation_configs --exec-print_acfg_list --show

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.annotation_configs import *  # NOQA
>>> import ibeis
>>> acfg_list = '?'
>>> expanded_aids_list = None
>>> ibs = None
>>> combined = False
>>> result = print_acfg_list(acfg_list, expanded_aids_list, ibs, combined)
>>> print(result)
>>> ut.quit_if_noshow()
>>> import plottool as pt
>>> ut.show_if_requested()

ibeis.expt.annotation_configs.shorten_to_alias_labels(key)[source]¶

ibeis.expt.annotation_configs.unflatten_acfgdict(flat_dict, prefix_list=[u'dcfg', u'qcfg'])[source]¶

ibeis.expt.cfghelpers module¶

Helper module that helps expand parameters for grid search

DEPRICATE: Most of this can likely be replaced by util_gridsearch TODO: rectify with versions in util_gridsearch

It turns out a lot of the commandlines made possible here can be generatd by using bash brace expansion. http://www.linuxjournal.com/content/bash-brace-expansion

ibeis.expt.cfghelpers.customize_base_cfg(cfgname, cfgopt_strs, base_cfg, cfgtype, alias_keys=None, valid_keys=None, offset=0, strict=True)[source]¶

DEPRICATE

ibeis.expt.cfghelpers.parse_argv_cfg(argname, default=[u''], named_defaults_dict=None, valid_keys=None)[source]¶

simple configs

Parameters:	argname – default (list) – (default = []) named_defaults_dict (dict) – (default = None) valid_keys (None) – (default = None)
Returns:	cfg_list
Return type:	list

CommandLine:

python -m ibeis.expt.cfghelpers --exec-parse_argv_cfg --filt :foo=bar
python -m ibeis.expt.cfghelpers --test-parse_argv_cfg

Example

>>> # ENABLE_DOCTET
>>> from ibeis.expt.cfghelpers import *  # NOQA
>>> argname = '--filt'
>>> cfg_list = parse_argv_cfg(argname)
>>> result = ('cfg_list = %s' % (str(cfg_list),))
>>> print(result)

ibeis.expt.cfghelpers.parse_cfgstr_list2(cfgstr_list, named_defaults_dict=None, cfgtype=None, alias_keys=None, valid_keys=None, expand_nested=True, strict=True, special_join_dict=None, is_nestedcfgtype=False, metadata=None)[source]¶

Parses config strings. By looking up name in a dict of configs

DEPRICATE

Parameters:	cfgstr_list (list) – named_defaults_dict (dict) – (default = None) cfgtype (None) – (default = None) alias_keys (None) – (default = None) valid_keys (None) – (default = None) expand_nested (bool) – (default = True) strict (bool) – (default = True) - used for annot configs so special joins arent geometrically combined (is_nestedcfgtype) –

Note

Normal Case:

–flag name

Custom Arugment Cases:

–flag name:custom_key1=custom_val1,custom_key2=custom_val2

Multiple Config Case:

–flag name1:custom_args1 name2:custom_args2

Multiple Config (special join) Case:

(here name2 and name3 have some special interaction) –flag name1:custom_args1 name2:custom_args2::name3:custom_args3

Varied Argument Case:

–flag name:key1=[val1,val2]

Returns:	cfg_combos_list
Return type:	list

CommandLine:

python -m ibeis.expt.cfghelpers --exec-parse_cfgstr_list2
python -m ibeis.expt.cfghelpers --test-parse_cfgstr_list2

Example

>>> # ENABLE_DOCTET
>>> from ibeis.expt.cfghelpers import *  # NOQA
>>> cfgstr_list = ['name', 'name:f=1', 'name:b=[1,2]', 'name1:f=1::name2:f=1,b=2']
>>> #cfgstr_list = ['name', 'name1:f=1::name2:f=1,b=2']
>>> named_defaults_dict = None
>>> cfgtype = None
>>> alias_keys = None
>>> valid_keys = None
>>> expand_nested = True
>>> strict = False
>>> special_join_dict = {'joined': True}
>>> cfg_combos_list = parse_cfgstr_list2(cfgstr_list, named_defaults_dict,
>>>                                      cfgtype, alias_keys, valid_keys,
>>>                                      expand_nested, strict,
>>>                                      special_join_dict)
>>> print('cfg_combos_list = %s' % (ut.list_str(cfg_combos_list, nl=2),))
>>> print(ut.depth_profile(cfg_combos_list))
>>> cfg_list = ut.flatten(cfg_combos_list)
>>> cfg_list = ut.flatten([cfg if isinstance(cfg, list) else [cfg] for cfg in cfg_list])
>>> result = ut.repr2(ut.get_varied_cfg_lbls(cfg_list))
>>> print(result)
['name:', 'name:f=1', 'name:b=1', 'name:b=2', 'name1:f=1,joined=True', 'name2:b=2,f=1,joined=True']

ibeis.expt.cfghelpers.remove_prefix_hack(cfg, cfgtype, cfg_options, alias_keys)[source]¶

ibeis.expt.draw_helpers module¶

class ibeis.expt.draw_helpers.IndividualResultsCopyTaskQueue[source]¶

Bases: object

append_copy_task(fpath_orig, dstdir=None)[source]¶

helper which copies a summary figure to root dir

flush_copy_tasks()[source]¶

ibeis.expt.draw_helpers.make_individual_latex_figures(ibs, fpaths_list, flat_case_labels, cfgx2_shortlbl, case_figdir, analysis_fpath_list)[source]¶

ibeis.expt.experiment_configs module¶

In this file dicts specify all possible combinations of the varied parameters and lists specify the union of parameters

Rename to pipe_cfgdef

ibeis.expt.experiment_configs.apply_CircQRH(cfg)[source]¶

ibeis.expt.experiment_configs.apply_Ell(cfg)[source]¶

ibeis.expt.experiment_configs.apply_EllQRH(cfg)[source]¶

ibeis.expt.experiment_configs.apply_k(cfg)[source]¶

ibeis.expt.experiment_configs.apply_knorm(cfg)[source]¶

ibeis.expt.experiment_configs.apply_param(cfg, **kwargs)[source]¶

ibeis.expt.experiment_configs.augbase(basedict, updatedict)[source]¶

ibeis.expt.experiment_configs.best(metadata)[source]¶

Infer the best pipeline config based on the metadata

ibeis.expt.experiment_configs.get_candidacy_dbnames()[source]¶

ibeis.expt.experiment_drawing module¶

./dev.py -t custom:affine_invariance=False,adapteq=True,fg_on=False –db Elephants_drop1_ears –allgt –index=0:10 –guiview # NOQA

ibeis.expt.experiment_drawing.draw_annot_scoresep(ibs, testres, f=None, verbose=None)[source]¶

Draws the separation between true positive and true negative name scores.

TODO:

plot the difference between the top true score and the next best false score?

CommandLine:

ib
python -m ibeis --tf draw_annot_scoresep --show
python -m ibeis --tf draw_annot_scoresep --db PZ_MTEST --allgt -w --show --serial
python -m ibeis --tf draw_annot_scoresep -t scores --db PZ_MTEST --allgt --show
python -m ibeis --tf draw_annot_scoresep -t scores --db PZ_Master0 --allgt --show
python -m ibeis --tf draw_annot_scoresep --db PZ_Master1 -a timectrl -t best --show
python -m ibeis --tf draw_annot_scoresep --db PZ_Master1 -a timectrl -t best --show -f :without_tag=photobomb

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.experiment_drawing import *  # NOQA
>>> from ibeis.init import main_helpers
>>> defaultdb = 'PZ_MTEST'
>>> ibs, testres = main_helpers.testdata_expts(defaultdb, a=['timectrl'], t=['best'])
>>> f = ut.get_argval(('--filt', '-f'), type_=list, default=[''])
>>> draw_annot_scoresep(ibs, testres, f=f, verbose=ut.VERBOSE)
>>> ut.show_if_requested()

ibeis.expt.experiment_drawing.draw_case_timedeltas(ibs, testres, falsepos=None, truepos=None, verbose=False)[source]¶

CommandLine:

python -m ibeis.dev -e draw_case_timedeltas --show
python -m ibeis.dev -e draw_case_timedeltas --show -t default \
    -a unctrl:num_names=1,name_offset=[1,2]
python -m ibeis.dev -e draw_case_timedeltas --show -t default \
    -a unctrl:num_names=1,name_offset=[1,2],joinme=1
python -m ibeis.dev -e draw_case_timedeltas --show -t default \
    -a unctrl:num_names=1,name_offset=[1,2] \
       unctrl:num_names=1,name_offset=[3,0]


python -m ibeis.dev -e timedelta_hist --show -t baseline \
    -a unctrl ctrl:force_const_size=True unctrl:force_const_size=True \
    --consistent --db PZ_MTEST

# Testing
python -m ibeis.dev -e timedelta_hist --show -t baseline \
    -a unctrl ctrl:force_const_size=True unctrl:force_const_size=True \
    --consistent --db PZ_Master1
python -m ibeis.dev -e timedelta_hist --show -t baseline \
    -a unctrl ctrl:sample_rule_ref=max_timedelta --db PZ_Master1 \
    --aidcfginfo

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.experiment_drawing import *  # NOQA
>>> from ibeis.init import main_helpers
>>> ibs, testres = main_helpers.testdata_expts('PZ_MTEST')
>>> draw_case_timedeltas(ibs, testres)
>>> ut.show_if_requested()

ibeis.expt.experiment_drawing.draw_casetag_hist(ibs, testres, f=None, with_wordcloud=True)[source]¶

Parameters:	ibs (ibeis.IBEISController) – ibeis controller object testres (TestResult) – test result object

CommandLine:

ibeis --tf -draw_casetag_hist --show

# Experiments I tagged
ibeis --tf -draw_casetag_hist -a timectrl -t invarbest --db PZ_Master1  --show

ibeis -e taghist -a timectrl -t best --db PZ_Master1  --show

ibeis -e taghist -a timequalctrl -t invarbest --db PZ_Master1  --show
ibeis -e taghist -a timequalctrl:minqual=good -t invarbest --db PZ_Master1  --show
ibeis -e taghist -a timequalctrl:minqual=good -t invarbest --db PZ_Master1  --show --filt :fail=True

# Do more tagging
ibeis -e cases -a timequalctrl:minqual=good -t invarbest --db PZ_Master1 \
    --filt :orderby=gfscore,reverse=1,min_gtrank=1,max_gf_tags=0 --show
ibeis -e print -a timequalctrl:minqual=good -t invarbest --db PZ_Master1 --show
ibeis -e cases -a timequalctrl -t invarbest --db PZ_Master1 \
    --filt :orderby=gfscore,reverse=1,max_gf_tags=0,:fail=True,min_gf_timedelta=12h --show

ibeis -e cases -a timequalctrl -t invarbest --db PZ_Master1 \
    --filt :orderby=gfscore,reverse=1,max_gf_tags=0,:fail=True,min_gf_timedelta=12h --show
python -m ibeis -e taghist --db PZ_Master1 -a timectrl -t best \
    --filt :fail=True --no-wordcloud --hargv=tags  --prefix "Failure Case " --label PZTags  --figsize=10,3  --left=.2

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.experiment_drawing import *  # NOQA
>>> from ibeis.init import main_helpers
>>> ibs, testres = main_helpers.testdata_expts('PZ_Master1', a=['timequalcontrolled'])
>>> f = ut.get_argval(('--filt', '-f'), type_=list, default=[''])
>>> draw_casetag_hist(ibs, testres, f=f)
>>> ut.show_if_requested()

ibeis.expt.experiment_drawing.draw_match_cases(ibs, testres, metadata=None, f=None, show_in_notebook=False, annot_modes=None, figsize=None, case_pos_list=None, verbose=None, interact=None, **kwargs)[source]¶

Parameters:	ibs (ibeis.IBEISController) – ibeis controller object testres (TestResult) – test result object metadata (None) – (default = None)

CommandLine:

python -m ibeis --tf draw_match_cases
python -m ibeis.dev -e draw_match_cases --figdir=figure
python -m ibeis.dev -e draw_match_cases --db PZ_Master1 -a ctrl \
    -t default --filt :fail=True,min_gtrank=5,gtrank_lt=20 --render

# Shows the best results
python -m ibeis.dev -e cases --db PZ_Master1 -a timectrl \
    -t invarbest --filt :sortasc=gtscore,success=True,index=200:201 --show

# Shows failures sorted by gt score
python -m ibeis.dev -e cases --db PZ_Master1 -a timectrl \
    -t invarbest --filt :sortdsc=gfscore,min_gtrank=1 --show

# Find the untagged photobomb and scenery cases
python -m ibeis.dev -e cases --db PZ_Master1 -a timectrl \
    -t invarbest --show --filt \
    :orderby=gfscore,reverse=1,min_gtrank=1,max_gf_td=24h,max_gf_tags=0

# Find untagged failures
python -m ibeis.dev -e cases --db PZ_Master1 -a timectrl \
    -t invarbest \
    --filt :orderby=gfscore,reverse=1,min_gtrank=1,max_gf_tags=0 --show

# Show disagreement cases
ibeis --tf draw_match_cases --db PZ_MTEST -a default:size=20 \
    -t default:K=[1,4] \
    --filt :disagree=True,index=0:4 --show

ibeis --tf draw_match_cases --db humpbacks_fb \
    -a default:has_any=hasnotch,mingt=2 \
    -t default:proot=BC_DTW,decision=max,crop_dim_size=500,crop_enabled=True,manual_extract=False,use_te_scorer=True,ignore_notch=True,te_net=annot_simple default:proot=vsmany \
    --qaids-override 12 --show

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.experiment_drawing import *  # NOQA
>>> from ibeis.init import main_helpers
>>> ibs, testres = main_helpers.testdata_expts('PZ_MTEST')
>>> filt_cfg = main_helpers.testdata_filtcfg()
>>> metadata = None
>>> analysis_fpath_list = draw_match_cases(ibs, testres, metadata, f=filt_cfg)
>>> ut.show_if_requested()

ibeis.expt.experiment_drawing.draw_rank_cdf(ibs, testres, verbose=False, test_cfgx_slice=None, do_per_annot=True, draw_icon=True, numranks=3, kind=u'bar', cdfzoom=False)[source]¶

Parameters:	ibs (ibeis.IBEISController) – ibeis controller object testres (TestResult) –

CommandLine:

python -m ibeis.dev -e draw_rank_cdf
python -m ibeis.dev -e draw_rank_cdf --db PZ_MTEST --show -a timectrl
python -m ibeis.dev -e draw_rank_cdf --db PZ_MTEST --show -a timectrl -t invar --kind=cmc
python -m ibeis.dev -e draw_rank_cdf --db PZ_MTEST --show -a timectrl -t invar --kind=cmc --cdfzoom
python -m ibeis.dev -e draw_rank_cdf --db PZ_MTEST --show -a varypername_td   -t CircQRH_ScoreMech:K=3
#ibeis -e rank_cdf --db lynx -a default:qsame_imageset=True,been_adjusted=True,excluderef=True -t default:K=1 --show

python -m ibeis.dev -e draw_rank_cdf --db lynx -a default:qsame_imageset=True,been_adjusted=True,excluderef=True -t default:K=1 --show

python -m ibeis --tf draw_rank_cdf -t best -a timectrl --db PZ_Master1 --show

python -m ibeis --tf draw_rank_cdf --db PZ_Master1 --show -t best \
    -a timectrl:qhas_any=\(needswork,correctable,mildviewpoint\),qhas_none=\(viewpoint,photobomb,error:viewpoint,quality\) \
    --acfginfo --veryverbtd

ibeis --tf draw_match_cases --db GZ_ALL -a ctrl \
    -t default:K=1,resize_dim=[width],dim_size=[700,750] \
    -f :sortdsc=gfscore,without_tag=scenerymatch,disagree=True \
    --show

ibeis --tf autogen_ipynb --db GZ_ALL --ipynb -a ctrl \
    -t default:K=1,resize_dim=[width],dim_size=[600,700,750] \
     default:K=1,resize_dim=[area],dim_size=[450,550,600,650]

ibeis draw_rank_cdf --db GZ_ALL -a ctrl -t default --show
ibeis draw_match_cases --db GZ_ALL -a ctrl -t default -f :fail=True --show

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.experiment_drawing import *  # NOQA
>>> from ibeis.init import main_helpers
>>> #ibs, testres = main_helpers.testdata_expts(
>>> #    'seaturtles', a='default2:qhas_any=(left),sample_occur=True,occur_offset=[0,1,2,3,4,5,6,7,8],num_names=None')
>>> ibs, testres = main_helpers.testdata_expts('PZ_MTEST')
>>> kwargs = ut.argparse_funckw(draw_rank_cdf)
>>> result = draw_rank_cdf(ibs, testres, **kwargs)
>>> ut.show_if_requested()
>>> print(result)

ibeis.expt.experiment_drawing.draw_rank_surface(ibs, testres, verbose=None, fnum=None)[source]¶

Draws n dimensional data + a score / rank The rank is always on the y axis.

The first dimension is on the x axis. The second dimension is split over multiple plots. The third dimension becomes multiple lines. May need to clean this scheme up a bit.

Parameters:	ibs (ibeis.IBEISController) – ibeis controller object testres (TestResult) – test result object

CommandLine:

ibeis --tf draw_rank_surface --db PZ_Master1 -a varysize_td -t CircQRH_K --show

ibeis --tf draw_rank_surface --show -t best -a varysize --db PZ_Master1 --show

ibeis --tf draw_rank_surface --show -t CircQRH_K -a varysize_td --db PZ_Master1 --show
ibeis --tf draw_rank_surface --show -t CircQRH_K -a varysize_td --db PZ_Master1 --show

ibeis --tf draw_rank_surface --show  -t candidacy_k -a varysize  --db PZ_Master1 --show --param-keys=K,dcfg_sample_per_name,dcfg_sample_size
ibeis --tf draw_rank_surface --show  -t best \
    -a varynannots_td varynannots_td:qmin_pername=3,dpername=2  \
    --db PZ_Master1 --show --param-keys=dcfg_sample_per_name,dcfg_sample_size
ibeis --tf draw_rank_surface --show  -t best -a varynannots_td  --db PZ_Master1 --show --param-keys=dcfg_sample_size

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.experiment_drawing import *  # NOQA
>>> from ibeis.init import main_helpers
>>> ibs, testres = main_helpers.testdata_expts('PZ_MTEST')
>>> result = draw_rank_surface(ibs, testres)
>>> ut.show_if_requested()
>>> print(result)

ibeis.expt.experiment_drawing.scorediff(ibs, testres, f=None, verbose=None)[source]¶

Parameters:	ibs (ibeis.IBEISController) – image analysis api testres (ibeis.TestResult) – test result object f (None) – (default = None) verbose (bool) – verbosity flag(default = None)

CommandLine:

python -m ibeis.expt.experiment_drawing scorediff --db PZ_Master1 -a timectrl -t best --show

python -m ibeis.expt.experiment_drawing scorediff --db humpbacks_fb \
    -a default:has_any=hasnotch,mingt=2 \
    -t default:proot=BC_DTW,decision=max,crop_dim_size=500,crop_enabled=True,use_te_scorer=False,manual_extract=True,ignore_notch=True,te_net=annot_simple --show

Example

>>> # DISABLE_DOCTEST
>>> # DISABLE_DOCTEST
>>> from ibeis.expt.experiment_drawing import *  # NOQA
>>> from ibeis.init import main_helpers
>>> defaultdb = 'PZ_MTEST'
>>> ibs, testres = main_helpers.testdata_expts(defaultdb, a=['timectrl'], t=['best'])
>>> f = ut.get_argval(('--filt', '-f'), type_=list, default=[''])
>>> scorediff(ibs, testres, f=f, verbose=ut.VERBOSE)
>>> ut.show_if_requested()

ibeis.expt.experiment_helpers module¶

Helper module that helps expand parameters for grid search TODO: move into custom pipe_cfg and annot_cfg modules

ibeis.expt.experiment_helpers.filter_duplicate_acfgs(expanded_aids_list, acfg_list, acfg_name_list, verbose=True)[source]¶

Removes configs with the same expanded aids list

CommandLine:

# The following will trigger this function:
ibeis -e print_acfg -a timectrl timectrl:view=left --db PZ_MTEST

ibeis.expt.experiment_helpers.get_annotcfg_list(ibs, acfg_name_list, filter_dups=True, qaid_override=None, daid_override=None, initial_aids=None, use_cache=None, verbose=None)[source]¶

For now can only specify one acfg name list

TODO: move to filter_annots

Parameters:	annot_cfg_name_list (list) –

CommandLine:

python -m ibeis.expt.experiment_helpers --exec-get_annotcfg_list:0
python -m ibeis.expt.experiment_helpers --exec-get_annotcfg_list:1
python -m ibeis.expt.experiment_helpers --exec-get_annotcfg_list:2

ibeis -e print_acfg --ainfo
ibeis -e print_acfg --db NNP_Master3 -a viewpoint_compare --nocache-aid --verbtd
ibeis -e print_acfg --db PZ_ViewPoints -a viewpoint_compare --nocache-aid --verbtd
ibeis -e print_acfg --db PZ_MTEST -a unctrl ctrl::unctrl --ainfo --nocache-aid
ibeis -e print_acfg --db testdb1 -a default --ainfo --nocache-aid
ibeis -e print_acfg --db Oxford -a default:qhas_any=query --ainfo --nocache-aid
ibeis -e print_acfg --db Oxford -a default:qhas_any=query,dhas_any=distractor --ainfo --nocache-aid

Example0:

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.experiment_helpers import *  # NOQA
>>> import ibeis
>>> from ibeis.expt import annotation_configs
>>> ibs = ibeis.opendb(defaultdb='PZ_MTEST')
>>> filter_dups = not ut.get_argflag('--nofilter-dups')
>>> acfg_name_list = testdata_acfg_names()
>>> _tup = get_annotcfg_list(ibs, acfg_name_list, filter_dups)
>>> acfg_list, expanded_aids_list = _tup
>>> print('\n PRINTING TEST RESULTS')
>>> result = ut.list_str(acfg_list, nl=3)
>>> print('\n')
>>> #statskw = ut.parse_func_kwarg_keys(ibs.get_annot_stats_dict, with_vals=False)
>>> printkw = dict(combined=True, per_name_vpedge=None,
>>>                per_qual=False, per_vp=False, case_tag_hist=False)
>>> annotation_configs.print_acfg_list(
>>>     acfg_list, expanded_aids_list, ibs, **printkw)

ibeis.expt.experiment_helpers.get_pipecfg_list(test_cfg_name_list, ibs=None)[source]¶

Builds a list of varied query configurations. Only custom configs depend on an ibs object. The order of the output is not gaurenteed to aggree with input order.

Parameters:	test_cfg_name_list (list) – list of strs ibs (IBEISController) – ibeis controller object (optional)
Returns:	(cfg_list, cfgx2_lbl) - cfg_list (list): list of config objects cfgx2_lbl (list): denotes which parameters are being varied. If there is just one config then nothing is varied
Return type:	tuple

CommandLine:

python -m ibeis.expt.experiment_helpers --exec-get_pipecfg_list:0
python -m ibeis.expt.experiment_helpers --exec-get_pipecfg_list:1 --db humpbacks

Example

>>> # ENABLE_DOCTEST
>>> from ibeis.expt.experiment_helpers import *  # NOQA
>>> import ibeis
>>> ibs = ibeis.opendb(defaultdb='testdb1')
>>> #test_cfg_name_list = ['best', 'custom', 'custom:sv_on=False']
>>> #test_cfg_name_list = ['default', 'default:sv_on=False', 'best']
>>> test_cfg_name_list = ['default', 'default:sv_on=False', 'best']
>>> # execute function
>>> (pcfgdict_list, pipecfg_list) = get_pipecfg_list(test_cfg_name_list, ibs)
>>> # verify results
>>> assert pipecfg_list[0].sv_cfg.sv_on is True
>>> assert pipecfg_list[1].sv_cfg.sv_on is False
>>> pipecfg_lbls = get_varied_pipecfg_lbls(pcfgdict_list)
>>> result = ('pipecfg_lbls = '+ ut.list_str(pipecfg_lbls))
>>> print(result)
pipecfg_lbls = [
    'default:',
    'default:sv_on=False',
]

Example1:

>>> # DISABLE_DOCTEST
>>> import ibeis_flukematch.plugin
>>> from ibeis.expt.experiment_helpers import *  # NOQA
>>> import ibeis
>>> ibs = ibeis.opendb(defaultdb='humpbacks')
>>> test_cfg_name_list = ['default:pipeline_root=BC_DTW,decision=average,crop_dim_size=[960,500]', 'default:K=[1,4]']
>>> (pcfgdict_list, pipecfg_list) = get_pipecfg_list(test_cfg_name_list, ibs)
>>> pipecfg_lbls = get_varied_pipecfg_lbls(pcfgdict_list)
>>> result = ('pipecfg_lbls = '+ ut.list_str(pipecfg_lbls))
>>> print(result)
>>> print_pipe_configs(pcfgdict_list, pipecfg_list)

ibeis.expt.experiment_helpers.get_varied_pipecfg_lbls(cfgdict_list, pipecfg_list=None)[source]¶

ibeis.expt.experiment_helpers.parse_acfg_combo_list(acfg_name_list)[source]¶

Parses the name list into a list of config dicts

Parameters:	acfg_name_list (list) – a list of annotation config strings
Returns:	acfg_combo_list
Return type:	list

CommandLine:

python -m ibeis.expt.experiment_helpers --exec-parse_acfg_combo_list
python -m ibeis.expt.experiment_helpers --exec-parse_acfg_combo_list:1

Example

>>> # ENABLE_DOCTET
>>> from ibeis.expt.experiment_helpers import *  # NOQA
>>> import ibeis
>>> from ibeis.expt import annotation_configs
>>> acfg_name_list = testdata_acfg_names(['default', 'uncontrolled'])
>>> acfg_combo_list = parse_acfg_combo_list(acfg_name_list)
>>> acfg_list = ut.flatten(acfg_combo_list)
>>> printkw = dict()
>>> annotation_configs.print_acfg_list(acfg_list, **printkw)
>>> result = list(acfg_list[0].keys())
>>> print(result)
[u'qcfg', u'dcfg']

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.experiment_helpers import *  # NOQA
>>> import ibeis
>>> from ibeis.expt import annotation_configs
>>> # double colon :: means expand consistently and force const size
>>> acfg_name_list = testdata_acfg_names(['unctrl', 'ctrl::unctrl'])
>>> acfg_name_list = testdata_acfg_names(['unctrl', 'varysize', 'ctrl::unctrl'])
>>> acfg_name_list = testdata_acfg_names(['unctrl', 'varysize', 'ctrl::varysize', 'ctrl::unctrl'])
>>> acfg_combo_list = parse_acfg_combo_list(acfg_name_list)
>>> acfg_list = ut.flatten(acfg_combo_list)
>>> printkw = dict()
>>> annotation_configs.print_acfg_list(acfg_list, **printkw)

ibeis.expt.experiment_helpers.print_pipe_configs(cfgdict_list, pipecfg_list)[source]¶

ibeis.expt.experiment_helpers.testdata_acfg_names(default_acfg_name_list=[u'default'])[source]¶

ibeis.expt.experiment_printres module¶

displays results from harness

TODO: save a testres variable so reloading and regenration becomes easier.

ibeis.expt.experiment_printres.get_diffmat_str(rank_mat, qaids, nConfig)[source]¶

ibeis.expt.experiment_printres.get_diffranks(rank_mat, qaids)[source]¶

Find rows which scored differently over the various configs FIXME: duplicated

ibeis.expt.experiment_printres.print_latexsum(ibs, testres, verbose=True)[source]¶

Parameters:	ibs (IBEISController) – ibeis controller object testres –

CommandLine:

python -m ibeis.expt.experiment_printres --exec-print_latexsum
python -m ibeis.scripts.gen_cand_expts --exec-gen_script

python -m ibeis --tf print_latexsum -t candidacy --db PZ_Master0 -a controlled --rank-lt-list=1,5,10,100
python -m ibeis --tf print_latexsum -t candidacy --db PZ_MTEST -a controlled --rank-lt-list=1,5,10,100

Example

>>> # SCRIPT
>>> from ibeis.expt.experiment_printres import *  # NOQA
>>> from ibeis.init import main_helpers
>>> ibs, testres = main_helpers.testdata_expts()
>>> tabular_str2 = print_latexsum(ibs, testres)

ibeis.expt.experiment_printres.print_results(ibs, testres)[source]¶

Prints results from an experiment harness run. Rows store different qaids (query annotation ids) Cols store different configurations (algorithm parameters)

Parameters:	ibs (IBEISController) – ibeis controller object testres (test_result.TestResult) –

CommandLine:

python dev.py -e print --db PZ_MTEST -a default:dpername=1,qpername=[1,2]  -t default:fg_on=False

python dev.py -e print -t best --db seals2 --allgt --vz
python dev.py -e print --db PZ_MTEST --allgt -t custom --print-confusion-stats
python dev.py -e print --db PZ_MTEST --allgt --noqcache --index 0:10:2 -t custom:rrvsone_on=True --print-confusion-stats
python dev.py -e print --db PZ_MTEST --allgt --noqcache --qaid4 -t custom:rrvsone_on=True --print-confusion-stats
python -m ibeis --tf print_results -t default --db PZ_MTEST -a ctrl
python -m ibeis --tf print_results -t default --db PZ_MTEST -a ctrl
python -m ibeis --tf print_results --db PZ_MTEST -a default -t default:lnbnn_on=True default:lnbnn_on=False,bar_l2_on=True default:lnbnn_on=False,normonly_on=True

CommandLine:

python -m ibeis.expt.experiment_printres --test-print_results
utprof.py -m ibeis.expt.experiment_printres --test-print_results

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.experiment_printres import *  # NOQA
>>> from ibeis.init import main_helpers
>>> ibs, testres = main_helpers.testdata_expts(
>>>     'PZ_MTEST', a='default:dpername=1,qpername=[1,2]', t='default:fg_on=False')
>>> result = print_results(ibs, testres)
>>> print(result)

ibeis.expt.experiment_printres.rankscore_str(thresh, nLess, total, withlbl=True)[source]¶

ibeis.expt.harness module¶

Runs many queries and keeps track of some results

ibeis.expt.harness.get_big_test_cache_info(ibs, cfgx2_qreq_)[source]¶

Parameters:	ibs (ibeis.IBEISController) – cfgx2_qreq (dict) –

ibeis.expt.harness.get_qres_name_result_info(ibs, cm, qreq_)[source]¶

these are results per query we care about

• gt (best correct match) and gf (best incorrect match) rank, their score and the difference

ibeis.expt.harness.get_query_result_info(qreq_)[source]¶

Helper function.

Runs queries of a specific configuration returns the best rank of each query

Parameters:	qaids (list) – query annotation ids daids (list) – database annotation ids
Returns:	qx2_bestranks

CommandLine:

python -m ibeis.expt.harness --test-get_query_result_info
python -m ibeis.expt.harness --test-get_query_result_info:0
python -m ibeis.expt.harness --test-get_query_result_info:1
python -m ibeis.expt.harness --test-get_query_result_info:0 --db lynx -a default:qsame_imageset=True,been_adjusted=True,excluderef=True -t default:K=1
python -m ibeis.expt.harness --test-get_query_result_info:0 --db lynx -a default:qsame_imageset=True,been_adjusted=True,excluderef=True -t default:K=1 --cmd

Example

>>> # ENABLE_DOCTEST
>>> from ibeis.expt.harness import *  # NOQA
>>> import ibeis
>>> qreq_ = ibeis.main_helpers.testdata_qreq_(a=['default:qindex=0:3,dindex=0:5'])
>>> #ibs = ibeis.opendb('PZ_MTEST')
>>> #qaids = ibs.get_valid_aids()[0:3]
>>> #daids = ibs.get_valid_aids()[0:5]
>>> #qreq_ = ibs.new_query_request(qaids, daids, verbose=True, cfgdict={})
>>> cfgres_info = get_query_result_info(qreq_)
>>> print(ut.dict_str(cfgres_info))

Example

>>> # ENABLE_DOCTEST
>>> from ibeis.expt.harness import *  # NOQA
>>> import ibeis
>>> ibs = ibeis.opendb('PZ_MTEST')
>>> #cfgdict = dict(codename='vsone')
>>> # ibs.cfg.query_cfg.codename = 'vsone'
>>> qaids = ibs.get_valid_aids()[0:3]
>>> daids = ibs.get_valid_aids()[0:5]
>>> qreq_ = ibs.new_query_request(qaids, daids, verbose=True, cfgdict={})
>>> cfgres_info = get_query_result_info(qreq_)
>>> print(ut.dict_str(cfgres_info))

ibeis.expt.harness.make_single_testres(ibs, qaids, daids, pipecfg_list, cfgx2_lbl, cfgdict_list, lbl, testnameid, use_cache=None, subindexer_partial=<class 'utool.util_progress.ProgressIter'>)[source]¶

CommandLine:

python -m ibeis.expt.harness --exec-run_test_configurations2

ibeis.expt.harness.run_test_configurations2(ibs, acfg_name_list, test_cfg_name_list, use_cache=None, qaid_override=None, daid_override=None, initial_aids=None)[source]¶

Loops over annot configs.

Try and use this function as a starting point to clean up this module. The code is getting too untenable.

CommandLine:

python -m ibeis.expt.harness --exec-run_test_configurations2

Example

>>> # SLOW_DOCTEST
>>> from ibeis.expt.harness import *  # NOQA
>>> import ibeis
>>> ibs = ibeis.opendb(defaultdb='PZ_MTEST')
>>> default_acfgstrs = ['controlled:qsize=20,dpername=1,dsize=10', 'controlled:qsize=20,dpername=10,dsize=100']
>>> acfg_name_list = ut.get_argval(('--aidcfg', '--acfg', '-a'), type_=list, default=default_acfgstrs)
>>> test_cfg_name_list = ut.get_argval(('-t', '-p')), type_=list, default=['custom', 'custom:fg_on=False'])
>>> use_cache = False
>>> testres_list = run_test_configurations2(ibs, acfg_name_list, test_cfg_name_list, use_cache)

ibeis.expt.old_expt_configs module¶

ibeis.expt.old_storage module¶

class ibeis.expt.old_storage.ResultMetadata(metadata, fpath, autoconnect=False)[source]¶

Bases: object

clear(metadata)[source]¶

close(metadata)[source]¶

connect(metadata)[source]¶

get_cfgstr_list(metadata)[source]¶

get_column_keys(metadata)[source]¶

get_square_data(metadata, cfgstr=None)[source]¶

rrr(verbose=True)¶

special class reloading function

set_global_data(metadata, cfgstr, qaid, key, val)[source]¶

sync_test_results(metadata, testres)[source]¶

store all test results in the shelf

write(metadata)[source]¶

ibeis.expt.old_storage.draw_results(ibs, testres)[source]¶

Draws results from an experiment harness run. Rows store different qaids (query annotation ids) Cols store different configurations (algorithm parameters)

Parameters:	testres (TestResult) –

CommandLine:

python dev.py -t custom:rrvsone_on=True,constrained_coeff=0 custom --qaid 12 --db PZ_MTEST --show --va
python dev.py -t custom:rrvsone_on=True,constrained_coeff=.3 custom --qaid 12 --db PZ_MTEST --show --va --noqcache
python dev.py -t custom:rrvsone_on=True custom --qaid 4 --db PZ_MTEST --show --va --noqcache
python dev.py -t custom:rrvsone_on=True,grid_scale_factor=1 custom --qaid 12 --db PZ_MTEST --show --va --noqcache
python dev.py -t custom:rrvsone_on=True,grid_scale_factor=1,grid_steps=1 custom --qaid 12 --db PZ_MTEST --show --va --noqcache

CommandLine:

python dev.py -t best --db seals2 --allgt --vz --fig-dname query_analysis_easy --show
python dev.py -t best --db seals2 --allgt --vh --fig-dname query_analysis_hard --show

python dev.py -t pyrscale --db PZ_MTEST --allgt --vn --fig-dname query_analysis_interesting --show
python dev.py -t pyrscale --db testdb3 --allgt --vn --fig-dname query_analysis_interesting --vf
python dev.py -t pyrscale --db testdb3 --allgt --vn --fig-dname query_analysis_interesting --vf --quality

python -m ibeis.expt.experiment_drawing --test-draw_results --show --vn
python -m ibeis.expt.experiment_drawing --test-draw_results --show --vn --db PZ_MTEST
python -m ibeis.expt.old_storage --test-draw_results --show --db PZ_MTEST --gv

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.old_storage import *  # NOQA
>>> from ibeis.init import main_helpers
>>> ibs, testres = main_helpers.testdata_expts('PZ_MTEST')
>>> result = draw_results(ibs, testres)
>>> # verify results
>>> print(result)

ibeis.expt.old_storage.make_metadata_custom_api(metadata)[source]¶

CommandLine:

python -m ibeis.expt.experiment_drawing --test-make_metadata_custom_api --show

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.experiment_drawing import *  # NOQA
>>> import guitool
>>> guitool.ensure_qapp()
>>> metadata_fpath = '/media/raid/work/Elephants_drop1_ears/_ibsdb/figures/result_metadata.shelf'
>>> metadata = ResultMetadata(metadata_fpath, autoconnect=True)
>>> wgt = make_metadata_custom_api(metadata)
>>> ut.quit_if_noshow()
>>> wgt.show()
>>> wgt.raise_()
>>> guitool.qtapp_loop(wgt, frequency=100)

ibeis.expt.old_storage.make_test_result_custom_api(ibs, testres)[source]¶

ibeis.expt.precomputer module¶

ibeis.expt.precomputer.precfg(ibs, acfg_name_list, test_cfg_name_list)[source]¶

Helper to precompute information

Parameters:	ibs (IBEISController) – ibeis controller object qaids (list) – query annotation ids daids (list) – database annotation ids test_cfg_name_list (list) –

CommandLine:

python -m ibeis.expt.precomputer --exec-precfg -t custom --expt-preload

python -m ibeis.expt.precomputer --exec-precfg -t candidacy -a default:qaids=allgt --preload
python -m ibeis.expt.precomputer --exec-precfg -t candidacy_invariance -a default:qaids=allgt --preload

python -m ibeis.expt.precomputer --exec-precfg --delete-nn-cache

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.precomputer import *  # NOQA
>>> import ibeis
>>> ibs = ibeis.opendb(defaultdb='PZ_MTEST')
>>> default_acfgstrs = ['default:qaids=allgt']
>>> acfg_name_list = ut.get_argval(('--aidcfg', '--acfg', '-a'), type_=list, default=default_acfgstrs)
>>> test_cfg_name_list = ut.get_argval('-t', type_=list, default=['custom'])
>>> result = precfg(ibs, acfg_name_list, test_cfg_name_list)
>>> print(result)

ibeis.expt.precomputer.precfg_dbs(db_list)[source]¶

Helper to precompute information Runs precfg on multiple databases

Parameters:	db_list (list) –

CommandLine:

python -m ibeis.expt.precomputer --exec-precfg_dbs
python -m ibeis.expt.precomputer --exec-precfg_dbs --dblist testdb1 PZ_MTEST
python -m ibeis.expt.precomputer --exec-precfg_dbs --dblist testdb1 PZ_MTEST --preload -t custom
python -m ibeis.expt.precomputer --exec-precfg_dbs --dblist=PZ_MTEST,NNP_MasterGIRM_core,PZ_Master0,NNP_Master3,GZ_ALL,PZ_FlankHack --preload --delete-nn-cache

#python -m ibeis.expt.precomputer --exec-precfg_dbs --dblist=PZ_Master0 -t candidacy1 --preload-chip --controlled --species=primary
python -m ibeis.expt.precomputer --exec-precfg_dbs --dblist=candidacy --preload

python -m ibeis.expt.precomputer --exec-precfg_dbs --dblist=candidacy -t candidacy --preload-chip --species=primary --controlled
python -m ibeis.expt.precomputer --exec-precfg_dbs --dblist=candidacy -t candidacy --preload-chip --species=primary --allgt
python -m ibeis.expt.precomputer --exec-precfg_dbs --dblist=candidacy -t candidacy --preload-feat
python -m ibeis.expt.precomputer --exec-precfg_dbs --dblist=candidacy -t candidacy --preload-featweight
python -m ibeis.expt.precomputer --exec-precfg_dbs --dblist=candidacy -t candidacy --preload
python -m ibeis.expt.precomputer --exec-precfg_dbs --dblist=candidacy --delete-nn-cache

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.precomputer import *  # NOQA
>>> db_list = ut.get_argval('--dblist', type_=list, default=['testdb1'])
>>> result = precfg_dbs(db_list)
>>> print(result)

ibeis.expt.test_result module¶

class ibeis.expt.test_result.TestResult(testres, cfg_list, cfgx2_lbl, cfgx2_cfgresinfo, cfgx2_qreq_)[source]¶

Bases: object

case_sample2(testres, filt_cfg, qaids=None, return_mask=False, verbose=None)[source]¶

Filters individual test result cases based on how they performed, what tags they had, and various other things.

Parameters:	filt_cfg (dict) –
Returns:	case_pos_list (list of (qx, cfgx)) or isvalid mask
Return type:	list

CommandLine:

python -m ibeis --tf TestResult.case_sample2
python -m ibeis --tf TestResult.case_sample2:0
python -m ibeis --tf TestResult.case_sample2:1 --db GZ_ALL --filt :min_tags=1
python -m ibeis --tf TestResult.case_sample2:1 --db PZ_Master1 --filt :min_gf_tags=1

python -m ibeis --tf TestResult.case_sample2:2 --db PZ_Master1

Example0:

>>> # ENABLE_DOCTEST
>>> # The same results is achievable with different filter config settings
>>> from ibeis.expt.test_result import *  # NOQA
>>> from ibeis.init import main_helpers
>>> verbose = True
>>> ibs, testres = main_helpers.testdata_expts('PZ_MTEST', a=['ctrl'])
>>> filt_cfg1 = {'fail': True}
>>> case_pos_list1 = testres.case_sample2(filt_cfg1)
>>> filt_cfg2 = {'min_gtrank': 1}
>>> case_pos_list2 = testres.case_sample2(filt_cfg2)
>>> filt_cfg3 = {'min_gtrank': 0}
>>> case_pos_list3 = testres.case_sample2(filt_cfg3)
>>> filt_cfg4 = {}
>>> case_pos_list4 = testres.case_sample2(filt_cfg4)
>>> assert np.all(case_pos_list1 == case_pos_list2), 'should be equiv configs'
>>> assert np.any(case_pos_list2 != case_pos_list3), 'should be diff configs'
>>> assert np.all(case_pos_list3 == case_pos_list4), 'should be equiv configs'
>>> ibs, testres = main_helpers.testdata_expts('PZ_MTEST', a=['ctrl'], t=['default:sv_on=[True,False]'])
>>> filt_cfg5 = filt_cfg1.copy()
>>> mask5 = testres.case_sample2(filt_cfg5, return_mask=True)
>>> case_pos_list5 = testres.case_sample2(filt_cfg5, return_mask=False)
>>> assert len(mask5.shape) == 2
>>> assert not np.all(mask5.T[0] == mask5.T[1])
>>> filt_cfg6 = {'fail': True, 'allcfg': True}
>>> mask6 = testres.case_sample2(filt_cfg6, return_mask=True)
>>> assert np.all(mask6.T[0] == mask6.T[1])
>>> print(mask5)
>>> print(case_pos_list5)
>>> filt_cfg = filt_cfg7 = {'disagree': True}
>>> case_pos_list7 = testres.case_sample2(filt_cfg7, verbose=verbose)
>>> print(case_pos_list7)

Example1:

>>> # SCRIPT
>>> from ibeis.expt.test_result import *  # NOQA
>>> from ibeis.init import main_helpers
>>> ibs, testres = main_helpers.testdata_expts('PZ_MTEST', a=['ctrl'])
>>> filt_cfg = main_helpers.testdata_filtcfg()
>>> case_pos_list = testres.case_sample2(filt_cfg)
>>> result = ('case_pos_list = %s' % (str(case_pos_list),))
>>> print(result)
>>> # Extra stuff
>>> all_tags = testres.get_all_tags()
>>> selcted_tags = ut.take(all_tags, case_pos_list.T[0])
>>> print('selcted_tags = %r' % (selcted_tags,))

Example1:

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> from ibeis.init import main_helpers
>>> ibs, testres = main_helpers.testdata_expts('PZ_MTEST', a=['ctrl'], t=['default:K=[1,2,3]'])
>>> ut.exec_funckw(testres.case_sample2, globals())
>>> filt_cfg = {'fail': True, 'min_gtrank': 1, 'max_gtrank': None, 'min_gf_timedelta': '24h'}
>>> ibs, testres = main_helpers.testdata_expts('humpbacks_fb', a=['default:has_any=hasnotch,mingt=2,qindex=0:300,dindex=0:300'], t=['default:proot=BC_DTW,decision=max,crop_dim_size=500,crop_enabled=True,manual_extract=False,use_te_scorer=True,ignore_notch=True,te_net=annot_simple', 'default:proot=vsmany'], qaid_override=[12])
>>> filt_cfg = ':disagree=True,index=0:8,min_gtscore=.00001,require_all_cfg=True'
>>> #filt_cfg = cfghelpers.parse_argv_cfg('--filt')[0]
>>> case_pos_list = testres.case_sample2(filt_cfg, verbose=True)
>>> result = ('case_pos_list = %s' % (str(case_pos_list),))
>>> print(result)
>>> # Extra stuff
>>> all_tags = testres.get_all_tags()
>>> selcted_tags = ut.take(all_tags, case_pos_list.T[0])
>>> print('selcted_tags = %r' % (selcted_tags,))

print(‘qaid = %r’ % (qaid,)) print(‘qx = %r’ % (qx,)) print(‘cfgxs = %r’ % (cfgxs,)) # print testres info about this item take_cfgs = ut.partial(ut.take, index_list=cfgxs) take_qx = ut.partial(ut.take, index_list=qx) truth_cfgs = ut.hmap_vals(take_qx, truth2_prop) truth_item = ut.hmap_vals(take_cfgs, truth_cfgs, max_depth=1) prop_cfgs = ut.hmap_vals(take_qx, prop2_mat) prop_item = ut.hmap_vals(take_cfgs, prop_cfgs, max_depth=0) print(‘truth2_prop[item] = ‘ + ut.repr3(truth_item, nl=2)) print(‘prop2_mat[item] = ‘ + ut.repr3(prop_item, nl=1))

cfgx2_daids¶

cfgx2_qaids¶

compare_score_pdfs(testres)[source]¶

CommandLine:

python -m ibeis.expt.test_result --exec-compare_score_pdfs --show --present
python -m ibeis.expt.test_result --exec-compare_score_pdfs --show --present --nocache
python -m ibeis.expt.test_result --exec-compare_score_pdfs --show --present -a timectrl:qindex=0:50

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> import ibeis
>>> defaultdb = 'PZ_MTEST'
>>> defaultdb = 'PZ_Master1'
>>> ibs, testres = ibeis.testdata_expts(
>>>     defaultdb=defaultdb, a=['timectrl'], t=['best'])
>>> testres.compare_score_pdfs()
>>> ut.quit_if_noshow()
>>> import plottool as pt
>>> ut.show_if_requested()

draw_annot_scoresep(testres, f=None)[source]¶

draw_failure_cases(testres, **kwargs)[source]¶

>>> from ibeis.other.dbinfo import *  # NOQA
>>> import ibeis
>>> ibs, testres = ibeis.testdata_expts(defaultdb='PZ_MTEST', a='timectrl:qsize=2', t='invar:AI=[False],RI=False', use_cache=False)

draw_feat_scoresep(testres, f=None, disttype=None)[source]¶

SeeAlso:

ibeis.algo.hots.scorenorm.train_featscore_normalizer

CommandLine:

python -m ibeis --tf TestResult.draw_feat_scoresep --show
python -m ibeis --tf TestResult.draw_feat_scoresep --show -t default:sv_on=[True,False]
python -m ibeis --tf TestResult.draw_feat_scoresep --show --db PZ_Master1
python -m ibeis --tf TestResult.draw_feat_scoresep --show --db PZ_Master1 --disttype=L2_sift,fg
python -m ibeis --tf TestResult.draw_feat_scoresep --show --db PZ_Master1 --disttype=L2_sift
python -m ibeis --tf TestResult.draw_feat_scoresep --show --db PZ_MTEST -t best:lnbnn_on=True --namemode=True
python -m ibeis --tf TestResult.draw_feat_scoresep --show --db PZ_MTEST -t best:lnbnn_on=True --namemode=False

python -m ibeis --tf TestResult.draw_feat_scoresep --show --db PZ_MTEST --disttype=L2_sift
python -m ibeis --tf TestResult.draw_feat_scoresep --show --db PZ_MTEST --disttype=L2_sift -t best:SV=False

utprof.py -m ibeis --tf TestResult.draw_feat_scoresep --show --db PZ_Master1
utprof.py -m ibeis --tf TestResult.draw_feat_scoresep --show --db PZ_Master1 --fsvx=1:2
utprof.py -m ibeis --tf TestResult.draw_feat_scoresep --show --db PZ_Master1 --fsvx=0:1

utprof.py -m ibeis --tf TestResult.draw_feat_scoresep --show --db PZ_Master1 -t best:lnbnn_on=False,bar_l2_on=True  --fsvx=0:1

# We want to query the oxford annots taged query
# and we want the database to contain
# K correct images per query, as well as the distractors

python -m ibeis --tf TestResult.draw_feat_scoresep  --show --db Oxford -a default:qhas_any=\(query,\),dpername=1,exclude_reference=True,minqual=ok
python -m ibeis --tf TestResult.draw_feat_scoresep  --show --db Oxford -a default:qhas_any=\(query,\),dpername=1,exclude_reference=True,minqual=good

python -m ibeis --tf get_annotcfg_list  --db PZ_Master1 -a timectrl --acfginfo --verbtd  --veryverbtd --nocache-aid

python -m ibeis --tf TestResult.draw_feat_scoresep --show --db PZ_MTEST --disttype=ratio

Example

>>> # SCRIPT
>>> from ibeis.expt.test_result import *  # NOQA
>>> from ibeis.init import main_helpers
>>> disttype = ut.get_argval('--disttype', type_=list, default=None)
>>> ibs, testres = main_helpers.testdata_expts(
>>>     defaultdb='PZ_MTEST', a=['timectrl'], t=['best'])
>>> f = ut.get_argval(('--filt', '-f'), type_=list, default=[''])
>>> testres.draw_feat_scoresep(f=f)
>>> ut.show_if_requested()

draw_match_cases(testres, **kwargs)[source]¶

Wrapper

draw_rank_cdf(testres)[source]¶

Wrapper

draw_score_diff_disti(testres)[source]¶

CommandLine:

python -m ibeis --tf TestResult.draw_score_diff_disti --show -a varynannots_td -t best --db PZ_Master1
python -m ibeis --tf TestResult.draw_score_diff_disti --show -a varynannots_td -t best --db GZ_Master1
python -m ibeis --tf TestResult.draw_score_diff_disti --show -a varynannots_td1h -t best --db GIRM_Master1

python -m ibeis --tf TestResult.draw_score_diff_disti --show -a varynannots_td:qmin_pername=3,dpername=2 -t best --db PZ_Master1

python -m ibeis --tf get_annotcfg_list -a varynannots_td -t best --db PZ_Master1
13502
python -m ibeis --tf draw_match_cases --db PZ_Master1 -a varynannots_td:dsample_size=.01 -t best  --show --qaid 13502
python -m ibeis --tf draw_match_cases --db PZ_Master1 -a varynannots_td -t best  --show

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> import ibeis
>>> ibs, testres = ibeis.testdata_expts('PZ_Master1', a=['varynannots_td'], t=['best'])
>>> result = testres.draw_score_diff_disti()
>>> print(result)
>>> ut.show_if_requested()

find_score_thresh_cutoff(testres)[source]¶

FIXME DUPLICATE CODE rectify with experiment_drawing

get_X_LIST(testres)[source]¶

get_all_qaids(testres)[source]¶

get_all_tags(testres)[source]¶

CommandLine:

python -m ibeis --tf TestResult.get_all_tags --db PZ_Master1 --show --filt :
python -m ibeis --tf TestResult.get_all_tags --db PZ_Master1 --show --filt :min_gf_timedelta=24h
python -m ibeis --tf TestResult.get_all_tags --db PZ_Master1 --show --filt :min_gf_timedelta=24h,max_gt_rank=5

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> from ibeis.init import main_helpers
>>> ibs, testres = main_helpers.testdata_expts('PZ_Master1', a=['timectrl'])
>>> filt_cfg = main_helpers.testdata_filtcfg()
>>> case_pos_list = testres.case_sample2(filt_cfg)
>>> all_tags = testres.get_all_tags()
>>> selected_tags = ut.take(all_tags, case_pos_list.T[0])
>>> flat_tags = list(map(str, ut.flatten(ut.flatten(selected_tags))))
>>> print(ut.dict_str(ut.dict_hist(flat_tags), key_order_metric='val'))
>>> ut.quit_if_noshow()
>>> import plottool as pt
>>> pt.word_histogram2(flat_tags, fnum=1, pnum=(1, 2, 1))
>>> pt.wordcloud(' '.join(flat_tags), fnum=1, pnum=(1, 2, 2))
>>> pt.set_figtitle(ut.get_cfg_lbl(filt_cfg))
>>> ut.show_if_requested()

get_all_varied_params(testres)[source]¶

Returns the parameters that were varied between different configurations in this test

Returns:	varied_params
Return type:	list

CommandLine:

python -m ibeis --tf -get_all_varied_params

Example

>>> # ENABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> import ibeis
>>> testres = ibeis.testdata_expts('PZ_MTEST', t='default:K=[1,2]')[1]
>>> varied_params = testres.get_all_varied_params()
>>> result = ('varied_params = %s' % (ut.repr2(varied_params),))
>>> print(result)
varied_params = ['K', '_cfgindex']

get_annotcfg_args(testres)[source]¶

CommandLine:

# TODO: More robust fix
# To reproduce the error
ibeis -e rank_cdf --db humpbacks_fb -a default:mingt=2,qsize=10,dsize=100 default:qmingt=2,qsize=10,dsize=100 -t default:proot=BC_DTW,decision=max,crop_dim_size=500,crop_enabled=True,manual_extract=False,use_te_scorer=True,ignore_notch=True,te_score_weight=0.5 --show

get_cfgstr(testres, cfgx)[source]¶

just dannots and config_str

get_cfgx_groupxs(testres)[source]¶

Groupxs configurations specified to be joined

Example

>>> # ENABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> from ibeis.init import main_helpers
>>> #ibs, testres = main_helpers.testdata_expts('testdb1')
>>> ibs, testres = main_helpers.testdata_expts(
>>>    'PZ_MTEST',
>>>     a=['default:qnum_names=1,qname_offset=[0,1],joinme=1,dpername=1',  'default:qsize=1,dpername=[1,2]'],
>>>     t=['default:K=[1,2]'])
>>> groupxs = testres.get_cfgx_groupxs()
>>> result = groupxs
>>> print(result)
[[7], [6], [5], [4], [0, 1, 2, 3]]

get_cfgx_with_param(testres, key, val)[source]¶

Gets configs where the given parameter is held constant

get_common_qaids(testres)[source]¶

get_fname_aug(testres, **kwargs)[source]¶

get_full_cfgstr(testres, cfgx)[source]¶

both qannots and dannots included

get_gf_tags(testres)[source]¶

Returns:	case_pos_list
Return type:	list

CommandLine:

python -m ibeis --tf TestResult.get_gf_tags --db PZ_Master1 --show

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> from ibeis.init import main_helpers
>>> ibs, testres = main_helpers.testdata_expts('PZ_Master1', a=['timectrl'])
>>> filt_cfg = main_helpers.testdata_filtcfg()
>>> case_pos_list = testres.case_sample2(filt_cfg)
>>> gf_tags = testres.get_gf_tags()

get_gt_annot_tags(testres)[source]¶

get_gt_tags(testres)[source]¶

get_gtquery_annot_tags(testres)[source]¶

get_infoprop_list(testres, key, qaids=None)[source]¶

key = ‘qx2_bestranks’ key = ‘qx2_gt_rank’ qaids = testres.get_test_qaids()

get_infoprop_mat(testres, key, qaids=None)[source]¶

key = ‘qx2_gf_raw_score’ key = ‘qx2_gt_raw_score’

get_nLessX_dict(testres)[source]¶

get_param_basis(testres, key)[source]¶

Returns what a param was varied between over all tests key = ‘K’ key = ‘dcfg_sample_size’

get_param_val_from_cfgx(testres, cfgx, key)[source]¶

get_pipecfg_args(testres)[source]¶

get_query_annot_tags(testres)[source]¶

get_rank_cumhist(testres, bins=u'dense')[source]¶

get_rank_histogram_bin_edges(testres)[source]¶

get_rank_histogram_bins(testres)[source]¶

easy to see histogram bins

get_rank_histogram_qx_binxs(testres)[source]¶

get_rank_histogram_qx_sample(testres, size=10)[source]¶

get_rank_histograms(testres, bins=None, asdict=True, jagged=False, key=None)[source]¶

get_rank_mat(testres, qaids=None)[source]¶

get_rank_percentage_cumhist(testres, bins=u'dense', key=None, join_acfgs=False)[source]¶

Parameters:	bins (unicode) – (default = u’dense’)
Returns:	(config_cdfs, edges)
Return type:	tuple

CommandLine:

python -m ibeis --tf TestResult.get_rank_percentage_cumhist
python -m ibeis --tf TestResult.get_rank_percentage_cumhist -t baseline -a uncontrolled ctrl

python -m ibeis --tf TestResult.get_rank_percentage_cumhist \
    --db lynx \
    -a default:qsame_imageset=True,been_adjusted=True,excluderef=True \
    -t default:K=1 --show --cmd

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> from ibeis.init import main_helpers
>>> #ibs, testres = main_helpers.testdata_expts('testdb1')
>>> ibs, testres = main_helpers.testdata_expts('testdb1', a=['default:num_names=1,name_offset=[0,1]'])
>>> bins = u'dense'
>>> key = None
>>> (config_cdfs, edges) = testres.get_rank_percentage_cumhist(bins)
>>> result = ('(config_cdfs, edges) = %s' % (str((config_cdfs, edges)),))
>>> print(result)

get_short_cfglbls(testres, join_acfgs=False)[source]¶

Labels for published tables

cfg_lbls = [‘baseline:nRR=200+default:’, ‘baseline:+default:’]

CommandLine:

python -m ibeis --tf TestResult.get_short_cfglbls

Example

>>> # SLOW_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> import ibeis
>>> #ibs, testres = ibeis.testdata_expts('PZ_MTEST', a=['unctrl', 'ctrl::unctrl_comp'], t=['default:dim_size:[450,550]'])
>>> ibs, testres = ibeis.testdata_expts('PZ_MTEST', a=['ctrl:size=10'], t=['default:dim_size=[450,550]'])
>>> #ibs, testres = ibeis.testdata_expts('PZ_MTEST', a=['ctrl:size=10'], t=['default:dim_size=450', 'default:dim_size=550'])
>>> cfg_lbls = testres.get_short_cfglbls()
>>> result = ('cfg_lbls = %s' % (ut.list_str(cfg_lbls),))
>>> print(result)
cfg_lbls = [
    'default:dim_size=450+ctrl',
    'default:dim_size=550+ctrl',
]

get_sorted_config_labels(testres)[source]¶

helper

get_test_qaids(testres)[source]¶

get_title_aug(testres, with_size=True, with_db=True, with_cfg=True, friendly=False)[source]¶

Parameters:	with_size (bool) – (default = True)
Returns:	title_aug
Return type:	str

CommandLine:

python -m ibeis --tf TestResult.get_title_aug --db PZ_Master1 -a timequalctrl::timectrl

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> import ibeis
>>> ibs, testres = ibeis.testdata_expts('PZ_MTEST')
>>> with_size = True
>>> title_aug = testres.get_title_aug(with_size)
>>> res = u'title_aug = %s' % (title_aug,)
>>> print(res)

get_total_num_varied_params(testres)[source]¶

get_truth2_prop(testres, qaids=None, join_acfg=False)[source]¶

Returns:	(truth2_prop, prop2_mat)
Return type:	tuple

CommandLine:

python -m ibeis.expt.test_result --exec-get_truth2_prop --show

Example

>>> # ENABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> import ibeis
>>> ibs, testres = ibeis.testdata_expts('PZ_MTEST', a=['ctrl'])
>>> (truth2_prop, prop2_mat) = testres.get_truth2_prop()
>>> result = '(truth2_prop, prop2_mat) = %s' % str((truth2_prop, prop2_mat))
>>> print(result)
>>> ut.quit_if_noshow()
>>> import plottool as pt
>>> ut.show_if_requested()

get_varied_labels(testres, shorten=False, join_acfgs=False)[source]¶

Returns labels indicating only the parameters that have been varied between different annot/pipeline configurations.

Helper for consistent figure titles

CommandLine:

python -m ibeis --tf TestResult.make_figtitle  --prefix "Seperability " --db GIRM_Master1   -a timectrl -t Ell:K=2     --hargv=scores
python -m ibeis --tf TestResult.make_figtitle

Example

>>> # SLOW_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> import ibeis
>>> ibs, testres = ibeis.testdata_expts('PZ_MTEST', t='default:K=[1,2]', a='timectrl:qsize=[1,2],dsize=3')
>>> varied_lbls = testres.get_varied_labels()
>>> result = ('varied_lbls = %r' % (varied_lbls,))
>>> print(result)
varied_lbls = [u'K=1+qsize=1', u'K=2+qsize=1', u'K=1+qsize=2', u'K=2+qsize=2']

get_worst_possible_rank(testres)[source]¶

has_constant_daids(testres)[source]¶

has_constant_length_daids(testres)[source]¶

has_constant_length_qaids(testres)[source]¶

has_constant_qaids(testres)[source]¶

ibs¶

interact_individual_result(testres, qaid, cfgx=0)[source]¶

make_figtitle(testres, plotname=u'', filt_cfg=None)[source]¶

Helper for consistent figure titles

CommandLine:

python -m ibeis --tf TestResult.make_figtitle  --prefix "Seperability " --db GIRM_Master1   -a timectrl -t Ell:K=2     --hargv=scores
python -m ibeis --tf TestResult.make_figtitle

Example

>>> # ENABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> import ibeis
>>> ibs, testres = ibeis.testdata_expts('PZ_MTEST')
>>> plotname = ''
>>> figtitle = testres.make_figtitle(plotname)
>>> result = ('figtitle = %r' % (figtitle,))
>>> print(result)

nConfig¶

nQuery¶

print_acfg_info(testres, **kwargs)[source]¶

Prints verbose information about the annotations used in each test configuration

CommandLine:

python -m ibeis --tf TestResult.print_acfg_info

Kwargs;

see ibs.get_annot_stats_dict hashid, per_name, per_qual, per_vp, per_name_vpedge, per_image, min_name_hourdist

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> import ibeis
>>> ibs, testres = ibeis.testdata_expts('PZ_MTEST', a=['ctrl::unctrl_comp'], t=['candk:K=[1,2]'])
>>> ibs = None
>>> result = testres.print_acfg_info()
>>> print(result)

print_config_overlap(testres, with_plot=True)[source]¶

print_pcfg_info(testres)[source]¶

Prints verbose information about each pipeline configuration

>>> from ibeis.expt.test_result import *  # NOQA

print_percent_identification_success(testres)[source]¶

Prints names identified (at rank 1) / names queried. This combines results over multiple queries of a particular name using max

OLD, MAYBE DEPRIATE

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA

print_results(testres)[source]¶

CommandLine:

python -m ibeis --tf TestResult.print_results

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> from ibeis.expt import harness
>>> ibs, testres = harness.testdata_expts('PZ_MTEST')
>>> result = testres.print_results()
>>> print(result)

print_unique_annot_config_stats(testres, ibs=None)[source]¶

Parameters:	ibs (IBEISController) – ibeis controller object(default = None)

CommandLine:

python -m ibeis --tf TestResult.print_unique_annot_config_stats

Example

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> import ibeis
>>> testres = ibeis.testdata_expts('PZ_MTEST', a=['ctrl::unctrl_comp'])
>>> ibs = None
>>> result = testres.print_unique_annot_config_stats(ibs)
>>> print(result)

qaids¶

rank_mat¶

reconstruct_test_flags(testres)[source]¶

rrr(verbose=True)¶

special class reloading function

ibeis.expt.test_result.combine_testres_list(ibs, testres_list)[source]¶

combine test results over multiple annot configs

CommandLine:

python -m ibeis --tf combine_testres_list

python -m ibeis --tf -draw_rank_cdf --db PZ_MTEST --show
python -m ibeis --tf -draw_rank_cdf --db PZ_Master1 --show
python -m ibeis --tf -draw_rank_cdf --db PZ_MTEST --show -a varysize -t default
python -m ibeis --tf -draw_rank_cdf --db PZ_MTEST --show -a varysize -t default

>>> # DISABLE_DOCTEST
>>> from ibeis.expt.test_result import *  # NOQA
>>> from ibeis.expt import harness
>>> ibs, testres_list = harness.testdata_expts('PZ_MTEST', ['varysize'])
>>> combine_testres_list(ibs, testres_list)

Module contents¶

ibeis.expt.reassign_submodule_attributes(verbose=True)[source]¶

why reloading all the modules doesnt do this I don’t know

ibeis.expt.reload_subs(verbose=True)[source]¶

Reloads ibeis.expt and submodules

ibeis.expt.rrrr(verbose=True)¶

Reloads ibeis.expt and submodules